Databricks Engineer

Role Overview

We are seeking a skilled Databricks Engineer to design, implement, and maintain efficient and scalable data pipelines using Azure Databricks and PySpark. The role involves building ETL/ELT processes, developing data transformation logic, and collaborating closely with business and technical teams to meet data needs.

Key Responsibilities

• Design, build, and maintain scalable pipelines for processing data utilizing Azure Databricks and PySpark technologies.
• Create and optimize ETL/ELT workflows to efficiently ingest data from various source systems.
• Implement business rule logic and complex data transformations using Spark frameworks.
• Collaborate with business stakeholders, data analysts, and data architects to accurately capture data requirements and translate them into technical solutions.
• Manage and deploy Delta Lake architecture to achieve reliable, consistent, and high-performance data processing.
• Optimize Databricks components such as jobs, notebooks, clusters, and Spark code to improve runtime performance and reduce operational costs.
• Develop and maintain data ingestion mechanisms leveraging Azure Data Factory (ADF) and related integration tools.
• Monitor data pipeline production environments to promptly identify and resolve failures, ensuring continuous data availability.
• Establish and enforce data validation, quality checks, and reconciliation procedures for reliable datasets.
• Uphold data governance policies including security and compliance standards throughout all processes.
• Participate actively in code reviews and technical design sessions promoting adherence to industry best practices.
• Provide second and third-level support for critical production data pipelines and deployments.

Candidate Eligibility

Applicants must have completed any graduate degree to be considered for this position.