Senior Data Engineer - Databricks

Role Overview

We are looking for an experienced Senior Data Engineer skilled in data engineering with a strong focus on Databricks, PySpark, Scala-Spark, and advanced SQL. The position involves working on data pipeline development, migration projects to Databricks Unity Catalog, performance tuning, and orchestration of workflows with modern data tools.

Key Responsibilities

• Deliver data engineering and analytics projects with a minimum of four years of experience.
• Lead or participate in at least one migration project to Databricks from platforms like Hadoop, Teradata, Oracle, or Talend.
• Design, develop, and enhance ETL workflows and data pipelines using Databricks and Apache Spark frameworks.
• Optimize Spark performance on Databricks for efficient data processing and management.
• Formulate and validate data delivery for large-scale Big Data initiatives.
• Collaborate with multiple teams to define and implement robust data solutions that satisfy organizational needs.
• Perform complex query tuning and optimize data models to improve performance.
• Orchestrate data workflows using tools such as Databricks Workflow, Azure Data Factory, Apache Airflow, or AWS Glue.
• Ensure data governance, security, and quality standards are upheld throughout the data lifecycle.

Technical Requirements

• Advanced proficiency in PySpark and Scala-Spark development.
• Strong expertise in writing advanced SQL queries for complicated data transformations.
• Proven track record optimizing Databricks Spark performance in at least three projects.
• Experience validating data formulations and deliveries specific to Big Data projects.
• Practical experience with at least two orchestration platforms among Databricks Workflow, Azure Data Factory, Apache Airflow, and AWS Glue.

Preferred Skills and Knowledge

• Familiarity with major cloud providers like AWS, Azure, or Google Cloud Platform.
• Understanding of data governance frameworks and security best practices.
• Additional experience with other Big Data technologies and frameworks is advantageous.