Senior Data Engineer

Overview

The Senior Data Engineer role entails designing, building, and maintaining efficient, secure, and scalable data platforms. This position plays a vital role in developing critical data pipelines and architectures that support analytics, reporting, machine learning applications, and real-time data insights.

Key Responsibilities

• Develop and operate high-performance data platforms and pipelines using Azure Data Lake Gen2, Synapse, and Databricks.
• Optimize PySpark and Spark SQL jobs for performance and cost efficiency within Databricks environments.
• Utilize advanced SQL skills, including window functions, common table expressions (CTEs), and query execution plan analysis, to support complex data transformations.
• Mentor junior and mid-level engineers, providing guidance on best practices and technology trade-offs.
• Work collaboratively with architects, product teams, analytics, data science, and business units to deliver impactful data solutions.
• Engage in building data platforms that power advanced analytics and real-time insights.

Desired Skills and Qualifications

• Proven experience developing production-grade pipelines with Azure Data Lake Gen2, Synapse, and Databricks.
• Expertise in PySpark and Spark SQL, including performance tuning and optimization.
• Strong SQL knowledge involving complex queries with window functions and CTEs.
• Experience with version control and continuous integration systems such as Azure DevOps or GitHub Actions.
• Mentorship abilities to guide less experienced engineers effectively.
• Nice to have: Knowledge of real-time streaming and low-latency pipeline design.
• Exposure to data governance practices, including metadata management and data classification with sensitivity labels.
• Familiarity with advanced Python engineering techniques focusing on performance optimization.
• Awareness of analytics and business intelligence domains.
• Understanding of cloud cost management and FinOps strategies.

Additional Information

This position demands a blend of technical leadership, architectural oversight, and collaboration with cross-functional teams to ensure engineering excellence and tangible business outcomes.