Data Engineer

About the Role

We seek a seasoned Data Engineer with over eight years of professional experience, proficient in Azure cloud services, Python, PySpark, and ETL pipeline development. The candidate will be responsible for constructing and refining robust, scalable data pipelines, managing voluminous datasets, and enforcing data governance and quality protocols using Soda and Alation tools.

Key Responsibilities

• Architect, implement, and sustain scalable data pipelines employing Azure technologies, Python, and PySpark.
• Develop and enhance ETL/ELT pipelines to handle data extraction, transformation, and loading efficiently.
• Craft and fine-tune advanced SQL queries to ensure optimal database performance.
• Leverage PySpark for processing large-scale datasets effectively.
• Integrate data quality monitoring, validation, and checks utilizing the Soda platform.
• Facilitate data cataloguing, metadata oversight, data lineage tracking, and governance through Alation.
• Maintain data integrity, security, consistency, and regulatory compliance with governance standards.
• Diagnose and resolve pipeline failures while enhancing performance and system reliability.
• Work closely with data analysts, architects, and business teams to provide trustworthy and efficient data solutions.

Eligibility and Education

Applicants must hold degrees in BE, B.Tech, ME, M.Tech, MCA, or M.Sc fields.

Experience and Notice Period

The role requires a minimum of 8 years of relevant experience. Candidates should be able to join immediately or within 15 days.