Azure Data Engineer

Role Overview and Responsibilities

This role involves designing, creating, and managing data pipelines and Extract, Transform, Load (ETL) workflows utilizing Azure Data Factory and Databricks platforms. The engineer will facilitate the integration of data from diverse sources into Azure Data Lake and Synapse Analytics environments.

• Design, implement, and sustain data pipelines and ETL processes using Azure Data Factory and Azure Databricks.
• Integrate data from multiple origins into Azure Data Lake Storage and Synapse to support analytics and operations.
• Collaborate with senior team members to develop and refine data transformation scripts primarily written in Python or PySpark.
• Take part in projects focused on migrating data systems and modernizing infrastructure to Azure cloud services.
• Establish robust data validation, implement error handling, and maintain logging practices within pipelines.
• Apply fundamental security setups employing Azure Key Vault and role-based access control to safeguard data.
• Prepare detailed documentation covering pipeline architecture, configuration settings, and operation guidelines.
• Assist with DevOps practices including deployment automation, continuous integration/continuous delivery (CI/CD) processes, and version control management.
• Diagnose and fix issues related to data flow and pipeline functionality during development and testing phases.
• Engage with business analysts, system architects, and quality assurance teams to ensure comprehensive and accurate data delivery solutions.
• Pursue ongoing education in Azure cloud technologies and tools to enhance solution performance and adaptability.

Candidate Profile and Eligibility

Applicants should be graduates in any discipline and possess a keen interest in data engineering and cloud technologies, particularly within the Azure environment.