Data Engineer

Overview

Embark on a professional journey as a Data Engineer specializing in Azure technologies, contributing to the creation and upkeep of resilient data architectures within the aviation industry.

Key Responsibilities

• Architect, develop, and sustain scalable ETL/ELT data pipelines designed for processing diverse datasets.
• Extract and convert both structured and unstructured data from multiple sources including APIs, databases, files, and third-party systems.
• Construct and manage data models and refined datasets aimed at supporting analytics and artificial intelligence applications.
• Establish mechanisms for data quality assurance, validation, ongoing monitoring, and fault tolerance.
• Enhance pipeline performance to ensure scalability, reliability, and cost efficiency.
• Develop batch and, as necessary, real-time streaming workflows for data processing.
• Utilize Azure data services alongside data lake and storage infrastructure to build robust solutions.
• Create and maintain datasets to be employed in Large Language Models (LLM), Retrieval-Augmented Generation (RAG), evaluation tools, and autonomous agent-based applications.
• Apply data lineage tracking, metadata management, and governance standards.
• Design and implement AI-driven services and APIs utilizing Python 3.12+ and FastAPI frameworks.

Qualifications

• A university degree in Computer Science, Engineering, or a related technical discipline.
• Solid expertise with Azure Data Factory, Azure Databricks, Azure SQL Database, and other relevant Azure cloud components.
• Demonstrated experience in designing data models, warehousing solutions, and managing ETL workflows.
• Advanced programming proficiency in Python, SQL, and Spark technologies.
• Familiarity with LangGraph technology and retrieval-based systems such as RAG.
• Hands-on experience with the Azure data ecosystem and data lake platforms.
• Knowledge of data orchestration and processing frameworks.
• Industry-specific insight into airline and irregular operations (IRREG) domains.
• Awareness of the EU261 regulations and airline fare rules.
• Proven track record as a Data Engineer focused on Azure data technologies.
• Proficient communication skills in English.