Senior Data Engineer

Position Overview

We are looking for a seasoned Senior Data Engineer to design, develop, and maintain scalable data pipelines and solutions within an enterprise context. This role demands expertise in Python, SQL, cloud data platforms, and modern data technologies, with a focus on handling large and intricate datasets.

Key Responsibilities

• Architect, build, and sustain scalable ETL/ELT data pipelines to support business needs.
• Develop data workflows and solutions using Python and SQL effectively.
• Optimize data processing and transformation procedures for efficiency and accuracy.
• Leverage cloud-based platforms and services for data management and processing.
• Design and upkeep data models, warehouses, and lakes to ensure robust data infrastructure.
• Support both batch and real-time data processing pipelines.
• Implement rigorous data quality controls, monitoring systems, and error handling mechanisms.
• Enhance performance of data pipelines and queries to ensure reliability.
• Collaborate closely with Data Architects, Analysts, Developers, and other key stakeholders.
• Provide support for production issues, conduct troubleshooting, and drive continuous pipeline improvements.

Required Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or related field.
• Over 8 years of experience in Data Engineering or similar data-focused roles.
• Proven track record working with enterprise-level data platforms and applications.
• Experience operating within Agile and collaborative technical teams.

Essential Technical Skills

• Hands-on expertise in Python programming.
• Advanced SQL skills including query optimization and thorough data manipulation.
• Development of ETL/ELT data pipelines.
• Strong background in data modeling, warehousing, and maintaining data lakes.
• Experience with cloud platforms such as AWS or Azure.
• Proficiency with Apache Spark, PySpark, or equivalent distributed data processing frameworks.
• Familiarity with data orchestration tools including Airflow, Azure Data Factory, or similar.
• Working knowledge of Git and continuous integration/continuous deployment (CI/CD) pipelines.
• Understanding of principles around data quality, governance, and security.

Additional Beneficial Skills

• Experience with Databricks or Snowflake cloud data platforms.
• Knowledge of Kafka or other streaming data technologies.
• Exposure to Terraform or infrastructure-as-code tools.
• Familiarity with analytics tools like Power BI or Tableau.
• Relevant cloud or data engineering certifications.
• Background in banking, financial services, or other large enterprise sectors is advantageous.