Data Engineer

Position Overview

The Data Engineer will take charge of designing, constructing, maintaining, and refining the data infrastructure and processing pipelines that allow organizations to gather, transform, store, and provide trustworthy data for analytics, business intelligence, AI, and operational uses. This role requires collaboration with data scientists, analysts, software developers, BI teams, cloud engineers, and business stakeholders to create scalable and secure data solutions.

Key Responsibilities

• Create and implement scalable data pipelines, ETL/ELT workflows, and data ingestion processes that handle structured, semi-structured, and unstructured data from multiple sources including databases, APIs, cloud services, files, streaming platforms, and external providers.
• Develop batch and real-time data processing solutions and maintain data warehouses, lakes, lakehouses, marts, and analytical platforms.
• Design and optimize data models, schemas, and semantic layers for analytical and operational efficiency.
• Write and enhance SQL queries, stored procedures, and data transformation logic.
• Employ technologies such as Python, Java, Scala, Spark, Kafka, Airflow, dbt, or comparable tools in data solutions development.
• Implement data quality validation, monitoring, error handling, and reconciliation processes to ensure pipeline reliability and scalability.
• Monitor pipeline operations by tracking performance metrics such as data freshness, processing times, failures, and system uptime.
• Troubleshoot data pipeline failures, performance issues, data inconsistencies, and infrastructure challenges.
• Design data integration across cloud, on-premises, hybrid, and distributed environments using platforms like AWS, Azure, Google Cloud, Snowflake, Databricks, BigQuery, or Redshift.
• Implement data governance, metadata management, data lineage, access control, privacy, security, and compliance standards.
• Collaborate to prepare datasets supporting analytics, machine learning, reporting, and experiments.
• Support machine learning and AI workloads through reliable feature pipelines, training datasets, and model-serving infrastructure where appropriate.
• Utilize automation, Infrastructure as Code, CI/CD pipelines, version control, testing, deployment, and orchestration tools for efficient data engineering workflows.
• Optimize storage, compute resources, query performance, and costs within the data architecture.
• Maintain detailed technical documentation, data dictionaries, pipeline specs, architecture diagrams, and standard operating procedures.
• Contribute to disaster recovery planning, backup strategies, data retention, and business continuity efforts.
• Participate in modernization initiatives, cloud migrations, technology updates, and architectural advancements.
• Engage with security, infrastructure, DevOps, application, and business teams to enhance overall data platform capabilities.

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, Software Engineering, IT, Mathematics, Statistics, or similar fields.
• Comprehensive knowledge of data engineering practices including data architecture, ETL/ELT, data integration, modeling, and distributed processing.
• Excellent command of SQL and familiarity with relational and NoSQL databases.
• Proficient programming skills in Python, Java, Scala, or related languages.
• Experience with Apache Spark, Kafka, Airflow, dbt, Hadoop, or comparable data engineering tools.
• Understanding of data warehouse, data lake, lakehouse, data mart, and analytical data platform management.
• Familiarity with cloud platforms like AWS, Microsoft Azure, Google Cloud, Snowflake, Databricks, BigQuery, or Redshift.
• Knowledge of batch and real-time processing, event-driven architectures, APIs, and data ingestion systems.
• Expertise in data quality management, validation, observability, metadata, governance, and security protocols.
• Experience with Git, CI/CD, Docker, Kubernetes, Terraform, and Infrastructure as Code is a plus.
• Awareness of DataOps, DevOps, automation, workflow orchestration, and software engineering best practices.
• Understanding of machine learning and AI data pipeline requirements, feature engineering, and large-scale dataset handling is advantageous.
• Familiarity with generative AI, vector databases, retrieval-augmented generation, and AI data infrastructure is beneficial.
• Strong analytic, debugging, troubleshooting, optimization, and problem-solving capabilities.
• Capable of designing data systems that are scalable, reliable, maintainable, and cost-effective.
• In-depth understanding of data security, privacy, access control, encryption, compliance, and ethical data handling.
• Effective collaboration skills with interdisciplinary technical and business teams.
• Excellent skills in technical writing and communication.
• Knowledge of data monitoring and observability tools is a plus.
• Relevant certifications in cloud engineering, data engineering, or analytics are valued.
• High motivation for technical curiosity, attention to detail, responsibility, and continual learning.
• Strong dedication to staying updated on cloud data platforms, distributed computing, AI infrastructure, automation, and emerging technologies.