Senior Data Engineer

Overview

Devsinc seeks an accomplished Senior Data Engineer with 4 to 6 years of experience to architect, develop, and maintain efficient, scalable data pipelines supporting analytics, AI/ML projects, and data products. The role demands expertise in Python, SQL, Apache Airflow, Apache Spark, and Redis to manage extensive and evolving datasets.

Key Responsibilities

• Create, implement, and sustain robust ETL/ELT pipelines for data ingestion, transformation, validation, and delivery processes.
• Develop and manage production-grade data workflows using Apache Airflow for orchestration and scheduling.
• Construct distributed data-processing tasks leveraging Apache Spark.
• Craft clean, reusable Python scripts focused on data processing, automation, and pipeline maintenance.
• Formulate and optimize complex SQL queries for data transformations, analyses, and validation checks.
• Design pipelines that effectively handle large volumes of structured, semi-structured, and unstructured data.
• Utilize Redis to enable high-speed caching and meet intensive data application needs.
• Combine data sourced through APIs, relational databases, external providers, and various files.
• Implement comprehensive data validation, monitoring, logging, error management, alert systems, and ensure pipeline observability.
• Enhance performance of pipelines, data storage, processing efficiency, and reduce infrastructure expenditure.
• Develop reusable frameworks for data ingestion and transformation instead of single-use scripts.
• Diagnose and resolve pipeline breakdowns, bottlenecks, and data quality concerns promptly.
• Partner with Business Intelligence, Product, Data Science, and Engineering teams to deliver dependable production datasets.
• Set and uphold data engineering standards, maintain detailed technical documentation, and promote best development practices.

Minimum Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Data Science, or a related field.
• 4 to 6 years of practical experience in Data Engineering or a closely related position.
• Solid experience engineering production-level ETL/ELT pipelines.
• Proficiency in Python applied to data workflows, processing, and automation.
• Advanced SQL capabilities and strong knowledge of relational database systems.
• Hands-on expertise with Apache Airflow for scheduling and orchestration in production.
• Experience in distributed data processing utilizing Apache Spark.
• Deep understanding of data modeling, transformation patterns, and pipeline architecture.
• Knowledge of Redis and caching strategies for optimizing high-performance data access.
• Proven ability handling large-scale datasets with a focus on pipeline and query performance tuning.
• Familiarity with data quality assurance, validation techniques, performance monitoring, and observability.
• Basic working knowledge of Linux environments, Git version control, containerization with Docker, and contemporary software engineering methodologies.
• Strong analytical mindset, excellent troubleshooting skills, and ability to collaborate effectively across teams.

Preferred Skills

• Experience with cloud-based data platforms and object storage on AWS, Azure, or GCP.
• Familiarity with PostgreSQL, data warehouses, or analytical database systems.
• Knowledge in processing geospatial or large-scale location-specific data sets.
• Exposure to analytical tools like DuckDB, Apache Sedona, Trino, or Presto.
• Experience working with high-volume event, mobility, transactional, or geospatial data streams.
• Understanding of CI/CD methodologies and infrastructure-as-code practices.
• History supporting data product pipelines, analytics frameworks, or AI/ML workflows.