- Experience
- 5–8 yrs
- Salary
- —
- Openings
- 1
- Posted
- 34 seconds ago
- Work mode
- In office
- Education
- Any graduate
- Eligibility
- Applicants must have completed any graduate degree to be eligible for this position.
- Resume
- Required to apply
Where you'll work
Sign in to tell us what does and doesn't work for you here — it sharpens every match we show you.
Job description
Role Overview
Seeking an experienced Data Engineer proficient in SQL, Python, AWS Glue, Databricks, and ETL processes to develop and maintain scalable data pipelines and integration solutions supporting analytics and business intelligence initiatives. The role involves developing cloud-based ETL/ELT workflows, optimizing data transformations, and ensuring robust and reliable data processing.
Key Responsibilities
- Develop and maintain scalable ETL/ELT workflows leveraging Python, SQL, AWS Glue, and Databricks.
- Design efficient data integration pipelines for processing structured and semi-structured datasets.
- Write advanced SQL queries, including joins, aggregations, stored procedures, and implement performance optimizations.
- Utilize Python for scripting automation, data validation, and pipeline development.
- Manage AWS Glue jobs, crawlers, and Data Catalog components for seamless data ingestion and processing.
- Construct data transformation workflows in Databricks using PySpark, Spark SQL, and Delta Lake technologies.
- Integrate diverse data sources such as databases, cloud storage, and enterprise applications.
- Implement rigorous data quality checks, error handling protocols, logging, monitoring, and data reconciliation strategies.
- Optimize ETL efficiency, improve query execution, and manage distributed data workloads.
- Collaborate with data architects, business analysts, developers, and stakeholders to gather and translate data requirements.
- Adhere to security best practices including data access control, version control, and deployment standards.
- Provide troubleshooting and support for production data pipelines ensuring data accuracy, availability, and reliability.
Required Qualifications and Experience
- Bachelor's degree in Computer Science, Information Technology, Engineering, or related discipline.
- Approximately 5-8 years of practical experience in data engineering, ETL development, and data processing.
- Strong hands-on expertise in SQL and Python programming.
- Experience working with AWS Glue and Databricks platforms.
- Solid understanding of data warehousing principles, data modeling techniques, and ETL/ELT architectures.
- Proven capability to handle large datasets and build scalable data pipelines.
- Excellent analytical, troubleshooting, and problem-solving skills.
Preferred Skills
- Advanced SQL features including complex queries, window functions, CTEs, and query tuning.
- Python skills focused on scripting, automation, and ETL pipeline development.
- AWS Glue components experience: jobs, crawlers, data catalog, and workflow orchestration.
- Proficiency with Databricks notebooks, PySpark, Spark SQL, and distributed data processing.
- Comprehensive understanding of ETL/ELT pipelines: data ingestion, validation, extraction, transformation, and loading.
- Knowledge of Amazon S3, Delta Lake, Lakehouse architecture, and AWS data ecosystem services.
- Familiarity with AWS Lambda, Amazon Redshift, CloudWatch monitoring, IAM policies.
- Experience with Apache Spark, Git version control, CI/CD pipelines, and deployment automation.
- Exposure to workflow orchestration tools like Airflow and Agile development methodologies.
- Understanding of dimensional modeling, star schemas, and data lake designs.
Additional Information
Work Location: Hyderabad, Bangalore, or Chennai
Job Type: Full-time
Eligibility: Open to any graduate
Minimum education
Bachelor's Degree
Skills
Tools & software
Amazon Simple Storage Service S3
required
Databricks
· 5 to 8 years required