PySpark Data Engineer

Role Overview

We are seeking a skilled PySpark Data Engineer responsible for designing and developing scalable data pipelines to handle large volumes of both structured and unstructured datasets. The role involves building ETL workflows, optimizing Spark applications, and collaborating with multidisciplinary teams to ensure data integrity and performance.

Key Responsibilities

• Architect and implement efficient data pipelines using PySpark.
• Create ETL workflows and data transformation processes.
• Process extensive datasets, structured and unstructured.
• Optimize Spark applications for enhanced performance.
• Utilize SQL and Spark SQL for comprehensive data analysis and manipulation.
• Maintain data quality, ensure scalability, and guarantee pipeline reliability.
• Diagnose and fix issues within data pipelines promptly.
• Work closely with business stakeholders and technical teams.

Candidate Profile and Requirements

• Between 2 to 5 years of experience in data engineering roles.
• Demonstrated expertise in PySpark and Apache Spark frameworks.
• Strong programming abilities in Python.
• Proficient understanding of SQL.
• Hands-on experience with ETL processes and big data technologies.
• Familiarity with Databricks, Hadoop, or Hive is an advantage.
• Excellent analytical thinking and problem-solving skills.

Eligibility Criteria

Applicants should hold one of the following degrees: B.Tech or B.E. in any specialization, B.Sc in any discipline, or B.C.A. in any specialization.

Work Location

This full-time opportunity is based in Hyderabad, India, at Infosys Limited, located on the 3rd Cross of Electronic City, Bangalore, Karnataka.