Data Engineer

Overview

This opportunity is with an Indian and global engineering and manufacturing organization seeking an experienced Data Engineer. The role involves designing, developing, and optimizing data pipelines primarily on Google Cloud Platform (GCP).

Key Responsibilities

• Create and fine-tune data processing workflows using PySpark on GCP infrastructure.
• Construct and manage both batch and real-time streaming data pipelines using Dataflow and Dataproc.
• Load datasets into BigQuery and uphold the data quality and integrity.
• Coordinate pipeline workflows through Cloud Composer (Airflow).
• Handle data storage and transfer leveraging Cloud Storage and perform SQL-based data transformations.

Requirements

• 6 to 9 years of professional experience in data engineering roles.
• Proficient with PySpark and Python programming for data processing tasks.
• Extensive experience working with Google BigQuery for data warehousing.
• Hands-on expertise in Google Cloud's Dataflow and Dataproc services.
• Experience using Cloud Composer (Airflow) to orchestrate workflows.
• In-depth knowledge of Google Cloud Storage solutions.
• Strong command over SQL, particularly in data transformation and performance optimization.
• Proven ability to develop and maintain scalable data engineering pipelines.

Preferred Qualifications

• Familiarity with developing ETL or ELT pipelines.
• Experience working on projects for American Express (AMEX) or within the financial services domain.

Educational Background

• Bachelor's degree in Computer Science or a related discipline.
• Master's degree in Computer Science or related field is also acceptable.

Eligibility

Applicants must hold at least a graduate degree to be eligible for this role.