Data Engineer

Job Overview and Required Skills

We are seeking skilled candidates to apply for a Data Engineer role requiring extensive expertise in cloud data engineering technologies. Applicants must possess strong experience working with SQL, Python or PySpark, and Google Cloud Platform (GCP) services with a focus on BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Functions. Proficiency in building and managing monitoring dashboards as well as knowledge of Apache Kafka and Terraform scripting are essential for this position. Experience with Airflow or its alternatives such as Astronomer or Cloud Composer is also mandatory.

Required Qualifications and Technologies

• Minimum 4 years of experience with SQL, and Python or PySpark programming languages.
• At least 3 years of hands-on use of GCP services including BigQuery, Dataflow/Dataproc, Pub/Sub, Scheduled Queries, Cloud Functions, and monitoring tools.
• Proven skills in operating Apache Kafka for data streaming.
• At least 2 years of experience writing infrastructure as code using Terraform scripting.
• Experience working with workflow orchestration tools like Airflow, Astronomer, or Cloud Composer for 2 or more years.

Additional Preferred Skills

• Familiarity with Data Fusion and Tekton Pipeline.
• Experience setting up CI/CD pipelines.
• Knowledge of Cloud Secret Manager to secure sensitive information.
• Interest or experience in prompt engineering and generative AI/large language models (GenAI/LLM).
• Understanding or proficiency in LangGraph, CrewAI, OpenAI SDK, and Google ADK.

Application Instructions

Candidates who meet the above criteria are encouraged to submit their updated resumes for consideration.