Data Engineer

Job Overview

We are seeking a skilled Databricks Developer with extensive experience in Spark and PySpark to architect, develop, and optimize data processing systems. The role involves creating scalable data engineering solutions that facilitate smooth data transitions across different platforms.

Key Responsibilities

• Create and maintain highly scalable data pipelines using Python, PySpark, and Apache Spark technologies.
• Work closely with data engineers, scientists, and various stakeholders to translate business needs into robust technical solutions.
• Maintain data integrity, boost pipeline performance, and ensure reliability in all data workflows.
• Produce clean, efficient, and maintainable code adhering to industry best practices and standards.
• Conduct thorough data analyses and implement validation methods to uphold data quality.
• Identify and resolve bottlenecks or issues impacting pipeline performance.
• Develop and manage Continuous Integration and Continuous Deployment (CI/CD) workflows for automated testing and deployment processes.

Required Qualifications

• Possess a Bachelor’s or Master’s degree in Computer Science, Engineering, or a closely related discipline.
• Demonstrable experience as a Databricks Developer with proficient skills in Spark and PySpark.
• Strong command of SQL with practical experience working on relational database systems.
• Robust programming capabilities in Python and PySpark.
• Knowledge of version control tools, particularly Git.
• Excellent analytical thinking and problem-solving skills.
• Effective communication and teamwork competencies.

Eligibility

This position is open to individuals holding at least a graduate-level qualification in any stream.