Data Engineer

Job Overview

We are seeking a skilled Data Engineer to design, build, and fine-tune data ingestion, transformation, and consumption processes on a state-of-the-art on-premises lakehouse platform based in Saudi Arabia. The candidate must possess substantial practical experience in Spark development, Informatica integration solutions, and enterprise-level data platforms.

Key Responsibilities

• Architect and develop efficient and scalable data ingestion and transformation pipelines.
• Build, enhance, and maintain ETL/ELT workloads using Spark technology.
• Create and administer data models and datasets leveraging Apache Iceberg tables.
• Implement and manage data integration workflows through Informatica tools.
• Facilitate analytics and data accessibility using platforms such as Denodo, Starburst, and Dremio.
• Establish secure, governed, and scalable data pipelines adhering to enterprise standards.
• Support both batch processing and near real-time data operations.
• Enforce data quality measures, metadata management, and validation protocols.
• Coordinate with Platform Engineers for deployment, integration, and operational management.
• Assist in migration, testing procedures, and production-level rollout activities.
• Troubleshoot and optimize data pipelines for performance, scalability, and reliability.

Required Skills and Knowledge

• In-depth experience with Apache Spark, including PySpark and/or Scala Spark.
• Proficiency in working with Apache Iceberg.
• Strong command of Informatica Data Integration tools.
• Knowledge of MinIO object storage technology.
• Experience using Denodo for data virtualization and analytics enablement.
• Advanced SQL skills and data modeling expertise.
• Understanding of data warehousing concepts and design.
• Hands-on experience with ETL and ELT development methodologies.
• Familiarity with Git version control systems.

Preferred Skills

• Experience with Starburst and/or Dremio platforms.
• Knowledge of data governance frameworks and metadata management.
• Implementing data quality frameworks.
• Utilizing CI/CD pipelines for data workflows.
• Proficiency in Python programming.
• Experience with Kafka or similar streaming technologies.
• Understanding of lakehouse architecture principles.
• Track record with enterprise data lake and lakehouse deployment.
• Expertise in tuning and optimizing performance for large-scale data processing workloads.