Senior Data Architect

Role Overview

We are seeking a seasoned Senior Data Architect specializing in Databricks, with a strong background in designing, implementing, and optimizing enterprise-scale data platforms on Microsoft Azure Databricks. This role involves overseeing the architecture, development, governance, integration, and performance management of the organization's Databricks data platform, especially its coordination with Dynamics 365 CRM and other enterprise systems.

Key Responsibilities

• Architect and deploy large-scale data solutions leveraging Azure Databricks.
• Establish and uphold Databricks platform architecture and data standards.
• Create ingestion and integration pipelines from varied source systems.
• Build scalable ETL/ELT workflows using Databricks and Apache Spark.
• Implement Medallion Architecture concepts (Bronze, Silver, Gold layers) appropriately.
• Manage and enhance Delta Lake tables and pipeline performance.
• Design data models and transformations to support analytical and operational reporting.
• Integrate Databricks with Microsoft Dynamics 365 / Dataverse and other enterprise applications.
• Develop batch and near-real-time data processing streams and optimize them.
• Enforce data quality, validation, monitoring, lineage, and governance best practices.
• Optimize Spark jobs, clusters, SQL queries, Delta tables, and pipelines to enhance cost-effectiveness and performance.
• Implement and maintain robust data security protocols and access controls.
• Leverage Unity Catalog for data governance, access management, cataloging, and lineage tracking.
• Support analytical tools like Power BI through carefully optimized data models and services.
• Diagnose and solve complex data engineering and performance challenges.
• Define development, testing, and production standards within Databricks environments.
• Adopt CI/CD and DevOps methodologies to streamline Databricks deployments.
• Provide technical mentorship and conduct code reviews for data engineering teams.
• Ensure that data platform solutions comply with security, scalability, availability, and disaster recovery standards.

Mandatory Technical Requirements

• Over 8 years of experience in data engineering and data architecture roles.
• At least 4 years of practical, hands-on experience working with Azure Databricks.
• Extensive expertise in Apache Spark and PySpark programming.
• Strong skills with Delta Lake technology.
• Significant experience designing comprehensive enterprise data platforms and pipelines.
• In-depth knowledge of Databricks Medallion Architecture.
• Proficiency with Unity Catalog, including data governance, access control, and data lineage functionalities.
• Advanced SQL capabilities and experience managing large-scale data processes.
• Familiarity with Azure data services ecosystem.
• Experience using Azure Data Factory and Azure Data Lake Storage Gen2.
• Ability to integrate Databricks with APIs, CRM systems such as Microsoft Dynamics 365, and other enterprise data sources.
• Experience with data and dimensional modeling techniques.
• Strong understanding of data quality assurance and governance frameworks.
• Expertise in optimizing performance and costs in Databricks environments.
• Familiarity with CI/CD pipelines, Git, Azure DevOps, and automation of deployments.
• Experience managing and analyzing large enterprise datasets.
• Strong troubleshooting skills for complex production data challenges.

Additional Information

Please share your CV at the contact numbers: +974 50459977 or +974 50473388.