Senior Databricks Specialist / Data Architect

Role Overview

Seeking an experienced Senior Databricks Specialist to design, develop, and optimize high-scale data platforms leveraging Microsoft Azure Databricks. The role focuses on establishing robust data architecture, integration with Dynamics 365 CRM and other enterprise systems, and ensuring peak performance and governance of data pipelines.

Key Responsibilities

• Architect and deploy enterprise-grade data solutions with Azure Databricks technology.
• Establish and uphold data platform standards and architecture guidelines specific to Databricks.
• Create and maintain data ingestion and integration workflows spanning diverse source systems.
• Build scalable ETL/ELT pipelines utilizing Databricks and Apache Spark frameworks.
• Implement Medallion Architecture layers (Bronze, Silver, Gold) where applicable for organized data processing.
• Manage and optimize Delta Lake tables and data flow processes for efficiency.
• Design data models and transformation procedures to support both operational and analytical reporting requirements.
• Integrate Databricks platform seamlessly with Microsoft Dynamics 365, Dataverse, and other organizational applications.
• Develop and fine-tune batch as well as near real-time data pipelines to support business needs.
• Introduce and maintain processes for data quality assurance, validation, lineage tracking, and governance.
• Optimize performance and cost factors by tuning Spark jobs, clusters, SQL queries, Delta Lake tables, and pipelines.
• Implement robust data security frameworks and control access rigorously.
• Utilize Unity Catalog for enhanced data governance, access permissions, cataloging, and lineage tracking.
• Provide optimized data models and services enabling platforms like Power BI and other analytics tools.
• Diagnose and resolve complex engineering and data performance challenges.
• Define and enforce development, testing, and production best practices for Databricks environment.
• Support continuous integration/continuous deployment (CI/CD) and DevOps workflows for Databricks solutions.
• Perform technical evaluations and mentor junior data engineering staff.
• Guarantee that data platform meets security, availability, scalability, and disaster recovery standards.

Technical Requirements

• At least 8 years of experience in data engineering or architecture roles.
• Minimum of 4 years hands-on expertise with Azure Databricks.
• Proficient in Apache Spark and PySpark programming.
• Extensive experience working with Delta Lake technology.
• Demonstrated skill designing enterprise data platforms and complex pipelines.
• Familiarity with Databricks Medallion Architecture concepts.
• Knowledgeable with Unity Catalog for governance, access control, and lineage documentation.
• Strong SQL capabilities; skilled in managing large-scale data processing tasks.
• Experience with various Azure data services, including Azure Data Factory and Azure Data Lake Storage Gen2.
• Skilled in integrating Databricks with APIs, CRM systems, and other enterprise software.
• Competence in data modeling techniques including dimensional modeling.
• Strong focus on ensuring data quality and governance policies.
• Expertise in optimizing performance and reducing costs related to Databricks workloads.
• Hands-on with CI/CD pipelines, using tools like Git, Azure DevOps, and automation for deployment.
• Ability to manage and troubleshoot data issues in large enterprise environments.

Preferred Certifications

• Databricks Certified Data Engineer Professional
• Databricks Certified Data Engineer Associate
• Microsoft Certified: Azure Data Engineer Associate
• Other applicable Databricks or Microsoft Azure certifications