Databricks Data Engineer (Data & AI)

About Deloitte

At Deloitte, we value your unique curiosity, passion, and collaboration alongside exceptional people working on meaningful projects. We aim to create significant impact, embracing diverse perspectives and fostering inclusion in line with our commitment to Te Tiriti o Waitangi and the indigenous Māori people of Aotearoa. Our culture emphasizes trust, balance, and wellbeing, offering a distinct work experience designed around flexibility and support.

Role Overview

We are seeking a motivated Databricks Data Engineer to join our growing AI & Data Engineering team based in Wellington. This role is tailored for individuals passionate about modern data engineering, enthusiastic about tackling challenging technical and business problems, and eager to utilize AI-assisted engineering tools to accelerate delivery quality and speed.

Key Responsibilities

• Architect, develop, and maintain efficient, scalable data and AI solutions leveraging Databricks technology.
• Create robust data ingestion, ELT/ETL pipelines, and data processing workflows within Databricks environments.
• Build reusable, high-quality data transformations using Lakeflow Declarative Pipelines.
• Design and implement Lakehouse architectures to support diverse data and AI applications.
• Develop dimensional and feature-rich data models suitable for analytics, reporting, and machine learning including Generative AI cases.
• Integrate Databricks with multiple cloud platforms, APIs, databases, streaming systems, and enterprise applications.
• Deploy CI/CD pipelines and employ modern DevOps methodologies for Databricks solutions.
• Orchestrate data workflows using Lakeflow Jobs, Azure Data Factory, Airflow, or analogous orchestration tools.
• Implement governance, security, data quality controls, lineage tracking, and continuous monitoring for data pipelines.
• Optimize Databricks workloads focusing on performance, scalability, reliability, and cost-efficiency.
• Diagnose and resolve complex pipeline and platform issues across development and production environments.
• Collaborate with cross-functional teams including architects, analysts, data scientists, ML engineers, and stakeholders to translate business needs into technical solutions.
• Contribute to establishing engineering standards, reusable frameworks, and comprehensive technical documentation.
• Leverage AI tools such as Genie Code and GitHub Copilot to expedite development processes, improve code quality, testing, and documentation.
• Identify automation opportunities to minimize repetitive tasks and speed up the development lifecycle.

Required Skills and Experience

• Proven hands-on expertise with Databricks platforms.
• Experience designing and delivering modern Lakehouse data architectures.
• Strong knowledge of data engineering concepts including data modeling, data warehousing, and distributed data processing.
• Practical experience in building batch and streaming data pipelines.
• Familiarity with orchestration tools such as Lakeflow Jobs, Azure Data Factory, or Airflow.
• Understanding cloud data services primarily on Azure and/or AWS.
• Competence with Git, CI/CD pipelines, and contemporary software engineering practices.
• Python programming capability within data engineering contexts is advantageous.
• Experience with infrastructure-as-code tools including Terraform or equivalent technologies.
• Excellent analytical, problem-solving, and troubleshooting skills.
• Strong verbal and written communication skills with effective stakeholder management.

Preferred Qualifications

• Background in Generative AI, Machine Learning, or MLOps implementations on Databricks.
• Experience with Databricks Declarative Automation Bundles.
• Professional certifications such as Databricks Certified Data Engineer Associate or Professional.

AI-Assisted Engineering Approach

This role embraces AI-powered engineering practices, utilizing tools like Genie Code, Claude, GitHub Copilot, and others to streamline development, reduce repetitive coding tasks, enhance code quality, and maintain top-tier security and maintainability standards.

Success Profile

The ideal candidate effectively owns data engineering projects end to end, delivering solutions that are scalable, automated, secure, and maintainable. They thrive in collaborative environments and continuously seek improvements by applying reusable patterns, automation, and AI-driven development aids.

Why Join Deloitte?

• Engage with diverse, high-impact projects for leading New Zealand clients.
• Work on modern enterprise-scale Databricks Lakehouse platforms integrating Data Engineering, Analytics, AI, and Generative AI.
• Collaborate with expert data engineers, architects, and technology leaders.
• Contribute to developing reusable frameworks and accelerate engineering innovations.
• Access supportive culture with flexible working, health benefits, professional development subsidies, social and wellness initiatives, and environmental sustainability focus.

Additional Information

Deloitte Aotearoa champions diversity and inclusion, encouraging applications even if some qualifications or ideal experiences are not fully met. We strive to maintain secure recruitment with official communications originating strictly from designated Deloitte domains. Candidates should be cautious of phishing scams.