Data Engineer

About the Role

We are seeking an experienced Data Engineer for an initial 12-month contract position in Brisbane. This role is crucial for supporting the delivery of a significant enterprise data platform program within a large organization. It's a hands-on position focused on the design, construction, and operation of secure and dependable data pipelines and curated data products. You will utilize Microsoft Fabric, Azure, SQL, and PySpark to contribute to a centralized data platform aimed at enhancing analytics, reporting, and business insights. Collaboration with data governance, reporting, and business stakeholders is key to implementing privacy-by-design ingestion, ensuring high-quality datasets, and developing performant semantic models for Power BI.

Key Responsibilities

• Construct and manage data pipelines utilizing Microsoft Fabric components such as Pipelines, Dataflows Gen2, Notebooks/Spark, Lakehouse/OneLake, and SQL Endpoints.
• Architect and implement data ingestion and transformation processes across Bronze, Silver, and Gold data layers.
• Develop curated datasets, dimensional models, and semantic models tailored for Power BI reporting.
• Work with Microsoft Purview to facilitate metadata discovery, classification, sensitivity labeling, lineage tracking, and governance adherence.
• Integrate data quality checks, validation rules, automated testing procedures, and maintain technical documentation.
• Enforce data security and privacy principles, including least-privilege access, managed identities, data masking, and audit logging.
• Support Continuous Integration and Continuous Deployment (CI/CD) and deployment automation using Azure DevOps or similar tools.
• Configure monitoring, logging, and alerting systems to ensure optimal reliability, performance, and cost efficiency.
• Engage collaboratively with data stewards, custodians, analysts, product owners, and other technical teams.

About You

The ideal candidate will possess over 5 years of experience in data engineering or comparable roles. You should have substantial experience in building production-ready data pipelines and curated datasets. Demonstrated proficiency with Microsoft Fabric or closely related Microsoft data platform technologies is essential. Strong skills in SQL and PySpark are required, along with experience in Azure data services like Azure Data Factory, Synapse, or Databricks. Familiarity with medallion architecture, dimensional modeling, and Power BI semantic models is expected. Exposure to Microsoft Purview, data governance, data quality, or lineage practices is beneficial. Experience with Azure DevOps, CI/CD, Git-based workflows, or environment promotion is also valuable. A solid understanding of data security, privacy, Role-Based Access Control (RBAC), managed identities, and audit logging is necessary. Excellent documentation, stakeholder engagement, and communication abilities are a must.

Additional Information

This is an initial 12-month contract position based in Brisbane with a hybrid working arrangement. To apply, please submit your resume or contact Abbie for a confidential discussion at [email protected] or 0413 444 157.