Lead Data Engineer

Overview

XPT Software Australia is seeking an accomplished Lead Data Engineer to spearhead the design, delivery, and enhancement of large-scale data platforms hosted on Microsoft Azure. This role requires blending advanced technical knowledge with leadership skills to create scalable, secure, and cost-effective data solutions aligned with business goals.

Mandatory Skills

• Proficient in SQL, Python, and PySpark programming languages.
• Extensive hands-on experience with Azure Data Factory and Azure Synapse Analytics services.
• Expertise with Delta Lake architecture and modern lakehouse data implementations.
• Comprehensive knowledge of data modelling and warehousing principles.
• Familiarity with Microsoft Fabric ecosystem components.
• Experience designing cloud-native, enterprise-grade data architectures.
• Strong problem-solving skills with a design thinking approach.
• Understanding of CI/CD pipelines and DevOps methodologies.
• Skilled in optimizing cloud expenditure, performance tuning, and resource governance.

Primary Responsibilities

• Lead complete lifecycle ownership of enterprise data platforms including architecture, development, deployment, and ongoing improvements.
• Develop and execute data platform roadmaps aligned with strategic priorities.
• Architect and manage scalable data pipelines, warehouses, and lakehouse solutions within Azure environment.
• Provide leadership through design and code reviews while enforcing engineering best practices.
• Champion the use of Microsoft Fabric, API services, Azure Functions, and Python-based tools.
• Ensure platform reliability, security, and scalability while managing cost efficiency.
• Implement standards for data governance, quality, security, and compliance.
• Promote and integrate CI/CD, DevOps, and Agile practices across teams.
• Collaborate with business units and technical teams to translate requirements into robust technical solutions.
• Mentor and cultivate data engineering talent, fostering innovation and continuous learning.
• Utilize AI-enabled tools like GitHub Copilot to enhance productivity and code quality.
• Evaluate cutting-edge technologies to drive platform innovation and modernization.

Experience Requirements

• Over 10 years of IT experience, with at least 10 years dedicated to data engineering roles.
• Minimum of 5 years leading large-scale data engineering programs and teams.
• Proven success in managing complex enterprise data transformations and cross-functional teams.
• Established record of building and scaling high-performance data engineering groups.
• Experience working within multi-vendor and globally distributed environments.
• Strong capabilities in stakeholder engagement, delivery governance, and executive communication.

Preferred Qualifications & Skills

• Background building enterprise APIs and managing Azure Function Apps.
• Experience applying framework-driven data engineering development.
• Familiarity with solution architecture, stakeholder interaction, and governance.
• Azure certifications such as Data Engineer Associate and Solutions Architect Expert.
• Knowledge of MLOps, Data Science practices, and AI-enabled analytics.
• Practical experience with Power BI and enterprise-level reporting platforms.
• Integration of AI and Generative AI capabilities within data engineering workflows.

Leadership Attributes

• Strong ownership mentality emphasizing accountability and outcomes.
• Excellent communication, influence, and stakeholder management skills.
• Ability to balance technical leadership with delivery and team management duties.
• Strategic mindset with a focus on impeccable execution.
• Conflict resolution adeptness and cross-team collaboration strengths.
• Adaptability in dynamic, fast-moving settings.
• Passionate about mentoring and fostering innovation-driven cultures.