AI Engineer, AI & Applications

Role Summary

Firmus Technologies is looking for an experienced AI Engineer to architect, build, and uphold the MLOps and AIOps infrastructures supporting Firmus AI Factory, their AI platform. This pivotal position involves establishing end-to-end MLOps workflows that convert model development into a structured, controlled software release process with clear governance, automated evaluation checkpoints, and dependable production promotion.

You will be tasked with enabling the Model Arena initiative by crafting and deploying evaluation pipelines and standards to ensure model selection for RAG and agentic applications is data-driven, reproducible, and production-ready. Your responsibilities extend to the design, testing, upkeep, and dependability of AI platform features such as training workflows, inference services, and RAG systems, including the implementation of quality gates, model promotion methods, monitoring, and incident response procedures.

Your role ensures that AI platform components are engineered to match the trustworthiness of core infrastructure—emphasizing speed, reliability, maintainability, and observability. Collaboration across teams is essential, working alongside engineers on CI/CD gates, data scientists on quality metrics, and operations teams handling incident response.

Responsibilities

• Architect and maintain comprehensive MLOps workflows covering training, evaluation, registry, deployment, monitoring, and retraining/retirement across development, staging, and production environments with defined standards and ownership.
• Manage the model registry and promotion lifecycle utilizing MLflow, incorporating stage/alias strategies, approval processes, environment segregation, access control, rollback options, and integration with software release pipelines.
• Guarantee reproducibility and lineage in the model lifecycle through versioned code/configuration, artifact tracking, dataset/version references, and repeatable evaluation processes.
• Create automated model quality gates for production targeting accuracy, latency, cost, safety, and reliability criteria.
• Establish Service Level Objectives and Indicators (SLOs/SLIs) for AI platform capabilities, including training success rates, inference latency at the 99th percentile, RAG retrieval accuracy, availability, and cost metrics.
• Develop and sustain production monitoring dashboards to observe model performance, data drift, and operational health, integrating alert systems such as PagerDuty and Slack.
• Compose on-call runbooks and incident triage protocols for AI services and lead postmortem analyses to enhance software reliability and maintainability.
• Implement instrumentation in AI services for debugging purposes—tracking request traces, GPU usage per model, retrieval speeds, and communication bottlenecks.
• Integrate evaluation frameworks (benchmarking, RAGAS, LLM-as-judge) into CI/CD pipelines to facilitate automated testing, deployment, and release management.

Skills & Experience

• Bachelor’s degree or higher in Software Engineering, Computer Science, Information Technology, or a related discipline.
• Minimum of three years’ hands-on experience in MLOps, ML platform engineering, software engineering, or production engineering with responsibility for production ML pipelines.
• Comprehensive knowledge of the ML software lifecycle including model versioning, promotion tactics, evaluation automation, governance, deployment approaches, monitoring, and drift detection.
• Practical expertise with MLflow Model Registry workflows, including handling stages/aliases, approval processes, traceability, and integration within software release cycles.
• Experience in designing and operationalizing model evaluation systems that encompass metric standards, orchestration, logging, and reproducibility.
• Strong foundation in software engineering and production principles encompassing metrics, logging, tracing, alerting systems, incident handling, and reliability engineering.
• Familiarity with CI/CD tooling, model packaging, deployment automation, and software testing; ability to collaborate effectively with ML engineers, SRE/platform teams, and application developers.
• Understanding of distributed systems concepts, resource management, and failure modes in both training and inference environments.

Key Competencies

• Production Ownership: Demonstrated ability to take charge of services in production with a proactive approach to monitoring and issue prevention.
• Reliability Engineering: Experience setting SLOs, managing error budgets, and promoting a blameless postmortem culture.
• Cross-Functional Leadership: Proven skill in collaborating and removing blockers among ML engineers, data scientists, and platform teams.
• Incident Response: Capable of efficient triage, root cause analysis, and systemic problem-solving beyond firefighting.
• Automation: Emphasis on programmatic solutions to minimize repetitive work while maintaining a balance between speed and rigor.
• Communication: Ability to clearly articulate complex ML and systems topics to both technical and non-technical audiences.

Location & Reporting

Position based in Sydney, Australia, reporting directly to the Head of AI & Applications.

Employment Details

This is a full-time role providing an opportunity to be part of Firmus’ mission to innovate sustainable AI infrastructure with a diverse and inclusive culture.