Staff Applied AI Scientist

About Culture Amp

Culture Amp leads the employee experience platform industry, influencing how over 25 million employees across more than 6,000 companies enhance workplace culture, engagement, performance, and team development. Trusted by major firms like Canva, Asana, and Nasdaq, Culture Amp operates globally with offices in the US, UK, Germany, and Australia. The company is recognized for its innovation and cloud technology excellence.

Role Overview

As a Staff Applied AI Scientist, you will address the complex challenge of continuously evaluating AI products in production, diagnosing performance changes real-time, investigating causes, and implementing quality improvements at scale. Your primary focus will be on the Coach AI system, developing frameworks that enable ongoing robust system enhancements and guiding the engineering team to independently sustain these processes.

Key Responsibilities

• Own a comprehensive feedback loop involving prompt engineering, scalable evaluation, and continuous refinement by developing LLM-driven diagnostic tools that detect performance shifts, offer insights, and automate improvement suggestions.
• Innovate in context engineering by optimizing data retrieval, session memory management, context compression, and handling context budget for complex multi-turn interactions; ensure all modifications are validated through systematic evaluations.
• Design and conduct evaluation methodologies such as sampling, LLM-as-judge techniques, and human labeling on anonymized production data to establish ongoing monitoring and alert systems.
• Enhance agentic systems through evaluation-driven orchestration including planning, tool usage, routing, and verification steps, adjusting architecture based on identified failure modes.
• Lead decisions on AI model and provider selection balancing quality, latency, and cost, determining when to use prompting, fine-tuning, or model replacement.
• Develop and supervise safety measures including input/output guardrails, personally identifiable information handling, content safety, and jailbreak resistance, respecting sensitive coaching and personnel data.
• Empower other teams with reusable frameworks, tools, and documentation to manage their own evaluations, leading the initiative before transitioning ownership.
• Collaborate closely with AI Coach teams, product, data science, and people science to link quality metrics with actual customer value.
• Keep abreast of the latest advancements in evaluation techniques, observability, and large language model operations (LLMOps).

Qualifications and Experience

• Proven experience in building and deploying agentic AI systems incorporating context engineering, retrieval-augmented generation, memory, cost optimization, model selection, and performance monitoring.
• Demonstrated expertise in analyzing AI or data product performance in production and effecting improvements to maintain and enhance outcomes.
• Hands-on work with LLM evaluation methods such as LLM-as-judge, curated evaluation datasets, human-in-the-loop labeling, and threshold-based scoring in live environments.
• Familiarity with observability tools tailored to LLM and agentic systems, including trace analysis, prompt management, and production monitoring platforms like Langfuse or equivalents.
• Experience in longitudinal performance measurement, including baseline establishment, regression detection, and sustained quality tracking.
• Comfort with AI-native development tools (e.g., Claude Code, Cursor, Codex) for complex, multi-step tasks, with sound judgment on agent versus manual coding balance.
• Strong written and verbal communication skills, with a record of enabling others through documentation and capability building.
• Additional advantages include establishing evaluation and observability practices across teams, evolving enterprise codebases with AI, deep involvement in production agentic systems, advanced degrees in machine learning or related fields, and public contributions to the evaluation or LLMOps community.

Personal Attributes

• Passionate about scaling AI system effectiveness and adoption in operational settings, showing resilience and a growth mindset.
• Driven by empowerment of peers, gaining satisfaction from creating tools and systems for others to adopt, even if not personally owning the end product.

Work Culture and Approach

Engineers at Culture Amp increasingly leverage agentic systems that autonomously perform coding tasks under guided oversight, focusing on architecture, quality, and strategic trade-offs. The team invests in shared tooling and standards to enable safe and effective AI-driven product development.

Additional Information

This role requires legal authorization to work in Australia and is based in either the Melbourne or Sydney office. Candidates typically work onsite approximately two days per week to foster connection and cultural alignment.

Benefits and Perks

• Equity participation via an Employee Share Option Program fostering long-term engagement.
• Access to learning programs and personal coaching to support professional growth.
• Quarterly refresh days, extended holiday breaks, and monthly allowances for wellbeing and lifestyle.
• Inclusive parental leave available from the first day of employment.
• Provision of a MacBook and budget for home office setup to enable flexible work environments.
• Opportunity to participate in five annual social impact days supporting charitable causes.
• Medical insurance options for employees and family members available in the US and UK.

Diversity and Inclusion

Applicants from diverse and underrepresented backgrounds are encouraged to apply. The company values unique experiences and is focused on building an inclusive workforce.

Accommodation and Privacy

Reasonable accommodations are available for applicants with disabilities during the application and interview process upon request. Candidate data is retained for up to two years (four years in the US) for future job opportunities, handled in accordance with company privacy policies.