Principal Research Scientist - Evaluations

About Canva and the Role

Join Canva's innovative team in Sydney, dedicated to transforming the global design experience. This role centers on pioneering evaluation methods for generative models, tackling complex challenges in assessing design quality across multiple modalities such as image, video, audio, and agentic workflows.

Key Responsibilities

• Lead the formulation and implementation of Canva's evaluation strategy, setting standards for measuring generative quality and balancing computational costs, human data collection, and evaluation accuracy.
• Advance the science behind measurement by correlating automated evaluators and large language models with human preferences and product outcomes, while identifying bias and saturation in benchmarks.
• Ensure evaluations closely mirror real user experience and business impact, distinguishing between quality metrics and reward models for reinforcement learning.
• Establish consistent evaluation protocols across international teams to guarantee comparability and foster collaboration.
• Design and manage human preference assessments at scale, accounting for cultural and market differences in aesthetic judgment.
• Develop learned quality models and reward signals to support post-training and inference selection processes, clarifying distinctions between judges and reward models.
• Innovate in vision-language models to understand design effectiveness and integrate these insights into feedback loops.
• Implement automated evaluation systems using MLLM judges and scalable infrastructure to conduct numerous evaluations in training cycles.
• Expand rigorous evaluation approaches to emerging modalities like photo AI and video, enabling segmentation by document type, user group, and locale.
• Deploy evaluation frameworks in production with offline and online monitoring to detect quality regression proactively.
• Enhance external credibility through benchmarking against industry leaders, publishing findings, and contributing to the broader evaluation community.
• Make critical launch decisions based on robust evaluation gates and confidently adjudicate ambiguous signals.
• Diagnose irregularities during production trainings, distinguishing genuine model issues from infrastructure artifacts efficiently.
• Collaborate directly with design, foundation model, and product teams to integrate evaluation into training and inference stages.
• Mentor senior scientists and engineers to elevate evaluation expertise across the organization.
• Advocate for Canva’s evaluation vision with leadership and industry peers.

Candidate Profile

• Proven experience designing foundational evaluation frameworks, especially for generative AI, with tangible impact on team decisions rather than purely academic contributions.
• Demonstrated skill linking evaluation results to actual business or user outcomes and diagnosing discrepancies.
• Expertise in converting subjective human judgments into objective, reliable evaluation signals, including rubric development and human data pipeline orchestration.
• Strong knowledge of multimodal generative architectures such as diffusion models, transformers, vision-language and large language models, with insight into their evaluation challenges.
• Background in reward modeling, preference learning, or alignment techniques involving human feedback.
• Experience guiding technical strategies across multiple teams or specialties within a global organization, taking initiative to address unowned problems.

Preferred Qualifications

• Research experience in human perception, psychophysics, aesthetics, or human-computer interaction.
• Familiarity with evaluating harm, bias, and safety along with quality.
• A publication record in evaluation, alignment, or generative modeling fields.
• Interest or background in visual arts and graphic design.

Additional Information

If you feel excited by this role but don’t meet every criterion, we encourage you to apply as diverse backgrounds and a passion for learning are highly valued.

Employee Benefits

• Equity packages enabling you to share in the company’s success.
• Family-friendly parental leave policies supporting all parents and carers.
• An annual allowance to enhance wellbeing, social engagement, and home office setup.
• Flexible leave policies promoting personal recharge, social good, and work-life balance.

Other Details

Hiring decisions are based on skills, experience, and cultural fit. Candidates should indicate pronouns and request any necessary accommodations for the interview process, which is conducted virtually.