Senior Research Scientist, Model Evaluation

About Cohere

Cohere is a pioneering security-focused enterprise AI company that develops advanced foundational AI models and comprehensive products aimed at solving tangible business challenges. We are dedicated to advancing AI adoption by training and deploying cutting-edge models for enterprises building AI applications. Our team comprises passionate researchers, engineers, and designers committed to enhancing model capabilities and delivering significant value to our customers. With global headquarters in Toronto and offices in multiple major cities worldwide, we foster a collaborative and diverse work environment.

Role Overview

This position is vital for driving progress in measuring and scaling AI intelligence. As AI models achieve superhuman performance in numerous real-world scenarios, there is an ongoing need to establish innovative evaluation techniques that not only capture current model abilities but also guide the capabilities of future models. In this role, you will design next-generation evaluation frameworks and infrastructures to assess the advancement of large language models (LLMs).

Key Responsibilities

• Design and build ambitious new evaluation benchmarks that test the upper limits of model abilities.
• Collaborate extensively with cross-functional teams to convert model feedback into reliable, repeatable evaluation metrics.
• Conduct research to push forward the state-of-the-art in LLM evaluation methods, including developing LLM-based judges, enhancing data synthesis pipelines, and increasing evaluation efficiency.
• Develop scalable, reusable tools for in-depth analysis of model performance.

Candidate Profile

• Aptitude for quickly building prototypes that demonstrate the boundaries of LLM capabilities and creating methods to measure these competencies.
• Extensive experience reviewing complex datasets and LLM outputs to assure high quality and accuracy.
• Strong commitment to precise and meaningful measurement of AI capabilities, ensuring alignment between metrics and actual model performance.
• Proficient software engineering skills.

Employee Benefits

• A weekly lunch allowance of $75 or equivalent local currency.
• Comprehensive health and dental insurance including a dedicated mental health budget.
• Retirement benefit plans such as RRSP matching, 401K, or pension schemes.
• Fully paid parental leave top-up for up to six months for either parent.
• Annual enrichment funds for arts, culture, fitness, wellness, workspace improvements, and learning (conferences, courses, coaching).
• Generous paid vacation totaling six weeks (30 working days).
• Travel budget for visiting other offices and attendance at an annual company-wide offsite event.

Work Environment and Flexibility

• Remote-friendly work culture with physical offices located globally including Toronto, London, New York City, San Francisco, Montreal, Paris, Berlin, and Seoul.
• Office-based employees enjoy daily lunch programs, snacks, and frequent social and community activities.
• Remote employees receive co-working benefits for local shared workspace access.
• All employees receive a $500 stipend for home office setup.

Inclusion and Equal Opportunity

We encourage applications from diverse backgrounds and are committed to fostering an inclusive workplace. Applicants requiring accommodations during the recruitment process can request assistance to ensure equitable participation.

Recruitment Integrity

We utilize AI-assisted tools to screen candidates but maintain full recruiter oversight. Please be vigilant against fraudulent job listings; official communications come only from @cohere.com or @cw.cohere emails.