Senior Research Scientist, Model Evaluation

About Cohere

Cohere is a leading security-focused enterprise AI company developing state-of-the-art foundation AI models and comprehensive products aimed at solving practical business challenges. Our work plays a pivotal role in accelerating AI adoption globally. Our team is composed of passionate researchers, engineers, and designers dedicated to crafting advanced AI capabilities that provide measurable value to our clients.

We maintain a global presence with headquarters in Toronto and offices in London, New York City, San Francisco, Montreal, Paris, Berlin, and Seoul, embracing a diverse and inclusive workplace.

Role Summary

As a Senior Research Scientist specializing in model evaluation, you will drive the creation of innovative evaluation benchmarks that test and expand the abilities of large language models (LLMs). The position involves building sophisticated infrastructure and methods to assess model progress, ensuring that our evaluation techniques both accurately reflect current model strengths and inform future model development.

Key Responsibilities

• Design and develop ambitious benchmarks to challenge and quantify model capabilities.
• Collaborate across multifunctional teams to transform model feedback into reliable and reproducible assessments.
• Advance research in LLM evaluation by training LLM evaluators, optimizing data synthesis pipelines, and enhancing evaluation efficiency.
• Create scalable, reusable diagnostic tools to analyze and understand model behaviors and performance.

Candidate Qualifications

• Proven enthusiasm for quickly prototyping tools that demonstrate the limits of LLM abilities and crafting resources to measure them.
• Extensive experience thoroughly reviewing complex datasets and LLM outputs to maintain high data integrity.
• Strong commitment to precise and meaningful measurement of AI capabilities, ensuring evaluations align with intended competencies.
• Excellent software engineering proficiency to support development and implementation.

Work Arrangement

This role offers flexibility to operate fully remotely or from any of our global office locations. There is no mandatory onsite presence requirement; however, timezone and working hour alignment may be expected based on the posting location.

Employee Benefits

• Weekly lunch allowance of $75 (currency adjusted to your locale).
• Comprehensive health and dental insurance with an additional mental health care budget.
• Retirement savings plans including RRSP matching, 401K, or Pension Schemes.
• Full parental leave salary top-up up to six months for any parent.
• Annual enrichment benefits covering arts, culture, wellness, quality time, and workspace upgrades.
• Education stipend for conferences, courses, and coaching.
• Generous paid vacation totaling six weeks (30 working days).
• Travel budget for in-person visits to other offices and annual company offsite gatherings.

Additional Workplace Perks

• Remote-friendly company with modern offices in multiple international cities.
• Daily lunch programs, snacks, and regular social events for office-based employees.
• Co-working benefits for remote workers to collaborate locally.
• Home office setup stipend of $500.

Commitment to Inclusion and Fair Hiring

Cohere fosters an inclusive environment welcoming candidates from diverse backgrounds and is dedicated to providing equal opportunities. Accommodation requests are supported during the recruitment process. AI tools may assist in evaluating applicants but recruiters consider all submissions comprehensively.

Compensation

Annual salary ranges from CA$250,000 to CA$535,000.

Security Notice

Cohere never requests payments or third-party services during hiring processes. Official communications use company email domains. Verify authenticity through official career pages.