Staff Software Engineer, Inference Infrastructure

Company Overview

Cohere stands as a leading enterprise AI firm prioritizing security, dedicated to creating state-of-the-art foundational AI models and comprehensive products aimed at solving practical business challenges. Operating globally with headquarters in Toronto and multiple international offices, Cohere fosters a collaborative environment comprising researchers, engineers, and designers passionate about advancing AI capabilities.

Role Overview

This position invites software engineers driven by building efficient, scalable, and dependable machine learning systems to join the Model Serving team. The core mission is to develop, deploy, and maintain AI platforms that serve large language models through user-friendly API endpoints. Collaboration across various teams is integral to optimize NLP model deployments in environments that demand low latency, high throughput, and robust availability. The role also offers interaction with customers for tailored deployment solutions.

Key Responsibilities

• Design, implement, and maintain scalable AI infrastructure for running large language models.
• Deploy optimized natural language processing models in production ensuring minimal latency and high availability.
• Collaborate closely with cross-functional teams to support customized customer deployments.
• Manage and troubleshoot complex distributed systems and GPU workloads on Kubernetes clusters.
• Ensure efficient compute, storage, and network resource allocation while managing costs.

Candidate Qualifications

• At least five years of practical experience managing production-scale infrastructure.
• Proven expertise in architecting distributed systems with Kubernetes and handling GPU-based workloads.
• Hands-on experience with Kubernetes development, deployment, and support in production environments.
• Familiarity with cloud platforms such as GCP, Azure, AWS, OCI, and hybrid or multi-cloud serving models.
• Proficiency in operating and troubleshooting Linux-based computing environments.
• Skill in managing resources and costs for compute, storage, and networking.
• Strong collaboration and problem-solving abilities to maintain mission-critical systems.
• Ability to adapt to evolving technical challenges requiring persistence and grit.
• Understanding of accelerator hardware (GPUs, TPUs, custom) and their impact on latency and throughput during inference.
• Solid background or working knowledge in distributed systems design and operations.
• Programming proficiency in Golang, C++, or other languages suited for high-performance, scalable server development.

Benefits and Work Environment

• Weekly lunch allowance of $75 (or local equivalent).
• Comprehensive health and dental insurance, including dedicated mental health benefits.
• Retirement schemes such as RRSP matching, 401K, or Pension plans.
• Fully paid parental leave top-up for six months applicable to any parent.
• Annual enrichment budgets for arts, culture, fitness, wellness, workspace upgrades, education, and learning opportunities.
• Generous vacation policy of six weeks (30 working days).
• Travel budget for visiting company offices and participation in annual company offsite events.
• Options to work remotely with co-working benefits and subsidies for home office setup ($500 stipend).

Additional Information

Cohere supports inclusiveness and equal opportunity hiring from all backgrounds and is prepared to provide accommodations during the recruitment process. AI-assisted applicant screening may be used alongside recruiter assessment. Job fraud is actively guarded against; genuine communications originate solely from official company emails.

Annual Compensation: $190,000 to $325,000 USD

Location: Remote opportunity with offices located worldwide, including Montreal.