Senior Evaluation Algorithm Engineer

Company Overview

Binance is the world's largest cryptocurrency exchange by trading volume and user registrations, serving over 300 million users across more than 100 countries. Recognized for leading security standards, transparent user fund management, rapid trading engine, extensive liquidity, and a wide array of digital asset products, Binance provides diverse services ranging from trading and finance to research, education, payments, and institutional offerings. Our mission is to promote financial freedom and access globally.

Role Description

Amid the rise of large language models (LLMs) transforming core business functionalities like dialogue and trading, we are looking for a seasoned evaluation algorithm engineer to develop robust evaluation systems. This position will focus on building trustworthy, scientific evaluation methodologies to gauge model quality, facilitating precise R&D guidance for continuous model advancements in various scenarios including financial trading and conversational AI.

Key Responsibilities

• Develop comprehensive end-to-end evaluation strategies tailored to business domains such as dialogue systems and financial trading, employing metric frameworks and rubrics that transform subjective assessments into reliable, repeatable, and transparent evaluations.
• Lead the design and construction of specialized evaluation datasets by defining key evaluation metrics and scenarios, instituting high-quality annotation protocols and quality assurance processes, and establishing benchmarks that accurately reflect business requirements and provide discriminative insights.
• Interpret evaluation outcomes to identify model weaknesses and limits; generate actionable recommendations and collaborate closely with algorithm and product teams to incorporate feedback into iterative model improvements.
• Drive the creation and automation of scalable evaluation platforms and toolchains capable of supporting frequent and stable assessment cycles critical for rapid LLM development.
• Partner with cross-functional teams including algorithm, product management, and data specialists to align business objectives with precise evaluation criteria and translate insights into well-defined R&D initiatives.

Qualifications

• Possess a Master's degree or higher in Computer Science, AI, Mathematics, Statistics, or related disciplines, along with a strong algorithmic background and solid understanding of LLM architecture, training, and fine-tuning.
• Demonstrated hands-on experience with LLM evaluation in a commercial setting, having actively contributed to deployment assessments beyond academic or purely offline benchmarks.
• Proficient in various evaluation methodologies including human judgment, automated model-based scoring (LLM-as-judge), and metric calculations, while understanding their limitations and selecting suitable evaluation frameworks for different business use cases.
• Expertise in ensuring representativeness and consistency in evaluation datasets and annotation procedures, guaranteeing credible and scientific evaluation results.
• Advanced Python programming skills with experience in automating evaluation workflows, building benchmarks, or developing evaluation platforms, capable of independently managing data processing, script development, and results interpretation.
• Strong business acumen and interpersonal communication skills to effectively translate evaluation findings into actionable improvement plans and foster collaborative efforts across teams.

Preferred Additional Skills

• Experience with evaluation of dialogue systems, AI agents, or financial-sector LLMs.
• Background in constructing high-quality AI training and evaluation datasets or annotation infrastructures.
• Familiarity with Reinforcement Learning with Human Feedback (RLHF), reward modeling, or preference data processes.

Why Join Binance?

• Influence the future direction of finance technologies.
• Engage with global top-tier talent in a user-focused, flat organizational structure.
• Work autonomously on challenging projects within an innovative and fast-paced environment.
• Enjoy career growth opportunities and continuous professional development.
• Competitive remuneration alongside comprehensive employee benefits.
• Remote-first work policy promoting work-from-home arrangements contingent on team role and requirements.

Equal Opportunity and Privacy Notice

Binance is an equal opportunity employer upholding diversity as a core value. Applicant information may be processed with AI tools for efficiency in recruitment phases, with final decisions made by human evaluators. For inquiries on data processing, please contact Binance directly.