Senior Data Labeling Specialist - Remote (Ottawa)

Overview

Rex.zone is seeking experienced Senior Data Labeling specialists to contribute to comprehensive AI and machine learning model training processes. This full-time remote position, ideally suited for candidates based in Ottawa, Canada, involves rigorous involvement in data labeling, reinforcement learning from human feedback (RLHF), prompt and quality assurance evaluation, as well as auditing the quality of training data primarily for large language models and computer vision tasks.

Key Responsibilities

• Perform expert-level data annotation across text, image, and multimodal datasets ensuring high accuracy and consistent decisions.
• Conduct RLHF-style preference labeling and ranking based on detailed rubric guidelines.
• Execute prompt and quality assurance evaluations to detect error patterns and policy shortcomings.
• Audit the quality of training data through sampling, identifying errors, and conducting root cause analysis.
• Enforce adherence to annotation protocols; suggest clarifications and examples for challenging cases.
• Assist in named entity recognition (NER)/span labeling and content safety classification tasks as needed.
• Mediate disagreements among annotators, align annotator calibrations, and promote stable agreement metrics.
• Keep detailed documentation of labeling decisions and criteria to support downstream model training.
• Engage in creating gold standard datasets, consensus labeling, and disagreement analysis.
• Maintain quality control systems including spot checks, batch validation, error tracking, and corrective actions.
• Evaluate datasets for large language models involving rubric scoring and identifying adversarial or safety issues.
• Where applicable, perform computer vision annotations such as bounding boxes, polygons, and segmentation.

Candidate Profile

• Proven experience in data labeling and annotation with a strong ability to handle edge cases and maintain labeling consistency.
• Familiarity and comfort working with RLHF concepts, prompt evaluations, and quality assurance assessments.
• Capable of providing mentorship to peers and effectively communicating when annotation guidelines conflict with real-world data.