Senior Data Labeling Specialist - Remote (Ottawa)

About the Role

Rex.zone is seeking experienced professionals for Senior Data Labeling positions to contribute comprehensively to AI and machine learning training workflows. This full-time, remote role based in Ottawa focuses on text, image, and multimodal annotation tasks supporting large language model evaluation and computer vision projects.

Key Responsibilities

• Perform advanced data labeling on diverse data types with precision and consistent judgment.
• Conduct preference labeling and ranking following reinforcement learning from human feedback (RLHF) guidelines.
• Execute prompt and quality assurance evaluations to detect failure points and policy inconsistencies.
• Audit training datasets by sampling, categorizing errors, and investigating root causes.
• Ensure strict adherence to annotation standards and suggest guideline improvements for edge cases.
• Assist with named entity recognition (NER) and content safety annotations as required.
• Manage disagreement resolution, annotator calibration, and maintain inter-annotator consistency.
• Record decision rationales and criteria to support downstream model training functions.
• Create gold-standard datasets, perform consensus labeling, and analyze disagreement patterns.
• Maintain quality assurance mechanisms including spot checks, batch validation, error monitoring, and corrective measures.
• Develop evaluation datasets for large language models incorporating rubric scoring and adversarial safety tests.
• Contribute to computer vision labeling tasks involving bounding boxes, polygons, and segmentation where applicable.

Candidate Profile

• Proven experience in data labeling/annotation with strong capabilities handling edge cases and ensuring consistency.
• Familiarity with RLHF approaches as well as prompt and QA evaluation methodologies.
• Strong communication and mentorship skills to guide peers and resolve guideline ambiguities in practical data scenarios.