- Experience
- Any
- Salary
- USD 30 – USD 50 / hour
- Openings
- 1
- Posted
- 3 days ago
- Work mode
- Work from home
- Resume
- Required to apply
Sign in to tell us what does and doesn't work for you here — it sharpens every match we show you.
Job description
About the Role
We are seeking an experienced Senior Data Annotator to remotely support AI and machine learning training workflows. The primary focus is to generate and verify top-notch annotated data which will serve as the foundation for training and assessing models. Tasks include annotation under strict guidelines, quality assurance, reinforcement learning from human feedback preference evaluations, and prompt assessments to enhance large language model training pipelines involving natural language processing, computer vision, and content safety.
Key Responsibilities
- Carry out advanced annotation and data review tasks for textual, visual, and multimodal datasets.
- Conduct reinforcement learning human feedback (RLHF) preference rankings and evaluate large language models based on established criteria.
- Perform quality assurance evaluations, audits, and calibration sessions to maintain data integrity.
- Detect deficiencies in annotation guidelines, record atypical cases, and suggest improvements and clarifications to rubrics.
- Track and resolve differences in annotations through inter-annotator agreement monitoring and calibration procedures.
- Assist with evaluating prompts to ensure they are helpful, safe, and comply with relevant policies.
- Contribute to labeling related to content safety and the management of sensitive subjects.
- Collaborate with engineering and data operations teams to enhance annotation tools, increase throughput, and improve error detection mechanisms.
Required Qualifications
- Proven experience in annotating data, labeling datasets, or performing quality assurance tasks for machine learning systems.
- Ability to apply and adhere to detailed rubrics and maintain strict compliance with annotation guidelines.
- Knowledge of RLHF methodologies, ranking preference data, or evaluation procedures for large language models.
- Strong written analytical skills for making judgment calls and documenting complex or edge cases.
- Experience in natural language processing annotation tasks such as named entity recognition, and/or computer vision annotation tasks including bounding boxes and segmentation.
Quality Assurance Approach
Quality is governed through audits, inter-annotator reliability checks, rubric compliance, calibration exercises, and monitoring of error rates to safeguard the performance of downstream models.
Compensation
The position offers a competitive wage ranging from $30 to $50 per hour (USD).
Level
Senior