Data Annotator

About the Role

Join an innovative company with over 115,000 inspired minds dedicated to advancing digital transformation for some of the world’s leading brands. We combine technology with human ingenuity to revolutionize how businesses operate, striving to improve the world through relentless innovation and continuous learning.

We invite passionate individuals to apply for the Data Annotator position, supporting the development, evaluation, and enhancement of AI systems, including large language models (LLMs), by performing detailed data annotation and content assessments. This role emphasizes complex annotation tasks, prompt creation, and involvement in setting and refining quality and evaluation standards.

Key Responsibilities

• Perform annotation, evaluation, and validation duties on AI-powered systems following established guidelines and quality benchmarks.
• Validate AI outputs against predefined golden questions and expected results to maintain accuracy.
• Maintain, create, and review annotated datasets essential for ongoing AI model assessment and enhancement.
• Test and assess responses generated by Databricks Genie to ensure accuracy, relevance, and consistency.
• Develop and run SQL queries to verify responses generated by AI against source data.
• Analyze any discrepancies between the expected data, source information, and AI-generated results and document these findings.
• Lead efforts to design, optimize, and standardize annotation techniques and prompt strategies across different AI use cases.
• Work closely with AI teams to establish annotation schemas, taxonomy, evaluation frameworks, and quality control standards.
• Review complex or sensitive AI outputs and provide expert feedback and quality recommendations.
• Maintain consistent, traceable, and auditable annotation and evaluation documentation.
• Create reusable prompt frameworks and define best practices aligned with compliance and responsible AI initiatives.
• Assess prompt efficiency and annotation quality metrics to identify areas for improvements using data insights.
• Set quality benchmarks, conduct calibration processes, and manage escalation protocols.
• Identify and address AI risks such as bias and hallucinations, suggesting appropriate mitigation strategies.
• Support governance, compliance, and ethical AI efforts within the organization.
• Participate in calibration meetings to align on annotation standards, quality expectations, and evaluation criteria.
• Collaborate with AI engineers, data scientists, product managers, and business stakeholders to optimize processes.
• Drive continuous improvement by learning new AI technologies, annotation techniques, and delivery methods.
• Provide mentorship and coaching to junior and intermediate annotators and prompt engineers.

Candidate Profile

• A bachelor’s degree in fields such as Communications, Information Science, Computer Science, Data Analytics, Linguistics, or a closely related discipline; equivalent experience accepted.
• Fluent in both written and spoken English with strong command of grammar, syntax, semantics, and content evaluation.
• Ability to understand complex business inquiries and interpret data-informed responses.
• Background in data annotation, content review, quality assurance, testing, or similar fields.
• Proficient in writing, debugging, and optimizing SQL queries.
• Experience reviewing AI, generative AI outputs, large language models (LLMs), or conversational AI systems.
• Proficient in validating data against source systems and investigating inconsistencies.
• Understanding of software quality assurance methods, testing procedures, and defect management.
• Experienced in creating detailed test plans, cases, and validation documentation.
• Strong analytical capabilities and meticulous attention to detail.
• Self-driven and able to work both independently and collaboratively in dynamic environments.
• Excellent communication and stakeholder engagement skills.
• Capable of managing confidential and sensitive information with utmost professionalism.

Preferred Skills

• Familiarity with prompt engineering, AI evaluation frameworks, and annotation platforms.
• Experience with Databricks Genie or comparable natural language query and AI analytics tools.
• Background in data validation, business intelligence, analytics, or data quality initiatives.

Diversity and Inclusion

The employer is committed to equal opportunity, valuing diversity, inclusion, respect, integrity, customer focus, and innovation. All applicants are considered without regard to race, color, religion, gender, age, nationality, marital status, veteran status, sexual orientation, physical or mental disability, or any legally protected characteristics.