AI Benchmarking Specialist, Performance Benchmarking Evaluation

Overview

At Amazon, our aim is to enhance the support provided to sellers and customers by enabling quicker resolution of queries through AI. This role is integral as the team benchmarks multiple AI models by evaluating their outputs for relevance, accuracy, and completeness.

Key Responsibilities

• Execute routine annotation tasks within machine learning data pipelines, choosing appropriate methods and occasionally modifying procedures to meet requirements.
• Serve as the main contact for one or more simple annotation tasks, ensuring high-quality execution.
• Comprehend the interdependencies in ML workflows, articulate their impact on customers, and refine standard operating procedures (SOPs) as needed. Document any revisions, obtain approval, disseminate knowledge, and audit SOP adherence.
• Test new SOPs and tools, providing constructive feedback for enhancement to facilitate integration of these innovations.
• Design data collection frameworks, analyze results, and recommend SOP improvements.
• Aggregate, monitor, and relay progress on critical metrics with stakeholders such as program managers and applied scientists.
• Detect operational inefficiencies related to processes and tools, proposing strategies to boost productivity and quality metrics.

Qualifications

• Must possess a Bachelor's degree.
• Fluent in English across speaking, reading, and writing.
• Proficient with Microsoft Office applications, especially Excel.
• Preferred candidates have 0 to 1 year of related experience involving annotation tasks in machine learning data processes.
• Technical skills in SQL and Python for data analysis are highly desirable.

Additional Information

Amazon values inclusivity and provides workplace accommodations during recruitment and onboarding processes for applicants with disabilities. Candidates are encouraged to consult available resources for support or reach out to their recruiting partner if local accommodations information is unavailable.

Company: ADCI HYD 13 SEZ

Job ID: A10565474