AI/ML Data Operations Manager - TikTok LIVE

About the Role

The global livestream sector has rapidly expanded and serves as a key growth channel for TikTok worldwide, providing users with enjoyment and creators a platform for expression.

We are looking for an AI/ML Data Operations Manager tasked with overseeing the complete quality and operational framework for prompts, policy application, and evaluation datasets that fuel multimodal large language model (LLM) and Agent moderation features. This role demands translating policies and domain expertise into executable model strategies, managing iterative improvements from problematic cases to deployment, and setting standards to ensure consistent, measurable, and scalable output across multiple task queues.

Situated at the nexus of Prompt Engineering, Data Operations, Evaluation Design, and Cross-functional Program Management, this position collaborates closely with teams in Algorithms, Product, Engineering, Data Science, Operations, and Labeling.

Key Responsibilities

• Lead prompt engineering and data operations by managing the entire lifecycle of prompts, skills, and policy settings for multimodal LLM Agents related to video and image handling for TikTok LIVE, including prompt creation, version and release management, online maintenance, and quality control.
• Develop training and evaluation data methodologies: define sampling strategies, construct golden and benchmark datasets, implement blind-label procedures, design offline and online evaluation toolkits, conduct A/B testing, and devise alternatives when instrumentation is unavailable.
• Translate policies and expert judgment into operational prompts and Agent skills, setting standards for prompt writing, author guidelines, criteria for admission, operating procedures, quality benchmarks, explainability, and structured decision frameworks such as chain-of-thought or title/subtitle levels.
• Collaborate with labeling teams to ensure high-quality data acquisition through clear labeling instructions, quality control procedures, alignment with ground truth, identification of edge cases, maintenance of throughput and quality service levels, performing error analysis, dataset refresh activities, and iterative model enhancements.
• Define tooling needs in partnership with Product and Engineering for functionalities including case retrieval, batch testing, prompt debugging, online results backfill, guidance tools, self-service root cause analysis, analysis assistants, and data-iteration workflows.
• Manage evaluation, iteration, and deployment by diagnosing false positives and false negatives, analyzing metrics such as accuracy, risk, false rejection, and coverage, and prioritizing remediation actions with appropriate launch review documentation to support various testing and rollout phases.
• Construct and scale the Prompt Engineering operational system to standardize workflows encompassing intake, sampling, labeling, quality control, evaluation, and release processes.
• Create program tracking and multi-queue delivery structures to ensure prompt and maintenance cycles delivery is predictable and scalable.
• Drive ongoing improvement using production feedback by turning online problem cases, root cause analyses, and policy constraints into actionable prompt and skill updates, reusable playbooks, and clearly differentiate confirmed conclusions from hypotheses.

Qualifications

Minimum requirements:

• Bachelor's degree or higher.
• At least 3 years of experience in AI/ML data operations, prompt engineering, content moderation strategies, annotation operations, technical program management, or related fields.
• Direct experience with designing or managing prompts and workflows for LLM production systems, including training, evaluation, and online inference.
• Demonstrated expertise in building evaluation frameworks featuring golden sets, benchmarks, labeling standard operating procedures, quality control measures, offline and online metrics, and experiment-ready evaluation suites, along with a proactive mindset towards process enhancement.
• Substantial experience working with labeling vendors or internal annotation teams.
• Knowledge in taxonomy and specification design.
• Proven skills in quality control and delivery management.

Preferred qualifications:

• Experience within content governance, Trust & Safety, or systems requiring high-stakes decision making.
• Familiarity with multimodal model evaluation across video, images, text, URLs, or webpage content.
• Knowledge of Agent systems including planners, skills, tools, and structured decision making such as chain-of-thought processes.
• Experience with A/B experimentation, metric breakdowns, numerator/denominator analysis, mix-shift evaluation, and case-level root cause analysis.
• Background in productizing internal tools focused on prompt debugging, case replay, and iterative data platforms.
• Proven record of developing zero-to-one operating systems including SOPs, workflow tools such as Meego or Jira, dashboards, and scalable delivery methods.
• Basic proficiency with SQL and data analysis.
• Python scripting expertise for evaluation pipelines is advantageous.
• Project management credentials like PMP or equivalent are a plus.

About TikTok

TikTok is the premier destination for short-form mobile video, driven by a mission to inspire creativity and bring joy. With headquarters in Los Angeles and Singapore and offices worldwide, TikTok fosters diverse and global teams dedicated to groundbreaking innovation.

Why Work With Us

We prioritize creativity and authenticity and believe in empowering our teams to deliver significant impact. We value humility, curiosity, resilience, and continuous learning and strive to turn challenges into opportunities.

Diversity & Inclusion

We are dedicated to generating an inclusive workforce where diverse perspectives are celebrated, and all employees receive equal respect regardless of their background.