Senior Product Manager

About the Role

This position is with a partner company seeking a Senior Product Manager based in Ireland to lead the strategy for assessing and enhancing the quality of AI-generated advertising creatives at scale. The successful candidate will be responsible for the entire Evals & Quality product domain, encompassing defining quality standards, developing reliable measurement systems, and driving evaluation methodologies and customer enablement.

Responsibilities

• Lead the overarching vision and roadmap for Evals & Quality, establishing long-term goals, success criteria, priorities, and clear boundaries.
• Develop and refine core quality metrics such as first-pass-right accuracy, evaluation coverage, and regression detection rates.
• Create and oversee evaluation methodologies including rubrics and calibration techniques that align automated judges with human reviewers.
• Design scalable human quality-control workflows, identifying where human input adds significant value and enhancing automated evaluation tools through these insights.
• Manage and maintain the golden dataset, ensuring representativeness across formats and client scenarios, with robust versioning and upkeep.
• Construct reusable evaluation loops enabling product and engineering teams to independently appraise new features and capabilities.
• Oversee the public evaluation interface to maintain high usability, credibility, and design standards.
• Provide evidential quality data to support commercial processes, including RFP responses and creative enablement content.
• Collaborate with Agent Architects to implement dynamic client-specific quality standards using configurable rubrics instead of custom evaluation code.
• Drive the complete product lifecycle from initial discovery and hypothesis formation through launch and subsequent review cycles.
• Implement a consistent practice of hypothesis documentation pre-release and outcome review approximately two weeks post-launch.
• Leverage customer insights and product analytics to influence cross-team decisions and continuously enhance evaluation strategies.

Key Requirements

• Proven ownership of quality, evaluation, experimentation, or machine learning measurement products with defined roadmaps, user bases, and measurable impacts.
• Experience defining and quantifying quality in complex or subjective domains lacking definitive correct answers.
• Practical knowledge of golden datasets, evaluation rubrics, calibration of judges, human review processes, and measurement reliability techniques.
• Ability to diagnose deficiencies in measurement systems and implement pragmatic enhancements for improved reliability and scale.
• Strong product discovery capabilities, including customer needs analysis, domain learning, hypothesis testing, and evidence-based decision making.
• Track record of managing metrics that influence decisions beyond immediate teams or functions.
• Comfort engaging directly with clients by presenting methodologies, clarifying evidence, and fielding inquiries from buyers and stakeholders.
• Deep expertise in full product lifecycle management—from discovery and prioritization to launch, measurement, and post-release learning.
• Collaborative aptitude to work effectively with AI, data, design, technical, and customer-facing teams in a fast-paced environment.
• Strong sense of ownership with a focus on achieving measurable outcomes rather than merely delivering outputs.
• Comfort with ambiguity, rapid iteration, and simplifying complex challenges into scalable product patterns.
• Empathy towards users and colleagues, prioritizing usability, workflow quality, and meticulous attention to detail.
• Genuine enthusiasm for AI-driven products, creative technologies, measurement methodologies, and the challenges of evaluating generative AI outputs.

Benefits

• Generous paid time off including 25 vacation days plus public holidays within a flexible leave policy.
• Comprehensive health insurance and private medical coverage.
• Monthly allowance supporting wellness, fitness, and professional development.
• Remote work from anywhere within your home country.
• Flexible working hours accommodating diverse work styles and schedules.
• Enhanced parental leave benefits for birth, adoption, and surrogacy.
• Access to a London office location for eligible UK-based staff.
• Opportunity to work on AI-powered products focused on generative AI, creative tech, evaluations, and advertising.
• Collaborate internationally with multidisciplinary teams and clients worldwide.

Additional Information

This role is offered on behalf of a partner company responsible for managing applications and next steps. The hiring process involves AI-supported candidate matching, but final decisions are made by human reviewers. Applicants' data will be processed according to data protection laws, including GDPR, with rights to access, correction, erasure, and objection available. AI tools may be used to help review applications and identify inconsistencies, but these aid human judgment and do not replace it.