Consultant

Role Overview

This position focuses on designing specialized tools for AI agents, creating prompt validation mechanisms, and developing comprehensive automated evaluation systems that ensure high levels of accuracy and safety in AI outputs.

Primary Responsibilities

• Design semantic boundaries, create tailored system instructions, and develop schemas enabling external tool execution for individual AI agents.
• Develop and operate automated pipelines that systematically test active prompt configurations against datasets to detect any regressions.
• Quantitatively assess and monitor key performance metrics such as precision and recall, addressing failure modes including entity misclassification, context leakage, and formatting inaccuracies.

Tools and Platforms

• Utilize Azure AI Studio and Prompt Flow for AI model design and management.
• Work with Azure AI Content Understanding and AI Builder platforms.
• Employ Python programming, Jupyter Notebooks for experiments and analysis, and Git for version control.