Automation Test Engineer (AI & Chatbot Testing)

Overview

We are seeking a seasoned Automation Test Engineer to enhance and validate AI-driven customer support and troubleshooting systems. This role demands expertise in automating tests, API validation, AI performance assessment, and ensuring the robustness of conversational AI solutions.

Key Responsibilities

• Create automated test cases targeting AI chatbots and Retrieval-Augmented Generation (RAG) frameworks.
• Assess the precision, pertinence, and effectiveness of AI-generated responses.
• Quantify AI quality indicators, including hallucination frequency and answer relevance.
• Conduct regression testing following updates to AI models or their knowledge databases.
• Automate tests for troubleshooting procedures and decision workflows.
• Confirm the integrity of complete customer support processes through simulation.
• Develop and perform automated API testing to verify integrations with backend systems.
• Build mock services and prepare relevant test data as needed.
• Perform load, performance, and security evaluations on conversational AI platforms.
• Examine AI safeguards against prompt injections and unsafe replies.
• Identify and troubleshoot performance and stability problems.
• Embed automated testing within Azure DevOps CI/CD pipelines and maintain quality metrics and reports.
• Collaborate extensively with developers, AI specialists, and product teams to ensure quality outputs.

Qualifications

• Minimum of 4 years’ involvement in Quality Assurance Automation or Software Development Engineer in Test (SDET) capacities.
• At least 1 year dedicated to testing Artificial Intelligence components, chatbots, large language models, or agentic AI.
• Proficient programming skills in Python, TypeScript, or C#.
• Hands-on experience with RESTful APIs, GraphQL, WebSockets, Postman, and WireMock.
• Familiarity with Azure DevOps and continuous integration/continuous delivery workflows.
• Understanding of Retrieval-Augmented Generation, semantic search, and vector database concepts.
• Additional advantage if experienced with MQTT protocols or IoT telemetry data.
• Preferably experienced in embedded systems or consumer electronics environments.