🕒 7 days ago
Build and maintain reliable automated tests for desktop and mobile applications, integrate them into CI/CD pipelines, and perform hands-on functional, exploratory, and regression testing.
AI Agentic Tester
📅 Aug 5
AI Agentic TestingFunctional TestingGenerative AI TestingLarge Language Models (llms)
📜 Description
- Perform functional testing of AI agent workflows and end-to-end AI applications.
- Evaluate LLM responses for accuracy, relevance, consistency, and hallucinations.
- Test RAG pipelines, prompt engineering, and AI agent orchestration.
- Run API tests using Postman, Swagger, and REST APIs.
- Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
- Validate multi-agent interactions, tool calling, and workflow execution.
🛠️ Requirements
- AI Agentic Testing
- Functional Testing
- Generative AI Testing
- Large Language Models (LLMs)
- AI Agents & Agentic AI
- Retrieval-Augmented Generation (RAG)
- Prompt Engineering Validation
- AI Model Validation & Evaluation
- API Testing (Postman, REST APIs, Swagger)
- Test Automation (Selenium / Playwright / PyTest)
Full job description
AI Agentic Tester
Denver, MO (Remote)
Must-Have Skills
* AI Agentic Testing
* Functional Testing
* Generative AI Testing
* Large Language Models (LLMs)
* AI Agents & Agentic AI
* Retrieval-Augmented Generation (RAG)
* Prompt Engineering Validation
* AI Model Validation & Evaluation
* API Testing (Postman, REST APIs, Swagger)
* Python
* Test Automation (Selenium / Playwright / PyTest)
* AI Evaluation Metrics (Accuracy, Hallucination, Relevance, Consistency)
* End-to-End Workflow Testing
* Multi-Agent Testing
* Azure OpenAI / AWS Bedrock / Gemini
Core Responsibilities
* Perform functional testing of AI agent workflows and end-to-end AI applications.
* Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
* Test RAG pipelines, prompt engineering, and AI agent orchestration.
* Execute API testing using Postman, Swagger, and REST APIs.
* Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
* Validate multi-agent interactions, tool calling, and workflow execution.
* Apply AI testing methodologies and evaluation metrics to ensure response quality.
* Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.
Similar jobs
Search more QA Engineer jobs
🕒 12 days ago
The QA Engineer strengthens product quality across mobile apps and backend services through risk-based exploratory testing, maintainable automation, API validation, and release readiness reviews.
🕒 13 days ago
Test completed campaign builds end to end, manage QC cases from intake to documented close, and communicate defects so teams can resolve issues before campaigns go live.
🕒 yesterday
Own quality across a product area by reviewing requirements, directing AI-driven test agents, exploring for defects, verifying daily releases, and improving test coverage, environments, and QA tooling.
QA Analyst II
🕒 5 days ago
55564patriot334567software868575745👥 51 - 200 employees🏢 Financial Services
🕒 5 days ago
Own software quality across multiple applications by planning and executing front- and back-end tests, investigating defects, and reporting test coverage and progress with agile development teams.