Remote Jobs RockRemote Jobs Rock

QA Engineer (AI & Feature Testing)

🕒 3 days ago
📍 🇧🇷 Brazil - Remote🎸 Senior🧳 QA Engineer💼 Full-Time
PlaywrightSoftware TestingAI And LLM TestingAI Evaluation
Apply Now

📜 Description

  • Design and execute test plans, cases, and scripts for software features and AI-powered components.
  • Evaluate AI and LLM outputs for accuracy, relevance, robustness, edge cases, and safety using metrics, benchmarks, and human review.
  • Choose deterministic or evaluation-based testing methods to match expected feature behavior.
  • Perform functional, integration, regression, performance/load, and user acceptance testing across web, mobile, and APIs.
  • Document defects and investigate issues using browser developer tools, application logs, and debugging utilities.
  • Maintain test records and collaborate with engineering and product teams to improve QA practices and responsible AI quality.

🛠️ Requirements

  • Professional experience: 5–7 years of hands-on software testing experience across web applications, mobile platforms, and APIs.
  • Playwright expertise: Proven production experience using Playwright for software testing. This is a primary requirement for the position.
  • AI and LLM testing: Practical experience testing AI-powered products or LLM-integrated features, including evaluating accuracy, relevance, robustness, adversarial prompt handling, edge cases, and safety or toxicity risks.
  • Testing methodology: Clear understanding of the differences between deterministic testing, which relies on defined inputs and expected outputs, and evaluation-based testing for probabilistic AI systems.
  • AI evaluation techniques: Familiarity with human-in-the-loop evaluations, scoring metrics, benchmark datasets, semantic evaluation, and other approaches for assessing AI-generated responses.
  • Software testing fundamentals: Demonstrated experience with functional, integration, regression, performance/load, and user acceptance testing.
  • Web and mobile development knowledge: Strong understanding of application development lifecycles, including web technologies and platform-specific considerations for iOS and Android.
  • Debugging and diagnostics: Hands-on experience with browser developer tools, application logs, and technical investigation techniques for identifying and resolving defects.
  • Test management and documentation: Advanced proficiency in Jira and Confluence, along with experience using test management systems to organize test cases, execution results, and defect tracking.
  • Analytical thinking: Strong attention to detail and the ability to investigate complex issues, recognize unexpected patterns, and distinguish between software defects and acceptable variations in AI-generated outputs.

✨ Benefits

  • 100% remote work: Work remotely from Brazil while collaborating with an international technology team.
  • International collaboration: Work with colleagues across countries and cultures, contributing to projects with global reach.
  • Meaningful AI projects: Help build and validate real-world AI-powered solutions and contribute to the evolution of AI-enabled products.
  • Professional development: Strengthen your expertise in software quality assurance, advanced testing methodologies, and AI/LLM evaluation.
  • Continuous learning: Join an environment that encourages technical development, knowledge sharing
  • Responsible AI focus: Participate in work that emphasizes ethical AI standards, product reliability, and user trust.
  • Career growth opportunities: Expand your technical capabilities by working on challenging projects spanning software engineering, AI
  • Jobgether - QA Engineer (AI & Feature Testing)
Full job description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a QA Engineer (AI & Feature Testing) based in Brazil.

Join a technology-driven team developing and validating innovative features for a leading personal finance platform, combining traditional software testing with next-generation AI quality assurance. As a QA Engineer, you will take ownership of quality across web applications, mobile platforms, and APIs, ensuring reliable, secure, and seamless user experiences. You will play a key role in testing AI-powered features, including large language model (LLM) integrations, where outputs can be probabilistic rather than strictly predictable. By combining deterministic testing techniques with AI evaluation methodologies, you will help identify inaccuracies, hallucinations, edge cases, and potential safety risks before they affect users. Working closely with engineers, product stakeholders, and international teams, you will contribute to robust testing strategies and continuously improve quality standards. This role offers an opportunity to apply your hands-on testing expertise to emerging AI technologies while promoting responsible AI development and dependable software delivery. If you are a detail-oriented QA professional who enjoys solving complex problems and exploring the evolving intersection of software quality and artificial intelligence, this opportunity offers meaningful technical challenges and room for growth.

How Jobgether works: We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best!  Why Apply Through Jobgether?    Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.     #LI-CL1

Accountabilities

  • Plan and execute comprehensive testing: Design, develop, and execute test plans, test cases, and test scripts for traditional software features and AI-powered components across web, mobile, and API environments.
  • Validate AI-driven functionality: Test AI and LLM-integrated features for accuracy, relevance, consistency, robustness, and safety, identifying hallucinations, unexpected responses, and failures in edge cases.
  • Apply appropriate testing methodologies: Determine when to use deterministic pass/fail testing and when to apply evaluation-based approaches for probabilistic AI outputs, ensuring test strategies reflect the expected behavior of each feature.
  • Evaluate AI model performance: Assess AI-generated outputs using appropriate evaluation metrics, benchmark datasets, semantic assessments, and human-in-the-loop review processes.
  • Translate requirements into test scenarios: Analyze business requirements, user journeys, process documentation, and AI behavior specifications to develop comprehensive and effective testing coverage.
  • Conduct manual feature testing: Execute functional tests across web and mobile applications, verify expected behavior, validate fixes, and ensure new releases meet quality expectations.
  • Perform regression and integration testing: Validate interactions between application components and APIs while conducting regression, performance/load, and user acceptance testing (UAT).
  • Identify and document defects: Create clear, detailed bug reports that include reproducible steps, expected and actual results, supporting evidence, and relevant technical information.
  • Investigate technical issues: Use browser developer tools, application logs, and debugging utilities to investigate failures, trace unexpected behavior, and support efficient root-cause analysis.
  • Support mobile quality assurance: Validate mobile application behavior across relevant iOS and Android environments, considering platform-specific functionality and potential compatibility issues.
  • Improve QA practices: Contribute to testing standards, methodologies, and best practices for emerging technologies, including AI frameworks and LLM-based applications.
  • Maintain testing documentation: Use Jira, Confluence, and test management systems to track defects, organize test cases, document results, and maintain clear project records.
  • Collaborate across teams: Partner with engineers, functional specialists, product stakeholders, and IT teams to resolve issues, align testing priorities, and ensure quality throughout the development lifecycle.
  • Develop subject-matter expertise: Build a strong understanding of core systems, application interfaces, business processes, and reporting tools to strengthen test coverage and improve the reliability of delivered solutions.
  • Promote responsible AI quality: Help ensure AI-powered features meet appropriate standards for reliability, robustness, safety, and user trust.

    • Requirements

  • Professional experience: 5–7 years of hands-on software testing experience across web applications, mobile platforms, and APIs.
  • Playwright expertise: Proven production experience using Playwright for software testing. This is a primary requirement for the position.
  • AI and LLM testing: Practical experience testing AI-powered products or LLM-integrated features, including evaluating accuracy, relevance, robustness, adversarial prompt handling, edge cases, and safety or toxicity risks.
  • Testing methodology: Clear understanding of the differences between deterministic testing, which relies on defined inputs and expected outputs, and evaluation-based testing for probabilistic AI systems.
  • AI evaluation techniques: Familiarity with human-in-the-loop evaluations, scoring metrics, benchmark datasets, semantic evaluation, and other approaches for assessing AI-generated responses.
  • Software testing fundamentals: Demonstrated experience with functional, integration, regression, performance/load, and user acceptance testing.
  • Web and mobile development knowledge: Strong understanding of application development lifecycles, including web technologies and platform-specific considerations for iOS and Android.
  • Debugging and diagnostics: Hands-on experience with browser developer tools, application logs, and technical investigation techniques for identifying and resolving defects.
  • Test management and documentation: Advanced proficiency in Jira and Confluence, along with experience using test management systems to organize test cases, execution results, and defect tracking.
  • Analytical thinking: Strong attention to detail and the ability to investigate complex issues, recognize unexpected patterns, and distinguish between software defects and acceptable variations in AI-generated outputs.
  • Communication skills: Ability to document technical findings clearly, communicate quality risks effectively, and collaborate with cross-functional stakeholders.
  • Teamwork and adaptability: Comfortable working in a distributed, international environment, coordinating with engineering and functional teams across different locations.
  • Preferred technical experience: Familiarity with HTTP proxy and debugging tools such as Charles Proxy or Fiddler, as well as Xcode logs and Android Debug Bridge (ADB) logs.
  • Preferred industry experience: Previous experience in financial services, fintech, or personal finance technology is an advantage.
  • Preferred education: A bachelor's degree in Computer Science or an equivalent technical qualification.
  • Quality and ethics mindset: Commitment to continuous improvement, responsible AI practices, and delivering reliable, secure, and user-focused software.

    • Benefits

  • 100% remote work: Work remotely from Brazil while collaborating with an international technology team.
  • International collaboration: Work with colleagues across countries and cultures, contributing to projects with global reach.
  • Meaningful AI projects: Help build and validate real-world AI-powered solutions and contribute to the evolution of AI-enabled products.
  • Professional development: Strengthen your expertise in software quality assurance, advanced testing methodologies, and AI/LLM evaluation.
  • Continuous learning: Join an environment that encourages technical development, knowledge sharing, and experimentation with emerging technologies.
  • Inclusive workplace: Contribute to a collaborative culture that values diversity, equal opportunity, and different perspectives.
  • Responsible AI focus: Participate in work that emphasizes ethical AI standards, product reliability, and user trust.
  • Career growth opportunities: Expand your technical capabilities by working on challenging projects spanning software engineering, AI, and enterprise technology.
    • Apply Now

    Similar jobs

    Search more QA Engineer jobs
    View more QA Engineer jobs

    Trusted by Remote Workers