AI Quality & Evaluation Engineer, AI product

About Woven by Toyota Woven by Toyota is enabling Toyota’s once-in-a-century transformation into a mobility company. Inspired by a legacy of innovating for the benefit of others, our mission is to challenge the current state of mobility through human-centric innovation — expanding what “mobility” means and how it serves society. Our work centers on four pillars: AD/ADAS, our autonomous driving and advanced driver assist technologies; Arene, our software development platform for software-defined vehicles; Woven City, a test course for mobility; and Cloud & AI, the digital infrastructure powering our collaborative foundation. Business-critical functions empower these teams to execute, and together, we’re working toward one bold goal: a world with zero accidents and enhanced well-being for all. ========================================================================= ========================================================================= Important Points ・All interviews will be arranged via Google Meet, unless otherwise stated. ・The same job descriptions are available in both English and Japanese; therefore, we kindly ask that you apply to only one version. ・We kindly request that you submit your resume in English, if possible. However, Japanese resumes are also acceptable. Please note that, depending on the English proficiency requirements of the role, we may request an English version of your resume later in the process. WHAT WE OFFER ・Competitive Salary - Based on experience ・Work Hours - Flexible working time ・Paid Holiday - 20 days per year (prorated) ・Sick Leave - 6 days per year (prorated) ・Holiday - Sat & Sun, Japanese National Holidays, and other days defined by our company ・Japanese Social Insurance - Health Insurance, Pension, Workers’ Comp, and Unemployment Insurance, Long-term care insurance ・Housing Allowance ・Retirement Benefits ・Rental Cars Support ・In-house Training Program (software study/language study) Our Commitment ・We are an equal opportunity employer and value diversity. ・Any information we receive from you will be used only in the hiring and onboarding process. Please see our privacy notice for more details.

TEAM

WHO ARE WE LOOKING FOR?

RESPONSIBILITIES

Define quality dimensions for AI products, including accuracy, consistency, safety, fairness, and UX

Conduct scenario-based testing, exploratory testing, and red teaming

Analyze LLM outputs to identify behavioral trends and failure patterns

Design quality evaluation processes using user feedback such as logs, ratings, and inquiries

Standardize evaluation processes and structure them in a reusable, scalable manner

Automate quality evaluation workflows

Collect and aggregate evaluation data

Establish mechanisms for continuous and recurring quality checks

Build and operate quality feedback loops in collaboration with development, MLOps, and data engineering teams

Document quality issues and risks, and communicate them clearly to relevant teams

MINIMUM QUALIFICATIONS

3+ years experience in QA or testing with products that use LLMs

Experience in test design, including defining test perspectives and creating test cases

Ability to evaluate systems with ambiguous specifications or non-unique “correct” answers by establishing clear judgment criteria

Experience automating test or evaluation processes

Experience collaborating with development teams to drive quality improvements

Business-level proficiency in both English and Japanese

NICE TO HAVES

Experience in scenario testing and red teaming

Experience evaluating and analyzing user feedback and logs

Proactive mindset with the ability to identify problems independently

Basic understanding of natural language processing, particularly in the context of the Japanese language