Remote Jobs RockRemote Jobs Rock

Human Data Evals Lead (Remote/US/LATAM)

πŸ“… Jun 19
AI/ML DataLLM BenchmarkingCode-model EvaluationModel Evaluation
Apply Now

πŸ“œ Description

  • Turn public benchmarks, evaluation targets, and lab requests into proposals and sample packages that demonstrate capability.
  • Design benchmark samples with subject-matter experts, including verified ground truth, reasoning trajectories, multi-model evaluations, and rigorous quality controls.
  • Recruit, brief, calibrate, and review experts across coding, agentic and tool-use tasks, and STEM reasoning.
  • Serve as a direct contact for lab partners, communicate their needs, and coordinate with the CEO and subject-matter experts.
  • Own pilots end to end, from scoping and statements of work through staffing, production, quality control, and delivery.

πŸ› οΈ Requirements

  • Originated data or benchmark proposals for AI labs, translated eval targets into sample tasks that demonstrate capability, and owned the engagement through delivery.
  • Deep evaluation and quality expertise: LLM benchmarking, with real strength in code-model evaluation.
  • Built QC processes and artifact standards that met enterprise or lab requirements, and set a quality bar a team of experts was held to.
  • Thrives in ambiguous, fast-moving environments where the rules are still being written, and delivers under pressure.
  • 5+ years in technical delivery, quality, or program management, with recent experience in AI/ML data, model evaluation, or benchmarking.
  • Hands-on experience delivering data or evaluation work to AI labs or enterprise ML teams, scoping through delivery.
  • Working fluency with how frontier models are evaluated: benchmarks, rubrics, pass rates, headroom, and what makes a task discriminate a model.
  • Proven people/vendor leadership, you've recruited, calibrated, and held a team or expert pool to a quality standard.
  • Fluent English. Spanish is a nice to have.
Full job description

Reports to: CEO

Owns: data proposals, sample development, quality, and pilot delivery

Location: Remote / Latam / US


The role

You will own Anyone AI’s data initiatives and proposals to AI labs, from the data proposal or responding to requests, through pilot delivery. You own how we build proposals and develop the sample packages and benchmarks: frontier-grade packages across reasoning, coding, agents, and tool use, multi-modal and others, produced in collaboration with subject-matter experts, with expert-verified ground truth, multi-model headroom results, and QC that survives buyer-side scrutiny. You are the person who designs the sample that demonstrates our quality, converts pilots into production engagements. On a small team, this is the operational center of the Human Data Division.

Responsibilities

  • Proposals & requests. Study public benchmarks and eval targets, and turn them into proposals and sample packages that demonstrate capability and win the work. Respond to lab data requests and pilots.

  • Sample & benchmark development. Design and build the sample packages, working with subject-matter experts. Every package meets the bar of our current sample set:

    • Expert-verified, exact-match-checkable ground truth and gold reasoning trajectories.

    • Multi-model evaluation showing real headroom, and proof the task discriminates the model, not just that it's hard.

    • Rigorous QC structure: calibration layers, severity-weighted rubrics, deterministic verifiers, evidence maps, etc.

  • Subject-matter experts. Recruit, brief, calibrate, and review a pool of experts across coding, agentic/tool-use, and STEM/reasoning. Raise their output to our standard and keep it there; be the arbiter of what "correct" and "frontier-difficulty" mean.

  • Lab relationships. Be a direct point of contact for lab partners on Slack and calls, with support from the CEO and the wider team. Keep senior lab contacts informed, surface what they actually need, and pull in the CEO and subject-matter experts when the conversation calls for it.

  • Pilot delivery. Own pilots end to end: scoping, SOW, staffing, production, QC, and delivery. Nothing ships before it's lab-ready, and nothing comes back rejected as "not frontier-level" without us already knowing why.

Experience

  • Originated data or benchmark proposals for AI labs, translated eval targets into sample tasks that demonstrate capability, and owned the engagement through delivery.

  • Deep evaluation and quality expertise: LLM benchmarking, with real strength in code-model evaluation.

  • Built QC processes and artifact standards that met enterprise or lab requirements, and set a quality bar a team of experts was held to.

  • Thrives in ambiguous, fast-moving environments where the rules are still being written, and delivers under pressure.

Qualifications

  • 5+ years in technical delivery, quality, or program management, with recent experience in AI/ML data, model evaluation, or benchmarking.

  • Hands-on experience delivering data or evaluation work to AI labs or enterprise ML teams, scoping through delivery.

  • Working fluency with how frontier models are evaluated: benchmarks, rubrics, pass rates, headroom, and what makes a task discriminate a model.

  • Proven people/vendor leadership, you've recruited, calibrated, and held a team or expert pool to a quality standard.

  • Fluent English. Spanish is a nice to have.

6sense

Staff Data Scientist, Forward Deployed

πŸ•’ 18 days ago
6senseπŸ‘₯ 501 - 1000 employees🏒 Computer Software

Lead customer-facing machine learning engagements from discovery through production, tailor predictive models to customer data and go-to-market needs, and turn recurring solutions into reusable product capabilities.

Machine LearningPredictive ModelingXgboostRandom Forest
Wiley-VCH

Principal Data Scientist

πŸ•’ 25 days ago
Wiley-VCHπŸ‘₯ 11 - 50 employees🏒 Publishing

Own end-to-end NLP and applied AI modeling for scientific research intelligence, building and evaluating production pipelines that enrich millions of journal articles and support agentic applications.

PythonNLPLLMsTransformers
Conga

Staff AI Data Scientist

πŸ•’ 29 days ago
CongaπŸ‘₯ 1001 - 5000 employees🏒 Computer Software

Research, build, evaluate, and deploy NLP, machine learning, and Generative AI capabilities for commercial products, taking solutions from experimentation to production while shaping technical direction and mentoring colleagues.

PythonNatural Language ProcessingMachine LearningGenerative AI
Life360

Staff Decision Scientist, Direct Revenue (AI Native)

πŸ•’ 4 days ago
Life360πŸ‘₯ 501 - 1000 employees🏒 Individual & Family Services

Lead decision science for subscription revenue, shaping pricing, packaging, new offerings, and international expansion through rigorous causal analysis and executive recommendations while mentoring analysts and advancing AI-enabled.

Statistical InferenceCausal InferenceA/B TestingPricing Experiments
Mercury

Senior and Staff Data Scientist - Credit and Lending ML

πŸ•’ 10 days ago
MercuryπŸ‘₯ 10,000+ employees🏒 Financial Services🀝 B2B

Build and own production machine-learning models for small-business lending, turning financial data into defensible credit decisions while shaping underwriting policy and lending products with Credit and Product teams.

Machine LearningPythonSQLData Warehousing

Trusted by Remote Workers