This a Full Remote job, the offer is available from: United States, Latin America, United Kingdom, France, Uruguay, Brazil, Ecuador, Colombia, Argentina, Chile, Peru, Mexico, Delaware (USA)
Research Scientist, Evaluations
Anyone AI Labs — Human Data Division
Reports to: CEO · Remote / LatAm / US
The role
You will own how Anyone AI measures frontier model capability. This is a research role at heart: you decide what a good evaluation is, design the benchmarks that prove it, and defend the methodology under lab scrutiny. You'll build frontier-grade evaluation packages across reasoning, coding, agents, tool use, and multi-modal — grounded in expert-verified truth, validated against multiple models, and QC'd to survive buyer-side review.
Responsibilities
• Evaluation research. Turn public benchmarks and eval targets into original evaluation designs. Own the hard questions: construct validity, discrimination, headroom, and contamination.
• Benchmark development. Build evaluation packages with subject-matter experts, each with expert-verified ground truth, multi-model headroom results, and rigorous QC (calibration layers, severity-weighted rubrics, deterministic verifiers).
• Experts. Recruit, calibrate, and review a pool across coding, agentic/tool-use, and STEM/reasoning. Be the final arbiter of correctness and frontier difficulty.
• Lab relationships. Be a technical point of contact for labs, with CEO support. Understand what they're trying to measure and translate it into an evaluation design.
• Delivery. Turn lab requests into winning sample packages, then own pilots end to end. Nothing ships before it's lab-ready.
What we're looking for
• Research background in ML evaluation or benchmarking — published/open benchmarks, eval research, or equivalent hands-on work labs relied on.
• Deep LLM benchmarking expertise, with real strength in code-model evaluation.
• Fluency with how frontier models are measured: rubrics, pass rates, headroom, contamination, and what makes a task discriminate a model.
• Proven ability to hold a team or expert pool to a rigorous standard.
• Fluent English. Spanish a nice to have.
This offer from "Anyone AI" has been enriched by Jobgether.com and got a 77% flex score.