Off-the-shelf datasets

Based on SWE-bench: SWE-bench++, a scalable, contamination-resistant software-engineering benchmark addressing weak test oracles and dataset aging, to evaluate and train models on real issue resolution.

Based on SWE-bench: verified human-in-the-loop solution trajectories for the hardest SWE-bench tasks, capturing full problem-solving paths and human corrections to train agentic software-engineering models.

Based on LiveCodeBench: coding tasks graded into difficulty tiers via a Difficulty Reference Score from frontier-model (Gemini, Sonnet, GPT) pass@5 results, to evaluate and train code generation across difficulty.

Based on MLE-bench: 2,305 Kaggle-style machine-learning engineering tasks packaged as Harbor directories for end-to-end agentic evaluation and training on real ML competition workflows.

Based on FinanceBench: an enterprise-finance agent benchmark of expert-authored, multi-step tasks over real 10-Ks, 10-Qs, spreadsheets, and deal materials, requiring long-context reasoning and sandboxed tool use.

Rubric-verified Finance OTS: expert-crafted set measuring LLM multi-step financial reasoning over messy real-world documents across 16+ finance domains, with answers validated against expert grading rubrics.

Expert-crafted evaluation and training set measuring LLM multi-step financial reasoning over messy real-world documents across 16+ domains, from investment banking to corporate finance.

Computer-use annotation (CUA) datapack capturing structured representations of real human computer interactions—app setup, feature navigation, end-to-end workflows—to train and evaluate computer-use agents.

Based on SciCode: SciCode++, 5,000+ expert-designed scientific-coding tasks built to challenge frontier models, to evaluate and train advanced scientific programming and computational problem-solving.

PhD-level multimodal STEM reasoning dataset across physics, math, biology, and chemistry with deterministic, auto-gradable answers over single-image, multi-panel, and caption-free multi-image formats.

Based on Humanity's Last Exam (HLE): self-contained, verifiable advanced STEM reasoning problems across physics, math, biology, and chemistry requiring multi-step symbolic reasoning, to evaluate frontier models.

Based on Humanity's Last Exam (HLE): PhD-level multimodal STEM reasoning problems across physics, math, biology, and chemistry, designed to break frontier models, for advanced multimodal evaluation.