FEATURES AND BENEFITS

Real engineering work, not just coding puzzles

Real-world tasks to evaluate and improve model reasoning across the entire software development lifecycle.

Reasoning datasets

Competitive programming tasks and rubric-aligned prompts that evaluate logic depth, planning, and correctness in code.

Full coding traces

Stepwise code generation prompts with human-verified chain-of–thought traces, useful for reward modeling and SFT.

Multimodality

Applied coding problems with multimodal inputs including image, video, audio, and molecular structures.

CAPABILITIES

Off-the-shelf and custom datasets

Browse our catalog for software engineering.

Software Engineering

SWEBench Public

BenchmarkBug FixingDockerizedRepository Tasks

pass@1:35%volume

Based on SWE-bench: SWE-bench++, a scalable, contamination-resistant software-engineering benchmark addressing weak test oracles and dataset aging, to evaluate and train models on real issue resolution.

SWEBench Trajectory

BenchmarkDockerizedHuman-in-the-loopSWE-bench

Based on SWE-bench: verified human-in-the-loop solution trajectories for the hardest SWE-bench tasks, capturing full problem-solving paths and human corrections to train agentic software-engineering models.

LiveCodeBench

BenchmarkCoding ProblemsCompetitive ProgrammingLiveCodeBench

pass@5:33%volume

Based on LiveCodeBench: coding tasks graded into difficulty tiers via a Difficulty Reference Score from frontier-model (Gemini, Sonnet, GPT) pass@5 results, to evaluate and train code generation across difficulty.

MLE-Bench++

BenchmarkData ScienceDockerizedKaggle

Mean Reward:0.3529volume

Based on MLE-bench: 2,305 Kaggle-style machine-learning engineering tasks packaged as Harbor directories for end-to-end agentic evaluation and training on real ML competition workflows.

RL environments

MCP tools, computer-use, browser-use, and terminal  environments for software engineering.

Software engineering