Terminal-Bench 3.0: Hard for the right reasons
Terminal-Bench 3.0 evaluates agents on hard command-line work. Turing contributed to the benchmark, with some of the tasks authored and verified by our team. We share how we built these tasks and where the benchmark is headed.
It's rewarding to see the past few months of work at Turing become part of Terminal-Bench 3.0. But saying Turing "contributed tasks to a benchmark as one of the top data partners" leaves out most of what actually goes into building one.
The biggest change in Terminal-Bench 3.0 isn't just that the tasks got harder. It's that the definition of a good task got much stricter. Difficulty is no longer the sole objective -- it matters more to be hard for the right reasons.
Building to a higher standard
Standard | Terminal-Bench 2.0 | Terminal-Bench 3.0 |
Difficulty philosophy | Difficult, realistic problems | Genuine capability gaps, not arbitrary complexity |
Domain coverage | SWE, sysadmin, security, scientific computing | Any economically valuable computer work that can be performed through a terminal |
Domain expertise | Subject-matter knowledge encouraged | Contributors document professional experience for review |
Compute | Mostly CPU | GPU-enabled tasks (up to one H100) |
Environment | Primarily isolated containers | Multi-container systems, databases, filesystems, microservices |
Verification | Programmatic tests | A verifier that passes if and only if the instruction was completed |
Terminal benchmarks started around git, builds, and test suites. Terminal-Bench 3.0 starts to look much closer to how people actually use computers for work. Expanding beyond software engineering shifts the bottleneck from engineering capacity to domain expertise. GPU-enabled environments make an entirely new class of AI research and engineering workflows benchmarkable. An if-and-only-if verifier raises the standard again, ensuring a passing score reflects genuine capability rather than pattern matching.
What a task consists of
Every task consists of four pieces:
- The instruction: what a competent practitioner would actually be told
- The environment: a Docker environment that recreates the real starting state, whether that's a broken script, production-shaped data, or misleading documentation
- The reference solution: a working solution authored by someone with domain expertise
- The verifier: an isolated grading environment that evaluates only the final outcome
Each task includes written design rationales explaining why it's difficult, how the reference solution works, what the verifier measures, and how much expert effort the task would require without an agent. In practice, writing the verifier specification is where many task design decisions get stress-tested. It forces you to think through what success actually looks like, why plausible-looking solutions should still fail, and whether the task captures the capability you intended to measure.
Proving a task is fair
Building the task isn't the expensive part. Validating it is.
Harbor provides the benchmark-level quality checks: oracle solvability, deterministic execution-based verification, instruction-verifier alignment, reward-hacking resistance, near-miss analysis, and schema validation.
On top of that, Turing built our own validation pipeline. Static validation catches packaging and environment issues before execution. Frontier model evaluations and adversarial probes stress-test tasks under realistic agent behavior. Additional automated and human review validate instruction quality, verifier correctness, determinism, test coverage, and whether failures reflect the intended capability rather than artifacts of the benchmark itself.
Looking beyond the leaderboard
A leaderboard tells you who won, but it doesn't tell you why. The more probing question is what training signals and behavioral patterns these tasks reveal about the models.
When do models confidently declare success despite violating the specification? Which implicit domain rules and expert-level nuances consistently get abstracted into generic solution patterns? When do models optimize for a measurable proxy instead of the actual business objective? Which failures come from harness and infrastructure, and which expose genuine capability limitations?
These questions are harder to answer than a pass @ n rate, and we’ll be sharing more of this analysis in a subsequent post.
Where this goes
I am genuinely (!) excited by where benchmark development is heading.
Not because the tasks are harder, but because they're starting to measure a broader notion of computer use. GPU-native workflows, domain-specific professional work, distributed systems, and long-horizon execution expose capabilities that traditional software engineering benchmarks largely miss. It still feels like we're only beginning to map what competent computer use looks like for frontier agents.
The Terminal-Bench 3.0 leaderboard, sample tasks, environments, and verifiers are all publicly available. The tasks we contributed are part of a broader effort at Turing to build training and evaluation data across ML, post-training, and professional knowledge work. Those same design principles continue to shape how we build training and evaluation data internally at Turing. Contact us to learn more about our research.
