Terminal-Bench 3.0: Why capable agents fail confidently

Two patterns in agent failure are already reasonably well established:

1. Successful runs often contain errors that agents recognize and recover from.
2. Failed runs often end without agents recognizing that anything went wrong.

Self-verification has a structural limitation in solving the latter. The solution and the check meant to validate it are both downstream of the agent’s interpretation of the task. When that interpretation is wrong, the mistake propagates through the trajectory: the agent builds the wrong thing, verifies it against the same misinterpretation, and confidently signs off.

Misinterpretation itself isn’t too surprising. What’s more meaningful to observe is how far it propagates, turning the agent’s own verification into confirmation. Figure 1 shows the underlying problem of the solution and self-check inheriting the same reading of the task.

We took the 45 most challenging tasks in our Terminal-Bench 3.0 set, a cut that approximately matches the official Terminal-Bench 3.0 benchmark in difficulty: gpt-5.6-sol + Codex resolves 33.9% of these tasks, compared with 34.4% on the official Terminal-Bench 3.0 leaderboard (the 45-task hard cut).

Across 45 tasks × 10 configurations (harness + reasoning settings) × 8 attempts, we ran 3,600 trials. Of the failures with evidence to classify, 2,980 are agent-attributable after removing harness and verifier faults.

Every failure is labeled along two independent axes. The mechanism, or failure mode, is what the task trapped, determined from the verifier's verdict against the failure mode that task was reviewed for. The claim is the agent's own account of what happened, read from its final message and shell commands.

Almost nothing that reaches the verifier here looks like a run-of-the-mill implementation bug. The six largest subtypes account for 72% of the 2,980 failures, and most are errors about scope: which rows, which namespace, which sources of state, which region of the solution space is legal, and which reading of a written rule.

The code is usually doing a competent job, but on the wrong problem. A regular implementation bug can reveal itself because the implementation deviates from what the agent intended to build. A scope error is different: if the agent's interpretation of the task is itself wrong, the implementation follows that interpretation and the check tests against it, so both can agree while the final output is still wrong.

By the time a trajectory reaches the verifier, the obvious failures have mostly surfaced. Only 1.8% fail at delivery, while the remaining 98.2% produce a usable artifact that is wrong on content, as shown in Figure 2.

Whether an agent checked its work tells us practically nothing about whether it was right. 76.4% of passing trials ran a check, compared with 76.8% of failing ones. The other signals point in the wrong direction, with agents asserting success more often when they failed and hedging more often when they passed.

Separately, we labeled what the agent itself believed had happened. 65.4% of failures ended in confident false completion, where the agent ran its own checks, read them as passing, and declared success. This pattern holds across every reasoning failure, with missing the numeric bar the clearest case at 77%.

38.4% of failures are coherent solutions to a different reading of the spec: the agent gets what it covers right, but misses part of what the task actually requires. The largest group is predicate-scope errors, with 528 failures across 8 tasks where the computation is correct but runs over the wrong set.

In one clinical derivation task, the agent applies the right treatment-emergent predicate to a population the protocol scoped differently, producing "Y" where the correct flag is "N". It then verifies the output against its reading of the rules and declares success.

Other failure subtypes include 395 failures that treat two identifier spaces as one and attach entities to the wrong owner, while another 220 translate a rule stated in domain prose into the wrong formula.

Agents can pass their own checks and still be confidently wrong. 28.8% of failures take the right general approach but omit a requirement explicitly named in the spec. The most common case, 285 failures across 4 tasks, is a second source of state that was never opened.

In a replicated key-value store, deletes are recorded as tombstones so that a replica that missed the delete cannot resurrect an old value during reconciliation. The agent never consulted that state, so a deleted key came back alive:

read_repair_consults_local_graveyard: expected None, got v1

The agent's final message reads:

all 14 end-to-end tests pass, with identical output across 5 consecutive runs.

None of those runs involved a replica that had missed a delete.

Other cases include constraints stated but never enforced, dropped boundary rules, non-reversible transforms, and unimplemented edge cases.

22.1% of failures produce a well-formed artifact but miss a numeric threshold stated explicitly in the task. These should be relatively easy to catch, yet at 77%, they are the failures agents are most likely to check and confidently sign off on.

462 failures across 7 tasks land on the wrong side of a declared threshold. In one forecasting task, the agent reports “2,400 rows, no nulls, sensible levels,” while the verifier returns:

Bias exceeds threshold

Everything the agent checked was true, but none of it was the actual instructed bar. The task asked whether forecast error stayed below a specific threshold, which the agent never computed. Another 123 failures optimize for parity rather than the required accuracy, while 74 produce rendered output outside the allowed diff.

The same pattern gets harder to catch under re-entry and concurrency, where 9.0% of failures are correct on a clean pass but break when operations interleave, retry, replay, or run in a different order. A clean pass is exactly what the agent can observe, while the verifier tests whether the invariant survives beyond it.

Three tasks account for 196 of these failures, all holding serially but breaking under interleaving. The fourth, with 71 failures, comes from single-process nondeterminism involving garbage collection, weak references, and hash-seed ordering. At 90%, it has the highest confident sign-off rate of any task in the corpus.

Failure mechanisms remain remarkably stable across configurations despite a 4.7× spread in capability. What varies is whether the system recognizes the failure, with little relationship to capability (Figure 3). This is a configuration-level effect, where the model, harness, environment, task, and verifier jointly determine what evidence gets produced and trusted.

Configurations: gpt-5.6-sol (Codex, max); fable-5 (Claude Code, high); glm-5.2 (Terminus-2, high); opus-4.8 (Terminus-2, xhigh); grok-4.5 (Terminus-2, no reasoning flag); kimi-k2.7 (Terminus-2, max); kimi-k3 (Terminus-2, thinking enabled); inkling (Terminus-2, xhigh); nemotron-ultra (mini-SWE-agent, no reasoning flag); deepseekv4-flash (Mistral Vibe, no reasoning flag).

If the solution and its check come from the same mistaken interpretation, more checking is not enough. The check needs a way to challenge the interpretation that produced the solution.

The common objective is falsification. Verification should be capable of disproving the agent’s reading of the task, not merely confirming it. Independent requirement decomposition, counterexample generation, and separate verifier policies are a few possible ways to introduce that independence, though we do not test them here.

One case captures the argument. In a concurrency task, the agent reports:

all 100 runs are byte-identical (single unique md5)

The verifier answers:

FAIL: output depends on chunk-to-task assignment order

The agent did run the determinism check 100 times, but every run held the chunk assignment fixed. The check was real and its result was correctly interpreted, but every run tested the wrong proposition.

A test derived from a wrong representation of the task cannot establish that the representation is right. And when 98.2% of failures still produce a well-formed artifact, 84.3% arrive with no warning, and self-verification barely separates success from failure, runtime health, output shape, and self-reported confidence are weak evidence of correctness.

Independent verification has to test what the agent did not think to test. For evaluation, that means verifiers and harnesses that introduce evidence independently of the agent’s solution path. For training, the challenge is teaching models to generate that kind of falsifying evidence themselves.

The official Terminal-Bench 3.0 leaderboard, sample tasks, environments, and verifiers are all publicly available. The tasks we contributed are part of a broader effort at Turing to build training and evaluation data for increasingly capable agents. The failure modes here continue to shape how we design that data, particularly around independent verification and the kinds of behavior a model’s own checks fail to expose. Contact us to learn more about our research.