A leaderboard tells you who won, but not why. This week we look past the score at how capable agents actually fail. On Turing's hardest Terminal-Bench 3.0 tasks, models increasingly solve the wrong problem, verify against their own misreading, and confidently sign off. We share a failure analysis spanning 3,600 trials that shows why an agent's own checks are weak evidence of correctness, and why independent verification is the harder problem worth solving. We also celebrate a new Turing Frontier Fellows partner in Granola, and explore fresh research on benchmarking agents in a realistic marketplace, executable red-teaming for agent safety, and hypergraph-based failure attribution for LLM reasoning.
What we're delivering
This week, we're sharing a failure analysis from Turing's hardest Terminal-Bench 3.0 tasks, a look past the leaderboard at why capable agents fail. Across the 45 most challenging tasks, we ran 3,600 trials (10 harness and reasoning configurations, 8 attempts each) and labeled every agent-attributable failure by both what the task trapped and what the agent claimed happened.
The pattern isn't weak execution. It's confident failure: agents solve the wrong problem, verify against their own misreading, and sign off.
- Failure has moved upstream. Only 1.8% of failures happen at delivery. The other 98.2% produce a clean, usable artifact that's simply wrong on content, most often a scope error (which rows, which namespace, which reading of a written rule).
- An agent's own checks barely predict correctness. 76.4% of passing trials ran a validation command, versus 76.8% of failing ones. Agents actually asserted success more often when they were wrong (84.3% vs 80.6%).
- 65% of failures end in confident false completion. The agent runs its checks, reads them as passing, and declares the task done. The clearest case, missing an explicitly stated numeric bar, reaches a 77% confident sign-off rate.
💡 More checking isn't the fix. When the solution and the check come from the same misreading, only independent verification (evidence that can falsify the agent's own interpretation) separates real success from confident failure.
Explore Turing OTS Datasets
🔬 Need better eval data this quarter?
Turing’s off-the-shelf (OTS) datasets are built for teams that need verifiable, high-signal data where frontier models still break — across multimodal STEM, HLE++ STEM, coding evaluation, and rubric-based reasoning.
Use them for reward modeling, RL post-training, outcome-supervised fine-tuning, frontier benchmarking, and failure-mode analysis.
What we're celebrating
🎉 Turing Frontier Fellows × Granola
The Turing Frontier Fellows program, an invite-only community bringing together domain experts and $500K+ in AI tools and learning experiences, has a new partner: Granola. Fellows now receive three months of Granola Business for up to five team members, in addition to Claude, Wispr Flow, and Airtable. Fellows work with Granola Business through hands-on workshops for building AI-native knowledge-work and note-taking workflows.
What we're reading
- Business Arena: Benchmarking LLM Agents in a Realistic Marketplace Researchers argue the next evaluation frontier is environments that are noisy, non-stationary, and lack a single verifiable answer, and build Business Arena to test it: a marketplace simulation where success isn't binary and the world keeps moving while the agent acts. It pairs skill-level metrics with an attribution toolkit that traces realized gains and losses back to the specific actions that produced them, giving fine-grained credit assignment rather than one aggregate score.
Across 15 frontier models, mean final net worth ranges from $20,856 to $188,488, a wide spread showing how far apart leading models still are on sustained business judgment. Mechanism ablations check whether high scores reflect genuine competence or simulator-specific shortcuts. - REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems Researchers note that agent-safety evaluation often collapses into a single attack-success rate, conflating real violations with what's merely visible. REDAgentBench instead derives attacks from explicit safety constraints, runs them in isolated service sandboxes, and confirms harm from service receipts and final-state changes, spanning 1,661 cases across five service surfaces.
The headline result is practical: a training-free "policy reminder" cuts confirmed violations by more than 70 percentage points in matched replay. The authors argue executable evaluation both sharpens safety measurement and points to concrete places to intervene. - HPFA: Hypergraph-Based Paired Failure Attribution for LLM Reasoning Researchers tackle why a reasoning trajectory failed. Existing attribution methods either run expensive step-by-step counterfactual tests that scale poorly, or flatten reasoning traces and ignore their non-linear logical dependencies. HPFA instead pinpoints the root-cause step by comparing the hyperedges of a failed reasoning path against a reference successful one.
On mathematical-reasoning and agentic-coding tasks, HPFA improves both attribution accuracy and efficiency, and the trained attributor also lifts reasoning accuracy at test time, outperforming prior attribution baselines. It's a useful companion to this week's confident-failure analysis: a way to locate the exact point where an agent's reasoning went wrong.
Where we’ll be
🔹 Turing After Hours | AI Researchers in Mountain View
📍 Mountain View, United States | 🗓️ Aug 19
Join Turing for an evening of networking with researchers and engineers advancing frontier AI across reasoning, coding, multimodality, and beyond.
Stay ahead with AGI Advance
Turing is leading the charge in bridging AI research with real-world applications. Subscribe to AGI Advance for weekly insights into breakthroughs, research, and industry shifts that matter.
Ready to Optimize Your Model for Real-World Needs?
Partner with Turing to fine-tune, validate, and deploy models that learn continuously.
