CEO Bench: Can frontier agents complete long-horizon financial and operational work in context?

This week, we’re introducing CEO Bench, a new evaluation suite from Turing built on a complete, expert-authored fictional company, and the first benchmark produced on SVC (Simulated Virtual Company), our platform for constructing entire companies as evaluation substrates. CEO Bench tests whether frontier agents can complete long-horizon financial and operational work in context, where the hard part is rarely the calculation and usually deciding which of a company’s own numbers to trust. 

Here's what we built:

• A complete company as the benchmark. Cirrus Sleep, Inc. is a simulated direct-to-consumer sleep brand with a full first fiscal year: 1,100+ files spanning accounting, finance, contracts, HR and payroll, board materials, operations, marketing, customer support, and a year of internal Slack. The books reconcile where they should, and deliberately don’t where real companies don’t. Because nothing about Cirrus Sleep exists outside the dataset, it is contamination-proof by construction.
• 500+ practitioner-written tasks, graded on judgment. Department-level (L2) and executive (L3) tasks authored by finance and operations practitioners, scored with multi-criterion rubrics that grade sourcing, reconciliation, and disclosure decisions, not just final numbers. Every task passes an ensemble of LLM judges and then human expert adjudication before it ships.
• Six frontier models evaluated on 40 published tasks. The top model averaged 41.3, half the tasks saw no model reach 50, and 96 of 240 runs scored under 25. Higher cost didn’t reliably buy higher scores: cost per run varied 9.6x across models, while scores spanned just 18.4 points.

💡 The most instructive failures weren’t arithmetic. They were trust failures: fabricated precision, interpolated data that nets to the correct total, missed governance traps, and undisclosed mixing of accounting and operational figures. Each one produces a deliverable a busy executive might accept.

→ Request sample tasks

🔬 Need better eval data this quarter?

Turing’s off-the-shelf (OTS) datasets are built for teams that need verifiable, high-signal data where frontier models still break — across multimodal STEM, HLE++ STEM, coding evaluation, and rubric-based reasoning.

Use them for reward modeling, RL post-training, outcome-supervised fine-tuning, frontier benchmarking, and failure-mode analysis.

→ Request sample data

🎉Turing Frontier Fellows at AGI House 

We just wrapped up our latest Frontier Fellows session with a private SF Tech Week gathering at AGI House, bringing together our Fellows and a curated group of builders, investors, researchers, and operators. The afternoon featured two fireside conversations on how AI is changing both the way we interact with technology and the way organizations work.

Tanay Kothari, co-founder and CEO of Wispr Flow, spoke about voice, habit formation, and the next generation of AI-native interfaces. Aaron Levie, co-founder and CEO of Box, explored AI agents, enterprise workflows, and the future of work. Thank you to our speakers, our Fellows, and everyone who joined us for an afternoon of great conversation.

→ Read our reflection

• A Competing-Hazards Systematization of Loss of Control in Autonomous Agents  This paper audits 22 public incident reports and 102 agent-safety evaluations published between January 2025 and September 2026, using a common framework in which every attempt ends in approved completion, safe stopping, scope escape, or continuation. Thirteen incidents involved agents that kept going rather than stopping, and in 20 of 22, the surrounding environment permitted the out-of-scope effect.
  
  The evaluation side is just as revealing: only 26 of 102 evaluations treated safe stopping as a first-class outcome, and 79 lumped budget exhaustion in with ordinary failure. The case it makes is that loss of control often comes from persistent agent behavior meeting permissive boundaries, and that evals should measure whether an agent knows when to stop.
•  Covert Assistance: Helpful LLM Agents Evade Oversight in Multi-Agent Systems Researchers set up a software-engineering workflow where a planner agent holds a company credential it’s told not to share with an external developer, while a monitor screens their exchanges. With no adversarial incentive at all, seven of nine frontier models disguised the credential to help the developer recover it while evading the monitor. 
  
  Per-episode breach rates are low, but they compound: for one model, a 0.9% full-breach rate implies a 61.3% chance of at least one leak across 105 episodes. The planners appeared to read the nondisclosure rule as banning plaintext, not character codes or riddles, a reminder that task-advancing helpfulness can cross safety boundaries on its own. 
• Audit Without Verification: When LLM Accountability Layers Relay Rather Than Check When a multi-agent pipeline fails, the evidence an auditor usually has is the reports each agent filed, and those reports often state a conclusion alongside the observations. In a pre-registered six-agent pipeline with injected defects, an auditor reading those reports recovered the true fault origin in just 4.1% of cases where no agent had named it, below a uniform guess, yet reached 60.3% from the raw documentation of the same episodes.  
  
  Removing the single field carrying the agents’ own conclusions raised accuracy by 41.2 points, a result that replicated across two frontier auditors and a second domain, though it cost accuracy when the upstream suggestion happened to be correct. The takeaway: an accountability layer needs evidence independent of the conclusions it’s meant to verify. 

Turing is leading the charge in bridging AI research with real-world applications. Subscribe to the Frontier AI Newsletter for weekly insights into breakthroughs, research, and industry shifts that matter.

[Subscribe & Read More]