Workflow-level grading reshapes enterprise AI benchmarks

TL;DR: Raw model scores are becoming meaningless for enterprise buyers. The real signal is how a model performs across a specific sequence of steps: finding the right filing, running the right calculation, flagging the right exception. Finance is the first vertical to operationalize this, and other enterprise domains will follow.

When Big Finance Bench dropped on May 27, it added some useful texture to our thesis from last month: aggregate model scores don't tell you much about what happens in production. 

The top models were nearly tied in aggregate with a best score of 58.8%. That number is worth knowing, but the real finding is what happened when researchers routed by workflow type and source. Each model was strongest in different tasks, and the aggregate score didn't show that.

The question worth asking now is which model handles this class of problem, at this step, against this source type, at a cost that makes the workflow viable. Finance is the first vertical to produce a benchmark that actually asks it. Every other enterprise domain will need to do the same.

Rogo had 928 questions written by people who actually worked in finance. The agent runs with tools a real analyst uses. The questions require the same retrieval, calculation, and judgment that would apply in a real scenario.

The grading is where it gets interesting. Most benchmarks score the final answer: right or wrong. Big Finance Bench grades against 15,656 rubric criteria spread across every step of each derivation. A model that finds the right 10-K filing but then miscalculates a ratio earns credit for the retrieval and loses it at the calculation. You can see exactly where it failed.

That step-level visibility is what makes this useful for deployment decisions. An aggregate score tells you a model got 58.8% of answers right. Step-level grading tells you it's reliable at document retrieval, inconsistent at multi-period comparisons, and prone to a specific class of calculation error. 

Those are actionable findings. An aggregate score obscures all of it: the retrieval reliability, the calculation errors, the workflow-specific gaps. That's why a model can score 58.8% and still tell you almost nothing useful about whether to deploy it.

The aggregate scores at the top of the Big Finance Bench leaderboard are close, but the models weren't equally good at everything. Each was strongest in different workflow types, and that's where the deployment finding lives.

A coarse router that directed queries by workflow type and source added 4.5 points over the best single model. An oracle router added 13.2. That gap between 4.5 and 13.2 is the design space most enterprise teams haven't touched yet. The floor is roughly where most deployments sit today: one model, applied uniformly, regardless of task. The ceiling is what's available if the system knows which model to use for which problem.

Getting from one to the other requires better architecture around the models you already have.

Routing matters, but it's not the whole architecture. Four other factors show up in any serious finance deployment, and they apply across verticals:

• Routing by workflow and source type. A model that's strong on earnings call analysis may be weaker on covenant compliance. Treating them as the same task leaves points on the table.
• Step-level verification. A final answer that looks correct can rest on a retrieval error or a flawed intermediate calculation. Checking only the output misses where things actually break.
• Cost per correct answer. Cost per API call is the wrong unit. A cheaper model that gets the right answer 60% of the time on a given task costs more than a pricier one that gets it 85%, once you account for the downstream work the failures create.
• Audit trails. Finance has compliance requirements that make this non-optional, but the principle applies anywhere a decision needs to be explained or reconstructed. A model that produces the right answer with no record of how it got there isn't deployable in most enterprise contexts.

Anthropic's financial services agents point to the same design pattern. So does most of what serious practitioners report from production. These four factors are table stakes for any deployment that needs to hold up under real conditions.

Finance didn't end up first by accident. The conditions that make workflow-level evaluation necessary are more visible there.

The data is structured: filings, spreadsheets, earnings calls, regulatory disclosures. The decisions are high-value and auditable. Compliance requirements mean every step in a workflow needs a record. When a model gets something wrong in a credit analysis or a portfolio review, the consequences are specific and traceable. That combination of structured inputs, high stakes, and hard accountability requirements creates pressure to measure model performance the way Big Finance Bench does: at the step level, against real source material, with credit and penalties attached to exactly where things go right or wrong.

Other verticals are close behind, and they'll follow the same logic for the same reasons.

Legal work has the same accountability requirements and similarly structured source material: contracts, case law, regulatory filings. A model handling due diligence or contract review needs step-level verification for the same reasons a finance model does. Healthcare has high-stakes decisions, strict compliance, and source material (clinical notes, billing codes, treatment protocols) that doesn't tolerate the kind of errors a clean benchmark wouldn't catch. Government procurement has long task horizons, complex policy constraints, and audit requirements that map almost directly onto what finance already deals with.

The evaluation design finance is operationalizing sets a standard for other enterprise deployments within a few years. Finance just got there first because the cost of getting it wrong is obvious. As that evaluation design spreads to other verticals, the deployments will follow. And deployments are where the next advantage gets built.

Every serious deployment surfaces failures the benchmark didn't anticipate. A model misreads a filing structure it hasn't seen before. It loses the thread on a 12-step calculation. It applies the wrong judgment to a grey-area compliance question. In isolation, each failure is a problem to fix. Across a deployment, they're a dataset.

Those failures become eval sets. Eval sets become RL environments. RL environments become training data for the next model. That model gets deployed into harder problems, surfaces new gaps, and the loop runs again. 

Most organizations sit on one side of this. Data companies see training and evaluation; they know what went into the model and what it was tested against. They don't see what happens when it meets a real workflow at enterprise scale. Deployment companies see the failures and the edge cases, but they don't have access to the data infrastructure to act on what they're learning. Each side has half the picture.

The organizations that connect both sides are running a faster iteration cycle than those that don't, and that gap widens over time. A model that learns from production failures in finance this year is better positioned to handle the harder credit and compliance workflows that open up next year. The next problem only becomes viable because the current one got solved.

That's the advantage worth watching. The organizations that close it fastest will have something no leaderboard can measure: a model that keeps getting better at the exact workflows their business runs on.

At Turing, we work with leading AI labs to train frontier models, then apply that expertise to enterprise deployments. Sitting between both sides gives us a clearer view of what actually works in production, and where the gap between benchmark performance and real-world results tends to show up. 

We're not tied to any single model. We run evaluations against your specific workflows to find out which model handles your use case best, and where the routing and verification need to happen. 

If you’re working through these decisions, talk to a Turing Strategist.