Enterprise agents must operate across systems with stateful actions, strict policies, and error propagation across workflows. Existing benchmarks focus on short tool sequences or static tasks and fail to capture:

• Long-horizon planning across multiple steps
• Stateful interactions with persistent system changes
• Cross-domain orchestration between tools and data systems
• Policy adherence and access constraints in enterprise environments

ServiceNow aimed to build a benchmark that could evaluate whether agents can reliably complete real workflows, not just generate correct responses.

Turing contributed to the EnterpriseOps-Gym benchmark by designing structured evaluation tasks, execution trajectories, and validation mechanisms aligned with real enterprise workflows.

1. Enterprise workflow prompt design

Turing designed 1,000+ prompts grounded in real operational scenarios, including:

• Customer service case management
• HR workflows such as onboarding and case handling
• IT service operations and incident resolution
• Productivity workflows across Email, Calendar, Drive, and Teams
• Hybrid workflows requiring cross-domain coordination

Each prompt required agents to retrieve data, apply policies, and execute multi-step workflows.

2. Long-horizon execution modeling

Tasks were constructed with 7 to 30 step execution paths, requiring:

• Sequential tool invocation
• Dependency resolution across steps
• State updates across systems
• Correct ordering of actions

This allowed controlled evaluation of how performance changes as planning complexity increases.

3. Expert reference trajectories

Turing produced expert-grounded execution paths for each task, including:

• Step-by-step reasoning aligned to system constraints
• Tool calls with parameters and expected outcomes
• Logged execution traces capturing system interactions

These trajectories served as reference standards for evaluating model behavior.

4. Verifier-based evaluation framework

Instead of evaluating outputs alone, tasks were validated using deterministic verification scripts:

• SQL-based checks confirmed whether workflows completed correctly
• Verification covered task completion, system state integrity, policy compliance, and side effects
• Multiple valid execution paths were allowed as long as the final state conditions were satisfied

This ensured that evaluation measured actual execution success, not just plausible responses.

5. Multi-layer quality assurance

Turing applied a structured QA pipeline:

• Reviewer audits for task correctness and realism
• Feasibility checks to ensure tasks were solvable within the environment
• Consistency validation across prompts, trajectories, and verification logic

This ensured that tasks reflected real enterprise workflows and supported reliable benchmarking.

• Contributed 1,000+ enterprise workflow prompts to EnterpriseOps-Gym
• Enabled evaluation across 8 enterprise domains with realistic system interactions
• Established execution-grounded benchmarking using verifier-based validation
• Modeled long-horizon planning (7–30 steps) to stress-test agent reasoning
• Supported benchmarking of stateful, multi-system workflows rather than isolated tasks

ServiceNow used EnterpriseOps-Gym to benchmark frontier models and uncovered clear limitations in real-world agent performance. Even the top-performing model achieved only 37.4% task completion, with performance dropping further in policy-heavy and cross-domain workflows .

The evaluation revealed that:

• Planning, not tool use, is the primary bottleneck, with human-authored plans improving performance by 14–35 percentage points
• Performance declines with longer workflows, highlighting challenges in maintaining multi-step reasoning
• Policy compliance and safe refusal remain unreliable, even for top models