This week, we introduce CompanyBench, Turing's long-horizon benchmark and training dataset built on five years of a real fintech company's operational history. By evaluating agents on realistic enterprise workflows, CompanyBench reveals where today's frontier models still fall short and provides the training signal needed to build agents that can complete real work from start to finish. We also celebrate Wispr Flow joining the Turing Frontier Fellows program and highlight new research on multimodal recommendation agents, agent skill optimization, and reducing reasoning hallucinations.
What we're delivering
This week, we're introducing CompanyBench, Turing's long-horizon benchmark and training dataset for enterprise knowledge work, built on five years of a real fintech company's operational history: hundreds of database tables, thousands of files, and millions of Slack messages, emails, and Jira tickets.
Across 64 challenging enterprise tasks run 10 times each, today's leading models still struggle; GPT-5.5 resolves 44% and Claude Opus 4.8 resolves 36%. Four failure patterns emerged consistently:
- Guessing instead of consulting documentation
- Searching for documents that don't exist
- Applying filters that silently remove valid data
- Producing the correct answer but failing to complete the final step such as posting to Slack or filing a ticket
Enterprise AI needs more than strong reasoning. It needs judgment, discipline, and thoroughness to complete real work from start to finish.
Explore Turing OTS Datasets
🔬 Need better eval data this quarter?
Turing’s off-the-shelf (OTS) datasets are built for teams that need verifiable, high-signal data where frontier models still break — across multimodal STEM, HLE++ STEM, coding evaluation, and rubric-based reasoning.
Use them for reward modeling, RL post-training, outcome-supervised fine-tuning, frontier benchmarking, and failure-mode analysis.
What we're celebrating
🎉 Wispr Flow × Turing Frontier Fellows Program
The Turing Frontier Fellows program, which brings together more than $500,000 in AI tools and learning experiences has a new partner: Wispr Flow.
Frontier Fellows receive three months of Wispr Flow Pro, along with exclusive workshops on building voice-first AI workflows, including how to use Wispr as an input layer alongside tools like Claude Code and Codex.
What we're reading
- Seeing and Reflecting: Multimodal Memory-Enhanced Agent Collaboration for Recommendation
Researchers introduce MMEACR, a recommendation framework that combines LLM-based agent reasoning with multimodal memory to better model user preferences. The system maintains separate User and Item Memory Agents that evolve through attribute-guided reinforcement and reflection, while a parallel multimodal embedding track preserves fine-grained text–image signals. The two rankings are fused using Reciprocal Rank Fusion (RRF) to improve both interpretability and recommendation quality.
Across three Amazon domains (CDs, Cell Phones, and Fashion), MMEACR consistently outperforms strong LLM and agent-based baselines. On the Fashion dataset, it improves NDCG@1 by 45.45%, NDCG@5 by 23.33%, and MRR by 27.14% over the strongest baseline, while also reducing inference time by 6–16% compared to AgentCF. Ablation studies further show that attribute-guided memory updates and iterative agent collaboration are key contributors to performance. - Compression, Structure, and Executor Capability: A Controlled Real-Cost Decomposition of Language-Model Agent Skill Optimisation
Researchers conduct a controlled study of 1,200 agent rollouts across 40 software engineering tasks to isolate the effects of skill compression, structured formatting, scoped loading, compiler model, and executor model on both task success and real monetary cost. Rather than relying on token counts, the study measures actual API costs and quality simultaneously, finding that most skill optimization strategies fail to deliver meaningful cost savings or performance gains.
The results show that executor capability is the dominant factor: upgrading the executor model improves pass rate by 27 percentage points, but at roughly 5× higher real cost. In contrast, deterministic skill shortening performs similarly to the raw baseline without proving non-inferiority, while structured rendering, scoped loading, and stronger compiler models provide no robust improvement and often reduce performance on compact models. - Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLMs
Researchers identify context-sensitive factual hallucinations, where LLMs produce factual errors during multi-step reasoning despite already knowing the correct facts. Analyzing reasoning traces shows this is the dominant hallucination type, accounting for about 67% of hallucination cases across Qwen3 reasoning models. To address this, they propose SSC-GRPO, which assigns step-level reinforcement learning rewards based on the self-consistency of individual reasoning steps across multiple rollouts, without relying on external knowledge.
Across Qwen3-4B-Base, Qwen3-4B-Instruct, and Llama3-8B-Instruct, SSC-GRPO achieves the best overall performance on both math reasoning and hallucination benchmarks, outperforming GRPO-based baselines by up to 1.8%. It also reduces the proportion of context-sensitive factual hallucinations from 22.8% to 16.9% on Qwen3-4B-Instruct while steadily increasing reasoning consistency throughout training.
Where we’ll be
🔹 Turing After Hours | AI Researchers in Mountain View
📍 Mountain View, United States | 🗓️ Aug 19
Join Turing for an evening of networking with researchers and engineers advancing frontier AI across reasoning, coding, multimodality, and beyond.
Stay ahead with AGI Advance
Turing is leading the charge in bridging AI research with real-world applications. Subscribe to AGI Advance for weekly insights into breakthroughs, research, and industry shifts that matter.
Ready to Optimize Your Model for Real-World Needs?
Partner with Turing to fine-tune, validate, and deploy models that learn continuously.
