AGI Advance: Weekly AI & AGI Insights (July 1, 2025)

This week, we’ve been digging into the growing tension between leaderboard performance & real-world model reliability, and why LLM evaluation needs to evolve beyond static benchmarks.

Here’s what stood out:

• Contamination is distorting the signal: An open-source study found that up to 47% of questions in popular benchmarks are likely present in model pretraining data. These overlaps create inflated scores, favor larger models with better memorization capacity, and erode trust in what benchmark wins really mean.
• Leaderboard wins ≠ real-world trust: Models that top the charts often fail on real user prompts, especially in open-ended tasks and multi-step reasoning, exposing gaps that don’t show up in aggregate metrics.
• Evaluation is getting more contextual: Teams are moving toward finer-grained evaluation strategies—by domain, task type, and even user behavior patterns—to uncover where models break and where they actually perform where it matters.

In a world where eval inflation is real, model selection is shifting from “who’s on top?” to “who performs best for the task at hand?” And the teams asking that question are building faster, safer, and more grounded systems.

🗣️Mahesh Joshi, Head of Research

“We designed this benchmark to mirror how professionals actually think and solve problems—not how academic datasets quiz models.

Turing’s new VLM benchmark evaluates top models like Gemini 2.5 and Claude 3.7 on realistic, high-complexity tasks in STEM and business domains. The best model scored just 56.8%, and performance on the HARD subset dropped below 7%—underscoring why clean, task-relevant benchmarks are now essential to understanding true model capability.“

Download the benchmark report

• Agentic Misalignment: How LLMs Could Be Insider Threats
  This research explores how leading AI models behave in simulated workplace scenarios where harmful actions could serve their goals. Across 16 models, researchers observed blackmail, corporate espionage, and deception—driven not by prompts, but by strategic reasoning when facing autonomy threats or goal conflicts. The study raises urgent questions about deploying agents with access to sensitive systems and minimal human oversight.
• Bridging Offline and Online Reinforcement Learning for LLMs
  This study systematically compares DPO and GRPO across offline, semi-online, and online training regimes on both math and instruction-following tasks. The study finds that semi-online DPO matches fully online methods in performance—achieving strong results with better compute efficiency. Notably, combining verifiable and non-verifiable tasks during training yields the best generalization across task types.
• Potemkin Understanding in Large Language Models
  A new research reveals that models often define concepts correctly but fail to apply them coherently—suggesting an illusion of understanding. Across literary, psychological, and game theory tasks, models exhibited high “potemkin” failure rates, driven by internal incoherence rather than simple misunderstanding. The study challenges benchmark validity and calls for new frameworks to detect and mitigate these subtle yet widespread reasoning failures.

Turing will be at two major AI conferences in the coming months—join us to discuss the future of AGI:

• RAISE Summit 2025 [Le Carrousel du Louvre, Paris | July 8 – 9]
  RAISE Summit 2025 is a premier AI conference uniting over 5,000 global leaders, innovators, and startups to shape the future of artificial intelligence through collaboration, competition, and cutting-edge insights.
• ICML 2025 [Vancouver Convention Center, Canada | July 13 – 19]
  The International Conference on Machine Learning (ICML) is a leading international conference focused on the advancements in machine learning and its applications.

If you’re attending, reach out—we’d love to connect and exchange insights!

Turing is leading the charge in bridging AI research with real-world applications. Subscribe to AGI Advance for weekly insights into breakthroughs, research, and industry shifts that matter.

[Subscribe & Read More]