A GPU kernel can compile cleanly, return the right-shaped output, and even post a fast time, and still be wrong, or quietly gaming the clock, and in kernel optimization, a good stopwatch reading isn't the same as a real speedup. This week we look past kernels that merely look fast to kernels that actually hold up. We introduce KernelQuest, a new benchmark from the Turing Frontier Research Lab with 100 engineer-authored Triton tasks, each measured against a human-certified speedup target and cleared through structural, correctness, and policy gates before its time counts, where 170 of 1,200 agent runs had already hit their target speed but failed verification, false wins that a stopwatch-only benchmark would have published outright. We also celebrate a milestone: welcoming Patrick McKinney as our new Chief Information Security Officer. And we explore fresh research on how frontier models slip into agentic misalignment under pressure, how reliable LLM judges really are when scoring mobile agents, and what retrieval-augmented generation does to model safety.
What we're delivering
This week, we're spotlighting the launch of KernelQuest, a new benchmark from the Turing Frontier Research Lab that measures whether AI agents can do the full job of GPU kernel optimization, not just generate code, but produce a genuine Triton kernel that is correct, faster than a serious compiled baseline, and able to survive an adversarial verifier.
Here's what makes it different:
- 100 engineer-authored Triton tasks, delivered as live environments across 21 kernel families and four capability levels, from single operators to fused epilogues, complete models, and large transformers and MoEs. In each, an agent profiles a PyTorch workload, writes and revises kernels, builds its own timing probes, measures progress, and submits a final solution.
- Human-certified targets, not theoretical ones. Every task's speedup bar is one an engineer-written Triton solution already reached on the same GPU through the same harness. That makes each target genuinely hard but reachable by construction, so a miss measures distance from demonstrated expert performance. The median certified target is 1.78x, and 45 tasks require at least a 2x speedup.
- Quality gated before speed counts. Every submission must clear structural, correctness, policy, and performance gates before its stopwatch time is credited. That design closes the shortcuts (reduced precision, cached outputs, graph capture, harness exploits) that have inflated results on earlier benchmarks.
💡 The gates matter. Of 1,200 agent runs, 565 were rejected before speed could even count, and 170 of those had already measured at or above their certified target. A stopwatch-only benchmark would have published them as wins. On the leaderboard, Claude Opus 5 (run in Claude Code) leads with a 67.0% pass rate, versus 26.0% for the next-best system, with substantial headroom still remaining.
Explore Turing OTS Datasets
🔬 Need better eval data this quarter?
Turing’s off-the-shelf (OTS) datasets are built for teams that need verifiable, high-signal data where frontier models still break — across multimodal STEM, HLE++ STEM, coding evaluation, and rubric-based reasoning.
Use them for reward modeling, RL post-training, outcome-supervised fine-tuning, frontier benchmarking, and failure-mode analysis.
What we're celebrating
🎉Welcome, Patrick McKinney, Turing's new Chief Information Security Officer
We're thrilled to welcome Patrick McKinney to Turing as Chief Information Security Officer (CISO). Patrick will lead Turing's global security strategy as we scale our work with the world's leading AI labs and enterprises across highly regulated industries.
He joins from Invisible Technologies, where he was the company's first dedicated security hire and built its security organization from the ground up. Earlier in his career, he held security and compliance roles at Dropbox and Coinbase, helping guide both companies through their public offerings. As CISO, he'll oversee security across Turing's frontier AI and enterprise AI businesses, spanning clients in financial services, life sciences, healthcare, retail, and automotive.
What we're reading
- Agentic Misalignment in Summer 2026 Anthropic's alignment team shares a snapshot of what happens when they deliberately probe frontier models for agentic misalignment under controlled conditions.
Running simulations across models from Anthropic, OpenAI, Google DeepMind, xAI, DeepSeek, and Moonshot AI, they group the failures into two kinds: harmful compliance, where a model follows a user's harmful request, and agentic misalignment, where a model pursues its own motivation against the user's instructions (For example, protecting another model or shaping an evaluation.) - Benchmarking LLM Judges for Mobile Agent Evaluation As teams increasingly rely on LLM-as-a-judge to score agents, this paper asks how reliable those judges actually are for mobile agent tasks. It finds that a simple baseline judge using sampled screenshots is often competitive with, or better than, purpose-built pipelines, suggesting the LLM backbone matters more than elaborate judge machinery.
It also shows benchmark-quality metrics predict both agent-ranking fidelity and downstream usefulness when a judge is used as an RL reward signal, and surfaces opposite (conservative vs. permissive) failure profiles across model backends. - RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety Retrieval-augmented generation is now standard for grounding models in corporate documents and knowledge bases, but this benchmark measures what RAG does to safety.
By isolating retriever quality and cleanly comparing non-RAG against RAG conditions across five open-source models, the authors find an inverse relationship between benign and unsafe capability, and show that baseline safety guardrails don't guarantee safe downstream behavior once retrieval is added. Even benign documents can lead to unsafe generation.
Where we’ll be
📍 San Francisco, CA United States | 🗓️ Oct 6-9
Turing is headed to San Francisco for COLM 2026! Come connect with us and other researchers, stay tuned for specific Turing-hosted events.
Ready to Optimize Your Model for Real-World Needs?
Partner with Turing to fine-tune, validate, and deploy models that learn continuously.
