The client needed a hard-tier RL task suite that could train and evaluate agents on real codebase-onboarding workflows: locating where a concept is implemented, tracing a data flow across files, understanding an architecture, or diagnosing a defect.

Agents needed to complete these tasks using native repository-exploration tools, including file reading, content search, file-pattern matching, a read-only shell, Git tools, and a planning scratchpad, rather than the templated bash-wrapper harnesses common in other coding benchmarks.

This required:

• Reproducible, verifiable task packages that could be run consistently across training and evaluation infrastructure
• Coverage across multiple reasoning families, not just bug-fixing
• A verifier design robust enough to reward genuine exploration and penalize reward-hacking or shortcut-taking
• A human-authored gold standard rigorous enough to support grading and downstream training

Turing deployed a team of software engineers with codebase deep-research experience, supported by Python engineers for verifier and harness work and dedicated QA reviewers, to design, author, and validate the environment end to end.

1. Task family design and taxonomy coverage

Tasks were authored across four families, each targeting a distinct reasoning pattern:

• Root-cause & behavioral diagnosis: tracing a reported failure or symptom through code path, runtime behavior, and configuration to identify its cause.
• PR review reasoning: reconstructing the code change a reviewer was requesting, or judging whether a reviewer's concern was valid and properly addressed.
• Test & invariant reasoning: identifying missing test coverage and unstated invariants with cross-module implications.
• Design tradeoff analysis: explaining why an approach changed or what design tension a review history reveals.

2.  Reproducible read-only exploration workspace

Tasks ran in a standardized, lightweight Linux Docker workspace containing Git and the required repository-exploration tools.

The relevant repository snapshot was added to the workspace for each task. No application dependencies were installed, and the repository itself was not configured for application or test execution; the workspace was designed specifically for read-only code discovery and analysis.

A consistent container image, fixed repository state, standardized tool surface, and read-only access supported reproducibility across runs while preventing repository or host-filesystem mutation.

3. Weighted, multi-stage verifier design

Each task's verifier combined several layers rather than a single pass/fail check:

• A workspace-integrity gate confirming that the repository remained unmodified
• A weighted rubric spanning six verifier types: exact match, contains, regex, keyword set, LLM rubric, and scripted verifier checks
• Deterministic grounding scoring confirming that the agent's answer was supported by evidence it had gathered, using file and line-range matching within a tight tolerance
• A multiplicative distractor penalty reducing the score for unnecessary reliance on tools or exploration paths designated as distractors

This layered design was deliberately built so that a run could not be gamed through tool-call volume alone. The substantive answer, evidence grounding, and distractor avoidance were evaluated independently.

4. Human-authored gold trajectories

Domain experts performed the code discovery themselves inside the repository workspace. They used the available exploration tools to inspect files, search repository content, trace code paths and Git history, gather supporting evidence, and develop the final answer. Their tool calls, evidence, reasoning notes, and conclusions were captured as the task's gold trajectory.

The authoring interface supported the recording and structuring of this work, but the investigation and final judgment remained human-led.

5. Hard-tier difficulty calibration

Rather than mixing difficulty levels, the task suite was calibrated to a single Hard tier. Pass@10 sampling was conducted against an agreed calibration model, producing more than 5,000 model-generated rollouts. These rollouts provided broader sampling coverage for profiling task difficulty, estimating pass-rate signals, and identifying tasks that required recalibration.

6. Multi-layer quality assurance

Every task moved through a four-role review workflow: trainer, reviewer, calibrator, and auditor. Tasks were evaluated against a structured set of acceptance gates covering schema and identifier integrity, verifier hygiene, prompt realism, reward-hackability, and gold-answer coherence.

A human calibrator also periodically re-scored samples to compare automated and human assessments, helping surface noisy scoring or verifier inconsistencies before the tasks reached production.

• Delivered more than 1,000 codebase exploration RL tasks using a standardized Docker-based exploration workspace, with each task including its repository content, verifier package, and human-authored gold trajectory
• Produced more than 1,000 human-authored gold trajectories; all trajectories included in the final delivery cleared the gold-validation gate and scored above 99.9% through the full verifier pipeline
• Generated more than 5,000 model rollouts through pass@10 sampling to improve sampling coverage, profile task difficulty, and support hard-tier calibration

The client received a reproducible, framework-independent RL task suite for codebase exploration, comprising a standardized read-only Docker workspace, task format, verifier pipeline, grounding logic, and reward-composition methodology.

With hard-tier calibration, a layered anti-gaming verifier design, and a rigorous human-authored gold-validation standard, the task suite was built to produce genuine training signals rather than metrics that saturate or reward shortcut-taking.

This foundation supports:

• Training and evaluating agents on realistic, multi-step codebase reasoning rather than templated bug-fix benchmarks
• Distinguishing model strength cleanly across a wide capability range
• Extending into additional repository sources or task families without redesigning the underlying task and verifier architecture
• Feeding directly into supervised fine-tuning and reinforcement learning pipelines