Existing code benchmarks rarely test model solvability across realistic patch distributions. The client sought to stress-test their model's ability to resolve real-world bugs, rather than hand-picked or synthetic examples.

Key challenges included:

• Finding testable, standalone issues in mature Java repositories
• Ensuring issue descriptions were clear but not solution-revealing
• Balancing solvable and hard samples while staying within a real PR distribution
• Controlling for data leakage and trivial patches

Repository and task identification

Turing identified 20+ open-source GitHub repositories that met strict criteria:

• ≥95% code written in Java
• Maven-based builds with standardized test output paths
• ≥50 PRs modifying both test and source code
• Pass/fail test behavior aligned with patch application

Each PR met the following validation criteria:

• Include meaningful logic changes (excluding trivial renames or version bumps)
• Contain reproducible test failures that passed after patch application
• Avoid changes to high-level interfaces and class structures in ways that risk compilation errors
• Provide a self-contained issue or derivable problem from the PR content
• Originate from pull requests created after January 2024

Dockerized execution and test validation

For each selected repository, the team created a dedicated Docker image capable of running the full test suite in a clean, isolated environment. This ensured consistent behavior across machines and eliminated environment-specific variance.

Within this environment, the team:

• Executed tests against the original commit to confirm reproducible failures
• Executed tests against the patched commit to confirm resolution
• Recorded both fail-to-pass and pass-to-pass outcomes
• Verified that compilation succeeded before and after patch application

Only pull requests exhibiting stable, reproducible test behavior were retained.

Issue description curation

When no original issue was linked to the PR, Turing’s trainers:

• Authored new issue descriptions based on the PR diff and intent
• Wrote problem-focused, not fix-guiding, descriptions
• Added context only when necessary for reproducibility
• Aligned descriptions with the test diff to preserve evaluability without revealing the solution

This format intentionally diverged from prior benchmarks (like SWE-bench), where issue hints can bias solutions. Prompts were designed to state the problem clearly without implying the fix.

Task balance

To maintain evaluation granularity:

• ≈30% of tasks were designated as model-solvable
• ≈70% exposed failure points, including complex logic changes, ambiguous context, or multi-file patches

• Delivered a 200-task Java dataset containing real bugs balanced for solvability and realism
• Authored neutral, reproducible prompts across PRs with and without original issues
• Used automated patch and test validation pipelines, ensuring fail-to-pass and pass-to-pass logic
• Enabled controlled evaluation of model successes and failure modes
• Delivered Dockerized, test-ready samples integrated with the client’s patch evaluation framework

This dataset enables model builders to evaluate performance across real-world Java software tasks:

• Tests model patching ability on authentic bugs
• Distinguishes between solvable tasks and hard breakpoints
• Reveals failure patterns in LLM-generated code under controlled conditions
• Supports deeper analysis of generalization, instruction following, and patch realism

Trainer-authored prompts, paired with reproducible test cases, make the dataset a valuable resource for fine-tuning, evaluation, and failure-mode diagnosis.