Evaluating AI systems for software Q&A requires high-quality, real-world tasks. The client needed a dataset that could:

• Capture non-trivial programming questions grounded in real repositories
• Annotate both question quality and answer completeness
• Ensure traceability from answer claims to documentation and code
• Include code executability checks across multiple programming languages and environments

Synthetic Q&A pairs or single-review annotations were insufficient. The dataset needed to meet academic publishing standards and support both benchmarking and fine-tuning research.

Dataset 

Turing curated 500+ GitHub issue-answer pairs using a structured, three-stage annotation strategy designed to assess repository-specific question quality, answer completeness, and reasoning depth:

• Languages covered: Python, Java, TypeScript, and JavaScript
• Sources: Real GitHub repositories and discussions
• Targets: Instructional Q&A covering repo-specific bugs, behaviors, and best practices

Each sample included:

• A rewritten version of the original GitHub question
• A structured answer divided into atomic claims
• Direct traceability to repository documentation (not web links)
• Code snippet validation and executable test status where applicable

Evaluation

To ensure annotation integrity, Turing implemented a multi-layer QA process:

Level 1: Question quality assessment

Annotators evaluated each GitHub discussion for:

• Relevance to the repository
• Learning value via implementation-level depth
• Clarity and self-contained framing

Level 2: Answer quality assessment

For accepted discussions, annotators assessed the provided answer for:

• Coverage of all question aspects
• Clarity of explanation, including reasoning and methodology
• Executable code, if included
• Code-to-claim traceability, ensuring assertions matched repo docs or code comments

Level 3: Question-answer rewrite and reasoning trace

• Rewrote questions to include all constraints and context
• Extracted atomic claims into a short answer list and rewrote a long, fluent answer
• Labeled each claim by importance and reasoning complexity
• Collected supporting evidence from the repository and documentation to ensure answers were self-contained and grounded in accessible context

Each sample was annotated by three to five contributors, followed by a final resolver who consolidated feedback, adjudicated disagreements, and ensured alignment with rubric standards.

• Created a 500+ sample QA dataset with structured annotations and full repo grounding
• Applied a multi-annotator consensus model to improve objectivity and research alignment
• Ensured every answer maintained claim-to-documentation traceability 
• Verified code quality and executability through language-specific test setups across Python, Java, JavaScript, and TypeScript

This project produced a benchmark-ready dataset enabling researchers and model developers to:

• Evaluate AI assistants on real-world, traceable GitHub Q&A tasks
• Analyze instruction-following failures across code, reasoning, and documentation layers
• Train or fine-tune models on multi-layer verified tasks to reduce hallucinations and off-target completions
• Extend coverage to new languages or domains using the provided QA rubric

The multi-layer annotation pipeline ensures the dataset remains precise and reproducible, meeting the standards required for rigorous model evaluation.