Salesforce AI Research developed Hard2Verify, a benchmark to evaluate how well verifiers can assess step-level correctness in long-form math solutions. The benchmark uses open-ended, Olympiad-level problems sourced from recent competitions like IMO, Putnam, and INMO. Responses were generated directly by frontier models such as GPT-5, Gemini 2.5 Pro, and Claude Sonnet 4, with no modifications, preserving natural reasoning errors for realistic evaluation.

Salesforce needed a trusted partner to verify multi-step reasoning chains and the evaluation required:

• Strict binary judgments (Correct/Incorrect) for each reasoning step
• Zero tolerance for carry-forward errors; if one step was incorrect, all dependent steps were marked incorrect
• Short written justifications for each step, explaining correctness judgments
• High reviewer alignment due to the interpretive difficulty of parallel versus dependent reasoning branches

Turing’s role was to execute high-precision annotation, reviewer alignment, and delivery at scale, following Salesforce’s internal evaluation rubric.

Turing evaluated over 200 long-form math responses, each containing 8–15 structured reasoning steps, generated by frontier reasoning models. For every step, annotators issued a binary Correct/Incorrect judgment and authored a short written justification, following a strict zero-tolerance rubric:

• Correct: Logically and computationally sound, aligned with prior steps
• Incorrect: Any flaw, such as logical, procedural, or inherited via carry-forward, led to rejection
• Carry-forward logic: Once a step was marked Incorrect, all dependent steps were also considered invalid
• No partial credit: Incomplete logic, vague claims, or unverifiable “hand-wavy” statements were rejected

Annotation was performed by a team of rigorously vetted experts, including graduates, postgraduates, and PhDs in mathematics, and followed Salesforce's benchmark rubric with multi-round review for accuracy and auditability.

• Annotated 200+ model responses, each broken down and evaluated step-by-step
• Applied error-carry-forward logic and justification writing with full reviewer alignment
• Achieved 100% delivery pass-through via Salesforce’s internal 3-tier review pipeline
• Maintained average annotation time of 90 minutes per model output and 63 minutes per review

Salesforce used this dataset to:

• Evaluate LLM performance on open-domain math tasks with strict reasoning fidelity
• Detect early-stage reasoning flaws that derail multi-step logic chains
• Align model outputs with human-grade reasoning standards for future fine-tuning and scoring research