The client needed a benchmark dataset to evaluate LLM performance on tasks requiring:

• Spatial and visual reasoning grounded in diagrams, graphs, and technical illustrations
• Expert-authored question design spanning multiple scientific disciplines
• Resistance to web search and AI model shortcuts
• Structured explanations and gold-standard justifications for preference and reward modeling

Existing VQA datasets are primarily focused on captioning, classification, or object recognition, rather than domain-specific reasoning grounded in images.

Turing implemented a multi-phase creation and validation pipeline purpose-built for evaluating multimodal reasoning through images. The process ensured each ImageQA task was validated by experts and resistant to shortcuts via search or AI models.

1. Expert-level question authoring

• Domain experts, primarily PhDs, designed questions grounded in non-Googleable images: diagrams, graphs, and custom visuals
• Each task required spatial or symbolic reasoning and was formatted as a multiple-choice question with one correct answer and three distractors
• Questions were paired with explanations and source references

2. Multi-stage validation

Expert review:

Two independent domain experts attempted each question without access to the author’s answer key. A question advanced only when both experts independently selected the same answer, and that answer matched the author’s answer key. After comparison with the answer key, experts either confirmed the correct response and provided structured feedback or flagged discrepancies for revision with detailed guidance.

Revision cycle:

• The original author reviewed peer feedback and revised questions for clarity, difficulty, or objectivity
• Only questions that passed expert validation advanced to the next stage

Novice adversarial testing:

• Two independent reviewers from unrelated domains attempted to solve each question using unrestricted web search and general-purpose tools
• Tasks solvable through search engines or general-purpose LLMs were flagged for revision or removal

3. Structural compliance and final QA

• All tasks were reviewed for benchmark-aligned criteria including spatial reasoning, image dependency, non-searchability, objectivity, and answerability
• Questions were formatted for consistency and markdown compatibility, with embedded images and answer logic structured per the client’s standard

This pipeline ensured each task was human-authored, adversarially tested, image-dependent, and validated for multimodal evaluation.

• Over 200 questions curated by PhDs across 20+ spatial-reasoning-heavy domains
• 100% client acceptance rate
• Structured explanations and reference links provided for every answer
• Used as a benchmark to evaluate frontier multimodal models

The resulting dataset provides a new standard for multimodal evaluation, offering:

• Image-grounded tasks for spatially grounded, symbolic, and scientific reasoning
• Signal for identifying failure modes in multimodal models
• A vetted pool of questions used to evaluate frontier multimodal systems across disciplines

Every task met defined criteria for image dependency, reasoning depth, domain correctness, and adversarial robustness.