The client needed high-quality preference signals to train and improve multimodal AI systems capable of accurately reasoning over images. Unlike text-only RLHF, image-grounded evaluation introduces unique challenges: responses must be assessed not only for linguistic quality but for factual grounding in visual content that may require active research, reverse image search, or domain knowledge to verify.

Key challenges included:

• Verifying factual accuracy against visual evidence, including charts, diagrams, scientific graphs, and structured data, where model responses could appear plausible while misreading key values or visual relationships
• Establishing consistent preference standards across subjective dimensions such as coherence, signal-to-noise ratio, and instruction-following, where both models might be correct but differ in quality of reasoning or structure
• Preventing annotation drift across a large-scale workforce operating on diverse prompt types, from mathematical problem solving and data interpretation to creative description and open-ended information seeking
• Handling edge cases systematically, including prompts answerable without the image, blurred visuals, non-English image text, and cases where neither model response was correct

Turing deployed a structured annotation and review pipeline built around factuality-first evaluation, graded preference reasoning, and a multi-layer QA process designed to maintain signal quality across a high-volume, subjective task type.

1. Task validity and image-relevance screening

Before evaluation began, annotators screened each task for validity. Tasks were marked improper and excluded if the image failed to load, if both responses could not be assessed, or if the prompt could be answered without the image. This last filter ensured that every evaluated task genuinely required visual reasoning, preserving the integrity of the preference signal for multimodal training.

Tasks with non-English image text were handled by translating the content rather than rejecting the task, ensuring broad coverage without sacrificing evaluability.

2. Factuality-first response evaluation

Each response was evaluated independently for factual correctness before a preference was selected. Annotators were expected to verify claims against the image using reverse image search, the image filename, and external sources where necessary.

For descriptive tasks, a structured tolerance rule was applied; responses with fewer than 20% inaccurate observations relative to correct ones were still marked factually correct, preventing over-penalization of minor errors while flagging responses with substantively misleading content.

Mathematical and scientific responses required step-by-step verification, not just final-answer checking, with LaTeX rendering validated using a dedicated formatting tool.

3. Structured preference selection and reasoning

Annotators selected from a five-point preference scale, ranging from "much better" to "both equal", with the "both equal" option reserved strictly for cases where no differentiating factor could be identified.

When both responses were factually correct, a priority-ordered tiebreaker framework guided preference decisions: logical flow, coherence and structure, signal-to-noise ratio, and finally intuition. When one response followed instructions more completely than the other, that response was rated as "much better" regardless of other similarities; a key rule that prevented instruction-following failures from being obscured by surface-level quality.

Annotators were required to write specific, evidence-grounded explanations for every preference decision, identifying the precise reason one response outperformed the other rather than relying on generic comparisons.

4. Dimension-level reasoning

In addition to preference selection and explanation, annotators selected the specific dimensions that drove their preference from structured helpfulness and honesty criteria, covering factual accuracy, coherence, instruction-following, conversational tone, signal-to-noise ratio, creativity, and honesty-related categories, including neutrality, transparency, and non-invasiveness.

These dimension-level selections served as structured metadata for the preference signal, enabling downstream analysis of what quality attributes most consistently differentiated model outputs across task types.

5. Multi-layer QA and calibration

A dedicated QA team reviewed completed annotations using a star-based rating rubric covering four dimensions: 

• Factuality evaluation correctness
• Model preference selection accuracy
• Preference explanation quality
• Dimension selection accuracy 

Each dimension was scored independently, with specific deductions applied for missing categories, incorrectly added categories, and mismatched reasoning.

Error trends were tracked regularly across annotator cohorts, with recurring issues, including overly generic explanations, incorrect use of "much better" when "slightly better" applied, and misapplied distinguishing factors, addressed through targeted calibration sessions.

• Delivered more than 50,000 preference-annotated evaluations across image-grounded multimodal prompts spanning scientific reasoning, data interpretation, descriptive analysis, and information-seeking tasks
• Applied a factuality-first evaluation framework combining active image research, tolerance-based correctness thresholds, and step-by-step mathematical verification
• Produced dimension-level preference metadata across helpfulness and honesty categories, providing structured signal beyond binary preference labels
• Maintained annotation consistency at scale through weekly error trend analysis, targeted calibration, and a star-rated QA rubric with explicit deduction rules

The client received a large-scale, structured preference dataset grounded in visual reasoning across diverse image types and prompt categories. By combining factuality verification, evidence-based preference reasoning, and dimension-level annotation, the dataset provides richer training signal than standard preference labeling, distinguishing not just which response was better, but why, and along which quality dimensions.

This foundation supports:

• Multimodal RLHF training with high-fidelity preference signal across visual reasoning tasks
• Identification of systematic model weaknesses in factuality, instruction-following, and coherence
• Scalable evaluation of multimodal AI performance across diverse image domains and prompt types
• Iterative model improvement informed by structured, dimension-level quality analysis