The client needed a reliable dataset to:

• Evaluate how well the model triggers the right built-in extension tool or tools for complex tasks
• Train preference models that can distinguish correct tool selection, chaining, and grounding
• Capture reasoning failures in under-triggered or missequenced responses
• Benchmark tool-specific behavior such as fallback to search or clarification prompts

Existing SFT and evaluation data did not offer:

• Complex prompt coverage across CUJs
• Tool-chaining logic across real-world queries
• RLHF-style comparisons for instruction tuning

Prompt generation

Turing designed more than 1000 novel prompts spanning key CUJs, including Browse, YouTube, Search, and Maps. Each CUJ included both positive and negative examples to test the model's ability to select the correct extensions and make sequence calls correctly.

Prompts were labeled by CUJ type, category, tool type, and complexity (trivial, moderate, complex). The goal was to surface challenging edge cases such as multi-tool workflows, missing or inferred tool usage, and incorrect or redundant tool activation. Prompts were also designed to trigger fallback reasoning when URLs were broken or incomplete.

RLHF preference collection

The team collected side-by-side preference ratings for the model’s tool-based responses. Annotators compared model outputs, rated them on a rubric, and supplied user-written demonstrations when both responses were suboptimal.

Each example was annotated with structured blocks: Hidden Thought, Code, and Explain. These hidden thoughts framed tool rationale, planned multi-step chaining, and distinguish fallback intent from over-triggering. Metadata included whether the step was incomplete or marked as the final response, aligning with the client’s internal data conventions.

Edge case and failure modeling

The team explicitly designed prompts and demonstrations to surface failure cases, including:

• Clarifying questions for underspecified prompts
• Handling unsupported tool behaviors such as liking or saving YouTube videos
• Inaccessible or suboptimal links
• Tool over-triggering and fallback logic such as fallback from Browse to Search

Fallback strategies were structured to reflect realistic conversational repair behaviors, including user re-prompts and alternate tool paths.

• Built a benchmark-grade dataset of more than 1000 tool-centric prompts and RLHF examples
• Covered 8 CUJs with tool-specific tags and reasoning formats
• Generated hundreds of demonstrations where models failed to trigger the correct tool
• Enabled training of instruction-tuned and reward models grounded in tool accuracy
• Delivered structured examples with hidden thoughts, fallback logic, and chained responses

This dataset supports:

• The client’s preference model training on real-world tool usage
• Chain-of-thought training for multi-tool reasoning
• Structured evaluation for CUJ coverage and fallback handling
• Research on tool grounding, misuse, and recovery behaviors

Turing’s work enabled the client to accelerate toolchain maturity across Browse, YouTube, Search, and Maps with traceable, fine-grained evaluation data.