AI agents are generally trained on web or mobile UIs, but real-world enterprise users rely on complex desktop applications. The client set out to build the first large-scale benchmark for testing AI agents in desktop software environments.

The benchmark needed to support:

• Element grounding: Identify the correct icon, menu item, or input field from a text query
• Layout grounding: Group elements by region or function
• Action prediction: Predict full trajectories of GUI interactions such as clicks, scrolls, and hotkeys

The annotation process required precise, tool-specific actions, exhaustive labeling, and frame-accurate screenshot capture, which were not achievable with generic data vendors or crowdsourcing.

Dataset structure

Turing deployed a team of more than 70 annotators based in India and Latin America. All contributors held technical degrees, were fluent in English and technical writing, and had prior experience in UI research and data labeling. 

Each task consisted of three parts:

1. Instruction design: Create realistic desktop-use prompts using open-source applications, such as “Open a new spreadsheet and apply conditional formatting to column B.”
2. Action recording: Perform the action while screen recording and capture GUI events such as CLICK, DRAG_TO, SCROLL, and TYPING
3. Frame annotation: Extract screenshots before each action and annotate all visible UI elements with bounding boxes, tooltips, and alt text

All actions followed a defined taxonomy of GUI events. Screenshot annotations followed strict standards for element coverage, cursor labeling, grouping logic, and text accuracy. Annotators worked in full-screen resolution and recorded actions using a proprietary annotation platform.

Task coverage

• Applications: The dataset covered 83 applications across categories such as productivity (LibreOffice, Zotero), creativity (GIMP, Blender), development (VSCode, Eclipse), and communication (Signal, Mastodon)
• Task types: Ranging from atomic (e.g., “Insert an image”) to complex (e.g., “Create a table, apply filters, export as PDF”)
• Interactions captured: Element location, motion (e.g., drag-and-drop), layout structure, and tool-based interactions

QA and review process

To maintain high quality across all the delivered tasks, Turing implemented a dual-layer manual QA pipeline. All annotations, including bounding boxes, text labels, and action logs, were manually verified to ensure screen-event fidelity, correct screenshot timing, and annotation consistency.

• Delivered more than 10,000 end-to-end tasks covering diverse interaction types across 83 open-source applications
• Annotated multiple keyframes per task, including UI components, tooltips, mouse events, and layout regions
• Supported benchmark tasks such as element grounding, layout grouping, and multi-step action prediction

The resulting dataset was used to build UI-Vision, the first large-scale benchmark for multimodal agents in desktop environments. The dataset reveals:

• Where models struggle with dense, multi-layered layouts
• How they fail at multi-step planning and execution
• The challenges of grounding instructions in complex desktop interfaces

Turing’s work enabled the client to:

• Scale the dataset rapidly with high-quality, reproducible annotations
• Maintain fidelity across complex user flows
• Benchmark frontier models on real-world, GUI-intensive tasks