How Turing and ServiceNow Collaborated on UI-Vision: A New Benchmark for GUI-Based AI Agents

GUIs are visual interfaces—toolbars, icons, dropdowns—that make modern software intuitive. Unlike web-based environments with structured HTML or mobile screens optimized for touch, desktop GUIs are complex, inconsistent, and more challenging for agents to interpret.

Today’s agents often struggle with visual grounding, spatial reasoning, and dynamic actions like drag-and-drop. These limitations have slowed progress in building fully autonomous systems that can navigate software like humans.

• UI-Vision is the first large-scale benchmark specifically designed to evaluate intelligent agents on desktop GUI tasks. Created by ServiceNow researchers in collaboration with academic and industry partners like Turing, UI-Vision covers 83 real-world applications across categories such as productivity, development, creativity, and more.
• What makes UI-Vision different?
• Dense, Human-Annotated Data: Every GUI interaction—including clicks, drags, and keystrokes—is paired with bounding boxes and labels.
• Three Core Tasks:
• Element Grounding: Locate specific UI elements from a query.
• Layout Grounding: Identify functional regions (e.g., toolbar or navigation pane).
• Action Prediction: Given a task, predict the agent’s next interaction.

These tasks help assess an agent’s perception, reasoning, and action capabilities, laying the foundation for more capable, adaptive AI.

Turing has been essential in crafting the high-quality training and evaluation data that powers UI-Vision. Our global annotation team, based primarily in India and Latin America, skillfully managed the most intricate aspects of developing this benchmark.

Key Contributions:

• Expert Annotation: Annotators with degrees in engineering and data science received training to perform and record tasks across 83 applications.
• Layered QA Process: Each annotation undergoes a rigorous, multi-stage review pipeline, including verification by subject matter experts and project leads.
• Diversity by Design: The annotation team was intentionally global, allowing the dataset to reflect how users from various geographies interact with software, thus ensuring broader applicability for downstream models.

This human-in-the-loop effort reflects Turing’s belief that human intelligence is a core differentiator in building real-world AI systems.

UI-Vision enables structured, measurable evaluation of GUI agents across three levels of understanding: identifying elements, interpreting layouts, and predicting interactions.

Potential Impacts:

• Better Training Signals: Developers can train agents on various real-world software beyond browser-based tasks.
• Agent Benchmarking: Researchers can now compare model performance across standard tasks, highlighting areas where systems fall short (e.g., drag actions or spatial reasoning).
• Enterprise Readiness: Desktop software is prevalent across various industries, and UI-Vision brings us closer to deploying agents in finance, design, logistics, and more.

Limitations:
UI-Vision is currently an offline benchmark, meaning it doesn't evaluate real-time interactions or explore multi-agent collaboration. Future extensions could address these gaps.

UI-Vision is crucial in assessing and enhancing intelligent agents in complex desktop environments. Turing’s role in powering the annotation process with rigorous human oversight highlights our commitment to improving AI infrastructure, enabling practical applications, and boosting real-world model performance. 

This demonstrates how Turing AGI Advancement and Turing Intelligence contribute to advancing AI, linking foundational research with applied enterprise impact.