Tools & retrieval
What is Computer Use?
Computer use is when an agent drives a GUI: click, type, scroll, and read the screen.
It is slower and brittle compared to APIs. Use it when no typed tool exists and a human would use the same UI.
Why it matters
Computer use allows agents to interact with GUIs — clicking buttons, typing text, scrolling pages, and reading the screen. It extends agent capabilities to applications that have no API.
Computer use is slower and more brittle than API calls. It should be used as a last resort when no typed tool exists, not as a default interaction pattern.
When to use computer use
Use computer use when: the target application has no API, the workflow requires visual verification (screenshots), or you are testing a UI. Avoid it when: an API exists, the task is purely data processing, or reliability is critical.
Computer use adds a camera-like observation loop: take a screenshot, identify elements, act, take another screenshot, verify the result.
Key takeaways
- 1Computer use is for GUIs without APIs — a last resort, not a default.
- 2It adds a visual observation loop: screenshot → identify → act → verify.
- 3Computer use is slower and more brittle than API-based tool calls.
Common mistakes
- ✕Using computer use when a simple API call would work.
- ✕Not handling UI changes that break element identification.
Related terms
Concept neighborhood
Terms linked from Computer Use in the glossary graph.
- Computer Use
- Tool Calling
- Observation
- Sandbox