AI coding · Advanced

Computer Use

Also called: GUI agent, Browser use, Desktop control

Let the AI look at the screen, move the pointer, click, and type—operating a browser or desktop app the way a person would.

In detail

Computer use (sometimes called a GUI agent) turns "see UI + click" into a tool: the model decides the next click or keystroke from a screenshot or accessibility tree. It helps when there is no stable API, on third-party sites, or for end-to-end checks.

It is slow, costly, and brittle: layout shifts miss-click; it might hit pay or delete. Pair it with human confirmation, sandbox accounts, and allowlists. Prefer APIs via Tool calling/MCP when they exist; GUI is a fallback.

For Vibe Coding, "walk through signup in staging and screenshot each step" beats clicking around production.

Developer infoTerm ID, DOM cues, match priority

Term ID: ai-computer-use
DOM selectors: No DOM cues. This concept isn't detected directly on a page.
Priority: 1 · when several match at the same level, the higher priority wins
Version: v1 · updated Oct 5, 2026