Debugging Is Also Clicking: Why AI Agents Need to Interact with UI

In previous explorations, agents were allowed to debug via DAP—handling breakpoints, stepping over code, and continuing execution. While that constitutes genuine debugging, it only covers half of the equation. True debugging often requires physical interaction: pressing a button and observing the result, reading a dialog box, or noticing that a toggle switch is grayed out. No backtrace will ever inform a developer that the Save button was never enabled. The critical question is whether an AI agent can handle this second half of the task as well.

Web browsers are inherently automatable by design. Most agent frameworks ship with built-in browsers or drive external ones; pointing Playwright at any web page gives every element a stable, queryable handle. The Document Object Model functions essentially as an accessibility tree wearing a different guise, exposing roles, labels, and structural hierarchies ready for parsing. Consequently, for web applications, this half of the debugging puzzle is largely solved.

Conversely, in environments lacking a DOM, when agents run out of options, they resort to a purely visual approach: capturing a screenshot, having the model analyze it, and sometimes running OCR or preprocessing passes to label on-screen items. Although this method functions and occasionally serves as the sole option, it remains brittle—miss a few pixels and the click completely misses its target—while consuming excessive tokens just to describe images.

This limitation emerged accidentally when developing a tiny utility designed to capture screenshots of a running 4D form and stitch them into an animated GIF for a README file, known as 4d-capture-gif. It was later observed that Claude Code utilized this utility for debugging because the script also reported minor structural details of the form, such as button placements, informing the agent where to click. For rudimentary scenarios, this approach genuinely functions, yet screenshot-plus-coordinate mapping is far from an ideal foundation.

Rather than relying purely on visual inspection, developers can read the UI tree directly. On macOS, the Accessibility API can be scripted using Python through pyobjc, supported by automation libraries. This enables an agent to click element #37—specifically targeting the Save button—instead of blindly guessing coordinates like (412, 260).