Mano-CUA 1.1 Scores 41.7 on WebRetriever, Beating Claude
Specialized web agent Mano-CUA 1.1 scored 41.7 on WebRetriever Protocol I, outperforming Gemini 2.5 Pro and Claude 4.5 on form-filling tasks.
Stock photo for illustration only, not from the actual event
- Mano-CUA 1.1 scored 41.7 on WebRetriever, beating Claude 4.5's 31.3
- The model uses pure vision-driven processing without relying on DOM parsers
- Runs locally on Apple M5 Pro Mac hardware at 80 tokens per second
General-purpose LLMs struggle significantly with browser automation tasks. Recent benchmark results on WebRetriever Protocol I, which tests real browser navigation and form-filling, show that Mano-CUA 1.1 achieved a NavEval score of 41.7. Meanwhile, Gemini 2.5 Pro Computer Use scored 40.9, and Claude 4.5 Computer Use landed at 31.3. The complete project is available on GitHub at github.com/Mininglamp-AI/Mano-P.
The roughly 10-point gap between Mano-CUA and Claude highlights a fundamental difference in handling multi-step form interactions, dropdown selections, and paginated navigation. While general models excel at text reasoning, browser automation requires spatial awareness, tracking moving click targets, and interpreting diverse UI elements across different websites.
Stock photo for illustration only, not from the actual event
Most general models rely on parsing the DOM or accessibility APIs, which breaks down quickly on complex web apps with dynamic rendering or heavy JavaScript. In contrast, Mano-P adopts a pure vision-driven approach. The model analyzes screenshots like a human user, identifying clickable elements directly from pixels and planning actions based purely on visual context without DOM or API dependencies.
"When the page structure changes but the visual layout stays the same, the vision approach keeps working while the DOM parser breaks."
Mininglamp-AI
Analytical Context: The core limitation of general LLMs in web automation is losing state tracking after 4 to 5 steps in complex forms. Mano-CUA's architecture incorporates a think-act-verify loop, forcing the model to check if each action succeeded before proceeding. If a field rejects input, it reads the error message from the screenshot and adjusts, explaining why specialized training on GUI interaction sequences yields superior completion rates.
Another major differentiator is local execution support. The 4B quantized model runs entirely on a Mac equipped with Apple silicon and Apple M5 Pro hardware, achieving roughly 80 tokens per second decode speed. Using the --local flag ensures all data stays on-device without transmitting screenshots to external cloud servers.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment