@aralroca/gui-agent weekly downloads statistics

gui-agent

An open-source, WebMCP-powered GUI agent that lives inside your web app. Let a natural-language agent operate your UI — click, fill, navigate, call your app's own actions — driven by any LLM you bring.

The optional visualizer in action: status chips per tool call, an animated gradient glow that tours every element the agent touches, and a backdrop veil that keeps the user's focus on the action.

Inspired by page-agent, but built on the emerging WebMCP standard via a polyfill.

• 📦 Just an npm package. npm i @aralroca/gui-agent and import. No script injection, no browser extension, no headless browser. • 🧩 Standards-based. Tools are registered on document.modelContext (WebMCP). The polyfill (@mcp-b/webmcp-polyfill) installs it where browsers don't natively support it yet. • 🎯 Producer + consumer in one. Expose your app's actions as precise tools and fall back to text-based DOM driving for everything else. • 🧠 Bring your own LLM. A tiny provider-agnostic interface, plus a ready-made Vercel AI SDK adapter. • 🪶 Headless core. No UI imposed — wire it to your own chat, command bar, or voice. • ✨ Opt-in visualizer. Status chips per action, an animated gradient glow around the element being acted on, and a backdrop blur that spotlights it (@aralroca/gui-agent/ui).

⚠️ Early/experimental. WebMCP is a moving W3C draft; APIs may change.

Thinking is cheap. But what if your agents didn't have to think so much—and instead had TOTAL CONTROL of your app (AUTOMATICALLY)? Take a look Janux Framework for more than gui-agent lib

Table of Contents

• Install • Quick start • How it works • How the agent picks a tool • Compared to page-agent • API • Core • Visualizing the agent • Bring your own LLM • AI SDK adapter • React • Safety • Demo • Develop • License

Install

Quick start

Importing the package installs the WebMCP polyfill automatically — no tag.

How it works

gui-agent unifies two approaches:

1. Producer (WebMCP). Your app calls defineTool(...) to register reliable, structured actions on document.modelContext. Any WebMCP agent — including a browser's native one — can use them. 2. DOM fallback (page-agent style). For anything not exposed, the agent builds a compact text snapshot of the page (roles, labels, values, stable refs like e7) and gets synthesized read_page / click / fill / select_option / drag / wait_for_text tools. No screenshots, no multimodal model needed.

The built-in loop discovers all available tools, asks your LLM what to do, runs the calls (gated by your optional confirm), feeds results back, and repeats until done.

How the agent picks a tool

For each step the agent prefers a purpose-built WebMCP tool if one fits, and only infers the DOM when nothing is exposed for the job:

The agent is handed a single merged tool list and is instructed to prefer your app's WebMCP tools; the synthesized DOM tools are the always-on fallback (a same-named app tool wins). WebMCP tools live on document.modelContext — your app registers them (so any WebMCP agent, including the browser's native one, can call them) and discoverExternalTools() can pull in tools other code registered there. upload_file and navigate DOM tools are opt-in.

Compared to page-agent

gui-agent is inspired by page-agent and shares its core idea — a natural-language agent that drives a real web page from a text-based DOM snapshot (no screenshots, no multimodal model) with a bring-your-own-LLM design. The difference is what gui-agent adds by building on the emerging WebMCP standard instead of DOM-driving alone:

| Capability | gui-agent | page-agent |

| WebMCP standard (document.modelContext) | ✅ Built on the W3C WebMCP draft + polyfill | ❌ Not WebMCP — its "MCP" is a separate Node server that lets external agents drive the browser | | App actions as standard tools | ✅ defineTool registers structured actions on document.modelContext, callable by any WebMCP agent — including the browser's native one | ⚠️ Custom tools are client-side config, marked @experimental, and not exposed via a page standard | | Interop with other in-page tools | ✅ discoverExternalTools() picks up tools other code registered on document.modelContext | ❌ Not available | | Human-in-the-loop safety gate | ✅ Built-in confirm() gate — every non-readOnlyHint tool is routed through approval before it runs | ⚠️ No mandatory approval gate and no read-only/write distinction (runs autonomously up to maxSteps); offers ask_user, lifecycle hooks and data-masking instead | | First-class React bindings | ✅ GuiAgentProvider, useGuiAgent, useTool, | ❌ None (vanilla panel UI) | | Provider-agnostic LLM | ✅ Any shape via a one-function Llm interface, plus a Vercel AI SDK adapter and a remote-LLM helper | ⚠️ Bring-your-own, but the endpoint must be OpenAI-compatible | | Producer + consumer in one | ✅ Precise structured producer tools and a DOM fallback in a single loop | ➖ Primarily a DOM driver (custom tools experimental) |

When page-agent is the better fit. page-agent is excellent and has real strengths gui-agent doesn't: it drives any existing site's DOM without the app opting in, so it works on legacy or third-party apps (ERP, CRM, admin panels) you don't control — whereas gui-agent's producer tools need the app to register them (its DOM fallback still works anywhere, but the WebMCP value comes from instrumentation). page-agent also ships an optional Chrome extension for cross-tab tasks and a Node MCP server so external agents (Claude Desktop, Cursor) can drive a real browser. Both projects are MIT-licensed, TypeScript, text-based, and ship an activity visualizer.

In short: reach for page-agent to automate sites you don't own; reach for gui-agent when you do own the app and want standards-based, structured, human-gated tools — with a DOM fallback for everything else.

API

Core (@aralroca/gui-agent)

| Export | Purpose |…