The In-Page GUI Agent That Controls Web Interfaces with Natural Language
alibaba/page-agent · 12.9k stars · TypeScript · MIT
agent37.club
The web page already knows its own structure. Why send screenshots to a vision model when the DOM is right there?
A pure JavaScript library that runs inside the page, reads the DOM as text, sends it to any LLM, and executes the returned actions directly. No extensions, no Python, no headless browsers.
Client-side web enhancement, not server-side automation. Page Agent is designed to be embedded into products, not to scrape them from outside.
page-controller traverses the live DOM, identifies interactive elements (buttons, inputs, links, selects), and produces a text representation with element indices.
The text DOM + user instruction + action history are sent to the LLM. The model returns a structured action: which element index to interact with and how.
page-controller dispatches the action (click, type, scroll, select) on the real DOM element. The page updates.
The agent re-extracts the DOM, checks if the task is done, and either loops back to step 1 or returns the result.
The page-controller package (derived from browser-use) walks the DOM tree and produces an indexed text map:
// Simplified output [1] <button> Login </button> [2] <input type="email" placeholder="Email"> [3] <input type="password"> [4] <a href="/forgot"> Forgot password? </a> [5] <select> English | Chinese </select>
| Dimension | Page Agent (text DOM) | Vision agents (screenshots) |
|---|---|---|
| Input size | ~2-5 KB text | ~200-500 KB image |
| Model requirement | Any text LLM | Multimodal (GPT-4V, etc.) |
| Latency per step | Fast (small payload) | Slow (image encoding) |
| Element precision | Exact (DOM index) | Approximate (x,y coords) |
| Dynamic content | Reads live DOM state | Point-in-time snapshot |
| Canvas / images | Cannot "see" pixels | Can interpret visuals |
| Cost per step | Low (text tokens only) | High (image tokens) |
The llms package exposes an OpenAI-compatible interface. Any model that speaks the chat completions API works:
const agent = new PageAgent({ // Alibaba's Qwen model: 'qwen3.5-plus', baseURL: 'https://dashscope... /compatible-mode/v1', apiKey: 'sk-...', }) // Or use OpenAI, Anthropic, local models const agent = new PageAgent({ model: 'gpt-4o', baseURL: 'https://api.openai.com/v1', apiKey: 'sk-...', })
A demo CDN build ships with a free testing API key for quick evaluation.
npm install page-agent
import { PageAgent } from 'page-agent' const agent = new PageAgent({ model: 'qwen3.5-plus', baseURL: 'https://dashscope...', apiKey: 'YOUR_KEY', language: 'en-US', }) await agent.execute( 'Click the login button' )
<!-- Global CDN --> <script src="https://cdn.jsdelivr.net /npm/[email protected]/dist /iife/page-agent.demo.js" crossorigin> </script> <!-- China mirror --> <script src="https://registry .npmmirror.com/page-agent /1.6.0/files/dist/iife /page-agent.demo.js"> </script>
This is the core package that bridges the gap between the DOM and the LLM. It handles two jobs:
DOM processing components and prompts are derived from the browser-use project (MIT licensed), adapted for in-page execution rather than external control.
The llms package provides a unified interface for talking to any LLM that supports the OpenAI chat completions format.
{
model: 'model-name',
baseURL: 'endpoint',
apiKey: 'key',
language: 'en-US',
}
Any OpenAI-compatible endpoint works. The model just needs to be good at structured output and instruction following.
Page Agent ships with a built-in chat panel that provides:
The UI is fully customizable and can be themed to match your product.
Unlike fully autonomous agents, Page Agent's default UI shows each planned action and can require user approval. This is critical for production SaaS deployments where you cannot let an AI make unchecked changes.
The mcp package enables Page Agent to expose its capabilities as MCP tools, allowing external AI systems to control web pages through it.
MCP turns Page Agent from a standalone library into an interoperable tool in the broader AI ecosystem. An agent running in Claude Desktop could call Page Agent to interact with a web app as one step in a larger workflow.
The core library works within a single page. For workflows that span multiple pages or tabs, the optional Chrome extension bridges the gap:
The extension reintroduces the browser extension dependency that Page Agent otherwise avoids. It is optional and only needed for multi-page workflows.
Embed Page Agent into your product to give users a natural language interface. "Show me last quarter's revenue chart" triggers navigation and data display without building custom NL commands.
Complex ERP/CRM forms with dozens of fields. "Fill this purchase order with the data from the attached invoice" -- one sentence replaces 15 minutes of data entry.
Voice-controlled web interfaces for users who cannot use a mouse or keyboard. The agent translates spoken commands into precise page interactions.
With the Chrome extension: "Copy the tracking number from the shipping page and paste it into the customer's order in our CRM."
| Aspect | Page Agent | browser-use |
|---|---|---|
| Runtime | In-browser JavaScript | Python + Playwright |
| Installation | npm install / CDN script tag | pip install + browser binary |
| DOM access | Direct (same process) | Remote (CDP / WebSocket) |
| Target use case | Product integration (SaaS copilot) | External automation / scraping |
| Multi-page | Requires Chrome extension | Built-in (controls browser) |
| Server needed | No | Yes (Python process) |
| Vision support | Text-only DOM | Text + screenshot modes |
Page Agent acknowledges browser-use as the source of its DOM processing components and prompts.
Ships as both an npm package (ESM/CJS) and a single IIFE script for drop-in CDN usage. The IIFE build bundles all dependencies into one file.
For production use, always proxy LLM calls through your backend to avoid exposing API keys in the browser.
| Tool | Approach | Runtime | Best for |
|---|---|---|---|
| Page Agent | In-page text DOM | Browser JS | SaaS copilots, embedding |
| browser-use | External text DOM | Python | Automation, scraping |
| Anthropic CUA | Screenshots + vision | API | General computer use |
| LaVague | Hybrid vision/DOM | Python | Web testing, QA |
| Playwright MCP | Programmatic control | Node.js | Deterministic automation |
| Stagehand | Vision + DOM | TypeScript | Web agents, scraping |
Page Agent's unique position: it is the only major agent that runs inside the page itself rather than controlling it from outside. This makes it ideal for product builders rather than automation engineers.
1. Page Agent proves that browser agents do not need screenshots, extensions, or Python. A text DOM + any LLM is sufficient for most web interactions.
2. The in-page architecture makes it uniquely suited for product integration (SaaS copilots, accessibility) rather than external scraping.
3. MCP support (v1.6.0) positions it as an interoperable tool in the AI ecosystem, not just a standalone library.
4. The text-only approach trades visual understanding for speed, cost, and model flexibility. The right choice depends on your use case.
In-page GUI agent for controlling web interfaces with natural language
github.com/alibaba/page-agent
12.9k stars · TypeScript · MIT License · v1.6.0
agent37.club