# Browser Automation Roadmap

HyperDesk already supports visible Chrome controls for launching Chrome, opening tabs, and letting agents drive browser interactions with approval gates. The next browser milestone is a first-class automation layer that combines fast Agent Browser/CDP workflows with visible Chrome review.

Tracking issue: https://github.com/hyper63/work/issues/24

Reference handoff: https://zenbin.org/p/hyperdesk-browser-automation-handoff

## Goal

Build a native HyperDesk browser automation layer that uses Agent Browser/CDP for fast work and visible Chrome for user review.

The preferred live automation mode is visible Chrome over CDP:

```bash
agent-browser --cdp 9222 open https://example.com
agent-browser --cdp 9222 snapshot -i
agent-browser --cdp 9222 click @e3
agent-browser --cdp 9222 get url
```

## Why this matters

Agent Browser snapshots, element refs, form actions, screenshots, eval, and persistent sessions are fast and model-friendly. CDP mode preserves the user's visible Chrome session, cookies, and logged-in context while still allowing the user to watch or inspect the final result.

## HyperDesk browser tools

Native browser tools expose one structured result model across visible Chrome and background Agent Browser backends:

- `browser_open` with `waitUntil`, `waitForSelector`, `waitForText`, `waitMs`, `timeoutMs`, and `autoConnect`
- `browser_snapshot` returning page text plus stable `@eN` refs for interactive elements; use `interactive: false` or default `fullText: true` when reading long chat/conversation pages
- `browser_click`, `browser_fill`, `browser_press`, and `browser_scroll` accepting refs/selectors, bounded wait options, and `autoConnect`
- `browser_wait` for load state, selector/ref, text, or fixed-delay waits
- `browser_screenshot` with `fullPage`, `annotate`, selector/ref options, and `autoConnect` where supported
- `browser_get` for `url`, `title`, `text`, `html`, `value`, `attr`, `count`, `box`, and `styles`
- `browser_connect_chrome` for Agent Browser `--auto-connect` against an existing visible Chrome tab/session
- `browser_get_url` as a compatibility wrapper around URL/title retrieval
- `browser_show_in_chrome` for handoff from background to visible Chrome
- future `browser_eval` once policy and approval semantics are finalized

## Default strategy

- Use visible Chrome with CDP for authenticated or user-visible tasks.
- Use Agent Browser `--auto-connect` (`browser_connect_chrome` or `autoConnect: true`) when a site blocks background/headless automation but works in an already-authenticated Chrome tab.
- Use an isolated background browser session for fast private automation when visible Chrome is not needed.
- Hand off final URLs, files, or screenshots to visible Chrome for review.
- Keep Chrome MCP as an experimental visible-backend candidate behind `HYPER_DESK_CHROME_BACKEND=mcp`; the prototype expects an injected MCP client and preserves the same HyperDesk browser result schema.

## Troubleshooting notes

- Browser results include `cloudflare: true/false`. When a Cloudflare challenge is detected, switch to visible Chrome or run `browser_connect_chrome`, then retry reads/actions with `autoConnect: true`.
- `chrome_fill` and visible Chrome fill paths treat `contenteditable` elements specially via focus/selection plus `insertText`; still verify text before submitting in chat apps.
- For chat apps and long conversations, prefer `browser_snapshot` with default `fullText: true` (or explicitly `interactive: false`) over targeted `browser_get text @ref`, which can time out on virtualized DOMs.

## Safety gates

Require explicit user approval for high-risk browser actions, including:

- purchases,
- deletes,
- sends,
- publishes,
- permission changes,
- password changes,
- billing changes,
- API key or secret handling.

## Google Workspace helpers

High-value helpers to build later:

- `sheets_create_and_fill(title, tsv)`
- `docs_create_and_paste(title, text)`
- `drive_create(type, title)`
- `workspace_wait_for_saved()`
- `browser_policy_confirm(action, risk)`

Sheets should support TSV paste. Docs may need specialized paste handling for canvas-heavy or iframe-backed editing surfaces.

## UI direction

Render browser actions as compact HyperDesk cards showing:

- action name,
- URL or page title,
- ref or selector,
- screenshot thumbnail when available,
- verification status,
- final URL.

## Implementation plan

1. Vendor/depend on Agent Browser, or port its CDP/Playwright command layer.
2. Add native typed Pi/HyperDesk browser tools.
3. Default to visible Chrome CDP mode when available.
4. Support isolated background browser sessions.
5. Add final-result handoff to visible Chrome.
6. Add Google Docs/Sheets helpers.
7. Add UI renderers and approval policies.
8. Add regression tests for open/snapshot/click/fill, Sheets TSV paste, Docs text paste, final Chrome display, and secret/destructive-action safety.
