# Browser Agent — Implementation Plan

> **Status**: Ready to implement.
> **Date**: 2026-05-16
> **Owner**: aion0
> **Related**:
> - `docs/RFC-Browser-Connectors.md` — detailed connector design (this is the abridged action plan)
> - `../temper` — memory service (existing)
> - Forge main repo (this) — orchestration platform (existing)

## TL;DR

Build a **Chrome extension** that does two things:

1. **Chat UI** — daily-driver enterprise AI assistant with Temper-backed memory
2. **Site connectors** — installable per-company adapters that re-use the user's browser sessions to read/write internal systems (Mantis, Teams, JIRA, internal CRM, …)

The extension is **independent of Forge** — composable, mutually enhancing. Forge stays as today (`localhost:8403` web UI) and is invoked when long tasks / code work are needed; the extension handles day-to-day chat + system queries.

## Architecture (Final)

```
                         ┌──────────────────────────────────┐
                         │  Browser (user's daily Chrome)   │
                         │ ┌──────────────────────────────┐ │
                         │ │ Forge Browser Extension      │ │
                         │ │                              │ │
                         │ │  ┌────────────────────────┐  │ │
                         │ │  │ Chat side-panel UI     │  │ │
                         │ │  │ (React)                │  │ │
                         │ │  └────────────────────────┘  │ │
                         │ │                              │ │
                         │ │  ┌────────────────────────┐  │ │
                         │ │  │ Agent loop             │  │ │
                         │ │  │ (LLM + tool calling)   │  │ │
                         │ │  └────────────────────────┘  │ │
                         │ │                              │ │
                         │ │  ┌────────────────────────┐  │ │
                         │ │  │ Connector marketplace  │  │ │
                         │ │  │ (install/uninstall)    │  │ │
                         │ │  │ • Mantis               │  │ │
                         │ │  │ • Teams                │  │ │
                         │ │  │ • <yours>              │  │ │
                         │ │  └────────────────────────┘  │ │
                         │ └─┬──────────┬─────────┬───────┘ │
                         └───┼──────────┼─────────┼─────────┘
                             │          │         │
                  HTTP REST  │          │ HTTP    │ HTTP (optional)
                             ▼          ▼         ▼
                    ┌─────────────┐ ┌──────┐ ┌──────────────┐
                    │   Temper    │ │ LLM  │ │   Forge      │
                    │  (memory)   │ │ API  │ │ (when need   │
                    │  existing   │ │      │ │  pipelines/  │
                    │             │ │      │ │  long tasks) │
                    └─────────────┘ └──────┘ └──────────────┘
```

**Three products with clear, asymmetric roles** (not three equal peers — each plays a distinct part):

| Product | Status | Role | Required when |
|---|---|---|---|
| **Browser Extension** | 🆕 to build | **Agent entry point** — main user-facing surface. Connector hub for company internal + external SaaS systems. | Always (the agent's "face") |
| **Forge** | ✅ existing | **Agent backend + dev platform** — pipelines, crafts, workspaces, code work, long-running tasks | When you need orchestration, code work, scheduled flows (not always for pure chat) |
| **Temper** | ✅ existing | **Agent memory** — supplements Forge's memory when agent chat is involved | Only when you have an agent that needs cross-session memory. Forge-only (no chat agent) usage doesn't need it. |

### Reading the trinity

- **Extension** is the most user-visible. If a non-technical user only sees one thing, it's the extension.
- **Forge** sits behind the extension as the backend / dev platform. Power users open Forge UI directly when they need pipeline editing, multi-agent workspaces, or code work.
- **Temper** is purely a memory backplane. Optional. **Forge without an agent doesn't need Temper** — you can run pipelines / crafts / Forge UI fine without any memory service. Temper kicks in once a conversational agent (e.g. the extension's chat) needs to remember things across sessions.

### Dependency direction

```
Browser Extension ───depends on───▶  Forge (when long task)
                  ───depends on───▶  Temper (for memory)
                  ───depends on───▶  LLM API (always)

Forge             ──optionally uses──▶  Temper (only when an agent runs)

Temper            stands alone, dependent on nothing else
```

**Nobody is a peer of "depends on me"** — Temper is the deepest, Forge is middle, Extension is the apex / user-facing layer.

## Component Responsibilities

### Browser Extension (new)

| Responsibility | How |
|---|---|
| Chat UI | React in side panel (`chrome.sidePanel`) |
| Agent loop | LLM tool-use loop (lightweight — direct API calls, no pi-coding-agent) |
| Memory | HTTP client → Temper REST API |
| Site access | Per-domain content scripts (connectors) + `fetch` with credentials |
| Connector management | Marketplace UI in extension popup (install/remove/update) |
| Forge bridge (optional) | `fetch('http://localhost:8403/api/…')` when user wants pipeline / craft / long task |

**Does NOT do:**
- Code editing / file ops (Forge job)
- tmux / terminal (Forge job)
- Long-running cron / scheduled tasks (Forge job)
- Cross-machine sync (Temper job)

### Temper (existing, no changes needed)

- HTTP REST memory API
- KV memory blocks (preferences, profile, current state)
- Graph episodes (third-party facts, recall over time)
- Multi-tenant, multi-user

Extension is a Temper client; nothing new to build on Temper side for v1.

### Forge (existing, minor additions later)

- Web UI for pipelines / crafts / workspace (as today)
- Backend services (terminal, workspace, telegram, MCP)
- Extension can call Forge's HTTP API when it needs heavy orchestration
- **No required changes for extension v1** — composable, the extension works without Forge

Later (post-v1):
- Forge "Browser Bridge" service for extension-to-Forge MCP bridging (per RFC)
- Forge UI showing connected extensions / paired devices

## Connector Model

Connectors are **installable per-company**:
- Each company has their own internal systems
- A connector is a JS module: `id`, `matches: [urlPattern]`, `tools: {...}`, optional `healthCheck`
- Installable from a public registry (`forge-browser-connectors` GitHub repo, mirrors `forge-crafts`)
- Or sideload-able from a company's private registry (URL configurable in extension settings)

**v1 ships with empty marketplace.** Users add their own (or pick from public ones once registry exists).

## Tech Stack

| Layer | Choice | Why |
|---|---|---|
| Extension manifest | MV3 | Chrome's mandatory modern format |
| Background | Service Worker | Required for MV3; OK because we don't need long-lived processes (the agent loop is request/response) |
| UI framework | React 18 + TypeScript + Vite | Match Forge stack; Vite for fast SPA build |
| State | Zustand (or jotai) | Lightweight; React Context for connector registry |
| LLM SDK | `@anthropic-ai/sdk` + optional OpenAI | User config; Anthropic primary |
| Memory client | Hand-rolled REST (or generated OpenAPI client) → Temper | Temper has stable API |
| Connector SDK | Local TS package | Defines `defineBrowserConnector({...})` |
| Bundler | Vite + `@crxjs/vite-plugin` | Standard MV3 toolchain |
| Package layout | Monorepo? Or single repo? | Decision: single repo `forge-browser-extension`, with `packages/sdk` workspace for the connector SDK to be published to npm |

## Phase Plan (4 weeks → MVP)

### Week 1: Skeleton

- New repo: `forge-browser-extension`
- MV3 manifest, side panel, background SW, content script
- Chat UI shell (input → static placeholder response)
- Settings page: Temper URL, LLM API key
- Build pipeline (Vite + crxjs) producing `dist/` loadable as unpacked extension
- **Deliverable**: extension loads in Chrome, side panel opens, you can type something + see your own message echoed back

### Week 2: Agent + Memory

- Wire LLM API (Anthropic streaming)
- Implement minimal tool-use loop (5-10 hand-coded tools, no connectors yet)
- Temper memory:
  - on session boot: load relevant memory blocks
  - after every turn: extract & write back any "fact about user" → memory blocks
  - episodes: write per-session to Temper graph
- **Deliverable**: chat with persistent memory; close Chrome, reopen, agent remembers what you told it last time

### Week 3: Connector framework + first connector

- `@forge/browser-connector-sdk` (in `packages/sdk/`):
  - `defineBrowserConnector({...})` factory
  - `ctx.fetch / $ / $$ / waitFor / storage / log / AuthExpiredError`
- Connector loader in extension:
  - Discover connectors in `chrome.storage` (installed list)
  - On tab match, inject content script
  - Register tools globally
- **Mantis connector** (first one, since you specifically mentioned it):
  - Tier 1: `/api/rest/issues` with cookies
  - Tier 2: scrape `view_all_bug_page.php`
  - Tools: `mantis.list_my_bugs`, `mantis.view_bug`, `mantis.add_comment`
- **Deliverable**: extension chat can say "list my open Mantis bugs" → returns real data

### Week 4: Marketplace + second connector

- Connector marketplace UI in extension popup
- `forge-browser-connectors` registry (GitHub repo, schema mirrors `forge-crafts`)
- Install / uninstall / update flows
- **Second connector** (pick one: Teams or your highest-value internal system)
- Connector health checks + auth-expired notification
- **Deliverable**: end-to-end real use; ready to use day-to-day on at least 2 systems

### Beyond MVP

- Forge bridge (Phase 0-1 of RFC) — extension can call Forge MCP for heavy tasks
- More connectors
- Vision-tier fallback for awkward sites
- Possible: smith engine folded in as alternative agent loop

## Code Locations

| Code | Repo | Path |
|---|---|---|
| Extension app | NEW `forge-browser-extension` | `src/` |
| Connector SDK | NEW `forge-browser-extension` | `packages/sdk/` |
| Connector registry index | NEW `forge-browser-connectors` (mirror `forge-crafts`) | `registry.json` + `<connector>/` |
| Forge integration (later) | Existing Forge repo | `lib/browser-bridge-standalone.ts` (Phase 0+ per RFC) |
| Temper changes | None | N/A (use existing API) |

## Decisions Resolved

| Question | Resolution |
|---|---|
| Bundle backend into extension? | No — backend stays in Forge node process; extension is UI + connectors only |
| Run browser in Docker? | No — re-use user's session is the whole point |
| Connectors per-company or generic? | Per-company, installable via marketplace, with public + private registries |
| Forge required for extension to work? | No — composable. Extension works standalone for chat + connectors |
| Smith engine folded into extension? | Deferred — start with simpler hand-rolled agent loop, fold smith later if needed |
| MV3 service worker concerns? | Acceptable — chat is request/response, no long-lived state needed. Agent loop is per-message |
| Chat UI location | Side panel (Chrome 117+). Popup for marketplace + settings |

## Open Questions

| Question | Need to answer by |
|---|---|
| Which LLM as default? Anthropic only, or also support OpenAI / local? | Week 1 (config) |
| Memory granularity: every turn → memory? Or only on important facts? | Week 2 |
| Connector code signing / supply-chain security | Week 3 (when registry goes public) |
| Public connector registry URL — host on github.com/aiwatching/forge-browser-connectors? | Week 3 |
| Auth expiry UX — toast? Inline chat message? Auto-resume? | Week 3 |
| Chrome Web Store publication strategy | Week 4 (or unsigned sideload for first internal users) |

## Tomorrow's First Concrete Step

```bash
mkdir -p ~/IdeaProjects/forge-browser-extension
cd ~/IdeaProjects/forge-browser-extension
git init
gh repo create aiwatching/forge-browser-extension --private --source . --remote origin

# Scaffold:
pnpm init
pnpm add -D vite @crxjs/vite-plugin @types/chrome typescript react @types/react @types/react-dom
pnpm add @anthropic-ai/sdk zustand

# Files to create on day 1:
# manifest.json (MV3)
# vite.config.ts (crxjs preset)
# src/background.ts (SW skeleton)
# src/sidepanel/index.html + sidepanel.tsx (chat UI shell)
# src/popup/index.html + popup.tsx (settings + marketplace placeholder)
# src/content/index.ts (per-tab content script bootstrap)
```

Aim by end of day 1: load unpacked in Chrome, click extension icon → side panel opens with React app saying "Hello".

## Success Criteria (4 weeks)

A user can:
1. Install the extension once
2. Configure Temper URL + LLM API key once
3. Open the side panel from any tab
4. Type: "list my open Mantis bugs"
5. See real bug data from their Mantis without re-authenticating
6. Continue: "summarise the top 3 by severity" — agent uses LLM
7. Close Chrome, come back tomorrow — agent remembers user preferences
8. Install a second connector from the marketplace with one click

If all 8 work end-to-end, MVP is done.

## Out of Scope (defer)

- Mobile / Safari / Firefox
- Centralized headless browser farm
- Forge UI inside extension (Forge stays at `localhost:8403`)
- Connector code signing infrastructure
- Multi-user / SSO at extension level (Temper handles user separation)
- Computer Use vision-tier fallback
- Smith engine integration

## ADR — 2026-05-16: Connectors are Forge plugins, not extension code

After initial RFC, decided to **move connector implementations out of the
extension and into Forge as plugins** (the existing `lib/builtin-plugins/*`
system: `docker`, `jenkins`, `slack`, `llm-vision`, …).

### Why plugins not crafts
- Crafts are per-project + UI-heavy. Connectors are system-level, stateless,
  pure capabilities.
- Plugins already have manifest/marketplace machinery in Forge.
- Plugins are already the "system capability" abstraction agents call.

### Why not keep them in extension
- Extension stays thin (chat UI + memory client + thin RPC to Forge).
- Connector marketplace = Forge plugin marketplace (don't double-build).
- Adding a connector = adding a YAML to Forge. No JS compiled into extension.

### Plugin execution modes (new addition to Forge plugin schema)
```yaml
mode: server-side    # default; Forge backend calls external API
mode: browser-side   # Forge dispatches to extension which fetches via tab cookies
mode: hybrid         # per-tool: some server, some browser
```

### Data flow (Mantis, browser-side mode)
```
Extension chat → POST /api/plugins/mantis/invoke
              ← Forge sees mode=browser-side → WS dispatch to extension
              ← Extension runs content-script fetch in mantis tab w/ cookies
              ← Mantis JSON → back up the chain → LLM → user
```

### Dependency revision
With this model, the extension's connector capabilities **require Forge**.
Pure-chat mode (no connectors) still works without Forge. The earlier
"all composable, all standalone" framing was too theoretical — for any
real enterprise use, the three-product stack is needed together.

### Forge changes required (small but real)
1. Plugin manifest accepts `mode: browser-side | server-side | hybrid`
2. Plugin manifest accepts `category: connector | pipeline-node | mcp-server | tool | other` — marketplace + extension filter on this
3. Plugin manifest accepts a `connectors: [...]` array (default 1:1 plugin↔connector; 1:N as escape hatch for same-vendor suites with shared auth — see RFC "Packaging in Forge")
4. New plugin executor: when `mode=browser-side`, dispatch to extension via WS
5. WebSocket back-channel: extension ↔ Forge persistent connection
6. Extension capability registry: Forge tracks "which extension has which tabs matching which plugins"
7. Plugin settings UI: show extension connectivity state + paired connectors

### Defaults shipped with Forge
- `gmail` (server-side, OAuth) — built-in
- `teams` (browser-side, MSAL token extraction) — built-in
- `jira` (hybrid — cloud OAuth, self-hosted cookie) — built-in
- `github` (server-side, PAT) — built-in
- `gitlab` (server-side, PAT) — built-in
- User-installed: `mantis` and others from plugin marketplace

### What this changes in the 4-week plan
- **Week 3 deliverable changes**: instead of "connector SDK + first connector in extension", it becomes "first browser-side plugin in Forge + extension dispatch mechanism".
- Extension code becomes thinner — no `defineBrowserConnector` SDK in extension itself; that lives as a Forge plugin author guide.
- `forge-browser-connectors` registry → **not needed**; reuse Forge plugin marketplace.

## ADR — 2026-05-16: Data architecture (what goes into Temper)

Key rule: **opportunistic memory, not bulk mirroring.** Temper is the
agent's *memory*, not a data lake. Mantis / Teams / Gmail stay as the
sources of truth; we don't sync everything wholesale.

### Data categories

| Category | Example | Storage | Why |
|---|---|---|---|
| Conversation | "user asked X, agent replied Y" | Temper **episodes** (graph) | Core memory asset |
| First-person facts | "user prefers concise answers", "user's team lead is Alice" | Temper **memory blocks** (KV) | Graphiti not great at these |
| Agent-processed content | bug #142 full text, meeting notes | Temper **wiki** (new) | RAG / long-form retrieval |
| Tool-call events | "called mantis.list_my_bugs, got 12 results" | Temper episode | Agent's own experience log |
| Transient search results | current Mantis result page, Teams query | Extension `chrome.storage` (TTL min) | Source is authoritative, refetch cheap |
| Files / code | git diff, file tree | NOT Temper — Forge workspace owns | Different concern |
| Credentials / cookies | tokens, passwords | NEVER STORED | Security red line |

### Three-tier data flow

```
Source system (Mantis / Teams / Gmail / GitHub)  ← single source of truth
    │ fetch on demand only
    ▼
Extension chrome.storage / IndexedDB (TTL minutes)  ← per-session cache
    │ promote to persistent memory when:
    │   - agent actually used the data (auto)
    │   - user said "remember this" (explicit)
    │   - agent flagged as significant (auto, heuristic)
    ▼
Temper (cross-session, cross-device, cross-agent)
    ├── episodes (graph) — chat + tool calls + agent events
    ├── memory blocks (KV) — first-person facts
    ├── wiki — agent-processed long-form content
    └── postgres (Temper internal) — user/auth/meta
```

### Write strategy (Week 2 implementation)

Every chat turn end, async write a batch to Temper:

1. **Episode for the conversation turn** — always (cheap, builds context)
2. **Episode per tool call** — always (summary, no raw secrets)
3. **Memory blocks** for first-person facts extracted by LLM from turn
4. **Wiki entry** when:
   - User says "记住 / remember this / save this"
   - Agent processes a specific document/ticket end-to-end (auto)
   - Agent generates a long-form artifact (status report, summary)

Sensitive fields (tokens, cookies, password-shaped strings) sanitized
before any Temper write.

### Read strategy (every turn start)

1. Episode semantic search by user message → top N (~10)
2. Load all memory blocks for current user
3. Wiki keyword/semantic search → top 3
4. Build LLM system context from above

### What to NEVER do

- ❌ Background cron mirroring Mantis / Teams full content into Temper
- ❌ Storing raw tool-call JSON unfiltered (noise + secrets risk)
- ❌ Using Temper as BI / aggregation store — it's per-user agent memory
- ❌ Storing files / code / large binaries — that's Forge workspace's job

### Wiki's specific role (new Temper primitive)

Wiki sits between KV (short, structured) and episodes (medium, eventful):

| Primitive | Length | Update freq | Retrieval | Example |
|---|---|---|---|---|
| KV blocks | sentence | low | by key | "team lead = Alice" |
| Episodes | paragraph | every turn | semantic recall | "5/16 fetched bug #142, user satisfied" |
| **Wiki** | document | when meaningful | full-text + semantic | "bug #142 full content + agent's deep analysis" |

Use wiki for:
- Specific ticket content the agent worked on (cached with backlink to source)
- "Save this Confluence page" requests
- Agent-generated reports / status updates / decision records
- Team-level SOPs and shared docs

### Integration timeline

| Week | What |
|---|---|
| Week 2 | Episodes (chat + tool calls) + KV blocks (extracted facts) write/read on every turn |
| Week 3 | Wiki writes when agent processes connector content end-to-end |
| Week 4+ | Smart promotion (episode → wiki by access frequency / agent tags) |
| Later | Forge pipelines also write episodes — chat and task share one memory plane |

### One-line principle

**Not "everything the connectors fetch goes to Temper" but "everything
the agent actually experiences goes to Temper".** Fetching is data
movement (don't auto-store); experiencing is agent memory (auto-store).

## ADR — 2026-05-16: DOM extraction is the default execution path

Surfaced after testing Mantis: REST API returned 403 because MantisBT's
`/api/rest/*` requires an `Authorization` token, not a cookie. Same
applies to GitLab `/api/v4/*`, JIRA Server REST, Teams Graph, etc.

**Decision**: connector tools run via **DOM extraction through the
extension's Chrome MCP capabilities**, not REST API calls.

### Why

If we route through REST, every connector requires the user to mint and
manage a per-system API token. That's exactly the friction the browser
extension model exists to remove. The whole product story is "use the
browser session you already have, no setup". DOM-first preserves that.

### Implementation rules

- Tool descriptions describe **observable behavior** ("list bugs assigned
  to me"), never URL paths.
- Extension's connector runtime navigates tabs and parses HTML — does NOT
  call `/api/rest/*` or `/api/v4/*`.
- `ctx.fetch` is reserved for same-origin XHR endpoints the rendered page
  itself uses (cookie-authenticated). Not for documented REST APIs.
- Token-based fallback (`api_token` setting) is **opt-in per connector**,
  only when DOM extraction is genuinely unworkable (canvas SPAs etc.).
  Off by default.

### Cost we accept

- Selector maintenance when sites redesign — pin stable attributes
  (`data-testid`, role-based queries, schema.org microdata).
- Slower (200ms-2s vs 50-500ms for REST) — acceptable for interactive use.
- Some pages need scrolling / interactions before all data is in DOM —
  connector handlers wait for selectors and trigger interactions as needed.

### What this changes

- RFC's three-tier model rewritten: DOM is Tier 1, vision is Tier 2,
  optional user-provided REST token is Tier 3 (escape hatch).
- Connector examples updated (no more "Tier 1 REST → Tier 2 DOM" — flipped).
- Mantis + GitLab manifests reflect this in their descriptions.

## Changelog

- **2026-05-16**: Initial plan
- **2026-05-16 (later)**: ADR — connectors live in Forge as plugins, not in extension. Major simplification.
- **2026-05-16 (later 2)**: ADR — data architecture: opportunistic memory not bulk mirroring; wiki for long-form agent-processed content.
- **2026-05-16 (later 3)**: ADR — DOM extraction is the default execution path, not REST. Token-based fallback is opt-in per connector.
