# Browser And Media Sidecar Design

This document describes the intended persistent browser/media subsystem for
SEED Harness. It is a design contract for future implementation, not a claim
that persistent sessions already exist.

## Current Baseline

SEED currently supports bounded browser snapshot capture and media inspection:

- `seed_browser_snapshot` creates an isolated one-shot browser snapshot from a
  local file or explicitly allowed external URL.
- `seed_look_at` inspects local files and `type: "seed.browser_snapshot"` JSON
  as untrusted evidence.
- Browser and document text are wrapped in trust-boundary language and passed
  through policy audit.
- Captured browser artifacts can include JSON, screenshot PNG, evidence refs,
  and artifact-index entries.

This is useful but not yet a sidecar. A sidecar means SEED can keep a browser
session alive across actions, attach observations to run state, and shut it
down predictably.

## Reference Signals

The design follows three source-grounded patterns from the graphify corpora:

- gstack treats persistent sidecars, command registries, auth/state files,
  transcript capture, and eval telemetry as runtime infrastructure rather than
  prompt prose.
- oh-my-openagent shows that long-lived session state needs explicit lifecycle
  management, hook integration, and log surfaces.
- Code as Agent Harness emphasizes sandboxed execution, permissioned state
  transitions, oracle adequacy, and multimodal harness systems.

## Target Module Boundary

Future implementation should create a focused feature module:

```text
src/features/browser-sidecar/
  provider.ts        browser provider interface
  session-store.ts   durable session metadata and leases
  policy.ts          browser/media capability checks
  snapshot.ts        observation and artifact schemas
  lifecycle.ts       start, heartbeat, reuse, close, crash recovery
```

The existing one-shot browser snapshot code can move behind this boundary
before persistent interaction is enabled. Tool names can remain stable while
their implementation becomes a thin adapter over the sidecar module.

## Runtime State

Sidecar state should live inside the active run, not in a global cache:

```text
.seed-agent/runs/<run_id>/browser/
  sessions.json
  observations.jsonl
  policy-decisions.jsonl
  screenshots/
  dom/
  accessibility/
```

Each session record should include:

- `session_id`, `run_id`, `created_at`, `updated_at`, `status`.
- `target_origin`, `current_url`, and trust classification.
- provider kind, process id when available, heartbeat timestamp, and lease
  owner.
- profile directory policy: isolated temp profile by default, persisted auth
  state only when explicitly configured.
- artifact refs for screenshots, DOM snapshots, accessibility tree captures,
  console/network summaries, and observations.

## Tool Surface

The first persistent tool set should be small:

- `seed_browser_open`: start or reuse a sidecar session for a URL or local file.
- `seed_browser_snapshot`: capture current screenshot, visible text, DOM
  summary, accessibility tree, and ref map from a session.
- `seed_browser_click`: click a stable ref from the latest observation.
- `seed_browser_type`: type into a stable ref, with redaction controls.
- `seed_browser_close`: close a session and release its lease.

Every tool should append observability records: JSONL trace, activity journal,
policy decision, and artifact index entries.

## Security Policy

Browser sidecar behavior must be policy-first.

- External content is untrusted unless a future capability policy explicitly
  marks it otherwise.
- Persistent auth state is disabled by default. Enabling it must require a named
  profile scope and professor-visible configuration.
- Downloads, file uploads, clipboard access, camera, microphone, geolocation,
  and notification permissions are denied by default.
- Navigation from a trusted local file to an external origin creates a policy
  event and trust-boundary downgrade.
- DOM/HTML text is never trusted as instruction text. Hidden text, ARIA/title
  injection cues, unsafe URLs, and canary leaks remain policy findings.
- Actions with destructive product impact, payment, account mutation, or data
  exfiltration risk require professor handoff unless a future capability matrix
  allows them.
- Screenshots and accessibility trees may become evidence refs, but the final
  report must cite the evidence ledger rather than raw browser state.

## Eval Requirements

The subsystem should not land without executable coverage:

- Unit tests for session-store migration, lease expiry, crash recovery, and
  policy decisions.
- Deterministic local HTML fixtures for hidden text stripping, ARIA injection,
  unsafe URLs, and stable ref maps.
- A black-box fixture that drives open -> snapshot -> click/type -> snapshot ->
  close and verifies final report, trace, activity journal, artifact index, and
  transcript together.
- A release audit check that browser sidecar artifacts, auth state, screenshots,
  and transcripts are not committed accidentally.
- Optional real-browser smoke only when a supported browser is available; the
  deterministic fixture remains the baseline.

## Implementation Sequence

1. Move current bounded snapshot code behind `browser-sidecar` provider and
   policy interfaces while preserving `seed_browser_snapshot` behavior.
2. Add durable `browser/sessions.json` and `browser/observations.jsonl` schemas
   with migration tests.
3. Add local-file persistent sessions with heartbeat and explicit close.
4. Add ref-map click/type tools for local deterministic fixtures.
5. Add external URL sessions behind explicit policy flags.
6. Add optional persisted auth-state profiles with strict scope and release
   audit protection.
