# 20 — Stability Benchmark Suite (oracle-derived)

Date: 2026-06-08
Status: active
Derived from: oracle vs agbrowse gap analysis (docs 01–08)

## Purpose

Task-completion benchmarks (WebVoyager, WebArena) measure whether the agent
answers correctly. Stability benchmarks measure whether the browser automation
layer performs its operations reliably — independent of the planner or model.

These benchmarks directly measure the surface area hardened in the P0 patches
and serve as regression gates for future changes.

## Benchmark Categories

### B1 — Send Button Reliability

**What it measures:** Success rate of send button click across DOM variations.

| Metric | Definition | Target |
|--------|-----------|--------|
| `send.selector_coverage` | % of known ChatGPT DOM variations matched by SEND_BUTTON_SELECTORS | ≥95% |
| `send.timeout_adequacy` | Whether timeout values match oracle parity (20s text, 45s attachment) | exact match |
| `send.fallback_chain` | Button click → Enter key fallback path exists | boolean true |
| `send.commit_signals` | Count of independent commit verification signals | ≥4 |

**Testable offline:** selector matching, timeout values, fallback chain existence.
**Requires live browser:** actual send success rate across models (instant/thinking/pro).

### B2 — Attachment Upload Reliability

**What it measures:** Chip detection and upload readiness across file types.

| Metric | Definition | Target |
|--------|-----------|--------|
| `attach.chip_wait_timeout` | Max wait for attachment chip readiness | 45 s |
| `attach.preflight_rejects` | Whether oversized files are rejected before upload | boolean true |
| `attach.file_type_routing` | Image vs non-image file input scoring correctness | 100% |

**Testable offline:** timeout values, preflight logic, scoring function.
**Requires live browser:** actual chip appearance across file types/sizes.

### B3 — Error Classification Coverage

**What it measures:** Whether the error taxonomy covers oracle's failure modes.

| Metric | Definition | Target |
|--------|-----------|--------|
| `error.code_count` | Number of distinct error codes in taxonomy | ≥12 |
| `error.oracle_coverage` | % of oracle error categories mapped to agbrowse codes | ≥80% |
| `error.retry_hint_present` | % of error codes with `retryHint` guidance | 100% |

Oracle error categories (3-tier):
- OracleUserError → maps to `provider.*`, `context.*`
- OracleTransportError → maps to `cdp.*`
- OracleResponseError → maps to `provider.poll-timeout`, `source-audit.failed`

### B4 — Session Resilience (live-only)

**What it measures:** Recovery success after CDP disconnection.

| Metric | Definition | Target |
|--------|-----------|--------|
| `session.reconnect_success` | % of successful reconnections after Chrome restart | ≥90% |
| `session.state_preserved` | Whether page URL survives reconnection | boolean |
| `session.stale_ref_detection` | Whether stale refs from pre-disconnect are rejected | boolean true |

**Requires live browser:** all metrics.

### B5 — Response Capture Completeness (live-only)

**What it measures:** Whether full responses are captured vs truncated.

| Metric | Definition | Target |
|--------|-----------|--------|
| `capture.full_text_rate` | % of responses captured in full (vs truncated) | ≥95% |
| `capture.streaming_recovery` | Whether polling recovers mid-stream after stall | boolean |
| `capture.exactness_score` | answerArtifact exactnessScore distribution | mean ≥0.9 |

**Requires live browser:** all metrics.

## Offline Test Coverage (unit-testable)

Benchmarks B1–B3 have offline-testable components. These are implemented in:
`test/unit/stability-benchmarks.test.mjs`

Running: `npx vitest run test/unit/stability-benchmarks.test.mjs`

## Live Test Protocol

B4–B5 and the live components of B1–B3 require:
1. Chrome with ChatGPT session active
2. agbrowse connected via CDP
3. Pro account (for model switching tests)

These are run manually during QA or via the smoke test harness at
`devlog/_smoke/`.

## Scoring Rules

Same policy as existing benchmarks — no leaderboard claims:
- ✅ Internal regression tracking with pass/fail gates
- ✅ Comparison against oracle baseline thresholds
- ❌ No published "agbrowse stability score" or "X% reliable"
- ❌ No head-to-head claims against oracle or other tools
