# QA runbook lifecycle (W-mpeiwz6k0005bf34)

Validation runbooks dispatched against live managed instances. Mirrors the
managed-spawn lifecycle (declare → engine spawns → healthcheck → observable)
but optimized for human/agent-driven smoke + E2E flows. Surfaced on the
`/qa` dashboard page (`dashboard/pages/qa.html`, `dashboard/js/qa.js`); full
live-process inventory remains on `/engine` (do NOT mirror it; see
W-mpdad3mq000m53bb).

## Runbook location

`qa-runbooks.json` (engine state, JSON list keyed by `id`). Each entry:
`{ id, name, target, steps, expectedArtifacts, createdAt, createdBy }`.
CRUD via `GET/POST /api/qa/runbooks` (POST returns the new runbook with
engine-assigned `id`). `target` is a name from `/api/managed-processes` or
`/api/keep-processes` (deduped by `<project>::<name>`, managed wins).

## Run-record path

`qa-runs.json` (newest-first, capped). Each run:
`{ id, runbookId, runbookName, target, status, startedAt, completedAt, workItemId, agentId, artifacts }`.
`status` ∈ `pending|dispatched|running|passed|failed|error`. Created by
`POST /api/qa/runbooks/run` (`{ id }`), which dispatches a `qa-validate`
work item and seeds the run with `dispatched`. Read via
`GET /api/qa/runs?limit=50` — UI polls every 5s while the QA page is active
and clears the interval on page navigation via the `switchPage` wrapper in
`dashboard/js/qa.js` (matches `_stopPlanPoll`/`_stopMeetingPoll` pattern in
`dashboard/js/state.js`).

### Persistence (Phase 8: SQLite source of truth)

Since the Phase 8 migration (`migrate-qa-state-to-sqlite`), both
`qa-runs.json` and `qa-sessions.json` are dual-written sidecars and SQLite
(`engine/state.db`, tables `qa_runs` + `qa_sessions`) is the source of
truth. Every mutation goes through `shared.mutateQaRuns` /
`shared.mutateQaSessions`, which apply inside a SQLite transaction
(`BEGIN IMMEDIATE`) then best-effort mirror to JSON under a `withFileLock`
on the sidecar path so cross-process writers serialize. Reads go through
`small-state-store.readQaRuns` / `readQaSessions`, which compare the SQL
content-hash to the JSON sidecar and rehydrate SQL from JSON when an
external editor (operator hand-edit, manual rollback) has diverged.

The JSON mirror is gated by `engine.qaDualWriteJson` (default true). Once
operators trust SQL as the source of truth and have migrated external
tooling off the JSON files, flip it false via Dashboard → Settings →
Advanced → "QA: dual-write JSON sidecars" to halve write I/O. With the
toggle off, `qa-runs.json` / `qa-sessions.json` stop receiving updates
(stale snapshot at the moment of the flip), so don't disable until any
external readers have moved to the SQL tables or the dashboard `/api/qa/*`
endpoints. The qa-runs cap (`QA_RUNS_MAX_RECORDS=2000`) and qa-sessions
cap (`QA_SESSIONS_MAX_RECORDS=500`) still apply, enforced in-memory at
createRun/createSession time.

### Deleting runs

`DELETE /api/qa/runs/<id>` hard-deletes a single terminal-status run
(`passed`/`failed`/`errored`) and best-effort removes its artifact
directory under `engine/qa-artifacts/<runId>/`. Non-terminal runs return
`409 not_terminal` — operators must cancel/complete the run via the
lifecycle helpers first. Invalid ids return `400 invalid_id`. The /qa
dashboard page renders a per-row Delete button conditionally (terminal
only), confirms with the user, and surfaces success/error via
`showToast()` rather than `alert()`.

## Artifact contract

`engine/qa-artifacts/<runId>/<file>`, served via
`GET /api/qa/artifacts/<runId>/<file>`. Files are agent-uploaded screenshots,
video recordings, and log captures listed in the run record's
`artifacts: [{ file, kind }]`. UI auto-detects:
`screenshot`/`png|jpg|jpeg|gif|webp|svg` → `<img>`;
`video`/`mp4|webm|ogg|mov` → `<video controls>`; everything else → log
preview (first 40 lines fetched lazily) with a `View full` link to the same
endpoint. **No direct filesystem paths are exposed** — every artifact URL
goes through `/api/qa/artifacts/` so path traversal is server-gated.
Optional config: `engine.qaArtifactsMaxBytes` caps per-file upload size;
when set, dashboard Settings exposes a matching toggle (CLAUDE.md
best-practice #15).

## Agent sidecar shape

The `qa-validate` agent writes `agents/<id>/qa-run.json` before exit:

```json
{ "runId": "qa-run-<id>",
  "status": "passed|failed|error",
  "summary": "...",
  "artifacts": [ { "file": "dashboard.png", "kind": "screenshot" },
                 { "file": "test.log",      "kind": "log" } ],
  "written_by": "<agentId>", "wi_id": "<workItemId>" }
```

The engine reads the sidecar in `onAgentClose`, copies the listed files into
`engine/qa-artifacts/<runId>/`, and stamps the run record with `status`,
`completedAt`, and the recorded `artifacts`. Sidecar validation failure is
non-retryable (`failure_class: invalid-qa-run`); listed files outside the
agent worktree or larger than `qaArtifactsMaxBytes` are rejected without
stamping the run.

## Entry point

`playbooks/qa-validate.md`. Dispatched by `POST /api/qa/runbooks/run`;
receives `target`, `steps`, `expectedArtifacts` as template vars; required
to write the sidecar above before exit. Routing line in `routing.md` maps
the synthetic `qa-validate` task-type to the playbook so manual dispatches
work too.

# QA Sessions (PL-qa-sessions)

Natural-language QA flows: the user describes what they want tested in
plain prose, the engine sets up a live target, an agent drafts a
runner-native test file, and (with user approval) a second agent executes
it. Sessions are a thin orchestration layer on top of the same
`managed-spawn` + `qa-run-result.json` infrastructure that powers
runbooks above — they reuse `qa-runs.json`, `engine/qa-artifacts/`, and
the existing `engine/lifecycle.js#runPostCompletionHooks` qa-run sidecar
hook. Surfaced on `/qa` (sessions card list above the runbooks/runs
tables) and proxied by the Command Center natural-language shortcut.

## Use case (vs. runbooks)

- **Runbook**: a stable, named, reusable test plan stored as JSON. Same
  steps run repeatedly against the same target. Edit once, dispatch
  many times.
- **Session**: an ad-hoc, single-shot, natural-language QA prompt. The
  user describes intent ("smoke the homepage after my PR"), engine
  resolves the target + spawns it, an agent drafts a *new* runner-native
  test file just for this run. The drafted file lives at
  `engine/qa-tests/<sessionId>/test.<ext>` and is the audit artifact; the
  session record is the orchestration breadcrumb.

If the same flow is going to be re-run repeatedly, promote the drafted
test file into a saved runbook. Sessions are *not* meant to replace
runbooks for repeat traffic.

## State machine

Eight states, source of truth `engine/qa-sessions.js#QA_SESSION_STATE`:

```
pending ──▶ spawning ──▶ drafting ──▶ awaiting-approval ──▶ executing ──▶ done
   │            │             │                │                │           ╲
   ╰────────────┴─────────────┴────────────────┴────────────────┴───────────▶ failed
   ╰────────────┴─────────────┴────────────────┴────────────────┴───────────▶ killed
```

Forward transitions only; `done` / `failed` / `killed` are terminal (zero
outgoing edges). Two non-linear edges:

- `awaiting-approval → drafting` — `POST /api/qa/sessions/<id>/edit` re-fires
  DRAFT with the user's natural-language feedback as steering.
- `drafting → executing` — `auto` mode skips approval and chains EXECUTE
  directly from DRAFT-done.

Cancellation paths (`pending → killed`, `spawning → killed`, etc.) cover
the `POST /api/qa/sessions/<id>/cancel` and `/kill` endpoints. Illegal
transitions throw; `dashboard.js` translates the thrown error to HTTP 409
via `_qaSessionsErrorToStatus` (see endpoint table).

## Three-phase work-item chain

A session orchestrates three sequential work items, each rendered with a
phase-specific playbook (`meta.playbook` override beats the work-type
routing default):

1. **SETUP** (`playbooks/qa-session-setup.md`, `meta.managed_spawn: true`)
   resolves `target.kind ∈ {pr, branch, current, commit}` to a worktree,
   inspects the codebase for the dev-up command, and writes
   `agents/<agentId>/managed-spawn.json` with
   `name=qa-session-<sessionId>`. The engine spawn-validates the sidecar
   then awaits `waitForFirstHealth` per the existing managed-spawn
   lifecycle (see `docs/managed-spawn.md`). Healthcheck-passing transitions
   the session `spawning → drafting`. First-health failure → `failed` with
   `failure_class: qa-session-setup-failed`.

   **Multi-project fan-out (W-mpq6xqzj000606d0):** when `spec.projects`
   contains more than one project name, SETUP is fanned out — one work item
   per project, queued in parallel into each project's own
   `work-items.json`. The first project (`spec.projects[0]`) is the
   **primary** (`meta.qaSession.primary === true`); the rest are
   **co-services** (`primary === false`). Each WI's
   `meta.qaSession.coServices` lists the co-service project names (only the
   primary carries the full list; co-services see `[]`) and
   `meta.qaSession.primaryProject` carries the canonical primary name.
   The primary keeps the unsuffixed managed-spawn name; each co-service
   gets `qa-session-<sessionId>-<project>`. The session tracks per-project
   completion in `session.setupStatus[<project>]: { state, wiId, error,
   completedAt }`, where `state ∈ {pending, success, failed}`. **All**
   projects must succeed before `spawning → drafting`; any failure flips
   the session to `failed` with `failure_class: qa-session-setup-failed`
   and a JSON-encoded per-project error array in `session.error`. DRAFT
   queues on `session.primaryWiPath` (stamped at queueSetup time).
   Single-project sessions skip `setupStatus` entirely (fast path uses
   `session.state` directly).

2. **DRAFT** (`playbooks/qa-session-draft.md`) reads the live spawn
   metadata via `/api/managed-processes/by-name/qa-session-<id>`, calls the
   resolved runner's `generateBrief({target, flowsRaw, capture})` hook, and
   writes the runner-native test file to
   `engine/qa-tests/<sessionId>/test.<ext>`. The runner's
   `validateOutputDir` hook gates DRAFT success — a missing or malformed
   test file lands the session in `failed`. In `confirm` mode the session
   parks at `awaiting-approval`; in `auto` mode it auto-chains to EXECUTE.

3. **EXECUTE** (`playbooks/qa-session-execute.md`) runs the drafted test
   against the live spawn via the runner's `executeBrief` hook, captures
   artifacts per `spec.capture: { video, screenshots, logs }`, and writes
   `agents/<agentId>/qa-run-result.json` — the same sidecar the
   `qa-validate` runbook flow uses. The existing
   `engine/lifecycle.js#runPostCompletionHooks` `meta.qaRunId` hook ingests
   the sidecar and marks the linked `qa-runs` record terminal; the
   session-level `handleExecuteComplete` then reads the `qa-run` terminal
   status and transitions `executing → done` (or `failed`).

Each phase WI carries `meta.sessionId`, `meta.sessionPhase`, `meta.qaSession`
(target + flowsRaw + mode + capture + runner), and `meta.playbook`. The
EXECUTE WI additionally carries `meta.qaRunId` so the existing qa-run
lifecycle hook fires.

## Endpoints

Documented in `dashboard.js`; routes are visible at `GET /api/routes`.

| Method | Path                                       | Behavior                                                                                                                  |
|--------|--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
| POST   | `/api/qa/session`                          | Create session; validates spec, calls `createSession` + `queueSetup` (`pending → spawning`). Body accepts `project: string` (single) OR `projects: string[]` (multi, ≤5; first is primary, rest are co-services). Returns `sessionId`, `setupWorkItemId` (primary's WI), `managedSpawnName`, and (multi only) `projects: string[]`. |
| GET    | `/api/qa/sessions`                         | List sessions newest-first. Optional `?limit=N` and `?state=pending\|spawning\|drafting\|awaiting-approval\|executing\|done\|failed\|killed`. |
| GET    | `/api/qa/sessions/<id>`                    | Fetch a single session record by id.                                                                                      |
| POST   | `/api/qa/sessions/<id>/approve`            | `awaiting-approval → executing`. Server-side creates the linked `qa-runs` record (synthetic `runbookId='qa-session-<id>'`), queues EXECUTE WI, stamps `qaRunId` on the session. |
| POST   | `/api/qa/sessions/<id>/edit`               | `awaiting-approval → drafting`. Body: `{ feedback }`. Re-fires DRAFT with the reviewer feedback threaded into the prompt. |
| POST   | `/api/qa/sessions/<id>/cancel`             | Non-terminal → `killed`. Optional `{ reason }`. Does NOT touch the managed-spawn — use `/kill` for that.                  |
| POST   | `/api/qa/sessions/<id>/kill`               | Non-terminal → `killed` AND `removeManagedSpec(<name>)`. Best-effort on the spawn kill (no-op if no spawn).                |
| POST   | `/api/qa/sessions/<id>/dismiss`            | Non-terminal → `done`. Accept the draft as final; leaves spawn alive. Optional `{ summary }`.                              |
| GET    | `/api/qa/runners`                          | List registered runner adapters (built-ins + `qa-runners.d/` plugins). Metadata only — hooks (functions) are stripped.    |
| POST   | `/api/qa/runners/reload`                   | Clear in-process registry, re-register built-ins, re-scan `qa-runners.d/` for plugin edits. Returns the fresh runner list. |
| DELETE | `/api/qa/runs/<id>`                        | Hard-delete a single QA run record (terminal-status runs only) and best-effort wipe its artifact directory under `engine/qa-artifacts/<runId>/`. Returns `{ ok: true, id, artifactsRemoved }` on success, `409 not_terminal` for in-flight runs, `404 not_found` for unknown ids. |

The single-session POSTs share `_qaSessionAction` in `dashboard.js`; module
errors are mapped to HTTP via `_qaSessionsErrorToStatus`:

- `'session not found'` → 404
- `'unsafe sessionId'` / `'invalid spec'` / `'requires …'` (size cap) → 400
- `'illegal state transition'` / `'requires state …'` / `'requires non-terminal'` → 409

## File locations

- **Session state**: `engine/qa-sessions.json` (single file, all projects,
  capped at `QA_SESSIONS_MAX_RECORDS = 500` via `createSession`-time
  rotation by `createdAt`).
- **Drafted test files**: `engine/qa-tests/<sessionId>/test.<ext>`. Created
  by `createSession` (idempotent `mkdirSync recursive`); written by the
  DRAFT agent. Path is sandboxed by `_isSafeSessionId` (kebab-case ≤ 64
  chars, no `..`, no separators) before every read/write that maps id →
  path. Gitignored.
- **Managed-spawn**: `engine/managed-processes.json` entry with
  `name='qa-session-<sessionId>'`. Cleaned up by `/kill` or by the
  session's `cancelSession` if the operator opts for the cancel-without-
  spawn-kill path.
- **EXECUTE artifacts**: same as runbooks — `engine/qa-artifacts/<runId>/`
  served via `GET /api/qa/artifacts/<runId>/<file>`.

## Runner adapters (P-c4a9e7f3 / P-b8e1d4a6)

Pluggable test-runner registry at `engine/qa-runners.js`. Built-in
adapters: `playwright` (priority 100, detects `playwright.config.*`),
`maestro` (priority 80, detects `.maestro/`). Each adapter exports five
hooks:

- `detect(target, project)` → `boolean`. Auto-detect for blank-runner
  sessions.
- `generateBrief({target, flowsRaw, capture, project})` → instructions
  string. Handed to the DRAFT agent so it emits a runner-native test file.
- `executeBrief({sessionId, target, capture, project})` → instructions /
  command for the EXECUTE agent.
- `validateOutputDir(dir)` → `{ ok, errors[] }`. Gates the DRAFT → EXECUTE
  transition.
- `installHint` → string shown when `detect()` returns true but the
  runner CLI is missing.

Resolution order in `detectRunner(target, project, explicitRunner)`:
explicit-name (no `detect` call, unknown names return null), then
priority-desc iteration. Plugin folder: `<MINIONS_DIR>/qa-runners.d/*.js`
(same trust level as `playbooks/` and `watches.d/`). Hot-reload via
`POST /api/qa/runners/reload` (clears registry → re-registers built-ins →
re-scans plugin dir) so plugin edits take effect without an engine
restart.

## Fast-state slice

`/api/status.qaSessions = { total, sig }` — the unsorted summary helper
`engine/qa-sessions.js#summarizeSessionsForStatus()`. Mirrors `qaRuns` so
the sidebar activity-dot lights up on any new session or state
transition within one `/api/status` poll cycle (~4s). Do NOT call
`listSessions({limit:50})` from this hot path — it sorts O(N log N) on
every fast-state rebuild.

## Dashboard UI (P-h7e4f9b2)

`/qa` page (`dashboard/pages/qa.html`, `dashboard/js/qa.js`) gains a
**QA Sessions** section above the existing Targets/Runbooks/Runs trio:

- **Start QA Session form** — target kind dropdown with conditional
  sub-fields (PR id, branch, commit SHA, worktree path), flows textarea,
  mode toggle (`confirm`|`auto`), capture checkboxes, runner dropdown
  auto-populated from `GET /api/qa/runners`, project input.
- **Sessions list** — composite cards with phase chips
  (🔧 setup → 📝 draft → ▶ execute → ✅ done). Cards visually flip the
  chip classes (`--done` / `--active` / `--pending` / `--failed` /
  `--killed`) per `session.state`. State-driven left-border color
  (red=failed, green=done, yellow=awaiting-approval, blue=active).
- **Action buttons** —
  `awaiting-approval` cards show `[Approve & run]` `[Edit]` `[Cancel]`;
  every non-terminal card shows `[Dismiss]` `[Kill spawn]` in the footer;
  terminal cards show no actions.
- **Polling** — `_startQaSessionsPoll` runs a 3000 ms
  `setInterval(loadQaSessions)`; `_qaAfterSessionsRender` auto-stops the
  poll once every cached session is in
  `QA_SESSION_TERMINAL_STATES = {done, failed, killed}`. Form-submit and
  action handlers each call `_startQaSessionsPoll` to re-activate polling
  when a new non-terminal session appears. Wired into the canonical page
  lifecycle: `loadQaRunners` + `loadQaSessions` + `_startQaSessionsPoll`
  live in `PAGE_LAZY_LOADERS.qa`; `_stopQaSessionsPoll` lives in
  `PAGE_LEAVE_HOOKS`.

## Command Center shortcut (PR12: CC system prompt)

The CC system prompt teaches natural-language → `POST /api/qa/session`.
The user can type "smoke test the home page in PR #2887" and CC builds
the spec (`target.kind='pr'`, `target.prId='github:yemi33/minions#2887'`,
`flowsRaw='smoke test the home page'`, default `mode='confirm'`) and
fires the endpoint. CC must include the
`X-Minions-Agent: cc-<turn-id>` header so the session's `createdBy`
field gets the right audit trail.

## When something goes wrong

- **SETUP managed-spawn won't validate** → session lands in `failed` with
  `failure_class: 'invalid-managed-spawn'` (from
  `evaluateManagedSpawnAcceptance` in `onAgentClose`) or
  `'managed-spawn-healthcheck-failed'`. The `error` field carries the
  reason; check `engine/managed-logs/qa-session-<sessionId>.log` for the
  failing healthcheck.
- **DRAFT validateOutputDir rejects** → `failure_class:
  'qa-session-draft-failed'`. Inspect the drafted file at
  `engine/qa-tests/<sessionId>/`; the runner's `validateOutputDir.errors`
  array tells you what's missing.
- **EXECUTE qa-run terminal status is `failed`/`errored`** →
  `failure_class: 'qa-session-execute-failed'` /
  `'qa-session-execute-errored'`. The linked `qa-runs` record (joined via
  `session.qaRunId`) carries the agent's `summary` and artifact list.
- **Want to start over after seeing a bad draft** → POST
  `/api/qa/sessions/<id>/edit` with `{ feedback: "…" }`; do NOT
  `/cancel` + create a new session unless the original spec was wrong
  (the test file from the prior DRAFT round stays on disk under
  `engine/qa-tests/<sessionId>/` as `test.<ext>.bak.<round>` so you can
  diff iterations).
- **Spawn is wedged but session is still in `executing`** → POST
  `/api/qa/sessions/<id>/kill` (NOT `/cancel` — the latter leaves the
  spawn alive).
