Build Observability

Why Build Observability exists

When several agents (and humans) build a project in parallel worktrees, the reasoning behind the build evaporates. A decision gets made in a throwaway branch, a blocker is hit and worked around, a story is silently dropped — and none of it survives the squash-merge. The docs say one thing; the code drifts to another. Build Observability turns those ephemeral moments into a durable, queryable record, then audits the record against the planning docs.

What goes wrong without it

• Decisions die at teardown. An agent records "we switched the cache to Redis" in its head, opens a PR, the worktree is removed — and the rationale is gone. The next agent re-litigates it.
• Blockers go unaddressed. A dependency block is hit at hour 0 and nobody notices it is still open at hour 6.
• Phase boundaries pass unaudited. tech-stack.md is marked complete while it still contradicts the PRD; the contradiction is found three phases later.
• Doc/code drift accumulates. A component is used outside its declared layer; an acceptance criterion never gets a test; a coding-standard erodes file by file.

The classic failure mode is a worktree teardown that takes the ledger with it: an agent worktree's .scaffold/activity.jsonl is lost the moment the worktree is removed, unless it is harvested into the primary archive first. The harvester (src/observability/engine/harvester.ts) and scripts/teardown-agent-worktree.sh exist precisely to close this gap — see Harvest, recover & teardown.

What good build observability produces

• Decisions captured the moment they're made (decision_recorded), surviving teardown via harvest.
• Blockers surfaced when they linger past a threshold (stall detection).
• Phase-boundary audits that run automatically when a planning doc is completed, non-gating so they never block progress.
• Doc/code drift caught by the nine-lens audit and routed into code review via the MMR doc-conformance channel.

The system ships 9 durable event types, 8 adapters fusing external signals, a 9-lens audit (A–I), and 5 active stall signals (a 6th reserved) on the "Needs Attention" surface.

System map

One write path, one fusion point, two read paths. Agents write events to the append-only ledger; adapters synthesize events from git, GitHub, MMR and pipeline state; the synthesizer fuses them with correlation-id dedup; the engine API drives the audit lenses and the progress timeline; renderers emit terminal, markdown, dashboard and MMR-findings output.

The boxes labelled probe (pipeline_docs, beads, audit_history) contribute availability checks — and, for audit_history, trend data — but do not emit events into the replay timeline. The five remaining adapters (git, gh, mmr, state, tests) synthesize replay events that flow into the synthesizer.

The ledger

Every durable observation is one JSON object on one line of .scaffold/activity.jsonl. Writes are append-only and lock-guarded so parallel worktrees never corrupt the file; each event is capped at 4 KiB; and secrets and home-directory paths are scrubbed both when the event is written and again when output is rendered.

How a write happens

• Validate against the per-type payload allow-list (src/observability/engine/event-schemas.ts). Unknown payload keys are dropped (reported in dropped_fields); a bad ts or missing required field rejects the event.
• Redact the payload (src/observability/engine/redact.ts) — see below.
• Size-check: Buffer.byteLength(line, 'utf8') must be ≤ 4096, else the write throws (src/observability/engine/ledger-writer.ts:10, src/observability/engine/ledger-writer.ts:58).
• Append under a proper-lockfile lock (10 retries, exponential backoff) so concurrent agents serialize cleanly.
• Link (fail-soft): a task_claimed event with a task_id also links the task into Beads via bd (beadsAdapter.claimWithEvent); any Beads error is swallowed — the ledger write already succeeded.

The nine event types

Each event carries a common envelope — event_id (ULID), worktree_id, actor_label, branch, task_id (string or null), type, ts (ISO-8601 UTC) — plus a type-specific payload. The CLI verb is scaffold observe event <type> ….

* marks required fields. Allow-lists live in EVENT_PAYLOAD_KEYS at src/observability/engine/event-schemas.ts:3-13.

Building a scaffold observe event command

--branch is required; --task-id is optional (it correlates the event to a claimed task) but must be omitted for finding_acknowledged (whose task_id must be null), and a task_claimed without one must pass --unplanned true. Payload keys are passed as --kebab-case flags, snake-cased before validation; unknown keys are dropped.

scaffold observe event task_claimed --branch <branch> --task-id <id> \
  --task-title "<title>" [--story-id <id>] [--wave <n>] [--unplanned true]
scaffold observe event task_completed --branch <branch> --task-id <id> \
  --outcome pr_submitted --pr-number <n> [--commit-sha <sha>]
scaffold observe event decision_recorded --branch <branch> --task-id <id> \
  --key <key> --summary "<text>" --affects "src/**,docs/**" [--links "ADR-1,ADR-2"]
scaffold observe event blocker_hit --branch <branch> --task-id <id> \
  --kind dependency --summary "<text>"
scaffold observe event blocker_resolved --branch <branch> --task-id <id> \
  --summary "<text>" --references "ref1,ref2"
scaffold observe event pr_opened --branch <branch> --task-id <id> --pr-number <n>
scaffold observe event progress_heartbeat --branch <branch> --task-id <id> --note "<text>"
scaffold observe event finding_acknowledged --branch <branch> \
  --finding-id <id> --status acknowledged [--note "<text>"]
scaffold observe event knowledge_gap_signal --branch <branch> \
  --topic <kebab-slug> --source agent_search --project-id <sha256> \
  [--step-name <name>] [--agent-excerpt "<text>"]

finding_acknowledged is normally written by scaffold observe ack, not by hand.

Redaction — write time and render time

Redaction runs twice. Write-time redaction (recursive over the payload tree) scrubs the event before it lands on disk; render-time redaction (redactEngineOutput / redactRendered) scrubs string values again before any report is shown, catching anything synthesized after the fact.

Order matters: a bare ghp_… matches the GitHub-token pattern, but inside a token=… pair the key/value rule wins first — so token=ghp_… redacts to [REDACTED:kv-secret], not [REDACTED:github-token] (leftmost-match alternation, src/observability/engine/redact.ts:16).

A redacted decision_recorded event on disk (one line, shown pretty-printed):

{
  "event_id":    "01H5ZABCDEFGHJKMNPQRSTVWX",
  "worktree_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "actor_label": "agent-alice",
  "branch":      "alice-feat/cache",
  "task_id":     "T-014",
  "type":        "decision_recorded",
  "ts":          "2026-05-04T12:00:00Z",
  "payload": {
    "key":     "cache-backend",
    "summary": "Switched to Redis; token=[REDACTED:kv-secret]",
    "affects": ["~/repo/src/cache/**"],
    "links":   ["ADR-021"]
  }
}

Worktree identity

Each worktree gets a stable identity on first write — .scaffold/identity.json holds worktree_id (a UUID), worktree_label (derived from the directory name, or primary), and created_at (src/observability/engine/identity.ts). The id is what lets the harvester tell one worktree's events from another's, and what stamps every event's worktree_id.

Adapters

The ledger only holds what agents chose to record. Adapters fill in the rest by synthesizing events from the surrounding tools — commits, PRs, MMR jobs, pipeline state, test runs. There are eight adapters (AdapterId at src/observability/engine/types.ts:69), but only five emit events into the timeline; the other three are availability probes / trend helpers.

Source priority

When two events describe the same thing (same correlation_id), the synthesizer keeps the one from the most authoritative source. The order is fixed in SOURCE_PRIORITY at src/observability/engine/synthesizer.ts:254 — lower number wins, ties broken by earliest ts:

The three probe-only adapters are intentionally absent from this table — they never produce a replay event to dedup.

The nine-lens audit

The audit reads the planning docs (PRD, stories, tech-stack, standards, design system, plan, playbook) into a document graph, then runs a set of independent lenses over the graph, the code, the ledger and adapter data. The design spec calls it the "eight-lens audit" (A–H); a ninth lens — I-knowledge-gaps — joined the suite after v3.26.0.

Scope, order and skips

• Scope → lenses (src/observability/engine/api.ts:72): --scope code runs A–G; --scope docs runs H, I; --scope all runs all nine. An explicit --lens <id> overrides the scope selection.
• Registry order: A → B → C → D → E → F → G → H → I. G-decisions declares depends_on: ['D-stack'] so it runs after D and can consume D's unsanctioned-dependency findings.
• Missing adapter → skip: if a lens's required adapters are unavailable it is skipped and emits a P3 finding with evidence.kind = lens_skipped. Skipped lenses (not blocking findings) are what turn a pass into a degraded-pass.

What feeds the doc-graph

Before any lens runs, the planning docs and the working tree are parsed into a single graph (src/observability/engine/doc-graph/index.ts). Every lens's "Reads:" line refers to nodes and edges built here:

All nine at a glance

Lens by lens

A-tdd — TDD violations (src/observability/checks/lens-a-tdd.ts). Are there skipped tests? Reads: the doc-graph's discovered tests (with last_status), ACs, stories, and ac_to_test edges. Tunes: none. Severity: P0 skipped test on a must-priority story; P1 otherwise.

B-ac-coverage — AC completion (src/observability/checks/lens-b-ac-coverage.ts). Is every acceptance criterion covered by a passing test? Reads: ACs, tests, ac_to_test edges; optional tests adapter for live status. Tunes: none. Severity: P0 AC's test is failing; P1 AC has no test edge or unknown status.

C-standards — coding-standards drift (src/observability/checks/lens-c-standards.ts). Are coding standards being violated across files? Reads: graph rules + files. Tunes: lenses.C-standards.rule_overrides (per-rule severity), escalation_threshold (default 5 — more than N violations of a rule escalates to P1). (enforce_via_linter is accepted in config but not read by the lens.) Severity: rule-specific P0–P3.

D-stack — tech-stack drift (src/observability/checks/lens-d-stack.ts). Unsanctioned dependencies, or components used outside their layer? Reads: file_to_component_use edges, components (with layer), and decision_recorded ledger events (which can sanction a dependency). Tunes: lenses.D-stack.path_to_layer (default 7 glob→layer mappings). Severity: P0 unsanctioned dependency; P1 out-of-layer use.

E-design — design-system drift (src/observability/checks/lens-e-design.ts). Are ad-hoc design values bypassing tokens? Reads: file_to_token_use edges, tokens (with priority + category). Tunes: lenses.E-design.ad_hoc_token_threshold (default 3), ui_glob (default src/components/**/*.tsx, consumed by the doc-graph builder). Severity: P1 ad-hoc count over threshold; P0 must-priority token category bypassed.

F-scope — missing scope (src/observability/checks/lens-f-scope.ts). Is high-priority work missing stories/plans, or are planned stories untouched? Reads: features, stories, plan-tasks and their edges; task_claimed ledger events; optional state adapter. Tunes: lenses.F-scope.untouched_story_grace_hours (default 168 = 7 days). Severity: P0 must-feature without story/plan; P1 should-feature; P2 planned-but-untouched story past grace.

G-decisions — undocumented decisions (src/observability/checks/lens-g-decisions.ts). Decisions made in the ledger/commits but not documented (or vice versa)? Reads: graph decisions, decision_recorded events, recent git commits (keyword scan), and D-stack's findings. Tunes: none effective (see note). Severity: P0 unsanctioned dep from D with no decision; P1 ledger↔doc mismatch; P2 decision-keyword commit with no event/doc.

H-cross-doc — cross-doc inconsistency (src/observability/checks/lens-h-cross-doc.ts). Are features, stories, plans, playbook and decisions internally consistent? Reads: the full structural doc-graph + unresolved globs. Severity: P0 decision supersedes a non-existent decision, or a must-priority story uncovered by plan/playbook; P1 feature with no story / orphan story / should-priority uncovered story; P2 plan task not in playbook, unresolved glob.

Under --profile=full only, Lens H runs three extra checks via an LLM dispatcher (all silently return [] if the dispatcher fails, so the audit never breaks):

• tech-stack ↔ PRD — contradictions (P0) or tensions (P2) between PRD prose and the chosen stack.
• PRD → stories coverage — a prose feature with no covering story (P1).
• terminology drift — one concept named inconsistently across docs (P2).

I-knowledge-gaps — the ninth lens (src/observability/checks/lens-i-knowledge-gaps.ts). What topics have agents repeatedly needed (over a 90-day window) with no knowledge-base entry? Reads: knowledge_gap_signal ledger events + synthetic signals scanned from tasks/lessons.md (src/observability/checks/lens-i-lessons-scanner.ts). Severity: P1 ≥5 signals from ≥3 projects; P2 ≥3 signals from ≥2 projects.

The observe audit CLI

One command runs the lenses and reports findings. Its exit code is the verdict gate: 0 on a non-blocked verdict (or any mmr-findings run), 1 when blocked or when a --fix pass still failed, 3 on an audit error.

Every flag

# A docs-only full audit including LLM-graded checks
scaffold observe audit --scope docs --profile full

# Audit, then dispatch a fix agent for each blocking finding
scaffold observe audit --fix --fix-threshold P1

# Lens I only, with an explicit knowledge root for suppression
scaffold observe audit --lens I-knowledge-gaps --knowledge-root content/knowledge

Verdict taxonomy

The engine computes exactly three verdicts (Verdict at src/observability/engine/types.ts:6):

fix_threshold

A finding is blocking when its status is open and its severity is at or above the threshold (src/observability/engine/checks/fix-threshold.ts). Resolution order: --fix-threshold flag → .mmr.yaml (audit_fix_threshold / fix_threshold) → default P2. The threshold never hides findings — every finding is still reported; it only decides which ones count toward blocked.

Companion verb — scaffold observe ack

Acknowledge (or reopen) a finding by ID prefix; it writes a finding_acknowledged event the aggregator reads on the next audit.

Branch is auto-derived (git rev-parse --abbrev-ref HEAD, fallback main); the event's task_id is null.

Exit codes across observe verbs

Progress, replay & stall

scaffold observe progress shows what's in flight. With --replay it fuses the ledger with synthesized git/gh/mmr/state/tests events into one timeline; when scaffold observe progress runs it also evaluates stall detection and surfaces a "Needs Attention" panel (suppress with --no-stall-check).

Fusion — how the timeline is built

The synthesizer's composeReplay (src/observability/engine/synthesizer.ts:294) converts each ledger Event to a ReplayEvent, concatenates the adapter events, filters to the time window, then dedups:

• Events with a null correlation_id pass through untouched.
• Events that share a correlation_id collapse to one — the synthesizer keeps the lowest SOURCE_PRIORITY number (ledger 0 → tests 5), breaking ties by earliest ts.
• The surviving events sort by (ts, source-priority, sort_id).

So a PR that the agent recorded (pr_opened in the ledger) and that the gh adapter also reports collapses to the single ledger event, because ledger (0) outranks gh (2).

Stall detection — the six signals

When scaffold observe progress runs (unless --no-stall-check) the engine checks for staleness and builds needs_attention (src/observability/engine/stall.ts). The type declares six signals; five fire today — pr_review_stale is reserved (deferred pending a per-PR correlation_id on the mmr adapter, src/observability/engine/stall.ts:119). Thresholds accept 'Nh', 'Nd', or 'off' under stall.*:

Flags

Stall thresholds are configurable under stall: in .scaffold/observability.yaml — see Config reference.

Phase-boundary triggers

Completing a planning document should re-check the docs for drift — automatically, but never blocking the build. When a pipeline step at a phase boundary is marked complete, the state manager fires a docs audit and prints a one-line summary.

The hook

StateManager.markCompleted() is async (src/state/state-manager.ts:175): it saves state, then awaits runPhaseAudit({ primaryRoot, step }) for every completed step — the phase-boundary gate lives inside runPhaseAudit, which returns a no-op result unless the step is one of the six boundaries (src/observability/engine/phase-subsets.ts): user-stories, tech-stack, coding-standards, design-system, implementation-plan, implementation-playbook.

The audit is fixed to H-cross-doc, scope=docs, profile=fast (src/observability/engine/phase-audit.ts). It is non-gating: markCompleted catches any audit error and returns a result object, so a state transition never fails because of the audit.

The stdout line:

# normal
[audit] 3 findings (1 blocking, verdict=blocked) — see docs/audits/audit-…-fast-lens-H-cross-doc-….md
# detached (fire-and-forget)
[audit] dispatched in background (step: tech-stack)
# timed out
[audit] timed out after 60500ms — partial findings may not be written

Config & timestamps

Step state (src/types/state.ts) now carries completed_at (set once, on first completion) and in_progress_started_at (set on first entry to in-progress). The state adapter's replayEvents prefers these per-step timestamps and only falls back to the file's mtime when they're absent — so the timeline reflects when each step actually transitioned, not when the file was last written.

MMR doc-conformance channel

The audit plugs into multi-model review as a built-in channel. scaffold observe audit --output-mode=mmr-findings emits findings in MMR's Finding shape (src/observability/renderers/mmr-findings.ts), always exiting 0 so the dispatcher captures stdout regardless of verdict. Only open findings are emitted (acknowledged and skipped are dropped).

Finding translation

The composite location — <source_doc>::<lens_id>::<short_id> — gives each finding a stable identity across audit runs, so MMR can track the same defect over time.

A Lens-B engine Finding like this:

{
  "id": "3a8c1f0211223344", "lens_id": "B-ac-coverage", "severity": "P0",
  "title": "AC has failing test", "description": "Test refresh.spec.ts is failing.",
  "source_doc": "docs/user-stories.md#user-auth-1",
  "fix_hint": { "prompt": "Re-enable the test", "target": "src/auth/test.spec.ts" }
}

translates to this MMR Finding:

{
  "location":    "docs/user-stories.md#user-auth-1::B-ac-coverage::3a8c1f02",
  "severity":    "P0",
  "description": "[doc-conformance/B-ac-coverage] AC has failing test — Test refresh.spec.ts is failing.",
  "suggestion":  "Re-enable the test",
  "category":    "doc-conformance"
}

Enabling it

The doc-conformance channel is disabled by default. Enable it per-project in .mmr.yaml or for a single run with --channels=doc-conformance; if it's globally enabled you can opt a project out via channels_disabled: ["doc-conformance"].

mmr review --channels=doc-conformance --diff <(git diff main...HEAD) --sync --format json

Under the hood the built-in channel runs scaffold observe audit --profile=full --scope=all --output-mode=mmr-findings and parses its stdout. See the MMR guide (../mmr/index.md) for the channel architecture this plugs into.

The --fix flow

scaffold observe audit --fix doesn't just report blocking findings — it dispatches an agent to fix each one, verifies the fix with a single-lens re-audit, and writes a post-fix report. The loop is abort-aware: an interrupt un-stages what the loop staged and re-applies the original stash (see the limitation in Abort safety).

The five phases

• 1 · audit — run the audit to get findings and the verdict (src/observability/engine/fix-flow.ts).
• 2 · buildFixPlan (src/observability/engine/fix-plan.ts) — keep findings that are open and at/above fix_threshold, ordered by severity ascending (P0 first) then lens_id.
• 3 · dispatchFixAgent (src/observability/engine/fix-agent-dispatcher.ts) — spawn the dispatcher command, write the per-finding prompt to stdin, stream stdout/stderr to the user.
• 4 · single-lens verify — re-run only the finding's lens; if the finding is gone, mark fixed; else retry, up to per_finding_max_attempts (default 3).
• 5 · post-fix report — a final scope=all audit, written to docs/audits/<id>-postfix.{md,json}.

Abort safety

Before dispatching, captureSnapshot (src/observability/engine/abort-snapshot.ts) records a git stash create SHA plus the set of pre-existing staged files. Newly staged paths are tracked per attempt. On SIGINT, restoreSnapshot un-stages the paths the loop recorded as newly staged and re-applies the original stash tree. Limitation: edits the dispatcher made but that the loop did not record as staged are not actively reverted and may need manual cleanup — recovery is scoped to the recorded staged paths plus the stash, not a guaranteed full reset.

Config — the fix: block

Exit code: 1 if any blocking finding survived the loop, else 0.

Harvest, recover & teardown

A worktree's ledger is local to that worktree. Harvest copies it into the primary repo's archive before the worktree is removed; recover sweeps up ledgers whose worktrees vanished without being harvested; teardown does the whole dance in one command.

Commands

• scaffold observe harvest --worktree=<path> — read the worktree's identity.json, then copy its activity.jsonl to .scaffold/activity-archive/active/<worktree_id>.jsonl in the primary repo (atomic temp-file + rename under a lock; keyed by worktree_id, idempotent full-file replace). Requires the primary root to be a real repo, not itself a worktree.
• scaffold observe harvest --recover — list live worktrees (git worktree list --porcelain), read each one's worktree_id, then scan active/ for entries whose worktree is no longer live and rotate each into activity-archive/<YYYY-MM>.jsonl (bucketed by the file's mtime), deleting the active entry. Returns the rotated ids.
• scripts/teardown-agent-worktree.sh <path> — derives the primary repo via git -C <path> rev-parse --git-common-dir (so it works from anywhere), harvests (fail-soft), runs git worktree remove, then deletes the workspace branch if it differs from the primary HEAD.

Renderers

One EngineOutput, several surfaces. The terminal renderer is for humans at a prompt; the markdown renderer writes a durable report plus a machine-readable JSON sidecar; the dashboard-fragment renderer emits an HTML panel that scripts/generate-dashboard.sh injects at named anchors.

Default audit/progress runs write both a markdown report and a JSON sidecar: audits to docs/audits/<id>.{md,json}, progress to docs/build-status/<id>.{md,json} (src/observability/renderers/sidecar.ts). --render and --output-mode bypass the files and print to stdout instead.

The same blocked audit renders three ways. As terminal ANSI:

build observability — audit
(profile: fast · scope: all)

  [P0] AC has failing test
       lens: B-ac-coverage
       fix:  Re-enable the test

verdict: blocked
(blocking 1 · ack 0 · total 1)

As markdown (docs/audits/<id>.md):

# Build Observability — Audit

**Verdict:** blocked

## Summary
1 findings · 1 blocking (at or above P2) · 0 acknowledged · 0 skipped lenses

### [P0] B-ac-coverage — AC has failing test
`3a8c1f02` · *source:* `…#user-auth-1` · *confidence:* high

As a dashboard fragment, an HTML <section id="build-audit" data-verdict="blocked"> panel injected into the dashboard.

Config reference

All tunables live in .scaffold/observability.yaml. The file is optional — when absent, DEFAULT_CONFIG applies wholesale (src/observability/engine/checks/observability-config.ts). A malformed or unreadable file silently falls back to defaults. Merging is a deep merge where objects merge key-by-key but arrays and scalars replace wholesale — so setting disabled_lenses or rule_overrides overwrites the default rather than extending it.

Example:

lenses:
  C-standards:
    enforce_via_linter: true
    rule_overrides:
      no-console: P1
  E-design:
    ad_hoc_token_threshold: 5
disabled_lenses: []
phase_audit:
  enabled: true
  timeout_s: 60
stall:
  task_stale: '6h'
fix:
  per_finding_max_attempts: 3
# shared with the knowledge-freshness subsystem
knowledge_freshness:
  link_check:
    skip:
      - platform.openai.com

Where it lives

The whole system lives under src/observability/, plus the state-manager hook and two shell scripts.

The full design lives in docs/superpowers/specs/2026-04-30-build-observability-design.md.

Shipping history

Build Observability shipped as eight plans (Foundation → Fix-flow), released together as v3.26.0 — Build Observability on 2026-05-07, with Lens I and refinements following after.

Known divergences

The audit's own rule — "code is ground truth where it disagrees with the spec or the docs" — surfaced these mismatches. They are deferred findings to resolve in the code/docs:

• 9 lenses, not 8. The design spec and CLAUDE.md enumerate A–H; I-knowledge-gaps shipped and is registered.
• pr_opened vs pr_open. CLAUDE.md abbreviates the event type as pr_open; the code only emits pr_opened.
• Three engine verdicts, not four. needs-user-decision is an MMR review verdict; the audit engine emits only pass / degraded-pass / blocked.
• Two dispatchers — don't conflate them. CLAUDE.md's "not project-config-overridable" note is about the Lens H LLM dispatcher (llm-dispatcher.ts, hard-coded claude -p by design). It does not apply to the separate --fix agent dispatcher, whose fix.dispatcher_command is project-configurable. The divergence is that the two are easy to conflate.
• pr_review_stale reserved. The sixth stall signal has a config default but is not yet emitted (deferred pending a per-PR correlation_id on the mmr adapter).
• Inert config keys. lenses.G-decisions.keywords_file and lenses.H-cross-doc.skip_phase_subsets are typed but unread; the bundled decision-keywords.txt is not loaded.
• No F-scope.wave_budget key. CLAUDE.md implies a wave budget in observability.yaml; in fact no wave/phase budget is read anywhere — Lens F reads only untouched_story_grace_hours.
• ui_glob default narrower than spec. Spec default covers {tsx,jsx,vue,svelte} + styles; shipped default is just src/components/**/*.tsx.