# Diagnostics: memory, GC, event-loop, heap snapshots, CPU/heap profiles

Operator runbook for the in-process memory + perf observability surface shipped
under PRD `minions-opg-2026-06-12.json` (plan items `P-a1b2c3d4` →
`P-e5f6a7b8`). Covers what the API exposes, how to read the engine log
emissions, how to capture and analyse heap snapshots, how to drive a
`--cpu-prof` / `--heap-prof` session, and how to retune the
`test/perf/soak.test.js` regression gate.

If you just want to *see* the live numbers, open the dashboard's `/engine`
page — the Memory panel renders the same data this doc describes.

## 1. `GET /api/diagnostics/memory` and `GET /api/diagnostics/memory/history`

Both routes are registered in the dashboard route registry
([`dashboard.js:13079-13080`](../dashboard.js)) and exposed at
`GET /api/routes`. They are read-only, do no shelling-out, and complete in
well under 10 ms warm.

### `GET /api/diagnostics/memory`

Returns the latest in-process **dashboard** sample plus the latest **engine**
sample (read fresh from `engine/diagnostics-memory.json` via
`safeJsonObj`) plus a staleness flag. Engine staleness is `true` when the
sidecar is missing/unparseable OR when its `capturedAt` is more than
5 minutes old (`DIAGNOSTICS_MEMORY_STALE_MS`,
[`dashboard.js:117`](../dashboard.js)).

```bash
curl -s http://localhost:7331/api/diagnostics/memory | jq
```

```json
{
  "dashboard": {
    "rss": 184250368,
    "heapUsed": 92847232,
    "heapTotal": 134217728,
    "external": 4214528,
    "arrayBuffers": 532480,
    "heapSizeLimit": 4341563392,
    "eventLoopLagP50": 1.42,
    "eventLoopLagP99": 18.36,
    "eventLoopLagMax": 64.20,
    "lastGcPauseMs": 4.83,
    "lastGcKind": "minor",
    "gcPausesTotalMs": 312.47,
    "gcCount": 87,
    "uptime": 612.4,
    "pid": 18324,
    "label": "dashboard",
    "capturedAt": 1749758480123
  },
  "engine": {
    "rss": 152784896,
    "heapUsed": 78643200,
    "heapTotal": 117440512,
    "external": 2228224,
    "arrayBuffers": 131072,
    "heapSizeLimit": 4341563392,
    "eventLoopLagP50": 0.94,
    "eventLoopLagP99": 12.18,
    "eventLoopLagMax": 41.62,
    "lastGcPauseMs": 2.11,
    "lastGcKind": "minor",
    "gcPausesTotalMs": 198.34,
    "gcCount": 64,
    "uptime": 603.7,
    "pid": 18301,
    "label": "engine",
    "capturedAt": 1749758460112
  },
  "engineStale": false
}
```

Field reference (every value originates from
[`engine/diagnostics-memory.js`](../engine/diagnostics-memory.js)
`sampleSelf()`):

| Field | Source | Units |
|---|---|---|
| `rss` | `process.memoryUsage().rss` | bytes |
| `heapUsed` | `process.memoryUsage().heapUsed` | bytes |
| `heapTotal` | `process.memoryUsage().heapTotal` | bytes |
| `external` | `process.memoryUsage().external` | bytes |
| `arrayBuffers` | `process.memoryUsage().arrayBuffers` | bytes |
| `heapSizeLimit` | `v8.getHeapStatistics().heap_size_limit` | bytes |
| `eventLoopLagP50` / `P99` / `Max` | `perf_hooks.monitorEventLoopDelay({ resolution: 20 })` | ms (fractional) |
| `lastGcPauseMs` / `lastGcKind` | `PerformanceObserver({ entryTypes: ['gc'] })` last entry | ms / `'minor'\|'major'\|'incremental'\|'weakcb'` |
| `gcPausesTotalMs` / `gcCount` | accumulated from same observer | ms / int |
| `uptime` | `process.uptime()` | seconds |
| `pid` | `process.pid` | int |
| `label` | constant per process | `'engine'` or `'dashboard'` |
| `capturedAt` | `Date.now()` at sample time | epoch ms |

When the engine has not yet written the sidecar (cold dashboard, engine
down, or `memoryBaselineEveryTicks <= 0`), `engine` is `null` and
`engineStale` is `true`.

### `GET /api/diagnostics/memory/history`

In-memory ring buffer of samples. Two backing rings:

- `process=dashboard` — the dashboard's own collector (populated by
  `diagnosticsMemory.startPeriodicSampling` on dashboard boot,
  `DIAGNOSTICS_MEMORY_SAMPLE_INTERVAL_MS = 60 000` ms,
  [`dashboard.js:118`](../dashboard.js)). Capacity is
  `RING_BUFFER_CAP = 1440` samples (≈ 24 h at 60 s cadence,
  [`engine/diagnostics-memory.js:28`](../engine/diagnostics-memory.js)).
- `process=engine` — the dashboard's **polled accumulation** of the
  engine's sidecar. Engine.js only persists the latest sample to
  `engine/diagnostics-memory.json`, so engine-side history is rebuilt by
  the dashboard poller (dedup by `capturedAt`). Capacity is
  `DIAGNOSTICS_MEMORY_ENGINE_RING_CAP = 1440` samples
  ([`dashboard.js:119`](../dashboard.js)).

```bash
# All samples
curl -s "http://localhost:7331/api/diagnostics/memory/history?process=engine" | jq '.count'

# Newest 60 samples (≈ last hour at 60s cadence)
curl -s "http://localhost:7331/api/diagnostics/memory/history?process=dashboard&limit=60" | jq
```

```json
{
  "process": "engine",
  "count": 60,
  "samples": [
    { "rss": 152784896, "heapUsed": 78643200, "...": "...", "capturedAt": 1749754860111 },
    { "rss": 153038848, "heapUsed": 79233024, "...": "...", "capturedAt": 1749754920119 }
  ]
}
```

`samples` is ordered oldest-first. `limit` clamps to the **newest** N.
Both rings are in-process — restarting the dashboard zeroes them. There
is no on-disk replay for history (by design — the soak gate
`test/perf/soak.test.js` covers the regression case the disk history
would otherwise serve).

A 400 is returned when `process` is missing or not in
`{engine, dashboard}`.

## 2. `MEMORY_BASELINE engine ...` log lines

Every `ENGINE_DEFAULTS.memoryBaselineEveryTicks` ticks (default `6`,
defined at
[`engine/shared.js:2540`](../engine/shared.js)), the engine emits one
structured log line and writes the latest sample to the sidecar. At the
default `tickInterval: 10` the cadence is ≈ 60 s.

```
[2026-06-12T20:01:00.115Z] MEMORY_BASELINE engine rss=152784896 heapUsed=78643200 eventLoopLagP99=12.18ms gcPauses=64/198.34ms tickN=360
```

Anatomy ([`engine.js:9539-9542`](../engine.js)):

| Token | Meaning |
|---|---|
| `rss` | Resident set size in bytes — total memory the OS has mapped for the process. |
| `heapUsed` | V8 heap actually in use in bytes (after the most recent allocation; not GC-stabilised). |
| `eventLoopLagP99` | 99th-percentile event-loop lag in ms across the histogram's lifetime. **The most useful single number** — when it stays above ~50 ms you have a synchronous hot path. |
| `gcPauses` | `<count>/<totalMs>` since process start. Rapidly-growing `count` with low `totalMs` is healthy (lots of cheap minor GCs); slow-growing `count` with large `totalMs` per increment is concerning (long major GCs). |
| `tickN` | Engine tick number when the sample was taken. Useful for correlating against the same tick's other log lines. |

### Cadence + opt-out

`ENGINE_DEFAULTS.memoryBaselineEveryTicks` is the only knob:

- Set to `6` (default) → emit every 6 ticks ≈ 60 s at 10 s tick interval.
- Set to `1` → emit every tick (≈ 10 s). Useful during a leak hunt; turns
  the log into ~6× more noise.
- Set to `0` or any non-positive integer → **both** the log emission and
  the sidecar write are disabled cleanly. The dashboard's
  `/api/diagnostics/memory` will surface `engineStale: true` and
  `engine: null` after the sidecar ages past
  `DIAGNOSTICS_MEMORY_STALE_MS` (5 minutes).

Override via `config.json`:

```json
{
  "engine": {
    "memoryBaselineEveryTicks": 1
  }
}
```

Restart the engine after editing — `ENGINE_DEFAULTS` is resolved once at
process start.

The sidecar file (`engine/diagnostics-memory.json`) is a passive
single-object cache, gitignored, and intentionally **exempt from the
SQL-first state rule** — same exemption as `engine/dashboard-port.json`.
Do not promote it to SQLite.

## 3. `POST /api/diagnostics/heap-snapshot`

Captures `v8.writeHeapSnapshot()` for **both** the dashboard and engine
processes. This is the heavyweight diagnostic — read this section in
full before invoking.

### ⚠ Stall warning

`v8.writeHeapSnapshot` is synchronous and stalls the calling process for
several seconds (≈ 2–6 s typical, longer on a hot engine with 100 MB+
heap). It also writes a 50–200 MB file per process. During the capture:

- The dashboard does not serve any other HTTP requests.
- The engine does not service ticks (no new dispatches, no PR polling,
  no timeout checks). Existing child agents continue running, but their
  output is buffered until the engine resumes.

Do not invoke during a live incident unless you've already accepted the
≈ 10–15 s engine pause as worth the diagnostic.

### Invocation

The endpoint is guarded by a literal confirm token
(`HEAP_SNAPSHOT_CONFIRM_TOKEN`,
[`dashboard.js:185`](../dashboard.js)) so accidental POSTs return 400
with the warning text:

```bash
curl -X POST -s \
  "http://localhost:7331/api/diagnostics/heap-snapshot?confirm=YES_I_UNDERSTAND_THIS_STALLS_THE_ENGINE" \
  | jq
```

```json
{
  "dashboardSnapshot": "D:\\squad-opg\\engine\\diagnostics\\heap-dashboard-2026-06-12T20-15-30-123Z.heapsnapshot",
  "engineSnapshot": "D:\\squad-opg\\engine\\diagnostics\\heap-engine-2026-06-12T20-15-30-123Z.heapsnapshot",
  "engineTimedOut": false,
  "timeoutMs": 30000
}
```

- Without the `?confirm=` token: `400` with hint.
- Within `HEAP_SNAPSHOT_RATE_LIMIT_MS` (60 000 ms) of the previous
  successful capture: `429` with `Retry-After` and `retryAfterSeconds` in
  the body. The window stamps on completion (not request start), so a
  back-to-back POST won't double-stall.
- If the engine doesn't write its snapshot within 30 s
  (`HEAP_SNAPSHOT_ENGINE_TIMEOUT_MS`): `engineSnapshot: null` and
  `engineTimedOut: true`. The dashboard snapshot still landed. Common
  causes: engine is paused/stopping, or a tick is in a long phase
  (`pollPrStatus` against a slow remote).

### Where files land

`engine/diagnostics/heap-<process>-<iso>.heapsnapshot`, where the
timestamp comes from the dashboard handler (engine reuses the same ISO
from the sentinel at
`engine/diagnostics/heap-snapshot-request.json` so the pair shares a
predictable filename suffix). ISO colons + the millis dot are replaced
with `-` for NTFS compatibility.

### Loading in Chrome DevTools

1. Open Chrome (or Edge) → DevTools → **Memory** tab.
2. Pick the **Heap snapshot** profiling type.
3. Right-click in the snapshot list → **Load…** → select the
   `.heapsnapshot` file.
4. Switch the dropdown from *Summary* to *Comparison* and load a second
   snapshot to diff retained sizes across captures.

For a leak hunt, capture **three** snapshots ≥ 2 h apart while the
process is under representative load, then diff snapshot 1 → 2 → 3.
Look for object classes whose `# Delta` and `Size Delta` are both
positive and monotonically growing. The Phase 2 explore item
(`P-f6a7b8c9`) documents this workflow end-to-end and lists the
canonical retention paths to inspect inside the Minions engine
(`activeProcesses` Map, CC session map, `_buildStatusCache`, etc.).

### Retention policy

`HEAP_SNAPSHOT_RETAIN_COUNT = 5` per process
([`dashboard.js:189`](../dashboard.js)). On every successful capture,
`_heapSnapshotPrune` keeps the 5 newest `heap-dashboard-*.heapsnapshot`
and the 5 newest `heap-engine-*.heapsnapshot` by mtime, deleting the
rest. Best-effort: a prune failure is swallowed and retried on the next
capture, so transient Windows file locks don't break the endpoint.

If you need to preserve a snapshot, **copy it out** of
`engine/diagnostics/` before invoking the endpoint a sixth time.

## 4. CPU + heap allocation profile via `NODE_OPTIONS`

The heap-snapshot endpoint gives you retained sizes at a single moment.
`--cpu-prof` and `--heap-prof` give you the orthogonal signal: where CPU
time and allocations are *spent* during a window of execution.

Both are Node built-ins; no extra deps. Restart the engine with both
flags pointed at `engine/diagnostics`:

```powershell
# PowerShell
$env:NODE_OPTIONS = '--cpu-prof --cpu-prof-dir=engine/diagnostics --heap-prof --heap-prof-dir=engine/diagnostics'
minions restart
```

```bash
# bash
NODE_OPTIONS='--cpu-prof --cpu-prof-dir=engine/diagnostics --heap-prof --heap-prof-dir=engine/diagnostics' minions restart
```

Drive 5–10 minutes of representative load:

- Manually dispatch ≥ 5 short fix WIs (mocked agents are fine).
- Poll `/api/status` every 4 s in a tight loop:
  `while ($true) { curl -s http://localhost:7331/api/status > $null; Start-Sleep 4 }`.
- Trigger ≥ 3 PR poller refreshes (or wait through ≥ 3 polling cycles).

Then **stop the engine cleanly** (`minions stop`) — the `.cpuprofile`
and `.heapprofile` files are only flushed on graceful exit.

The two file types land in `engine/diagnostics/`:

- `CPU.<date>.<pid>.<tid>.<seq>.cpuprofile` — V8 CPU sampler output.
- `Heap.<date>.<pid>.<tid>.<seq>.heapprofile` — V8 sampling heap
  profiler output (allocations, *not* live retention; complements the
  heap-snapshot endpoint).

### Loading in Chrome DevTools

1. Open DevTools → **Performance** tab → ⋯ menu → **Load profile…** for
   `.cpuprofile`. Look at the flame chart for any synchronous frame
   > 50 ms (these violate the `/api/status` isolation invariant — the
   tick should never block the dashboard event loop that long).
2. Open DevTools → **Memory** tab → pick **Allocation sampling** →
   right-click → **Load…** for `.heapprofile`. Sort by *Self size* to
   identify the top allocation sites by byte count.

The Phase 3 explore item (`P-a7b8c9d0`) calls out the specific hot
paths to inspect (route-registry rebuild in `dashboard.js`,
`dashboard-build.js` HTML assembly, per-tick allocation hotspots in
`engine.js#tick`, regex / JSON.parse frames).

Remember to **clear** `NODE_OPTIONS` before the next normal start —
`--cpu-prof` writes a new file on every process exit and the directory
will fill up.

## 5. `test/perf/soak.test.js` threshold tuning

The soak test drives 30 simulated engine ticks against an isolated
`MINIONS_TEST_DIR` and asserts that `heapUsed` neither grows unboundedly
across the soak nor exceeds an absolute ceiling. Source:
[`test/perf/soak.test.js`](../test/perf/soak.test.js).

```bash
node test/perf/soak.test.js                    # RSS fallback, looser bound
node --expose-gc test/perf/soak.test.js        # heap measurement, tight bound
```

### The two thresholds

| Constant | Default | What it gates |
|---|---|---|
| `HEAP_DELTA_MAX` | 8 MB | `heapUsed(30t) - heapUsed(5t)` — the delta between the post-warmup baseline (5 ticks) and the end of the soak (30 ticks). Catches compounding leaks. |
| `HEAP_ABSOLUTE_MAX` | 200 MB | `heapUsed(30t)` after the final forced GC. Catches "engine's resting footprint regressed", not just "per-tick leak". |
| `HEAP_DELTA_MAX_NO_GC` | 32 MB (= 4× `HEAP_DELTA_MAX`) | Fallback delta bound when `--expose-gc` is absent. Measured against RSS (includes uncollected garbage), so 4× slack. |

The absolute-ceiling test is informational (warning only) when GC is not
forced — without `--expose-gc`, `heapUsed` can include arbitrary
uncollected garbage that doesn't reflect a real leak.

### When to retune

**Lower the bounds** (tighten the gate) when you've shipped a real
reduction in per-tick allocations and want to lock it in:

1. Run the soak ≥ 5 times locally with `--expose-gc`.
2. Take the **max** observed delta across the runs.
3. Set the new `HEAP_DELTA_MAX` to `ceil(maxObservedDelta * 1.5)` MB,
   rounded to the nearest MB.
4. Same recipe for `HEAP_ABSOLUTE_MAX` against the max post-soak
   `heapUsed`.
5. Land the constant change with the PR that earned the headroom — link
   the 5 sample runs in the PR description.

**Raise the bounds** (loosen the gate) only with evidence that the
existing limit is too tight for the *legitimate* per-tick cost (not to
mask a regression):

1. The CI gate must be flaking *across multiple unrelated PRs* — a
   single PR failing the gate is a signal to fix the PR, not the test.
2. Capture ≥ 10 runs on the CI host (or a comparable Windows
   self-hosted runner) with `--expose-gc` enabled. Record max delta and
   max post-soak heapUsed.
3. If the variance is the issue (max - min > 50% of the current bound),
   **add samples or extend warmup** before bumping. The header comment
   on the test file is explicit about this: "If they flake on a noisy
   CI host, prefer adding samples / longer warmup over raising the
   bound — the gate's value is detecting compounding growth."
4. If max + 2σ truly exceeds the current bound, bump to
   `ceil((max + 2σ) * 1.25)` MB. Link the 10 runs in the PR
   description. Add a comment line above the constant referencing the
   PR that justified the bump.

### Acceptable variance

On a quiet developer workstation with `--expose-gc`:

- `HEAP_DELTA_MAX` should clear with 1–3 MB observed delta. > 5 MB
  means there is something to investigate, even if it still passes.
- `HEAP_ABSOLUTE_MAX` should clear with 60–110 MB observed post-soak
  heapUsed. Approaching 150 MB means the resting footprint has grown
  recently.

On a noisy CI host without `--expose-gc`, RSS variance can be 4–8 MB
between runs — this is exactly why the no-GC fallback uses 4× slack.

## 6. Phase 2 and Phase 3 findings notes

The diagnostics surface this doc describes exists to make two follow-up
explores (and any future ones) *cheap to run*. Findings live in
`knowledge/architecture/` once the KB sweep promotes them; until then,
they sit in `notes/inbox/` with predictable slugs:

| Phase | Plan item | Inbox slug (pre-sweep) | Promoted path (post-sweep) |
|---|---|---|---|
| 2 — heap dominators + leak candidates | `P-f6a7b8c9` | `notes/inbox/ripley-memory-audit-phase-2-heap-dominators*.md` | `knowledge/architecture/2026-06-12-ripley-memory-audit-phase-2-heap-dominators.md` |
| 3 — CPU + allocation hot frames | `P-a7b8c9d0` | `notes/inbox/ripley-memory-audit-phase-3-cpu-and-allocations*.md` | `knowledge/architecture/2026-06-12-ripley-memory-audit-phase-3-cpu-and-allocations.md` |

Use [`engine/kb-sweep.js`](../engine/kb-sweep.js) (see
[`docs/kb-sweep.md`](kb-sweep.md)) or the dashboard's "Sweep" button to
trigger promotion. The cheap-win PR items downstream of the explores
(`P-b8c9d0e1`, `P-c9d0e1f2`, `P-d0e1f2a3`) consume the prioritized lists
from these notes — read them before opening one of those PRs to avoid
re-deriving the same findings.

## Related modules and tests

- Module: [`engine/diagnostics-memory.js`](../engine/diagnostics-memory.js) — sampler + ring buffer + GC observer.
- Engine wiring: [`engine.js`](../engine.js) `emitMemoryBaseline` + `processHeapSnapshotRequest`.
- Dashboard wiring: [`dashboard.js`](../dashboard.js) `handleDiagnosticsMemory`, `handleDiagnosticsMemoryHistory`, `handleDiagnosticsHeapSnapshot`.
- Frontend: [`dashboard/pages/engine-memory-panel.html`](../dashboard/pages/engine-memory-panel.html) + [`dashboard/js/memory-panel.js`](../dashboard/js/memory-panel.js).
- Tests:
  - [`test/unit/diagnostics-memory-api.test.js`](../test/unit/diagnostics-memory-api.test.js)
  - [`test/unit/heap-snapshot-api.test.js`](../test/unit/heap-snapshot-api.test.js)
  - [`test/unit/memory-panel.test.js`](../test/unit/memory-panel.test.js)
  - [`test/perf/memory-baseline-tick-overhead.test.js`](../test/perf/memory-baseline-tick-overhead.test.js)
  - [`test/perf/soak.test.js`](../test/perf/soak.test.js)
