# Upgrade Guide — vNEXT

<!-- assembled-by: assemble-next-md -->
<!-- bump: patch -->

## What Changed

A revoked mesh machine (the stale Mac Mini) came back **active** after an update
(2026-06-07, topic 21816) and resumed sending mesh/live-tail chatter. Root cause:
`MachineIdentityManager.registerMachine` rebuilt the entry with `status: 'active'`, so a
re-register (re-join / re-pair / post-update self-registration) clobbered a `revoked`
status back to active. The merge path (`mergeRegistry`) already kept revocation sticky;
this was the other door. Fix: `registerMachine` now refuses to resurrect a revoked entry
— if the existing entry is `status: 'revoked'` or has `revokedAt`, it logs loudly and
leaves the entry untouched. A revoked machine stays revoked across updates; the only path
back to active is an explicit un-revoke.

The exactly-once message recovery re-ran an already-answered Telegram message every
~10 minutes — tagged "from Unknown" — re-spawning a session each cycle and piling load
onto an already-overloaded machine (the 2026-06-07 "server temporarily down on every
message" incident, topic 21816). Three fixes, all "don't replay a handled message":

1. **Reply-evidence guard** — before re-running a message stuck in `processing`,
   stuck-recovery now checks the durable ledger for a reply committed on that topic
   at/after the message arrived. If found, the entry is committed (it was effectively
   handled — its own reply just failed to record during the server flap, or it was a
   duplicate) instead of being re-injected. The no-loss path (re-running a genuinely
   unanswered turn) is unchanged.
2. **Sender preserved through replay** — the inbound sender is captured at ingress and
   replayed, so a legitimate re-run no longer shows "from Unknown" (Know Your Principal).
3. **Lifeline queue dedup** — `MessageQueue.enqueue` is idempotent on message id and
   refuses to re-queue an id already delivered/dropped this run, killing the "stale
   already-delivered copies kept getting retried" half.

`SessionWatchdog` polled every 30s over every running session, running several
`ps`/`pgrep` probes per session via synchronous `spawnSync` — each blocking the single
Node event loop for its full duration. Under load (dozens of sessions on a busy box) the
cumulative stall made the server miss its own `/health` window, so the supervisor
declared it down and restarted it in a loop (2026-06-07 "server temporarily down on
every message" incident, topic 21816). The watchdog's process scans are now async
(`execFile`) and the poll yields the event loop between sessions, so health checks stay
responsive while scans run. The watchdog's escalation/kill DECISIONS are unchanged —
only the I/O mechanism of its read-only probes changed.

## What to Tell Your User

If they revoked a machine and it kept coming back after updates: it won't anymore. A
revoked machine stays revoked until they explicitly un-revoke it. Nothing to do.

If they saw the same message echoed back repeatedly, or "server temporarily down" on
nearly every message during an overload: that replay loop is fixed — an already-answered
message is no longer re-run, and the queue won't re-deliver something it already sent.
Nothing for them to do; their messages were never lost (they queued and delivered).

If they saw "server temporarily down" on nearly every message during a busy period,
with the server restarting repeatedly: a big contributor was the watchdog freezing the
event loop while scanning processes. It no longer does. Nothing for them to do.

## Summary of New Capabilities

- `MachineIdentityManager.registerMachine` is now revocation-sticky: re-registering a
  machine whose existing entry is revoked is refused (logged, entry left intact) instead
  of silently flipping it back to active. `ensureSelfRegistered` already no-ops on any
  existing entry; explicit un-revoke remains the deliberate restore path.

- `MessageProcessingLedger.hasReplyCommittedForTopicSince(topic, sinceISO)` — durable
  reply-evidence query used to recognize an already-answered topic.
- `MessageProcessingLedger` stores the inbound sender envelope (additive, nullable
  column; idempotent in-place schema upgrade — no migrator step).
- `recoverStuckMessages` reply-evidence guard + `alreadyHandled` result counter;
  re-injection preserves the real sender.
- `MessageQueue.enqueue` is id-idempotent (returns boolean); `MessageQueue.markDelivered`
  records delivered/dropped ids so a redelivery can't re-queue them.

- `SessionWatchdog` process scans (`getFrameworkPid`/`getClaudePid`/`getChildProcesses`/
  `hasActivePipelineSibling`/`checkCompactionIdle`) are async (`execFile`) instead of
  blocking `spawnSync`.
- `poll()` yields the event loop (`setImmediate`) between sessions so a scan over many
  sessions can't monopolize the loop and starve `/health`.

## Evidence

`machine-identity.test.ts` (new cases): a revoked machine re-registered stays revoked
(status/role/revokedAt/revokedBy preserved); a brand-new machine still registers active.
75 machine-identity unit tests green; `tsc --noEmit` clean; no-silent-fallbacks budget
unaffected (the guard adds a `return`, no catch). causalAutopsy: latent — registerMachine
always rebuilt with `status: 'active'`; only surfaced once a revoked machine re-registered
after an update (the Mac Mini case). Symmetric with the already-sticky merge path.

`tests/unit/stuck-message-recovery.test.ts`, `MessageProcessingLedger.test.ts`,
`lifeline/MessageQueue-durability.test.ts`, `lifeline/version-skew-recovery.test.ts`:
reply-evidence guard commits-not-reruns an answered topic; a genuinely-unanswered entry
still re-runs; sender round-trips through a recovered replay; the queue refuses to
re-enqueue an already-delivered id; wiring guards assert routes.ts captures the sender
and reinjectStuck forwards it. causalAutopsy: latent (the stuck==unanswered assumption
surfaced under sustained CPU starvation that made replies fail to commit).

`tests/unit/SessionWatchdog-nonblocking.test.ts`: scanning helpers return Promises; a
0ms timer fires during an in-flight probe (loop stays live); source guards assert no
`spawnSync(` call, `execFileAsync` present, the three helpers async, and `poll()` yields
via `setImmediate`. Existing compaction/pipeline/mcp/rate-limit watchdog suites updated
to `await` and pass. 72 watchdog unit tests green; `tsc --noEmit` clean. causalAutopsy:
latent — synchronous process scans were always event-loop-blocking, but only surfaced as
a health-miss restart loop once session count × machine load grew large enough.

## Scope (honest)

Contained Tier-1 guard in one method (`src/core/MachineIdentity.ts`). Multi-machine only
(a single-machine agent never hits this). Normal joins / active re-registers are
unaffected; only resurrection of a revoked entry is blocked. No new API/route/config/
migration. Justin already re-quarantined the Mini live during the incident; this is the
durable code fix so it can't recur.

Contained Tier-1 fix to the existing exactly-once ingress + lifeline replay subsystems.
No new HTTP route, config default, or migration step. The reply-evidence false-negative
case (two distinct rapid questions, the second crashes before the first's reply commits)
is bounded — that message was already routed once, so it isn't lost, only a redundant
re-run is skipped. 603 messaging/lifeline unit tests green; `tsc --noEmit` clean.

Contained Tier-1 refactor of one monitor (`src/monitoring/SessionWatchdog.ts`) + its
tests. Behavior-equivalent (same commands, parsing, thresholds, LLM gate); only the
sync→async I/O mechanism changed. The watchdog stays opt-in. The same blocking-scan
pattern exists in OrphanProcessReaper / mcpProcessReaper (lower cadence — memory-pressure
and 30-min triggers); those are a candidate follow-up, not in this PR.
