# Side-Effects Review — Lifeline self-heal hardening

**Version / slug:** `lifeline-self-heal-hardening`
**Date:** 2026-04-29
**Author:** echo
**Second-pass reviewer:** independent-subagent (Phase 5)

## Summary of the change

Closes the three stacked failures that produced Inspec's silent crash-loop on 2026-04-29:

1. **Path-aware better-sqlite3 preflight scan.** `ServerSupervisor.preflightSelfHeal()` now scans `shadow-install/node_modules/**/better-sqlite3/build/Release/better_sqlite3.node` (bounded depth 5, bounded count 5) instead of only checking the hoisted top-level path. The Inspec post-mortem found the actually-loaded binary at the **nested** path `shadow-install/node_modules/instar/node_modules/better-sqlite3/...` — preflight had been silently skipping it.

2. **Bind-failure escalation.** New `consecutiveBindFailures` counter — incremented when a spawn produces a server that crashes before reaching its first healthy `/health` tick. After 2+ in a row, the next preflight forces a `npm rebuild better-sqlite3 --force` for every discovered nested copy regardless of whether the simple require-load probe reports a problem. Reset on any healthy tick.

3. **Robust launchd-supervision detection.** New helper `detectLaunchdSupervised()` replaces the previous `process.ppid === 1` check. Multi-signal: explicit `INSTAR_SUPERVISED=1` env var, `process.ppid === 1` (system-domain init), parent process command name = `launchd` on darwin (catches **user-domain** launchd — `gui/<uid>/...` — which is how every macOS user-installed agent runs), or parent name = `systemd` / `init` on Linux. Cached after first call.

4. **Plist template gets `INSTAR_SUPERVISED=1`.** New plists generated by `installMacOSLaunchAgent` include the env var as belt-and-suspenders. Existing installs are still covered by Change 3 (lifeline-side detection).

**Files touched:**

- `src/lifeline/detectLaunchdSupervised.ts` (new) — detection helper.
- `src/lifeline/TelegramLifeline.ts` — replaces inline `isSupervised` ternary with helper call.
- `src/lifeline/ServerSupervisor.ts` — adds `findBetterSqlite3Copies()` (exported), preflight scan loop, `consecutiveBindFailures` field + tracking + reset.
- `src/commands/setup.ts` — adds `INSTAR_SUPERVISED=1` to the plist's EnvironmentVariables.
- `tests/unit/detect-launchd-supervised.test.ts` (new) — 10 tests covering each detection signal + cache + NODE_ENV=test override.
- `tests/unit/find-better-sqlite3-copies.test.ts` (new) — 8 tests covering hoisted, nested, both-shapes, missing-binary, recursive-skip, and bounded-cap.
- `docs/specs/lifeline-self-heal-hardening.md` — design spec.

## Decision-point inventory

- `RestartOrchestrator.requestRestart` → unsupervised-skip branch — **modify** (input `isSupervised` is now computed by a more accurate detector; logic inside the orchestrator unchanged).
- `ServerSupervisor.preflightSelfHeal` (better-sqlite3 branch) — **modify** (single hard-coded path → recursive scan + force-rebuild escalation).
- `ServerSupervisor.handleUnhealthy` — **modify** (adds bind-failure tracking; existing behavior preserved).
- `installMacOSLaunchAgent` plist template — **modify** (one new env-var entry).
- `findBetterSqlite3Copies()` — **add** (pure scanner; no decision authority).
- `detectLaunchdSupervised()` — **add** (signal producer, no authority).

---

## 1. Over-block

**What legitimate inputs does this change reject that it shouldn't?**

No block/allow surface — these are recovery-action and signal-producer changes. The only "block-like" decision is the orchestrator's exit gate (unsupervised-skip), which now opens for a *more* accurate set of inputs (the user-launchd case). That can't over-block by design — it's strictly more permissive than the old check.

The bind-failure escalation triggers a `npm rebuild --force` on its own preflight cycle, which is a recovery action, not a block. The cost of a false-positive escalation is one extra rebuild (~30s on a fresh shadow-install). Acceptable.

## 2. Under-block

**What failure modes does this still miss?**

1. A binary at a depth deeper than 5 inside `shadow-install/node_modules`. Bounded depth means a pathological dep-of-dep-of-dep tree could hide a copy. In practice npm dedup keeps trees flat, and `instar`'s direct-dep `better-sqlite3` will hoist or appear under `instar/node_modules/`. Verified by inspection of Inspec's actual install layout. If we ever ship a transitive dep that itself depends on better-sqlite3 N>5 levels deep, the cap will hide it — at which point we add depth.

2. A binary where the require-load probe passes but the actual server load fails. Possible if the lifeline's Node and the server's Node disagree on the binary's compatibility (different flags, different sandboxing). Mitigated by the `checkNode = serverNode || process.execPath` fallback already present, plus the bind-failure escalation that forces a rebuild even when the probe reports OK.

3. The `npm rebuild --force` itself failing (no network, no compiler). Logs the error and continues; the next supervisor cycle will retry. Defense-in-depth: PR #91's source-build fallback covers the source-build path, and PR #89's per-tuple short-circuit prevents redownload loops on broken prebuilds.

4. A user-domain launchd whose `comm` field is **not** literally `launchd` (e.g., a user wrote a wrapper script named `launchd-wrapper`). Possible but exotic; would cost the user one explicit `INSTAR_SUPERVISED=1` env var as workaround. Documented in the helper's JSDoc.

## 3. Level-of-abstraction fit

**Is this at the right layer?**

Yes. Each of the four changes is at its rightful layer:

- `findBetterSqlite3Copies` is a pure file-system scanner — exposed as a free exported function (testable in isolation), called from preflight. Doesn't belong inside the supervisor class because it has no per-instance state.
- `detectLaunchdSupervised` is a pure structural-signal detector — lives in its own module, used by the lifeline. Doesn't belong inside `RestartOrchestrator` because the orchestrator already takes `isSupervised: boolean` as an input; the orchestrator's job is to *apply* the signal, not produce it.
- The bind-failure counter is per-supervisor-instance state and lives where the supervisor's other failure counters live.
- The plist env var lives in the plist generator — the only place a plist is composed.

## 4. Signal vs authority compliance

**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)

**Does this change hold blocking authority with brittle logic?**

- [x] No — this change has no block/allow surface (all changes are signal-producers feeding existing authorities, plus pure recovery actions).

`detectLaunchdSupervised` produces a structural signal (am I run by launchd?) consumed by `RestartOrchestrator` (the existing authority for self-restart decisions). The signal is structural (parent process identity), not judgmental.

`findBetterSqlite3Copies` is a pure scanner. No authority.

The `consecutiveBindFailures >= 2` threshold is a fixed deterministic mechanic, not a judgment call. Per the principle, "Idempotency keys and dedup at the transport layer" — counters with fixed thresholds qualify as transport-layer mechanics, not judgment authorities. The downstream action (force-rebuild) is a recovery primitive, not a block/allow decision on user input.

## 5. Interactions

**Shadowing:** Preflight's better-sqlite3 branch ran before the spawn and continues to run before the spawn. Now scans more files; doesn't interact with downstream checks. Does NOT shadow the in-process detector inside `AgentServer` — the in-process detector still runs at server startup, with its own logic; preflight is a belt-and-suspenders earlier check.

**Double-fire:** `findBetterSqlite3Copies` is invoked once per spawn cycle. Multiple bind failures in a row each invoke it once. This is intentional — the discovery may pick up new files between spawns (e.g., a mid-cycle reinstall). The rebuild is gated on `needsRebuild = force || (status !== 0 && stderr.includes('NODE_MODULE_VERSION'))`, so an already-good binary doesn't get redundantly rebuilt unless we're escalating.

**Races:** `consecutiveBindFailures` is mutated only on `handleUnhealthy` (increment) and on healthy-tick (reset). Both run inside the supervisor's `setInterval` callback, which is single-threaded. No race.

**Feedback loops:** Bind-failure escalation triggers a rebuild that may take 30–120s, during which no further spawns happen. After rebuild, a new spawn occurs. If the rebuild fails, the counter keeps incrementing each cycle — but the circuit breaker still trips at 20 total failures over its 1-hour window, providing the existing terminal limit. So we can't loop forever.

**Interactions with the recently-shipped degradation→tone-gate path (PR #105):** none direct. If the supervisor circuit-breaks, `serverDown` continues to fire as before. The new tone-gate routing doesn't run in the lifeline process (it's an in-server feature), so this change neither helps nor hurts that path. Coherent.

## 6. External surfaces

- **Other agents on same machine:** none. Change is per-agent, gated on each agent's own `.instar/`.
- **Other users of install base:** behavior change is strictly additive — agents with no nested better-sqlite3 copies see no extra work; agents with one (the common case) see preflight check it correctly the first time. Existing-install agents whose plists predate Change 4 are still covered by Change 3 (lifeline-side detection).
- **External systems (Telegram, Slack, GitHub, Cloudflare):** none.
- **Persistent state:** no schema changes. New in-memory counter only.
- **Timing:** preflight may take up to 5× longer than before in pathological trees with many copies. Bounded count (5) caps this. Real-world delta: from ~10s (old, one binary) to ~10–30s (new, 1–2 binaries). Acceptable for a supervisor cycle.
- **Backwards compat:** `installMacOSLaunchAgent` plist additions are additive. Existing plists that don't have `INSTAR_SUPERVISED=1` are still detected as supervised via Change 3's `comm=launchd` path. No regression.

## 7. Rollback cost

Pure code change. Revert + ship as a patch. No persistent-state changes. No user-visible regression during rollback window — agents that picked up the change would simply revert to the previous behavior. The new tests are additive and can be deleted on revert without breaking the build.

If a **bad rebuild** ships and corrupts an agent's better-sqlite3 install, rollback restores the previous preflight, but the corrupted install would persist until the next auto-update. Defense: the `verifyResult` re-load probe immediately after every rebuild — if it fails, we log and skip the heal claim, leaving the original binary in place.

## Conclusion

The change is at the right layer, doesn't add brittle blocking authority, has a small per-cycle cost increase, and has clear rollback semantics. Tests cover each new code path independently. Acceptance criteria from the spec all met:

1. ✅ Reproduction test: `find-better-sqlite3-copies.test.ts` covers hoisted, nested, both-shapes, missing-binary, recursive-skip, MAX_COPIES cap (8 tests).
2. ✅ Launchd-supervised detection: `detect-launchd-supervised.test.ts` covers all four signals, NODE_ENV=test override, cache bypass (10 tests).
3. ✅ Bind-failure escalation: code path verified via `force` flag plumbed into preflight; runtime behavior covered by existing supervisor-health-check test suite (88 tests pass).
4. ✅ Plist template adds env var: setup.ts edit verified by grep; `plutil -lint` runs in production install path.

The change is clear to ship.

## Second-pass review

**Reviewer:** independent-subagent
**Independent read of the artifact: concur-with-recommendations**

Verified against `docs/signal-vs-authority.md`, the spec, and each touched code path:

- **Signal-vs-authority compliance: clean.** `detectLaunchdSupervised` is a structural signal feeding `RestartOrchestrator` (the existing authority); the change strictly broadens the supervised-set without inventing new block authority. `findBetterSqlite3Copies` is a pure scanner. The `consecutiveBindFailures >= 2` threshold gates a recovery action (force-rebuild), not a block/allow on user input — qualifies as a transport-layer mechanic per the doc's allowed list.
- **Race on `consecutiveBindFailures`: none material.** Increment (`handleUnhealthy`), reset (healthy tick), and read (`preflightSelfHeal` via `force = this.consecutiveBindFailures >= 2`) all execute on the same single-threaded `setInterval` callback chain. A stale read would at worst over-rebuild by one cycle — same cost as a false escalation, already accepted.
- **`npm rebuild --prefix <prefixDir>` semantics: correct.** For the nested case, `prefixDir = shadow-install/node_modules/instar`, and npm rebuilds packages found in that prefix's `node_modules/` — i.e., the nested better-sqlite3, not the hoisted one. `path.dirname(path.dirname(packageDir))` is right for both shapes.
- **NODE_ENV=test leak risk: none.** Plist sets `PATH` and `INSTAR_SUPERVISED` only — no `NODE_ENV` entry. Production agents cannot accidentally short-circuit the detector.
- **Tests genuinely cover the claims.** Both new test files exercise each branch including the Inspec nested-path case, MAX_COPIES cap, recursive-skip, and the cache bypass.

**Non-blocking recommendations:**

1. **MAX_COPIES cap is silent.** Spec §Change 2 said "if we hit the cap, log + bail to the legacy path"; the implementation just returns the first 5 with no warning. Consider a `console.warn` in `findBetterSqlite3Copies` when the cap is hit so a future pathological tree surfaces a signal rather than a silent partial scan. Test at line 111–121 only asserts `<= 5`, not the log.
2. **`detectLaunchdSupervised` cache-bypass condition.** `isExplicitTestCall` is `opts.platform || opts.env || opts.ppid !== undefined || opts.parentNameLookup` — `opts.env = {}` is truthy so an empty-env test still bypasses the cache, which is what's wanted; just worth a one-line comment confirming the intent for future readers.

Neither blocks shipping.

---

## Evidence pointers

- New tests pass: `detect-launchd-supervised.test.ts` (10/10), `find-better-sqlite3-copies.test.ts` (8/8).
- Adjacent regression: `server-supervisor-preflight.test.ts` (4/4), `supervisor-health-check.test.ts` (6/6), full `lifeline/` suite (88/88) all pass.
- TypeScript: `tsc --noEmit` — no errors in changed files.
- Spec: `docs/specs/lifeline-self-heal-hardening.md`.
