# Managed-spawn primitive

> Engine-owned long-running services with sidecar declaration, healthcheck-gated dispatch SUCCESS, and dashboard discovery.
> Plan: [`plans/plan-w-mp7k1r760003b5dd-2026-05-15.md`](../plans/plan-w-mp7k1r760003b5dd-2026-05-15.md) · Module: [`engine/managed-spawn.js`](../engine/managed-spawn.js) · Dashboard panel: `/engine` → "Managed Processes".

## Why this exists

Before managed-spawn, Constellation (and similar multi-process projects) lost dev servers every time `minions restart` ran. The original Lambert/Ralph failure pattern:

1. Agent spawned `bun dev` as a detached child.
2. Wrote down PID in `keep-pids.json` so the engine wouldn't kill it.
3. `minions restart` killed the engine but left the child running — *on Node*. **On `bun` + Windows the child died with the parent** because the Windows detached-spawn semantics differ across runtimes.
4. Next dispatch landed against a dead URL with no signal it had died.

Managed-spawn moves the spawn ownership from the agent into the engine. Agents *describe* services in a sidecar; the engine spawns them using the proven [`bin/minions.js spawnDashboard`](../bin/minions.js) pattern (works for Node, bun, python, docker, …), runs healthchecks, gates dispatch SUCCESS on first-healthy, sweeps dead PIDs / expired TTL, auto-injects live processes into downstream agent prompts, and exposes everything through `/api/managed-processes` + the dashboard.

## When to use managed-spawn vs `keep_processes`

| Need | Use this |
|---|---|
| Long-running dev server / emulator / build daemon that downstream agents need to hit | **managed-spawn** |
| Short-lived helper the *same* agent needs alive past its exit (e.g. `gradle --daemon` for the next gradle invocation) | `meta.keep_processes` (existing per-agent `keep-pids.json`) |
| One-shot script that exits on its own | Neither — just run it inline |

Both can coexist. There's no plan to deprecate `keep_processes`; a future cleanup PR can revisit if managed-spawn fully subsumes its use cases.

## Sidecar schema

The sidecar lives at `<MINIONS_DIR>/agents/<agentId>/managed-spawn.json` and is read **once** by the engine in `onAgentClose`, after the agent's process exits. (Per-tick polling for sidecar edits is explicitly out of scope — pick a final shape before exit.)

```jsonc
{
  "specs": [
    {
      "name": "constellation-host",          // kebab-case, ≤64 chars, unique within file
      "cmd": "bun",                          // must be on engine.managedSpawn.executableAllowlist
      "args": ["run", "dev"],                // ≤64 entries
      "cwd": "D:/repos/constellation",       // must be inside a real git worktree (requireGitWorkdir: true) — monorepo subdirs ok, ancestor walked up to gitWorktreeMaxParentDepth parents
      "env": { "CONSTELLATION_SERVER": "http://localhost:3000" },  // ≤32 keys; POSIX-shape + denylist enforced
      "ports": [3001],                       // 1024-65535; ≤20 per spec; advisory only (engine doesn't bind)
      "ttl_minutes": 240,                    // ≤1440 (24h hard cap); defaults to 720 (12h)
      "attrs": {                             // opaque per-spec metadata, ≤2048 bytes serialized
        "base_url": "http://localhost:3001",
        "framework": "vite"
      },
      "healthcheck": {                       // required for SUCCESS gating
        "type": "http",                      // "http" | "command"
        "url": "http://localhost:3001/health",
        "expect_status": 200,
        "interval_s": 1,
        "timeout_s": 60                      // total wait for first-healthy before failing dispatch
      }
    }
  ],
  "written_by": "<your-agent-id>",
  "wi_id": "<this-work-item-id>"
}
```

The renderer in [`buildManagedSpawnHint`](../engine/managed-spawn.js) at `engine/managed-spawn.js:419` emits this exact shape (with allowlist + cap reminders) into the agent's prompt whenever the work item has `meta.managed_spawn: true`. Treat the rendered hint as the source of truth — if this doc and the hint drift, the hint wins.

### Smoke-test before writing the sidecar (mandatory, W-mpbpexrg00110661)

The hint mandates that agents run each `cmd args` in the declared `cwd` for at least 5 seconds AND confirm the healthcheck endpoint returns the expected status BEFORE writing `managed-spawn.json`. Sidecars written from guessed-at commands are how dispatches silently lose specs: the engine spawns what the agent declared; if the command crashes on first launch, the engine kills the spec, removes it from state, and (for partial failures) marks the WI with `_managedSpawnPartial` rather than demoting it — the agent's primary work succeeded by validator standards. The cost of guessing wrong is the operator finding out hours later that half the stack is down.

Real failure class to internalize: workspace-filter resolution (`bun -F <name>`, `pnpm --filter <name>`) failing because workspace deps were never installed in the worktree. Exits 1 in <1s with `No packages matched the filter`. Smoke-test catches it in seconds; an operator finding it catches it in hours (incident W-mpbolwvt000gb9b1: 1 of 3 Constellation specs survived, WI showed `done`).

Both a PowerShell (Windows) and bash (macOS/Linux) example are inlined in the hint — see `buildManagedSpawnHint` for the canonical patterns.

## Healthcheck examples

### HTTP — most common

```jsonc
"healthcheck": {
  "type": "http",
  "url": "http://localhost:3001/health",
  "expect_status": 200,
  "interval_s": 1,
  "timeout_s": 60
}
```

Probes `GET <url>` on `interval_s` cadence until `response.status === expect_status` or `timeout_s` elapses. No body assertion (deferred — see plan rejected items). Auth headers / cookies not supported (local dev only).

### Command — anything that exits 0 + matches a regex

```jsonc
"healthcheck": {
  "type": "command",
  "cmd": "curl",
  "args": ["-fsS", "http://localhost:3001/ready"],
  "shell": false,
  "expect_regex": "^ready$",
  "interval_s": 2,
  "timeout_s": 30
}
```

Runs the command with `child_process.exec` / `spawn`, asserts both exit code 0 AND stdout matching `expect_regex`. The `cmd` must be on the same executable allowlist as `spec.cmd`. Use `command` healthchecks for:
- Raw TCP — `Test-NetConnection -Port 3001 -ComputerName localhost` on Windows / `nc -z localhost 3001` on POSIX.
- File presence — `Get-Content -Tail 1 log.txt` matching `^ready$`.
- Database connectivity — `docker exec pg pg_isready`.

`tcp` and `log-match` healthcheck types are intentionally not implemented — see the plan's Rejected items for the reasoning.

## Lifecycle

```
┌───────────────────────────────────────────────────────────────────┐
│ 1. Agent runs                                                     │
│    - Playbook injects the managed-spawn hint when                 │
│      work_item.meta.managed_spawn === true                        │
│    - Agent writes agents/<id>/managed-spawn.json before exit      │
└───────────────────────────────────┬───────────────────────────────┘
                                    │ engine.onAgentClose
                                    ▼
┌───────────────────────────────────────────────────────────────────┐
│ 2. evaluateManagedSpawnAcceptance(agentId)                        │
│    - Validates schema, allowlists, caps                           │
│    - Rejects → ERROR + alert + sidecar deleted                    │
│      failure_class: invalid-managed-spawn (non-retryable)         │
└───────────────────────────────────┬───────────────────────────────┘
                                    │ accepted
                                    ▼
┌───────────────────────────────────────────────────────────────────┐
│ 3. spawnManagedSpec(spec) per spec                                │
│    - Uses the proven Windows detached-spawn pattern               │
│    - Records each in engine/managed-processes.json (locked write) │
│    - Stdio → engine/managed-logs/<name>.log (append fd, NOT pipe) │
└───────────────────────────────────┬───────────────────────────────┘
                                    │
                                    ▼
┌───────────────────────────────────────────────────────────────────┐
│ 4. waitForFirstHealth(spec) per spec, in parallel                 │
│    - Probes every interval_s, gives up at timeout_s               │
│    - First success → state.healthy = true, last_health_at = now   │
│    - Any failure → dispatch ERROR + sibling spawns left alive     │
│      failure_class: managed-spawn-healthcheck (retryable)         │
└───────────────────────────────────┬───────────────────────────────┘
                                    │ all healthy
                                    ▼
┌───────────────────────────────────────────────────────────────────┐
│ 5. Dispatch SUCCESS                                               │
│    - Spec survives engine restart (PID detached + boot reconcile) │
│    - Visible at /api/managed-processes + dashboard "Managed       │
│      Processes" panel                                             │
│    - Downstream agent prompts auto-inject a "## Live managed      │
│      processes" block scoped to this project                      │
└───────────────────────────────────────────────────────────────────┘
```

Background work the engine does without further agent involvement:
- **Per-tick** (cadence `engine.managedSpawn.sweepEvery: 30`, ~30 min): `sweepManagedSpawn()` drops dead-PID rows, kills + unlinks TTL-expired specs, rotates `managed-logs/<name>.log` past `logRotateBytes` (10MB).
- **Per-engine-boot**: `bootReconcileManagedSpawn()` (timeout-bounded by `bootReconcileMaxMs: 2000` ms) drops dead PIDs, kills expired, runs one immediate healthcheck per survivor.
- **Per-project-removal**: `removeManagedSpecsForProject(name)` (called from `engine/projects.js removeProject`) kills + unlinks specs whose `owner_project` matches.

## WI 1 (build) → WI 2 (test) chained-validation pattern

The canonical use case: split a build + smoke-test workflow across two work items so each can be retried / restarted independently.

**WI 1 — build & host:**
- `meta.managed_spawn: true`
- Playbook: `implement` (or whichever fits)
- Sidecar spec: dev server, emulator, or whatever the next WI needs.
- `task_description`: build + verify locally, write the sidecar.
- Engine gates SUCCESS on healthcheck → spec is healthy when this WI completes.

**WI 2 — test against the live service:**
- `dependencies: [WI-1-id]` (engine won't dispatch until WI 1 is SUCCESS)
- `meta.managed_spawn: false` (or omit — no new specs to write)
- Playbook auto-inject (from `engine/playbook.js`) ships a `## Live managed processes for project <name>` block in WI 2's prompt with WI 1's specs' `name`, `pid`, `attrs`, `log_path`, `ttl_expires_at`, and base URL.
- WI 2 reads `attrs.base_url` from its prompt context, hits the live service, runs whatever test it needs to.

Outcome: if WI 2 fails, the spec stays alive (TTL-managed) and the next dispatch of WI 2 reuses the same instance — no rebuild churn. If WI 1 fails healthcheck, WI 2 never dispatches; you fix WI 1 and retry.

## Operating

### Inspect at runtime

| What you want | Where |
|---|---|
| All live specs (JSON) | `GET /api/managed-processes` |
| Filtered to one project | `GET /api/managed-processes?project=<name>` |
| One spec | `GET /api/managed-processes/by-name/<name>` |
| Live log tail (SSE) | `GET /api/managed-processes/log-stream/<name>` |
| Dashboard | http://localhost:7331/engine → "Managed Processes" |

The list endpoint returns an ETag; pass it as `If-None-Match` on follow-up polls to get `304 Not Modified` when nothing changed. The dashboard panel does this automatically.

### Force a restart / kill

| Action | Endpoint | Dashboard |
|---|---|---|
| Kill (PID terminated, removed from state) | `POST /api/managed-processes/kill` `{"name":"…"}` | Per-row "Kill" button |
| Restart (kill old PID, respawn from saved state, kick first healthcheck) | `POST /api/managed-processes/restart` `{"name":"…"}` | Per-row "Restart" button |

Killing a spec from outside Minions (raw `Stop-Process`) leaves a stale row in `managed-processes.json` until the next 30-tick sweep notices the dead PID. Prefer the API.

### View logs

`engine/managed-logs/<name>.log` — append-only, rotated to `<name>.log.1` at 10MB. The dashboard's "Log" button opens an SSE stream of the live tail.

## Configuration

All knobs live under `engine.managedSpawn` in `engine/shared.js` (`ENGINE_DEFAULTS.managedSpawn`). Override per install via `config.json`:

| Key | Default | Notes |
|---|---|---|
| `enabled` | `true` | Global kill switch. `false` makes the engine ignore all sidecars + skip the sweep. |
| `maxSpecsPerFile` | `5` | Per-agent cap. |
| `maxTtlMinutes` | `1440` | Hard cap (24h). |
| `defaultTtlMinutes` | `720` | Fallback when `ttl_minutes` omitted (12h). |
| `sweepEvery` | `180` | Ticks between sweeps. Default tick = 10s ⇒ ~30 min. |
| `defaultHealthIntervalSec` | `1` | Healthcheck cadence pre-first-healthy. |
| `healthBackoffSec` | `30` | Healthcheck cadence post-first-healthy. |
| `logRotateBytes` | `10485760` | Rotation threshold for `<name>.log`. |
| `bootReconcileMaxMs` | `2000` | Boot-time reconcile hard timeout. |
| `promptContextMaxBytes` | `2048` | Auto-injected `## Live managed processes` block cap. |
| `requireGitWorkdir` | `true` | Reject specs whose `cwd` isn't inside a git worktree (root or any ancestor up to `gitWorktreeMaxParentDepth`). |
| `gitWorktreeMaxParentDepth` | `6` | How many parent directories `shared.isValidGitWorktree` walks when probing for `.git`. Lets monorepo specs pin a per-package `cwd` (`<root>/packages/<pkg>/...`); set to `0` to disable the walk. |
| `executableAllowlist` | `[node, bun, npm, …]` | Single global. Applies to `spec.cmd` AND `command` healthcheck `cmd`. |
| `envKeyDenyPatterns` | `[^AWS_, ^AZURE_, _SECRET, _TOKEN, _API_KEY, …]` | Regex source strings, matched case-insensitively. Keys matching ANY pattern are rejected unless exact-listed in `envKeyDenyOverrides`. Threat model: credential leakage, not env-key enumeration — plain project vars (`CONSTELLATION_SERVER`, `DATABASE_URL`, …) pass with no config. |
| `envKeyDenyOverrides` | `[AWS_REGION, AWS_DEFAULT_REGION, AZURE_REGION, GCP_REGION, AWS_PROFILE]` | Exact-match exemptions for known-safe keys that would otherwise be caught by a broad prefix pattern. Case-sensitive. |

## Failure modes

| Symptom | Likely cause | Fix |
|---|---|---|
| Dispatch ERROR `failure_class: invalid-managed-spawn` | Sidecar schema/allowlist violation | Read inbox alert; the validator includes a precise reason. Non-retryable — fix and re-dispatch. |
| Dispatch ERROR `failure_class: managed-spawn-healthcheck` | `timeout_s` elapsed before any spec became healthy | Check `engine/managed-logs/<name>.log` for the child's crash output. Sibling spawns are left alive. Retryable. |
| WI shows yellow `⚠ managed-spawn: N/M healthy` chip on dashboard | Partial healthcheck failure: ≥1 spec passed, ≥1 failed. WI keeps `status: done` (the agent's primary work — getting an accepted sidecar — succeeded); annotation `_managedSpawnPartial = { healthy, failed[{name, reason, log_tail}], evaluated_at }` lives on the WI. Click the row to see per-spec failure reasons + last 20 log lines. Dispatch is still recorded ERROR with `failure_class: managed-spawn-healthcheck-failed`. Restart the failing spec via `POST /api/managed-processes/restart` once you've fixed the root cause (often: workspace deps not installed; see smoke-test rule in the hint). W-mpbpexrg00110661. |
| Spec gone after `minions restart` | Bun child died with parent (the original failure mode) | Should be fixed by item 2's detached-spawn pattern. If it recurs, verify `bin/minions.js spawnDashboard` semantics still work for the runtime — that's the canonical reference. |
| Spec listed `alive: true, healthy: false` for >30s | Healthcheck loop self-detected service degradation | The spec did not pass a subsequent healthcheck. Inspect the service; restart via API once recovered. |
| Stale row sticks around with dead PID | Spec killed outside Minions | Wait one sweep cycle (~30 min) or call `POST /api/managed-processes/kill` manually. |

## Performance budget

- **Per-tick contribution**: zero unless the 30-tick sweep fires. The sweep is `O(N)` over specs in `managed-processes.json` with one `process.kill(pid, 0)` per spec.
- **Healthcheck loops**: per-spec, self-scheduled (not tick-coupled). 10 specs × 1s pre-healthy cadence ⇒ 10 probes/s peak. Post-healthy drops to one probe per `healthBackoffSec` (30s).
- **Lock contention** on `managed-processes.json`: writes batch on transitions (healthy ↔ unhealthy), and `last_health_at` only persists every `healthBackoffSec` to avoid flap. See [`test/perf/managed-spawn-load.test.js`](../test/perf/managed-spawn-load.test.js) for the 10-spec / 3-project load assertion.
- **Hard cap recommendation**: 50 specs across one engine. Above that, `process.kill(pid, 0)` shells out enough on Windows to dent tick latency.

## Source map

| File | Purpose |
|---|---|
| `engine/managed-spawn.js` | Schema, validator, spawn, healthcheck, sweep, state-file I/O. All pure helpers; no side effects on import. |
| `engine.js` `onAgentClose` (line ~2247) | Acceptance gate (item 2) + healthcheck gate (item 3) wired into dispatch result. |
| `engine.js` `tickInner` (line ~5687) | Per-30-tick `sweepManagedSpawn()` invocation. |
| `engine/cli.js` | Boot-path `bootReconcileManagedSpawn()` invocation. |
| `engine/playbook.js` | Auto-inject `## Live managed processes` block + `managed_spawn` hint section. |
| `engine/projects.js` `removeProject` | `removeManagedSpecsForProject` cleanup hook. |
| `dashboard.js` | 5 routes (`/api/managed-processes`, `…/by-name/<n>`, `…/kill`, `…/restart`, `…/log-stream/<n>`). |
| `dashboard/js/render-managed.js` + `dashboard/pages/engine.html` | "Managed Processes" panel + log-viewer modal. |
| `engine/shared.js` `ENGINE_DEFAULTS.managedSpawn` | All configurable knobs in one place. |

## Plan items

| # | PRD id | Lands what |
|---|---|---|
| 1 | P-7a3b1c92 | Schema, validator, sidecar reader, defaults |
| 2 | P-2d5e8f04 | Engine spawn + locked state file |
| 3 | P-9c1f47a6 | Healthcheck implementations + dispatch SUCCESS/ERROR gate |
| 4 | P-4b8d2e57 | Discovery API (list / by-name / kill / restart + ETag) |
| 5 | P-6e2a8b13 | Dashboard panel + SSE log viewer |
| 6 | P-1f9c3a45 | Playbook auto-inject + dispatch hint |
| 7 | P-8a4d6f29 | TTL sweep + boot reconcile + project-removal + log rotation |
| 8 | P-3c7e9d18 | Load test + e2e + docs + Constellation smoke run *(this doc)* |
