# Autonomy / Goal System Reliability Audit — 2026-05-28

Triage of every reliability hole in `src/autonomy/` and the gateway entry
points that feed it. Each entry has a confirmed file:line, the failure mode
it produces in production, and the fix (or a "left unfixed" rationale).

Branch: `feat/activity-page-running-agents-restyle`.
Audit scope: `src/autonomy/**`, `src/gateway/server-methods/autonomy.ts`,
`src/agents/tools/goal-tool.ts`, `scripts/test-goal-loop.ts`.

---

## Holes confirmed + FIXED in this pass

### H1 — `runLoop` swallows top-level throws and strands the task

**File:** `src/autonomy/orchestrator.ts:346` (function body) and `:296` (the
`startOrchestratedTask` caller).

`runLoop` wraps every `spawnSubagent` / `waitForSubagent` in `try/catch` and
calls `deps.failTask` on each path. But the loop body itself has no
top-level catch: a throw in `notifications.iterationStarted`,
`syncToKanban`, `parseTesterVerdict`, or any other helper bypasses every
inner handler. The outer `runLoop(task, deps).finally(() => running.delete(task.id))`
removes the in-flight marker, but the queue task stays `status: "running"`
with no `completedAt`, no audit failure entry, no Telegram notification — it
is silently abandoned.

**Fix:** wrap the entire `runLoop` body in `try/catch`. The catch sets
`task.currentPhase = "failed"`, fires `deps.failTask`, notifies, and
re-throws (so the gateway-side `console.error` still triggers).

**Test:** new test in `orchestrator.test.ts` that makes
`notifications.iterationStarted` throw; asserts `failTask` is called and the
running map is cleared.

---

### H2 — Builder/tester `timeout` is treated as success → spawns tester on top of a still-running builder

**File:** `src/autonomy/orchestrator.ts:480-512` (build phase) and
`:572-593` (test phase).

`deps.waitForSubagent` returns one of three statuses: `ok | error | timeout`.
The build branch only short-circuits on `"error"` (`:499`). A `"timeout"`
result falls through to the tester phase, but the actual subagent is still
running (the timeout is the *wait* timeout — the agent has 1 h default
`runTimeoutSeconds`, vs the 30 min wait). The tester then runs against
nothing.

The tester branch has the same shape (`:572`): a timeout there is silently
written into `iteration.testerVerdict` as a `parseTesterVerdict("Tester timeout: …")`
string, which then either silently passes or fails depending on regex.

**Fix:** treat `"timeout"` like `"error"` — stop the iteration, fail the
task with a `builder timed out (waited Xs)` reason, do NOT spawn the
tester. Same in the tester branch (cap iteration, surface the timeout as a
failed verdict).

**Test:** new test in `orchestrator.test.ts` simulating a builder `"timeout"`
and asserting `failTask` is called with a `timed out` reason and that the
tester spawn is never invoked.

---

### H3 — Concurrency-cap math double-counts in-flight spawns

**File:** `src/autonomy/goal-orchestrator.ts:233-250` (the `spawn_subagent`
branch of `executeToolCall`).

```ts
const all = listSubagentRunsByGoalId(context.goalId);
if (all.length + context.spawnedRunIds.length >= MAX_TOTAL) { … }
const active = all.filter((r) => !r.endedAt);
if (active.length + context.spawnedRunIds.length >= MAX_PARALLEL) { … }
```

`setSubagentLineage` is called inside `spawnGoalSubagent` (line 203) so by
the time the *next* iteration of the loop runs, the freshly-spawned run is
already in `all` (and in `active`). We then add `context.spawnedRunIds.length`
again on top — double counting every spawn from this turn.

In practice: with `MAX_PARALLEL=4`, the orchestrator caps at 2 fresh
spawns per turn instead of 4. With `MAX_TOTAL=20` and 10 prior spawns, the
3rd spawn this turn already trips the cap.

**Fix:** drop the `context.spawnedRunIds.length` term from both checks —
the registry view already reflects this turn's spawns.

**Test:** extend `autonomy.e2e.test.ts` scenario 5 to seed
`listSubagentRunsByGoalIdMock` with the in-flight runs (mirroring real
behavior) and assert the cap fires at 5 with a `MAX_PARALLEL=4` env, not
at 3.

---

### H4 — `validatePrefire` accepts a 0/negative `maxIterations` from the dynamic form

**File:** `src/autonomy/orchestrator.ts:63-121` (prefire validator) and
`:398` (consumer).

The legacy questionnaire validator rejects `maxIterations < 1`
(orchestrator.ts:111). The dynamic-form validator never checks
`maxIterations` at all — if the operator answers `0` (or omits the field),
`runLoop`'s `Math.max(1, prefire.maxIterations)` rescues it… but only
because it also falls back to `5` when `answerNum` returns `undefined`.
If the form requires `maxIterations` and the operator submits `0`, the
loop runs exactly once silently, which is not what they asked for.

**Fix:** in the dynamic-form branch of `validatePrefire`, if a field with
id `maxIterations` exists and has a numeric value `< 1`, add it to the
missing list. Cheap, surgical.

**Test:** add a case to `spawn-params.test.ts`… actually no — this is in
`orchestrator.ts`. Add a new `describe` block in `orchestrator.test.ts`
covering the dynamic-form path of `validatePrefire`.

---

### H5 — Heartbeat path can race with the orchestrator path on `currentTask`

**File:** `src/autonomy/manager.ts:131-193` (heartbeat) and
`:309-344` (startTask) and gateway `autonomy:queue.start` in
`src/gateway/server-methods/autonomy.ts:238-278`.

`doHeartbeatWork` checks `if (currentTask) return`, then sets
`currentTask = task; queue.start(task.id)`. The orchestrator path (gateway
`queue.start`) calls `queue.start()` directly without touching the
manager's `currentTask` singleton (comment at manager.ts:384-388 admits
this). If both fire at the same time:

1. Heartbeat reads `currentTask == null`, picks task A.
2. Gateway `queue.start` fires for task B in parallel; `queue.start(B)`
   flips B to running.
3. Heartbeat then calls `queue.start(A)`, flips A to running.

Result: two simultaneously-running tasks even though the manager promised
exactly one. This is the bug behind "we tried to run two autopilot tasks
at once and one stalled."

**Fix:** in `doHeartbeatWork`, after `if (currentTask) return`, also bail if
any queue entry already has `status === "running"`. Cheap and correct
because the heartbeat is a single-concurrency cap by design.

**Test:** new test in a new `describe("doHeartbeatWork")` block in
`autonomy.e2e.test.ts`-style mocked setup. Simulate one queued task that's
already `running` and assert heartbeat returns `didWork: false`.

---

### H6 — Goal orchestrator's `await_subagent` returns `ok: false` on bad runId without a hint

**File:** `src/autonomy/goal-orchestrator.ts:274-304`.

If the planner emits an `await_subagent` with a runId the gateway can't
find, `agent.wait` returns `{status: "timeout"}` (no error). The orchestrator
maps that to `ok: false` but with no message — the synthesis turn sees
`await_subagent: FAILED {"runId":"...","status":"timeout","reply":""}` and
the model can't decide what to do.

Minor, but fixable: if `status === "timeout"`, the error field becomes
`"await timed out after Xs (runId may be unknown or still running)"`. The
synthesis prompt then has a real signal.

**Fix:** populate `error` on timeout. ~3 LOC.

---

## Holes confirmed but NOT FIXED (deferred — over budget or out of scope)

### H7 — `autonomy:goal.*` gateway handlers do NOT EXIST

**File:** `src/gateway/server-methods/autonomy.ts` (entire file — no `goal`
methods registered), referenced by `src/agents/tools/goal-tool.ts:74,92,100,127,143`
and `scripts/test-goal-loop.ts:33,56`.

The goal-tool calls `autonomy:goal.enqueue`, `autonomy:goal.status`,
`autonomy:goal.list`, `autonomy:goal.clear`, `autonomy:goal.subtask_add`.
None of these are registered as gateway handlers anywhere. The
`runOrchestratorTurn` function in `goal-orchestrator.ts` is exported but
the *only* consumer in production code is `autonomy.e2e.test.ts` —
**the entire orchestrator-mode goal system is unreachable from the
runtime** because the goal-tool's gateway calls would just hard-fail with
"method not found".

This explains "the goal system is not reliable" — it doesn't actually run
goals; every goal-tool call against the live gateway fails immediately.

**Why not fixed here:** building the missing `autonomy:goal.{enqueue,
status, list, clear, subtask_add}` handlers + persistent goal queue +
goal-loop runner that calls `runOrchestratorTurn` until the judge says
done, plus storage shape, plus UI wiring, plus tests, is way over the
~150 LOC budget the task limits this audit to. This is a feature gap
(the goal system was started but the wiring never landed on this branch),
not a reliability bug. Flagged as **BLOCKED** per the task contract.

### H8 — Mid-run crash recovery doesn't exist

**File:** `src/autonomy/orchestrator.ts:11-13` (comment claims "manager
will re-pick the task up on resume"), `src/autonomy/queue.ts:144-163`
(`getNextTask` only returns `pending` / `approved` — never `running`),
`src/autonomy/manager.ts:23-74` (`initAutonomy` does no recovery).

If the gateway dies while a task is `running`, that task stays `running`
forever:

- `getNextTask` filters by `(status === "pending" || status === "approved")`,
  so it's never re-picked.
- `getStatusForUI` shows the task as still running in the UI.
- The orchestrator's in-memory `running` Map is empty after restart, so
  `startOrchestratedTask(task)` would actually re-spawn — except nobody
  calls it on boot.

**Why not fixed here:** a correct fix is policy-laden (do we re-run from
iteration 1? Continue from the last persisted iteration? Mark failed and
let the operator decide?). It cuts across the queue, manager, and
orchestrator. Designing this correctly is outside the budget and would
likely need product input. Flagged as **BLOCKED**.

A cheap stopgap (mark every `status === "running"` task on boot as
`failed` with reason "gateway restarted mid-run") would be a behavior
change visible to the UI; we are not making that call without operator
sign-off.

### H9 — Heartbeat/orchestrator concurrency split is architecturally confusing

Same area as H5. `getCurrentTask()` is exported but only the heartbeat
path actually maintains it; the orchestrator path bypasses it (manager.ts:384-388
admits this). The fallback in `getStatusForUI` (filter by `status === "running"`)
works but means two sources of truth. A real fix would unify on the queue
status as the singleton truth and remove `currentTask`. Out of budget;
the H5 fix is sufficient to plug the race in practice.

---

## Holes investigated and found to be fine

- **Mocked-out goal-mode self-heal test** (`orchestrator.test.ts:177-220`)
  is `describe.skip`'d because the `kind: "goal"` branch it tests doesn't
  exist in `orchestrator.ts`. It's correctly skipped; not a bug.
- **`spawn-params` validators** are already tight and tested in
  `spawn-params.test.ts`. No regressions found.
- **`retryWithBackoff` non-retryable classification** was widened in commit
  8998dca and now matches Hermes — verified against `NON_RETRYABLE_SUBSTRINGS`
  and the existing `retry.test.ts` suite.
- **`tryParsePlanResponse`** and the repair-retry path are exercised by
  `autonomy.e2e.test.ts` scenarios 4a/4b. Correct behavior.
- **Wallclock cap short-circuit** (`goal-orchestrator.ts:503-526`) fires
  before credential resolution, exactly as designed. Covered by scenario 6.
- **Prefire form draft persistence** (commit 741b9f0) — verified
  `autonomy:queue.answerPrefireForm` mutates `form.answers` in place and
  calls `queue.persist()`; the regenerate path preserves prior answers via
  `{ ...result.form, answers: { ...prior } }`. No leak.
- **`sanitizeToolResult` framing-escape guard** — tested in
  `spawn-params.test.ts`. Correct.

---

## How to run

```bash
# Unit + e2e tests for autonomy stack
pnpm vitest run src/autonomy/

# Full goal-loop E2E (needs the gateway running)
pnpm tsx scripts/test-goal-loop.ts
```

The vitest suite is hermetic — gateway/pi-ai/subagent-registry are mocked
at the module boundary, so it runs in ~1s and proves the H1–H6 fixes.
