# Worktree Lifecycle

Deep-dive for the four engine pieces that own git-worktree state for dispatch:
the **pool** (recycling), the **live guard** (don't wipe an agent's work),
the **quarantine path** (dirty/divergent → quarantine dir + retry), and the
**Windows file-lock retry** (EPERM/EBUSY footgun). CLAUDE.md → Worktree
Lifecycle keeps the cross-cutting invariants; the detail lives here.

> Source of truth: `engine/worktree-pool.js`, `engine/shared.js#removeWorktree`
> + `_retryFsOp`, `engine.js` (`_quarantineDirtyWorktree`, `_renameWithRetry`,
> `_killGitDescendantsForWorktree`, `pruneOrphanWorktrees*`, `gcDispatchWorktreeIfOrphan`),
> `engine/cleanup.js`. Last verified: 2026-06-09.

## Live-worktree guard (W-mq5rwwss000f30a7)

**Invariant:** every code path that wipes, renames, or recycles a worktree
MUST call `shared.isWorktreePathLive(path, { db?, excludeDispatchId? })`
first and skip on `true`. Originally added after W-mq5n1zx5000hcfb5 — an
agent's worktree was wiped 4× consecutively by the reaper while the
dispatch was still active.

- **Backed by SQL** against `dispatches` (status IN ('pending','active')),
  reading `json_extract(data, '$.worktreePath')` with a `data.meta.worktreePath`
  fallback. The pending → active transition persists `item.worktreePath`
  on the dispatch row so the guard has a path to correlate.
- **Fails OPEN** (returns `true`) when SQLite is unreachable or the query
  throws — better to leak a worktree than nuke an agent's unpushed work.
- **Wired sites:** `shared.removeWorktree` (accepts `{ excludeDispatchId }`
  forwarded to the guard); pool-return chain in `engine.js`
  (`excludeDispatchId: id` so the dispatch can clean up its own worktree);
  dispatch-end orphan GC (`gcDispatchWorktreeIfOrphan`); `_quarantineDirtyWorktree`
  (returns `{ skipped: true, quarantinedPath: null }` on skip — callers
  MUST honor `skipped` and not set `quarantined: true`); `engine/cleanup.js`
  orphan-dir sweep.
- **On skip:** drops a deduped operator note at
  `notes/inbox/engine-worktree-skip-live-<basename>-<date>.md`.

## Worktree pool (opt-in)

`ENGINE_DEFAULTS.worktreePoolSize > 0` enables `engine/worktree-pool.js` to
recycle worktree dirs across branches.

- **Borrow** in `spawnAgent` only when (a) the new branch doesn't exist on
  origin AND (b) the dispatch is not shared-branch / `useExistingBranch`.
- **Return** in `onAgentClose` BEFORE `completeDispatch`:
  `git reset --hard HEAD` → `git clean -fd` → `git fetch origin <main>`
  → `git checkout --detach origin/<main>` → mark IDLE.
- **State** at `engine/worktree-pool.json`; git ops outside any lock.

## Quarantine path (dirty / divergent)

When `discoverFromWorkItems` finds a worktree in a `WORKTREE_DIRTY`,
`WORKTREE_DIVERGENT`, or post-stuck state, `_quarantineDirtyWorktree`
moves it to `<root>/.quarantined/<basename>-<utc>` and lets the next tick
recreate a clean worktree. Six layered defenses cover the Windows EBUSY
race when git status descendants still hold packfile handles
(W-mq5n1zx5000hcfb5; PR #3156).

### Layer 1b — `--no-optional-locks` on every status probe (load-bearing)

`_statusPorcelainCmd()` emits `git --no-optional-locks status …`. Skips
the `.git/index.lock` acquire around the untracked-cache refresh inside
`status`. Typical probe duration under AV scanning drops from 6–12s to
<500ms, which removes the timeout that leaks the child in the first
place. **Most incidents are closed by 1b alone.**
Toggle: `ENGINE_DEFAULTS.statusProbeUseNoOptionalLocks` (default `true`).

### Layer 1a — `_renameWithRetry` with jittered backoff

6 attempts × 250 ms base × 2^N exponential + 200 ms random jitter
(~16 s worst-case). Only retries on `EBUSY|EPERM|EACCES|ENOTEMPTY`;
rethrows other codes immediately. Toggles:
`ENGINE_DEFAULTS.quarantineRenameRetryAttempts` (6),
`quarantineRenameRetryBaseMs` (250).

### Layer 2a — `_killGitDescendantsForWorktree` (Windows-only)

PowerShell sweep, 2 s timeout. Shell-out built via single-quoted literal
(`'` escaped as `''`) so worktree-path interpolation is injection-safe.
POSIX no-op. Best-effort — failure logged, never throws.
Toggle: `ENGINE_DEFAULTS.statusProbeKillDescendantsWin32` (default `true`).

### Layer 2b — `git worktree remove --force` fallback

Triggers only when 1a exhausts retries; destroys worktree contents (no
quarantine dir preserved). Toggle:
`ENGINE_DEFAULTS.quarantineForceRemoveFallback` (default `true`).

### Layer 1c — `FAILURE_CLASS.WORKTREE_QUARANTINE_ENV_BLOCKED`

Routed when `cleanResult.quarantineError && !cleanResult.quarantined`.
Added to `dispatch.js#isRetryableFailureReason` never-retry set so the
per-agent retry counter doesn't bump (env failure, not the agent's fault).
The auto-recovery loop in `engine.js#discoverFromWorkItems` recognizes
the new class via both enum check and regex on legacy `failReason`
strings, then re-queues under the existing
`ENGINE_DEFAULTS.quarantineAutoRecoveryMax` (default 2) cap.

### Layer 3a/3b — metrics + inbox alert

Counters at
`metrics._engine.worktreeQuarantineOutcomes.{attempts, success, successAfterRetry, fallbackForceRemove, totalFailure}`.
Total-failure path writes a structured inbox alert with the exact
PowerShell/POSIX recovery commands and `git worktree prune` instructions.

### Auto-recovery cap

`discoverFromWorkItems` auto-recovers `WORKTREE_DIRTY`/`WORKTREE_DIVERGENT`
quarantine (#2996) up to `ENGINE_DEFAULTS.quarantineAutoRecoveryMax`
(default 2, tracked on `_quarantineRecoveryCount`); after the cap,
`_quarantineRecoveryGaveUp` triggers a warn-once.

## Windows file-lock on removal (footgun #6 detail)

`fs.rmSync` against an active git worktree can lose to lingering file
handles (CC log streams, virus scanners, Explorer previews).
`shared.removeWorktree` retries via `shared._retryFsOp`
(`worktreeRemoveRetryAttempts` × exponential `worktreeRemoveRetryBaseMs`,
codes `EPERM|EBUSY|EACCES|ENOTEMPTY`).

After `worktreeStuckThreshold` consecutive failures the path is
**escalated**:

- A `notes/inbox/engine-worktree-stuck-<basename>-<date>.md` note with
  Windows holder-identification hints is written (deduped per UTC day).
- Per-tick warns are suppressed for `worktreeStuckSuppressMs`.
- The retry cadence drops to `worktreeStuckSlowRetryMs`.
- When the holder finally releases, a
  `worktree-recovered-<basename>` note clears the alert.

**Don't fight this with `fs.rmSync({force:true})` outside
`removeWorktree`** — you'll bypass the retry, escalation, and metrics
layers.

## Periodic prune (W-mq5o6bvy000x7191)

`pruneWorktreesPeriodic` runs `pruneOrphanWorktrees` (in-root) +
`pruneOrphanWorktreesFromGitRegistry` (out-of-root via
`git worktree list --porcelain`) per project at
`ENGINE_DEFAULTS.worktreePruneIntervalTicks` cadence — catches Windows
EPERM/EBUSY stragglers that the dispatch-end GC couldn't reap and sweeps
the `git worktree list` registry for OUT-of-root entries the in-root
scanner is blind to.

## Holder identification + opt-in auto-reap (W-mq6f2fe0000557fa)

Orphan-sweep escalations also run `shared.findProcessesWithCwdInside(wt)`
(cross-platform: PowerShell `Get-CimInstance Win32_Process` on Windows;
`/proc/*/cwd` walk on Linux; `lsof -d cwd` + `ps` on macOS) and append a
`## Live holders` section listing pid / cmdline / age to the
`engine-worktree-stuck-<basename>` escalation note.

Setting `engine.autoReapOrphanWorktreeHolders: true`
(Settings → Worker Pool & Worktrees → "Auto-reap orphan worktree holders")
additionally kills any holder whose `cmdline` matches `spawn-agent.js`
AND references the worktree basename AND whose age exceeds
`engine.agentTimeout * 2`, then retries `removeWorktree` once. Default
**OFF** — killing a foreign process is destructive.

- Scan timeout: `engine.orphanHolderScanTimeoutMs` (default 5000ms,
  clamped 1000–30000).
- Motivating incident: W-mq1k8z6o003acd89 / stuck `spawn-agent.js`
  PID 16828 that the periodic prune couldn't shake loose without manual
  intervention.
