# Long-Running Scripts — Discipline for Agents

When a shell script is expected to take more than ~5 minutes (audit harnesses, full test suites, benchmarks, multi-stage builds, vault sweeps), agents must follow a strict launch / poll / inspect protocol. Violating it produces parallel chains of expensive work, CPU spinlocks, and 50-minute false hangs.

Applies to every agent that orchestrates shell work — including but not limited to: `gmd-executor`, `gmd-debugger`, `gmd-verifier`, `gmd-integration-checker`, `gmd-nyquist-auditor`, `gmd-phase-researcher`.

---

## Iron rules

1. **Capture the PID** when launching a long-running script. Without a PID handle the script's liveness is unobservable.
2. **Poll with bounded patience and a stall detector.** Sleep meaningfully (≥ 5s); detect when output has not advanced for N consecutive iterations.
3. **READ the existing log on stall.** Do not re-invoke the script to "see what it is doing." The output is already on disk.
4. **NEVER busy-wait** with `until …; do true; done` or any sleepless loop. That is a CPU spinlock, not polling.
5. **NEVER re-invoke** the same script as a "probe" while one is already running. `bash script.sh 2>&1 | tail -N` is not a probe — it runs the entire script first.
6. **Surface to the user** when polling exceeds the bounded budget. A long stall is human-decision territory, not agent-self-help territory.

---

## Launch pattern

```bash
LOG=/tmp/script-$(date +%s).log
PID_FILE=/tmp/script-$(date +%s).pid

# Foreground (preferred — Bash tool blocks until completion):
bash path/to/script.sh > "$LOG" 2>&1 &
echo $! > "$PID_FILE"
PID=$(cat "$PID_FILE")
```

Use `$(date +%s)` (or `$BASHPID`) to avoid log/pid collisions across parallel runs.

If the script is launched as part of a single `Bash(…)` call that you expect to block, you don't need the `&` + PID dance — the Bash tool itself already waits. The launch pattern matters when you want to background-run and poll explicitly.

---

## Poll pattern

```bash
LAST_LINES=0
STALL_COUNT=0
MAX_ITER=60        # 60 × 10s = 10 min cap; tune per expected runtime
STALL_THRESHOLD=3  # 3 consecutive no-progress iterations = ~30s of silence

for i in $(seq 1 $MAX_ITER); do
  sleep 10

  # Liveness check
  if ! kill -0 "$PID" 2>/dev/null; then
    echo "[poll] process exited after $i iterations"
    break
  fi

  # Progress check
  LINES=$(wc -l < "$LOG" 2>/dev/null || echo 0)
  if [ "$LINES" -eq "$LAST_LINES" ]; then
    STALL_COUNT=$((STALL_COUNT + 1))
  else
    STALL_COUNT=0
  fi
  LAST_LINES=$LINES

  # Stall detection
  if [ $STALL_COUNT -ge $STALL_THRESHOLD ]; then
    echo "[poll] stalled at $LINES lines for $((STALL_COUNT * 10))s — inspecting log"
    break
  fi
done
```

**Constraints:**
- `sleep 10` minimum. 1s polling burns context on no-op echoes; sub-second polling burns CPU.
- `kill -0 "$PID"` is the only reliable liveness check. File-mtime is unreliable (some scripts work silently, then dump output at the end).
- The `MAX_ITER` cap ensures the loop terminates even if the stall detector misfires.

---

## On stall — what to do

Order of operations, **strictly read-only until the user weighs in:**

1. `tail -100 "$LOG"` — show recent output. What was the last thing the script printed?
2. `ps --pid "$PID" -o pid,ppid,etime,cmd` (or `ps -ef | grep $PID`) — confirm still alive, see how long it has been running.
3. `ps --ppid "$PID"` — list children. Is it spawning subprocesses? Are any of them stuck (zombie, deep recursion)?
4. **Read the script source** (`tests/audit-foo.sh`) at the point matching the last log line. What is it likely doing now (network call, file walk, subprocess invocation)?
5. **Surface findings to the user.** Format: "stalled at line N; last output: '…'; subprocess tree: … ; suspected cause: … . Continue waiting / kill / investigate?"

**What NOT to do on stall:**
- Do not run the script again "to see what it does."
- Do not start a sibling instance to "compare."
- Do not `cat` the same output file in a tight loop.
- Do not invent a probe. If the script's output isn't telling you what's happening, the answer is in the script's source or the process tree, not in another invocation.

---

## Forbidden patterns (memorize these)

| Pattern | Why it's wrong | Right alternative |
|---|---|---|
| `until [ -s "$f" ]; do true; done` | CPU spinlock — burns 100% of one core. | `until [ -s "$f" ]; do sleep 5; done` (with a max-iter cap) |
| `bash script.sh 2>&1 \| tail -60` as a "probe" | Runs the entire script first, then pipes through tail. Not a probe — a full re-invocation. | `tail -60 "$LOG"` of the existing run's log. |
| Spawning a second `bash script.sh` while the first is running | Doubles workload, contends on shared state, makes the apparent hang worse. | `tail -100 "$LOG"`, then surface to user. |
| `while true; do …; sleep 1; done` | 1-second polling is too aggressive for >5min scripts; burns context. | `sleep 10` minimum, with bounded iteration. |
| Polling with no liveness check | Will spin forever if process died without writing exit marker. | `kill -0 "$PID"` every iteration. |
| Polling output file mtime alone | Some scripts work silently for minutes (file walks, subprocess audits) — false stall. | Track line count; if mtime stalls but the script is known to do silent work, increase `STALL_THRESHOLD`. |
| `cat "$LOG" \| wc -l` in a polling loop without a sleep | Same as spinlock. | Always include `sleep N` of N≥5 in any until/while polling. |

---

## Cross-platform notes

- **Windows / Git Bash:** mtime granularity can be ~1 second; `find -newer` may miss very-fast file modifications. If you're using mtime-based progress, use line-count instead. Native exes (node, sed) are slower than on Linux — be especially conservative with `MAX_ITER` for any script that invokes node many times.
- **Process visibility:** on Git Bash, `ps -ef` shows MSYS processes; native Windows processes (some node binaries) may not appear. Use `tasklist` as fallback for liveness.
- **`pkill` is unavailable on Git Bash by default.** Kill by PID: `kill <pid>` (use `kill -9` only if `SIGTERM` is ignored).
- **`set -euo pipefail`** at the top of any orchestration script catches silent failures early — but be aware it changes failure semantics for downstream `||`/`!` constructs.

---

## Escalation rule

If the script crosses **2× its expected runtime** without producing an exit marker, stop polling and surface to the user. Show the log tail, process tree, and the line in the script source where execution is suspected to be stuck. The user decides whether to wait, kill, or investigate.

The 2× rule is empirical: most genuinely-slow scripts finish within their expected runtime; a script at 2× is more likely to be stuck than to be working harder than expected. Surface, don't speculate.

---

## When this reference applies

- Any audit script in `tests/` directory of a GMD project (especially `tests/audit-*.sh`).
- Full test suite invocations (`npm test`, `pytest`, etc.).
- Build steps known to take >5 min (large compilations, vault sweeps, mass-doc-regen).
- `gmd-tools.cjs` operations that spawn subprocess chains (`vault-sync --all`, `vault-extract-knowledge`).
- Any subprocess where you wrote the launch as `bash foo.sh` or `node foo.js` and don't already know the runtime is fast.

When in doubt, treat the script as long-running. The discipline is cheap; the alternative (50-minute false hang from a fork-bomb of duplicate runs) is expensive.

---

## See also

**`references/anti-busy-wait-discipline.md`** — the complementary discipline for
agent-turn safety while a background process runs. This doc (long-running-scripts)
covers shell-script orchestration: how to launch a long-running script (PID +
log redirection), how to poll its log (`sleep 10` + stall detector), and what
to do on stall (read the existing log, do not re-invoke). The anti-busy-wait
doc covers what your TOOL CALLS may not do while you wait — no-op `echo wait`
loops, chained `date; kill -0` polls, sub-minute `sleep` chains. Complementary,
not duplicative; both are mandatory reading for agents that orchestrate shell
work.
