# E2E Validation Audit Report: autonomy/ Module

**Date:** 2026-05-28  
**Scope:** src/autonomy/*.ts — Full TypeScript codebase analysis  
**Target Length:** 1,500–3,000 words  
**Author:** Autonomous Build — Iteration 1

---

## 1. Overview

The `src/autonomy/` module constitutes InfiniBot's trust infrastructure for autonomous agent operation—a sophisticated closed-loop orchestration system that enables AI agents to execute software engineering tasks with minimal human supervision. At its core, the module implements a **build-test-orchestrate** paradigm: tasks flow from a persistent queue through pre-fire validation (questionnaire completion), then enter an iterative loop where builder agents implement changes and tester agents verify outcomes against success criteria using Playwright MCP integration.

The architecture embraces production-reliability principles with multiple safety layers: a tamper-evident audit chain (`audit.ts`) appends SHA256-hashed entries for every action; hardware monitoring (`hardware.ts`) gates intensive work behind CPU/memory/GPU thresholds and user idle detection; criticality classification (`criticality.ts`) requires Telegram approval for destructive operations; and a file-based pause mechanism (`pause.ts`) provides immediate operational kill-switches. The orchestrator (`orchestrator.ts` and `goal-orchestrator.ts`) implements bounded retry with exponential backoff, spawn-parameter validation, concurrency caps (max 4 parallel sub-agents per goal, max 20 total), and wallclock timeouts to prevent runaway execution.

This audit catalogs 21 source modules, examines their public API contracts, maps test coverage gaps, and identifies three high-severity reliability concerns derived from static analysis of control flow, asynchronous patterns, and failure handling across ~3,500 lines of TypeScript implementation.

---

## 2. Module Inventory

- **api-monitor.ts** (264 LOC): SQLite-backed API usage tracking with autonomous vs. interactive cost attribution; mirrors data to unified credit ledger via `recordLegacyApiUsage()`.
- **approval-registry.ts** (134 LOC): In-memory pending approval registry with 30-minute expiry timers; Telegram callback resolution via `resolveApproval()`.
- **audit.ts** (197 LOC): Tamper-evident append-only JSONL logging with SHA256 chain verification; supports daily log rotation and integrity verification via `verifyAuditChain()`.
- **criticality.ts** (84 LOC): Heuristic task classifier using keyword matching to assign "low"/"medium"/"critical" tiers from prompts; triggers Telegram approval gates for critical operations.
- **goal-judge.ts** (163 LOC): LLM-based goal completion evaluation using Claude Haiku; parses JSON verdicts with fallback "continue" semantics on parse/auth failures.
- **goal-orchestrator.ts** (463 LOC): Multi-turn planning/execution/synthesis loop with 4-tool surface (spawn_subagent, await_subagent, list_my_subagents, cancel_subagent); enforces wallclock caps and repair-retry for malformed JSON.
- **hardware.ts** (352 LOC): Cross-platform system monitoring (Windows PowerShell, Linux /proc, macOS ioreg) for CPU, memory, GPU, and user idle time; gates autonomous work behind threshold checks.
- **index.ts** (36 LOC): Barrel file exporting all public subsystems for external consumption.
- **kanban-sync.ts** (130 LOC): Bidirectional sync between autonomy task state and Kanban/Notion boards; maps status enums with best-effort error handling.
- **manager.ts** (388 LOC): Central autonomy manager with 20-second heartbeat scheduler; coordinates queue draining, hardware checks, pause state, and orchestrator dispatch.
- **notifications.ts** (318 LOC): Telegram Bot API integration with batched notification queue; supports HTML formatting and retry logic for failed deliveries.
- **orchestrator.ts** (647 LOC): Closed-loop build/test orchestrator implementing iterative builder/tester cycles; includes spawn-parameter validation, tester verdict parsing, and prefire gating with NO-BLOCKAGES self-heal fallback.
- **pause.ts** (116 LOC): File-based pause control using JSON state in `state/autonomy/PAUSE`; supports scheduled auto-resume via `resumeAt` timestamps.
- **prefire-form-agent.ts** (268 LOC): Dynamic form generator using LLM calls to produce task-specific questionnaires; evaluates readiness with secondary LLM pass.
- **queue.ts** (268 LOC): JSON-file-backed task queue with priority ordering, status lifecycle management, and audit ID tracking per task.
- **retry.ts** (166 LOC): Exponential backoff retry utility with jitter; iteration-level retry policy for failed builds/tests with configurable backoff schedules.
- **spawn-params.ts** (118 LOC): Pre-call parameter validator enforcing label ≤63 chars, task ≤32,000 chars, and timeout clamping; prevents round-trip gateway rejections.
- **telegram-prefire.ts** (169 LOC): Telegram command handler (`/prefire`) for questionnaire completion without web UI; supports field type coercion.
- **tester-agent.ts** (159 LOC): Tester prompt builder and verdict parser; extracts fenced JSON from LLM replies with fallback failure verdicts on parse errors.
- **types.ts** (312 LOC): Comprehensive type definitions for audit entries, task queue, hardware thresholds, notification schemas, and prefire forms.

---

## 3. Public API Surface

### 3.1 Audit & Logging (`audit.ts`)
- `initAudit(basePath?: string): void` — Initialize audit log directory and load last hash for chain continuity.
- `logAudit(actionType, description, options?): AuditEntry` — Append tamper-evident entry to today's JSONL log.
- `readAuditLog(date?): AuditEntry[]` — Retrieve entries for specific date (defaults to today).
- `getRecentAuditEntries(limit?): AuditEntry[]` — Fetch N most recent entries across all daily logs.
- `verifyAuditChain(date?): { valid: boolean }` — Verify SHA256 chain integrity for tamper detection.
- `getTodayAuditStats(): Stats` — Aggregated statistics by category and outcome.
- `logCageInterference(description, details?): AuditEntry` — Specialized audit entry for cage influence detection.
- `audit` (convenience object) — Namespace wrapper exposing all functions as methods.

### 3.2 Queue Management (`queue.ts`)
- `initQueue(path?: string): void` — Initialize queue file and load persisted tasks.
- `queueTask(title, description, options?): QueuedTask` — Create and append new task with generated ID.
- `getQueue(): QueuedTask[]` — Return shallow copy of all queued tasks.
- `getPendingTasks(): QueuedTask[]` — Filter to pending or approved status.
- `getNextTask(hardwareIntensiveAllowed?): QueuedTask | null` — Priority-ordered task selection.
- `approveTask(taskId): QueuedTask | null` — Transition task to approved status.
- `cancelTask(taskId, reason?): QueuedTask | null` — Cancel pending task with optional reason.
- `startTask(taskId): QueuedTask | null` — Mark task as running with timestamp.
- `completeTask(taskId, outcome?): QueuedTask | null` — Finalize task as completed.
- `failTask(taskId, error): QueuedTask | null` — Finalize task as failed with error.
- `getTask(taskId): QueuedTask | null` — Lookup by ID.
- `getQueueStats(): Stats` — Aggregated counts by status and priority.
- `cleanupQueue(olderThanDays?): number` — Remove completed/failed/cancelled tasks older than threshold.
- `persistQueue(): void` — Force-flush queue to disk.
- `queue` (convenience object) — Namespace wrapper with `add`, `get`, `start`, `complete`, `fail`, etc.

### 3.3 Orchestration (`orchestrator.ts`, `goal-orchestrator.ts`)
- `startOrchestratedTask(task, customDeps?): Promise<{ started: boolean; reason?: string }>` — Entry point for closed-loop task execution.
- `validatePrefire(task): PrefireValidation` — Check questionnaire completion; supports legacy and dynamic form modes.
- `truncateLabel(raw): string` — Truncate task labels to 63 characters with ellipsis.
- `synthesizeSuccessCriteria(task): string` — Fallback success criteria from task description/title.
- `runOrchestratorTurn(input): Promise<OrchestratorTurnResult>` — Execute one planning/execution/synthesis turn.
- `tryParsePlanResponse(text): ParsedPlan | null` — Extract tool calls from LLM JSON responses.
- `getRunningOrchestrationsForTests(): string[]` — Test-only introspection of in-flight loops.

### 3.4 Retry Logic (`retry.ts`)
- `retryWithBackoff(fn, opts?): Promise<T>` — Generic retry wrapper with exponential backoff and jitter.
- `isRetryableError(err): boolean` — Classify errors as retryable vs. permanent.
- `planIterationRetry(input): PlanIterationRetryDecision` — Compute retry delay and preamble for failed iterations.
- Constants: `DEFAULT_ITERATION_RETRY_POLICY`, `NON_RETRYABLE_SUBSTRINGS`.

### 3.5 Spawn Parameters (`spawn-params.ts`)
- `validateSpawnParams(input): SpawnParamsValidated` — Pre-call validation with auto-truncation.
- `sanitizeToolResult(s): string` — Strip XML/markdown/CDATA and cap length for safe prompt embedding.
- `SpawnParamsError` — Custom error class with field identification.
- Constants: `SPAWN_LABEL_MAX` (63), `SPAWN_TASK_MAX` (32,000), `SPAWN_TIMEOUT_*` (30–10,800 sec).

### 3.6 Manager & Control (`manager.ts`)
- `initAutonomy(stateDir, customConfig?, telegramConfig?): Promise<void>` — Full subsystem initialization.
- `startHeartbeatLoop(): void` / `stopHeartbeatLoop(): void` — Control heartbeat scheduling.
- `canDoAutonomousWork(): Promise<{ allowed: boolean; reason?: string }>` — Composite gate check (enabled, paused, hardware).
- `doHeartbeatWork(): Promise<HeartbeatResult>` — Synchronous work dispatch for manual invocation.
- `getStatusForUI(): Promise<StatusPayload>` — Comprehensive state snapshot for dashboard consumption.
- `autonomy` (namespace) — Exposes all manager functions plus subsystem accessors.

### 3.7 Subsystem APIs (`notifications.ts`, `hardware.ts`, `pause.ts`, `api-monitor.ts`)
- `notifications.send/notifyInfo/notifySuccess/notifyWarning/notifyError/cageAlert/taskStarted/taskCompleted/taskFailed/iterationStarted/testerVerdict/prefireBlocked/maxIterationsReached/cageInterference/status`
- `hardware.init/check/canRunIntensive/isUserIdle/getIdleTime/getLastState`
- `pause.init/isPaused/getState/activate/deactivate/checkExpiry`
- `apiMonitor.init/record/summary/recent/today/taskUsage/close`

### 3.8 Prefire Form (`prefire-form-agent.ts`, `telegram-prefire.ts`)
- `generatePrefireForm(ctx): Promise<GenerateResult>` — LLM-driven form generation.
- `evaluatePrefireReadiness(form, ctx): Promise<ReadinessResult>` — Structural + LLM readiness evaluation.
- `handlePrefireTelegramCommand(bot, chatId, rawText): Promise<void>` — Telegram bot command handler.

### 3.9 Tester Agent (`tester-agent.ts`)
- `buildTesterPrompt(task, prefire, iteration): string` — Construct tester system prompt.
- `buildBuilderPrompt(task, prefire, iteration, previous?): string` — Construct builder prompt with fix hints from prior verdict.
- `parseTesterVerdict(rawReply): TesterVerdict` — Extract structured verdict from LLM reply.

---

## 4. Test Coverage Snapshot

### 4.1 Modules WITH Test Coverage

| Module | Test File | Coverage Scope |
|--------|-----------|----------------|
| `orchestrator.ts` | `orchestrator.test.ts` | **Extensive** — 7 describe blocks covering: label truncation (≤63 chars), spawn integration with length validation, H1 crash handling, H2 timeout termination, H4 dynamic form validation, H5 heartbeat race prevention, NO-BLOCKAGES self-heal, success criteria synthesis |
| `retry.ts` | `retry.test.ts` | **Comprehensive** — 12 test cases spanning: non-retryable error classification (label validation, billing, auth, model_not_found), retryable errors (ECONNRESET, ETIMEDOUT, rate limits), exponential backoff timing, maxAttempts exhaustion, iteration policy planning |
| `spawn-params.ts` | `spawn-params.test.ts` | **Detailed** — Every validation rule tested: agentId/label/task emptiness, label truncation to SPAWN_LABEL_MAX, task length rejection (>32K), timeout clamping, SpawnParamsError field capture, sanitizeToolResult XML/markdown stripping |
| `goal-orchestrator.ts` | `autonomy.e2e.test.ts` | **Scenario-based** — 6 Hermes-mapped scenarios: happy path, transient ECONNRESET retry, non-retryable spawn-param rejection, malformed JSON repair retry, MAX_PARALLEL cap enforcement (H3), wallclock cap short-circuit |

### 4.2 Modules WITHOUT Test Coverage

| Module | Risk Assessment | Rationale |
|--------|-----------------|-----------|
| `api-monitor.ts` | **HIGH** | SQLite database operations, cost calculations, mirror to credit ledger — no unit tests for query logic or schema migrations |
| `approval-registry.ts` | **MEDIUM** | In-memory Map-based registry with timers; timer-based expiry logic not verified |
| `audit.ts` | **HIGH** | Filesystem I/O for JSONL append operations, SHA256 chain verification, daily log rotation — no mocks or temp directory tests |
| `criticality.ts` | **LOW** | Pure keyword matching with deterministic output; limited test value but should have regression tests |
| `goal-judge.ts` | **HIGH** | External Anthropic API calls with credential resolution (OAuth + API key); fallback logic not exercised |
| `hardware.ts` | **MEDIUM** | Platform-specific shell execution (PowerShell, /proc parsing, ioreg); hard to test but could mock child_process |
| `kanban-sync.ts` | **MEDIUM** | Notion API integration with lazy loading; error handling paths not verified |
| `manager.ts` | **CRITICAL** | Heartbeat scheduler with Interval timers, concurrency flags (`heartbeatTickInFlight`), `doHeartbeatWork` race conditions — most complex module at 388 LOC with zero coverage |
| `notifications.ts` | **MEDIUM** | Telegram fetch API calls with HTML formatting; retry logic for failed notifications not tested |
| `pause.ts` | **LOW** | Simple file I/O; atomicity concerns not tested |
| `prefire-form-agent.ts` | **HIGH** | LLM calls for form generation and readiness evaluation; no mocked pi-ai tests |
| `queue.ts` | **HIGH** | JSON file persistence with concurrent mutation risks; `persistQueue` boundary not tested |
| `telegram-prefire.ts` | **LOW** | Telegram command parsing with grammy types; integration-heavy |
| `types.ts` | **N/A** | Type definitions only — no runtime code to test |

### 4.3 Coverage Statistics

- **Total Source Modules:** 21 (excluding `index.ts` barrel)
- **Modules with Tests:** 4
- **Modules without Tests:** 16
- **Test Coverage Rate:** ~19% (by module count)
- **Critical Gaps:** `manager.ts` (heartbeat scheduler), `audit.ts` (tamper chain), `api-monitor.ts` (cost tracking)

---

## 5. Three Most Important Reliability Concerns

### 5.1 Concern #1: Heartbeat Scheduler Race Conditions and Stranded Tasks (`manager.ts:132-198`)

**Severity:** CRITICAL  
**Location:** `src/autonomy/manager.ts`, lines 132–198 (`runHeartbeatTick` function)

**Description:**  
The heartbeat scheduler implements a non-reentrant guard using `heartbeatTickInFlight` (line 135), but the orchestrator dispatch path (lines 171–192) contains a critical vulnerability: `startOrchestratedTask` is invoked without `await` in a `void` context, and if the orchestrator rejects startup (e.g., prefire incomplete, already running), the error handler at lines 182–191 attempts to call `queue.fail()` which may throw if the queue is corrupted or disk is full. If `queue.fail()` throws, the `finally` block clears `heartbeatTickInFlight` (line 196), but the task remains in "running" status despite never having started, permanently blocking the queue.

Furthermore, the orchestrator path does NOT set the manager-level `currentTask` global variable (unlike `doHeartbeatWork` at line 203), creating a split-brain condition where `alreadyRunning` checks at line 152 read from the queue while heartbeat work at line 203 reads from the global. A race between concurrent heartbeat ticks (despite the flag) and manual `startTask` calls from the UI could permit dual execution despite `maxConcurrentTasks: 1`.

**Evidence:**
```typescript
// manager.ts:171-191
const res = await startOrchestratedTask(task);
if (!res?.started) {
  queue.fail(task.id, res?.reason ?? "orchestrator declined to start"); // May throw
}
```

**Remediation:** Wrap the orchestrator dispatch and queue mutation in a try/catch/finally block that guarantees status reversion on any failure path. Unify the concurrency check to use a single source of truth (queue status only, eliminate `currentTask`).

---

### 5.2 Concern #2: Audit Log Integrity Failures on Concurrent Writes (`audit.ts:88-98`)

**Severity:** HIGH  
**Location:** `src/autonomy/audit.ts`, lines 88–98 (`logAudit` function)

**Description:**  
The tamper-evident audit chain relies on synchronous file appends via `appendFileSync` (line 95), but the implementation lacks file locking or atomic write mechanisms. Under concurrent execution—such as when the heartbeat spawns parallel sub-agents that each log actions, or when multiple gateway workers access the same state directory—interleaved writes could corrupt the JSONL format, breaking the hash chain and rendering `verifyAuditChain` unreliable for forensic analysis.

More critically, the `lastEntryHash` global variable (line 25) is used for chain continuity but is only updated in-memory (line 98). If the process crashes between the `appendFileSync` and the memory update, or if two concurrent `logAudit` calls interleave, the hash chain will contain mismatched `previousHash` references, causing `verifyAuditChain` to report false tampering positives.

**Evidence:**
```typescript
// audit.ts:88-98
appendFileSync(logPath, JSON.stringify(fullEntry) + '\n', 'utf-8'); // Race: interleaved writes possible
lastEntryHash = hash; // Race: stale/hash mismatch if concurrent calls
```

**Remediation:** Implement file-level locking using `proper-lockfile` or similar; batch audit writes through a single writer process; or use SQLite for audit storage with transactional guarantees. Alternatively, append to a WAL (write-ahead log) and compact asynchronously to daily JSONL.

---

### 5.3 Concern #3: Tester Timeout Misclassification as Verdict Failure (`orchestrator.ts:437-470`)

**Severity:** HIGH  
**Location:** `src/autonomy/orchestrator.ts`, lines 437–470 (`runLoop` function)

**Description:**  
When the tester agent wait times out (line 437: `testerResult.status === "timeout"`), the orchestrator synthesizes a synthetic `failVerdict` (lines 440–449) and immediately transitions the task to `currentPhase: "failed"` (line 456), calling `failTask` and sending Telegram failure notifications. However, a timeout from `waitForSubagent` indicates the tester subprocess is **still running** (or the gateway lost track of it), not that the tester actually returned a "fail" verdict.

This conflates infrastructure latency/tester slowness with actual test failure, causing tasks to be marked failed when they might have passed if given more time. Worse, if the tester eventually completes after the timeout, its orphaned output may contaminate the next task's environment or leave zombie processes. The H6 fix (lines 444–448) adds an explanatory error message but does not address the premature failure classification.

**Evidence:**
```typescript
// orchestrator.ts:437-456
if (testerResult.status === "timeout" || testerResult.status === "error") {
  const failVerdict: TesterVerdict = {  // Synthetic failure on timeout
    pass: false,
    summary: `Tester ${reasonNoun}: ${reasonDetail}`,
    // ...
  };
  iteration.testerVerdict = failVerdict;
  task.currentPhase = "failed"; // Premature termination
  deps.failTask(task.id, /* ... */); // Marks task failed
}
```

**Remediation:** Distinguish between "tester timed out" (infrastructure) and "tester returned fail" (legitimate verdict). For timeouts, either retry the await with extended timeout (up to a limit) or transition to a "suspended" state requiring operator intervention, rather than immediate failure. Implement subprocess reaping to cancel orphaned testers when the orchestrator gives up on them.

---

## 6. Conclusion

The autonomy module demonstrates mature engineering with clear separation of concerns, extensive defensive coding (retry logic, parameter validation, self-healing fallbacks), and conscientious safety mechanisms (audit chains, approval gates, pause controls). However, the test coverage gap—particularly for the heartbeat scheduler and audit persistence—creates blind spots where race conditions and file I/O edge cases could compromise reliability in production.

The three cited concerns represent the highest-risk intersections of async execution, shared mutable state, and external system dependencies. Addressing these through file locking, unified concurrency control, and improved timeout handling would significantly harden the module for unattended autonomous operation at scale.

---

**End of Report**

*Generated by Autonomous Build — Iteration 1*  
*Report Location: docs/autonomy-e2e-validation-report-2026-05-28.md*
