# Agentic Orchestrator Execution Mode Specification

**Status:** Completed  
**Created:** 2026-03-05  
**Author:** System  
**Scope:** Historical architectural refactor for worker execution mode support. `deterministic` worker execution is now deprecated; this document is retained as migration/reference context, while new implementation should standardize on interactive execution only.

---

## 0. Executive Summary

### 0.1 Problem Statement

The current architecture enforces a **deterministic, API-first execution model** where agents:

- Receive context bundles via tool calls (`feature.get_context`, `repo.read_file`, `repo.search`)
- Return structured patches via `repo.apply_patch` tool calls
- Cannot directly modify files in the worktree
- Must isolate all changes in their context window

This approach guarantees determinism and auditability but **handicaps modern coding agents** (Claude, Codex, Copilot, Kiro-CLI) that work best when they can:

- Directly edit files in their working directory
- Use native file system operations
- Iterate rapidly without context window constraints
- Leverage their built-in file editing capabilities

### 0.2 Proposed Solution

Standardize the orchestrator on **interactive worker execution**:

1. **Interactive Mode (Direct worktree access)** - Supported behavior
   - Agent works directly in feature worktree (or shadow workspace)
   - Agent modifies files using native file operations
   - Supervisor monitors worktree with file system watchdog
   - Periodic validation checkpoints replace per-patch validation
   - Supervisor captures diffs at checkpoints for audit trail
   - **Two strategies:** Direct worktree (simple) or Shadow workspace (safe)

2. **Deterministic Mode (API-first)** - Deprecated historical context
   - Retained here only to explain the legacy API-first design and migration path
   - New product and implementation specs should not present it as a normal operator-facing configuration

### 0.3 Key Design Principles

- **Interactive-first:** interactive worker execution is the intended operating model for new work
- **Preserve determinism guarantees:** Both modes enforce plan/policy/lock constraints
- **Migration-aware:** legacy deterministic references remain only where needed to explain compatibility or cleanup
- **Provider-agnostic:** Mode selection independent of provider (Claude, Codex, etc.)
- **Audit trail maintained:** Both modes produce complete change history
- **Safety first:** Shadow workspace strategy available for high-risk features
- **Graceful degradation:** Circuit breakers, fallbacks, and health checks throughout
- **Performance budgets:** Explicit latency/throughput targets with degradation strategies

### 0.4 User Benefits

**For API-first providers (Codex API, Claude API):**

- Continue using deterministic mode
- No changes required

**For interactive CLI providers (Kiro-CLI, local Claude, Copilot):**

- Enable interactive mode
- Agent works naturally in worktree
- Higher quality output (agent uses native capabilities)
- Faster iteration (no context window bottleneck)
- Choose safety level (direct worktree vs shadow workspace)

### 0.5 Critical Design Improvements (Post-Review)

This spec has been enhanced with critical mitigations for production readiness:

1. **Race condition handling:** Agent pause/resume protocol with acknowledgment
2. **Validation timing:** Shadow workspace strategy for validation-before-write
3. **Rollback strategies:** Full, partial, and smart rollback capabilities
4. **Security hardening:** Path validation, symlink blocking, disk quotas, `.git` protection
5. **Performance budgets:** Explicit latency targets with timeout and degradation strategies
6. **Graceful degradation:** Circuit breakers, health checks, automatic fallback to deterministic mode
7. **Audit trail completeness:** Continuous event log, checkpoint chains, incremental + cumulative diffs
8. **Concurrency control:** Per-feature service instances, validation queues, backpressure
9. **Agent communication:** Structured message protocol with acknowledgment and retry
10. **Monitoring and alerts:** Comprehensive metrics and alert thresholds

---

## 1. Current Architecture Analysis

### 1.1 Current Execution Flow

```
SupervisorRuntime
  └─> WorkerDecisionLoop
      └─> WorkerProvider.runWorker(context_bundle, instructions)
          └─> Agent receives:
              - feature.get_context (spec, state, plan, evidence)
              - repo.read_file (specific files)
              - repo.search (grep results)
          └─> Agent returns:
              - tool_calls: [{ name: "repo.apply_patch", args: { unified_diff: "..." } }]
          └─> Supervisor validates patch:
              - Plan enforcement (allowed_areas, forbidden_areas)
              - Policy enforcement (patch_policy.enforce_allowed_areas)
              - Lock enforcement (contracts require held locks)
              - Path rules (protected_areas)
          └─> Supervisor applies patch:
              - git apply in feature worktree
              - Update QA index
              - Update feature state
```

### 1.2 Key Components

**WorkerProvider Interface** (`apps/control-plane/src/providers/providers.ts`)

- `runWorker(input)` - Executes agent with context bundle
- Returns structured output with `tool_calls` array
- Implementations: `CliWorkerProvider`, `ApiWorkerProvider`, `NullWorkerProvider`

**PatchService** (`apps/control-plane/src/application/services/patch-service.ts`)

- `repoApplyPatch(featureId, unifiedDiff)` - Validates and applies patches
- Enforces plan, policy, locks, path rules
- Updates QA index after successful apply

**WorkerDecisionLoop** (`apps/control-plane/src/supervisor/worker-decision-loop.ts`)

- Orchestrates agent execution cycles
- Handles tool call dispatch
- Detects stalls, loops, no-progress conditions

**ToolRuntime** (`apps/control-plane/src/mcp/tool-runtime.ts`)

- Validates tool inputs/outputs against schemas
- Enforces RBAC (role-based access control)
- Tracks operation_id for idempotency

### 1.3 Constraints Enforced

**Plan Constraints:**

- `allowed_areas` - Paths agent may modify
- `forbidden_areas` - Paths agent must not touch
- `contracts` - Locks required (openapi, events, db)

**Policy Constraints:**

- `patch_policy.enforce_plan` - Require accepted plan before patches
- `patch_policy.enforce_allowed_areas` - Validate patch paths against plan
- `protected_areas` - Global read-only paths
- `path_rules` - Per-path RBAC rules

**Lock Constraints:**

- Features must hold locks for contract resources before modifying them
- Lock leases expire after TTL (default 300s)
- Heartbeat service renews leases during execution

---

## 2. Proposed Architecture

### 2.1 Execution Mode Enum

> **Deprecation note:** the `deterministic` enum member is legacy compatibility only. New behavior should resolve to `interactive`, and user-facing configuration should stop advertising deterministic worker mode as a normal choice.

**New type:** `ExecutionMode`

```typescript
type ExecutionMode = 'deterministic' | 'interactive';
```

**Configuration location:** `agents.yaml`

```yaml
runtime:
  execution_mode: interactive
  interactive:
    strategy: direct_worktree # or 'shadow_workspace'
    watchdog_poll_interval_ms: 2000
    checkpoint_interval_ms: 30000 # Hybrid trigger: time-based
    max_uncommitted_changes: 50 # Hybrid trigger: change-based
    validation_on_checkpoint: true
    revert_on_violation: false
    violation_severity: warning # info | warning | error | critical
    shadow_workspace:
      enabled: false # Enable shadow workspace strategy
      promotion_strategy: atomic # atomic | incremental
      cleanup_on_failure: true # Delete shadow after validation failure
      max_shadow_size_mb: 2048 # Disk quota for shadow workspace
```

**CLI override:** historical document value only; new UX should stop advertising `deterministic` as a selectable worker mode.

### 2.2 Mode Selection Precedence

> **Historical note:** this precedence ladder reflects the original dual-mode design. Going forward, runtime policy should converge on `interactive` by default and remove user-facing deterministic selection.

1. CLI flag `--execution-mode`
2. Feature-level override in `state.md` frontmatter (`execution_mode`)
3. `agents.yaml runtime.execution_mode`
4. Default: `deterministic`

### 2.3 Deterministic Mode (Historical Behavior)

**Deprecated.** This section is retained only to explain the legacy API-first design and migration path.

**Flow:**

1. Agent receives context via tool calls
2. Agent returns `repo.apply_patch` tool calls
3. Supervisor validates patch before applying
4. Supervisor applies patch with `git apply`
5. Supervisor updates QA index and state

**Guarantees:**

- Every change validated before application
- Complete audit trail in operation ledger
- Atomic patch application
- Immediate policy/plan enforcement

### 2.4 Interactive Mode (New Behavior)

**Two implementation strategies:**

#### Strategy A: Direct Worktree with Checkpoints (Proposed)

**Flow:**

1. Supervisor spawns agent with `cwd` set to feature worktree
2. Agent works directly in worktree (native file operations)
3. Supervisor monitors worktree with file system watchdog
4. Watchdog detects file changes in real-time
5. At checkpoints (time-based or change-count-based):
   - Supervisor pauses agent
   - Supervisor captures `git diff` snapshot
   - Supervisor validates diff against plan/policy/locks
   - If validation fails: notify agent, optionally revert changes
   - If validation passes: record checkpoint in audit log
   - Supervisor resumes agent
6. On agent completion:
   - Final checkpoint validation
   - Commit changes if valid
   - Update QA index and state

**Pros:**

- Agent works in real worktree (no sync overhead)
- Simple implementation
- Fast iteration for agent

**Cons:**

- Validation happens after writes (risky)
- Revert is destructive
- Race conditions during checkpoint
- Agent could corrupt worktree before validation

#### Strategy B: Shadow Workspace with Promotion (Alternative - RECOMMENDED)

**See detailed specification:** [Shadow Workspace Implementation Specification](../outstanding/shadow_workspace_implementation_spec.md)

**Flow:**

1. Supervisor creates shadow workspace (copy of feature worktree)
2. Supervisor spawns agent with `cwd` set to shadow workspace
3. Agent works in shadow (isolated from real worktree)
4. Supervisor monitors shadow with file system watchdog
5. At checkpoints:
   - Supervisor pauses agent
   - Supervisor captures diff between real worktree and shadow
   - Supervisor validates diff
   - If validation passes: promote shadow changes to real worktree (atomic)
   - If validation fails: discard shadow changes, notify agent
   - Supervisor resumes agent in fresh shadow
6. On agent completion:
   - Final validation
   - Promote shadow to real worktree if valid
   - Commit changes
   - Update QA index and state

**Pros:**

- Validation before promotion (safe)
- Real worktree never corrupted
- Easy rollback (just discard shadow)
- No race conditions (shadow is isolated)

**Cons:**

- Disk space overhead (2x worktree size)
- Promotion overhead (copy operation)
- More complex implementation
- Agent must restart in fresh shadow after revert

**Recommendation:** Start with Strategy A (simpler), migrate to Strategy B if validation-after-write proves problematic in practice.

**Hybrid approach:** Use Strategy A by default, Strategy B for high-risk features (e.g., features modifying contracts or schemas).

**Key Differences from Deterministic Mode:**

- Agent has direct file system access
- Validation happens at checkpoints, not per-patch
- Supervisor is reactive (watchdog) instead of proactive (gate-keeping)
- Agent can iterate freely between checkpoints

**Guarantees:**

- Plan/policy/lock constraints enforced at checkpoints
- Audit trail captured at checkpoints (not per-change)
- Rollback capability if validation fails
- Final state is deterministic (validated before commit)

---

## 3. Implementation Plan

### 3.1 Schema Changes

**File:** `agentic/orchestrator/schemas/agents.schema.json`

Add to `runtime` properties:

```json
{
  "execution_mode": {
    "type": "string",
    "enum": ["deterministic", "interactive"],
    "default": "deterministic",
    "description": "Execution mode for agent workers"
  },
  "interactive": {
    "type": "object",
    "description": "Configuration for interactive execution mode",
    "properties": {
      "watchdog_poll_interval_ms": {
        "type": "number",
        "default": 2000,
        "description": "File system watchdog polling interval"
      },
      "checkpoint_interval_ms": {
        "type": "number",
        "default": 30000,
        "description": "Time between validation checkpoints (hybrid trigger)"
      },
      "max_uncommitted_changes": {
        "type": "number",
        "default": 50,
        "description": "Trigger checkpoint after N file changes (hybrid trigger)"
      },
      "validation_on_checkpoint": {
        "type": "boolean",
        "default": true,
        "description": "Validate diffs at checkpoints"
      },
      "revert_on_violation": {
        "type": "boolean",
        "default": false,
        "description": "Auto-revert changes that violate plan/policy"
      },
      "violation_severity": {
        "type": "string",
        "enum": ["info", "warning", "error", "critical"],
        "default": "warning",
        "description": "Severity level for violation notifications sent to agent"
      }
    }
  }
}
```

**File:** `agentic/orchestrator/schemas/state.schema.json`

Add to properties:

```json
{
  "execution_mode": {
    "type": "string",
    "enum": ["deterministic", "interactive"],
    "description": "Execution mode override for this feature (optional)"
  },
  "checkpoints": {
    "type": "array",
    "description": "Validation checkpoints recorded during interactive mode",
    "items": {
      "type": "object",
      "required": ["checkpoint_id", "timestamp", "files_changed", "validation_status"],
      "properties": {
        "checkpoint_id": { "type": "string" },
        "timestamp": { "type": "string", "format": "date-time" },
        "files_changed": { "type": "array", "items": { "type": "string" } },
        "validation_status": { "type": "string", "enum": ["valid", "invalid", "skipped"] },
        "violations": { "type": "array", "items": { "type": "string" } },
        "severity": { "type": "string", "enum": ["info", "warning", "error", "critical"] },
        "diff_snapshot": { "type": "string", "description": "Path to diff file" }
      }
    }
  }
}
```

### 3.2 New Services

**File:** `apps/control-plane/src/application/services/worktree-watchdog-service.ts`

```typescript
interface WorktreeWatchdogService {
  startWatching(featureId: string): Promise<void>;
  stopWatching(featureId: string): Promise<void>;
  getChangedFiles(featureId: string): Promise<string[]>;
  getChangeCount(featureId: string): number;
  resetChangeCount(featureId: string): void;
  on(event: 'changeThreshold', handler: (featureId: string) => void): void;
}
```

**Responsibilities:**

- Monitor feature worktree for file changes
- Track changed file paths
- Emit `changeThreshold` event when `max_uncommitted_changes` reached
- Use `chokidar` for cross-platform file system monitoring
- Support concurrent monitoring of multiple feature worktrees

**File:** `apps/control-plane/src/application/services/checkpoint-service.ts`

```typescript
interface CheckpointService {
  createCheckpoint(featureId: string): Promise<CheckpointResult>;
  validateCheckpoint(featureId: string, diff: string): Promise<ValidationResult>;
  recordCheckpoint(featureId: string, checkpoint: Checkpoint): Promise<void>;
  getCheckpoints(featureId: string): Promise<Checkpoint[]>;
  notifyAgent(featureId: string, violations: string[], severity: ViolationSeverity): Promise<void>;
  rollbackToCheckpoint(featureId: string, checkpointId: string): Promise<void>;
  rollbackFiles(featureId: string, filePaths: string[]): Promise<void>;
}

type ViolationSeverity = 'info' | 'warning' | 'error' | 'critical';

interface CheckpointResult {
  checkpoint_id: string;
  timestamp: string;
  files_changed: string[];
  diff_snapshot: string;
  validation_status: 'valid' | 'invalid' | 'skipped';
  violations: string[];
  severity?: ViolationSeverity;
}
```

**Responsibilities:**

- Capture `git diff` snapshots at checkpoints (triggered by time OR change count)
- Validate diffs against plan/policy/locks (reuse PatchService validation logic)
- Record checkpoint metadata in feature state
- Store diff snapshots in `.aop/features/<id>/checkpoints/`
- Notify agent of violations with configurable severity level
- Optionally revert changes if `revert_on_violation: true`
- **Rollback to specific checkpoint (full or partial)**
- **Rollback specific files only (surgical revert)**

**Rollback strategies:**

1. **Full checkpoint rollback:** Restore worktree to exact state at checkpoint

   ```typescript
   async rollbackToCheckpoint(featureId: string, checkpointId: string): Promise<void> {
     const checkpoint = await this.getCheckpoint(featureId, checkpointId);
     const worktreePath = this.kernel.worktreePath(featureId);

     // Apply inverse diff to restore state
     const inverseDiff = this.invertDiff(checkpoint.diff_snapshot);
     await this.git.apply(worktreePath, inverseDiff);

     // Update state to reflect rollback
     await this.kernel.updateFeatureState(featureId, {
       last_checkpoint_id: checkpointId,
       rollback_count: (state.rollback_count || 0) + 1,
     });
   }
   ```

2. **Partial file rollback:** Revert only specific files that violated constraints

   ```typescript
   async rollbackFiles(featureId: string, filePaths: string[]): Promise<void> {
     const worktreePath = this.kernel.worktreePath(featureId);

     // Get last valid checkpoint
     const lastValidCheckpoint = await this.getLastValidCheckpoint(featureId);

     for (const filePath of filePaths) {
       // Extract file content from checkpoint diff
       const fileContent = this.extractFileFromDiff(
         lastValidCheckpoint.diff_snapshot,
         filePath
       );

       // Restore file
       await fs.writeFile(path.join(worktreePath, filePath), fileContent);
     }

     // Notify agent which files were reverted
     await this.notifyAgent(featureId, [
       `Reverted ${filePaths.length} files: ${filePaths.join(', ')}`
     ], 'warning');
   }
   ```

3. **Smart rollback:** Keep valid changes, revert only violations

   ```typescript
   async smartRollback(featureId: string, violations: ValidationViolation[]): Promise<void> {
     // Group violations by file
     const violatedFiles = new Set(violations.map(v => v.file_path));

     // Get all changed files
     const allChangedFiles = await this.watchdog.getChangedFiles(featureId);

     // Rollback only violated files, keep valid changes
     await this.rollbackFiles(featureId, Array.from(violatedFiles));

     // Notify agent
     await this.notifyAgent(featureId, [
       `Reverted ${violatedFiles.size} files with violations`,
       `Kept ${allChangedFiles.length - violatedFiles.size} valid changes`
     ], 'warning');
   }
   ```

**Notification message format:**

```json
{
  "type": "checkpoint_violation",
  "severity": "warning",
  "checkpoint_id": "ckpt-002-b7e4c1d9",
  "timestamp": "2026-03-05T16:29:05.127Z",
  "violations": [
    {
      "file": "src/config.ts",
      "constraint": "allowed_areas",
      "message": "Path 'src/config.ts' not in allowed_areas",
      "suggestion": "Add 'src/config.ts' to plan.allowed_areas or move changes to allowed path"
    }
  ],
  "files_changed": ["src/config.ts"],
  "action_taken": "none" // or "reverted" if revert_on_violation: true
}
```

### 3.3 Modified Services

**File:** `apps/control-plane/src/supervisor/worker-decision-loop.ts`

**Changes:**

- Add `executionMode` parameter to constructor
- Branch execution logic based on mode:
  - **Deterministic:** Current behavior (tool call loop)
  - **Interactive:** New behavior (spawn agent, start watchdog, checkpoint loop)

**New method:** `runInteractiveWorker(input: WorkerDecisionInput)`

```typescript
async runInteractiveWorker(input: WorkerDecisionInput): Promise<WorkerDecisionResult> {
  const { featureId, role, instructions } = input;

  // 1. Start watchdog
  await this.watchdog.startWatching(featureId);

  // 2. Spawn agent with cwd = worktree
  const worktreePath = this.kernel.worktreePath(featureId);
  const session = await this.provider.createSession(role, featureId, systemPrompt);

  // 3. Start checkpoint loop (hybrid: time-based OR change-based)
  const checkpointTimer = setInterval(async () => {
    await this.maybeCreateCheckpoint(featureId);
  }, this.checkpointIntervalMs);

  // Also trigger on change threshold
  this.watchdog.on('changeThreshold', async (fid) => {
    if (fid === featureId) {
      await this.maybeCreateCheckpoint(featureId);
    }
  });

  // 4. Run agent (non-blocking, agent works in worktree)
  const result = await this.provider.runWorker({
    role,
    feature_id: featureId,
    working_directory: worktreePath,
    instructions,
    execution_mode: 'interactive',
  });

  // 5. Stop checkpoint loop
  clearInterval(checkpointTimer);

  // 6. Final checkpoint
  await this.checkpointService.createCheckpoint(featureId);

  // 7. Stop watchdog
  await this.watchdog.stopWatching(featureId);

  return result;
}
```

**File:** `apps/control-plane/src/application/services/patch-service.ts`

**Changes:**

- Extract validation logic into reusable method: `validateDiff(featureId, parsedDiff)`
- Used by both `repoApplyPatch` (deterministic) and `CheckpointService.validateCheckpoint` (interactive)

**New method:** `validateDiff(featureId: string, parsedDiff: ParsedDiff): Promise<ValidationResult>`

```typescript
async validateDiff(featureId: string, parsedDiff: ParsedDiff): Promise<ValidationResult> {
  const plan = await this.loadAcceptedPlan(featureId);

  // Validate paths against plan
  await this.validatePatchPaths(featureId, parsedDiff, plan);

  // Validate locks held
  await this.assertPlanLocksHeld(featureId, plan);

  return { valid: true, violations: [] };
}
```

### 3.4 Provider Interface Changes

**File:** `apps/control-plane/src/providers/providers.ts`

**Extend `WorkerProvider` interface:**

```typescript
interface WorkerProvider {
  // Existing methods
  runWorker(input: WorkerRunInput): Promise<WorkerRunOutput>;
  createSession(role: string, featureId: string, systemPrompt: string): Promise<SessionInfo>;

  // NEW: Interactive mode support
  pauseAgent(featureId: string, timeout?: number): Promise<PauseAckResult>;
  resumeAgent(featureId: string): Promise<void>;
  sendMessage(featureId: string, message: AgentMessage): Promise<MessageAckResult>;
  getCapabilities(): WorkerProviderCapabilities;
}

interface PauseAckResult {
  acknowledged: boolean;
  timeout: boolean;
  latency_ms: number;
}

interface MessageAckResult {
  delivered: boolean;
  acknowledged: boolean;
  agent_response?: string;
}

interface AgentMessage {
  type: 'checkpoint_violation' | 'info' | 'warning' | 'error';
  severity: ViolationSeverity;
  content: string;
  structured_data?: Record<string, unknown>;
  requires_acknowledgment: boolean;
}

interface WorkerProviderCapabilities {
  supportsInteractiveMode: boolean;
  supportsWorkingDirectory: boolean;
  supportsPauseResume: boolean;
  supportsMessagePassing: boolean;
  supportsAcknowledgment: boolean;
}
```

**Extend `WorkerRunInput`:**

```typescript
interface WorkerRunInput {
  role: string;
  feature_id: string;
  context_bundle?: Record<string, unknown>;
  instructions?: string;
  last_tool_results?: Array<Record<string, unknown>>;
  runtime_selection?: {
    provider: string;
    model: string;
    provider_config_ref: string | null;
  };
  // NEW: Interactive mode fields
  execution_mode?: 'deterministic' | 'interactive';
  working_directory?: string; // Set to worktree path in interactive mode
  pause_resume_protocol?: 'signal' | 'message' | 'none'; // How to pause agent
}
```

**Provider implementations:**

- **CliWorkerProvider:**
  - Set `cwd` to `working_directory` when spawning agent process
  - Implement `pauseAgent` via SIGSTOP/SIGCONT or stdin message
  - Implement `sendMessage` via stdin JSON messages
  - Implement `getCapabilities` returning full support
- **ApiWorkerProvider:**
  - Include `working_directory` in context (API may not support it)
  - `pauseAgent` not supported (return `{ acknowledged: false, timeout: true }`)
  - `sendMessage` via API message endpoint (if available)
  - `getCapabilities` returning limited support
- **NullWorkerProvider:**
  - No changes (stub)
  - `getCapabilities` returning no support

**Agent-side protocol (for CLI agents):**

Agents must implement stdin message handling:

```json
// Supervisor -> Agent: Pause request
{
  "type": "pause_request",
  "checkpoint_id": "ckpt-003-xyz",
  "reason": "checkpoint_validation",
  "timeout_ms": 10000
}

// Agent -> Supervisor: Pause acknowledgment
{
  "type": "pause_ack",
  "checkpoint_id": "ckpt-003-xyz",
  "status": "paused",
  "pending_operations": []
}

// Supervisor -> Agent: Resume
{
  "type": "resume",
  "checkpoint_id": "ckpt-003-xyz"
}

// Supervisor -> Agent: Violation notification
{
  "type": "checkpoint_violation",
  "severity": "warning",
  "checkpoint_id": "ckpt-003-xyz",
  "violations": [
    {
      "file": "src/config.ts",
      "constraint": "allowed_areas",
      "message": "Path not in allowed_areas",
      "suggestion": "Add to plan.allowed_areas or move to allowed path"
    }
  ],
  "action_taken": "none",
  "requires_acknowledgment": true
}

// Agent -> Supervisor: Violation acknowledgment
{
  "type": "violation_ack",
  "checkpoint_id": "ckpt-003-xyz",
  "understood": true,
  "corrective_action": "will_move_file_to_allowed_path"
}
```

### 3.5 Supervisor Integration

**File:** `apps/control-plane/src/supervisor/runtime.ts`

**Changes:**

- Resolve `execution_mode` from config/CLI/state
- Pass `execution_mode` to `WorkerDecisionLoop`
- Pass `interactive` config to `CheckpointService`

**New method:** `resolveExecutionMode(featureId: string): ExecutionMode`

```typescript
private resolveExecutionMode(featureId: string): ExecutionMode {
  // 1. Check CLI override
  if (this.cliExecutionMode) {
    return this.cliExecutionMode;
  }

  // 2. Check feature state override
  const state = await this.kernel.readState(featureId);
  if (state.frontMatter.execution_mode) {
    return state.frontMatter.execution_mode;
  }

  // 3. Check agents.yaml
  const agentsConfig = this.kernel.getAgentsConfig();
  if (agentsConfig.runtime?.execution_mode) {
    return agentsConfig.runtime.execution_mode;
  }

  // 4. Default
  return 'deterministic';
}
```

### 3.6 CLI Changes

**File:** `apps/control-plane/src/cli/types.ts`

Add to `CliArgs`:

```typescript
interface CliArgs {
  // ... existing fields
  execution_mode?: 'deterministic' | 'interactive';
}
```

**File:** `apps/control-plane/src/cli/cli-argument-parser.ts`

Parse `--execution-mode` flag.

**File:** `apps/control-plane/src/cli/help-command-handler.ts`

Add to `run` command help:

```
--execution-mode <deterministic|interactive>
  Execution mode for agent workers (default: deterministic)
```

---

## 4. Validation and Audit Trail

### 4.1 Deterministic Mode Audit Trail

**Current behavior (unchanged):**

- Every `repo.apply_patch` call logged in operation ledger
- Full unified diff captured per operation
- Validation errors logged with rejection reason
- Complete history in `.aop/runtime/operation-ledger/run:<run_id>.json`

### 4.2 Interactive Mode Audit Trail

**New behavior:**

- Checkpoints logged in feature state (`checkpoints` array)
- Diff snapshots stored in `.aop/features/<id>/checkpoints/<checkpoint_id>.diff`
- Validation results recorded per checkpoint
- Final diff captured on agent completion
- Checkpoint metadata includes:
  - `checkpoint_id` (UUID)
  - `timestamp` (ISO 8601)
  - `files_changed` (array of paths)
  - `validation_status` (valid/invalid/skipped)
  - `violations` (array of error messages)
  - `severity` (info/warning/error/critical)
  - `diff_snapshot` (path to diff file)

**Audit trail location:**

```
.aop/features/<feature_id>/
  ├── state.md (includes checkpoints array)
  └── checkpoints/
      ├── checkpoint-001.diff
      ├── checkpoint-002.diff
      └── checkpoint-003.diff
```

### 4.3 Validation Enforcement

**Both modes enforce:**

- Plan constraints (allowed_areas, forbidden_areas)
- Policy constraints (patch_policy, protected_areas)
- Lock constraints (contracts require held locks)
- Path rules (RBAC per path)

**Difference:**

- **Deterministic:** Validation before application (gate-keeping)
- **Interactive:** Validation at checkpoints (monitoring)

**Interactive mode violation handling:**

1. **Notify agent:** Send message via `WorkerProvider.sendMessage` with:
   - Violation details (which files, which constraints violated)
   - Severity level (info/warning/error/critical) from `violation_severity` config
   - Suggested remediation actions
2. **Optional revert:** If `revert_on_violation: true`, run `git checkout -- <files>` to undo changes
3. **Block merge:** Feature cannot reach `ready_to_merge` if any checkpoint has `validation_status: invalid` with severity >= `error`

**Severity level behavior:**

- `info`: Log only, no blocking
- `warning`: Notify agent, no blocking (default)
- `error`: Notify agent, block merge
- `critical`: Notify agent, block merge, optionally revert

---

## 5. Migration Path

### 5.1 Phase 1: Schema and Config (Week 1)

- Add `execution_mode` to `agents.schema.json`
- Add `interactive` config to `agents.schema.json`
- Add `execution_mode` and `checkpoints` to `state.schema.json`
- Update `agents.yaml` with default `execution_mode: deterministic`
- Add CLI flag `--execution-mode`

### 5.2 Phase 2: Watchdog and Checkpoint Services (Week 2)

- Implement `WorktreeWatchdogService`
- Implement `CheckpointService`
- Extract `PatchService.validateDiff` for reuse
- Add unit tests (>= 90% coverage)

### 5.3 Phase 3: Interactive Mode Execution (Week 3)

- Add `runInteractiveWorker` to `WorkerDecisionLoop`
- Update `SupervisorRuntime.resolveExecutionMode`
- Update provider implementations to support `working_directory`
- Add integration tests

### 5.4 Phase 4: Audit Trail and Dashboard (Week 4)

- Store checkpoint diffs in `.aop/features/<id>/checkpoints/`
- Update dashboard to display checkpoints (new tab in RuntimeInspector)
- Add checkpoint timeline view
- Add validation status indicators

### 5.5 Phase 5: Documentation and Rollout (Week 5)

- Update README with execution mode documentation
- Update AGENTS.md and CLAUDE.md
- Add example configurations for interactive mode
- Update `aop init` wizard to ask about execution mode preference

---

## 6. Testing Strategy

### 6.1 Unit Tests

**WorktreeWatchdogService:**

- Detects file creation
- Detects file modification
- Detects file deletion
- Tracks change count correctly
- Resets change count on demand

**CheckpointService:**

- Creates checkpoint with valid diff
- Validates diff against plan
- Records checkpoint in state
- Stores diff snapshot in correct location
- Handles validation failures

**PatchService.validateDiff:**

- Rejects paths outside allowed_areas
- Rejects paths in forbidden_areas
- Requires locks for contract modifications
- Enforces protected_areas

### 6.2 Integration Tests

**Deterministic Mode (Regression):**

- Existing tests continue to pass
- No behavior changes

**Interactive Mode:**

- Agent spawns with correct working directory
- Watchdog detects agent file changes
- Checkpoints created at intervals
- Validation runs at checkpoints
- Violations notify agent
- Final checkpoint validates before commit
- Audit trail captured correctly

### 6.3 End-to-End Tests

**Scenario 1: Interactive mode with valid changes**

1. Start feature with `--execution-mode interactive`
2. Agent modifies files in worktree
3. Checkpoints validate successfully
4. Feature completes with all changes committed

**Scenario 2: Interactive mode with policy violation**

1. Start feature with `--execution-mode interactive`
2. Agent modifies file outside allowed_areas
3. Checkpoint validation fails
4. Agent receives violation notification
5. Agent corrects violation
6. Next checkpoint validates successfully

**Scenario 3: Mode switching**

1. Start feature in deterministic mode
2. Update state with `execution_mode: interactive`
3. Resume feature
4. Execution switches to interactive mode

---

## 7. Acceptance Criteria

### 7.1 Must Have

- [x] Schema changes for `execution_mode` and `checkpoints`
- [x] CLI flag `--execution-mode` implemented
- [x] `WorktreeWatchdogService` implemented
- [x] `CheckpointService` implemented
- [x] `PatchService.validateDiff` extracted and reused
- [x] `WorkerDecisionLoop.runInteractiveWorker` implemented
- [x] `SupervisorRuntime.resolveExecutionMode` implemented
- [x] Provider interface supports `working_directory`
- [x] Checkpoint diffs stored in `.aop/features/<id>/checkpoints/`
- [x] Validation enforced at checkpoints
- [x] Audit trail complete for both modes
- [x] All existing tests pass (deterministic mode regression)
- [x] New tests for interactive mode (>= 90% coverage)
- [x] TypeScript strict mode passes
- [x] ESLint zero warnings

### 7.2 Should Have

- [x] Dashboard displays checkpoints in RuntimeInspector
- [x] Checkpoint timeline view
- [x] Validation status indicators
- [x] Agent notification on violation
- [x] Optional auto-revert on violation
- [x] Documentation in README
- [x] Example configurations

### 7.3 Nice to Have

- [x] Real-time checkpoint streaming to dashboard
- [x] Diff viewer for checkpoint snapshots
- [x] Checkpoint comparison (diff between checkpoints)
- [x] Checkpoint rollback command (`aop rollback --checkpoint <id>`)
- [x] Interactive mode performance metrics

---

## 8. Open Questions

### 8.1 Checkpoint Frequency ✅ RESOLVED

**Question:** What is the optimal checkpoint interval?

**Options:**

- Time-based: Every 30 seconds (default)
- Change-based: Every 50 file changes (default)
- Hybrid: Whichever comes first

**Decision:** Hybrid approach with configurable thresholds.

- `checkpoint_interval_ms: 30000` (time trigger)
- `max_uncommitted_changes: 50` (change trigger)
- Checkpoint created when either threshold is reached

### 8.2 Violation Handling ✅ RESOLVED

**Question:** Should violations auto-revert or just notify?

**Options:**

- **Notify only:** Agent receives message, decides how to fix
- **Auto-revert:** Supervisor reverts changes, agent starts fresh
- **Configurable:** User chooses via `revert_on_violation` flag

**Decision:** Configurable with default `false` (notify only).

**Severity levels added:**

- `violation_severity: info | warning | error | critical`
- Default: `warning`
- Behavior varies by severity (see Section 4.3)

### 8.3 Provider Compatibility ✅ RESOLVED

**Question:** Which providers support interactive mode?

**Analysis:**

- **CliWorkerProvider:** Full support (can set `cwd`)
- **ApiWorkerProvider:** Limited support (API may not honor `working_directory`)
- **NullWorkerProvider:** N/A (stub)

**Decision:** Use recommendation - document provider compatibility in README. Warn users if provider doesn't support interactive mode.

### 8.4 Concurrent Checkpoints ✅ RESOLVED

**Question:** Can multiple features run in interactive mode simultaneously?

**Decision:** Yes, confirmed understanding is correct.

- Each feature has isolated worktree and watchdog instance
- Watchdog service maintains map of `featureId -> WatcherInstance`
- Checkpoint service handles concurrent validation requests
- File system watchdog overhead scales linearly with active features
- Monitor performance with `max_active_features > 5`

---

## 9. Risks and Mitigations

### 9.1 Risk: Race Conditions During Checkpoint Validation

**Problem:** Agent continues writing while checkpoint validation runs, causing:

- Validation of incomplete/inconsistent state
- File corruption if revert happens mid-write
- Duplicate checkpoints if change threshold triggers during time-based checkpoint

**Mitigation:**

- **Agent pause protocol:** Send `PAUSE` signal to agent before checkpoint, wait for `ACK`, then validate
- **Debounce checkpoint triggers:** Minimum 5s between checkpoints regardless of trigger source
- **File write detection:** Use `chokidar` `awaitWriteFinish` option (waits for file size to stabilize)
- **Checkpoint queue:** Serialize checkpoint operations per feature (no concurrent checkpoints)
- **Timeout:** If agent doesn't ACK pause within 10s, force checkpoint anyway and log warning

**Implementation:**

```typescript
interface WorkerProvider {
  pauseAgent(featureId: string): Promise<void>; // Send PAUSE signal
  resumeAgent(featureId: string): Promise<void>; // Send RESUME signal
}

class CheckpointService {
  private checkpointLocks = new Map<string, Promise<void>>();

  async createCheckpoint(featureId: string): Promise<CheckpointResult> {
    // Serialize checkpoints per feature
    const existingCheckpoint = this.checkpointLocks.get(featureId);
    if (existingCheckpoint) {
      await existingCheckpoint;
    }

    const checkpointPromise = this._doCheckpoint(featureId);
    this.checkpointLocks.set(featureId, checkpointPromise);

    try {
      return await checkpointPromise;
    } finally {
      this.checkpointLocks.delete(featureId);
    }
  }

  private async _doCheckpoint(featureId: string): Promise<CheckpointResult> {
    // 1. Pause agent
    await this.provider.pauseAgent(featureId).catch(() => {
      this.logger.warn(`Agent ${featureId} did not acknowledge pause`);
    });

    // 2. Wait for file writes to stabilize (100ms)
    await this.waitForStableFileSystem(featureId);

    // 3. Capture diff
    const diff = await this.captureDiff(featureId);

    // 4. Validate
    const validation = await this.validateDiff(featureId, diff);

    // 5. Resume agent
    await this.provider.resumeAgent(featureId);

    return validation;
  }
}
```

### 9.2 Risk: Validation After Write (Too Late)

**Problem:** Interactive mode validates AFTER agent writes files. If validation fails:

- Agent has already made dependent changes based on invalid state
- Revert is destructive and loses agent's work
- No atomic rollback guarantee

**Mitigation:**

- **Shadow workspace:** Agent writes to shadow directory, validation promotes to real worktree
- **Incremental validation:** Validate each file as it's written (watchdog triggers per-file validation)
- **Validation cache:** Cache validation results per file to avoid re-validating unchanged files
- **Partial revert:** Only revert files that violate constraints, keep valid changes
- **Agent guidance:** Send pre-emptive warnings when agent attempts to write to suspicious paths

**Implementation (Shadow Workspace):**

```typescript
class InteractiveExecutionService {
  async runInteractiveWorker(featureId: string): Promise<void> {
    const realWorktree = this.kernel.worktreePath(featureId);
    const shadowWorktree = `${realWorktree}.shadow`;

    // 1. Create shadow workspace (copy of real worktree)
    await fs.cp(realWorktree, shadowWorktree, { recursive: true });

    // 2. Agent works in shadow
    await this.provider.runWorker({
      working_directory: shadowWorktree,
      // ...
    });

    // 3. Validate shadow changes
    const diff = await this.git.diff(realWorktree, shadowWorktree);
    const validation = await this.validateDiff(featureId, diff);

    if (validation.valid) {
      // 4. Promote shadow to real (atomic)
      await fs.rm(realWorktree, { recursive: true });
      await fs.rename(shadowWorktree, realWorktree);
    } else {
      // 5. Discard shadow, notify agent
      await fs.rm(shadowWorktree, { recursive: true });
      await this.notifyAgent(featureId, validation.violations);
    }
  }
}
```

**Alternative (Incremental Validation):**

```typescript
class WorktreeWatchdogService {
  async startWatching(featureId: string): Promise<void> {
    const watcher = chokidar.watch(worktreePath, {
      awaitWriteFinish: { stabilityThreshold: 100, pollInterval: 50 },
    });

    watcher.on('change', async (path) => {
      // Validate single file immediately
      const validation = await this.validateSingleFile(featureId, path);

      if (!validation.valid) {
        // Immediate feedback to agent
        await this.notifyAgent(featureId, validation.violations, 'critical');

        if (this.config.revert_on_violation) {
          // Revert single file only
          await this.git.checkout(featureId, path);
        }
      }
    });
  }
}
```

### 9.3 Risk: Agent Modifies Files Outside Worktree

**Problem:** Agent could escape worktree via:

- Symlinks (`ln -s /etc/passwd ./passwd`)
- Parent directory traversal (`../../sensitive-file`)
- Absolute paths (`/etc/passwd`)
- `.git` directory modification

**Mitigation:**

- **Symlink detection:** Watchdog rejects symlink creation immediately
- **Path canonicalization:** Resolve all paths to absolute, verify they start with worktree prefix
- **`.git` protection:** Watchdog ignores `.git` directory, validation rejects any `.git` changes
- **Filesystem sandbox (advanced):** Run agent in container with bind-mount to worktree only

**Implementation:**

```typescript
class WorktreeWatchdogService {
  private async validatePath(featureId: string, filePath: string): Promise<boolean> {
    const worktreePath = this.kernel.worktreePath(featureId);
    const canonicalPath = path.resolve(filePath);

    // 1. Must be inside worktree
    if (!canonicalPath.startsWith(worktreePath)) {
      await this.notifyAgent(
        featureId,
        [`Path escape detected: ${filePath} is outside worktree`],
        'critical',
      );
      return false;
    }

    // 2. Must not be .git directory
    if (canonicalPath.includes('/.git/')) {
      await this.notifyAgent(
        featureId,
        [`Git directory modification blocked: ${filePath}`],
        'critical',
      );
      return false;
    }

    // 3. Must not be symlink
    const stats = await fs.lstat(canonicalPath);
    if (stats.isSymbolicLink()) {
      await this.notifyAgent(featureId, [`Symlink creation blocked: ${filePath}`], 'critical');
      // Auto-remove symlink
      await fs.unlink(canonicalPath);
      return false;
    }

    return true;
  }
}
```

### 9.4 Risk: Validation Overhead in Interactive Mode

**Problem:** Every checkpoint runs full `git diff` + validation. With large changesets:

- `git diff` on 1000 files takes seconds
- Parsing 1000 diffs is CPU-intensive
- Validating 1000 paths against complex plan patterns is slow
- Checkpoint blocks agent progress

**Mitigation:**

- **Incremental diff:** Only diff files changed since last checkpoint (use watchdog's changed file list)
- **Validation cache:** Cache validation results per file path + plan version
- **Validation budget:** Timeout validation after 5s, log warning, allow agent to continue
- **Async validation:** Run validation in background, don't block agent (notify on completion)
- **Sampling:** For large changesets (>100 files), validate random sample + high-risk files only

**Implementation:**

```typescript
class CheckpointService {
  private validationCache = new Map<string, ValidationResult>();

  async validateCheckpoint(featureId: string, diff: string): Promise<ValidationResult> {
    const plan = await this.loadPlan(featureId);
    const parsedDiff = this.parseDiff(diff);
    const changedFiles = parsedDiff.map((d) => d.path);

    // Check cache, validate uncached files with timeout
    const validationPromise = this.validateFiles(featureId, changedFiles, plan);
    const timeoutPromise = new Promise<ValidationResult>((resolve) =>
      setTimeout(
        () =>
          resolve({
            valid: false,
            violations: ['Validation timeout after 5s'],
            severity: 'warning',
          }),
        5000,
      ),
    );

    return await Promise.race([validationPromise, timeoutPromise]);
  }

  private async validateFiles(featureId: string, files: string[], plan: Plan) {
    // Sampling for large changesets (>100 files)
    if (files.length > 100) {
      const highRiskFiles = files.filter(
        (f) => f.includes('schema') || f.includes('contract') || f.includes('migration'),
      );
      const sampledFiles = this.randomSample(files, 20);
      files = [...new Set([...highRiskFiles, ...sampledFiles])];
    }

    return await pMap(files, (file) => this.validateSingleFile(featureId, file, plan), {
      concurrency: 10,
    });
  }
}
```

### 9.5 Risk: Incomplete Audit Trail

**Problem:** Checkpoints only capture state at intervals. Between checkpoints:

- No record of intermediate states
- No record of which files were attempted but reverted
- No record of agent's decision process
- Cannot reconstruct exact sequence of changes

**Mitigation:**

- **Continuous event log:** Watchdog logs every file change event with timestamp
- **Agent action log:** Capture agent's tool calls and responses (even in interactive mode)
- **Validation history:** Log all validation attempts (pass/fail) with details
- **Checkpoint chain:** Each checkpoint references previous checkpoint (linked list)
- **Diff accumulation:** Store both incremental diffs (since last checkpoint) and cumulative diffs (since start)

**Implementation:**

```typescript
interface CheckpointMetadata {
  checkpoint_id: string;
  timestamp: string;
  previous_checkpoint_id: string | null; // Chain checkpoints
  files_changed_since_last: string[]; // Incremental
  files_changed_total: string[]; // Cumulative
  diff_incremental: string; // Path to incremental diff
  diff_cumulative: string; // Path to cumulative diff
  validation_status: 'valid' | 'invalid' | 'skipped';
  violations: string[];
  severity: ViolationSeverity;
}

interface FileChangeEvent {
  timestamp: string;
  event_type: 'add' | 'change' | 'unlink';
  file_path: string;
  file_size: number;
  checkpoint_id: string | null; // Which checkpoint captured this change
}

class WorktreeWatchdogService {
  private eventLog = new Map<string, FileChangeEvent[]>();

  async startWatching(featureId: string): Promise<void> {
    const watcher = chokidar.watch(worktreePath);

    watcher.on('all', (event, path) => {
      const changeEvent: FileChangeEvent = {
        timestamp: new Date().toISOString(),
        event_type: event,
        file_path: path,
        file_size: fs.statSync(path).size,
        checkpoint_id: null, // Will be set when checkpoint captures it
      };

      this.eventLog.get(featureId)?.push(changeEvent);

      // Persist event log continuously
      this.persistEventLog(featureId);
    });
  }

  private async persistEventLog(featureId: string): Promise<void> {
    const logPath = `.aop/features/${featureId}/logs/file-changes.jsonl`;
    const events = this.eventLog.get(featureId) || [];

    // Append to JSONL file (one JSON object per line)
    await fs.appendFile(logPath, events.map((e) => JSON.stringify(e)).join('\n') + '\n');
  }
}
```

### 9.6 Risk: Provider Incompatibility

**Problem:** Not all providers support interactive mode:

- API-based providers (Claude API, Codex API) cannot set `cwd`
- Some CLI providers don't respect working directory
- Agent may not understand pause/resume protocol

**Mitigation:**

- **Provider capability detection:** Query provider for `supportsInteractiveMode` before starting
- **Automatic fallback:** If provider doesn't support interactive, fall back to deterministic mode
- **Clear error messages:** Warn user if they request interactive mode with incompatible provider
- **Provider adapter interface:** Standardize pause/resume/working_directory contract

**Implementation:**

```typescript
interface WorkerProviderCapabilities {
  supportsInteractiveMode: boolean;
  supportsWorkingDirectory: boolean;
  supportsPauseResume: boolean;
  supportsMessagePassing: boolean;
}

interface WorkerProvider {
  getCapabilities(): WorkerProviderCapabilities;
  // ... existing methods
}

class SupervisorRuntime {
  private resolveExecutionMode(featureId: string, requestedMode: ExecutionMode): ExecutionMode {
    const capabilities = this.provider.getCapabilities();

    if (requestedMode === 'interactive' && !capabilities.supportsInteractiveMode) {
      this.logger.warn(
        `Provider ${this.provider.name} does not support interactive mode, ` +
          `falling back to deterministic mode`,
      );
      return 'deterministic';
    }

    return requestedMode;
  }
}
```

### 9.7 Risk: Security - Agent Escapes Worktree

**Problem:** Agent with direct file system access could:

- Write malicious code that executes before checkpoint
- Modify `.git` directory to corrupt repository
- Create symlinks to escape worktree
- Exhaust disk space with large files
- Execute arbitrary commands via shell scripts

**Mitigation:**

- **Path validation:** Reject any file operations outside worktree (see 9.3)
- **`.git` protection:** Watchdog blocks all `.git` modifications immediately
- **Disk quota:** Set file size limits per feature (e.g., 1GB max)
- **Executable detection:** Flag creation of executable files for review
- **Sandboxing (advanced):** Run agent in Docker container with read-only bind mounts except worktree

**Implementation:**

```typescript
class WorktreeWatchdogService {
  private diskUsage = new Map<string, number>();
  private readonly MAX_DISK_USAGE = 1024 * 1024 * 1024; // 1GB

  async startWatching(featureId: string): Promise<void> {
    const watcher = chokidar.watch(worktreePath);

    watcher.on('add', async (path) => {
      // 1. Path validation
      if (!(await this.validatePath(featureId, path))) {
        await fs.unlink(path);
        return;
      }

      // 2. Disk quota
      const stats = await fs.stat(path);
      const currentUsage = this.diskUsage.get(featureId) || 0;

      if (currentUsage + stats.size > this.MAX_DISK_USAGE) {
        await this.notifyAgent(
          featureId,
          [
            `Disk quota exceeded: ${currentUsage + stats.size} bytes > ${this.MAX_DISK_USAGE} bytes`,
          ],
          'critical',
        );
        await fs.unlink(path);
        return;
      }

      this.diskUsage.set(featureId, currentUsage + stats.size);

      // 3. Executable detection
      if (stats.mode & 0o111) {
        // Has execute bit
        await this.notifyAgent(
          featureId,
          [`Executable file created: ${path} - requires review`],
          'warning',
        );
      }
    });
  }
}
```

### 9.8 Risk: No Graceful Degradation

**Problem:** If watchdog fails, checkpoint service fails, or validation times out:

- Agent continues running blind (no monitoring)
- Changes accumulate without validation
- System state becomes inconsistent
- No clear recovery path

**Mitigation:**

- **Circuit breakers:** After 3 consecutive checkpoint failures, pause agent and escalate to human
- **Health checks:** Periodic health checks for watchdog, checkpoint service, validation service
- **Fallback mode:** If interactive mode fails, automatically switch to deterministic mode
- **Agent timeout:** If agent doesn't complete within max_execution_time, force checkpoint and terminate
- **Recovery protocol:** On service failure, capture current state, notify human, wait for manual intervention

**Implementation:**

```typescript
class InteractiveExecutionService {
  private checkpointFailures = new Map<string, number>();
  private readonly MAX_CHECKPOINT_FAILURES = 3;

  async runInteractiveWorker(featureId: string): Promise<void> {
    try {
      // Health check before starting
      await this.healthCheck();

      // Start agent with timeout
      const agentPromise = this.provider.runWorker({...});
      const timeoutPromise = new Promise((_, reject) =>
        setTimeout(() => reject(new Error('Agent timeout')),
          this.config.max_execution_time_ms)
      );

      await Promise.race([agentPromise, timeoutPromise]);

    } catch (error) {
      // Circuit breaker check
      const failures = this.checkpointFailures.get(featureId) || 0;

      if (failures >= this.MAX_CHECKPOINT_FAILURES) {
        await this.escalateToHuman(featureId, error);
        throw new Error('Interactive mode failed, human intervention required');
      }

      // Fallback to deterministic mode
      this.logger.warn(
        `Interactive mode failed for ${featureId}, falling back to deterministic mode`
      );

      await this.runDeterministicWorker(featureId);
    }
  }

  private async healthCheck(): Promise<void> {
    const checks = [
      this.watchdog.healthCheck(),
      this.checkpointService.healthCheck(),
      this.validationService.healthCheck(),
    ];

    const results = await Promise.allSettled(checks);
    const failures = results.filter(r => r.status === 'rejected');

    if (failures.length > 0) {
      throw new Error(`Health check failed: ${failures.length} services unhealthy`);
    }
  }

  private async escalateToHuman(featureId: string, error: Error): Promise<void> {
    // 1. Capture current state
    const state = await this.captureState(featureId);

    // 2. Pause agent
    await this.provider.pauseAgent(featureId);

    // 3. Notify human via all channels
    await this.notificationService.send({
      severity: 'critical',
      title: `Interactive mode failure: ${featureId}`,
      message: `Agent execution failed after ${this.MAX_CHECKPOINT_FAILURES} checkpoint failures`,
      details: { error: error.message, state },
      actions: [
        { label: 'Review State', url: `/dashboard/features/${featureId}` },
        { label: 'Force Checkpoint', action: 'force_checkpoint' },
        { label: 'Terminate Agent', action: 'terminate' },
      ],
    });

    // 4. Update feature state to BLOCKED
    await this.kernel.updateFeatureState(featureId, {
      status: 'blocked',
      blocked_reason: 'interactive_mode_failure',
      requires_human_intervention: true,
    });
  }
}
```

### 9.9 Risk: Concurrent Feature Resource Contention

- Fallback to deterministic mode if provider doesn't support `working_directory`

---

### 9.9 Risk: Concurrent Feature Resource Contention

**Problem:** Multiple features in interactive mode simultaneously:

- Watchdog service is singleton with shared state
- Checkpoint service has no concurrency control
- Validation service could be bottlenecked
- File system events could be misattributed

**Mitigation:**

- **Per-feature service instances:** Each feature gets isolated watchdog/checkpoint instances
- **Resource pooling:** Limit concurrent interactive features (e.g., max 5)
- **Event attribution:** Watchdog uses separate `chokidar` instance per feature
- **Validation queue:** Serialize validation requests with priority queue
- **Backpressure:** If validation queue is full, pause low-priority features

**Implementation:**

```typescript
class WorktreeWatchdogService {
  private watchers = new Map<string, FSWatcher>(); // Per-feature watchers
  private eventLogs = new Map<string, FileChangeEvent[]>();

  async startWatching(featureId: string): Promise<void> {
    const worktreePath = this.kernel.worktreePath(featureId);
    const watcher = chokidar.watch(worktreePath, {
      ignored: /(^|[\/\\])\../,
      persistent: true,
      ignoreInitial: true,
    });

    this.watchers.set(featureId, watcher);
    this.eventLogs.set(featureId, []);

    watcher.on('all', (event, path) => {
      this.handleFileChange(featureId, event, path);
    });
  }
}

class CheckpointService {
  private validationQueue = new PQueue({ concurrency: 3 });

  async validateCheckpoint(featureId: string, diff: string): Promise<ValidationResult> {
    return this.validationQueue.add(() => this._validateCheckpoint(featureId, diff), {
      priority: this.getFeaturePriority(featureId),
    });
  }
}
```

---

## 10. Performance Requirements

### 10.1 Latency Budgets

| Operation                 | Target  | Maximum | Degradation Strategy         |
| ------------------------- | ------- | ------- | ---------------------------- |
| Checkpoint creation       | < 500ms | 2s      | Skip if timeout, log warning |
| Single file validation    | < 50ms  | 200ms   | Use cached result if timeout |
| Full diff validation      | < 1s    | 5s      | Sample validation if timeout |
| Agent pause/resume        | < 100ms | 500ms   | Force checkpoint if no ACK   |
| Watchdog event processing | < 10ms  | 50ms    | Buffer events if slow        |

### 10.2 Throughput Requirements

| Metric                           | Target          | Maximum         |
| -------------------------------- | --------------- | --------------- |
| File changes per second          | 100             | 500             |
| Concurrent interactive features  | 5               | 10              |
| Checkpoints per feature per hour | 120 (every 30s) | 360 (every 10s) |
| Validation cache hit rate        | > 80%           | N/A             |

### 10.3 Resource Limits

| Resource              | Limit per Feature | Limit Global |
| --------------------- | ----------------- | ------------ |
| Disk usage            | 1 GB              | 10 GB        |
| Memory (watchdog)     | 50 MB             | 500 MB       |
| CPU (validation)      | 10%               | 50%          |
| Open file descriptors | 1000              | 10000        |

### 10.4 Monitoring and Alerts

**Metrics to track:**

- Checkpoint latency (p50, p95, p99)
- Validation latency (p50, p95, p99)
- Watchdog event processing latency
- Validation cache hit rate
- Checkpoint failure rate
- Agent pause/resume success rate
- Disk usage per feature
- Memory usage per watchdog instance

**Alerts:**

- Checkpoint latency > 2s for 3 consecutive checkpoints
- Validation failure rate > 10%
- Disk usage > 80% of limit
- Memory usage > 80% of limit
- Watchdog event queue depth > 1000

---

## 11. Success Metrics

### 10.1 Functional Metrics

- [x] Both execution modes pass all tests
- [x] No regressions in deterministic mode
- [x] Interactive mode enforces all plan/policy/lock constraints
- [x] Audit trail complete for both modes

### 10.2 Quality Metrics

- [ ] Agent output quality improves in interactive mode (subjective, user feedback)
- [ ] Iteration speed increases in interactive mode (measured by time-to-completion)
- [ ] Context window usage decreases in interactive mode (fewer `repo.read_file` calls)

### 10.3 Performance Metrics

- [ ] Checkpoint validation completes in < 500ms
- [ ] Watchdog overhead < 5% CPU per active feature
- [ ] No memory leaks in long-running interactive sessions

---

## 11. Future Enhancements

### 11.1 Hybrid Mode

**Concept:** Agent can choose execution mode per task.

**Example:**

- Use deterministic mode for critical contract changes (requires explicit approval)
- Use interactive mode for implementation details (faster iteration)

### 11.2 Checkpoint Branching

**Concept:** Create git branches at checkpoints for easy rollback.

**Benefit:** Agent can experiment freely, supervisor can revert to any checkpoint.

### 11.3 Multi-Agent Interactive Mode

**Concept:** Multiple agents work in same worktree with conflict resolution.

**Challenge:** Requires sophisticated merge logic and coordination.

### 11.4 Real-Time Collaboration

**Concept:** Human operator can edit files in worktree while agent is running.

**Benefit:** Pair programming with AI agent.

**Challenge:** Requires conflict detection and resolution.

---

## 12. Appendix

### 12.1 Related Specifications

- **Shadow Workspace Implementation:** [Shadow Workspace Implementation Specification](../outstanding/shadow_workspace_implementation_spec.md) - Detailed implementation of shadow workspace strategy for validation-before-write guarantees
- **Runtime Inspection:** [Runtime Inspection Specification](./agentic_orchestrator_runtime_inspection_spec.md) - Dashboard integration for execution mode monitoring

### 12.2 File System Watchdog Libraries

**Options:**

- `chokidar` - Cross-platform, battle-tested, 20M+ downloads/week
- `fs.watch` - Native Node.js, no dependencies
- `watchman` - Facebook's file watching service (requires external daemon)

**Recommendation:** `chokidar` for reliability and cross-platform support.

### 12.2 Checkpoint Storage Format

**Diff snapshot format:** Unified diff (same as `git diff` output)

**Metadata format:** JSON in `state.md` frontmatter

**Example:**

```yaml
checkpoints:
  - checkpoint_id: 'ckpt-001-a3f2b9c4'
    timestamp: '2026-03-05T16:28:19.503Z'
    files_changed: ['src/app.ts', 'src/utils.ts']
    validation_status: 'valid'
    violations: []
    severity: 'info'
    diff_snapshot: '.aop/features/my_feature/checkpoints/ckpt-001-a3f2b9c4.diff'
  - checkpoint_id: 'ckpt-002-b7e4c1d9'
    timestamp: '2026-03-05T16:29:05.127Z'
    files_changed: ['src/config.ts']
    validation_status: 'invalid'
    violations: ["Path 'src/config.ts' not in allowed_areas"]
    severity: 'warning'
    diff_snapshot: '.aop/features/my_feature/checkpoints/ckpt-002-b7e4c1d9.diff'
```

### 12.3 Validation Reuse

**Shared validation logic:**

- `PatchService.validateDiff(featureId, parsedDiff)`
- Used by:
  - `PatchService.repoApplyPatch` (deterministic mode)
  - `CheckpointService.validateCheckpoint` (interactive mode)

**Validation steps:**

1. Parse diff into file operations (create/modify/delete)
2. Load accepted plan
3. Validate paths against `allowed_areas` and `forbidden_areas`
4. Validate locks held for contract modifications
5. Validate against `protected_areas` and `path_rules`
6. Return validation result with violations array

---

**End of Specification**
