# Feature Spec: Persistent Worker Runtime Sessions with Patch-Mediated Writes (AOP)

> **Purpose of this document**: Define an implementation-ready design for reusing long-lived worker runtime processes across planner/builder/qa turns while preserving deterministic orchestration, tool firewall constraints, and `repo.apply_patch` as the only code-mutation path.

**Version:** 1.0  
**Date:** 2026-03-05  
**Status:** Draft  
**Roadmap Mapping:** M47

---

## 0. Standards and Dependencies

### 0.1 Required Standards

All implementation MUST preserve:

- deterministic orchestration and normalized error envelopes
- MCP/in-process tool contract parity
- role-based tool authorization and `operation_id` idempotency on mutating tools
- canonical runtime artifacts under `.aop/**`
- existing feature topology invariants (`1 + 3N` logical sessions)
- coverage thresholds: Lines >=90%, Branches >=90%, Functions >=90%, Statements >=90%

### 0.2 Required Upstream Inputs

Implementing agents MUST read:

- `spec-files/outstanding/agentic_orchestrator_real_worker_provider_execution_spec.md`
- `spec-files/outstanding/agentic_orchestrator_worker_runtime_watchdog_resilience_spec.md`
- `apps/control-plane/src/providers/providers.ts`
- `apps/control-plane/src/providers/cli-worker-provider.ts`
- `apps/control-plane/src/providers/api-worker-provider.ts`
- `apps/control-plane/src/providers/worker-provider-factory.ts`
- `apps/control-plane/src/supervisor/session-orchestrator.ts`
- `apps/control-plane/src/supervisor/worker-decision-loop.ts`
- `apps/control-plane/src/supervisor/runtime.ts`
- `apps/control-plane/src/supervisor/build-wave-executor.ts`
- `apps/control-plane/src/supervisor/qa-wave-executor.ts`
- `apps/control-plane/src/application/services/patch-service.ts`
- `apps/control-plane/src/mcp/tool-runtime.ts`
- `agentic/orchestrator/agents.yaml`
- `agentic/orchestrator/schemas/agents.schema.json`
- `agentic/orchestrator/policy.yaml`
- `agentic/orchestrator/schemas/policy.schema.json`

### 0.3 Scope

This spec implements:

- persistent per-role worker runtime processes reused across turns
- explicit separation between logical session identity and runtime process lifecycle
- deterministic fallback and restart policy for persistent runtime failures
- runtime observability artifacts for boot/turn/restart/teardown events
- full preservation of patch-mediated write controls (`PATCH` -> `repo.apply_patch`)

Out of scope:

- direct filesystem write permission for worker providers
- replacing plan/patch/gate semantics
- changing role-level tool authorization rules
- changing merge governance and approval policy

### 0.4 OOP and Architecture Guardrails (Mandatory)

Implementation MUST follow:

- **KISS**: one provider execution abstraction with two strategies (`turn_spawn`, `persistent_runtime`)
- **SRP**: provider process management, output parsing, and wave policy decisions remain in separate modules
- **SOLID-oriented design**:
  - open/closed extension for new providers without supervisor branching
  - interface segregation between worker execution strategy and provider selection
  - dependency inversion from supervisor to provider/runtime ports
- **Hexagonal alignment**: process control and protocol details remain in `providers/*`, not `supervisor/*`
- **Composition over inheritance**: runtime manager composed into provider adapters
- **High cohesion / low coupling**: no direct patch/gate logic inside provider runtime manager

---

## 1. Problem Statement and Clarifications

### 1.1 Current Behavior

Current `runWorker(...)` behavior for CLI-backed providers spawns a new OS process per turn.

- logical role sessions are created once per feature (`planner`, `builder`, `qa`)
- OS worker process is currently short-lived per iteration

This means build round 1 and build round 2 typically reuse the same logical builder session id, but not the same process.

### 1.2 Why Change if Build Is 10-30 Minutes?

Spawn overhead is usually not the dominant runtime cost for long builds. However, repeated process churn still creates non-trivial costs:

- repeated process boot and protocol setup
- larger context replay burden per turn
- more failure surface at process startup boundaries
- weaker continuity for provider-side reasoning context

Primary objective for M47 is **context continuity and reliability**, with latency improvement as a secondary outcome.

### 1.3 Non-Negotiable Safety Constraint

This spec does **not** move to direct worktree writes by workers.

Workers continue to emit structured outputs (`PATCH`, `NOTE`, `REQUEST`, `PLAN_SUBMISSION`) and code changes still land only through `repo.apply_patch` policy checks.

---

## 2. Revised Architecture Fit

### 2.1 Key Distinction: Session vs Runtime

Define two independent lifecycle objects:

- **Logical worker session**: existing cluster identity (`session_id`) used for topology and audit
- **Worker runtime process**: long-lived provider subprocess bound to one `(run_id, feature_id, role)`

Logical session survives runtime restarts. Runtime process may restart without changing session assignment.

### 2.2 Execution Strategy Abstraction

Add explicit execution strategy contract:

```ts
export type WorkerExecutionMode = 'turn_spawn' | 'persistent_runtime';

export interface WorkerExecutionStrategy {
  runTurn(input: WorkerTurnInput): Promise<WorkerOutputEnvelope>;
  ensureReady?(ctx: RuntimeContext): Promise<void>;
  close?(ctx: RuntimeContext): Promise<void>;
}
```

Implementations:

- `SpawnPerTurnExecutionStrategy` (current behavior)
- `PersistentRuntimeExecutionStrategy` (new)

### 2.3 Runtime Manager

Create `PersistentWorkerRuntimeManager` (provider-layer component) that:

- owns runtime handles keyed by `(run_id, feature_id, role)`
- boots process once (lazy or eager)
- sends turns over a persistent protocol channel
- tracks health/turn count/restart count
- tears down process deterministically on feature close and run close

### 2.4 Provider Capability Model

Not all providers can support persistent process mode immediately.

Add provider capability negotiation:

```ts
interface ProviderRuntimeCapabilities {
  supports_turn_spawn: true;
  supports_persistent_runtime: boolean;
  persistent_protocol?: 'jsonl_stdio' | 'native_session_rpc';
}
```

Policy governs unsupported cases:

- fallback to `turn_spawn`
- block feature
- fail run

### 2.4.1 Provider-Specific Assumptions (M47 Baseline)

The following assumptions are mandatory for initial implementation to avoid ambiguous provider behavior.

| Provider               | Current run path                 | Baseline persistent support                         | Default M47 behavior                                               |
| ---------------------- | -------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------ |
| `codex`                | CLI command with session id      | **Assume unsupported unless explicitly configured** | use `turn_spawn`; preserve session-id continuity in prompt path    |
| `claude`               | CLI print/text command via stdin | **Assume unsupported**                              | use `turn_spawn` only                                              |
| `kiro-cli` / `copilot` | CLI command path                 | **Assume unsupported**                              | use `turn_spawn` only                                              |
| `custom`               | user-defined command templates   | **Config-driven**                                   | may opt into persistent runtime when protocol contract is declared |
| `gemini`               | API adapter (no CLI process)     | not applicable in M47                               | unchanged API request model                                        |

### 2.4.2 Codex and Claude Tightening Rules

- `codex`:
  - current default command path is treated as `turn_spawn` even with provider-side session ids.
  - persistent mode may only be enabled when explicit adapter config declares a machine-readable persistent protocol and passes protocol contract tests.
- `claude`:
  - current `--print --output-format text` path is treated as stateless and non-persistent.
  - persistent mode is disabled by default and cannot be inferred automatically.
- For both:
  - if `persistent_runtime_required_for_run=true` and provider capability is unsupported, fail deterministically with `provider_runtime_capability_unsupported`.
  - if `persistent_runtime_required_for_run=false`, apply policy-driven fallback (`fallback_turn_spawn` by default).

### 2.5 Runtime Granularity and Lifecycle

Runtime process granularity is fixed:

- one runtime per `(run_id, feature_id, role)`
- planner/build/qa each get separate runtime
- builder runtime is reused across build wave iterations and gate-repair iterations

This directly answers the lifecycle concern:

- not recreated between build round 1 and build round 2
- separate from planner and QA runtimes

### 2.6 Determinism Invariant

Persistent process memory is an optimization, not source of truth.

Authoritative state remains:

- `.aop/features/<id>/state.md`
- `.aop/features/<id>/plan.json`
- `.aop/features/<id>/qa_test_index.json`
- tool responses and event journals

Each turn still includes deterministic context bundle input so cold-restart replay remains correct.

---

## 3. Why Keep Patch-Mediated Writes (Not Direct Worktree Writes)

### 3.1 Advantages over Direct-Write + Post-hoc Diff

Keeping `PATCH` mediation preserves:

- path policy enforcement before mutation (`allowed_areas`, `forbidden_areas`, protected areas)
- plan-file-set validation before mutation
- required-lock checks before mutation
- mutating-tool idempotency via `operation_id`
- role-based tool firewall and auditable mutation envelopes

Direct worker writes plus post-hoc diff lose preventive controls and detect violations only after mutations occurred.

### 3.2 Consequence for M47

Persistent runtime affects **how turns are executed**, not how code writes are authorized.

Write path remains:

`worker output PATCH` -> `repo.apply_patch` -> `PatchService.validatePatchPaths/assertPlanLocksHeld` -> `git apply`

---

## 4. Contract and Schema Deltas

## 4.1 Agents Runtime Contract (`agents.schema.json`)

Additive runtime fields:

```yaml
runtime:
  worker_execution_mode: turn_spawn # turn_spawn | persistent_runtime
  worker_runtime_boot_mode: lazy # lazy | eager
  worker_runtime_boot_timeout_ms: 20000
  worker_runtime_idle_ttl_ms: 900000
  worker_runtime_max_turns: 200
  worker_runtime_max_restarts_per_feature: 3
```

Rules:

- all timeout/limits must be positive integers
- `worker_execution_mode` default remains `turn_spawn` for backward compatibility
- `eager` mode pre-warms runtimes on feature cluster initialization

## 4.2 Policy Contract (`policy.schema.json`)

Add execution policy controls:

```yaml
execution:
  persistent_runtime_required_for_run: false
  persistent_runtime_boot_failure_action: fallback_turn_spawn # fallback_turn_spawn | block_feature | fail_run
  persistent_runtime_protocol_failure_action: restart_runtime # restart_runtime | block_feature | fail_run
  persistent_runtime_restart_exhausted_action: block_feature # block_feature | fail_run
```

Rules:

- when `persistent_runtime_required_for_run=true`, unsupported providers are hard errors
- restart attempts must not exceed `worker_runtime_max_restarts_per_feature`

## 4.2.1 Provider Override Contract (Additive)

Add optional provider capability overrides in `agents.yaml` runtime section:

```yaml
runtime:
  persistent_provider_overrides:
    codex:
      persistent_enabled: false
      persistent_protocol: null
    claude:
      persistent_enabled: false
      persistent_protocol: null
    custom:
      persistent_enabled: true
      persistent_protocol: jsonl_stdio
```

Rules:

- overrides are additive and optional
- unknown providers in overrides are validation errors
- `persistent_enabled=true` requires explicit `persistent_protocol`
- default remains conservative (`false`) for codex/claude in M47 baseline

## 4.3 Runtime Artifact Contract

Create new runtime artifacts:

```text
.aop/runtime/worker-runtime-registry/<run_id>.json
.aop/runtime/worker-runtime-events/<run_id>.jsonl
```

Registry entry shape:

```json
{
  "runtime_id": "builder-feature_x-...",
  "run_id": "run:...",
  "feature_id": "feature_x",
  "role": "builder",
  "session_id": "builder-feature_x-...",
  "provider": "claude",
  "model": "local-default",
  "pid": 12345,
  "status": "ready",
  "turn_count": 14,
  "restart_count": 1,
  "started_at": "2026-03-05T00:00:00.000Z",
  "last_turn_at": "2026-03-05T00:12:34.000Z"
}
```

Event journal types:

- `runtime_boot_started`
- `runtime_boot_succeeded`
- `runtime_boot_failed`
- `runtime_turn_started`
- `runtime_turn_succeeded`
- `runtime_turn_failed`
- `runtime_restarted`
- `runtime_closed`

## 4.4 Error Taxonomy Extensions

Add deterministic error codes:

- `provider_runtime_boot_failed`
- `provider_runtime_protocol_error`
- `provider_runtime_restart_exhausted`
- `provider_runtime_capability_unsupported`
- `provider_runtime_fallback_engaged`

---

## 5. Runtime Flow (Normative)

### 5.1 Run Start and Cluster Init

1. `SessionOrchestrator.initializeFeatureCluster(featureId)` creates logical planner/builder/qa sessions.
2. If `worker_execution_mode=persistent_runtime` and `boot_mode=eager`, runtime manager boots runtimes for all cluster roles.
3. If boot fails, apply `persistent_runtime_boot_failure_action`.

### 5.2 Builder Turn (Including Build Round N)

1. `BuildWaveExecutor` calls `WorkerDecisionLoop.execute(role='builder')`.
2. Worker decision loop calls provider `runWorker(...)`.
3. Provider routes call through execution strategy:
   - `turn_spawn`: current short-lived process
   - `persistent_runtime`: reuse existing runtime process for that feature/role
4. Worker output is parsed and routed.
5. `PATCH` outputs continue through `repo.apply_patch`.

### 5.3 Runtime Restart Path

On protocol/runtime failure in persistent mode:

1. manager marks runtime unhealthy
2. manager attempts restart (bounded by restart limit)
3. turn is retried once after successful restart
4. if restart limit exceeded, apply `persistent_runtime_restart_exhausted_action`

### 5.4 Resume Semantics

`aop resume` does not require surviving old PIDs.

- runtime registry is rebuilt from active feature session assignments
- missing/dead runtimes are cold-started with same logical session ids
- deterministic context bundle reconstructs turn state

### 5.5 Feature Close and Run End

- `SessionOrchestrator.closeFeatureCluster(featureId)` closes all role runtimes first, then logical sessions
- global run shutdown closes remaining runtimes and writes final runtime registry snapshot

---

## 6. Implementation Plan (Normative)

## M47.1 Contracts and Defaults

Files:

- `agentic/orchestrator/agents.yaml`
- `agentic/orchestrator/schemas/agents.schema.json`
- `agentic/orchestrator/policy.yaml`
- `agentic/orchestrator/schemas/policy.schema.json`
- `apps/control-plane/src/core/error-codes.ts`

Deliverables:

- new runtime mode/config fields
- new execution policy fields for failure handling
- new persistent runtime error codes

## M47.2 Provider Runtime Core

Files:

- `apps/control-plane/src/providers/worker-execution-strategy.ts` (new)
- `apps/control-plane/src/providers/persistent-worker-runtime-manager.ts` (new)
- `apps/control-plane/src/providers/providers.ts`

Deliverables:

- execution strategy interface and concrete strategy implementations
- runtime handle state machine (`booting`, `ready`, `unhealthy`, `closing`, `closed`)
- bounded restart logic and protocol-safe turn dispatch

## M47.3 Adapter and Factory Integration

Files:

- `apps/control-plane/src/providers/cli-worker-provider.ts`
- `apps/control-plane/src/providers/api-worker-provider.ts`
- `apps/control-plane/src/providers/worker-provider-factory.ts`

Deliverables:

- provider capability negotiation
- mode selection wiring from runtime config
- policy-driven fallback or hard-failure behavior
- codex/claude baseline capability explicitly set to non-persistent unless override enables a validated protocol
- deterministic mismatch handling:
  - required persistent mode + unsupported capability -> `provider_runtime_capability_unsupported`
  - optional persistent mode + unsupported capability -> `provider_runtime_fallback_engaged` and `turn_spawn`

## M47.4 Supervisor Lifecycle Wiring

Files:

- `apps/control-plane/src/supervisor/runtime.ts`
- `apps/control-plane/src/supervisor/session-orchestrator.ts`
- `apps/control-plane/src/supervisor/worker-decision-loop.ts`
- `apps/control-plane/src/supervisor/build-wave-executor.ts`
- `apps/control-plane/src/supervisor/qa-wave-executor.ts`

Deliverables:

- eager/lazy runtime boot orchestration
- deterministic runtime cleanup on feature closure
- restart/fallback behavior propagated to feature policy actions

## M47.5 Runtime Observability and Reporting

Files:

- `apps/control-plane/src/supervisor/worker-decision-loop.ts`
- `apps/control-plane/src/application/services/reporting-service.ts`
- `apps/control-plane/src/core/path-layout.ts`

Deliverables:

- runtime registry + runtime event journal artifacts
- dashboard/report summary fields for runtime mode, restart count, unhealthy runtimes

## M47.6 Verification and Regression Hardening

Files:

- `apps/control-plane/test/worker-provider-adapters.spec.ts`
- `apps/control-plane/test/providers.spec.ts`
- `apps/control-plane/test/worker-execution-policy.spec.ts`
- `apps/control-plane/test/supervisor.unit.spec.ts`
- `apps/control-plane/test/supervisor.spec.ts`

Deliverables:

- persistent mode unit and integration coverage
- fallback and restart policy tests
- non-regression coverage for patch-mediation enforcement

---

## 7. Test Strategy (Must Be in `npm run verify` Path)

### 7.1 Unit - Runtime Manager

- boot success/failure state transitions
- turn dispatch against ready runtime
- dead runtime detection and restart
- restart exhaustion behavior
- idle TTL eviction and deterministic close

### 7.2 Unit - Provider Adapters

- `turn_spawn` and `persistent_runtime` produce equivalent normalized output envelopes
- unsupported persistent capability path obeys policy (fallback vs hard fail)
- persistent runtime protocol error maps to deterministic error code

### 7.3 Supervisor Policy Tests

- builder persistent runtime restart succeeds and build continues
- builder restart exhausted with `block_feature` blocks only that feature
- builder restart exhausted with `fail_run` fails run deterministically
- QA runtime lifecycle mirrors builder behavior
- planner runtime lifecycle does not interfere with builder/QA progression

### 7.4 Integration Tests

- multi-feature run with mixed runtime health: one feature restarts, others continue
- `resume` after process death reconstructs runtime and continues
- patch path remains mediated (`PATCH` required; no direct write side effects)

### 7.5 Non-Regression Tests

- role tool firewall unchanged
- `repo.apply_patch` policy checks still enforced in persistent mode
- operation ledger idempotency unchanged for mutating tools

---

## 8. Acceptance Criteria

1. Runtime process reuse is active for a configured persistent mode and verified by stable runtime id/pid across consecutive turns for the same `(run_id, feature_id, role)`.
2. Builder runtime is not respawned between build round 1 and build round 2 unless failure policy triggers restart.
3. Planner/build/QA keep separate runtimes and session ids; no cross-role runtime sharing.
4. Persistent runtime failures are categorized with explicit error codes and policy actions.
5. Unsupported provider capabilities are handled deterministically per policy (fallback or fail).
6. `repo.apply_patch` remains the only write path for worker-emitted code changes.
7. Runtime artifacts under `.aop/runtime/worker-runtime-*` are append-only and sufficient for RCA.
8. `aop resume` correctly reconstructs missing runtimes without corrupting feature status transitions.
9. `npm run verify` includes persistent-runtime regression coverage and remains green.

---

## 9. Risks and Mitigations

- Risk: provider CLI does not support stable persistent protocol semantics.
  - Mitigation: capability negotiation + policy-driven fallback to `turn_spawn`.

- Risk: long-lived runtimes leak memory or become stale.
  - Mitigation: `worker_runtime_max_turns`, idle TTL eviction, bounded restart policy.

- Risk: restart loops hide real provider outages.
  - Mitigation: restart exhaustion action and explicit error taxonomy.

- Risk: persistent memory diverges from authoritative feature artifacts.
  - Mitigation: context bundle remains deterministic input on every turn; runtime memory is non-authoritative.

- Risk: implementation complexity outpaces benefit for long-running builds.
  - Mitigation: phased rollout behind config flag with observability gates before default flip.

---

## 10. Definition of Done

A completed implementation MUST provide:

- schema/config deltas for runtime mode and failure policies
- provider execution strategy abstraction with both modes implemented
- persistent runtime manager with bounded restart/ttl/turn controls
- supervisor lifecycle integration for boot/close/resume
- runtime registry and event artifacts under `.aop/runtime`
- deterministic error codes and normalized envelopes for persistent runtime failures
- green `npm run verify` with persistent mode tests and no patch-safety regressions
- `spec-files/progress.md` entry documenting delivery, tests, and evidence

---

## 11. Spec Gap Tracker

| ID     | Name                                         | Priority | Status  | Milestone |
| ------ | -------------------------------------------- | -------- | ------- | --------- |
| M47-S1 | Runtime mode/schema contracts                | P0       | Planned | M47       |
| M47-S2 | Execution strategy + runtime manager         | P0       | Planned | M47       |
| M47-S3 | Provider capability negotiation and fallback | P0       | Planned | M47       |
| M47-S4 | Supervisor lifecycle wiring                  | P1       | Planned | M47       |
| M47-S5 | Runtime observability artifacts              | P1       | Planned | M47       |
| M47-S6 | Persistent-mode regression and integration   | P0       | Planned | M47       |
