# Feature Spec: Observability — Replay Timeline Foundation (AOP)

> **Split from**: `agentic_orchestrator_observability_integrity_diagnostics_spec.md` (v2.0)
> **Covers**: OID0 (full shared scaffolding) + OID1 (Q3 Replay Timeline) + Replay CLI surface
> **Roadmap Mapping:** M33 / Q3

**Version:** 1.1
**Date:** 2026-03-15
**Status:** Implemented in Repo (Retained Here Until Archived)

---

## 0. Standards and Dependencies

### 0.1 Required Standards

All implementation MUST preserve:

- deterministic tool behavior and normalized error envelopes
- MCP/in-process transport parity for all new tools
- strict contract validation (`validate:mcp-contracts`) and architecture boundaries (`validate:architecture`)
- runtime artifact ownership under `.aop/**`
- existing idempotency semantics (`operation_id` replay/mismatch) for mutating tools

### 0.2 Required Upstream Inputs

Implementing agents MUST read before writing any code:

- `apps/control-plane/src/mcp/tool-runtime.ts`
- `apps/control-plane/src/mcp/operation-ledger.ts`
- `apps/control-plane/src/core/kernel.ts`
- `apps/control-plane/src/core/path-layout.ts`
- `apps/control-plane/src/core/constants.ts`
- `apps/control-plane/src/core/error-codes.ts`
- `apps/control-plane/src/application/services/gate-service.ts`
- `apps/control-plane/src/cli/types.ts`
- `apps/control-plane/src/cli/cli-argument-parser.ts`
- `apps/control-plane/src/cli/help-command-handler.ts`
- `apps/control-plane/src/interfaces/cli/bootstrap.ts`
- `apps/control-plane/src/application/tools/tool-metadata.ts`
- `agentic/orchestrator/defaults/policy.defaults.yaml`
- `agentic/orchestrator/schemas/policy.schema.json`
- `agentic/orchestrator/schemas/policy.user.schema.json`
- `agentic/orchestrator/tools/catalog.json`
- `apps/control-plane/scripts/validate-mcp-contracts.ts`

### 0.3 Feature Scope

This spec implements:

- **OID0**: Shared scaffolding (policy schema, error codes, tool catalog stubs, path-layout helpers) for ALL observability tools including those in the companion spec `agentic_orchestrator_evidence_integrity_doctor_spec`
- **OID1**: Q3 Deterministic Replay + Time-Travel Diagnostics
- **Replay CLI**: `aop replay` command surface

The companion spec `agentic_orchestrator_evidence_integrity_doctor_spec` MUST NOT start OID2/OID3 until this spec's OID0 is merged, as it relies on the policy schema extensions and error codes landed here.

Out of scope for this spec:

- evidence integrity chain (OID2 — see companion spec)
- doctor diagnostics (OID3 — see companion spec)
- evidence/doctor CLI surface (see companion spec)

---

## 1. Architectural Context

### 1.1 Existing Runtime Touchpoints

- **Tool execution boundary**: `ToolRuntime.callTool()` is the authoritative event point for replay timeline instrumentation (covers in-process and MCP equally).
- **Idempotency source of truth**: `OperationLedger` already determines `new|replay|mismatch` for mutating operations. The replay writer reads this result from the response envelope.
- **Evidence write path**: `GateService.gatesRun()` is not touched by this spec.

### 1.2 Boundary Rules (Normative)

- `core/*` remains pure infrastructure/domain helpers; orchestration/policy decisions live in `application/services/*`.
- Replay timeline emission belongs in the MCP runtime layer (`tool-runtime`) to guarantee transport parity.
- All behavior toggles must be schema-backed and default-safe.
- Replay writes are always best-effort by default — they must never cause a tool call to fail.

---

## 2. Contracts and Schema Additions

### 2.1 Policy Extensions (All Observability Toggles — Shared with Companion Spec)

Add to `agentic/orchestrator/defaults/policy.defaults.yaml`:

```yaml
innovation:
  replay_debugger: true
  tamper_evident_evidence: false

observability:
  replay:
    enabled: true
    write_mode: best_effort # best_effort | required
    retention:
      max_runs: 100
      max_entries_per_run: 50000
  evidence_integrity:
    enabled: false
    verify_on_read: false
    algorithm_version: 1

diagnostics:
  doctor:
    enabled: true
    strict_fails_on: critical # critical | warning
```

Normative behavior:

- `innovation.replay_debugger` AND `observability.replay.enabled` must both be `true` for timeline writes to occur.
- `innovation.tamper_evident_evidence` AND `observability.evidence_integrity.enabled` must both be `true` for chain writes (companion spec).
- `write_mode=best_effort`: timeline write failures are logged but never fail the parent tool call.
- `write_mode=required`: timeline write failure propagates as tool error.
- If `diagnostics.doctor.enabled=false`, doctor returns a report with a deterministic `doctor_disabled` info finding (companion spec).

Update `agentic/orchestrator/schemas/policy.schema.json` and `agentic/orchestrator/schemas/policy.user.schema.json` to accept all three new top-level sections (`innovation`, `observability`, `diagnostics`).

Update `apps/control-plane/src/cli/init-command-handler.ts` advanced policy template to include these sections with commented defaults.

### 2.2 Replay Artifact Contracts

Paths:

```text
.aop/runtime/replay/<run_id>.timeline.jsonl
.aop/runtime/replay/<run_id>.meta.json
```

Timeline entry contract (one JSON line per tool call):

```json
{
  "ts": "2026-03-04T00:00:00.000Z",
  "sequence": 42,
  "run_id": "run-123",
  "session_id": "session-abc",
  "actor_type": "builder",
  "feature_id": "feature_x",
  "tool_name": "repo.apply_patch",
  "mutating": true,
  "operation_id": "repo_apply_patch__feature_x__uuid",
  "idempotency_status": "new",
  "request_hash": "sha256:...",
  "result": "ok",
  "error_code": null,
  "latency_ms": 317
}
```

Meta file contract (`.meta.json`):

```json
{
  "run_id": "run-123",
  "sequence_head": 42,
  "entry_count": 43,
  "started_at": "2026-03-04T00:00:00.000Z",
  "last_entry_at": "2026-03-04T00:05:00.000Z"
}
```

Rules:

- `sequence` strictly increases per `run_id`, starting at 0.
- Write path is append-only JSONL; no in-place mutation of prior lines.
- `idempotency_status` must be one of `none|new|replay|mismatch`.
  - `none` for read-only tools (no `operation_id` involved).
  - `new|replay|mismatch` for mutating tools (resolved by `OperationLedger`).
- Ordering for reads is by `sequence` ascending, never filesystem mtime.
- `sequence` allocation and meta update are performed under a per-run file lock.

### 2.3 New Tool: `replay.timeline_get`

Read-only tool. Queries the timeline for a run.

Input schema (`replay.timeline_get.input.schema.json`):

```json
{
  "type": "object",
  "required": ["run_id"],
  "properties": {
    "run_id": { "type": "string", "minLength": 1 },
    "from_sequence": { "type": "integer", "minimum": 0 },
    "limit": { "type": "integer", "minimum": 1, "maximum": 10000 },
    "feature_id": { "type": "string" },
    "tool_name": { "type": "string" }
  }
}
```

Output schema: array of timeline entries matching the query, sorted by `sequence` ascending. Returns `replay_run_not_found` error if `<run_id>.timeline.jsonl` does not exist.

### 2.4 Error Code Additions

Add to `apps/control-plane/src/core/error-codes.ts` — **all observability error codes** (including those used by companion spec):

```
replay_run_not_found
replay_timeline_corrupt
replay_timeline_write_failed
evidence_chain_missing
evidence_chain_artifact_missing
evidence_chain_hash_mismatch
evidence_chain_discontinuity
evidence_integrity_write_failed
doctor_check_failed
doctor_report_write_failed
```

### 2.5 Tool Catalog Stubs for All Observability Tools

Add all three new tools to `agentic/orchestrator/tools/catalog.json` in OID0, even if OID2/OID3 services aren't implemented yet. This prevents catalog drift between the two specs and satisfies `validate:mcp-contracts` from the start.

Tools to stub:

- `replay.timeline_get` — fully implemented in this spec
- `evidence.verify_chain` — stubbed in catalog; implemented in companion spec
- `doctor.run` — stubbed in catalog; implemented in companion spec

Create all input/output schema files (even minimal stubs) for all three tools in OID0.

---

## 3. Detailed Implementation Plan

### OID0: Contract Scaffolding and Policy Wiring

**Goals**: Land all schema/default/error-code/catalog scaffolding before any service implementation. Companion spec cannot start until this is merged.

**File targets**:

- `agentic/orchestrator/defaults/policy.defaults.yaml` — add `innovation`, `observability`, `diagnostics` sections
- `agentic/orchestrator/schemas/policy.schema.json` — add new sections
- `agentic/orchestrator/schemas/policy.user.schema.json` — add new sections
- `apps/control-plane/src/cli/init-command-handler.ts` — advanced policy template alignment
- `apps/control-plane/src/core/constants.ts` — add `TOOLS.REPLAY_TIMELINE_GET`, `TOOLS.EVIDENCE_VERIFY_CHAIN`, `TOOLS.DOCTOR_RUN`
- `apps/control-plane/src/core/error-codes.ts` — add all 10 new error codes (§2.4)
- `agentic/orchestrator/tools/catalog.json` — add all three tool entries
- `agentic/orchestrator/tools/schemas/input/replay.timeline_get.input.schema.json` (new)
- `agentic/orchestrator/tools/schemas/output/replay.timeline_get.output.schema.json` (new)
- `agentic/orchestrator/tools/schemas/input/evidence.verify_chain.input.schema.json` (new stub)
- `agentic/orchestrator/tools/schemas/output/evidence.verify_chain.output.schema.json` (new stub)
- `agentic/orchestrator/tools/schemas/input/doctor.run.input.schema.json` (new stub)
- `agentic/orchestrator/tools/schemas/output/doctor.run.output.schema.json` (new stub)
- `agentic/orchestrator/tools.md` — regenerate

**Mandatory checks after OID0**:

```
npm run validate:mcp-contracts
npm run validate:architecture
npm run typecheck
```

**Exit criteria**:

- Policy composition validates cleanly with new fields
- All three tool schemas pass contract validator
- No contract validator drift

---

### OID1: Q3 Replay Timeline Service

**Design**: Instrument `ToolRuntime.callTool()` finalization path with deterministic timeline writes via a dedicated `ReplayTimelineService`.

**Algorithm**:

1. `ToolRuntime.callTool()` finishes executing the tool and has the response envelope.
2. Construct replay event: extract `run_id`, `session_id`, `actor_type`, `feature_id`, `tool_name`, `mutating`, `operation_id`, `idempotency_status`, `request_hash`, `result`, `error_code`, `latency_ms`.
3. `ReplayTimelineService.append(run_id, event)`:
   a. Acquire per-run file lock on `<run_id>.meta.json`.
   b. Read current `sequence_head` from meta (0 if meta doesn't exist).
   c. Assign `sequence = sequence_head + 1`.
   d. Append JSONL line to `<run_id>.timeline.jsonl`.
   e. Write updated meta with new `sequence_head` and `last_entry_at`.
   f. Release lock.
4. If `write_mode=best_effort` (default), catch and log all write errors; never propagate.
5. If `write_mode=required`, propagate write errors as `replay_timeline_write_failed`.

**`replay.timeline_get` read algorithm**:

1. Check for `<run_id>.timeline.jsonl`; return `replay_run_not_found` if missing.
2. Read all lines; parse each as JSON; skip malformed lines (log as warning).
3. Apply filters: `from_sequence`, `feature_id`, `tool_name`.
4. Sort by `sequence` ascending.
5. Apply `limit`.
6. Return filtered entries.

**Path-layout helpers** — add to `AopPathLayout` in `apps/control-plane/src/core/path-layout.ts`:

```
replayDir(): string           // .aop/runtime/replay/
replayTimelinePath(runId): string   // .aop/runtime/replay/<runId>.timeline.jsonl
replayMetaPath(runId): string       // .aop/runtime/replay/<runId>.meta.json
replayLockPath(runId): string       // .aop/runtime/replay/<runId>.lock
```

**`ReplayTimelineService` retention enforcement**:

After a successful timeline write, check if the total number of timeline files in `replayDir()` exceeds `observability.replay.retention.max_runs`. If so, delete the oldest timeline + meta files (by creation timestamp in meta) until within limit. This check is best-effort (skipped on error).

Per-run `max_entries_per_run` enforcement: if `sequence_head` exceeds the limit, skip the append and log a warning. Do not truncate existing entries.

**Implementation targets**:

- `apps/control-plane/src/application/services/replay-timeline-service.ts` (new)
- `apps/control-plane/src/mcp/tool-runtime.ts` — instrument `callTool` finalization
- `apps/control-plane/src/core/path-layout.ts` — add replay path helpers
- `apps/control-plane/src/core/kernel.ts` — register `replay.timeline_get` tool handler

**Behavioral rules**:

- Replay emission runs for both success and failure responses.
- `idempotency_status=replay` entries must still be written (diagnostic visibility).
- Replay writes must not block or slow tool execution (write after response is assembled).
- `run_id` must be injected into `ToolRuntime` at construction time (already available via `SupervisorRuntimeState`).

**Test plan**:

- `apps/control-plane/test/replay-timeline-service.spec.ts` (new):
  - sequence strictly monotonic under sequential writes
  - concurrent write simulation: sequence allocation is collision-free under file lock
  - best-effort mode: write failure does not throw
  - required mode: write failure throws `replay_timeline_write_failed`
  - retention: excess runs are pruned to `max_runs`
  - per-run cap: entries beyond `max_entries_per_run` are skipped
- Extend `apps/control-plane/test/tool-runtime.spec.ts`:
  - idempotency status correctly propagated to timeline entry
  - read-only tools get `idempotency_status=none`
- Extend `apps/control-plane/test/mcp.spec.ts`:
  - `replay.timeline_get` parity (in-process vs MCP transport)
  - `from_sequence` and `feature_id` filters work correctly
  - `replay_run_not_found` returned for unknown run

**Exit criteria**:

- Identical tool call sequence produces identical timeline lines (except `ts` and `latency_ms`)
- Sequence is strictly monotonic
- `replay.timeline_get` returns entries in sequence order

---

### OID4 (Replay Portion): `aop replay` CLI Command

Add `aop replay` command surface.

**CLI specification**:

```
aop replay --run-id <id> [--from-sequence N] [--limit N] [--feature-id <id>] [--tool <name>] [--json]
```

- Without `--json`: pretty-print entries as table or structured text.
- With `--json`: emit raw `replay.timeline_get` response payload.

**Required file changes** (per CLAUDE.md):

1. `apps/control-plane/src/cli/types.ts` — add `CliCommand.Replay`; add `runId`, `fromSequence`, `limit` option fields to `CliOptions`
2. `apps/control-plane/src/cli/cli-argument-parser.ts` — parse `replay` command and its flags
3. `apps/control-plane/src/interfaces/cli/bootstrap.ts` — dispatch `CliCommand.Replay` → `ReplayCommandHandler`
4. `apps/control-plane/src/cli/help-command-handler.ts` — add `replay` entry to `COMMAND_HELP`
5. `apps/control-plane/src/cli/replay-command-handler.ts` (new) — calls `replay.timeline_get` tool and formats output

**Test additions**:

- Extend `apps/control-plane/test/cli.unit.spec.ts`: parser coverage for `replay` + all flags
- `apps/control-plane/test/replay-command-handler.spec.ts` (new): handler dispatch, JSON mode, error passthrough

---

## 4. Test Strategy

### 4.1 Unit Tests

- Replay sequence allocation under concurrent lock contention
- `idempotency_status` mapping: `none` (read-only), `new/replay/mismatch` (mutating)
- Retention pruning: oldest file deleted when `max_runs` exceeded
- Per-run cap: entry skipped at `max_entries_per_run`

### 4.2 Integration Tests

- End-to-end run emits replay timeline entries for at least one `feature.init` and one `repo.apply_patch` call
- `replay.timeline_get` returns entries in correct order with correct `idempotency_status`

### 4.3 Transport Parity

- `replay.timeline_get` parity: in-process vs MCP transport produce identical response shapes

### 4.4 CLI Tests

- Parser coverage for `replay` command and all flags
- `--json` flag produces parseable JSON
- `--run-id` missing produces a clear error

### 4.5 Regression Tests

- Existing tool call behavior unchanged (replay writes are non-blocking)
- `validate:mcp-contracts` passes with new tool schemas

---

## 5. Acceptance Criteria

1. Replay timeline is persisted under `.aop/runtime/replay` and queryable by `run_id` with deterministic ordering by `sequence`.
2. Replay entries include `idempotency_status` and distinguish replayed mutations (`replay`) from fresh execution (`new`).
3. Best-effort write failures do not surface as tool errors.
4. `replay.timeline_get` tool is fully wired: constants / kernel / tool-metadata / catalog / schemas / RBAC / tools.md.
5. `aop replay` command is fully wired: types / parser / help / bootstrap / handler.
6. All companion spec tool stubs are in catalog and pass `validate:mcp-contracts`.
7. Quality gates pass: `lint`, `typecheck`, `test`, `validate:mcp-contracts`, `validate:architecture`.

---

## 6. Risks and Mitigations

- **Risk**: Timeline file corruption under concurrent tool calls.
  - **Mitigation**: Per-run file lock around sequence allocation and JSONL append.

- **Risk**: Performance regression from per-call timeline writes.
  - **Mitigation**: Append-only JSONL (no seeks), bounded retention, best-effort mode by default.

- **Risk**: Policy schema addition breaks existing repo validation.
  - **Mitigation**: All new fields are optional with safe defaults; `additionalProperties: false` must be relaxed or extended correctly.

- **Risk**: Companion spec starts before OID0 is merged, causing catalog drift.
  - **Mitigation**: Companion spec declares explicit dependency on this spec's OID0 milestone.

---

## 7. Definition of Done

A completed implementation session MUST provide:

- Changed files and rationale by phase (OID0 → OID1 → OID4-replay)
- Schema/catalog/RBAC deltas and regenerated `tools.md`
- Test additions and executed command results (`npm test`, `npm run lint`, `npm run validate:mcp-contracts`)
- `spec-files/progress.md` update entry covering OID0 and OID1 milestones
