# Feature Spec: Runtime File Inspection (AOP Dashboard)

> Purpose: Enable direct inspection of `.aop/runtime` artifacts (Operation Ledger and Worker Events) through the dashboard with time-sorted, formatted views per feature.

**Version:** 1.0
**Date:** 2026-03-05
**Status:** Outstanding
**Roadmap Mapping:** M46
**Depends On:** M45 (Dashboard Advanced UX)

---

## 0. Scope and Standards

### 0.1 Primary User Job

**Operator/Debugger Job:** Understand what happened during a feature run by inspecting:

- Operation Ledger: idempotent tool invocations with request/response payloads
- Worker Events: agent lifecycle events (started, completed, failed, timeout)

**Key Questions This Feature Answers:**

1. What tools were called for this feature and what were the results?
2. When did the agent start/stop/fail/timeout?
3. What was the sequence of events leading to a failure?
4. Were there any retries or duplicate operations?

### 0.2 UX Principles

1. **Time-first ordering:** All events sorted by timestamp (newest first by default)
2. **Progressive disclosure:** Raw JSON collapsed by default, formatted summary visible
3. **Feature-scoped:** Filter to single feature ID, not global view
4. **Read-only:** No mutation of runtime files through UI
5. **Performance-bounded:** Limit display to last N events, paginate if needed

### 0.3 Quality Standards

- TypeScript strict mode passes
- ESLint zero warnings
- Coverage >= 90% for new code
- API response time < 200ms for typical feature (< 100 events)
- Graceful degradation when files are large (> 1000 events)
- Must pass `npm run verify` checks

---

## 1. Runtime File Structure

### 1.1 Run ID Tracking in Feature State

**Schema Change Required:** Add `run_history` field to `state.schema.json`

**Location:** `.aop/features/<feature_id>/state.md` (YAML frontmatter)

**New Field:**

```yaml
run_history:
  - run_id: 'run:8efe99b2-bba0-473a-ad2e-22c54153fdd8'
    started_at: '2026-03-05T22:24:39.635Z'
    ended_at: '2026-03-05T23:00:09.599Z'
    status: 'failed'
  - run_id: 'run:69c36d92-3255-435a-806a-6425461e5aa8'
    started_at: '2026-03-05T06:05:12.123Z'
    ended_at: '2026-03-05T06:07:17.975Z'
    status: 'completed'
```

**Schema Definition:**

```json
{
  "run_history": {
    "type": "array",
    "description": "Time-sorted list of run IDs associated with this feature (newest first). Updated by supervisor on run start/end.",
    "items": {
      "type": "object",
      "required": ["run_id", "started_at"],
      "properties": {
        "run_id": {
          "type": "string",
          "description": "Run ID from runtime_sessions"
        },
        "started_at": {
          "type": "string",
          "format": "date-time",
          "description": "When this run started working on this feature"
        },
        "ended_at": {
          "type": "string",
          "format": "date-time",
          "description": "When this run stopped working on this feature (null if still active)"
        },
        "status": {
          "type": "string",
          "enum": ["active", "completed", "failed", "timeout", "interrupted"],
          "description": "Final status of this run for this feature"
        }
      }
    }
  }
}
```

**Update Behavior:**

- **On feature.init:** Create `run_history` with current run_id, status `active`
- **On supervisor start:**
  1. Read existing `run_history`.
  2. If the latest entry (index 0) has `status: 'active'` or `ended_at: null`, mark it as `status: 'interrupted'` and set `ended_at` to the current timestamp. This handles crashes or manual restarts where the previous run didn't exit cleanly.
  3. Append the _new_ current run_id to `run_history` with `status: 'active'`.
- **On supervisor stop/complete:** Update latest run_history entry with `ended_at` and final `status` (`completed`, `failed`, or `timeout`).
- **On feature.delete:** Archive run_history before deletion (optional)

### 1.2 Operation Ledger Files

**Location:** `.aop/runtime/operation-ledger/run:<run_id>.json`

**Structure:**

```json
{
  "run_id": "run:69c36d92-3255-435a-806a-6425461e5aa8",
  "updated_at": "2026-03-05T06:07:17.975Z",
  "operations": {
    "<operation_id>": {
      "operation_id": "feature_delete__my_feature__uuid",
      "tool_name": "feature.delete",
      "request_hash": "sha256...",
      "response": {
        "ok": true,
        "data": { ... }
      }
    }
  }
}
```

**Key Fields:**

- `operation_id`: Unique ID for idempotency tracking
- `tool_name`: MCP tool invoked (e.g., `feature.init`, `plan.submit`, `gates.run`)
- `request_hash`: SHA256 of request payload
- `response`: Full tool response with `ok` and `data`/`error`

### 1.3 Worker Events Files

**Location:** `.aop/runtime/worker-events/run:<run_id>.jsonl`

**Structure:** JSONL (one JSON object per line)

```json
{"ts":"2026-03-05T22:24:39.635Z","run_id":"run:8efe99b2...","feature_id":"my_feature","role":"planner","phase":"execution","event_type":"worker_started","output_types":[],"patch_count":0,"plan_submission_count":0,"request_count":0,"note_count":0,"valid":true,"error_code":null,"provider":"claude","model":"local-default","watchdog_reason":null,"signal":null}
{"ts":"2026-03-05T22:25:17.441Z","run_id":"run:8efe99b2...","feature_id":"my_feature","role":"planner","phase":"execution","event_type":"worker_completed","output_types":["PLAN_SUBMISSION"],"patch_count":0,"plan_submission_count":1,"request_count":0,"note_count":0,"valid":true,"error_code":null,"provider":"claude","model":"local-default","watchdog_reason":null,"signal":null}
```

**Key Fields:**

- `ts`: ISO 8601 timestamp
- `feature_id`: Feature this event belongs to
- `role`: Agent role (`planner`, `builder`, `qa`)
- `event_type`: `worker_started`, `worker_completed`, `worker_failed`, `worker_timeout`
- `valid`: Boolean indicating success/failure
- `error_code`: Error code if `valid=false`
- `watchdog_reason`: Timeout/failure reason

---

## 2. API Contract

**Note:** The API routes in `packages/web-dashboard` must use Node.js `fs` module to directly read artifacts from the `.aop` directory. Ensure proper error handling for file access.

### 2.1 Operation Ledger Endpoint

**Route:** `GET /api/runtime/operations`

**Query Parameters:**

- `feature_id` (required): Feature ID to filter operations
- `run_id` (optional): Specific run ID (defaults to latest from `run_history`)
- `project` (optional): Project name for multi-project setups
- `limit` (optional, default 100): Max operations to return
- `offset` (optional, default 0): Pagination offset

**Run ID Resolution:**

1. If `run_id` provided, use it directly
2. Else read `.aop/features/<feature_id>/state.md`
3. Extract `run_history[0].run_id` (latest run)
4. If `run_history` is empty or missing, return `run_not_found`

**Response Envelope:**

```typescript
{
  ok: true,
  data: {
    run_id: string;
    feature_id: string;
    updated_at: string;
    run_info: {
      started_at: string;
      ended_at: string | null;
      status: 'active' | 'completed' | 'failed' | 'timeout' | 'interrupted';
    };
    operations: Array<{
      operation_id: string;
      tool_name: string;
      request_hash: string;
      timestamp: string; // extracted from operation_id or response
      response: {
        ok: boolean;
        data?: unknown;
        error?: {
          code: string;
          message: string;
          details?: unknown;
        };
      };
    }>;
    total_count: number;
    has_more: boolean;
  },
  metadata: {
    source: 'runtime_ledger';
  }
}
```

**Error Codes:**

- `feature_not_found`: Feature ID not found in index
- `run_not_found`: No run_history found for feature
- `ledger_not_found`: Operation ledger file missing for run_id
- `ledger_parse_error`: JSON parse failure

### 2.2 Worker Events Endpoint

**Route:** `GET /api/runtime/events`

**Query Parameters:**

- `feature_id` (required): Feature ID to filter events
- `run_id` (optional): Specific run ID (defaults to latest from `run_history`)
- `project` (optional): Project name for multi-project setups
- `limit` (optional, default 100): Max events to return
- `offset` (optional, default 0): Pagination offset
- `event_type` (optional): Filter by event type (comma-separated)
- `role` (optional): Filter by role (comma-separated)

**Run ID Resolution:**

1. If `run_id` provided, use it directly
2. Else read `.aop/features/<feature_id>/state.md`
3. Extract `run_history[0].run_id` (latest run)
4. If `run_history` is empty or missing, return `run_not_found`

**Response Envelope:**

```typescript
{
  ok: true,
  data: {
    run_id: string;
    feature_id: string;
    run_info: {
      started_at: string;
      ended_at: string | null;
      status: 'active' | 'completed' | 'failed' | 'timeout' | 'interrupted';
    };
    events: Array<{
      ts: string;
      run_id: string;
      feature_id: string;
      role: string;
      phase: string;
      event_type: string;
      output_types: string[];
      patch_count: number;
      plan_submission_count: number;
      request_count: number;
      note_count: number;
      valid: boolean;
      error_code: string | null;
      provider: string;
      model: string;
      watchdog_reason: string | null;
      signal: string | null;
      exit_code?: number;
      timeout_ms?: number;
      elapsed_ms?: number;
    }>;
    total_count: number;
    has_more: boolean;
  },
  metadata: {
    source: 'runtime_events';
  }
}
```

**Error Codes:**

- `feature_not_found`: Feature ID not found in index
- `run_not_found`: No run_history found for feature
- `events_not_found`: Worker events file missing for run_id
- `events_parse_error`: JSONL parse failure

### 2.3 Run History Endpoint (New)

**Route:** `GET /api/runtime/runs`

**Query Parameters:**

- `feature_id` (required): Feature ID to get run history
- `project` (optional): Project name for multi-project setups

**Response Envelope:**

```typescript
{
  ok: true,
  data: {
    feature_id: string;
    runs: Array<{
      run_id: string;
      started_at: string;
      ended_at: string | null;
      status: 'active' | 'completed' | 'failed' | 'timeout' | 'interrupted';
      has_operations: boolean;
      has_events: boolean;
    }>;
  },
  metadata: {
    source: 'feature_state';
  }
}
```

**Purpose:** Allow UI to show run selector dropdown with all historical runs for a feature

**Error Codes:**

- `feature_not_found`: Feature ID not found in index
- `state_not_found`: Feature state file missing
- `state_parse_error`: YAML parse failure

---

## 3. UI Components

### 3.1 Runtime Inspector Panel (New Component)

**File:** `packages/web-dashboard/src/components/runtime-inspector.tsx`

**Type:** Client Component (`"use client"`)

**Location:** Appears in Feature Detail Panel (right side) as a new tab

**Tabs:**

1. **Operations** - Operation Ledger view
2. **Events** - Worker Events view

**Run Selector:**

- Dropdown at top of panel showing all runs from `run_history`
- Format: `Run <short_id> - <started_at> - <status>`
- Default: Latest run (index 0)
- Fetches from `GET /api/runtime/runs?feature_id=<id>`

**Default State:**

- Operations tab selected
- Latest run selected
- Last 50 operations shown
- Newest first (reverse chronological)
- Raw JSON collapsed

**Interaction:**

- Select run from dropdown → refetch operations/events for that run
- Click operation row to expand/collapse JSON
- Copy button for operation_id and request_hash
- Filter by tool_name (dropdown)
- Filter by success/failure (toggle)
- Pagination controls at bottom

**Error & Loading States:**

- **Loading:** Display skeleton rows while fetching data.
- **Error:** Display user-friendly error message ("Failed to load runtime data") with a "Retry" button.

### 3.2 Operations List View

**Requirements:**

**Summary Row (Always Visible):**

```
[Icon] tool_name | timestamp | status_badge | expand_icon
```

**Status Badge:**

- ✅ Success (green) - `response.ok === true`
- ❌ Failed (red) - `response.ok === false`
- ⚠️ Partial (yellow) - `response.ok === true` but `response.data` contains warnings

**Expanded View:**

```
Operation ID: <operation_id> [Copy]
Request Hash: <request_hash> [Copy]
Tool: <tool_name>
Timestamp: <formatted_timestamp>

Response:
<formatted_json_with_syntax_highlighting>
```

**Formatting:**

- Use `JSON.stringify(response, null, 2)` for readability
- Syntax highlighting for JSON (optional, use `<pre>` with CSS if no library)
- Max height 400px with scroll for large responses

### 3.3 Events Timeline View

**Requirements:**

**Timeline Row (Always Visible):**

```
[timestamp] [role_badge] [event_type_icon] event_type | duration | status
```

**Role Badge:**

- 🎯 Planner (blue)
- 🔨 Builder (orange)
- ✅ QA (green)

**Event Type Icons:**

- ▶️ `worker_started`
- ✅ `worker_completed`
- ❌ `worker_failed`
- ⏱️ `worker_timeout`

**Duration Calculation:**

- For `worker_completed`/`worker_failed`/`worker_timeout`: show `elapsed_ms` if present
- For `worker_started`: calculate duration to next event for same role

**Expanded View:**

```
Timestamp: <formatted_timestamp>
Run ID: <run_id>
Feature: <feature_id>
Role: <role>
Phase: <phase>
Event Type: <event_type>
Valid: <valid>
Error Code: <error_code>
Provider: <provider>
Model: <model>
Watchdog Reason: <watchdog_reason>
Signal: <signal>
Exit Code: <exit_code>
Timeout: <timeout_ms>ms
Elapsed: <elapsed_ms>ms

Output Types: <output_types.join(', ')>
Patch Count: <patch_count>
Plan Submissions: <plan_submission_count>
Requests: <request_count>
Notes: <note_count>
```

### 3.4 Filter Controls

**Operations Filters:**

- Tool Name (dropdown, multi-select)
- Status (Success/Failed/All)
- Time Range (Last Hour/Last Day/All)

**Events Filters:**

- Role (Planner/Builder/QA/All)
- Event Type (Started/Completed/Failed/Timeout/All)
- Valid Only (toggle)
- Time Range (Last Hour/Last Day/All)

**Filter Persistence:**

- Store in URL query params
- Restore on page load
- Clear filters button

---

## 4. Implementation Details

### 4.0 Schema and State Management Changes

**File:** `agentic/orchestrator/schemas/state.schema.json`

**Add to properties:**

```json
{
  "run_history": {
    "type": "array",
    "description": "Time-sorted list of run IDs associated with this feature (newest first). Updated by supervisor on run start/end.",
    "default": [],
    "items": {
      "type": "object",
      "required": ["run_id", "started_at"],
      "additionalProperties": false,
      "properties": {
        "run_id": {
          "type": "string",
          "pattern": "^run:[a-f0-9-]+$",
          "description": "Run ID from runtime_sessions"
        },
        "started_at": {
          "type": "string",
          "format": "date-time",
          "description": "When this run started working on this feature"
        },
        "ended_at": {
          "type": ["string", "null"],
          "format": "date-time",
          "description": "When this run stopped working on this feature (null if still active)"
        },
        "status": {
          "type": "string",
          "enum": ["active", "completed", "failed", "timeout", "interrupted"],
          "description": "Final status of this run for this feature"
        }
      }
    }
  }
}
```

**File:** `apps/control-plane/src/application/services/feature-state-service.ts`

**Changes:**

- Add `appendRunHistory(featureId, runId, startedAt)` method
- Add `updateRunHistory(featureId, runId, endedAt, status)` method
- Ensure run_history is sorted newest-first on write

**File:** `apps/control-plane/src/supervisor/runtime.ts`

**Changes:**

- **On feature start:**
  - Retrieve current feature state.
  - Check `run_history[0]`. If it exists and `status === 'active'` or `ended_at` is null:
    - Call `featureStateService.updateRunHistory(featureId, oldRunId, new Date().toISOString(), 'interrupted')`
  - Call `featureStateService.appendRunHistory(featureId, this.runId, new Date().toISOString())`
- **On feature complete/fail:** Call `featureStateService.updateRunHistory(featureId, this.runId, new Date().toISOString(), status)`
- **On recovery:** Check if current run_id exists in run_history, append if missing.

**Migration Strategy:**

- Existing features without run_history: Initialize as empty array `[]`
- No data loss: run_history is additive only
- Backward compatible: Old state files without run_history still validate (default: `[]`)

### 4.1 Backend Implementation

**File:** `packages/web-dashboard/src/app/api/runtime/runs/route.ts` (New)

**Logic:**

1. Resolve `feature_id` and `project` from query params
2. Read `.aop/features/<feature_id>/state.md` using `fs`
3. Parse YAML frontmatter to extract `run_history`
4. For each run, check if ledger/events files exist (using `fs.existsSync`)
5. Return formatted response with run list

**File:** `packages/web-dashboard/src/app/api/runtime/operations/route.ts`

**Logic:**

1. Resolve `feature_id` and `project` from query params
2. If `run_id` provided, use it; else read state.md and extract `run_history[0].run_id`
3. If no run_history, return `run_not_found`
4. Read `.aop/runtime/operation-ledger/run:<run_id>.json` using `fs`
5. Parse JSON and extract operations
6. Filter operations by `feature_id` (check operation_id contains feature_id)
7. Sort by timestamp (extract from operation_id or use file `updated_at`)
8. Apply pagination (limit/offset)
9. Return formatted response with run_info from run_history

**File:** `packages/web-dashboard/src/app/api/runtime/events/route.ts`

**Logic:**

1. Resolve `feature_id` and `project` from query params
2. If `run_id` provided, use it; else read state.md and extract `run_history[0].run_id`
3. If no run_history, return `run_not_found`
4. Read `.aop/runtime/worker-events/run:<run_id>.jsonl` using `fs`
5. Parse JSONL (split by newline, parse each line as JSON)
6. Filter events by `feature_id`
7. Apply filters (event_type, role, valid)
8. Sort by `ts` (descending for newest first)
9. Apply pagination (limit/offset)
10. Return formatted response with run_info from run_history

**Error Handling:**

- Catch file read errors → return `ledger_not_found` or `events_not_found`
- Catch JSON parse errors → return `ledger_parse_error` or `events_parse_error`
- Handle missing run_history → return `run_not_found`
- Handle large files (> 10MB) → return `file_too_large` with suggestion to use CLI

### 4.2 Frontend Implementation

**File:** `packages/web-dashboard/src/components/runtime-inspector.tsx`

**State Management:**

```typescript
interface RuntimeInspectorState {
  activeTab: 'operations' | 'events';
  selectedRunId: string | null;
  runs: RunHistoryEntry[];
  operations: Operation[];
  events: WorkerEvent[];
  loading: boolean;
  error: string | null;
  filters: {
    toolName?: string[];
    status?: 'success' | 'failed' | 'all';
    eventType?: string[];
    role?: string[];
    validOnly?: boolean;
    timeRange?: 'hour' | 'day' | 'all';
  };
  pagination: {
    limit: number;
    offset: number;
    totalCount: number;
    hasMore: boolean;
  };
  expandedIds: Set<string>;
}

interface RunHistoryEntry {
  run_id: string;
  started_at: string;
  ended_at: string | null;
  status: 'active' | 'completed' | 'failed' | 'timeout' | 'interrupted';
  has_operations: boolean;
  has_events: boolean;
}
```

**Hooks:**

- `useRunHistory(featureId)` - Fetch run history list
- `useRuntimeOperations(featureId, runId, filters, pagination)` - Fetch operations
- `useRuntimeEvents(featureId, runId, filters, pagination)` - Fetch events
- `useExpandedState()` - Manage expanded/collapsed rows

**Component Lifecycle:**

1. Mount → fetch run history
2. Run history loaded → select latest run (index 0)
3. Run selected → fetch operations/events for that run
4. Run selector change → refetch operations/events

**Performance:**

- Virtualize lists if > 100 items (use `react-window` or similar)
- Debounce filter changes (300ms)
- Cancel in-flight requests on run/filter change
- Cache responses per (feature_id, run_id) pair (60s TTL)

### 4.3 Integration with Detail Panel

**File:** `packages/web-dashboard/src/components/detail-panel.tsx`

**Changes:**

1. Add new tab: "Runtime" (after "Evidence" tab)
2. Render `<RuntimeInspector featureId={selectedFeature.feature_id} />` when tab is active
3. Badge on tab shows total operation count (if available)

**Tab Order:**

1. Overview
2. Plan
3. Gates
4. Evidence
5. **Runtime** (new)
6. Logs

---

## 5. Testing Strategy

### 5.1 Unit Tests

**Schema Tests:**

- `schemas/state.schema.spec.ts`
  - Validates run_history field structure
  - Rejects invalid run_history entries
  - Allows empty run_history array
  - Validates date-time formats

**Backend Tests:**

- `api/runtime/runs/route.spec.ts`
  - Returns run history for valid feature_id
  - Returns 404 for missing feature
  - Returns 404 for missing state file
  - Handles YAML parse errors
  - Checks file existence for operations/events

- `api/runtime/operations/route.spec.ts`
  - Returns operations for valid feature_id
  - Uses latest run_id when not specified
  - Uses specified run_id when provided
  - Returns 404 for missing feature
  - Returns 404 for missing run_history
  - Returns 404 for missing ledger file
  - Handles JSON parse errors
  - Applies pagination correctly
  - Filters by tool_name
  - Sorts by timestamp descending

- `api/runtime/events/route.spec.ts`
  - Returns events for valid feature_id
  - Uses latest run_id when not specified
  - Uses specified run_id when provided
  - Returns 404 for missing feature
  - Returns 404 for missing run_history
  - Returns 404 for missing events file
  - Handles JSONL parse errors
  - Applies pagination correctly
  - Filters by event_type and role
  - Sorts by timestamp descending

**Frontend Tests:**

- `components/runtime-inspector.spec.tsx`
  - Renders operations tab by default
  - Fetches run history on mount
  - Selects latest run by default
  - Switches between tabs
  - Changes selected run
  - Refetches data on run change
  - Expands/collapses operation rows
  - Applies filters
  - Handles pagination
  - Shows loading state
  - Shows error state
  - Copies operation_id to clipboard

### 5.2 Integration Tests

**End-to-End:**

1. Create feature with multiple runs
2. Open feature detail panel
3. Click "Runtime" tab
4. Verify run selector appears with all runs
5. Verify latest run is selected
6. Verify operations list appears
7. Select older run from dropdown
8. Verify operations list updates
9. Expand operation row
10. Verify JSON response is visible
11. Switch to Events tab
12. Verify events timeline appears
13. Apply filter (e.g., "Failed only")
14. Verify filtered results

### 5.3 Performance Tests

**Benchmarks:**

- Load run history (10 runs) in < 50ms
- Load 100 operations in < 200ms
- Load 100 events in < 200ms
- Render 100 operations in < 100ms
- Filter 100 operations in < 50ms
- Pagination in < 50ms
- Run selector change in < 100ms

**Stress Tests:**

- Handle 50 runs in history gracefully
- Handle 1000 operations gracefully (virtualization)
- Handle 1000 events gracefully (virtualization)
- Handle 10MB ledger file (return error with suggestion)

---

## 6. Acceptance Criteria

### 6.1 Must Have

- [ ] Schema change: Add `run_history` field to `state.schema.json`
- [ ] Supervisor: Update run_history on feature start/stop
- [ ] Supervisor: Correctly mark interrupted runs on startup
- [ ] FeatureStateService: Support run_history read/write
- [ ] `GET /api/runtime/runs` endpoint implemented with `fs` access
- [ ] `GET /api/runtime/operations` endpoint implemented with `fs` access
- [ ] `GET /api/runtime/events` endpoint implemented with `fs` access
- [ ] Run ID resolution from run_history (latest by default)
- [ ] RuntimeInspector component renders in detail panel
- [ ] Run selector dropdown with all historical runs
- [ ] Operations tab shows last 50 operations by default
- [ ] Events tab shows last 50 events by default
- [ ] Operations sorted by timestamp (newest first)
- [ ] Events sorted by timestamp (newest first)
- [ ] Expand/collapse operation rows
- [ ] Expand/collapse event rows
- [ ] Copy operation_id and request_hash
- [ ] Filter operations by tool_name
- [ ] Filter operations by success/failure
- [ ] Filter events by role
- [ ] Filter events by event_type
- [ ] Pagination controls (Next/Previous)
- [ ] Loading state
- [ ] Error state with retry button
- [ ] Empty state ("No operations recorded")
- [ ] All tests pass (>= 90% coverage)
- [ ] TypeScript strict mode passes
- [ ] ESLint zero warnings

### 6.2 Should Have

- [ ] Syntax highlighting for JSON responses
- [ ] Duration calculation for events
- [ ] Time range filters (Last Hour/Last Day/All)
- [ ] Export operations to JSON file
- [ ] Export events to JSON file
- [ ] Search within operation responses
- [ ] Keyboard navigation (arrow keys to expand/collapse)

### 6.3 Nice to Have

- [ ] Real-time updates via SSE (append new operations/events)
- [ ] Diff view for plan.update operations
- [ ] Link from operation to related feature state change
- [ ] Flame graph for operation timing
- [ ] Event correlation (link worker_started to worker_completed)

---

## 7. Non-Goals

- **Mutation of runtime files:** This feature is read-only inspection
- **Global runtime view:** Always scoped to single feature
- **Historical run comparison:** Only shows latest run (or specified run_id)
- **Real-time streaming:** Initial version uses polling/manual refresh
- **Advanced analytics:** No aggregation, charts, or trend analysis (use Analytics page)

---

## 8. Rollout Plan

### 8.1 Phase 0: Schema and State Management (Week 1)

- Add `run_history` field to `state.schema.json`
- Update FeatureStateService to read/write run_history
- Update SupervisorRuntime to append run_id on feature start
- Update SupervisorRuntime to update run status on feature end
- Implement robust crash detection (mark interrupted runs)
- Add migration for existing features (backfill empty run_history)
- Add unit tests for schema validation
- Add unit tests for state service

### 8.2 Phase 1: Backend API (Week 2)

- Implement `/api/runtime/runs` endpoint
- Implement `/api/runtime/operations` endpoint with run_history resolution
- Implement `/api/runtime/events` endpoint with run_history resolution
- Add unit tests
- Add integration tests
- Document API in README

### 8.3 Phase 2: Frontend UI (Week 3)

- Implement RuntimeInspector component
- Add run selector dropdown
- Add Operations tab
- Add Events tab
- Add filter controls
- Add pagination
- Integrate with detail panel

### 8.4 Phase 3: Polish (Week 4)

- Add syntax highlighting
- Add duration calculations
- Add export functionality
- Add keyboard navigation
- Performance optimization
- Accessibility audit

### 8.5 Phase 4: Documentation (Week 4)

- Update state.schema.json documentation
- Update dashboard README
- Add screenshots to docs
- Create user guide
- Update CLAUDE.md and AGENTS.md

---

## 9. Success Metrics

### 9.1 Functional Metrics

- API response time < 200ms (p95)
- UI render time < 100ms (p95)
- Zero crashes on large files (< 10MB)
- 100% test coverage for new code
- 100% accurate identification of interrupted/crashed runs

### 9.2 User Metrics

- Time to find failure cause reduced by 50%
- Number of "What happened?" questions reduced
- Operator satisfaction score > 4/5

---

## 10. References

### 10.1 Related Documents

- `spec-files/completed/agentic_orchestrator_dashboard_advanced_ux_spec.md` - Dashboard UX standards
- `README.md` - CLI and tool contracts
- `CLAUDE.md` - Architecture patterns

### 10.2 Related Code

- `packages/web-dashboard/src/components/detail-panel.tsx` - Feature detail panel
- `packages/web-dashboard/src/lib/aop-client.ts` - API client utilities
- `apps/control-plane/src/mcp/tool-runtime.ts` - Operation ledger writer
- `apps/control-plane/src/supervisor/runtime.ts` - Worker events writer

---

## 11. Implementation Tasks

## Phase 1: Foundational State Management

> **Goal:** Ensure all feature runs are reliably tracked in `feature_state.run_history` to enable historical inspection.

- [ ] T001 [P] Implement `appendRunHistory` method in `apps/control-plane/src/application/services/feature-state-service.ts` to add new runs to the history list.
- [ ] T002 [P] Implement `updateRunHistory` method in `apps/control-plane/src/application/services/feature-state-service.ts` to update status/end time of existing runs.
- [ ] T003 Update `SupervisorRuntime` in `apps/control-plane/src/supervisor/runtime.ts` to call `appendRunHistory` on startup for the current run.
- [ ] T004 Update `SupervisorRuntime` in `apps/control-plane/src/supervisor/runtime.ts` to call `updateRunHistory` on graceful shutdown or completion.
- [ ] T005 [P] Update `FeatureLifecycleService` in `apps/control-plane/src/application/services/feature-lifecycle-service.ts` to initialize `run_history` with the creation run ID during `feature.init`.
- [ ] T006 Verify state updates by inspecting `.aop/features/<feature_id>/state.md` after a supervisor run.

## Phase 2: Backend API Endpoints

> **Goal:** Expose runtime artifacts (ledger, events, run history) to the frontend via Next.js API routes.

- [ ] T007 [P] Create utility `packages/web-dashboard/src/lib/runtime-files.ts` to resolve paths and read JSON/JSONL files from `.aop/runtime`.
- [ ] T008 [P] [US1] Create API route `GET /api/runtime/runs` in `packages/web-dashboard/src/app/api/runtime/runs/route.ts` to return run history for a feature.
- [ ] T009 [P] [US1] Create API route `GET /api/runtime/operations` in `packages/web-dashboard/src/app/api/runtime/operations/route.ts` to return operation ledger data.
- [ ] T010 [P] [US1] Create API route `GET /api/runtime/events` in `packages/web-dashboard/src/app/api/runtime/events/route.ts` to return worker events (parsing JSONL).
- [ ] T011 [US1] Implement run ID resolution logic in API routes: if `run_id` is missing, fetch latest from feature state.
- [ ] T012 [P] Add error handling for missing files (404) and permission errors in all runtime API routes.

## Phase 3: UI Components & Integration

> **Goal:** Visualize the runtime data in the dashboard with a user-friendly inspector panel.

- [ ] T013 [P] [US1] Create `RuntimeInspectorPanel` container component in `packages/web-dashboard/src/components/features/runtime-inspector/RuntimeInspectorPanel.tsx`.
- [ ] T014 [P] [US1] Create `OperationsListView` component in `packages/web-dashboard/src/components/features/runtime-inspector/OperationsListView.tsx` to display tool calls.
- [ ] T015 [P] [US1] Create `EventsTimelineView` component in `packages/web-dashboard/src/components/features/runtime-inspector/EventsTimelineView.tsx` to display worker events.
- [ ] T016 [US1] Implement run selection dropdown in `RuntimeInspectorPanel` to switch between historical runs.
- [ ] T017 [US1] Add "Load More" or pagination support for large event logs (progressive loading).
- [ ] T018 [US1] Integrate `RuntimeInspectorPanel` into the Feature Detail page (`packages/web-dashboard/src/app/features/[id]/page.tsx` or equivalent).

## Phase 4: Polish & UX Refinements

> **Goal:** Ensure the inspector is performant, responsive, and handles edge cases gracefully.

- [ ] T019 [P] Add collapsible JSON views for request/response payloads in `OperationsListView`.
- [ ] T020 [P] Add filtering controls (by event type: `worker_started`, `worker_failed`, etc.) to `EventsTimelineView`.
- [ ] T021 [P] specific styling for error states (red highlights for failed operations/events).
- [ ] T022 [P] Ensure timestamp formatting is consistent (local time vs UTC toggle if needed).
- [ ] T023 Run `npm run verify` to ensure all new code meets quality standards (linting, types).

## Dependencies

1. **Foundational State Management** must complete before **Backend API Endpoints** (specifically for `runs` endpoint).
2. **Backend API Endpoints** must complete before **UI Components**.
3. **UI Components** integration depends on the existence of the Feature Detail page.

## Implementation Strategy

- **MVP**: Focus on the _latest_ run first. Get the `run_history` tracking working, then the APIs for the current run, then the UI.
- **Incremental**: You can build the API routes in parallel with the State Management updates, using mock data or manually edited state files for testing.
