# Architecture: agentic-orchestrator project

## 🎯 Quick Context (For AI Agents)

- **What it does**: A CLI control plane (`aop`) that orchestrates multiple LLM agents through a deterministic, plan-gated workflow: planner, builder, and QA run per feature, while a single shared reconciler session handles mainline-divergence repair.
- **Who it's for**: Engineering teams building or experimenting with fully-automated, multi-agent code generation pipelines that require reproducible, auditable outcomes.
- **Key constraint**: All state mutations go through atomic file locks, schema validation (AJV), and optimistic concurrency — non-determinism is never allowed to corrupt persisted state.

---

## 🏗️ System Overview

<!-- AI-DRAFT: Review before treating as canonical -->

The orchestrator is a local-first control plane that drives an iterative, self-correcting execution loop across three per-feature AI agents (planner, builder, QA) plus one shared reconciler lane for mainline-divergence work. The reconciler is not part of the normal planning/build/QA iteration and is intentionally single-instance per run so conflicting merge repair is serialized across the repo. The supervisor runtime (deterministic TypeScript, `orchestrator` RBAC role) is the coordination brain — it drives the loop, manages session lifecycles, and makes kernel tool calls, while all agent interactions are mediated through a 46-tool MCP protocol with state persisted atomically to `.aop/features/`.

---

## 🧠 Mental Model

<!-- AI-DRAFT: Review before treating as canonical -->

Think of the orchestrator as a **self-correcting iterative loop**, not a one-shot pipeline. Each iteration runs three waves (plan → build → QA), and then the planner is re-invoked for a **post-QA reconciliation** step: it inspects the gate results and checks whether the implementation matches what the plan required. If discrepancies remain, the planner amends the plan and the next iteration picks up where the build left off. The loop converges when the planner confirms there are no remaining discrepancies, or terminates when the iteration limit (`max_iterations_per_phase` in `policy.yaml`) is exhausted.

The **Supervisor Runtime** (TypeScript, not an LLM) is the coordination brain. It drives the outer iteration loop, sequences the three agent waves, and manages session lifecycles, including the single shared reconciler session. The worker agents operate within the lanes the supervisor defines — they cannot reorder waves, skip gates, or modify state outside their permitted MCP tools.

```
[Supervisor Runtime — outer iteration loop (bounded by max_iterations_per_phase)]
│
│  ┌─── each iteration ─────────────────────────────────────────────────────┐
│  │  Planning wave ──────► [Planner LLM] ──► plan.json                    │
│  │  Build wave ──────────► [Builder LLM] ──► patches (kernel-validated)   │
│  │  QA wave ─────────────► [QA LLM] ──────► gate evidence                │
│  │  Post-QA reconciliation ► [Planner LLM] checks gate results:           │
│  │      discrepancies found? ──► PLAN_UPDATE ──► next iteration loops ────┤
│  │      no discrepancies? ─────► converged ──► feature → ready_to_merge ──┘
```

> **Important**: There is no 4th "orchestrator LLM agent." The orchestrator is the deterministic supervisor code. It appears as a role in the RBAC layer (`RuntimeRole = 'orchestrator' | 'planner' | 'builder' | 'qa' | 'reconciler'`) so the kernel can authorize its tool calls, but it never spawns an LLM worker session.

---

## 📂 Component Map

<!-- AI-DRAFT: Review before treating as canonical -->

| Component                  | Responsibility                                                                   | Location                                                    |
| :------------------------- | :------------------------------------------------------------------------------- | :---------------------------------------------------------- |
| **CLI**                    | User-facing command interface; parses `aop` commands and dispatches to bootstrap | `apps/control-plane/src/cli/`                               |
| **Bootstrap / Interfaces** | Wires CLI args to kernel and supervisor                                          | `apps/control-plane/src/interfaces/cli/bootstrap.ts`        |
| **Kernel** (`AopKernel`)   | Deterministic orchestration engine; state, locks, plans, patches, gates, merges  | `apps/control-plane/src/core/kernel.ts`                     |
| **Supervisor Runtime**     | Multi-agent wave executor; spawns provider sessions, manages heartbeats          | `apps/control-plane/src/supervisor/runtime.ts`              |
| **Worker Decision Loop**   | Per-agent turn loop; routes MCP tool calls from agent output                     | `apps/control-plane/src/supervisor/worker-decision-loop.ts` |
| **MCP Tool Runtime**       | Registry-based tool dispatch; input/output schema validation; RBAC               | `apps/control-plane/src/mcp/tool-runtime.ts`                |
| **Application Services**   | 35 domain services (plan, patch, gates, locks, merge, QA, cost, etc.)            | `apps/control-plane/src/application/services/`              |
| **Provider Adapters**      | LLM CLI wrappers for codex, claude, kiro-cli, copilot                            | `apps/control-plane/src/providers/`                         |
| **Configuration Layer**    | YAML policies, gate profiles, agent roles, MCP tool contracts                    | `config/agentic/orchestrator/`                              |
| **State Store**            | File-system runtime artifacts: index, per-feature state/plan/evidence            | `.aop/features/`                                            |
| **Web Dashboard**          | Optional real-time monitoring UI                                                 | `packages/web-dashboard/`                                   |

---

## 🔄 Core Data Flow

<!-- AI-DRAFT: Review before treating as canonical -->

1. **Spec ingestion**: User runs `aop run -fi <spec.md>`; CLI parses args, resolves spec file.
2. **Feature initialization**: Kernel creates entry in `.aop/features/index.json`; writes initial `state.md` (YAML frontmatter + markdown).
3. **Config validation**: Kernel loads and AJV-validates `policy.yaml`, `agents.yaml`, `gates.yaml` against JSON schemas.
4. **Iterative execution loop** (repeats up to `max_iterations_per_phase` from `policy.yaml`):
   - **Planning wave**: Supervisor spawns planner agent; agent emits `PLAN_SUBMISSION` JSON; kernel validates against plan schema and persists `plan.json`.
   - **Build wave**: Supervisor spawns builder agent; agent emits `PATCH` diffs; kernel validates each patch against plan, applies to worktree.
   - **QA wave**: Supervisor spawns QA agent; kernel runs configured gate commands (lint, typecheck, tests); evidence artifacts written to `.aop/features/<id>/evidence/`.
   - **Post-QA reconciliation**: Planner is re-invoked to inspect gate results against the plan. If implementation discrepancies are found, planner calls `PLAN_UPDATE` and the loop continues. If the planner reports no discrepancies (`no_update`), the feature converges and exits the loop.
5. **Merge promotion**: On convergence with all gates passing, kernel transitions feature to `ready_to_merge`; merge service creates PR or applies changes to main branch. If the iteration budget (`max_iterations_per_phase`) is exhausted before convergence, the supervisor transitions the feature to `paused_budget`.

```
aop run → init → ┌─── iteration 1..N ────────────────────────────────────┐
                  │  planning → build → QA gates → post-QA reconciliation  │
                  │              ↓           ↓              ↓              │
                  │          plan.json    patches       evidence/           │
                  │                                         │              │
                  │                              discrepancies? ──► loop ──┤
                  │                              converged? ───────────────┘
                  └──────────────────────────────── ready_to_merge → merge
```

---

## 🚢 Deployment & Infrastructure

<!-- AI-DRAFT: Review before treating as canonical -->

- **Environment**: Local CLI (primary), Docker container (`npm run publish:docker`), npm package (`npm run publish:npm`)
- **CI/CD**: GitHub Actions (`.github/workflows/`)
- **Database**: None — all state is file-system-based (JSON + Markdown in `.aop/`)
- **IaC**: N/A (local-first tool; Docker image for distribution)

---

## 📊 Observability & Monitoring

<!-- AI-DRAFT: Review before treating as canonical -->

**Logging**:

- **Strategy**: Structured JSON output to stdout; all CLI responses use `{ ok, data/error }` envelope
- **Levels**: Errors and warnings via `console.error`/`console.warn`; `console.log` is forbidden in source (lint-enforced)
- **Sensitive Data**: No PII or secrets in logs; provider tokens come from environment variables only

**Runtime Artifacts** (primary observability mechanism):

- `.aop/features/<id>/state.md` — current feature status and YAML metadata
- `.aop/features/global/state.md` — reserved orchestrator-global pseudo-feature state; auto-bootstrapped on first access for compatibility with older runtimes
- `.aop/features/<id>/plan.json` — versioned plan with all patch history
- `.aop/features/<id>/evidence/` — gate run outputs and coverage reports
- Worker event journal (`supervisor/worker-event-journal.ts`) — per-session turn log

**Metrics**: No external APM; `performance-analytics-service.ts` tracks cost and timing per feature.

---

## 📈 Scalability & Performance

<!-- AI-DRAFT: Review before treating as canonical -->

**Current Profile**: Local CLI; designed for parallel multi-feature development on a single machine with per-feature worktree isolation.

**Scaling Strategy**:

- **Parallelism**: Multiple features can run concurrently via separate worktrees and independent lock namespaces
- **Bottlenecks**: File I/O for state mutations; LLM provider API rate limits and latency
- **Known Limits**: Supervisor iteration limit configured in `policy.yaml` (`supervisor.max_iterations_per_phase`); lock TTL prevents deadlock

---

## 🔐 State Mutation Concurrency Contract

All writes to a feature's `state.md` frontmatter **must** go through the feature file lock to prevent lost-update races. Two write mechanisms exist:

| Mechanism                           | Lock              | Version Guard                  | When to Use                                                        |
| :---------------------------------- | :---------------- | :----------------------------- | :----------------------------------------------------------------- |
| `FeatureStateService.updateState()` | `withFeatureLock` | Optimistic (`expectedVersion`) | Default for all state transitions (gates, status changes, patches) |
| `kernel.writeState()` (raw)         | **None**          | **None**                       | Only inside a caller-managed `withFeatureLock` block               |

**Invariant**: Every read-modify-write cycle on `state.md` must be wrapped in `withFeatureLock(featureId, …)`. Code that reads frontMatter, mutates it in memory, and writes it back without holding the lock is a **lost-update bug** — a concurrent writer (e.g. gate service finishing while the reconciler is blocking) can overwrite the first writer's changes.

**Version bumping**: When using raw `writeState` inside a lock, the caller must increment `version` and set `last_updated` manually. `updateState` handles this automatically.

**Services that hold the lock**:

- `FeatureStateService` — via `updateState()` (all standard transitions)
- `GitReconciliationService` — via `withFeatureLock()` around `upsertMainlineConflictState`, `resolveMainlineConflictState`, and `updateReconciliationState`
- `kernel.featureStatePatch` — delegates to `FeatureStateService.updateState()`

---

## 🔒 Security Posture

<!-- AI-DRAFT: Review before treating as canonical -->

- **Authentication**: None (local CLI tool); MCP tool runtime uses token-based auth for remote scenarios (`mcp/token-auth-verifier.ts`)
- **Authorization**: RBAC defined in `policy.yaml`; MCP tool authorizer enforces role-based tool access (`mcp/tool-authorizer.ts`)
- **Data Safety**: No PII processed; provider API keys from environment variables only; never committed
- **Execution Safety**: Deterministic mode suppresses interactive LLM permission prompts; worktree isolation prevents cross-feature file contamination; plan-gated patches prevent out-of-scope mutations

---

## 🧪 Testing Strategy

<!-- AI-DRAFT: Review before treating as canonical -->

- **Unit Tests**: Vitest (`apps/control-plane/test/`) — covers all services, kernel operations, provider adapters
- **Validation Scripts**: `npm run validate:mcp-contracts`, `npm run validate:architecture`, `npm run validate:docker-mcp`
- **Coverage Thresholds**: Lines: 90% | Branches: 90% | Functions: 90% | Statements: 90%
- **Run Tests**: `npm run test` (all) or `npx vitest run apps/control-plane/test/<file>.spec.ts` (single file)

---

## 🛠️ Technology Decisions (ADR Summary)

<!-- AI-DRAFT: Review before treating as canonical -->

- **TypeScript 5.9 strict + Node.js ESM**: Type safety across all layers; `import type` enforced; `no-explicit-any` in source.
- **AJV for schema validation**: Fast, standards-compliant JSON Schema validation for all configs and runtime artifacts; validation failures return structured errors with retry hints.
- **File-system state (no database)**: Zero infrastructure dependencies; state is portable, inspectable, and git-committable.
- **Nx monorepo**: Single workspace with `apps/control-plane/` and `packages/web-dashboard/`; shared TypeScript config.
- **MCP tool protocol**: Agents communicate exclusively via structured tool calls, not free-form text, ensuring parseable and auditable interactions.
- **Port-based DI**: Services define interfaces; kernel implements them — decouples domain logic from infrastructure and simplifies testing.

For detailed rationale, see `docs/core/DECISIONS.md`.

---

## 📌 Key Files & Locations

<!-- AI-DRAFT: Review before treating as canonical -->

| File/Folder                                                 | Purpose                                                    | Notes                                        |
| :---------------------------------------------------------- | :--------------------------------------------------------- | :------------------------------------------- |
| `apps/control-plane/src/cli/aop.ts`                         | CLI entry point                                            | Delegates to `interfaces/cli/bootstrap.ts`   |
| `apps/control-plane/src/cli/types.ts`                       | `CliCommand` enum + arg types                              | Update when adding new commands              |
| `apps/control-plane/src/cli/help-command-handler.ts`        | `COMMAND_HELP` map                                         | Source of truth for `aop --help`             |
| `apps/control-plane/src/core/kernel.ts`                     | Deterministic orchestration engine (~1,530 lines)          | Central coordinator; delegates to services   |
| `apps/control-plane/src/core/error-codes.ts`                | 97 structured error codes                                  | Always use these; never throw raw strings    |
| `apps/control-plane/src/supervisor/runtime.ts`              | Multi-agent wave supervisor                                | Spawns/manages provider sessions             |
| `apps/control-plane/src/supervisor/worker-decision-loop.ts` | Per-agent turn loop                                        | Routes MCP tool calls; detects no-progress   |
| `apps/control-plane/src/mcp/tool-runtime.ts`                | MCP tool execution pipeline                                | Registry dispatch + schema validation        |
| `apps/control-plane/src/providers/provider-defaults.ts`     | LLM provider CLI templates (codex/claude/kiro-cli/copilot) | Single source of truth for provider CLI args |
| `apps/control-plane/src/application/services/`              | 35 domain services                                         | Each implements a port interface             |
| `config/agentic/orchestrator/policy.yaml`                   | Commit/merge policies, RBAC, TTLs                          | Governs kernel behavior                      |
| `config/agentic/orchestrator/agents.yaml`                   | Agent roles, provider defaults, execution modes            | `deterministic` vs `interactive`             |
| `config/agentic/orchestrator/gates.yaml`                    | Gate profiles: fast / full / merge                         | Commands + coverage thresholds               |
| `config/agentic/orchestrator/tools/catalog.json`            | 46-tool MCP catalog                                        | Drives registry dispatch                     |
| `spec-files/progress.md`                                    | Append-only work log                                       | Update after every meaningful change         |

---

## 🔗 Related Documentation

- **Patterns**: See `docs/core/PATTERNS.md` for code conventions and canonical examples
- **Domain Logic**: See `docs/core/DOMAIN-LOGIC.md` for business rules and invariants
- **Decisions**: See `docs/core/DECISIONS.md` for detailed ADRs
- **Troubleshooting**: See `docs/core/TROUBLESHOOTING.md` for known issues and workarounds
- **Spec**: See `spec-files/agentic_orchestrator_spec.md` for normative requirements

<!-- @cortex-tms-version 4.0.2 -->
