# Feature Spec: Phase-Specific Agent Profiles and Token Telemetry (AOP)

> **Purpose of this document**: Define per-phase provider/model selection for planner, builder, and QA, and add trustworthy token telemetry so operators can measure the cost impact of those choices.

**Version:** 1.0
**Date:** 2026-03-16
**Status:** Draft
**Roadmap Mapping:** M53
**Depends On:**

- `spec-files/outstanding/agentic_orchestrator_dynamic_agent_swapping_spec.md`
- `spec-files/completed/agentic_orchestrator_reactive_execution_enrollment_spec.md`
- `spec-files/completed/agentic_orchestrator_dashboard_advanced_ux_spec.md`

---

## 0. Scope

This spec covers the following user requests:

1. Support different agent/model selections per phase or role:
   - planner -> Agent/Model A
   - builder -> Agent/Model B
   - qa -> Agent/Model C
2. Provide the telemetry required to make those cost-saving choices observable.
3. Replace the current “token usage” placeholder behavior with real usage reporting when available and explicit `unavailable` reporting when not.

Out of scope:

- automatic blocked-state escalation and agent swapping policy logic
- dashboard route redesign
- policy-file splitting

### 0.1 Required Standards

All implementation derived from this spec MUST preserve:

- deterministic provider/session resolution per worker role
- normalized error handling for unsupported providers, missing credentials, and incompatible execution modes
- truthful usage reporting with explicit unavailable states where telemetry is not exposed
- compatibility with the existing cost, analytics, and reporting artifacts unless intentionally versioned

### 0.2 OOP and Architecture Guardrails (Mandatory)

Implementation MUST follow:

- **KISS**: treat per-role selection and token telemetry as two focused seams, not one catch-all “optimization” subsystem.
- **SRP**: provider resolution, usage capture, persistence, and dashboard presentation MUST remain separate responsibilities.
- **SOLID-oriented design**:
  - open/closed for future roles/providers/usage fields without branching through monolithic handlers
  - interface segregation between provider-selection contracts and usage-reporting contracts
  - dependency inversion so reporting consumes normalized usage data instead of provider-specific payloads
- **Hexagonal alignment**: provider-specific parsing stays in provider/runtime adapters; application services consume normalized metadata only.
- **Composition over inheritance**: layer role-specific selection and usage instrumentation through composed helpers and services.
- **High cohesion / low coupling**: token telemetry changes must not be tightly coupled to dashboard-only rendering concerns.

---

## 1. Problem Statement

The codebase already contains partial role-specific selection infrastructure:

- `resolveProviderSelectionsByRole(...)`
- `runtime.role_provider_overrides` in `agents.yaml`

But the current CLI contract still forces all per-role overrides to use the same provider as the base selection, which blocks the intended cost-optimization use case.

At the same time, token usage surfaces exist in reporting and dashboard UI, but no provider path is reliably recording real usage into `cost.record`. That makes the “Feature Usage” UI mostly decorative rather than operational.

The result is a half-finished optimization story:

- configuration hints exist
- cross-phase agent selection is not actually allowed end-to-end
- usage metrics do not prove whether the configuration is helping

---

## 2. Design Summary

### 2.1 Phase-Specific Profiles

Make per-role runtime selection a first-class supported feature.

Terminology:

- `role` means the runtime worker role: `planner`, `builder`, `qa`, and optionally `reconciler`
- `phase-specific` in this spec maps to those role selections because the orchestrator executes the phases through those role lanes

### 2.2 Supported Overrides

`agents.yaml` MUST support role-specific overrides for:

- `provider`
- `model`
- `provider_config_env`
- `agent_config`

The base runtime defaults remain the fallback.

### 2.3 Token Telemetry Truthfulness

Token telemetry MUST be truthful rather than inferred silently.

Normative rule:

- if the provider exposes token usage, record it
- if the provider does not expose token usage, mark usage as `unavailable`
- do not fabricate zeros for missing provider telemetry

Derived USD remains optional and secondary.

---

## 3. Functional Requirements

### FR-1 Allow Cross-Provider Role Overrides

The runtime MUST allow role-specific selections whose provider differs from the base runtime provider.

Acceptance criteria:

1. `planner`, `builder`, and `qa` can each resolve to a different provider/model pair.
2. CLI validation no longer rejects role overrides merely because the provider differs from the base selection.
3. Runtime session metadata persists the actual provider/model used for each worker lane.

### FR-2 Preserve Safe Selection Rules

Per-role selection MUST still preserve existing safety and validation rules.

Acceptance criteria:

1. Unsupported providers still fail through `ERROR_CODES.UNSUPPORTED_AGENT_PROVIDER`.
2. Missing credentials still fail per provider.
3. Execution mode compatibility is validated per selected provider.

### FR-3 Expose Role Usage in Runtime Artifacts

Each worker execution or session lane MUST persist enough metadata to answer:

- which provider/model handled planner?
- which provider/model handled builder?
- which provider/model handled qa?
- how many tokens were consumed by each lane?

Acceptance criteria:

1. Worker event journal entries capture provider/model and usage metadata when available.
2. Dashboard analytics can aggregate token usage by provider/model and by role.
3. Feature-level summaries can show planner/builder/qa usage breakdowns when available.

### FR-4 Real Token Recording

Implement actual token usage recording into the existing cost pipeline.

Recommended data shape:

```json
{
  "feature_id": "feature_checkout_flow",
  "tokens_used": 31244,
  "estimated_cost_usd": 0.62,
  "recorded_at": "2026-03-16T12:00:00.000Z",
  "by_role": {
    "planner": { "tokens_used": 4200, "status": "reported" },
    "builder": { "tokens_used": 22100, "status": "reported" },
    "qa": { "tokens_used": 4944, "status": "reported" }
  }
}
```

Acceptance criteria:

1. Provider adapters or decision-loop integration call `cost.record` with real deltas when the provider returns usage.
2. Missing provider usage is recorded as `status: unavailable`, not `0`.
3. Existing budget and dashboard reporting surfaces consume the new real values.

### FR-5 Dashboard and Analytics Updates

The dashboard MUST present token usage in a way that supports model-cost decisions.

Acceptance criteria:

1. Feature detail shows total tokens plus role-level breakdown when available.
2. Analytics route can compare providers/models by token usage, estimated cost, and success metrics.
3. UI states clearly distinguish `reported`, `estimated`, and `unavailable`.

---

## 4. Configuration Contract

### 4.1 `agents.yaml`

The existing `runtime.role_provider_overrides` field becomes normative rather than experimental.

Example:

```yaml
runtime:
  default_provider: codex
  default_model: gpt-5.2-codex
  role_provider_overrides:
    planner:
      provider: claude
      model: claude-sonnet-4
    builder:
      provider: codex
      model: gpt-5.2-codex
    qa:
      provider: openai
      model: gpt-5.2
```

### 4.2 Compatibility with Dynamic Agent Swapping

This spec is complementary to `agentic_orchestrator_dynamic_agent_swapping_spec.md`.

Boundary:

- this spec defines the baseline per-role selection
- the dynamic-agent-swapping spec defines post-block escalation to alternate profiles

Dynamic swapping MUST treat the role-specific baseline from this spec as the starting configuration.

---

## 5. Implementation Notes

### 5.1 Existing Constraint to Remove

Current `run-command-handler.ts` and `resume-command-handler.ts` reject role-provider overrides whose provider differs from the base provider. That guard must be replaced with compatibility validation, not provider equality validation.

### 5.2 Usage Instrumentation Points

Preferred instrumentation order:

1. provider adapter returns usage in structured form
2. `WorkerDecisionLoop` or runtime adapter converts it into a normalized usage record
3. runtime records `cost.record`
4. reporting/dashboard surfaces read the recorded artifact

### 5.3 CLI Provider Reality

Some CLI providers will not expose token usage.

For those providers, the correct v1 behavior is:

- show `usage unavailable`
- do not block execution
- do not silently imply zero cost

---

## 6. File-Level Targets

- `apps/control-plane/src/providers/providers.ts`
  - allow cross-provider per-role selection and normalize role metadata.
- `apps/control-plane/src/cli/run-command-handler.ts`
  - remove same-provider restriction and pass through per-role selections.
- `apps/control-plane/src/cli/resume-command-handler.ts`
  - same as run path.
- `apps/control-plane/src/supervisor/runtime.ts`
  - persist per-role provider/model metadata into run metadata and feature summaries.
- `apps/control-plane/src/supervisor/worker-decision-loop.ts`
  - record per-turn or per-step usage data when available.
- `apps/control-plane/src/providers/cli-worker-provider.ts`
  - expose usage metadata if the underlying CLI supports it.
- `apps/control-plane/src/providers/api-worker-provider.ts`
  - expose reported usage metadata from API responses.
- `apps/control-plane/src/application/services/cost-tracking-service.ts`
  - support richer role-aware usage structure without regressing total summaries.
- `apps/control-plane/src/application/services/reporting-service.ts`
  - surface role/provider usage in dashboard/report payloads.
- `apps/control-plane/src/application/services/performance-analytics-service.ts`
  - aggregate provider/model token and cost metrics.
- `packages/web-dashboard/src/components/feature-cost-panel.tsx`
  - render real usage status and per-role breakdown.
- `packages/web-dashboard/src/components/dashboard/triage-drawer.tsx`
  - stop implying token data exists when it does not.
- `packages/web-dashboard/src/app/analytics/page.tsx`
  - add usage-driven comparison surfaces.
- `packages/web-dashboard/src/lib/types.ts`
  - extend cost/usage types for availability and per-role breakdown.

---

## 7. Acceptance Criteria

1. Planner, builder, and QA can run on different providers/models under one execution.
2. The runtime persists the actual provider/model used per role.
3. Token usage is recorded when the provider supplies it.
4. Missing token data is surfaced explicitly as unavailable rather than as fake zero usage.
5. Dashboard and analytics surfaces show enough usage detail to evaluate cost-saving model choices.

---

## 8. Risks and Mitigations

- Risk: cross-provider overrides break assumptions in attach/send flows.
  - Mitigation: keep feature session metadata authoritative per role and validate all role selections at session-creation time.
- Risk: token telemetry differs across providers.
  - Mitigation: introduce a normalized usage contract with explicit availability status instead of forcing all providers into the same exact payload.
- Risk: dashboard cost widgets overstate precision.
  - Mitigation: always label `reported`, `estimated`, or `unavailable`.
