# Feature Spec: `.aop` Artifact Publishing to Configurable Audit Database (AOP)

> **Purpose of this document**: Define implementation-ready functionality to publish runtime artifacts from `.aop/**` to a user-configured database after execution, enabling durable audit history of what the orchestrator did and why it did it. The feature includes external database targets and a built-in local Docker-backed database mode (`type: local`).

**Version:** 1.0  
**Date:** 2026-03-04  
**Status:** Draft  
**Roadmap Mapping:** M40

---

## 0. Standards and Dependencies

### 0.1 Required Standards

All implementation MUST preserve:

- deterministic runtime behavior and MCP/in-process parity
- existing command behavior when publishing is disabled
- normalized error envelope (`{ ok: false, error: { code, message, details } }`)
- atomic file-write/lock guarantees for `.aop/**` artifacts
- Nx + Vitest + coverage thresholds (>= 90% per file)

### 0.2 Upstream Inputs

Implementing agents MUST read:

- `apps/control-plane/src/core/path-layout.ts`
- `apps/control-plane/src/core/kernel.ts`
- `apps/control-plane/src/supervisor/run-coordinator.ts`
- `apps/control-plane/src/cli/run-command-handler.ts`
- `apps/control-plane/src/interfaces/cli/bootstrap.ts`
- `apps/control-plane/src/cli/cli-argument-parser.ts`
- `apps/control-plane/src/cli/help-command-handler.ts`
- `apps/control-plane/src/core/constants.ts`
- `apps/control-plane/src/core/error-codes.ts`
- `apps/control-plane/src/mcp/operation-ledger.ts`
- `agentic/orchestrator/policy.yaml`
- `agentic/orchestrator/defaults/policy.defaults.yaml`
- `agentic/orchestrator/schemas/policy.schema.json`
- `agentic/orchestrator/tools/catalog.json`
- `spec-files/completed/agentic_orchestrator_dot_aop_generated_artifacts_spec.md`

### 0.3 Feature Scope

This spec implements three slices:

- **DB1** Configurable artifact publishing policy + database connection contract
- **DB2** Post-run and manual publishing pipeline with idempotent checkpoints
- **DB3** Local Docker database mode (`database.type: local`)

---

## 1. Objectives

### 1.1 Must-Have Outcomes

- Users can configure a database target in YAML and publish `.aop/**` artifacts to it.
- Publishing can run automatically after `aop run` execution and can also be triggered manually.
- Published records are queryable over time by `run_id`, `feature_id`, tool activity, gate results, and decisions.
- `database.type: local` starts/uses a local Docker database automatically and publishes there without external infra setup.
- Publishing is idempotent and safe to retry.

### 1.2 Non-Goals

- No replacement of file-backed `.aop/**` runtime storage.
- No requirement to support every SQL engine in first release.
- No real-time streaming pipeline in this phase (batch publish at execution boundaries).
- No dashboard UI implementation in this phase (data plane + CLI/tool contracts only).

---

## 2. Architecture Decisions

### 2.1 Source of Truth Remains `.aop/**`

Database rows are derived artifacts for analytics/audit. `.aop/**` remains authoritative runtime state.

### 2.2 First-Class Publisher Service

All publishing must route through a single service boundary (`ArtifactPublisherService`) to centralize:

- artifact enumeration
- normalization and hashing
- DB transaction semantics
- checkpointing/retry behavior

### 2.3 Idempotency by `(target_id, run_id, publish_revision)`

Publishing writes must be safely repeatable. Re-running publish for the same run must not duplicate logical records.

### 2.4 Local Mode Resolves to Docker-Managed Postgres

`database.type: local` is a convenience wrapper around a managed local Postgres container. External engines remain explicitly configured.

### 2.5 Failure Behavior is Configurable

Default is best-effort (run succeeds even if publish fails). Strict mode is available when audit persistence is mandatory.

---

## 3. Configuration Contract

### 3.1 Policy Additions (`policy.yaml`)

Add `artifact_publishing` section to policy (full + defaults + schema):

```yaml
artifact_publishing:
  enabled: false
  trigger: post_run # post_run | manual_only
  mode: best_effort # best_effort | required
  include_content:
    state_md: true
    plan_json: true
    decisions_md: true
    qa_index_json: true
    gate_logs_preview_bytes: 16384
  database:
    type: postgres # postgres | local
    target_id: default
    postgres:
      connection_uri: '${AOP_AUDIT_DB_URL}'
      schema: aop_audit
      ssl_mode: require # disable | require
      connect_timeout_ms: 8000
    local:
      engine: postgres
      image: postgres:16-alpine
      container_name: aop-audit-db
      port: 55432
      database: aop_audit
      username: aop
      password_env: AOP_LOCAL_AUDIT_DB_PASSWORD
      volume_name: aop_audit_db_data
      auto_start: true
      wait_timeout_ms: 30000
```

### 3.2 YAML Validation

`agentic/orchestrator/schemas/policy.schema.json` MUST validate the new section with:

- strict enums for `trigger`, `mode`, `database.type`
- required `postgres.connection_uri` when `type: postgres`
- required local fields when `type: local`
- numeric bounds for port/timeouts/preview bytes

### 3.3 Secret Handling

- connection strings and passwords MUST support env interpolation (`${ENV_VAR}`).
- raw password literals in committed policy SHOULD be rejected with explicit validation guidance.

### 3.4 Defaults

`agentic/orchestrator/defaults/policy.defaults.yaml` MUST include `artifact_publishing.enabled: false` so behavior remains opt-in.

---

## 4. Canonical Publish Payload

### 4.1 Artifacts Included Per Run

Minimum dataset for a run publish:

- `.aop/features/index.json`
- `.aop/runtime/operation-ledger/<run_id>.json`
- for each feature in run scope:
  - `state.md`
  - `plan.json` (if present)
  - `qa_test_index.json` (if present)
  - `cost.json` (if present)
  - `decisions.md` (if present)
  - gate/evidence metadata and selected content previews

### 4.2 Normalized Record Categories

Publisher MUST normalize into these logical categories:

- `run_summary`
- `feature_snapshot`
- `tool_execution`
- `gate_execution`
- `decision_log_entry`
- `artifact_manifest_entry`
- `publish_event`

### 4.3 Hashing and Determinism

For each artifact row:

- include `content_sha256`
- include `relative_path`
- normalize JSON key order before hashing JSON content
- ensure stable sort order by `feature_id`, then `relative_path`

---

## 5. Database Schema (v1)

### 5.1 Required Tables

1. `aop_publish_runs`

- `publish_id` (PK)
- `target_id`
- `run_id`
- `publish_revision`
- `status` (`started|succeeded|failed`)
- `started_at`, `finished_at`
- `error_code`, `error_message`

2. `aop_runs`

- `run_id` (PK)
- `project_root_hash`
- `started_at`
- `finished_at`
- `queue_depth`
- `runtime_status`

3. `aop_features`

- composite PK: (`run_id`, `feature_id`)
- `status`
- `branch`
- `gate_profile`
- `last_updated`
- `cost_estimated_usd`
- `tokens_used`

4. `aop_tool_executions`

- composite PK: (`run_id`, `operation_id`)
- `feature_id`
- `tool_name`
- `actor_type`
- `result`
- `error_code`
- `latency_ms`
- `request_hash`
- `created_at`, `updated_at`

5. `aop_gate_executions`

- surrogate PK + unique (`run_id`, `feature_id`, `gate_name`, `executed_at`)
- `result`, `exit_code`
- `mode`
- `evidence_path`
- `logs_preview`

6. `aop_decisions`

- surrogate PK + index (`run_id`, `feature_id`, `line_no`)
- `entry_text`
- `entry_hash`

7. `aop_artifact_manifest`

- composite PK: (`run_id`, `feature_id`, `relative_path`)
- `size_bytes`
- `content_sha256`
- `content_preview`
- `modified_at`

### 5.2 Migration Contract

- Create `schema_migrations` table and apply ordered SQL migrations.
- Publisher must run `ensureSchema()` before first write for a target.

---

## 6. Runtime Flow and Trigger Points

### 6.1 Automatic Post-Run Publishing

`RunCommandHandler.execute(...)` flow extension:

1. run supervisor cycle (`supervisor.start(...)`)
2. if `artifact_publishing.enabled && trigger == post_run`, invoke publisher
3. attach publish result under response payload:
   - `data.artifact_publish.status`
   - `data.artifact_publish.publish_id`
   - `data.artifact_publish.warning` (best-effort failure only)

### 6.2 Manual Publishing

Add CLI command:

- `aop publish [--run-id <id>] [--latest] [--dry-run] [--strict]`

Behavior:

- default `--latest` when no run id provided
- `--dry-run` builds manifest + stats without DB writes
- `--strict` forces non-zero exit on publish failure regardless of policy mode

### 6.3 MCP Tool Surface

Add tools:

- `artifacts.publish` (mutating: true)
- `artifacts.publish_status` (mutating: false)

`artifacts.publish` input includes `run_id`, `dry_run`, `strict`, `trigger_source`.

---

## 7. Local Docker Database Mode (`type: local`)

### 7.1 Behavior

When `database.type: local`:

1. verify Docker CLI availability
2. verify/start container using configured image/name/port/volume
3. wait for DB health/readiness
4. connect via computed Postgres URI
5. run schema migrations and publish

### 7.2 Local Runtime Artifacts

Persist local DB runtime metadata under:

- `.aop/runtime/artifact-publishing/local-db.json`

Contract example:

```json
{
  "target_id": "default",
  "container_name": "aop-audit-db",
  "engine": "postgres",
  "host": "127.0.0.1",
  "port": 55432,
  "database": "aop_audit",
  "username": "aop",
  "started_at": "2026-03-03T22:00:00Z"
}
```

### 7.3 Docker Failure Handling

Errors must map cleanly:

- `artifact_publish_local_docker_unavailable`
- `artifact_publish_local_start_failed`
- `artifact_publish_local_unhealthy`

If mode is `best_effort`, run continues and publish failure is reported in payload + logs.

---

## 8. Error Codes

Add to `apps/control-plane/src/core/error-codes.ts`:

- `artifact_publish_config_invalid`
- `artifact_publish_target_unconfigured`
- `artifact_publish_connect_failed`
- `artifact_publish_migration_failed`
- `artifact_publish_write_failed`
- `artifact_publish_checkpoint_conflict`
- `artifact_publish_local_docker_unavailable`
- `artifact_publish_local_start_failed`
- `artifact_publish_local_unhealthy`

---

## 9. Implementation Plan

## 9.1 DB1: Configuration + Validation

### DB1-T1 Policy contract

- update `policy.yaml` examples
- update `policy.defaults.yaml`
- extend `policy.schema.json`

### DB1-T2 Typed policy accessors

- add typed readers for `artifact_publishing` section in kernel/service layer

### DB1-T3 Config normalization

- implement env interpolation + normalization helper

## 9.2 DB2: Publisher pipeline

### DB2-T1 Artifact enumeration

- deterministic scanner for run/feature artifacts
- manifest builder with hashes and size

### DB2-T2 Store interface and Postgres adapter

- `ArtifactAuditStore` interface
- `PostgresArtifactAuditStore` implementation (transactional upsert)

### DB2-T3 Checkpointing and idempotency

- write local checkpoint under `.aop/runtime/artifact-publishing/checkpoints/<run_id>.json`
- support safe retries after partial failures

### DB2-T4 Trigger integration

- invoke publisher from run flow and manual command path

## 9.3 DB3: Local Docker orchestration

### DB3-T1 Local DB manager

- start/inspect/wait for container
- compute connection URI for local target

### DB3-T2 Local-only smoke checks

- optional smoke test command for local mode readiness

---

## 10. File Targets

### 10.1 New files

- `apps/control-plane/src/application/services/artifact-publisher-service.ts`
- `apps/control-plane/src/application/services/artifact-manifest-service.ts`
- `apps/control-plane/src/application/services/artifact-audit-store.ts`
- `apps/control-plane/src/application/services/postgres-artifact-audit-store.ts`
- `apps/control-plane/src/application/services/local-artifact-db-manager.ts`
- `apps/control-plane/src/cli/publish-command-handler.ts`
- `agentic/orchestrator/tools/schemas/input/artifacts.publish.input.schema.json`
- `agentic/orchestrator/tools/schemas/output/artifacts.publish.output.schema.json`
- `agentic/orchestrator/tools/schemas/input/artifacts.publish_status.input.schema.json`
- `agentic/orchestrator/tools/schemas/output/artifacts.publish_status.output.schema.json`
- `apps/control-plane/src/application/services/sql/artifact-publishing/postgres/001_init.sql`

### 10.2 Existing files to modify

- `apps/control-plane/src/core/kernel.ts`
- `apps/control-plane/src/core/constants.ts`
- `apps/control-plane/src/core/error-codes.ts`
- `apps/control-plane/src/core/path-layout.ts`
- `apps/control-plane/src/cli/cli-argument-parser.ts`
- `apps/control-plane/src/cli/help-command-handler.ts`
- `apps/control-plane/src/cli/types.ts`
- `apps/control-plane/src/interfaces/cli/bootstrap.ts`
- `apps/control-plane/src/cli/run-command-handler.ts`
- `agentic/orchestrator/tools/catalog.json`
- `agentic/orchestrator/tools.md`
- `agentic/orchestrator/policy.yaml`
- `agentic/orchestrator/defaults/policy.defaults.yaml`
- `agentic/orchestrator/schemas/policy.schema.json`
- `README.md`

### 10.3 Tests

- `apps/control-plane/test/artifact-publisher.spec.ts` (new)
- `apps/control-plane/test/local-artifact-db-manager.spec.ts` (new)
- `apps/control-plane/test/run-command.spec.ts` (extend post-run publish assertions)
- `apps/control-plane/test/cli.unit.spec.ts` (parse + publish command)
- `apps/control-plane/test/mcp.spec.ts` (new tool registration + parity)
- `apps/control-plane/test/kernel.spec.ts` (tool handler contracts)

---

## 11. Acceptance Criteria

### DB1 — Configurable DB contract

- [ ] `artifact_publishing` policy section schema-validates with strict enum/type checks
- [ ] env-variable interpolation is supported for DB secrets
- [ ] invalid config surfaces `artifact_publish_config_invalid` with actionable details

### DB2 — Publish pipeline

- [ ] `aop run` publishes artifacts post-run when enabled and trigger is `post_run`
- [ ] `aop publish --run-id <id>` publishes historical run artifacts on demand
- [ ] published dataset includes run summary, feature snapshots, tool executions, decisions, and artifact manifest
- [ ] re-running publish for same run is idempotent (no duplicate logical records)
- [ ] best-effort mode does not fail run command on publish failure; required mode does

### DB3 — Local Docker mode

- [ ] `database.type: local` auto-starts/uses configured Docker DB container
- [ ] publish succeeds to local DB without external DB provisioning
- [ ] explicit, typed errors are returned when Docker is missing/unhealthy

### Quality and regression

- [ ] existing commands continue to work unchanged when publishing is disabled
- [ ] `npm run typecheck` and `npm test` pass with coverage thresholds preserved
- [ ] `npm run validate:mcp-contracts` passes with updated tool catalog/schemas

---

## 12. Implementation Order

1. Add policy contract + schema + defaults (`artifact_publishing`, disabled by default)
2. Add new error codes and constants for publish tools
3. Implement manifest + hashing service (no DB yet)
4. Implement Postgres audit store + migrations
5. Add `artifacts.publish` / `artifacts.publish_status` tool handlers and schemas
6. Add `aop publish` CLI command and bootstrap wiring
7. Wire post-run trigger in `RunCommandHandler`
8. Implement `type: local` Docker manager and integrate with publisher
9. Add tests (unit + integration contracts)
10. Update docs and examples

---

## 13. Risks and Mitigations

- Risk: large artifact payloads cause slow publish times.
  - Mitigation: default to metadata + bounded previews for large logs, plus batched writes.
- Risk: local Docker startup flakiness creates noisy failures.
  - Mitigation: explicit readiness checks, bounded retries, and best-effort default mode.
- Risk: duplicate records on retries.
  - Mitigation: deterministic upsert keys and run-level publish checkpoints.
- Risk: secrets leakage in logs.
  - Mitigation: redact connection URI credentials and password fields from all logs/errors.

---

## 14. Open Questions

1. Should first release support only Postgres (`postgres` + `local`) or include MySQL in MVP?
2. Should `aop status` expose latest publish status summary by default?
3. Should strict mode be policy-only, CLI-only, or both (recommended: both)?

---

## 15. Progress Tracking Note

When implementation starts, add milestone entries to `spec-files/progress.md` and move this spec from `spec-files/outstanding/` to `spec-files/completed/` only after acceptance criteria are met.
