# Release Validation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Prove CacheLane's correctness rigorously before public release — every REQ-F invariant mapped to a test, every cost-formula multiplier verified, and every critical gate re-confirmed green.

**Architecture:** Validation is read-then-fill: parallel audit agents survey each subsystem against the spec, then targeted tests fill any gaps, then a final end-to-end green-light run confirms the whole suite is clean.

**Tech Stack:** TypeScript, Vitest, better-sqlite3, Node 20 (required for native bindings), ESLint, `npx tsc --noEmit`

---

## Baseline Assumptions (confirmed before this plan was written)

- 268 tests across 35 files, all green on `npm test`
- `npx tsc --noEmit` — clean
- `npm run lint` — clean (after fixing `_cc` unused-var)
- Pipeline smoke test (`src/proxy/__tests__/pipeline-smoke.test.ts`) exists and validates 5 assertions end-to-end
- All milestones M1–M9 shipped as v1.0.0
- Cache-stability gate: passed (SHA-256 byte-identical across 3 runs, 6 scenarios)
- Reference detection corpus gate: passed (precision 100%, recall 94.7%)

---

## File Map

| File | Action | Why |
|------|--------|-----|
| `src/proxy/__tests__/pipeline-smoke.test.ts` | Extend | Add 1h-TTL cost tier test case |
| `src/storage/__tests__/migrations.test.ts` | Extend | Cover migrations 001–003 explicitly |
| `src/proxy/__tests__/server.test.ts` | Extend | Add workspace-isolation assertion |
| `src/orchestrator/__tests__/cache-stability.test.ts` | Read + confirm | Verify 6 scenarios cover spec requirements |
| `src/tokenizer/__tests__/tokenizer.test.ts` | Read + confirm | Verify 4.6 vs 4.7 model-string lookup |
| `src/benchmark/__tests__/recorded.test.ts` | Read + confirm | Verify cost formula math in report |
| `src/agent-traces/__tests__/` | Read + confirm | Verify scenario integrity and redaction |

---

## Task 1: Verify the Effective-Cost Formula — 1h TTL Tier

**Why this matters:** The pipeline smoke test only exercises the `5m` creation tier (`cache_creation_5m_tokens * 1.25`). The `1h` tier uses a `2.0x` multiplier. If that branch is broken, sessions with large prefixes silently under-report costs. This must be explicitly asserted.

**Formula (from spec):**
```
effective_cost = input_tokens * 1.0
              + cache_creation_5m_tokens * 1.25
              + cache_creation_1h_tokens * 2.0
              + cache_read_tokens        * 0.1
```

**Files:**
- Modify: `src/proxy/__tests__/pipeline-smoke.test.ts`

- [ ] **Step 1: Read the current smoke test to understand the test harness**

Run: `cat src/proxy/__tests__/pipeline-smoke.test.ts`

The test uses `sseResponseBody(inputUsage)` helper and `postMessages()` to send through the proxy. Reuse both.

- [ ] **Step 2: Write the failing test — 1h TTL cost tier**

Add a second `describe` block after the existing one in `src/proxy/__tests__/pipeline-smoke.test.ts`:

```typescript
describe("Pipeline smoke test — 1h TTL cost tier (§cost-formula)", () => {
  it("computes effective_cost_units correctly for a 1h cache-creation turn", async () => {
    const req: AnthropicMessagesRequest = {
      model: "claude-opus-4-7",
      system: [{ type: "text", text: "system prompt" }],
      tools: [{ name: "Read", input_schema: { type: "object" } }],
      messages: [{ role: "user", content: [{ type: "text", text: "msg" }] }],
      max_tokens: 1024,
    };

    resetFakeUpstream(sseResponseBody({
      input_tokens: 200,
      cache_read_input_tokens: 0,
      cache_creation_5m_tokens: 0,
      cache_creation_1h_tokens: 100,
    }));

    await postMessages(proxyPort, JSON.stringify(req));
    await waitForTurn(dbPath, "test-session", 1);

    const db = openDatabase(dbPath);
    try {
      const turn = db.getTurnByNumber("test-ws", "test-session", 1)!;
      // effective = 200 * 1.0 + 100 * 2.0 = 400
      expect(turn.effective_cost_units).toBe(400);
    } finally {
      db.close();
    }
  });
});
```

- [ ] **Step 3: Run the test to verify it fails (or passes — confirm it's exercising the right code path)**

```bash
npx vitest run src/proxy/__tests__/pipeline-smoke.test.ts --reporter=verbose
```

Expected: either FAIL (proving the 1h path was untested) or PASS (proving it works). If FAIL, proceed to fix; if PASS, the coverage is confirmed real.

- [ ] **Step 4: If failing — locate and fix the cost formula in the proxy response handler**

The cost formula should live in `src/proxy/server.ts` (or similar). Find `effective_cost_units` computation and ensure:

```typescript
effective_cost_units =
  input_tokens * 1.0 +
  (cache_creation_5m_tokens ?? 0) * 1.25 +
  (cache_creation_1h_tokens  ?? 0) * 2.0 +
  (cache_read_tokens         ?? 0) * 0.1;
```

- [ ] **Step 5: Run full test suite to confirm no regressions**

```bash
npm test
```

Expected: all 268+ tests pass.

- [ ] **Step 6: Commit**

```bash
git add src/proxy/__tests__/pipeline-smoke.test.ts
git commit -m "test(smoke): add 1h TTL tier cost formula assertion"
```

---

## Task 2: Verify Database Migration Coverage (M001–M003)

**Why this matters:** `storage/__tests__/migrations.test.ts` only explicitly tests migration 004. Migrations 001–003 create the core schema (blocks, turns, block_references, turn_explanations). If they silently apply wrong column names or types, every other test is a lie — it's testing against an in-memory schema that never matches what real users get.

**Files:**
- Modify: `src/storage/__tests__/migrations.test.ts`
- Read first: `src/storage/migrations/` to discover the migration files

- [ ] **Step 1: List migration files**

```bash
ls src/storage/migrations/
```

Expected: files named `001_*.sql`, `002_*.sql`, etc.

- [ ] **Step 2: Read migrations.test.ts to understand the existing test pattern**

```bash
cat src/storage/__tests__/migrations.test.ts
```

Note the pattern used for migration 004 — replicate it.

- [ ] **Step 3: Read each migration file to know expected columns/indexes**

```bash
cat src/storage/migrations/001_*.sql
cat src/storage/migrations/002_*.sql
cat src/storage/migrations/003_*.sql
```

- [ ] **Step 4: Write tests for migrations 001–003**

For each migration, assert the schema is present after applying all migrations. Add to `migrations.test.ts`:

```typescript
describe("migration 001 — core schema", () => {
  it("creates blocks, turns, block_references tables", () => {
    const db = openDatabase(dbPath); // openDatabase applies all migrations
    try {
      const tables = db.raw
        .prepare("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")
        .all() as { name: string }[];
      const names = tables.map((t) => t.name);
      expect(names).toContain("blocks");
      expect(names).toContain("turns");
      expect(names).toContain("block_references");
    } finally {
      db.close();
    }
  });
});

describe("migration 002 — turn_explanations", () => {
  it("creates turn_explanations table with correct columns", () => {
    const db = openDatabase(dbPath);
    try {
      const cols = db.raw
        .prepare("PRAGMA table_info(turn_explanations)")
        .all() as { name: string }[];
      const colNames = cols.map((c) => c.name);
      expect(colNames).toContain("workspace_id");
      expect(colNames).toContain("session_id");
      expect(colNames).toContain("turn_number");
    } finally {
      db.close();
    }
  });
});

// Repeat pattern for migration 003
```

Adjust column names to match what you find in step 3.

- [ ] **Step 5: Run the migration tests**

```bash
npx vitest run src/storage/__tests__/migrations.test.ts --reporter=verbose
```

Expected: all pass.

- [ ] **Step 6: Run full suite**

```bash
npm test
```

Expected: all tests pass.

- [ ] **Step 7: Commit**

```bash
git add src/storage/__tests__/migrations.test.ts
git commit -m "test(storage): assert migrations 001-003 create correct schema"
```

---

## Task 3: Verify Workspace Isolation

**Why this matters:** REQ-F-017/018 and the Anthropic Feb 2026 mandate require that sessions in different workspaces never share prefix state. If `CacheStateTracker` or the DB allows cross-workspace leakage, two separate users (or Claude Code instances) could corrupt each other's cache state. No test currently asserts this.

**Files:**
- Modify: `src/proxy/__tests__/server.test.ts` OR `src/orchestrator/__tests__/cache-state-tracker.test.ts`
- Read first: `src/orchestrator/__tests__/cache-state-tracker.test.ts` to find the existing isolation tests

- [ ] **Step 1: Read cache-state-tracker.test.ts**

```bash
cat src/orchestrator/__tests__/cache-state-tracker.test.ts
```

Check whether workspace isolation is already asserted (look for tests with two different `workspace_id` values).

- [ ] **Step 2: If workspace isolation is NOT explicitly tested — write the test**

Add to `cache-state-tracker.test.ts`:

```typescript
it("isolates state between workspaces — different workspace_id never shares prefix state", () => {
  const tracker = new CacheStateTracker();
  tracker.update("ws-A", "session-1", {
    prefix_hash: "hash-A",
    middle_hash: null,
    ttl_class: "5m",
    cached_at_ms: Date.now(),
    last_read_at_ms: Date.now(),
    expected_expiry_ms: Date.now() + 300_000,
  });

  // ws-B should see no state even with same session ID
  const stateB = tracker.get("ws-B", "session-1");
  expect(stateB).toBeUndefined();

  // ws-A state unaffected
  const stateA = tracker.get("ws-A", "session-1");
  expect(stateA?.prefix_hash).toBe("hash-A");
});
```

- [ ] **Step 3: Also add a DB-level workspace isolation test in server.test.ts**

The proxy routes requests using a `workspace_id` extracted from headers. Add:

```typescript
it("stats are isolated per workspace — ws-A stats do not include ws-B turns", async () => {
  // Send one turn as ws-A
  await postMessages(proxyPortA, JSON.stringify(req)); // proxy started with workspace_id: "ws-A"
  await waitForTurn(dbPath, "session-1", 1);

  const db = openDatabase(dbPath);
  try {
    const statsA = db.getStats({ scope: "workspace", workspace_id: "ws-A" });
    const statsB = db.getStats({ scope: "workspace", workspace_id: "ws-B" });
    expect(statsA.turns).toBe(1);
    expect(statsB.turns).toBe(0); // ws-B has no turns
  } finally {
    db.close();
  }
});
```

- [ ] **Step 4: Run the tests**

```bash
npx vitest run src/orchestrator/__tests__/cache-state-tracker.test.ts --reporter=verbose
```

Expected: all pass.

- [ ] **Step 5: Run full suite**

```bash
npm test
```

- [ ] **Step 6: Commit**

```bash
git add src/orchestrator/__tests__/cache-state-tracker.test.ts
git commit -m "test(isolation): assert workspace_id never shares prefix state"
```

---

## Task 4: Confirm Cache-Stability Gate Coverage

**Why this matters:** The SHA-256 gate is the single hardest invariant to satisfy and the one that blocks merge on failure. We need to confirm it covers all 6 required scenarios and that the 3-run determinism check is actually enforced in CI (not just locally).

**Files:**
- Read only: `src/orchestrator/__tests__/cache-stability.test.ts`
- Read only: `.github/workflows/` (CI config)

- [ ] **Step 1: Read the cache-stability test**

```bash
cat src/orchestrator/__tests__/cache-stability.test.ts
```

Confirm the 6 scenarios from the spec are present:
1. `scenario-1-empty-schemas`
2. `scenario-2-large-schemas`
3. `scenario-3-middle-included`
4. At least 3 more (middle-empty, active-pruning, stub-just-created)

- [ ] **Step 2: Confirm 3-run determinism is enforced**

The test must run the same scenario 3 times and assert `hash1 === hash2 === hash3`. Verify this pattern exists — if it only runs once, the gate doesn't mean what the spec says.

- [ ] **Step 3: Check CI enforces it**

```bash
cat .github/workflows/*.yml | grep -i cache-stability
```

If the cache-stability test file is not explicitly named in a gate step, note it. It should be run on every PR.

- [ ] **Step 4: If any scenarios are missing — add them**

For each missing scenario, create a fixture file in the test fixtures directory (follow the pattern of existing fixtures) and add a `describe` block.

- [ ] **Step 5: Run the stability tests in isolation**

```bash
npx vitest run src/orchestrator/__tests__/cache-stability.test.ts --reporter=verbose
```

Expected: all pass, 3 runs per scenario byte-identical.

---

## Task 5: Confirm Tokenizer Model-String Coverage (4.6 vs 4.7)

**Why this matters:** The spec (REQ-F-003) requires a model-string lookup table so Opus 4.6 and 4.7 produce the right token counts. The tokenizer test has a "M1 gate" asserting 4.7 > 4.6 for the same input (up to 35% more tokens). If this is broken, TTL decisions are wrong.

**Files:**
- Read only: `src/tokenizer/__tests__/tokenizer.test.ts`
- Read only: `src/tokenizer/`

- [ ] **Step 1: Read the tokenizer test**

```bash
cat src/tokenizer/__tests__/tokenizer.test.ts
```

Confirm tests exist for:
- Counting tokens for model `claude-opus-4-6`
- Counting tokens for model `claude-opus-4-7`
- Assertion that 4.7 count ≥ 4.6 count for the same input
- Empty string returns 0
- Unknown model throws or returns a sensible default

- [ ] **Step 2: If the 4.7 > 4.6 gate test is missing — add it**

```typescript
it("M1 gate: claude-opus-4-7 produces >= tokens as claude-opus-4-6 for identical input", () => {
  const text = "Hello, world! This is a test message.";
  const count46 = countTokens("claude-opus-4-6", text);
  const count47 = countTokens("claude-opus-4-7", text);
  expect(count47).toBeGreaterThanOrEqual(count46);
});
```

- [ ] **Step 3: Run tokenizer tests**

```bash
npx vitest run src/tokenizer/__tests__/tokenizer.test.ts --reporter=verbose
```

Expected: all pass.

---

## Task 6: Confirm Agent-Traces Scenario Integrity and Benchmark Methodology

**Why this matters:** The `agent-traces` module generates realistic conversation traces for benchmarking. If the scenarios don't represent real Claude Code sessions (e.g., missing tool calls, wrong block ratios), the benchmark numbers don't generalise. The recorded benchmark must use the cost formula correctly.

**Files:**
- Read only: `src/agent-traces/scenarios.ts`
- Read only: `src/benchmark/recorded.ts`
- Read only: `src/benchmark/__tests__/recorded.test.ts`

- [ ] **Step 1: Read scenarios.ts to understand what trace types exist**

```bash
cat src/agent-traces/scenarios.ts
```

Check: do scenarios include realistic tool-call sequences (bash, file reads, writes)? Is there a mix of short (5-turn) and long (30+ turn) sessions?

- [ ] **Step 2: Read recorded.ts to verify cost formula**

```bash
cat src/benchmark/recorded.ts
```

Find the `effective_cost_units` computation in `generateRecordedBenchmarkReport`. Confirm it matches:
```
input * 1.0 + create_5m * 1.25 + create_1h * 2.0 + read * 0.1
```

- [ ] **Step 3: Read recorded.test.ts to confirm cost assertions**

```bash
cat src/benchmark/__tests__/recorded.test.ts
```

Check that the test has a concrete cost assertion with specific numbers (not just "greater than 0").

- [ ] **Step 4: If cost assertions are vague — tighten them**

Replace any `expect(report.savings_ratio).toBeGreaterThan(0)` with concrete expected values:

```typescript
// For a session with 100 input, 80 cache_read, 0 creation:
// effective = 100 + 8 = 108; baseline = 100 + 80 = 180; savings = 40%
expect(report.savings_ratio).toBeCloseTo(0.40, 2);
```

- [ ] **Step 5: Run benchmark tests**

```bash
npx vitest run src/benchmark/__tests__/recorded.test.ts --reporter=verbose
```

Expected: all pass.

---

## Task 7: Run the Recorded Benchmark and Validate Numbers

**Why this matters:** The live benchmark showed 80% savings, 90.8% cache hit ratio on a 30-turn session. The recorded benchmark should reproduce comparable numbers on synthetic traces to prove the savings are reproducible and not session-specific.

**Files:**
- Read: `scripts/benchmark/run-recorded.ts`
- Read: `docs/benchmarks/` for any existing benchmark output

- [ ] **Step 1: Check what recorded benchmark data exists**

```bash
ls docs/benchmarks/
```

- [ ] **Step 2: Run the recorded benchmark**

```bash
npx tsx scripts/benchmark/run-recorded.ts 2>&1 | tail -30
```

Expected output: a markdown report with `savings_ratio`, `cache_hit_ratio`, turn-by-turn breakdown.

- [ ] **Step 3: Validate the numbers make sense**

The recorded benchmark should show:
- `cache_hit_ratio` > 0.7 for sessions longer than 10 turns
- `savings_ratio` > 0.5 for sessions with stable system prompts
- No NaN or negative cost values
- Effective cost strictly ≤ baseline cost for every turn

If numbers are outside these bounds, investigate — either the scenarios are unrealistic or the cost formula has a bug.

- [ ] **Step 4: Document the benchmark result**

Save the output to `docs/benchmarks/recorded-validation-$(date +%Y-%m-%d).txt`.

---

## Task 8: Final Comprehensive Green-Light

**Why this matters:** Everything must be green simultaneously — tests, types, lint, recorded benchmark. This is the release gate.

- [ ] **Step 1: Full test suite**

```bash
npm test
```

Expected: all test files pass, 0 failures.

- [ ] **Step 2: TypeScript**

```bash
npx tsc --noEmit
```

Expected: no output (clean).

- [ ] **Step 3: Lint**

```bash
npm run lint
```

Expected: no output (clean).

- [ ] **Step 4: Cache-stability gate explicitly**

```bash
npx vitest run src/orchestrator/__tests__/cache-stability.test.ts --reporter=verbose
```

Expected: all scenario hashes byte-identical across 3 runs.

- [ ] **Step 5: Pipeline smoke test explicitly**

```bash
npx vitest run src/proxy/__tests__/pipeline-smoke.test.ts --reporter=verbose
```

Expected: both describe blocks (5m and 1h tier) pass all assertions.

- [ ] **Step 6: Print final summary**

Record the following as the release attestation:
```
Test files:  35 passed
Tests:       [N] passed
TypeScript:  clean
Lint:        clean
Cache-stability gate: passed (6 scenarios × 3 runs)
Pipeline smoke:       passed (5m + 1h tiers)
Recorded benchmark:   savings_ratio=[X]%, cache_hit_ratio=[Y]%
```

- [ ] **Step 7: Commit final state**

```bash
git add -p   # stage only the test additions
git commit -m "test(validation): release validation — all gates confirmed green"
```

---

## Correctness Invariants Checklist (REQ cross-reference)

Before signing off, confirm each invariant has a test pointing at it:

| Invariant | Tested by | Status |
|-----------|-----------|--------|
| I1 — stubs preserve block ID | `pruner.test.ts` (refetch handle) | Confirm |
| I2 — stubs carry refetch_handle | `pruner.test.ts` | Confirm |
| I3 — aging in turns not tool-calls | `pruner.test.ts` (K=3 at turn 4) | Confirm |
| I4 — turn = user+assistant pair | `storage.test.ts` / `server.test.ts` | Confirm |
| I5 — stubs same position as original | `pruner.test.ts` materialization | Confirm |
| I6 — STABLE blocks never ticked | `classifier.test.ts` + `pruner.test.ts` | Confirm |
| I7 — stubs don't tick further | `pruner.test.ts` (already-stubbed skip) | Confirm |
| I8 — tool-use/result pairs atomic | `request-mutator.test.ts` | Confirm |
| Cost formula 5m | `pipeline-smoke.test.ts` (existing) | ✅ |
| Cost formula 1h | `pipeline-smoke.test.ts` (Task 1) | After Task 1 |
| Workspace isolation | `cache-state-tracker.test.ts` (Task 3) | After Task 3 |
| Migration 001–003 | `migrations.test.ts` (Task 2) | After Task 2 |
| Cache stability 6 scenarios | `cache-stability.test.ts` (Task 4) | Confirm |
| Tokenizer 4.6 vs 4.7 | `tokenizer.test.ts` (Task 5) | Confirm |
