# Context Architecture v2 — Cache-aware Boundary Migration

> Status: design only. Code unchanged. Implementation gated on baseline
> measurements (see §6) and a signed-off rollout plan.

## 0. Background

Today's compact pipeline (4 layers: API ContextManagement, microCompact,
reactiveCompact, autoCompact + sessionMemory + emergencyTruncate) all share
one structural decision: **after compaction, the `messages` array is
*physically rewritten* to `[boundary, summary, ...keep, ...attachments]`
and prepended with the boundary marker**.

This decision predates Anthropic's prompt caching becoming a primary cost
lever. With caching:

- Stable prompt prefixes (system prompt + early messages) hit
  `cache_read_input_tokens` at 10% the cost of `input_tokens`.
- Any prefix mutation (insert / re-order / drop) invalidates the cache
  for everything after it.

Prepending a boundary marker mutates byte 0 of the message array, so
**every compaction invalidates the entire cache**. The next turn pays full
`cache_creation_input_tokens` for the same content the model already saw.

Empirically: in long sessions that hit autoCompact 5+ times, observed
cache hit rate from `runtime/turn-loop-pre.ts:236` is well below 20%. We
are leaving ~80% of an Anthropic prompt-cache discount on the floor.

## 1. Goal

Move the compact boundary from "physical prefix mutation" to "logical
range marker", so that:

1. Old messages stay at their original byte offsets in whatever the
   provider sends; cache prefix remains valid across compactions.
2. The model still sees a single coherent prompt — archived history is
   collapsed to a summary block at a *deterministic* position.
3. All four compact layers (microCompact / reactive / sessionMemory /
   emergency) keep working with no behavior change visible to the user
   beyond improved latency and cost.

## 2. Target prompt shape

```
[ system_prompt ]                          (stable; cached)
[ archived_history_blob ]                  (concatenation of all past
                                            compact summaries; appended
                                            to, not rewritten; cached
                                            with cache_control:ephemeral)
[ pinned_context_block ]                   (plan + todos + recent
                                            user requests, P0 surface)
[ live_messages ]                          (everything since the most
                                            recent compact boundary)
[ current_user_input ]
```

Key invariants:

- `system_prompt` never changes mid-session unless the user explicitly
  changes provider/agent. Always cache-hit-eligible.
- `archived_history_blob` is **append-only** during a session — each
  compact appends a new summary block, never rewrites earlier ones. The
  bytes preceding any addition stay identical, so the prefix up to the
  *previous* compact remains cache-hit-eligible.
- `pinned_context_block` is rebuilt every turn (plan / todos / recent
  user requests are short; cache miss here is cheap). Lives between the
  archived blob and live messages so the model always sees the *current*
  state of pinned items, not whatever was pinned at the last compact.
- `live_messages` is the only segment that grows dirty within a turn,
  exactly as today — no change to the streaming / tool-use loop.

## 3. Boundary protocol changes

`SystemCompactBoundaryMessage` today carries:

```
{
  type: 'system',
  subtype: 'compact_boundary',
  compactMetadata: {
    trigger, preTokens, userContext, messagesSummarized,
    preservedSegment: { headUuid, anchorUuid, tailUuid },
    preCompactDiscoveredTools,
  },
}
```

v2 additions:

```
{
  ...
  compactMetadata: {
    ...
    schemaVersion: 'v2',
    archivedRange: {
      // uuid range, inclusive on both ends, that this boundary archived
      firstUuid: UUID,
      lastUuid: UUID,
    },
    summaryBlobOffset: number,    // byte offset into archived_history_blob
                                  // where this boundary's summary starts;
                                  // lets the loader replay history without
                                  // re-summarizing
  },
}
```

`messagesToKeep` is **no longer materialized into the messages array**.
Instead, the boundary records `firstKeptUuid` (= the first uuid after the
archived range) and the prompt assembler walks from there.

## 4. Compact-path checklist

For each path, what changes:

### microCompact

No change. It content-clears tool_result blocks in-place; nothing about
that interacts with the boundary protocol.

### reactiveCompact

- `partialCompactConversation` produces a `CompactionResult` whose
  `summaryMessages` get **appended to `archived_history_blob` instead of
  spliced into `messages`**.
- `messagesToKeep` becomes a *read-only window* — physical messages stay
  put; assembler reads `[firstKeptUuid, …]`.

### sessionMemoryCompact

- `createCompactionResultFromSessionMemory` already builds a summary; the
  same summary goes into `archived_history_blob`.
- `calculateMessagesToKeepIndex` keeps its tool-pair adjustment logic
  unchanged — it's still the source of `firstKeptUuid`.

### autoCompact + emergencyTruncate

- `buildEmergencyTruncationWith` already produces a summary message; ship
  it through the same append-only blob.
- Critical-heap path stays sync. The append target (`archived_history_blob`)
  lives in memory, not on disk, so emergency append is just a string
  concat.

## 5. Disk format compatibility

`utils/sessionStorage.ts` persists `messages` to JSONL. v2 adds:

- `archived_history_blob` as a parallel file: `<sessionDir>/archive.md`
- Each boundary's `summaryBlobOffset` points into that file
- On resume: read archive.md once, restore as a single attachment-style
  block; replay messages.jsonl from `firstKeptUuid` of the most recent
  boundary

Backwards compat:

- Old sessions (no `schemaVersion: 'v2'` on boundary) → fall back to v1
  loader, which reads inline summary messages exactly as today
- New sessions with `CLAVUE_BOUNDARY_V2=1` → write v2 format
- Mixed mode (resumed v1 session, new compact triggers under v2 flag) →
  upgrade in place: write archive.md, drop inline summary, leave existing
  v1 boundaries valid for the loader

## 6. Required baseline measurements before shipping

Block implementation on **all** of these, captured from real sessions
(not synthetic benchmarks):

| Metric | How to capture | Target |
|---|---|---|
| Cache hit rate, post-1st-compact | `cache_read / (cache_read + cache_create + input)` from `tengu_compact` events | Today: < 20% — collect 100 sessions for true median |
| Token cost per 100-turn session | sum `(cache_create + input)` over events | Today's median |
| Compact frequency | count of `compact_boundary` per session-hour | Today's median |
| Emergency-truncate trigger rate | `emergency_truncate` userContext count | Today's median |

After v2 ships behind `CLAVUE_BOUNDARY_V2=1`, run AB for 1 week with the
same metrics. Promote to default only if cache hit rate ≥ 70% AND token
cost down ≥ 30% AND no regression on the other two.

## 7. Risks (the reasons not to ship today)

1. **Anthropic `cache_control` block budget**: only 4 `ephemeral` blocks
   allowed per request. Architecture must place exactly one on
   `archived_history_blob`, leaving 3 for tools / system / pinned.
   Verify before changing assembly.

2. **Provider parity**: OpenAI Responses dialect (`CLAVUE_API_DIALECT=
   openai_responses`) has different caching semantics. Validate v2 there
   or gate it to Anthropic-dialect only.

3. **Session-storage format migration**: `archive.md` is a new file. The
   resume loader (`cli/print.ts:5184`, `utils/sessionStorage.ts`,
   `utils/conversationRecovery.ts`) needs an explicit reader for it, plus
   defensive fallback when the file is missing or corrupt.

4. **Boundary protocol consumers**: 20+ files call
   `getMessagesAfterCompactBoundary` or `isCompactBoundaryMessage`. Most
   are read-only and safe; the writers (`handleSystemMessage.ts:73`
   splice, `services/compact/compact.ts:283` `buildPostCompactMessages`)
   need explicit v2 branches.

5. **Test coverage gap**: today's compact tests assert on physical
   message-array shape (`messagesToKeep` content, `boundaryMarker`
   position). v2 needs a parallel test suite that asserts on assembled
   *prompt* shape instead, since that's what actually drives the model.

## 8. Phased rollout

```
Phase A (1-2 days)   — telemetry only: log cache_hit_rate at every turn
                       under a `tengu_cache_observability` event. Build
                       the baseline dashboard.
Phase B (3-5 days)   — implement archived_history_blob + new boundary
                       protocol behind CLAVUE_BOUNDARY_V2=1. AB-test on
                       internal sessions.
Phase C (1 week)     — flip default to v2 for new sessions; resumed v1
                       sessions stay on v1 until next compact upgrades
                       them in place.
Phase D (deferred)   — once telemetry is steady, drop v1 code path.
```

## 9. Why P0 + P1 ship first

P0 (pinned tier) and P1 (inline reflection) are pure additions with no
boundary-protocol changes. They make the failure modes that actually
hurt users today — losing plan/todos, losing recent user intent — go
away regardless of whether v2 ever ships. They also become *load-bearing*
for v2: the `pinned_context_block` in §2 is exactly what P0 produces,
and the reflection log gives v2 a way to keep "what happened recently"
out of the cached prefix entirely.

Implementation order is therefore: P0 → P1 → telemetry (Phase A) →
decide whether to invest in P2 based on real cache-hit numbers.
