# Overlay-Load Telemetry

This doc describes the telemetry layer that records, **objectively**, whether skill overlays are actually loaded at runtime. The layer is **observation only** — it changes no behaviour. It exists so we can decide, with real data, whether skill overlays need a stronger delivery guarantee than the current runtime-concat model.

## Why this exists

Skill overlays and agent/command overlays have **opposite delivery guarantees**:

- **Agent / command overlays** are *compile-time merged* — the overlay is materially fused into the generated `.claude/agents/<x>.md` at `add`/`update` (`overlay-merger.js`). When Claude loads the subagent, the overlay is already there. The guarantee is structural.
- **Skill overlays** use *runtime concatenation* — `SKILL.md` stays a symlink, the overlay lives separately in `.baldart/overlays/<skill>.md`, and a line of prose tells the agent "load `.baldart/overlays/<skill>.md` if present". Nothing verifies the agent did. The guarantee is best-effort.

So the *most* important overlays for knowledge transfer (skills) have the *weakest* guarantee. The architectural fixes for that (SessionStart awareness injection, PreToolUse-on-Skill injection, or compile-time merge of `SKILL.md`) are all non-trivial — and we have **no evidence** overlays actually fail to load in practice. Before building any of them, we measure. This layer is that measurement.

## What it captures (and what it can't)

A single PostToolUse hook (`framework/.claude/hooks/overlay-telemetry.sh`, matcher `Read`) appends one JSON line to `.baldart/telemetry/overlay-loads.jsonl` every time a `.baldart/overlays/*.md` file is read:

```json
{"ts":"2026-05-29T08:00:00Z","event":"overlay-read","kind":"skill","name":"bug","session_id":"..."}
```

### Why a hook, not agent self-report

The failure mode we measure is *"the agent forgot to load the overlay."* A self-reported `loaded=yes/no` would be corrupted in exactly that case — an agent that forgot to load would also forget to log, or log `yes` without having done it. Only an objective `Read` observation is trustworthy here.

### Numerator only — the known limitation

We capture the **numerator** (overlay reads), not the **denominator** (skill runs that *should* have loaded an overlay). The denominator is not hook-observable: skills are *prompt-expansion* in Claude Code, not tool calls, so there is no `PreToolUse` on skill invocation and the tool-call payload does not carry the invoking skill's name (anthropics/claude-code [#43630](https://github.com/anthropics/claude-code/issues/43630), [#22655](https://github.com/anthropics/claude-code/issues/22655)).

This makes the metric a **conservative proxy that UNDER-counts**:

- Overlay content already in the agent's context (no explicit `Read`) is invisible.
- A `Read` is not proof the content influenced the output — only that it was loaded.
- Reads performed in ways the main-loop hook doesn't observe are missed.

Every bias points the same direction: the observed load-rate looks **worse** than reality. That matters because under-counting biases toward "build the enforcement" — the very thing this telemetry exists to gate. **Treat a low count as a reason to investigate, not as proof of a miss.** (Same honesty convention as `agent-discovery-info.sh`'s #10373 caveat and the "conservative proxy" language in `FIX-APPLICATION-TELEMETRY.md`.)

## How to read the data

```bash
node framework/scripts/analyze-overlay-loads.js
# or, non-default locations:
node framework/scripts/analyze-overlay-loads.js --log <path.jsonl> --overlays <dir>
```

The aggregator cross-references the read log against the overlays that **exist on disk** and prints per-overlay read counts. The headline signal:

| Column / footer | What it means |
|---|---|
| `Flag = NEVER-LOADED` | A **skill** overlay exists but was never read. The alarm signal — authored knowledge the agent isn't picking up. (Subject to the under-count caveat above.) |
| `Reads` low but non-zero | Overlay loads sometimes. Compare against how often the skill ran (you'll have to estimate the denominator from your own usage). |
| Agent/command rows, zero reads | **Expected, not a finding.** These overlays are compile-time merged into the generated file — they're not meant to be read at runtime. The table does not flag them. |
| `Reads logged for overlays no longer on disk` | Stale/renamed overlays. Cosmetic. |

## When to act on the data

After a few weeks of real usage with at least one authored skill overlay:

1. **`NEVER-LOADED` skill overlays.** If a skill overlay you actively rely on shows zero reads across many sessions where that skill clearly ran, that's the strongest available evidence that runtime concat is dropping it — and the case for an enforcement mechanism (SessionStart awareness, PreToolUse injection, or `SKILL.md` compile-time merge) becomes data-backed rather than speculative.
2. **Healthy load pattern.** If your authored skill overlays show steady reads, the runtime-concat model is working in practice and enforcement is not worth its cost. Ship the DX improvements and leave the architecture alone.

## Cost

Be honest about where the cost lands: the hook fires on **every `Read`** — the most frequent tool — not just on overlay reads. So the per-read cost is what matters, and it is paid hundreds of times per session in the PostToolUse critical path (the model waits for the hook before seeing the tool result).

- **Every read** (overwhelmingly non-overlay): one bash fork + a path-prefix match, then exit. This is why the hook is bash, not node — a node cold-start (~50ms) per read would be a real tax; a bash fork (~5ms) keeps the hot path cheap. There is no cheaper event: observing "an overlay was read" *requires* hooking `Read`, so the only lever is making the hook itself cheap.
- **Only on an actual overlay read** (rare): one `mkdir -p` + one ~120-byte append.

The hook is fail-safe (any error → exit 0, never blocks). The aggregator has zero dependencies and runs in well under a second. The log is append-only and rotates only by hand (delete the file to reset the window).

## Pointers

- Hook: `framework/.claude/hooks/overlay-telemetry.sh`
- Registration: `src/utils/hooks.js` → `HOOK_REGISTRY` entry `baldart-overlay-telemetry` (auto-registered by `add`, backfilled by `update`, reported by `doctor`)
- Aggregator: `framework/scripts/analyze-overlay-loads.js`
- Overlay model: `framework/agents/project-context.md` §5 (skills, runtime concat) and §5.bis (agent/command, compile-time merge)
- Sibling telemetry layer: `framework/docs/FIX-APPLICATION-TELEMETRY.md`
