# `InsightReport`: the report

The single shape every analysis call returns. `selfImprove()` embeds it in `SelfImproveResult.insight`; `analyzeRuns()` returns it directly. The hosted-tier wire format carries it on `EvalRunEvent.insightReport?`.

Use `summarizeExecution({ runs })` when observed traces have no task-quality labels.
It returns only `execution` and `costProvenance`, so callers do not need to fabricate a quality score to report runtime facts.

Every section is **opt-in based on what your data supports**: the function never invents signal. If your runs don't carry judge scores, `judges` is empty. If there's no baseline/candidate split, `lift` is undefined. The shape is consistent; population is honest.

This page walks every section with a real (synthetic) example and explains how to act on it.

---

## At a glance

```ts
interface InsightReport {
  n: number                              // runs analyzed
  execution: ExecutionInsight            // duration, tokens, models, failures
  composite: ScalarDistribution          // always
  perDimension: Record<string, ScalarDistribution>   // when judgeScores carry dimensions
  costQuality: { cost: ScalarDistribution; pareto: ParetoFigureSpec }   // always
  judges: Record<string, JudgeInsight>   // when runs carry judge scores
  interRater?: InterRaterInsight         // when raterScores supplied
  lift?: LiftInsight                     // when baseline + candidate present
  failureClusters?: FailureClusterInsight    // when AnalystRegistry wired
  contamination?: ContaminationInsight   // when canaryScenarios supplied
  outcomeCorrelation?: OutcomeCorrelationInsight   // when outcomeSignal supplied
  release: ReleaseSummary                // always
  recommendations: Recommendation[]      // always: read this FIRST
}
```

---

## `execution`: runtime facts, separate from quality

Always present.
It reports duration, optional queue time, direct input, output, reasoning, cache-read, and cache-write tokens, model-call coverage, model cohorts, explicit failures, and separately reported orchestration aggregates.
These fields describe what ran; they do not claim whether the task succeeded.

```jsonc
{
  "execution": {
    "durationMs": { "n": 30, "p50": 5400, "p95": 82000, "min": 900, "max": 190000 },
    "queueMs": { "n": 0, "histogram": [] },
    "tokenUsage": {
      "totals": { "input": 50132, "output": 471783, "reasoning": 12000, "cached": 60489565, "cacheWrite": 3032227 },
      "input": { "n": 30, "p50": 25, "p95": 56 },
      "output": { "n": 30, "p50": 230, "p95": 2651 },
      "reasoning": { "n": 12, "p50": 800, "p95": 2400 },
      "cached": { "n": 20, "p50": 94193, "p95": 310178 },
      "cacheWrite": { "n": 20, "p50": 3070, "p95": 11148 }
    },
    "aggregateUsage": {
      "runs": 2,
      "tokenUsage": {
        "totals": { "input": 5000, "output": 176829, "reasoning": 0, "cached": 0, "cacheWrite": 0 }
      },
      "costUsd": { "n": 0 },
      "totalCostUsd": 0
    },
    "modelCalls": { "runs": 20, "events": 42, "reportingRuns": 30 },
    "models": [{ "model": "claude-opus@2026-07-01", "runs": 20 }],
    "failures": { "runs": 2, "fraction": 0.067, "reportedErrorEvents": 3, "reportingRuns": 30 }
  }
}
```

Use `distribution.n` for optional fields to distinguish an uncaptured category from a recorded zero.
Never add `aggregateUsage` to direct `tokenUsage`: orchestration spans may repeat model-call usage from other traces.
Cost remains in `costQuality`, where observed, estimated, and uncaptured USD stay separate.

---

## `n` + `composite` + `perDimension`: distributional summary

Always present. The basic "where are my numbers" view.

```jsonc
{
  "n": 30,
  "composite": {
    "n": 30,
    "mean": 0.683, "p50": 0.667, "p95": 1.000, "stddev": 0.231,
    "min": 0.0, "max": 1.0,
    "histogram": [
      { "lo": 0.0,  "hi": 0.083, "count": 5 },
      { "lo": 0.083, "hi": 0.167, "count": 0 },
      // ...12 bins by default
    ]
  },
  "perDimension": {
    "clarity":   { "mean": 0.72, "p50": 0.75, "p95": 0.95, "stddev": 0.18, /* ... */ },
    "concision": { "mean": 0.65, "p50": 0.68, "p95": 0.88, "stddev": 0.21, /* ... */ }
  }
}
```

**Read first:** the `composite.mean`. If it's < 0.5, your agent has a ceiling problem, not a tuning problem.

**Read next:** `perDimension`. If `clarity` is high but `concision` is low, your prompts get the right ideas in too many words: different fix than "wrong ideas."

**Use the histogram for:** finding bimodal failure modes. A bin with `count > 0` near zero and another > 0 near 1 means your agent has two distinct behaviors, not one noisy one.

---

## `costQuality`: cost-vs-quality Pareto

Always present. `cost.histogram` is the per-run cost distribution; `pareto` is the substrate's `ParetoFigureSpec`.

```jsonc
{
  "costQuality": {
    "cost": {
      "mean": 0.024, "p95": 0.041,
      "histogram": [/* */]
    },
    "pareto": {
      "kind": "pareto-cost-quality",
      "split": "holdout",
      "axes": { "x": "costUsd", "y": "score" },
      "points": [
        { "candidateId": "baseline", "cost": 0.018, "quality": 0.58, "n": 20, "onFrontier": true },
        { "candidateId": "winner",   "cost": 0.027, "quality": 0.65, "n": 20, "onFrontier": true }
      ]
    }
  }
}
```

**Use this when:** comparing prompts, models, or candidate surfaces. The Pareto frontier is your menu of "best you can do at each cost level."

**Render with:** any chart library: `points` is plain JSON. Hosted-tier dashboards render this as a scatter with the frontier highlighted.

---

## `judges`: per-judge mean

Populated when run records carry `outcome.judgeScores`.

```jsonc
{
  "judges": {
    "domain-expert":   { "n": 30, "meanScore": 0.71 },
    "helpfulness-llm": { "n": 30, "meanScore": 0.62 }
  }
}
```

The substrate's full judge-calibration suite (positional bias, self-preference, verbosity bias) lives in `/reporting` and operates on **paired-by-condition** inputs that `analyzeRuns` doesn't synthesize from raw `RunRecord[]`. Wire them yourself when you have the paired data; the report's `judges` map is the corpus-level slice.

**Use this when:** comparing multiple judges over the same corpus. A big gap between two judges' means is the first signal that one of them is mis-calibrated.

---

## `interRater`: multi-rater agreement and disagreement review

Populated when `analyzeRuns({ raterScores })` is supplied: typically via `fromFeedbackTable()`.

```jsonc
{
  "interRater": {
    "raters": 3,
    "jointlyRated": 30,
    "kappa": 0.40,
    "icc": 0.42,
    "pearson": 0.43,
    "spearman": 0.41,
    "perPair": {
      "alice::bob":   0.53,
      "alice::carol": 0.47,
      "bob::carol":   0.19
    },
    "disagreementCases": [
      { "runId": "claim-7", "range": 1.00,
        "ratings": [{"rater":"alice","score":1},{"rater":"bob","score":1},{"rater":"carol","score":0}] },
      { "runId": "claim-13", "range": 1.00,
        "ratings": [{"rater":"alice","score":0},{"rater":"bob","score":0},{"rater":"carol","score":1}] }
      // ...top 20 by range
    ]
  }
}
```

**Read first:** `kappa` and `icc`, which measure absolute agreement.
Pearson and Spearman measure correlation and can remain high when raters use different score levels.
When absolute agreement is low, review the largest disagreement cases before automating the rubric.

**Use this when:** building per-rater LLM judges. Each rater's individual scores are the gold signal you calibrate against. Once a calibrated LLM matches the human ≥85%, you can auto-grade and escalate only the disagreement cases.

---

## `lift`: paired-bootstrap statistical lift

Populated when baseline + candidate candidates are present (auto-detected from two distinct `candidateId`s, or explicit via `baselineCandidateId` + `candidateCandidateId`).

```jsonc
{
  "lift": {
    "baselineMean": 0.58,
    "candidateMean": 0.65,
    "delta": 0.07,
    "ci95": [0.04, 0.10],          // bootstrap CI on the delta
    "pValue": 0.0008,              // paired t-test
    "n": 40,                       // paired observations
    "cohensD": 0.41,
    "mde": 0.06,                   // min detectable effect at current n, 80% power
    "requiredN": 38                // n needed for observed delta at 80% power
  }
}
```

**Decision rule:**
- `ci95[0] > threshold` → **SHIP.** Lower bound above your delta threshold means the lift is real at 95% confidence.
- `ci95[0] ≤ threshold < ci95[1]` → **INCONCLUSIVE.** Expand the corpus or wait for more data.
- `ci95[1] ≤ threshold` → **HOLD.** No evidence the candidate is better.

The `recommendations` array surfaces exactly this decision (`kind: 'ship' | 'hold' | 'expand-corpus'`): that's what consumers should read.

**Why bootstrap, not t-test alone:** paired bootstrap is distribution-free. Your judge scores are bounded in [0,1] and almost never normal; the bootstrap CI is the honest one.

---

## `failureClusters`: grouped failure modes

Populated when an `AnalystRegistry` is passed via `analyzeRuns({ analyst })`. The substrate runs each failed run through the registered analysts and groups findings by `analyst_id` / `area`.

```jsonc
{
  "failureClusters": {
    "totalFailures": 11,
    "clusters": [
      { "id": "off-topic-drift", "name": "off-topic-drift",
        "share": 0.45, "exemplars": ["run-12", "run-19", "run-33"] },
      { "id": "over-confidence", "name": "over-confidence",
        "share": 0.27, "exemplars": ["run-3", "run-21"] },
      { "id": "format-mismatch", "name": "format-mismatch",
        "share": 0.18, "exemplars": ["run-41", "run-44"] }
    ]
  }
}
```

**Read first:** the top cluster's `share`. If one cluster is > 40% of failures, fix that pattern before doing anything else.

**Use this when:** triaging a regression. Failure clusters tell you "fix this kind of thing first."

**To wire it:** register analysts in `AnalystRegistry`. See `src/analyst/registry.ts` and `src/analyst/kinds.ts` for the four built-in kinds (`failure-mode`, `improvement`, `knowledge-gap`, `knowledge-poisoning`).

---

## `contamination`: canary check

Populated when canary scenarios are passed via `analyzeRuns({ canaryScenarios })`. Each canary carries a sentinel string the agent should never emit; the report counts leaks.

```jsonc
{
  "contamination": {
    "leaks": 0,
    "holdoutAuditPassed": true,
    "details": []
  }
}
```

When `leaks > 0`:

```jsonc
{
  "contamination": {
    "leaks": 2,
    "holdoutAuditPassed": false,
    "details": [
      { "runId": "run-12", "canary": "xyz-secret-canary-123", "matched": "...the secret xyz-secret-canary-123 says..." }
    ]
  }
}
```

**When this fails:** your holdout corpus has leaked into training context. The `lift` number is **unreliable**. Investigate before shipping anything.

---

## `outcomeCorrelation`: closing the loop on real outcomes

Populated when `outcomeSignal: { metric, valueByRunId }` is supplied.

```jsonc
{
  "outcomeCorrelation": {
    "metric": "engagement_rate",
    "n": 80,
    "pearson": 0.72,           // linear correlation
    "spearman": 0.69,          // rank correlation (robust to monotonic nonlinearity)
    "rewardModel": {
      "intercept": 0.04,
      "slope": 1.93,
      "r2": 0.52               // share of outcome variance the judge explains
    }
  }
}
```

This is the layer that says **"does my judge's taste actually predict the metric the business cares about?"**

**Read first:** `spearman`. If it's < 0.3 in absolute value, your judges are scoring something different from what wins downstream. Refit the judges (use the customer's downstream signal as gold) or change the rubric.

**The reward model** is the simple linear `y = intercept + slope * composite`. Use it to:
- Predict the engagement of a new run from its composite score alone.
- Set a `composite` threshold for "must beat X to ship" based on the engagement equivalent.

---

## `release`: pass/warn/fail axes

Always present. Roll-up across three axes: quality lift, contamination, composite distribution.

```jsonc
{
  "release": {
    "status": "pass",
    "axes": [
      { "name": "quality-lift", "status": "pass",
        "detail": "delta=0.070, CI95=[0.040, 0.100], n=40" },
      { "name": "contamination", "status": "pass",
        "detail": "0 canary leak(s)" },
      { "name": "composite-distribution", "status": "pass",
        "detail": "mean=0.683, p50=0.667, p95=1.000 over n=30" }
    ],
    "issues": []
  }
}
```

Overall `status` is `fail` if any axis fails; `warn` if any warn; `pass` otherwise.

**Use this when:** wiring agent-eval into CI. A `status === 'pass'` from `analyzeRuns` on the candidate vs baseline is your green-light gate.

---

## `recommendations`: the actionable layer

Always present. Read this first.

```jsonc
{
  "recommendations": [
    { "priority": "critical", "kind": "ship",
      "title": "Ship: lift 0.070 (95% CI 0.040..0.100)",
      "detail": "Holdout lift exceeds threshold 0.02 with 95% bootstrap confidence (n=40, p=0.0008, d=0.41).",
      "evidencePath": "lift" },
    { "priority": "high", "kind": "investigate",
      "title": "Top failure cluster: off-topic-drift (45% of failures)",
      "detail": "11 runs failed. The largest cluster groups 3 exemplars under 'off-topic-drift'.",
      "evidencePath": "failureClusters.clusters[0]" }
  ]
}
```

| `kind` | When emitted |
|---|---|
| `ship` | lift CI lower bound > threshold |
| `hold` | lift CI upper bound ≤ threshold |
| `expand-corpus` | lift CI straddles threshold: more data needed |
| `fix` | canary contamination detected |
| `recalibrate` | inter-rater κ < 0.5, OR outcome correlation < 0.3 |
| `investigate` | top failure cluster > some-share |

`evidencePath` points back into the report (`"lift"`, `"contamination"`, `"failureClusters.clusters[0]"`) so a UI can deep-link from each recommendation to its evidence.

---

## How `analyzeRuns` populates each section

| Section | Required input |
|---|---|
| `composite`, `perDimension`, `costQuality`, `release`, `recommendations` | `runs` |
| `judges` | `runs` with `outcome.judgeScores` |
| `interRater` | `raterScores` (≥ 2 raters jointly rated some runs) |
| `lift` | two distinct `candidateId`s in `runs` (or explicit baseline/candidate ids) |
| `failureClusters` | `analyst` registry passed in |
| `contamination` | `canaryScenarios` passed in |
| `outcomeCorrelation` | `outcomeSignal` passed in |

All sections beyond the always-present ones are `T | undefined`, never empty objects. If a section is missing, your inputs didn't support it: the report is honest about that.
