# Extraction-Quality Recorded Number (QUAL-01 + QUAL-05)

This is the recorded **extraction-quality number** for the producer-agnostic
instrument (v1.8.1-B). It is the empirical output of running the 98-03 harnesses
over the **orchestrator-supplied frozen producer fixtures** (commit `ccb5d73`).
It is the **GOV-02 precondition-2 input** that the Phase-99 **GOV-04** auto-stage
ruling consults. The instrument scores the producer's emitted atoms/edges (the
OUTPUT) against a frozen gold set; it is producer-agnostic (scores outputs, not
authorship).

> **PROVISIONAL-NOT-RATIFIED — READ THIS FIRST.**
>
> - The gold this number is scored against is **AI-AUTHORED (Claude), NOT human.**
> - `human_ceiling: PENDING (human input required)` — the human-vs-human ceiling
>   is **DEFERRED** per the 2026-06-04 user ruling. No human-validated number is
>   asserted anywhere.
> - Every numeric pass-threshold derived from this number is therefore
>   **PROVISIONAL-not-ratified** (a human ceiling is the precondition to ratifying
>   them).
> - This number **BOUNDS — it does NOT certify** — the auto-stage decision. It is
>   **reported-not-gated**: GOV-04 consults it as a *clearly-provisional* bar; it
>   is never auto-acted-on.
> - **Same-model circularity** is acknowledged (not hidden): gold author and
>   default producer are the same model family (Claude); cross-model codex is a
>   *verifier*, NOT a producer. See
>   `references/extraction-quality-noncircularity-protocol.md` §3.

## Provenance (exactly how this number was produced)

- **Harness commands (re-run by the 98-04 executor over the frozen fixtures):**
  - `bash tests/audit-extraction-quality.sh` (QUAL-01)
  - `bash tests/audit-extraction-lift-rho.sh` (QUAL-05)
- **Frozen fixtures (orchestrator-generated, committed `ccb5d73`):**
  - QUAL-01: `tests/fixtures/extraction-quality/machine-runs/run-{1,2,3}/`
    (`candidate-atoms.tsv` + `candidate-edges.tsv`) — **N = 3** independent
    `gmd-vault-ingest` producer runs over the pinned corpus subset
    (`tests/fixtures/extraction-quality/corpus/`, 10 heterogeneous non-GMD docs).
    The 3 runs emitted 15 / 13 / 15 candidate atoms respectively (real
    agent-loop drift).
  - QUAL-05: `tests/fixtures/extraction-quality/rho-machine-atoms/` — 3 machine
    atoms re-derived from the retrieval-lift corpus **prose only** (the producer
    never saw `qrels.tsv`), substituted into a SANDBOX copy of
    `tests/fixtures/retrieval-lift/`.
- **Gold (scored against):** `tests/fixtures/extraction-quality/gold/`
  (`gold-atoms.tsv`, `gold-edges.tsv`, `negative-spans.tsv`, `adjudication-log.md`)
  — **AI-AUTHORED**, frozen-before-build, with adversarial negative spans and an
  adjudication log (QUAL-02/QUAL-03 disciplines).
- **Both harnesses exited 0** (reported-not-gated; a non-zero exit would signal a
  HARNESS error, not a quality failure).

## QUAL-01 — per-dimension P/R/F1 (raw numerator/denominator)

Scored over the frozen machine-runs. Claim matching is token-Jaccard with TAU
swept {0.5, 0.6, 0.7}. Raw num/den shown (never bare floats). The table below is
the **last scored run** (run-3); the cross-run mean/stdev is in the variance arm.

| Dimension                | TAU | Value | Raw num/den |
| ------------------------ | --- | ----- | ----------- |
| claim-Precision          | 0.5 | 0.667 | 10/15       |
| claim-Recall             | 0.5 | 0.313 | 10/32       |
| **claim-F1**             | 0.5 | 0.426 | —           |
| claim-Precision          | 0.6 | 0.600 | 9/15        |
| claim-Recall             | 0.6 | 0.281 | 9/32        |
| **claim-F1**             | 0.6 | 0.383 | —           |
| claim-Precision          | 0.7 | 0.400 | 6/15        |
| claim-Recall             | 0.7 | 0.188 | 6/32        |
| **claim-F1**             | 0.7 | 0.255 | —           |
| char ≤240 compliance     | —   | 1.000 | 15/15       |
| source-triad complete    | —   | 1.000 | 15/15       |
| edge-FORWARD-P           | —   | 0.000 | 0/5         |
| edge-FORWARD-R           | —   | 0.000 | 0/14        |
| edge-FORWARD-F1          | —   | 0.000 | —           |
| edge-AUTO-INVERSE-P      | —   | n/a   | 0/0         |
| edge-AUTO-INVERSE-R      | —   | 0.000 | 0/10        |
| edge-AUTO-INVERSE-F1     | —   | 0.000 | —           |

Forward and auto-inverse edges are scored in **SEPARATE buckets** (an inverse-pass
bug is not masked by forward success). The frozen producer runs emitted few/no
gold-matching closed-vocab edges, so the edge buckets read 0 — a real, honestly
reported producer characteristic, not a harness artifact.

### Atomicity — HARD sub-floor (never blended)

`atomicity multi-claim-violation-rate = 0.000 (0/15)` — **HARD sub-floor**. A high
violation rate is NOT buyable-back by high P/R elsewhere. Zero of 15 emitted atoms
packed multiple claims.

### Negative-span false-positive rate

`negative-span FP-rate = 0.000 (0/12)` — zero of the 12 adversarial negative spans
(boilerplate / two-claim-must-split / hedged-opinion) drew a false atom.

### Bounded-variance arm — agent-loop DRIFT (stdev across N=3 runs)

Reported (non-gated) run-level stability. Per-mille units 0..1000 for the F1
dimensions. This is the agent-loop drift metric (NOT md5-determinism, NOT a single
temp-0 call) — the auto-stage threshold must be robust to real agent
non-determinism.

| Dimension       | mean  | stdev | n |
| --------------- | ----- | ----- | - |
| CLAIM_F1PM@0.5  | 0.403 | 0.018 | 3 |
| CLAIM_F1PM@0.6  | 0.359 | 0.034 | 3 |
| CLAIM_F1PM@0.7  | 0.273 | 0.018 | 3 |
| CHAR_COMPLY     | 1.000 | 0.000 | 3 |
| TRIAD           | 1.000 | 0.000 | 3 |
| EDGE_FWD_F1PM   | 0.000 | 0.000 | 3 |
| EDGE_INV_F1PM   | 0.000 | 0.000 | 3 |
| ATOMICITY_VIOL  | 0.000 | 0.000 | 3 |
| NEG_FP          | 0.000 | 0.000 | 3 |

### Matcher is load-bearing (RED-on-mutation)

`MUTATION-RED`: garbling one gold claim dropped the recoverable-claim numerator
9 → 8 at TAU0.5 (the matcher actually discriminates). `MUTATION-ROUNDTRIP-MD5`:
the committed `gold-atoms.tsv` is byte-identical after the run
(md5 `27768fb6e05782bf82f48f6d759092ea`) — the sandbox mutation never touched the
real gold.

**QUAL-01 result:** 7 PASS, 0 FAIL (reported), 0 HARNESS-ERROR. EXIT 0.

## QUAL-05 — Extraction-Lift recovery ratio ρ

`ρ = recall@5(machine atoms) / recall@5(human atoms)` on the held-out qrels, over a
SANDBOX copy of `tests/fixtures/retrieval-lift/` (nothing lands in the live
immutable store). The producer **never saw `qrels.tsv`** — ρ is a fair recovery
ratio only because the machine atoms are produced from prose, not fitted to the
queries.

| Slice                         | recall@5 (human) | recall@5 (machine) | ρ     |
| ----------------------------- | ---------------- | ------------------ | ----- |
| FULL qrel set                 | 1.000 (11/11)    | 1.000 (11/11)      | 1.000 |
| HELD-OUT (neutral+off-winnable) | 1.000 (6/6)    | 1.000 (6/6)        | 1.000 |

The held-out 2nd-corpus slice (the `neutral` + `off-winnable` qrel rows, disjoint
from the `enrichment` rows the substituted atoms most directly affect) reads the
same as the full set — no single-corpus overfit signal. The committed
retrieval-lift fixture atoms are **md5-unchanged** after the run (sandbox-sink
invariant held).

**QUAL-05 result:** 5 PASS, 0 FAIL (reported), 0 HARNESS-ERROR. EXIT 0.

ρ = 1.000 means the AI-authored producer atoms recover the held-out retrieval
targets as well as the human-curated VREAD-01 atoms on this fixture. ρ **REPORTS**
a recovery ratio; it does **NOT** certify against a human ceiling.

## AI self-consistency note (NOT the human ceiling)

The N=3-run variance arm above is, in effect, an **AI self-consistency** signal
(how stably the same model family reproduces its own atomization across runs:
claim-F1 stdev 0.018–0.034 per-mille). It is reported **explicitly labeled
AI self-consistency (NOT the human ceiling)**. It measures model self-agreement,
NOT human task-difficulty, and is **never** placed under the "human ceiling"
label. The human-vs-human ceiling remains `PENDING`.

## How GOV-04 (Phase 99) consumes this (the GOV-02 precondition-2 link)

This recorded number — the gold claim-F1 floor (mean 0.403 @TAU0.5 across N=3
runs), the atomicity HARD sub-floor (0/15), the agent-loop variance (stdev
0.018–0.034 per-mille), and ρ (1.000 full + held-out) — is the **GOV-02
precondition-2** input. The Phase-99 **GOV-04** ruling consults it to set/lift the
**auto-stage confidence threshold** for large unsupervised bulk runs. GOV-04 MUST
state in writing that the bar it sets is **PROVISIONAL pending the human-ceiling
spike**. This number **BOUNDS — it does NOT certify** — that decision. Small
human-reviewed batches proceed without it (D-V181-05); trust derives from
volume × reversibility × human-attention, not authorship (D-V181-06).

---

*Harnesses: `tests/audit-extraction-quality.sh` (QUAL-01) +
`tests/audit-extraction-lift-rho.sh` (QUAL-05). Frozen fixtures:
`tests/fixtures/extraction-quality/machine-runs/` (N=3) +
`.../rho-machine-atoms/` (commit `ccb5d73`). Gold: AI-AUTHORED,
`tests/fixtures/extraction-quality/gold/`. Protocol + circularity:
`references/extraction-quality-noncircularity-protocol.md`. Consumer: Phase-99
GOV-04 (provisional bar) + Phase-100 closure audit (QUAL legs).
Recorded: 2026-06-04.*
