---
title: Extraction-Quality Dimensions Contract
type: reference
tags: [vault, atoms, edges, quality, extraction, QUAL-04, v1.8.1]
status: provisional
created: 2026-06-04
edges:
  refines:
    - "[[vault-atoms]]"
  derived-from:
    - "[[v1.8.1-SPEC]]"
  see-also:
    - "[[vault-edges]]"
    - "[[extraction-quality-noncircularity-protocol]]"
---

# Extraction-Quality Dimensions Contract

This is the single, producer-agnostic SCORING CONTRACT cited by BOTH the
extraction-quality instrument (Phase 98-03, QUAL-01) and the atom producer
(Phase 99). It defines EXACTLY the dimensions the deterministic matcher
(`tests/lib/extraction-quality-matcher.cjs`) computes and the meaning of
each — and it states, prominently, that **every numeric threshold below is
PROVISIONAL-not-ratified** (see [§ Thresholds are provisional](#thresholds-are-provisional)).

The instrument scores the **output** (the atoms and edges a run produces),
NOT the authorship. It therefore applies unchanged to agent-authored atoms
(D-V181-04 MERGE) and **bounds, never certifies**, the auto-stage decision
(D-V181-06: trust = volume × reversibility × human-attention, NOT authorship).
Per GOV-02 precondition-2 it produces a recorded NUMBER consulted by the
Phase-99 GOV-04 ruling; it is reported-not-gated (the harness exits non-zero
ONLY on harness error, exactly as VREAD-01's `tests/audit-retrieval-lift.sh`).

The contract and the matcher code are kept in **lockstep**: the matcher
implements exactly the dimensions defined here (TAU sweep, atomicity
sub-floor, edge forward/auto-inverse buckets, negative-span FP). A change to
one without the other is a defect.

## Inputs the matcher scores against

The gold set is AI-authored (no human available in `--auto`), honestly
labeled (the corpus README, the instrument output, and the Phase-100 closure
audit all state the gold is AI-authored — never assume human gold). It is
frozen-before-build per the
[non-circularity protocol](extraction-quality-noncircularity-protocol.md).
The matcher consumes these TSV shapes (authored by 98-01):

- `gold-atoms.tsv`: `source_doc \t span_anchor \t canonical_claim \t class(core|boundary)`
- `gold-edges.tsv`: `subject_atom \t verb \t object_atom \t class` — gold authors **FORWARD edges only**.
- `negative-spans.tsv`: `source_doc \t span_anchor \t negative_kind \t rationale` — spans that must NOT yield an atom (boilerplate / two-claim-must-split / hedged-opinion).

A candidate atom carries the source triad (`id` + `claim` + `source`) and a
`span_anchor` (so its overlap with a negative span is computable).

## The token-Jaccard matcher

Tokenization (the matcher's `tokenize`): lowercase, split on every
non-alphanumeric character, drop empty tokens. So `"The Closed Vocab!"` →
`["the","closed","vocab"]`.

Jaccard: `J(a,b) = |tokens(a) ∩ tokens(b)| / |tokens(a) ∪ tokens(b)|`.
Identical strings → `1.0`; fully disjoint → `0.0`; two empty strings → `0.0`
(defined as no-overlap, not a divide-by-zero).

A candidate claim **MATCHES** a gold claim at threshold **TAU** iff
`J >= TAU`. Assignment is **greedy best-match**: each gold claim is matched
to AT MOST ONE candidate; among candidates clearing TAU, pick the highest J,
ties broken by candidate order. Each candidate is consumed by at most one
gold claim.

## The scored dimensions

### 1. Atom claim-P/R/F1 (token-Jaccard, TAU swept {0.5, 0.6, 0.7})

What it measures: how well the candidate atom claims recover the gold
canonical claims. How computed: greedy best-match at TAU; P = matched / total
candidates; R = matched / total gold; F1 = harmonic mean. **Reported per-TAU
with RAW numerator/denominator** (mirrors VREAD-01's raw-num/den discipline —
never a bare float). Sweeping TAU upward CANNOT raise recall (a tighter
threshold cannot match MORE): recall is monotonically non-increasing as TAU
rises {0.5 → 0.6 → 0.7}; this monotonicity is asserted by the matcher test.
PROVISIONAL threshold: per-TAU F1 ≥ 0.70 at TAU=0.6 (the swept midpoint).

### 2. Atomicity = multi-claim-violation-RATE (HARD sub-floor, NOT averaged)

This is a **HARD sub-floor gate dimension reported on its own line — NEVER
blended into an aggregate**. A multi-claim violation is a candidate atom that
encodes more than one canonical claim, operationalized as: claim length > 240
chars OR the claim matches ≥ 2 DISTINCT gold claims at the TAU floor. The
violation RATE = violations / total candidate atoms. Because it is a separate
sub-floor, a producer cannot buy back an atomicity failure with high claim
precision elsewhere. PROVISIONAL threshold: violation-rate ≤ 0.05 (a hard
ceiling, separate from every P/R/F1 number).

### 3. ≤240-char compliance rate

What it measures: structural-proxy for atomicity at the field level
(`CLAIM_MAX_CHARS = 240`, the same cap `bin/lib/vault-frontmatter.js`
structurally enforces). Rate = candidates with `claim.length <= 240` / total
candidates. PROVISIONAL threshold: compliance ≥ 0.98.

### 4. Source-triad completeness rate

What it measures: provenance integrity — each candidate atom must carry
`id` + `claim` + `source` (the DEEP-03 required-field triad). Rate =
candidates with all three present and non-empty / total candidates.
PROVISIONAL threshold: completeness = 1.00 (every atom must be traceable).

### 5. Closed-10-verb edge-P/R — forward vs auto-inverse in SEPARATE buckets

What it measures: how well candidate edges recover the gold edges over the
closed-10-verb vocabulary (`supersedes, contradicts, supports, questions,
refines, example-of, derived-from, cited-by, see-also, broader`). Gold
authors FORWARD edges only. The matcher DERIVES the 8 auto-written inverses
via the EDGE_INVERSE_PAIRS table (`supersedes→superseded-by`,
`contradicts→contradicted-by`, `supports→supported-by`,
`questions→questioned-by`, `refines→refined-by`, `cited-by→cites`,
`broader→narrower`, `derived-from→derives`) and scores them as a **DISTINCT
bucket** so an inverse-pass bug is NOT masked by forward-edge success. The two
symmetric verbs (`see-also`, `example-of`) are inert / self-inverse and
contribute no separate inverse. Each bucket reports its own P/R num/den.
PROVISIONAL threshold: forward F1 ≥ 0.70; auto-inverse recall = 1.00 relative
to the inverses derivable from matched forward edges (the inverse pass is
mechanical — a miss there is a bug, not a judgment call).

### 6. Negative-span FP-rate

What it measures: false-positive discipline against the adversarial negative
spans. A candidate atom anchored to (span-overlapping) a negative span is a
false positive. FP-rate = such-atoms / total negative spans. PROVISIONAL
threshold: FP-rate ≤ 0.05.

### 7. Degenerate-arm contract (the metric MUST separate good from bad)

The instrument is only load-bearing if it SEPARATES good output from
degenerate output. This is proven WITHOUT any LLM by two deterministic test
doubles (NOT the producer):

- An **over-splitter** (`tests/fixtures/extraction-quality/synthetic/oversplitter/`)
  shatters a single gold claim into many fragment-atoms. It MUST trip the
  atomicity (precision side) / claim-precision arm — many fragment candidates
  drive claim-precision DOWN versus a faithful producer.
- An **under-splitter** (`tests/fixtures/extraction-quality/synthetic/undersplitter/`)
  welds multiple gold claims into one over-long atom (≥ 1 claim > 240 chars).
  It MUST trip the recall / atomicity (length side) arm — fewer, overlong
  candidates drive claim-recall DOWN and the multi-claim-violation-rate UP.

The matcher test scores both doubles against a faithful baseline and asserts
the faithful run scores STRICTLY BETTER on the relevant arm — that strict
separation is the proof the metric is not vacuous (the QUAL analogue of
VREAD-01's RED-on-mutation arm).

## Thresholds are provisional

**Every numeric threshold in this contract is PROVISIONAL-not-ratified.** The
precondition to ratifying them is the deferred human-vs-human ceiling spike
(the "human-ceiling spike"), which the user RULED deferred (2026-06-04): the
spike harness/scaffold ships, but the human ceiling number is recorded as
`human_ceiling: PENDING (human input required)`. An AI-vs-AI baseline is NOT
the ceiling (it measures model self-consistency, not human task-difficulty)
and must never be presented under the "human ceiling" label. Until the human
ceiling is measured, the Phase-99 GOV-04 ruling consults a clearly-provisional
bar and must state in writing that the threshold is provisional. See
[`references/extraction-quality-noncircularity-protocol.md`](extraction-quality-noncircularity-protocol.md)
for the freeze-before-build + adversarial-negative-span + adjudication-log
protocol and the human-ceiling spike scaffold.

---

*Implementation: `tests/lib/extraction-quality-matcher.cjs` (the deterministic
token-Jaccard scorer) + `tests/lib/extraction-quality-matcher.test.cjs` (the
behavior + degenerate-arm separation proof). Consumed by the Phase 98-03
QUAL-01 harness. Schema upstream: `references/vault-atoms.md` (240-char cap,
source triad) + `references/vault-edges.md` (closed-10 vocab, 8 inverse pairs,
2 symmetric verbs). Thresholds PROVISIONAL pending the deferred human-ceiling
spike.*
