# @adia-ai/a2ui-corpus

Corpus and operational-learning artifacts for the gen-UI pipeline —
patterns, fragments, compositions, exemplars, feedback, gaps, eval
fixtures. Pure data plus the scripts that maintain it. No runtime.

> The pipeline reads this package; this package never reads the pipeline.
> See [`@adia-ai/a2ui-compose`](../compose) for engine code,
> [`@adia-ai/web-components`](../../web-components) for UI atoms,
> [`@adia-ai/a2ui-runtime`](../runtime) for the A2UI runtime (renderer,
> registry, streams, wiring),
> [`@adia-ai/a2ui-mcp`](../mcp) for the MCP server.

## Dependency direction

```
a2ui-compose   ──reads──▶  a2ui-corpus
a2ui-mcp       ──reads──▶  a2ui-compose, a2ui-corpus
web-components ◀─reads──   a2ui-corpus (exemplars use AdiaUI tags)
```

No back-writes. No circular reads. Web-components ships UI atoms only;
corpus lives here; runtime lives in a2ui-compose.

## Glossary

Four overlapping concepts circulate through this package. Knowing which
is which prevents reading bugs from creeping into authoring.

| Term | Source of truth | Granularity | What it carries | Engine that consumes it |
| --- | --- | --- | --- | --- |
| **chunk** | `chunks/<id>.json` (one file per chunk) + `_index.json` | Smallest — a single labeled HTML region, harvested via `data-chunk` markers | `html`, `intent`, `domain`, `kind` (`block` / `data-chunk` / `field`), `keywords` | `chunk-zettel` (retrieve → synthesize) |
| **pattern** | `patterns/<domain>/<name>.json` | Full-canvas A2UI template (≈whole-page) | A2UI message tree, retrieval keywords, exemplar provenance | `monolithic-instant` / `monolithic-pro` (retrieve → emit verbatim) |
| **fragment** | `fragments/<domain>/<name>.json` | Atomic shape — re-usable across compositions | A2UI sub-tree, slot bindings, `metrics.leverage` | `zettel` (retrieve → compose into compositions) |
| **composition** | `compositions/<domain>/<name>.json` | Pre-assembled set of fragments | Ordered fragment refs, slot wiring, keywords + tags, `links.used_by` | `zettel` (retrieve → emit verbatim or refine) |

**One-line distinctions:**

- **Chunk vs fragment** — chunks are *harvested*, fragments are *authored*. A chunk is whatever HTML region a `data-chunk` marker pointed at; a fragment is a deliberately-shaped A2UI sub-tree with named slots and leverage metrics.
- **Fragment vs composition** — a fragment is *one shape*; a composition is *several fragments arranged together* into a coherent surface (e.g. `pricing-tiers` composition = header fragment + 3× tier-card fragments + footer fragment).
- **Composition vs pattern** — both are full-surface, but compositions are *fragment-graph* native (zettel reuses the fragment library) while patterns are *flat A2UI templates* (monolithic emits them whole).

**When in doubt, follow the engine:**

- Looking at `monolithic-*` retrieval? You want a **pattern**.
- Looking at `zettel` retrieval? You want a **composition** (or a fragment, if synthesis is composing one).
- Looking at `chunk-zettel`? You want a **chunk**.

## Layout

```
a2ui/corpus/
├── patterns/                ~97 full-canvas A2UI templates (retrieval corpus)
│   ├── <domain>/<name>.json   agent · data · forms · layout · navigation · settings
│   ├── _components.json       canonical name + aliases map (Carousel → Swiper)
│   └── _taxonomy.json         concept synonyms (auth → [login, signup, otp])
│
├── fragments/               atomic shapes — building blocks for Zettel
│   ├── action/                  button rows, toolbars, inline CTAs
│   ├── content/                 stat cards, list items, media blocks
│   ├── form/                    field, field-row, field-group recipes
│   └── layout/                  section, grid, card scaffolding
│
├── compositions/            named multi-fragment recipes
│   ├── agent/ data/ form/ layout/ navigation/ settings/
│                              each = pre-assembled fragment set with keywords + tags
│
├── exemplars/               hand-authored HTML sources
│   ├── ui/                      component-level exemplars
│   └── prose/                   prose + mixed-content exemplars
│                              extract.js compiles these → chunks/ + catalog
│
├── evals/                   held-out.jsonl + eval fixtures
├── chunks/                  gen-UI training chunks (702 unique, 852 instances)
│                              — harvested from data-chunk markers across
│                              site/pages/* + corpus exemplars (since 0.0.3).
│                              Powers compose_from_chunks in @adia-ai/a2ui-mcp.
├── pages/                   extracted full-page A2UI   (build output)
├── feedback/                daily JSONL — user feedback events
├── gaps/                    gap registry — prompts with missing coverage
├── scripts/                 maintenance tooling (extract, ingest, feedback, ticket)
│   └── chunk-library.js       in-memory loader + keyword/semantic search over chunks/
│
├── catalog-a2ui_0_9.json       aggregated artifact — what a2ui-compose + a2ui-mcp read
├── catalog-a2ui_0_9_rules.txt  natural-language composition rules (per component)
├── common_types.json           shared A2UI type shapes
├── pattern-embeddings.json     pre-computed embeddings for pattern semantic search
├── chunk-embeddings.json       pre-computed embeddings for chunk semantic search (0.0.4+)
├── functions.json              declarative wiring-engine function catalog
├── manifest.json               extraction metadata (what / when / counts)
├── pattern-specs.md            written specs for each pattern category
└── data-flow.md                how the 4 signal sources feed the pipeline
```

## What's committed vs generated vs published

| Kind                       | Committed | Published to npm | Source of truth                        |
|----------------------------|:---------:|:----------------:|----------------------------------------|
| Patterns (`patterns/`)     | ✓         | ✓                | Hand-authored JSON                     |
| Fragments/compositions     | ✓         | ✓                | Hand-authored JSON                     |
| Exemplars (`exemplars/`)   | ✓         | ✓                | Hand-authored HTML                     |
| Extracted chunks/pages     | ✓         | ✓                | `scripts/extract.js` (from exemplars)  |
| `catalog-a2ui_0_9.json`    | ✓         | ✓                | `scripts/build-catalog.js` (aggregated)|
| `chunk-embeddings.json`    | ✓         | **✗ since 0.2.1**| `scripts/build/embeddings-chunks.mjs`  |
| `pattern-embeddings.json`  | ✓         | **✗ since 0.2.1**| `scripts/build/embeddings.mjs`         |
| Feedback JSONL             | ✓         | ✓                | Written by `@adia-ai/a2ui-retrieval` at runtime |
| Gap registry               | ✓         | ✓                | Written by `@adia-ai/a2ui-retrieval` at runtime |

Extracted artifacts are committed for convenience (avoids a build step to
read the pipeline), but the scripts are authoritative — regenerate via
`npm run pipeline` if anything drifts.

### Why embeddings ship via git, not npm

`chunk-embeddings.json` (~20 MB) and `pattern-embeddings.json` (~3 MB) are
**committed in git** so the monorepo's own pipeline runs without a network
round-trip, but **excluded from the published npm tarball** — every
`npm i @adia-ai/a2ui-corpus` was pulling 23 MB of pre-computed float
arrays that consumers had no reliable way to address (the
`chunk-embedding-retriever` resolves them via a relative-path that breaks
under a `node_modules/@adia-ai/a2ui-corpus/` install layout).

Consumers who want embedding-based retrieval either:

1. Regenerate locally — `npm run build:embeddings:all` produces both
   files in your `node_modules/@adia-ai/a2ui-corpus/` checkout. Requires
   API access to your embedding provider.
2. Use the keyword-only fallback — `chunk-library.searchChunks()` and
   `pattern-library.searchPatterns()` work without embeddings; the
   embedding-aware `searchChunksAsync` / `searchPatternsAsync` paths
   fall through to keyword scoring when the index file is absent
   (`chunk-embedding-retriever.js` returns `null` gracefully).

### Embedding model pinning

The `provider` and `model` recorded in each `*-embeddings.json` header
are the **source of truth** at query time. The retrievers
(`chunk-embedding-retriever.js`, `embedding-retriever.js`) re-resolve
the same embedder from those header fields — they do **not** auto-pick
a different provider when the recorded one's API key is unset, because
cross-model cosine similarity is meaningless and same-provider/
different-model emits different-dim vectors that `cosine()`
short-circuits to 0 (silent retrieval failure).

Currently pinned defaults:

| Provider | Model | Dims | Env |
|---|---|---:|---|
| openai | `text-embedding-3-small` | 1536 | `OPENAI_API_KEY` |
| voyage | `voyage-3-lite` | 1024 | `VOYAGE_API_KEY` |

`detectProvider()` (in `packages/a2ui/retrieval/embedding/embedding-provider.js`)
prefers Voyage when both keys are present (denser vectors, lower cost).
The `build:embeddings*` scripts record the chosen provider/model into the
`.json` header, so subsequent reads always re-bind to the same model.

When upgrading to a new model (e.g. `text-embedding-3-small` →
`text-embedding-3-large`):

1. Update the default in `embedding-provider.js`.
2. Rebuild **both** indexes (`npm run build:embeddings:all`).
3. Verify with `npm run check:embeddings-fresh` that both index headers
   record the new model.
4. Re-run `npm run eval:diff -- --engine zettel` to confirm the new
   model doesn't regress retrieval quality (different models score
   queries differently — thresholds in `chunk-synthesizer.js` may
   need a re-look, though the absolute keyword score floor remains
   independent).

Don't mix models. If one index records `voyage-3-lite` and the other
records `text-embedding-3-small`, the retrievers will load both fine
but the rankings will be incomparable across the two corpora.

## Scripts

All run from repo root via npm:

```bash
npm run pipeline             # extract + ingest (full refresh of chunks/ + catalog)
npm run pipeline:stats       # report counts without writing
npm run build:catalog        # aggregate patterns/fragments/compositions → catalog
npm run build:catalog -- --verify   # fail if catalog is stale vs source

npm run feedback:report      # human-readable feedback digest
npm run feedback:promote     # promote high-confidence feedback → new patterns

npm run ticket               # open ticket tracker
npm run ticket:list          # list open tickets
npm run ticket:create        # create a ticket against corpus/pipeline
```

Script inventory (`scripts/`):

| Script                    | Purpose                                              |
|---------------------------|------------------------------------------------------|
| `extract.js`              | Exemplar HTML → A2UI chunks & pages                  |
| `ingest.js`               | Index extracted chunks into the retrieval corpus     |
| `run-pipeline.mjs`        | Orchestrates harvest → extract → ingest → index      |
| `build-pattern-index.mjs` | Computes the keyword inverted index (`patterns/_index.json`) |
| `chunk-library.js`        | In-memory loader + keyword/semantic search over `chunks/` |
| `feedback-report.js`      | Aggregates feedback JSONL into a readable digest     |
| `feedback-promote.js`     | Moves high-confidence feedback into the pattern lib  |
| `ticket.mjs`              | Corpus/pipeline issue tracker                        |

Repo-side build scripts (not in tarball; run from the workspace root):

| Script                              | Purpose                                                  |
|-------------------------------------|----------------------------------------------------------|
| `npm run harvest:chunks`            | Walks site/pages/* + corpus exemplars, harvests every `[data-chunk]` element, writes `chunks/<name>.json` + `_index.json` |
| `npm run build:embeddings`          | Generates `pattern-embeddings.json` (113 patterns × 1536d) |
| `npm run build:embeddings:chunks`   | Generates `chunk-embeddings.json` (702 chunks × 1536d)   |
| `npm run build:embeddings:all`      | Both embedding indices in sequence                       |

## Exports

```javascript
// Catalog — the aggregated read-target for engines
import catalog from '@adia-ai/a2ui-corpus';

// Alias map + taxonomy
import aliases from '@adia-ai/a2ui-corpus/patterns';

// Held-out eval fixture
import fixture from '@adia-ai/a2ui-corpus/evals/held-out.jsonl?raw';

// Exemplar source (for scripts + docs)
import html from '@adia-ai/a2ui-corpus/exemplars/ui/pricing-tiers.html?raw';

// Chunk corpus (since 0.0.3 / 0.0.4)
import chunkIndex from '@adia-ai/a2ui-corpus/chunks';   // _index.json (metadata only)
import { searchChunks, searchChunksAsync, getChunk }
  from '@adia-ai/a2ui-corpus/chunk-library';            // in-memory query API
```

## Authoring order — demo page → exemplar → training

When adding coverage for a new intent:

1. **Site demo page** — author the HTML in `site/pages/examples/**`
   using pure primitive composition (no `<style>` block, no inline
   styles, only real components). See repo-root `AGENTS.md`.
2. **Exemplar** — copy the working markup into
   `a2ui/corpus/exemplars/ui/<name>.html`. Strip non-essential chrome;
   keep attributes that encode intent.
3. **Extract + ingest** — `npm run pipeline`. The scripts write
   `chunks/`, `pages/`, and refresh the catalog.
4. **Verify** — `npm run eval:diff -- --engine zettel` should still hold
   coverage 100%, avgScore ≥ 88. Regression thresholds in `AGENTS.md`.

See `data-flow.md` for the full 4-signal pipeline (exemplars →
fragments → compositions → feedback).

## Regression floors

The pipeline must hold these thresholds — tracked in the held-out
benchmark:

- **Fragment reuse ratio ≥ 29.9%** — 167 refs / 559 composition nodes
- **Zettel**: coverage 100%, avgScore ≥ 88, MRR ≥ 0.94
- **Monolithic**: coverage 100%, avgScore ≥ 95
- **Dogfood**: 20/20 intents at avg ≥ 95

## What this package does NOT contain

- Pipeline runtime — `gen-ui/`
- UI custom elements — `web-components/`
- MCP transport — `gen-ui-mcp/`
- Site / playground UI — `/site/`

If a file here is `.js` / `.mjs`, it's a maintenance script, not runtime.
Runtime readers go through `gen-ui/retrieval/*`.

## License

MIT
