{"_id":"@apatureai/sigil","name":"@apatureai/sigil","dist-tags":{"latest":"0.1.1"},"versions":{"0.1.1":{"name":"@apatureai/sigil","version":"0.1.1","license":"MIT","type":"module","description":"Error bars for LLM-as-judge evals: calibration (ECE/Brier/reliability), finite-sample risk certificates (Clopper-Pearson, Learn-Then-Test), Pass^k reliability, anytime-valid drift monitoring, and a cost-at-equal-quality frontier gated by an exact McNemar ","author":{"name":"Aditya Prathapa"},"repository":{"type":"git","url":"git+https://github.com/apatureai/sigil.git"},"homepage":"https://github.com/apatureai/sigil#readme","bugs":{"url":"https://github.com/apatureai/sigil/issues"},"keywords":["llm-as-judge","calibration","conformal-prediction","e-process","model-risk","eval"],"engines":{"node":">=24"},"main":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"},"./package.json":"./package.json"},"bin":{"sigil":"dist/bin.js"},"publishConfig":{"access":"public"},"devDependencies":{"@eslint/js":"^10.0.1","@types/node":"^26.1.1","eslint":"^10.7.0","fast-check":"^4.9.0","typescript":"6.0.3","typescript-eslint":"^8.64.0","vitest":"4.1.10"},"scripts":{"typecheck":"tsc --noEmit","test":"vitest run","lint":"eslint .","build":"tsc -p tsconfig.build.json"},"_id":"@apatureai/sigil@0.1.1","_integrity":"sha512-K0KT7DJXnHeTHyE71FuHATDDrcdpYWLi9WDWkUbOrd1K+k8A/Hj3tM3WQue+o6ev/HFOl6bDvU2Ae4EWqLbHcA==","_resolved":"/tmp/98ca21e75c440e2a94d1837685110f9d/apatureai-sigil-0.1.1.tgz","_from":"file:apatureai-sigil-0.1.1.tgz","_nodeVersion":"24.19.0","_npmVersion":"11.17.0","dist":{"integrity":"sha512-K0KT7DJXnHeTHyE71FuHATDDrcdpYWLi9WDWkUbOrd1K+k8A/Hj3tM3WQue+o6ev/HFOl6bDvU2Ae4EWqLbHcA==","shasum":"bae32fc1df6d3da7c9e2cb64eaefb0e72a6eed80","tarball":"https://registry.npmjs.org/@apatureai/sigil/-/sigil-0.1.1.tgz","fileCount":52,"unpackedSize":193440,"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@apatureai%2fsigil@0.1.1","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIASozizz1urzgE4Hy1ryScqOwRY8o+PCTUDkw6clHxWLAiEAkAuwLnEeROhb5foNBF7UqlxAlxgWe90cgKQ63DP12gA="}]},"_npmUser":{"name":"aprathapa","email":"aprathapa01@gmail.com"},"directories":{},"maintainers":[{"name":"aprathapa","email":"aprathapa01@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/sigil_0.1.1_1787613389523_0.7780489027284492"},"_hasShrinkwrap":false}},"time":{"created":"2026-08-24T23:16:29.326Z","0.1.1":"2026-08-24T23:16:29.743Z","modified":"2026-08-24T23:16:30.255Z"},"maintainers":[{"name":"aprathapa","email":"aprathapa01@gmail.com"}],"description":"Error bars for LLM-as-judge evals: calibration (ECE/Brier/reliability), finite-sample risk certificates (Clopper-Pearson, Learn-Then-Test), Pass^k reliability, anytime-valid drift monitoring, and a cost-at-equal-quality frontier gated by an exact McNemar ","homepage":"https://github.com/apatureai/sigil#readme","keywords":["llm-as-judge","calibration","conformal-prediction","e-process","model-risk","eval"],"repository":{"type":"git","url":"git+https://github.com/apatureai/sigil.git"},"author":{"name":"Aditya Prathapa"},"bugs":{"url":"https://github.com/apatureai/sigil/issues"},"license":"MIT","readme":"# Sigil\n\n[![CI](https://img.shields.io/github/actions/workflow/status/apatureai/sigil/ci.yml?branch=main&label=CI)](https://github.com/apatureai/sigil/actions/workflows/ci.yml) [![license](https://img.shields.io/github/license/apatureai/sigil?color=blue)](https://github.com/apatureai/sigil/blob/main/LICENSE) [![node](https://img.shields.io/badge/node-%3E%3D24-brightgreen)](#requirements)\n\n> Part of the [Apature stack](https://github.com/apatureai): automated design review for rendered UI. The [org profile](https://github.com/apatureai/.github/blob/main/profile/README.md) maps how the pieces compose.\n\n**Error bars for LLM-as-judge evals: calibration, finite-sample risk certificates, and anytime-valid drift monitoring, in dependency-free TypeScript.**\n\nIf you use a model to grade another model's output, you have a ruler you have never measured.\nSigil measures it. Give it your judge's verdicts, a set of human-labeled examples, and a captured\nrun of the models you are comparing, and it will tell you how well calibrated the judge is, what\nconfidence threshold you can defensibly abstain at, how reliable each model is across repeated\nruns, and which cheaper model actually holds quality once a paired significance test has had its\nsay.\n\nIt is a pure measurement engine. It calls no model, holds no key, and opens no socket. You supply\nthe judge and the labels; Sigil scores them and emits a byte-reproducible report.\n\n```\npnpm install --frozen-lockfile && pnpm build\nnode dist/bin.js examples/credit-memo out/ && cat out/report.md\n```\n\n## Who this is for\n\n- **People running LLM-as-judge eval pipelines in Node** who need calibration numbers with error\n  bars rather than a single accuracy figure.\n- **Anyone shipping selective prediction** who is currently choosing an abstention threshold by\n  intuition and wants one with a finite-sample guarantee attached.\n- **Teams doing model-cost work** who need \"switch to the cheaper model\" to be an evidenced claim\n  rather than a comparison of two averages.\n- **JS/TS people locked out of the stats ecosystem.** Conformal prediction, e-processes and\n  sequential testing are almost entirely Python and R. This is one of very few JavaScript\n  implementations, and it has zero runtime dependencies.\n\nSigil is deliberately vendor-neutral: it has no opinion about which model you should use and no\ncode path that could profit from the answer.\n\n## Why it is technically interesting\n\nMost eval harnesses report a number. Sigil's premise is that **an unqualified number is not\nevidence**, and nearly every design decision follows from that.\n\n**1. The measurement is itself measured.** Before any quality claim, `metrics.ts` scores the judge\nagainst human labels: expected calibration error (does \"0.8 confidence\" hold about 80% of the\ntime?), Brier score, and a binned reliability table. A badly calibrated judge is surfaced, not\nhidden.\n\n**2. Finite-sample guarantees, not asymptotics.** `conformal.ts` computes exact one-sided\nClopper-Pearson binomial bounds on the judge's error rate, and certifies an *abstention threshold*\nvia fixed-sequence Learn-Then-Test over a data-independent confidence grid: walk thresholds from\nmost to least conservative, test at each with the exact bound, stop at the first failure. The\nfamily-wise guarantee survives because the tests are ordered a priori. The output is a sentence you\ncan file: *\"on the X% of cases the judge accepts, its error rate is at most alpha with confidence\n1 minus delta; the rest go to human review.\"* Exact bounds were chosen over Gaussian intervals\nbecause audit sample sizes are small. When nothing certifies, the module **refuses** rather than\nreturning an uncertified threshold. Split conformal plus LTT was chosen over full conformal because\nit needs only the labeled corpus you already have.\n\n**3. Anytime-valid sequential monitoring.** `drift.ts` implements betting supermartingales over\nbounded [0,1] observation streams (judge-error indicators), with two constructions whose guarantees\nare deliberately never blurred: a fixed-null **e-process** where Ville's inequality bounds the\nprobability of *ever* false-alarming over an unbounded horizon by alpha, and a changepoint\n**e-detector** (e-CUSUM style) that trades that for an average-run-length-to-false-alarm bound of\nat least 1/alpha in exchange for retained sensitivity to late changes. A lambda-grid mixture\nreplaces hyperparameter tuning (a mixture of supermartingales is a supermartingale). Classical Page\nCUSUM ships alongside as the explicitly weaker disclosed baseline. Monitor state is plain\nserializable data, so a run can be persisted, resumed, and replayed from the observation log.\n\n**4. Headline claims are gated by evidence, and can fail.** \"Switch model X to model Y, save 97% at\nequal quality\" is a point comparison of aggregate means, not evidence. `stats.ts` runs an exact\ntwo-sided **McNemar test** on *paired* per-task outcomes and marks the switch **not defensible**\nwhen the cheaper candidate is significantly worse, regardless of how good the cost delta looks. \"No\ndetected loss\" is reported as exactly that and never upgraded to \"equal\". Similarly, the Pass^k\npoint estimate gets a certified Clopper-Pearson floor, with the i.i.d.-runs assumption stated in\nthe output.\n\n**5. Ordinal and cardinal reliability are separated.** `rank-score.ts` measures Kendall's tau-b\nbetween judge scores and reference quality *and* the empirical width of the score-to-quality\nresidual interval. A judge frequently ranks well while its absolute numbers are noise; that is\nprecisely the regime where \"use the ordering, distrust the magnitude\" is the correct guidance, and\nit is invisible to ECE alone.\n\n**6. Determinism as an architectural constraint.** Canonical JSON (sorted keys, RFC-8785 style)\nplus SHA-256 content addressing for the corpus and the report. No RNG and no wall clock anywhere in\nthe analysis path; the only injected non-determinism is the report timestamp, which is recorded\nrather than read from a clock. The same frozen corpus plus the same frozen panel produces a\nbyte-identical report, which is what makes third-party reproduction possible at all.\n\n**7. Fail-closed data egress.** `egress.ts` refuses to release any artifact containing a raw model\noutput, a raw prompt, or a credential-shaped string. It searches both the artifact's own strings\n(keys, values, array elements) and its JSON serialization, because matching only the serialization\nmisses every needle holding a character JSON escapes, which is to say every multi-line or quoted\nmodel output. The end-to-end golden test deliberately plants a PII-leaking model output in the\ncorpus and asserts it cannot reach the exported deliverable.\n\n**8. Signed, offline-verifiable artifacts.** `bundle.ts` produces a detached Ed25519 signature over\nthe canonical `{documentHash, markdownHash}` payload. Verification re-derives everything with no\nnetwork and fails closed with named reasons (document-hash mismatch, markdown tamper, payload swap,\nsignature failure, unknown key id). Ed25519 is used because RFC 8032 signatures are deterministic,\nkeeping bundles byte-stable. Signer and verifier are injected ports; the repo ships no key.\n\n**9. Ports all the way down.** `Gateway`, `Judge`, `GroundTruth`, `BundleSigner`/`BundleVerifier`,\n`fetchImpl`, and the clock are all injected. That is why the whole test suite runs in a few seconds\nwith no model, key, or network, and why the property tests (`fast-check`) can hammer the\ncertificate math directly.\n\n## Requirements\n\n| Thing | Need | Check |\n|---|---|---|\n| Node | 24 or newer (`engines: >=24`; `.node-version` pins 24, which is what CI and every verification run used) | `node -v` |\n| pnpm | 9 or 10 (`lockfileVersion: 9.0`) | `pnpm -v` |\n| OS | verified on macOS 15 (Darwin 24.6.0); CI runs ubuntu-latest | n/a |\n\nIf pnpm is missing: `corepack enable pnpm` (ships with Node), or `npm install -g pnpm`.\n\n**No credentials, no API keys, and no network access are needed for anything in this README.** Sigil\nreads no environment variables at all; the only `process` use in `src/` is `argv`, `exit`, and\nwrites to stderr. Dependencies are pinned, `pnpm-lock.yaml` is committed, and every command below\ninstalls with `--frozen-lockfile`.\n\nNode 24 specifically is needed for one reason beyond `engines`: the library example below is a `.ts`\nfile run directly by Node's native type stripping.\n\n## Install\n\nFrom the repo root, on a clean clone:\n\n```\npnpm install --frozen-lockfile\npnpm build\n```\n\n`pnpm build` is part of installing, not an optional extra: `tsconfig.json` sets `noEmit: true`\n(so `pnpm typecheck` is the real compile gate), and the CLI runs from the emitted `dist/`.\n\nTranscript:\n\n```\n$ pnpm install --frozen-lockfile\nLockfile is up to date, resolution step is skipped\nPackages: +132\n++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\nProgress: resolved 132, reused 131, downloaded 0, added 132, done\n\ndevDependencies:\n+ @eslint/js 10.0.1\n+ @types/node 26.1.1\n+ eslint 10.7.0\n+ fast-check 4.9.0\n+ typescript 6.0.3\n+ typescript-eslint 8.64.0\n+ vitest 4.1.10\n\nDone in 804ms using pnpm v10.34.3\n\n$ pnpm build\n\n> @apatureai/sigil@0.1.1 build /path/to/sigil\n> tsc -p tsconfig.build.json\n\n```\n\n## Quickstart\n\nRun the shipped audit. `examples/credit-memo/` is a complete synthetic input bundle holding a\nfrozen corpus with human labels, a captured three-model panel run, and captured judge verdicts. No\ncredentials, no network.\n\n```\nnode dist/bin.js examples/credit-memo out/\ncat out/report.md\n```\n\nTranscript:\n\n```\n$ node dist/bin.js examples/credit-memo out/\nwrote report.{json,md}, router-policy.json, governance.json to out/\n\n$ cat out/report.md\n# Independent AI Quality & Efficiency Assurance: Example Bank\n\n- Corpus: `sha256:d0483ca33ca972f0651497eaa07e4c8d21abdc3c9743f9b92e9637ce39a974e1`\n- Panel: frontier, budget, thrifty\n- Generated: 2026-06-30T00:00:00.000Z\n- Document hash: `sha256:91f91ffe93aca39e2b031feb3007e83104c08a8b78df8fdd77f63c068df8be42`\n\n## Judge reliability (the number's own error bars)\n- ECE: 0.1 · Brier: 0.01 · sample: 12\n\n## Efficiency frontier (equal-quality savings)\n- **credit_memo**: switch frontier → thrifty, save 96.7% at equal-or-better measured quality\n\n## Run-to-run reliability exposure (Pass^k)\n- budget: worst-case Pass^k = 0.25 (inconsistent across runs)\n- frontier: worst-case Pass^k = 1\n- thrifty: worst-case Pass^k = 1\n```\n\n**Success criterion:** `out/` contains four files (`report.json`, `report.md`,\n`router-policy.json`, `governance.json`), and the document hash printed above is exactly\n`sha256:91f91ffe93aca39e2b031feb3007e83104c08a8b78df8fdd77f63c068df8be42`. That hash is\ncontent-addressed over the whole report, so matching it means your run reproduced the audit\nbyte for byte.\n\nIf `node dist/bin.js` reports `Cannot find module`, you skipped `pnpm build`.\n\n### Reading the result\n\nThe numbers are the point of the exercise:\n\n- `budget` costs 10x less than `frontier`, but it failed one of its four recorded runs, so its\n  Pass^3 collapses to 0.25. It is **not** recommended, and the cost delta does not rescue it.\n- `thrifty` held quality exactly, at 1/30th the cost of `frontier`. That is the recommendation.\n- ECE 0.1 is reported *before* any savings claim. If the judge were badly calibrated, you would see\n  that first rather than being sold a saving measured with a broken ruler.\n- The raw task input (`Summarize creditworthiness for applicant 4821`) and the raw model outputs\n  appear in **none** of the four written files. The egress guard runs before anything is written.\n\nThe other three artifacts:\n\n```\n$ cat out/router-policy.json\n{\n  \"policyVersion\": \"neutral-route/1\",\n  \"qualityFloor\": 0.9,\n  \"routes\": [\n    {\n      \"family\": \"credit_memo\",\n      \"primary\": \"thrifty\",\n      \"fallbacks\": []\n    }\n  ]\n}\n\n$ cat out/governance.json\n[\n  {\n    \"agentId\": \"credit-bot\",\n    \"code\": \"excess_scope\",\n    \"severity\": \"warning\",\n    \"detail\": \"agent holds scopes no observed task requires (least-privilege gap)\",\n    \"scopes\": [\n      \"read:ssn\"\n    ]\n  }\n]\n```\n\n`report.json` is the same content as `report.md` in structured form, ending in the `documentHash`.\n\n## Usage\n\n### CLI\n\n```\nnode dist/bin.js <bundle-dir> [out-dir]\n```\n\n`out-dir` defaults to `<bundle-dir>/out`, which writes inside the bundle directory, so pass an\nexplicit out-dir if you want the output somewhere else. Exit code `0` on success, `2` with a usage\nmessage when `<bundle-dir>` is omitted, `1` on any failure (including an egress violation, a bundle\nasking for more trials than it captured, or a `passK` larger than the recorded runs, in which case\nthe CLI writes nothing at all).\n\nThe package is not on npm yet, so there is no globally installable `sigil` command; invoke the\nbuilt entry point directly, as above. The usage line it prints on exit `2` reads\n`usage: audit <corpus-dir> [out-dir]`, using the tool's internal name. Publishing is on the\n[roadmap](#status-and-roadmap).\n\n### The input bundle\n\nA bundle directory contains four required JSON files and one optional one. To audit your own\nsystem, copy `examples/credit-memo/` and replace the contents.\n\n| File | Required | Contents |\n|---|---|---|\n| `config.json` | yes | Run configuration; see the table below |\n| `corpus.json` | yes | `{ rubric, tasks[] }`; every task carries human `labels` of `{ output, accept }`. This is the frozen, content-addressed benchmark set, and the ground truth is derived from it (an unlabeled output is conservatively treated as not-accepted) |\n| `panel.json` | yes | The captured panel run, per model: `{ costUsd, latencyMs, outputs: { [taskId]: string[] } }`. One array entry per **distinct recorded trial**, so run-to-run variance is expressible. Every `(model, taskId)` must hold at least `trialsPerTask` entries; see below |\n| `judge.json` | yes | Judge verdicts keyed by raw output string: `{ pass, confidence }`. An output with no verdict is treated as `{ pass: false, confidence: 0.5 }` |\n| `governance.json` | no | `{ agents[], requirements[] }` for the least-privilege overlay. Omit it and the overlay returns `[]` |\n\n### Configuration\n\nSigil reads **no environment variables**. All configuration is the `config.json` in the bundle:\n\n| Field | Required | Effect |\n|---|---|---|\n| `client` | yes | Name printed in the report header |\n| `models` | yes | The candidate panel; must match the keys in `panel.json` |\n| `trialsPerTask` | yes | How many recorded trials per task to consume. It may not exceed the shortest series in `panel.json`; the CLI refuses the bundle rather than repeat a recorded output (see [Asking for more trials than were captured](#asking-for-more-trials-than-were-captured)) |\n| `passK` | yes | The `k` in Pass^k. The reliability question is \"would `k` independent runs all pass?\" It may not exceed `trialsPerTask`, because the estimator needs at least `k` observed runs; the CLI refuses the bundle rather than leave the pairs unmeasured (see [Asking for a Pass^k the capture cannot answer](#asking-for-a-passk-the-capture-cannot-answer)) |\n| `currentModel` | yes | The incumbent the switch recommendation is measured against |\n| `qualityFloor` | yes | Minimum measured quality a model must clear to be the primary route in the exported policy |\n| `generatedAt` | yes | ISO timestamp recorded in the report. Supplied rather than read from the clock, so runs stay byte-identical |\n\n### Asking for more trials than were captured\n\n`trialsPerTask` names how many recorded trials to consume. It used to be allowed to exceed what\n`panel.json` holds: the stub gateway replayed the last recorded output for every extra trial and the\nharness counted each replay as a sample. On the shipped `examples/credit-memo` bundle, raising\n`trialsPerTask` from 4 to 8 turned 12 real judgements into `sample: 24` and lifted the budget\nmodel's worst-case Pass^3 from 0.25 to 0.625, at exit 0, with no warning, on a document that was\ncontent-addressed and signable.\n\nA replay is not evidence. It is the same recorded string handed back again, so it adds no\ninformation about anything, and Pass^k in particular reads agreement between copies of one output\nas consistency across runs. Both halves of the path now refuse it:\n\n- **The CLI refuses the bundle.** `runBundleAudit` throws before building any document, naming each\n  short `(model, task)` pair and the `trialsPerTask` value that would be honest. The process exits\n  `1` and writes nothing, so no partial report can be picked up without the reason it was rejected.\n  The missing trials are obtained by re-running the panel, never by repeating a recorded output.\n\n  ```\n  $ node dist/bin.js ./bundle-asking-for-8\n  audit refused: bundle asks for trialsPerTask 8 but the captured panel holds fewer for 3\n  (model, task) pairs: budget/memo-1 has 4, frontier/memo-1 has 4, thrifty/memo-1 has 4.\n  Repeating a recorded output would inflate the sample size and Pass^k. Set trialsPerTask to 4\n  or capture the missing trials.\n  ```\n\n- **The library counts only real trials and says what it dropped.** `runAudit` is usable with any\n  gateway, so it degrades instead of throwing: a response flagged `replayed` is excluded from the\n  judge's sample and from Pass^k, and `report.trialCoverage` records `realTrials`, `replayedTrials`\n  and a per-`(model, task)` shortfall list. `buildReportDocument` carries that into the document as\n  `trialCoverage` **only when the evidence fell short**, and `renderMarkdown` prints\n  `INCOMPLETE EVIDENCE` next to the sample size plus a `## Trial coverage (INCOMPLETE)` section. A\n  complete audit's document, and therefore its hash, is unchanged.\n\nA custom `Gateway` should set `replayed: true` on any response that is not a distinct observation.\nOmitting the field asserts that it is one.\n\n### Asking for a Pass^k the capture cannot answer\n\nThe companion mistake. `passK` is the `k` in \"would `k` independent runs all pass?\", and the\nunbiased estimator needs at least `k` observed runs, so a `(model, task)` holding fewer cannot be\nestimated at all. It used to be skipped in silence: no row, no note. On the shipped\n`examples/credit-memo` bundle, raising `passK` from 4 to 8 left the entire\n`## Run-to-run reliability exposure (Pass^k)` section empty, at exit 0, on a content-addressed\ndocument. The budget model's worst-case Pass^3 of 0.25, the one number in that report that rejects a\n10x-cheaper model, simply disappeared, and a heading with no rows under it reads as \"nothing to\nflag\".\n\nMissing evidence is not a passing result, so both halves refuse it the same way they refuse a\nreplayed trial:\n\n- **The CLI refuses the bundle**, naming each short pair and the `passK` that fits the capture,\n  exits `1`, and writes nothing.\n\n  ```\n  $ node dist/bin.js ./bundle-asking-for-passk-8\n  audit refused: bundle asks for Pass^8 but the captured panel holds fewer recorded runs for 3\n  (model, task) pairs: budget/memo-1 has 4, frontier/memo-1 has 4, thrifty/memo-1 has 4. Pass^k\n  over fewer than k runs is not a lower number, it is no measurement at all, and those pairs would\n  leave the reliability table with no row and no reason. Set passK to at most 4 or capture the\n  missing runs.\n  ```\n\n- **The library records what it could not measure.** `report.passKCoverage` gives `k`, the number of\n  pairs `measured`, and every `unmeasured` pair with its recorded run count.\n  `buildReportDocument` carries that into the document as `passKCoverage` **only when some pair was\n  unmeasured**, and `renderMarkdown` prints `NOT MEASURED` plus a line per pair under the\n  reliability heading. A fully measured audit's document, and therefore its hash, is unchanged.\n\n### As a library\n\nThe same audit through the API. Save this as `example.ts` in the repo root and run it with\n`node example.ts`. Node 24 strips the type annotations natively, and `./dist/index.js` exists\nbecause Install ended with `pnpm build`. `.gitignore` covers `example.ts` and `out/`, so this\nleaves your tree clean.\n\n```ts\nimport {\n  freezeCorpus, panelCorpus, groundTruthFrom, runAudit, StubGateway,\n  buildReportDocument, renderMarkdown, exportRouterPolicy,\n  type CorpusSpec, type Judge,\n} from \"./dist/index.js\";\n\nconst GOOD = \"Adequate coverage; approval reasonable with covenants.\";\nconst BAD = \"idk approve i guess\";\n\nconst corpus: CorpusSpec = {\n  rubric: { id: \"fs-credit-qa\", version: \"1\", criteria: [\"accurate\", \"policy-compliant\"] },\n  tasks: [{\n    taskId: \"memo-1\",\n    family: \"credit_memo\",\n    input: \"Summarize creditworthiness for applicant 4821\",\n    labels: [{ output: GOOD, accept: true }, { output: BAD, accept: false }],\n  }],\n};\n\nconst frozen = freezeCorpus(corpus);           // -> content-addressed\nconst gateway = new StubGateway({              // captured panel run, per trial\n  frontier: { costUsd: 0.03,  latencyMs: 1400, outputs: { \"memo-1\": [GOOD, GOOD, GOOD, GOOD] } },\n  budget:   { costUsd: 0.003, latencyMs: 400,  outputs: { \"memo-1\": [GOOD, GOOD, BAD,  GOOD] } },\n  thrifty:  { costUsd: 0.001, latencyMs: 300,  outputs: { \"memo-1\": [GOOD, GOOD, GOOD, GOOD] } },\n});\nconst judge: Judge = { judge: (_taskId, output) => ({ pass: output === GOOD, confidence: 0.9 }) };\n\nconst report = await runAudit({\n  corpus: panelCorpus(frozen),\n  models: [\"frontier\", \"budget\", \"thrifty\"],\n  gateway,\n  judge,\n  groundTruth: groundTruthFrom(frozen),\n  trialsPerTask: 4,\n  passK: 3,\n  currentModel: \"frontier\",\n});\n\nconst doc = buildReportDocument(report, {\n  client: \"Example Bank\",\n  corpusHash: frozen.contentHash,\n  panel: [\"frontier\", \"budget\", \"thrifty\"],\n  generatedAt: \"2026-06-30T00:00:00.000Z\",\n});\n\nconsole.log(\"ece:\", report.judgeReliability.ece);\nconsole.log(\"recommendation:\", JSON.stringify(report.families[0].recommendation));\nconsole.log(\"documentHash:\", doc.documentHash);\nconsole.log(\"policy:\", JSON.stringify(exportRouterPolicy(report, 0.9)));\nconsole.log(\"markdown lines:\", renderMarkdown(doc).split(\"\\n\").length);\n```\n\n```\n$ node example.ts\nece: 0.09999999999999976\nrecommendation: {\"fromId\":\"frontier\",\"toId\":\"thrifty\",\"savingsPct\":0.9666666666666667,\"latencyDeltaMs\":-1100,\"qualityDelta\":0}\ndocumentHash: sha256:91f91ffe93aca39e2b031feb3007e83104c08a8b78df8fdd77f63c068df8be42\npolicy: {\"policyVersion\":\"neutral-route/1\",\"qualityFloor\":0.9,\"routes\":[{\"family\":\"credit_memo\",\"primary\":\"thrifty\",\"fallbacks\":[]}]}\nmarkdown lines: 17\n```\n\nNote the `documentHash` is identical to the CLI's. The same frozen corpus plus the same frozen\npanel produces the same report, whichever entry point you use.\n\nTo make that recommendation *defensible* rather than merely observed, pass the paired per-task\noutcomes through `verifySwitchQuality` in `stats.ts` and feed the result into\n`buildReportDocument`'s optional third `evidence` argument.\n\n## What it does not do\n\n- It does not include an LLM judge. `Judge` is an interface; you supply the judge and the human\n  labels, Sigil scores them.\n- It does not call a model, hold a key, or open a network socket in the analysis path. The panel run\n  is captured upstream and supplied as data.\n- It does not route traffic, edit code, or mutate anything on your estate. Every artifact it\n  produces is derived facts only.\n- It is not a scheduler or a service. See [Status and roadmap](#status-and-roadmap).\n\n## How it works\n\nThe engine is a single pure pipeline over injected ports. It is designed to run inside your own\nnetwork boundary, with only derived numbers crossing out.\n\n```\n  your environment (your keys, your data)\n     |\n     |  Gateway adapter (OpenAI-compatible: LiteLLM proxy / OpenRouter)\n     |  outputs + cost + latency\n     v\n  +------------------------- Sigil -------------------------+\n  |                                                          |\n  |  freeze corpus  ->  run panel  ->  calibrated judge       |\n  |  (content-addressed)  (gateway)   (ECE / Brier / table)   |\n  |        |                               |                  |\n  |        |                               +-> risk certificates (Clopper-Pearson, LTT)\n  |        |                               +-> drift monitors (e-process / e-detector)\n  |        v                                                  |\n  |   Pass^k reliability  ->  Pareto frontier  ->  McNemar gate|\n  |        |                        |                    |    |\n  |        +------------------------+--------------------+    |\n  |                       |                     |             |\n  |              report document          router policy       |\n  |            (+ Ed25519 signed bundle)   (neutral export)    |\n  |                       |                     |             |\n  |                  [ egress guard: fail-closed ]             |\n  +------------------------------------------------------------+\n     |\n     v  scores, metrics, frontier, report, policy: nothing raw\n```\n\n### Boundaries enforced in code, not prose\n\n- **No key.** The production gateway takes a key *accessor* invoked per request; the key is never\n  stored on the adapter, never logged, and is scrubbed from every error the module can throw.\n- **No fabricated cost.** If the endpoint reports no cost (OpenRouter `usage.cost` or LiteLLM's\n  `x-litellm-response-cost` header), the adapter throws instead of defaulting to `0`, because a\n  zero would silently flatter every candidate's position on the frontier.\n- **No write.** Nothing in the codebase accepts a write or checkout credential; the governance\n  overlay reports least-privilege gaps and never mutates anything.\n- **Egress allowlist:** scores, calibration metrics, frontier, report, router policy. **Denylist:**\n  raw outputs, prompts, keys, corpus content beyond hashes.\n\n### Deployment modes\n\n1. **In your own environment** (the default): engine and gateway adapter both run inside your\n   boundary.\n2. **Hosted**, where the data sensitivity allows it.\n3. **Air-gapped signed-bundle exchange:** corpus and panel results exchanged as Ed25519-signed\n   bundles, verified fully offline (`bundle.ts`).\n\n### Directory map\n\nSingle package, no workspaces. All source is in `src/`, one module per pipeline stage.\n\n| Module | What it does |\n|---|---|\n| `corpus.ts` | Freezes and content-addresses the task set, rubric, and human labels; derives ground truth (unlabeled outputs are conservatively not-accepted) |\n| `gateway.ts` | The `Gateway` port plus `StubGateway`, a fixture map that can vary output per trial to exercise run-to-run variance |\n| `gateway-openai-compat.ts` | Adapter for any OpenAI-compatible chat-completions endpoint, behind a host allowlist checked before every call |\n| `metrics.ts` | ECE, Brier, reliability table: the judge's own calibration |\n| `reliability.ts` | Unbiased combinatorial Pass^k estimator (probability a random k-subset of observed runs all pass) |\n| `frontier.ts` | Pareto frontier over quality x cost x latency; cheapest candidate at equal-or-better measured quality |\n| `conformal.ts` | Exact Clopper-Pearson bounds; certified abstention threshold via fixed-sequence Learn-Then-Test |\n| `stats.ts` | Exact McNemar gate on paired outcomes; certified Pass^k lower bound |\n| `drift.ts` | E-process (Ville), changepoint e-detector (ARL), CUSUM baseline over bounded error streams |\n| `aci.ts` | Adaptive conformal intervals (Gibbs-Candes) for forecast bands on short series; abstains when history is too short |\n| `rank-score.ts` | Kendall tau-b + cardinal interval width (ordinal vs cardinal judge reliability) |\n| `ensemble.ts` | Multi-judge majority vote (ties resolve to FAIL) with inter-judge disagreement disclosed as a first-class signal |\n| `governance.ts` | Read-only agent to task to scope least-privilege gap map |\n| `report.ts` | Content-addressed report document + deterministic markdown render; certificate blocks are optional and additive |\n| `router-policy.ts` | Portable, vendor-neutral routing policy export (cheapest model clearing the quality floor, frontier as fallbacks) |\n| `bundle.ts` | Ed25519-signed report bundle + fail-closed offline verification |\n| `egress.ts` | Fail-closed guard: no raw output, prompt, or credential-shaped string may leave |\n| `canonical.ts` | Canonical JSON + SHA-256 content hashing, the reproducibility primitive |\n| `harness.ts` | `runAudit`: pure orchestration over the injected ports |\n| `cli.ts`, `bin.ts` | Offline runner over a captured results bundle |\n| `index.ts` | Public surface |\n\n`test/` mirrors `src/` one-to-one, plus `golden-fs.test.ts` (a full end-to-end deterministic\nregression on a finance-shaped fixture) and `certificates.property.test.ts` (property tests over\nthe certificate math via `fast-check`). `examples/credit-memo/` is the synthetic input bundle used\nby the quickstart. `fixtures/calibration-contract.golden.json` pins the calibration math against a\nsibling implementation; see [Status and roadmap](#status-and-roadmap).\n\n### Documented failure modes\n\n- Incomplete labels: calibration runs over the labeled subset only, and the report discloses the\n  sample size.\n- Nothing clears the quality floor: the router policy falls back to the highest-quality candidate,\n  never silently dropping a family, and that route carries a `note` saying the primary does not\n  clear the floor. Without it, a policy entry pairing a `qualityFloor` with a `primary` reads as the\n  claim that the primary met it.\n- Judge poorly calibrated (high ECE): surfaced before any savings claim, never hidden.\n- No abstention threshold certifies the target risk: the certificate says so explicitly (\"abstain\n  or collect more labels\"). A wide bound from a small sample is a finding, never rounded away.\n- Candidate significantly worse on paired tasks: the switch recommendation is reported **not\n  defensible** regardless of the cost delta.\n- Fewer recorded trials than `trialsPerTask` asks for: the CLI refuses the bundle and writes\n  nothing; through the library, replayed responses are excluded from the sample and from Pass^k and\n  the shortfall is stated in the report. The sample count is never padded to the requested number.\n- Fewer recorded runs than `passK` asks for: Pass^k is not computable for those pairs, so the CLI\n  refuses the bundle and writes nothing; through the library the pairs are listed in\n  `report.passKCoverage.unmeasured` and printed under `NOT MEASURED` in the report. A pair missing\n  from the reliability table is never left to read as a clean one.\n\n## Status and roadmap\n\nStatus values are exactly three: **Working**, **Partial**, **Planned**. Everything marked Planned is\na real gap today, described in enough detail to pick up.\n\n### Working today\n\n| Component | Notes |\n|---|---|\n| Calibration metrics (ECE / Brier / reliability) | Covered by tests and the quickstart |\n| Risk certificates (Clopper-Pearson, Learn-Then-Test) | Property-tested in `test/certificates.property.test.ts` |\n| Pass^k reliability + certified floor | Quickstart shows it rejecting a 10x-cheaper model |\n| Pareto frontier + McNemar switch gate | Quickstart shows both |\n| Report, router policy, governance overlay | Four artifacts written by the quickstart |\n| Signed bundle + offline verification | Signer and verifier are injected ports; the repo ships no key |\n| Egress guard | Golden test plants a PII leak and asserts it cannot escape |\n| Drift monitors (e-process, e-detector, CUSUM) | Implemented and unit-tested; external cross-validation is a roadmap item below |\n| Adaptive conformal intervals (`aci.ts`) | Abstains when the history is too short |\n| CLI over a captured bundle | `node dist/bin.js <bundle-dir> [out-dir]` |\n\n### Partial\n\n**Live gateway adapter (`src/gateway-openai-compat.ts`).** Fully implemented and tested against an\ninjected `fetchImpl`, including the host allowlist, key scrubbing, and the throw-on-missing-cost\nrule. It has never been run against a live LiteLLM or OpenRouter endpoint from this repo, so the\nwire-level details (header names, error shapes, cost field placement per provider) are unverified\nagainst a real server. **Good first contribution:** run it against your own endpoint, and send a\nrecorded-response fixture test for whatever needed fixing. The seam is `OpenAiCompatGatewayOptions`\nin that module.\n\n### Planned\n\n**Continuous / trend mode.** `drift.ts` and `aci.ts` are the machinery, but there is no scheduler,\npersistence layer, or service around them, so today drift monitoring is a library you drive\nyourself. Monitor state is deliberately plain serializable data: `EProcessState` and `EDetectorState`\nin `src/drift.ts` (and `AciState` in `src/aci.ts`) are values in and values out, with\n`initEProcess` / `updateEProcess` and `initEDetector` / `updateEDetector` as the whole interface.\nWhat is missing is the layer above: persist state between runs, replay an observation log,\nand emit a trend report across a series of audits rather than a single point-in-time one.\n\n**External cross-validation of `drift.ts`.** The e-process and e-detector guarantees are argued in\nthe module docs and property-tested internally, but never checked against an external reference\nimplementation. The concrete task: generate golden fixtures from CRAN's `stcpR6` on the same\nobservation streams and assert agreement to within tolerance, the same way\n`fixtures/calibration-contract.golden.json` pins the calibration math. This is the single most\nvaluable contribution anyone could make to this repo, because it converts an internal argument into\nexternal corroboration.\n\n**Live upstream calibration import.** `metrics.ts` deliberately *mirrors* the canonical ECE/Brier\nmath rather than importing it, pinned by `fixtures/calibration-contract.golden.json` (generated from\n`@apatureai/verdict-eval` in `apatureai/verdict`). If the contract test and the upstream ever disagree,\none side changed the math unilaterally, which is the failure the contract exists to catch. The\nfixture is a frozen manual copy today, not a live check; regenerating it on a cadence, or in CI,\nis open.\n\n**Anchor-set drift attribution (system vs judge).** When drift fires, it does not tell you whether\nthe system got worse or the judge did. The design sketch is a held-out anchor set with a\nrotate-with-overlap refresh policy, sitting on top of `drift.ts`. Nothing is implemented.\n\n**npm publication.** Nothing is on npm yet, so `npm i @apatureai/sigil` does not work today. The\npackaging is done and verified: the manifest is no longer `private`, it declares `bin`, `files`,\n`exports`, `engines` and a `prepublishOnly` build, and the packed tarball has been installed from\ndisk and its `sigil` command run end to end against the shipped example bundle. A tag-triggered\nrelease workflow (`.github/workflows/release.yml`) now publishes with provenance on any `v*` tag;\nwhat is left is the maintainer adding the `NPM_TOKEN` secret (see `CONTRIBUTING.md`) and pushing the\nfirst tag. The unscoped name `sigil` was taken (the\nregistry holds a tombstoned entry from a 2013 package unpublished in October 2024), so the package\nname is the scoped `@apatureai/sigil`.\n\n### Out of scope on purpose\n\n- **An LLM judge.** `Judge` is an interface. Sigil scores whatever judge you hand it, and the same\n  is true of ground truth: human labels, supplied by you. Bundling a judge would make the harness\n  non-neutral, which is the one thing it must not be.\n\n### Stated preconditions\n\nThese are properties of the statistics, not bugs, and each is stated in the output as well as here:\n\n- The abstention certificate assumes exchangeability between the calibration and deployment draws.\n  Drift breaks that assumption, which is why re-certification is meant to be a recurring cadence.\n- `certifiedPassKLowerBound` assumes i.i.d. runs.\n- The ACI guarantee is long-run average coverage; locally it can under-cover, and the implementation\n  clamps the adaptive level to [0.001, 0.999] to keep intervals finite, trading a corner of the\n  asymptotic argument for bounded artifacts.\n- Small samples give wide bounds, and that is the intended behavior. A wide interval is reported as\n  a finding, and when nothing certifies at the requested level the certificate says so explicitly\n  instead of returning a number.\n- The regulatory framing in some module docs (SR 11-7, SR 26-2) explains *why* the code is shaped\n  the way it is. It is not legal advice and not a compliance claim.\n\n## Development\n\n```\npnpm install --frozen-lockfile\npnpm typecheck   # tsc --noEmit\npnpm test        # vitest run  -> Test Files 23 passed (23), Tests 192 passed (192)\npnpm lint        # eslint .\npnpm build       # tsc -p tsconfig.build.json -> dist/\n```\n\nThose five commands, in that order, are exactly what CI runs (`.github/workflows/ci.yml`). All five\npass on a clean checkout, verified 2026-08-24 on Node 24.14.0 with pnpm 10.34.3: 23 test files, 192\ntests, ~2s.\n\nRun a single test file:\n\n```\npnpm vitest run test/golden-fs.test.ts\n```\n\nOne rule matters if you change the code: **Sigil is offline and deterministic by construction, and\nthe tests enforce it.** No test may call a real model, key, or network. The live adapter\n(`src/gateway-openai-compat.ts`) is exercised only through an injected `fetchImpl`. No wall clock\nand no RNG in the analysis path; `test/golden-fs.test.ts` and `test/canonical.test.ts` will catch\nyou. And do not widen what crosses the egress line.\n\n[CONTRIBUTING.md](CONTRIBUTING.md) has the full setup, conventions, and review process.\n\n## Prior work\n\nThe statistical machinery follows published lines rather than inventing any: split conformal\nprediction and Learn-Then-Test risk control (Angelopoulos, Bates et al.); selective classification\nwith a reject option (Geifman and El-Yaniv); betting supermartingales and e-detectors (Waudby-Smith\nand Ramdas; Shin, Ramdas and Rinaldo, arXiv:2203.03532); adaptive conformal inference under\ndistribution shift (Gibbs and Candes, NeurIPS 2021); Page's CUSUM; the exact McNemar test;\nClopper-Pearson intervals. The contribution here is putting them behind one deterministic,\nreproducible, fail-closed audit contract in TypeScript, not the estimators themselves.\n\n## Contributing\n\nContributions are welcome, and the roadmap above is the shortlist. Issues and pull requests are\nread. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, conventions, and how review works.\n\n## Security\n\nNo credentials, keys, or secrets are stored in this repository, and the analysis path opens no\nsockets. To report a vulnerability, use GitHub's private vulnerability reporting on this repository.\n[SECURITY.md](SECURITY.md) has the policy and the threat boundary.\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n","readmeFilename":"README.md","_rev":"1-b27f9c566aea8681270f6b9f5cb1fdea"}