{"_id":"@dagurupriyan/ctxd","name":"@dagurupriyan/ctxd","dist-tags":{"next":"0.1.0-beta.0","latest":"0.1.0-beta.0"},"versions":{"0.1.0-beta.0":{"name":"@dagurupriyan/ctxd","version":"0.1.0-beta.0","description":"Vendor-neutral context contract, semantic compiler, release validation, and MCP runtime for reliable data agents.","license":"MIT","type":"module","keywords":["mcp","metabase","agents","context","semantic-layer","sql","snapshot"],"bin":{"ctxd":"dist/cli.js"},"engines":{"node":">=20"},"main":"dist/index.js","exports":{".":"./dist/index.js","./cli":"./dist/cli.js"},"repository":{"type":"git","url":"git+https://github.com/thisis-gp/ctxd.git"},"bugs":{"url":"https://github.com/thisis-gp/ctxd/issues"},"homepage":"https://github.com/thisis-gp/ctxd#readme","publishConfig":{"access":"public"},"scripts":{"build":"tsc -p tsconfig.json","dev":"tsc -w -p tsconfig.json","start":"node dist/cli.js","serve":"node dist/cli.js serve","typecheck":"tsc --noEmit -p tsconfig.json","prepack":"npm run build","pretest":"npm run build","test":"node --test","test:integration":"node scripts/itest.mjs","test:eval":"yarn build && node scripts/generic-eval-check.mjs","test:semantic":"yarn build && node scripts/semantic-check.mjs","test:semantic:db":"yarn build && node scripts/semantic-check.mjs --db","benchmark:compare":"node dist/cli.js benchmark compare examples/benchmark.json examples/benchmark-direct-observations.json examples/benchmark-context-observations.json","release:gate":"node dist/cli.js release-gate","clean":"rimraf dist"},"dependencies":{"@modelcontextprotocol/sdk":"^1.4.1","better-sqlite3":"^11.3.0","commander":"^12.1.0","dotenv":"^16.4.5","node-sql-parser":"^5.3.3","zod":"^3.23.8"},"devDependencies":{"@types/better-sqlite3":"^7.6.11","@types/node":"^20.14.0","gpt-tokenizer":"^3.4.0","rimraf":"^6.0.1","ts-node":"^10.9.2","typescript":"^5.5.4"},"gitHead":"424b09821a5a9873cc1246a1d62579955bd895d5","types":"./dist/index.d.ts","_id":"@dagurupriyan/ctxd@0.1.0-beta.0","_nodeVersion":"24.13.1","_npmVersion":"11.8.0","dist":{"integrity":"sha512-IV8pxy1fpJyPCJUmRL9UO70D4xVCpiotsFZMJGG0IyA6MMG0Cpxv1ACLUm+518nW1O27k9CZzolYMPaVTrwHDw==","shasum":"590e576bb928080f81c01f5c9e5ff5ea0c8ac0c1","tarball":"https://registry.npmjs.org/@dagurupriyan/ctxd/-/ctxd-0.1.0-beta.0.tgz","fileCount":122,"unpackedSize":567085,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIDegYaUbUAUpNrDg98/6PYeSJaTFKSU2veMEuuPxJtxkAiBGesFD1sUCKyUtM9jg1ZdkbvnM046iTutSqlD8nkfu3w=="}]},"_npmUser":{"name":"dagurupriyan","email":"am400718@gmail.com"},"directories":{},"maintainers":[{"name":"dagurupriyan","email":"am400718@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/ctxd_0.1.0-beta.0_1785814645768_0.7496106634823023"},"_hasShrinkwrap":false}},"time":{"created":"2026-08-04T03:37:25.616Z","0.1.0-beta.0":"2026-08-04T03:37:25.899Z","modified":"2026-08-04T03:37:26.082Z"},"maintainers":[{"name":"dagurupriyan","email":"am400718@gmail.com"}],"description":"Vendor-neutral context contract, semantic compiler, release validation, and MCP runtime for reliable data agents.","homepage":"https://github.com/thisis-gp/ctxd#readme","keywords":["mcp","metabase","agents","context","semantic-layer","sql","snapshot"],"repository":{"type":"git","url":"git+https://github.com/thisis-gp/ctxd.git"},"bugs":{"url":"https://github.com/thisis-gp/ctxd/issues"},"license":"MIT","readme":"# ctxd\r\n\r\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\r\n[![Node.js](https://img.shields.io/badge/node-%3E%3D20-brightgreen)](package.json)\r\n\r\n**Stop your AI agent hallucinating against your production database.**\r\n\r\nAn agent that writes SQL from raw schema has no way to know which of your three\r\n`csat` tables is canonical, that `rating = 0` means the survey was never answered,\r\nor that `*_history` holds superseded rows. So it guesses — and answers\r\nconfidently, with no error and no warning.\r\n\r\nctxd puts a reviewed, release-versioned context layer between the agent and your\r\ndata: compact metadata, approved metric definitions, join paths, read-only\r\nenforcement, and PII masking.\r\n\r\n### The problem\r\n\r\n![An agent guessing against raw schema](docs/01-without-ctxd.gif)\r\n\r\nAsked for CSAT on settled claims, the agent inspects the schema, finds three\r\nplausible tables, picks one, and reports **2.70 out of 5**. It looks exactly like\r\na finished answer.\r\n\r\n### The same question, with ctxd\r\n\r\n![A real Claude Code session using ctxd](docs/03-real-claude-session.gif)\r\n\r\nAnswered from the reviewed metric `csat.settled_claims`: **4.67 out of 5** across\r\n900 answered surveys, citing the snapshot release it read.\r\n\r\n> A real Claude Code session over ctxd's MCP server — the tool calls, arguments,\r\n> returned values and final answer are verbatim, re-typeset for legibility rather\r\n> than screen-captured. The database is synthetic; all three takes returned 4.67.\r\n\r\nctxd deliberately does not replace Metabase, dbt, Cube, or a warehouse. It is the\r\ncontract, compiler, release gate, and evaluation layer between those systems and\r\nan agent.\r\n\r\n## Install\r\n\r\n**From npm** (CLI + library; requires a C++ toolchain for `better-sqlite3`):\r\n\r\n```bash\r\nnpm install -g @dagurupriyan/ctxd@next\r\nctxd init\r\n```\r\n\r\nThe package lives under the `@dagurupriyan` scope; the command it installs is plain\r\n`ctxd`. It is currently a prerelease on the `next` tag — drop `@next` once a\r\n`latest` release is published.\r\n\r\n**From source** (recommended for development and contributions):\r\n\r\n```bash\r\ngit clone https://github.com/thisis-gp/ctxd.git\r\ncd ctxd\r\nnpm install   # or: corepack enable && yarn install\r\nnpm test\r\nnpx ctxd init\r\n```\r\n\r\nRequires **Node 20 or newer**. `better-sqlite3` is a native module, so if you\r\nswitch Node versions after installing you will see\r\n`NODE_MODULE_VERSION ... does not match`. Rebuild it rather than reinstalling\r\neverything:\r\n\r\n```bash\r\nnpm rebuild better-sqlite3\r\n```\r\n\r\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for the full dev workflow. Report security\r\nissues privately — see [SECURITY.md](SECURITY.md).\r\n\r\n```\r\nAgent question\r\n    -> context search / plan / compile\r\n    -> trusted SQL (validated, version-stamped)\r\n    -> user runs SQL in Metabase (their own access)\r\n```\r\n\r\nBy default ctxd does **not** execute production queries. It is the context and SQL\r\ncompiler for agents; Metabase remains the query surface.\r\n\r\nIt talks to Metabase **via the Metabase API for metadata ingest** (and optionally\r\nfor tech-bot query execution). No direct database credentials. Nightly `refresh`\r\nuses a server-side API key; end users never see it.\r\n\r\n## Safety\r\n\r\nThe differentiator is not that an agent can reach your data — it's what it cannot do.\r\n\r\n- **Read-only by default.** Out of the box ctxd hands back context and *drafted*\r\n  SQL; nothing executes. Users run queries in Metabase under their own access.\r\n  `CTXD_ALLOW_QUERY=true` is opt-in, for trusted bots only.\r\n- **Fails closed.** With execution enabled, SQL must start with `SELECT`/`WITH`,\r\n  parse as a single statement, and reference only tables and columns present in\r\n  the active snapshot. Mutations, DDL, stacked statements and `SELECT ... INTO`\r\n  are rejected.\r\n- **PII stays masked.** Denylisted tables and columns remain indexed, so an agent\r\n  knows they exist, but carry no descriptions or sample values and cannot appear\r\n  in validated SQL. Wildcards like `*.email` cover every table at once.\r\n- **Every execution is auditable**, appended to `audit/queries.jsonl`.\r\n- **Snapshots hold metadata and query definitions only** — never production rows.\r\n\r\nFull details in [Safety reference](#safety-reference) below.\r\n\r\n## Who configures what (plug-and-play)\r\n\r\n| Role | What they do | What they never need |\r\n|------|--------------|----------------------|\r\n| Tech team | Set Metabase URL/API key on the server, run `ctxd refresh`, run `ctxd serve --http` | Per-user Metabase logins for agents |\r\n| Claude / Codex users | Add MCP URL + **their personal** token; get trusted SQL for Metabase | Metabase API keys, shared org token, running queries through ctxd |\r\n\r\n**Default product mode is context-only:** ctxd helps the LLM find tables/joins/metrics and draft/validate SQL. Users (or Metabase) run the query. Query execution tools stay off unless `CTXD_ALLOW_QUERY=true` (tech bots only).\r\n\r\n### Org quickstart (hosted + HTTPS)\r\n\r\n```bash\r\n# On the server (tech once)\r\nctxd init\r\n# edit .env: METABASE_URL, METABASE_API_KEY, CTXD_ADMIN_TOKEN, CTXD_DOMAIN, ACME_EMAIL\r\n\r\nctxd refresh --prune-keep 14   # must run on the SAME host/volume as serve\r\ndocker compose up -d --build       # Caddy HTTPS → ctxd\r\n```\r\n\r\n**Ops note:** Nightly refresh must run where the MCP server reads snapshots\r\n(e.g. `docker compose run --rm refresh` on that host). A GitHub Action that only\r\nuploads an artifact does **not** update the live server — that was the “ops gap.”\r\n\r\n```cron\r\n15 2 * * * cd /opt/ctxd && docker compose run --rm refresh >> /var/log/ctxd-refresh.log 2>&1\r\n```\r\n\r\nOpen **https://$CTXD_DOMAIN/admin**, sign in with `CTXD_ADMIN_TOKEN`, and create\r\n**one token per user**. Users paste into Claude/Codex:\r\n\r\n```json\r\n{\r\n  \"mcpServers\": {\r\n    \"ctxd\": {\r\n      \"url\": \"https://ctxd.example.com/mcp\",\r\n      \"headers\": {\r\n        \"Authorization\": \"Bearer ctxd_USER_SPECIFIC_TOKEN\"\r\n      }\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nHealth: `GET /health`. MCP: `POST /mcp` (per-user Bearer). Admin: `/admin`.\r\nFor production, put `/admin` behind your VPN, SSO proxy, or an IP allowlist even\r\nthough the dashboard also requires `CTXD_ADMIN_TOKEN`.\r\n\r\n### Local developer (stdio)\r\n\r\n```bash\r\nctxd serve\r\n```\r\n\r\nRegister with a local command entry as in `examples/mcp-config.json` (`ctxd-local`).\r\n\r\n## How it works\r\n\r\nA snapshot build queries Metabase once and produces an immutable, per-release\r\nsnapshot directory:\r\n\r\n```\r\nsnapshots/v8.19.0-0/\r\n├── manifest.json          # release, git commit, fingerprint, counts, status\r\n├── entities.jsonl         # databases / schemas / tables / columns (inspectable)\r\n├── relationships.jsonl    # foreign-key edges\r\n├── metabase-assets.jsonl  # questions / models / dashboards + query definitions\r\n├── semantic-definitions.json # versioned metric definitions + SQL templates\r\n├── search.sqlite          # FTS5 full-text index for sub-second retrieval\r\n└── changes.json           # diff vs the previous snapshot\r\n```\r\n\r\nThe MCP server then serves focused retrieval from the promoted `current` snapshot\r\ninstead of dumping the whole model into the conversation.\r\n\r\n### Architecture\r\n\r\n| Layer | Module | Responsibility |\r\n|-------|--------|----------------|\r\n| Adapter | `src/metabase/` | Only place that speaks the Metabase HTTP API |\r\n| Metadata indexer | `src/indexer/metadata-indexer.ts` | Normalize DB structure into entities + FK relationships |\r\n| Content indexer | `src/indexer/content-indexer.ts` | Normalize questions / models / dashboards |\r\n| Snapshot | `src/snapshot/` | Deterministic fingerprint, JSONL + SQLite, manifest, diff |\r\n| Release | `src/release/manager.ts` | `current` pointer lifecycle: validate → publish → promote → rollback |\r\n| Retrieval | `src/context-service.ts` | Search, entity context, freshness, SQL validation, read-only exec |\r\n| Hybrid ranker | `src/ranker.ts` | Rerank BM25 candidates with semantic ownership, synonyms, join distance, and safety penalties |\r\n| Intent planner | `src/intent.ts` | Classify questions, score plan confidence, and emit semantic-query fast paths |\r\n| MCP | `src/mcp/server.ts`, `src/mcp/http.ts` | Fourteen `context_*` tools over stdio or remote HTTP |\r\n| Adapters | `src/adapters/` | Import dbt, Cube, and MetricFlow JSON metadata into the same normalized model |\r\n| Contract draft | `src/contract-draft.ts` | Generate a reviewable starting contract from a snapshot or adapter export |\r\n| Evaluation | `src/benchmark.ts` | Compare direct-schema agent runs with context-layer runs on accuracy, tokens, calls, and time |\r\n\r\nThe adapter boundary is strict: nothing above `src/metabase/` sees a raw Metabase\r\nresponse — everything consumes the normalized model in `src/model.ts`. Swapping\r\nthe data source later means rewriting only the adapter.\r\n\r\n## Setup\r\n\r\n```bash\r\nyarn install\r\nyarn build\r\nnode dist/cli.js init      # scaffolds .env from .env.example\r\n# edit .env: set METABASE_URL and METABASE_API_KEY\r\nnode dist/cli.js doctor    # checks env, Metabase, snapshot, semantics, and HTTP config\r\n```\r\n\r\nThe checked-in semantic file is a generic example. By default, `refresh` skips\r\nsemantic definitions that do not match the connected database and still builds a\r\nusable metadata/search snapshot. Set `CTXD_STRICT_SEMANTICS=true` once your own\r\nsemantic definitions should fail the release when they drift from the database.\r\n\r\n## Building a snapshot\r\n\r\n```bash\r\n# Build, validate, publish, and promote a release snapshot.\r\nnode dist/cli.js snapshot build    --release v8.19.0-0 --git-commit \"$(git rev-parse HEAD)\"\r\nnode dist/cli.js snapshot validate --release v8.19.0-0\r\nnode dist/cli.js snapshot publish  --release v8.19.0-0\r\nnode dist/cli.js snapshot promote  --release v8.19.0-0   # advances `current`\r\n```\r\n\r\nOr run the whole pipeline (build → validate → publish → deploy/health → promote):\r\n\r\n```bash\r\nDEPLOY_SCRIPT=./scripts/deploy.sh HEALTHCHECK_SCRIPT=./scripts/healthcheck.sh scripts/release.sh v8.19.0-0 v8.18.0-0\r\n```\r\n\r\nRollback restores the previous pointer:\r\n\r\n```bash\r\nnode dist/cli.js snapshot rollback\r\n```\r\n\r\n## Querying from the terminal\r\n\r\n```bash\r\nnode dist/cli.js search \"active user subscriptions\"\r\nnode dist/cli.js search \"invoices\" --scope tables\r\nnode dist/cli.js join-path public.orders public.customers      # shortest FK join path\r\nnode dist/cli.js diff                                          # schema changes vs previous release\r\nnode dist/cli.js drift                                         # is the current snapshot stale vs live Metabase?\r\nnode dist/cli.js freshness\r\nnode dist/cli.js stats            # token/latency savings dashboard from recorded MCP usage\r\nnode dist/cli.js doctor           # readiness check without printing secrets\r\nnode dist/cli.js query \"SELECT COUNT(*) AS ticket_count FROM public.tickets;\"\r\n```\r\n\r\n## Importing other metadata tools\r\n\r\nThe core model is not Metabase-specific. Adapter inputs can be inspected or used\r\nto draft a contract:\r\n\r\n```bash\r\nnode dist/cli.js adapters list\r\nnode dist/cli.js adapters inspect dbt examples/dbt-manifest-mini.json\r\nnode dist/cli.js contract draft --adapter dbt --input examples/dbt-manifest-mini.json --project demo\r\n```\r\n\r\nSupported JSON importers:\r\n\r\n| Adapter | Input |\r\n|---------|-------|\r\n| `dbt` | `manifest.json` models/sources + columns |\r\n| `cube` | Cube schema JSON with cubes, dimensions, and measures |\r\n| `metricflow` | MetricFlow semantic manifest JSON |\r\n\r\nThese importers create the same `NormalizedModel` that snapshots use, so new\r\nconnectors do not need a separate agent-facing contract.\r\n\r\n## Serving to an agent\r\n\r\n```bash\r\nctxd serve                 # local MCP stdio (developer laptop)\r\nctxd serve --http          # shared org MCP at http://HOST:8787/mcp\r\n```\r\n\r\nRegister it with Claude Code / Codex (see `examples/mcp-config.json`).\r\nFor `--http`, users only need the URL and their **personal** Bearer token (issued at `/admin`) — not Metabase credentials.\r\n\r\n### MCP tools\r\n\r\n| Tool | Purpose |\r\n|------|---------|\r\n| `context_search` | Top-N tables, columns, and saved questions for a query |\r\n| `context_plan_query` | Resolve a question into candidate tables, columns, saved questions, and ambiguity warnings |\r\n| `context_compile_semantic_query` | Compile measures/dimensions/filters into SQL (no execution) |\r\n| `context_compile_contract_query` | Compile reviewed-contract SQL (no execution) |\r\n| `context_get_entity` | One table's columns, relationships, and referencing questions |\r\n| `context_get_relationships` | FK edges touching a table |\r\n| `context_get_join_path` | Shortest FK join path between two tables |\r\n| `context_find_saved_questions` | Reusable Metabase questions/models before writing SQL |\r\n| `context_get_changes` | Schema-change diff for a release |\r\n| `context_get_release_context` | Inspect a specific release snapshot |\r\n| `context_get_freshness` | Snapshot version, fingerprint, deployed-match status |\r\n| `context_validate_sql` | Check a statement is read-only (no execution) |\r\n| `context_run_*` | **Hidden by default.** Execute via Metabase only when `CTXD_ALLOW_QUERY=true` |\r\n\r\nEvery response embeds the source snapshot + release version. Planner responses\r\nalso include ranked table/column candidates with score reasons so agents can see\r\nwhy a candidate was selected instead of blindly trusting BM25 order.\r\nWhen a reviewed semantic metric answers the question directly, planner responses\r\nalso include intent, confidence, confidence reasons, and a `suggestedSemanticQuery`\r\npayload that can be compiled without scanning database metadata.\r\n\r\n### Context contract\r\n\r\n`context.contract.json` is the vendor-neutral project contract. It records entity\r\ngrain, measures, dimensions, approved joins, fanout risk, and query policies.\r\nThe contract is reviewed like code and can be validated without credentials:\r\n\r\n```bash\r\nnode dist/cli.js validate-contract context.contract.json\r\nnode dist/cli.js benchmark validate examples/benchmark.json examples/benchmark-observations.json\r\nnode dist/cli.js benchmark compare examples/benchmark.json examples/benchmark-direct-observations.json examples/benchmark-context-observations.json\r\nnode dist/cli.js snapshot validate --release v1.0.0-0 --contract context.contract.json\r\n```\r\n\r\nCross-entity compilation uses only approved contract joins. Inferred foreign keys\r\nremain useful for discovery, but they cannot silently become semantic truth.\r\nHigh-fanout and many-to-many paths fail closed until explicitly modeled.\r\n\r\nThe benchmark format is intentionally model-neutral. Record each agent's\r\nselected measures, dimensions, entities, calls, latency, and answer correctness,\r\nthen compare them against the same contract. This makes releases measurable on\r\ncorrectness, wrong joins, round trips, latency, and cost instead of relying on\r\ntoken estimates alone.\r\n\r\n### Release gate\r\n\r\nUse one command in CI to block a release when snapshot validation, contract\r\nvalidation, or benchmark validation fails:\r\n\r\n```bash\r\nnode dist/cli.js release-gate \\\r\n  --release v1.0.0-0 \\\r\n  --contract context.contract.json \\\r\n  --benchmark examples/benchmark.json \\\r\n  --observations examples/benchmark-observations.json\r\n```\r\n\r\nThe example GitHub Actions workflow in `.github/workflows/release.yml` runs this\r\nafter snapshot build and before publish/promote.\r\n\r\n### Demo\r\n\r\nOpen `docs/demo.html` for a static overview of the workflow and copyable\r\ncommands. It does not require credentials or a dev server.\r\n\r\n### Declarative semantic queries\r\n\r\nAgents should prefer semantic queries when a registered metric exists. They declare\r\nthe measure and optional dimensions/filters; the layer owns the table, canonical\r\nformula, default filters, row cap, and read-only execution.\r\n\r\n```json\r\n{\r\n  \"measures\": [\"tickets.csat_responses\"],\r\n  \"dimensions\": [\"status\"],\r\n  \"limit\": 100\r\n}\r\n```\r\n\r\nUse `context_compile_semantic_query` to get generated SQL, validate with\r\n`context_validate_sql`, then run it in Metabase. Cross-table measures are rejected\r\nuntil a reviewed join graph is registered. (`context_run_semantic_query` exists only\r\nwhen `CTXD_ALLOW_QUERY=true`.)\r\n\r\nThe sample support-ticket semantics include reviewed same-table dimensions such\r\nas `status`, `priority`, and `created_month`. This lets questions such as\r\n\"CSAT responses by ticket status\" compile without asking the agent to inspect\r\n`information_schema`.\r\n\r\n### Eval Checks\r\n\r\nRun the generic planner golden set:\r\n\r\n```bash\r\nyarn test:eval\r\n```\r\n\r\nTo compile every semantic measure/dimension combination:\r\n\r\n```bash\r\nyarn test:semantic\r\n```\r\n\r\nTo execute the compiled semantic SQL against your own database, configure `psql`\r\nthrough standard Postgres environment variables or set `CTXD_SEMANTIC_CHECK_PSQL`:\r\n\r\n```bash\r\nyarn test:semantic:db\r\n```\r\n\r\n## Safety reference\r\n\r\n- API keys come only from env / secret manager, are never logged, and are never\r\n  written into a snapshot.\r\n- **Default: no query execution through ctxd.** Agents get context + drafted SQL;\r\n  users run queries in Metabase under their own access. Set `CTXD_ALLOW_QUERY=true`\r\n  only for trusted tech bots.\r\n- When execution is enabled, SQL must start with `SELECT`/`WITH`, parse as a\r\n  single SELECT (fail closed), reject `SELECT ... INTO`, enforce row/time limits,\r\n  and validate table/column references against the active snapshot.\r\n- Sensitive tables/fields can be denylisted (`DENYLIST` env) — they stay indexed\r\n  but carry no descriptions and cannot be referenced in validated/executed SQL.\r\n  Supported entries include `schema.table`, `table`, `table.column`,\r\n  `schema.table.column`, `*.column`, and `schema.*.column`.\r\n- Reviewed semantic and contract definition files may contain raw SQL expressions;\r\n  treat those files like code.\r\n- Snapshots store metadata and query *definitions* only, never production rows.\r\n- When execution is enabled, every executed query is appended to `audit/queries.jsonl`.\r\n\r\n## Status\r\n\r\nImplements the Metabase adapter, indexers, SQLite snapshot + FTS search, MCP\r\nserver, semantic + contract compilation, release versioning, release gate,\r\nbenchmark harness, and multi-source metadata adapters (dbt/Cube/MetricFlow).\r\nIncremental snapshot refresh (Phase 5 in the BRD) is not yet implemented.\r\n","readmeFilename":"README.md","_rev":"1-d2089f67c7e07e37f9581d535c56c411"}