{"_id":"@adhd/sox-tokenguard-core","_rev":"2-761662a773ec1c97f8e56ded73cdf0ee","name":"@adhd/sox-tokenguard-core","dist-tags":{"latest":"0.2.1"},"versions":{"0.2.0":{"name":"@adhd/sox-tokenguard-core","version":"0.2.0","license":"MIT","_id":"@adhd/sox-tokenguard-core@0.2.0","maintainers":[{"name":"pseudosky","email":"skywinston.sk@gmail.com"}],"dist":{"shasum":"36cd206d6146af2fd73471820983fbf7add322f2","tarball":"https://registry.npmjs.org/@adhd/sox-tokenguard-core/-/sox-tokenguard-core-0.2.0.tgz","fileCount":27,"integrity":"sha512-80R+yHwwLU/gBqiJOMfZdoNBHr8k74TPGmYWTcDB2CkFH3bI/HgvpC0Fc+xD9vcc2K9uS4dWXkCvmPXxL4XDtg==","signatures":[{"sig":"MEUCIHQYHlVYhefvtiq95440qC0mHyqr/uSsmXnzvZG1lpIaAiEAgKwJ9/vyYEejUDCDGRTgtIK30sCWUDQfN+2wGDZGoow=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":85928},"main":"./dist/index.js","_from":"file:adhd-sox-tokenguard-core-0.2.0.tgz","types":"./dist/index.d.ts","engines":{"node":">=20"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.js"}},"_npmUser":{"name":"pseudosky","email":"skywinston.sk@gmail.com"},"_resolved":"/private/var/folders/yg/cfczgtx54bzfh74lx2_mv0z80000gp/T/38df24430d9b51f3c023f19a8bcb880a/adhd-sox-tokenguard-core-0.2.0.tgz","_integrity":"sha512-80R+yHwwLU/gBqiJOMfZdoNBHr8k74TPGmYWTcDB2CkFH3bI/HgvpC0Fc+xD9vcc2K9uS4dWXkCvmPXxL4XDtg==","_npmVersion":"11.6.2","description":"Pure, IO-light TypeScript port of the TokenGuard bijective tokenize/detokenize engine","directories":{},"_nodeVersion":"24.11.1","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/sox-tokenguard-core_0.2.0_1782451185497_0.7826446313036131","host":"s3://npm-registry-packages-npm-production"}},"0.2.1":{"_id":"@adhd/sox-tokenguard-core@0.2.1","bugs":{"url":"https://github.com/PseudoSky/adhd/issues"},"dist":{"shasum":"5bb4b7d3b6e80cf3af4b3b0ad065342bacba79c7","tarball":"https://registry.npmjs.org/@adhd/sox-tokenguard-core/-/sox-tokenguard-core-0.2.1.tgz","fileCount":28,"integrity":"sha512-rRM++3Llv6WKpbf73p9PFks+kgdmoOH7ZWZNtgNpCv3a2W5Fd2Tbm9Bp0b9dsTpZjlcFSsLfkstejeaHQLH60A==","signatures":[{"sig":"MEQCIAiTzylCaayT/KqzTnKniSr2aVcai9mNJLq0tk7rt9rYAiB2Tm/2FJQd2LECv5BOCmMfXhXixT49Bgz9jteVcMn7yQ==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"},{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIGvhEaNRVSTD0LjQZ6+VUvFfnOXp/Jsu1x181qFojIghAiEA9yooor8t9kucwru8dTYvRv888OTIAVECCV3J48BZfwI="}],"unpackedSize":93871},"main":"./dist/index.js","name":"@adhd/sox-tokenguard-core","_from":"file:adhd-sox-tokenguard-core-0.2.1.tgz","types":"./dist/index.d.ts","engines":{"node":">=20"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.js"}},"license":"MIT","version":"0.2.1","_npmUser":{"name":"pseudosky","email":"skywinston.sk@gmail.com"},"homepage":"https://github.com/PseudoSky/adhd","keywords":["token","security","tokenize","typescript"],"_resolved":"/private/var/folders/yg/cfczgtx54bzfh74lx2_mv0z80000gp/T/a98cc32d4bddc084e2b229ef7ae2432e/adhd-sox-tokenguard-core-0.2.1.tgz","_integrity":"sha512-rRM++3Llv6WKpbf73p9PFks+kgdmoOH7ZWZNtgNpCv3a2W5Fd2Tbm9Bp0b9dsTpZjlcFSsLfkstejeaHQLH60A==","repository":{"url":"git+https://github.com/PseudoSky/adhd.git","type":"git"},"_npmVersion":"11.6.2","description":"Pure, IO-light TypeScript port of the TokenGuard bijective tokenize/detokenize engine","directories":{},"maintainers":[{"name":"pseudosky","email":"skywinston.sk@gmail.com"}],"_nodeVersion":"24.11.1","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/sox-tokenguard-core_0.2.1_1788566542553_0.32259354319504885"}}},"time":{"created":"2026-06-26T05:19:45.343Z","modified":"2026-09-05T00:02:22.874Z","0.2.0":"2026-06-26T05:19:45.638Z","0.2.1":"2026-09-05T00:02:22.695Z"},"license":"MIT","description":"Pure, IO-light TypeScript port of the TokenGuard bijective tokenize/detokenize engine","maintainers":[{"name":"pseudosky","email":"skywinston.sk@gmail.com"}],"readme":"# @adhd/sox-tokenguard-core\n\nBijective (1:1, reversible) pseudonymization for text and LLM request/response bodies — pure\nTypeScript, synchronous, zero network calls, zero provider-specific logic. It detects sensitive\nidentifiers (hostnames, FQDNs, IPv4/IPv6, MAC addresses, emails, phone numbers), replaces each with\na stable pseudo-token like `<HOST_1>` or `<EMAIL_2>`, and reverses the substitution later — the\nsame real value always maps to the same token, and the same token always maps back to the same\nreal value, including across process restarts if you give it a persist path.\n\n```bash\npnpm add @adhd/sox-tokenguard-core\n```\n\n## Quick start\n\n```typescript\nimport { Mapper, tokenizeStr, detokenizeText } from '@adhd/sox-tokenguard-core';\n\n// persistPath is optional — pass one to make tokens reload-stable across restarts\nconst mapper = new Mapper('/tmp/tokens.json');\n\nconst text = 'Contact admin@prod.internal about 10.0.0.5';\nconst tokenized = tokenizeStr(text, mapper, null);\nconsole.log(tokenized);\n// \"Contact <EMAIL_1> about <IP_1>\"\n\nconst restored = detokenizeText(tokenized, mapper, null);\nconsole.log(restored);\n// \"Contact admin@prod.internal about 10.0.0.5\" — exact round-trip\n```\n\n## API reference\n\n### `Mapper` — the bijective store\n\n```typescript\nclass Mapper {\n  constructor(persistPath?: string);\n\n  // Insert / read\n  getOrCreate(real: string, type: string, source: Source): string;\n  registerExplicit(real: string, type: string, token: string, source: Source): string;\n  seed(items: ReadonlyArray<{ real?: string; type?: string; token?: string }>, source?: Source): void;\n\n  // Access\n  entries(): MapEntry[];\n  tokenOf(real: string): string | undefined;\n  realOf(token: string): string | undefined;\n  typeFor(real: string): string;\n  realsLongestFirst(): Array<[string, string]>;\n  tokensLongestFirst(): Array<[string, string]>;\n\n  // Persistence\n  load(filePath: string): void;\n  serialize(): TokenMap;\n\n  // Per-instance do-not-tokenize set (lower-cased)\n  readonly never: Set<string>;\n}\n```\n\n`getOrCreate` is idempotent: calling it again with the same `real` returns the existing token\nunchanged, and the original `source`/`type` are never overwritten by a later call. Passing a\n`persistPath` to the constructor makes every mutation durable — reopen the same path later and\nthe same reals still map to the same tokens, with per-type token counters continuing from where\nthey left off (IDs are never reassigned).\n\n### Types\n\n```typescript\ntype Source = 'seed' | 'proxy' | 'tooling' | 'custom';\ntype IdType = 'host' | 'fqdn' | 'ip' | 'ip6' | 'mac' | 'email' | 'phone' | 'id' | string;\n\ninterface MapEntry {\n  token: string;\n  real: string;\n  type: IdType;\n  source: Source;\n  created_ts: string;\n}\n\ninterface TokenMap {\n  version: 2;\n  entries: MapEntry[];\n}\n\ninterface DetectorConfig {\n  detectPhone?: boolean;\n  detectIpv6?: boolean;\n}\n```\n\n### Detectors — pattern recognition\n\nEach detector runs a single regex family over a string, tokenizing every match it finds via the\nsupplied `Mapper`, and returns the resulting string. `hits` is an optional counter map (pass `null`\nif you don't need it) that detectors increment per match — useful for auditing how many\nsubstitutions a call made.\n\n```typescript\ntype HitMap = Map<string, number>;\n\nfunction detectEmail(s: string, mapper: Mapper, hits?: HitMap | null): string;\nfunction detectFqdn(s: string, mapper: Mapper, hits?: HitMap | null): string;\nfunction detectIpv4(s: string, mapper: Mapper, hits?: HitMap | null): string;\nfunction detectIpv6(s: string, mapper: Mapper, hits?: HitMap | null): string;\nfunction detectMac(s: string, mapper: Mapper, hits?: HitMap | null): string;\nfunction detectPhone(s: string, mapper: Mapper, hits?: HitMap | null): string;\nfunction detectKnown(s: string, mapper: Mapper, hits?: HitMap | null): string;\n\n// Runs the full ordered pipeline: known reals → email → fqdn → ipv6 → ipv4 → mac → phone.\n// dynamicIp (default true) gates the auto-detectors; known reals are always applied regardless.\nfunction tokenizeStr(\n  s: string,\n  mapper: Mapper,\n  hits?: HitMap | null,\n  dynamicIp?: boolean,\n  config?: DetectorConfig,\n): string;\n```\n\n```typescript\nimport { Mapper, detectEmail, detectFqdn } from '@adhd/sox-tokenguard-core';\n\nconst mapper = new Mapper();\ndetectEmail('Contact admin@target.internal', mapper, null);\n// \"Contact <EMAIL_1>\"\ndetectFqdn('Server is webapp.internal.corp', mapper, null);\n// \"Server is <HOST_1>\" — FQDNs are tokenized under the \"host\" type, not a separate \"FQDN_n\" series\n```\n\n`detectPhone` and `detectIpv6` run as part of the full pipeline by default — pass\n`{ detectPhone: false }` / `{ detectIpv6: false }` as the `config` argument to `tokenizeStr` to\nturn either one off (e.g. if plain integers or version-like strings in your text are getting\nmatched as phone numbers).\n\n### Request / response tokenization\n\n```typescript\nfunction walkTokenize(obj: unknown, mapper: Mapper, hits?: HitMap | null, dynamicIp?: boolean, config?: DetectorConfig): unknown;\n\nfunction tokenizeRequest(req: unknown, mapper: Mapper, hits?: HitMap | null, config?: DetectorConfig): unknown;\n\nfunction wireLeaks(req: unknown, mapper: Mapper): string[];\n\nfunction detokenizeText(text: string, mapper: Mapper, hits?: HitMap | null): string;\n```\n\n`tokenizeRequest` is shaped for an Anthropic Messages-style request body: it tokenizes `system`,\n`messages`, and other data-bearing fields in two passes (a full detector pass, then a known-reals-only\npass to catch anything the first pass's dynamic detection newly added), while leaving `tools` — a\nJSON-Schema block — completely verbatim, so the API still receives a valid schema.\n\n`wireLeaks` scans the same scoped regions (`system`/`messages`/`metadata`, never `tools`) for any\n`real` already known to the `Mapper` that still appears un-tokenized, and returns the list of\nleaked reals — an empty array means the tokenization pass caught everything mapped.\n\n```typescript\nimport { Mapper, tokenizeRequest, wireLeaks, detokenizeText } from '@adhd/sox-tokenguard-core';\n\nconst mapper = new Mapper();\nconst request = {\n  system: 'You are assessing prod.internal.',\n  messages: [{ role: 'user', content: 'Contact admin@prod.internal for access.' }],\n  metadata: {},\n  tools: [{ name: 'bash', description: 'run a shell command', input_schema: { type: 'object', properties: {} } }],\n};\n\nconst tokenized = tokenizeRequest(request, mapper, null) as typeof request;\nconsole.log(tokenized.system);\n// \"You are assessing <HOST_1>.\"\n\nconst leaks = wireLeaks(tokenized, mapper);\nconsole.log(leaks);\n// [] — nothing mapped survived in system/messages/metadata\n\nconst restoredSystem = detokenizeText(tokenized.system, mapper, null);\nconsole.log(restoredSystem);\n// \"You are assessing prod.internal.\"\n```\n\n### SSE stream detokenization\n\nA token like `<HOST_1>` can be split across two consecutive `content_block_delta` events in an\nAnthropic-style stream (e.g. `<HOS` in one delta, `T_1>` in the next), so flat replacement on the\nraw bytes misses it. This module reassembles each content block's full value across its deltas\nbefore detokenizing, then re-emits it as a single delta — everything else (including `thinking`\nblocks, which are signed and must never be mutated) passes through byte-identical.\n\n```typescript\nfunction detokenizeSse(raw: string, reverse: (s: string) => string): string;\nfunction detokenizeSseWithMapper(raw: string, mapper: Mapper, hits?: HitMap | null): string;\n```\n\n```typescript\nimport { Mapper, detokenizeSseWithMapper } from '@adhd/sox-tokenguard-core';\n\nconst mapper = new Mapper();\nmapper.getOrCreate('vulntarget.internal', 'host', 'seed'); // → <HOST_1>\n\n// The token is split across two content_block_delta events for the same block\n// index (\"<HOS\" then \"T_1>\"); reassembly + detokenization happens once the\n// matching content_block_stop event is seen.\nconst rawSseBody =\n  'event: content_block_delta\\n' +\n  'data: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Target is <HOS\"}}\\n\\n' +\n  'event: content_block_delta\\n' +\n  'data: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"T_1>\"}}\\n\\n' +\n  'event: content_block_stop\\n' +\n  'data: {\"type\":\"content_block_stop\",\"index\":0}\\n\\n';\n\nconst restored = detokenizeSseWithMapper(rawSseBody, mapper, null);\nconsole.log(restored);\n// event: content_block_delta\n// data: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Target is vulntarget.internal\"}}\n//\n// event: content_block_stop\n// data: {\"type\":\"content_block_stop\",\"index\":0}\n```\n\n`detokenizeSse` is the same operation with the reversal function supplied directly, decoupling the\nmodule from a concrete `Mapper` — pass `(s) => detokenizeText(s, mapper, null)` if you don't want\nto construct `detokenizeSseWithMapper`'s dependency directly.\n\n### Identifier group variants\n\n```typescript\nfunction identifierGroupVariants(label: string, members: string[]): string[];\n```\n\nGiven a label and a set of associated hostnames/names, derives the specific (length ≥ 4,\nnon-generic) DNS components worth treating as related identifiers — dropping generic components\nlike `www`, `api`, or bare TLDs:\n\n```typescript\nimport { identifierGroupVariants } from '@adhd/sox-tokenguard-core';\n\nidentifierGroupVariants('webapp', ['webapp-prod.internal.corp', 'www.internal.corp']);\n// [\"webapp\", \"webapp-prod\"]\n```\n\n## Seeding known identifiers\n\n`Mapper.seed` pre-registers a batch of reals (optionally with explicit tokens) before any\ndetection runs — useful for guaranteeing a fixed identifier always gets the same token from the\nfirst request, rather than whichever one happens to be seen first:\n\n```typescript\nconst mapper = new Mapper();\nmapper.seed([\n  { real: 'prod.internal', type: 'host' },\n  { real: 'alice@example.com', type: 'email' },\n]);\n\nmapper.tokenOf('prod.internal'); // \"<HOST_1>\"\n```\n\nAn explicit `token` in a seed item is only honored when `source: 'custom'`; otherwise the token is\nallocated the normal way. After explicit items are seeded, `seed` also derives and seeds\nidentifier-group variants (via `identifierGroupVariants`) from any label-typed entries.\n\n## Design\n\n- **No I/O in the hot path** — every function above is synchronous; the only I/O is the optional\n  `Mapper` persist file, written on each mutation and read via `load()`.\n- **Bijective** — every real has exactly one token; every token maps back to exactly one real. A\n  `Mapper` that never saw a given token cannot reverse it (there's no global registry).\n- **Reload-stable** — construct a `Mapper` with the same `persistPath` in a later process and the\n  same reals still resolve to the same tokens; token counters resume rather than restart.\n- **Type-tagged** — every entry records its `type` (`host`, `email`, `ip`, …) and `source`\n  (`seed`/`proxy`/`tooling`/`custom`), so a persisted map is self-describing for audit.\n- **Provider-agnostic core** — this package has no HTTP client, no API keys, and no\n  provider-specific parsing; `tokenizeRequest`'s field scoping matches the Anthropic Messages API\n  shape, but every function operates on plain strings and parsed JSON values you hand it.\n","readmeFilename":"README.md","homepage":"https://github.com/PseudoSky/adhd","keywords":["token","security","tokenize","typescript"],"repository":{"url":"git+https://github.com/PseudoSky/adhd.git","type":"git"},"bugs":{"url":"https://github.com/PseudoSky/adhd/issues"}}