{"_id":"@codecai/web","_rev":"6-db24366f09797d72e9680100ab0cf82f","name":"@codecai/web","dist-tags":{"latest":"0.5.0"},"versions":{"0.1.0":{"name":"@codecai/web","version":"0.1.0","keywords":["codec","tokenizer","detokenizer","llm","binary","msgpack","protobuf","streaming","ai"],"license":"MIT","_id":"@codecai/web@0.1.0","maintainers":[{"name":"wdunn001","email":"wdunn001@gmail.com"}],"homepage":"https://github.com/wdunn001/Codec#readme","bugs":{"url":"https://github.com/wdunn001/Codec/issues"},"dist":{"shasum":"84b44f2713c4f37e183546b5937ce29044111df9","tarball":"https://registry.npmjs.org/@codecai/web/-/web-0.1.0.tgz","fileCount":32,"integrity":"sha512-N+fmjA104W0GgvvW0S2V/pDFYNGcL1xFkxcUC9kDF/hiC4Psr2V2GDSherS1OHLaUwFTu6NcIpCrHzWaEp6qsg==","signatures":[{"sig":"MEUCIQCxDH3KxJ02+tXjjGYa/DOIst/f29RaLtDWdhhCgXO8MwIga2ENCXAGjfStFzOpb6XGWDVVPfKMBX6RR8pwB5xOPAk=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":88828},"main":"./dist/cjs/index.cjs","type":"module","types":"./dist/types/index.d.ts","module":"./dist/esm/index.js","engines":{"node":">=18"},"exports":{".":{"types":"./dist/types/index.d.ts","import":"./dist/esm/index.js","default":"./dist/esm/index.js","require":"./dist/cjs/index.cjs"},"./map":{"types":"./dist/types/map.d.ts","import":"./dist/esm/map.js","require":"./dist/cjs/map.cjs"},"./stream":{"types":"./dist/types/stream.d.ts","import":"./dist/esm/stream.js","require":"./dist/cjs/stream.cjs"},"./tokenize":{"types":"./dist/types/tokenize.d.ts","import":"./dist/esm/tokenize.js","require":"./dist/cjs/tokenize.cjs"},"./detokenize":{"types":"./dist/types/detokenize.d.ts","import":"./dist/esm/detokenize.js","require":"./dist/cjs/detokenize.cjs"}},"gitHead":"eb3dcc1e02a284ba0e8df69a924e0878fb41f2d2","scripts":{"test":"node --test --import tsx test/*.test.ts","build":"npm run build:esm && npm run build:cjs && npm run build:types","clean":"node -e \"require('fs').rmSync('dist',{recursive:true,force:true})\"","build:cjs":"tsc -p tsconfig.cjs.json && node -e \"require('fs').readdirSync('dist/cjs').filter(f=>f.endsWith('.js')).forEach(f=>require('fs').renameSync('dist/cjs/'+f,'dist/cjs/'+f.replace(/\\.js$/,'.cjs')))\"","build:esm":"tsc -p tsconfig.esm.json","build:types":"tsc -p tsconfig.types.json"},"_npmUser":{"name":"wdunn001","email":"wdunn001@gmail.com"},"repository":{"url":"git+https://github.com/wdunn001/Codec.git","type":"git","directory":"packages/web"},"_npmVersion":"11.8.0","description":"Isomorphic edge tokenizer + lazy detokenizer for the Codec binary transport protocol. Works in browsers, Node 18+, and edge runtimes.","directories":{},"sideEffects":false,"_nodeVersion":"25.5.0","dependencies":{"@msgpack/msgpack":"^3.0.0"},"_hasShrinkwrap":false,"devDependencies":{"tsx":"^4.19.2","typescript":"^5.7.3"},"_npmOperationalInternal":{"tmp":"tmp/web_0.1.0_1778049330041_0.47879242695834656","host":"s3://npm-registry-packages-npm-production"}},"0.2.0":{"name":"@codecai/web","version":"0.2.0","keywords":["codec","tokenizer","detokenizer","llm","binary","msgpack","protobuf","streaming","ai"],"license":"MIT","_id":"@codecai/web@0.2.0","maintainers":[{"name":"wdunn001","email":"wdunn001@gmail.com"}],"homepage":"https://github.com/wdunn001/Codec#readme","bugs":{"url":"https://github.com/wdunn001/Codec/issues"},"dist":{"shasum":"ab71224fb081b8eb4ab0d817b2564cfee7a9f6cd","tarball":"https://registry.npmjs.org/@codecai/web/-/web-0.2.0.tgz","fileCount":42,"integrity":"sha512-Jf5Y2tCHUDwufWpA+f2AqUuYhcicsWFTvyNcflwr95OhNME+/kFh306dWU7gZxm1fZgn0orFt9+1qxvCWNvgKw==","signatures":[{"sig":"MEUCIQDXacZgUMzZW+zq4aCIe4XoMLqDE+mzeLnp0txsWDHndwIgNO6i7POmrHCjIeJyblQUEFnhhAX3H4F8VjZQP5tHqqY=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":155896},"main":"./dist/cjs/index.cjs","type":"module","types":"./dist/types/index.d.ts","module":"./dist/esm/index.js","engines":{"node":">=18"},"exports":{".":{"types":"./dist/types/index.d.ts","import":"./dist/esm/index.js","default":"./dist/esm/index.js","require":"./dist/cjs/index.cjs"},"./map":{"types":"./dist/types/map.d.ts","import":"./dist/esm/map.js","require":"./dist/cjs/map.cjs"},"./stream":{"types":"./dist/types/stream.d.ts","import":"./dist/esm/stream.js","require":"./dist/cjs/stream.cjs"},"./tokenize":{"types":"./dist/types/tokenize.d.ts","import":"./dist/esm/tokenize.js","require":"./dist/cjs/tokenize.cjs"},"./detokenize":{"types":"./dist/types/detokenize.d.ts","import":"./dist/esm/detokenize.js","require":"./dist/cjs/detokenize.cjs"}},"gitHead":"eb3dcc1e02a284ba0e8df69a924e0878fb41f2d2","scripts":{"test":"node --test --import tsx test/*.test.ts","build":"npm run build:esm && npm run build:cjs && npm run build:types","clean":"node -e \"require('fs').rmSync('dist',{recursive:true,force:true})\"","build:cjs":"tsc -p tsconfig.cjs.json && node -e \"require('fs').readdirSync('dist/cjs').filter(f=>f.endsWith('.js')).forEach(f=>require('fs').renameSync('dist/cjs/'+f,'dist/cjs/'+f.replace(/\\.js$/,'.cjs')))\"","build:esm":"tsc -p tsconfig.esm.json","build:types":"tsc -p tsconfig.types.json"},"_npmUser":{"name":"wdunn001","email":"wdunn001@gmail.com"},"repository":{"url":"git+https://github.com/wdunn001/Codec.git","type":"git","directory":"packages/web"},"_npmVersion":"11.8.0","description":"Isomorphic edge tokenizer + lazy detokenizer for the Codec binary transport protocol. Works in browsers, Node 18+, and edge runtimes.","directories":{},"sideEffects":false,"_nodeVersion":"25.5.0","dependencies":{"@msgpack/msgpack":"^3.0.0"},"_hasShrinkwrap":false,"devDependencies":{"tsx":"^4.19.2","typescript":"^5.7.3"},"_npmOperationalInternal":{"tmp":"tmp/web_0.2.0_1778052172180_0.5487690266836349","host":"s3://npm-registry-packages-npm-production"}},"0.3.0":{"name":"@codecai/web","version":"0.3.0","keywords":["codec","tokenizer","detokenizer","llm","binary","msgpack","protobuf","streaming","ai"],"license":"MIT","_id":"@codecai/web@0.3.0","maintainers":[{"name":"wdunn001","email":"wdunn001@gmail.com"}],"homepage":"https://github.com/wdunn001/Codec#readme","bugs":{"url":"https://github.com/wdunn001/Codec/issues"},"dist":{"shasum":"2d69b3622ff374b62c51fe12d6ee9c34cc04f60f","tarball":"https://registry.npmjs.org/@codecai/web/-/web-0.3.0.tgz","fileCount":47,"integrity":"sha512-DXL8oXPCxfRyhfMpcHqErimxMXa9o5RwnEN49Pl9ayEqhmYupD0v06x08iZu32Oiqx/D63CD9vsuNqlt0XywPQ==","signatures":[{"sig":"MEQCIBpIqgleuZn38ZUvOVSERKbIJOIX0XjUz914m6D1gYm/AiAh8x/JT2HnX9VMPNKdj3YsSqPE93ZdoZSXQPTMVZ7PaA==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":182484},"main":"./dist/cjs/index.cjs","type":"module","types":"./dist/types/index.d.ts","module":"./dist/esm/index.js","engines":{"node":">=18"},"exports":{".":{"types":"./dist/types/index.d.ts","import":"./dist/esm/index.js","default":"./dist/esm/index.js","require":"./dist/cjs/index.cjs"},"./map":{"types":"./dist/types/map.d.ts","import":"./dist/esm/map.js","require":"./dist/cjs/map.cjs"},"./stream":{"types":"./dist/types/stream.d.ts","import":"./dist/esm/stream.js","require":"./dist/cjs/stream.cjs"},"./tokenize":{"types":"./dist/types/tokenize.d.ts","import":"./dist/esm/tokenize.js","require":"./dist/cjs/tokenize.cjs"},"./detokenize":{"types":"./dist/types/detokenize.d.ts","import":"./dist/esm/detokenize.js","require":"./dist/cjs/detokenize.cjs"}},"gitHead":"429a31fa147a117acb237d55534d4b8842a4c898","scripts":{"test":"node --test --import tsx test/*.test.ts","build":"npm run build:esm && npm run build:cjs && npm run build:types","clean":"node -e \"require('fs').rmSync('dist',{recursive:true,force:true})\"","build:cjs":"tsc -p tsconfig.cjs.json && node -e \"require('fs').readdirSync('dist/cjs').filter(f=>f.endsWith('.js')).forEach(f=>require('fs').renameSync('dist/cjs/'+f,'dist/cjs/'+f.replace(/\\.js$/,'.cjs')))\"","build:esm":"tsc -p tsconfig.esm.json","build:types":"tsc -p tsconfig.types.json"},"_npmUser":{"name":"wdunn001","email":"wdunn001@gmail.com"},"repository":{"url":"git+https://github.com/wdunn001/Codec.git","type":"git","directory":"packages/web"},"_npmVersion":"11.8.0","description":"Isomorphic edge tokenizer + lazy detokenizer for the Codec binary transport protocol. Works in browsers, Node 18+, and edge runtimes.","directories":{},"sideEffects":false,"_nodeVersion":"25.5.0","dependencies":{"@msgpack/msgpack":"^3.0.0"},"_hasShrinkwrap":false,"devDependencies":{"tsx":"^4.19.2","typescript":"^5.7.3"},"_npmOperationalInternal":{"tmp":"tmp/web_0.3.0_1778071745443_0.6263923658464379","host":"s3://npm-registry-packages-npm-production"}},"0.4.0":{"name":"@codecai/web","version":"0.4.0","keywords":["codec","tokenizer","detokenizer","llm","binary","msgpack","protobuf","streaming","ai"],"license":"MIT","_id":"@codecai/web@0.4.0","maintainers":[{"name":"wdunn001","email":"wdunn001@gmail.com"}],"homepage":"https://github.com/wdunn001/Codec#readme","bugs":{"url":"https://github.com/wdunn001/Codec/issues"},"dist":{"shasum":"187a013ce594a83770b6455e8acc541ded02d29e","tarball":"https://registry.npmjs.org/@codecai/web/-/web-0.4.0.tgz","fileCount":62,"integrity":"sha512-UYKbnaRH7cX8WY8b6CkOWsE+Hlze501WY0nT+bpnw6aLlwQsKAFGxED39WyvNmll0BN8UJlJqbkv1Ot92oqt6g==","signatures":[{"sig":"MEYCIQD5Gc+s2bojoghM/fBw7wW6EgDQQxrtBFwefsx00NpxggIhAJyVuseOt/xrgqgo4tuvAGqJBT8/QtkUtFHMAIrDxgXY","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":273837},"main":"./dist/cjs/index.cjs","type":"module","types":"./dist/types/index.d.ts","module":"./dist/esm/index.js","engines":{"node":">=18"},"exports":{".":{"types":"./dist/types/index.d.ts","import":"./dist/esm/index.js","default":"./dist/esm/index.js","require":"./dist/cjs/index.cjs"},"./map":{"types":"./dist/types/map.d.ts","import":"./dist/esm/map.js","require":"./dist/cjs/map.cjs"},"./stream":{"types":"./dist/types/stream.d.ts","import":"./dist/esm/stream.js","require":"./dist/cjs/stream.cjs"},"./discover":{"types":"./dist/types/discover.d.ts","import":"./dist/esm/discover.js","require":"./dist/cjs/discover.cjs"},"./tokenize":{"types":"./dist/types/tokenize.d.ts","import":"./dist/esm/tokenize.js","require":"./dist/cjs/tokenize.cjs"},"./detokenize":{"types":"./dist/types/detokenize.d.ts","import":"./dist/esm/detokenize.js","require":"./dist/cjs/detokenize.cjs"}},"gitHead":"56f7721bb74fcf739d05732addaeee1eed8782ed","scripts":{"test":"node --test --import tsx test/*.test.ts","build":"npm run build:esm && npm run build:cjs && npm run build:types","clean":"node -e \"require('fs').rmSync('dist',{recursive:true,force:true})\"","build:cjs":"tsc -p tsconfig.cjs.json && node -e \"require('fs').readdirSync('dist/cjs').filter(f=>f.endsWith('.js')).forEach(f=>require('fs').renameSync('dist/cjs/'+f,'dist/cjs/'+f.replace(/\\.js$/,'.cjs')))\"","build:esm":"tsc -p tsconfig.esm.json","build:types":"tsc -p tsconfig.types.json"},"_npmUser":{"name":"wdunn001","email":"wdunn001@gmail.com"},"repository":{"url":"git+https://github.com/wdunn001/Codec.git","type":"git","directory":"packages/web"},"_npmVersion":"11.8.0","description":"Isomorphic edge tokenizer + lazy detokenizer for the Codec binary transport protocol. Works in browsers, Node 18+, and edge runtimes.","directories":{},"sideEffects":false,"_nodeVersion":"25.5.0","dependencies":{"@msgpack/msgpack":"^3.0.0"},"_hasShrinkwrap":false,"devDependencies":{"tsx":"^4.19.2","typescript":"^5.7.3"},"_npmOperationalInternal":{"tmp":"tmp/web_0.4.0_1778248268020_0.8089364234404972","host":"s3://npm-registry-packages-npm-production"}},"0.4.1":{"name":"@codecai/web","version":"0.4.1","keywords":["codec","tokenizer","detokenizer","llm","binary","msgpack","protobuf","streaming","ai"],"license":"MIT","_id":"@codecai/web@0.4.1","maintainers":[{"name":"wdunn001","email":"wdunn001@gmail.com"}],"homepage":"https://github.com/wdunn001/Codec#readme","bugs":{"url":"https://github.com/wdunn001/Codec/issues"},"dist":{"shasum":"e683fef736e831a2423e230cf739cd768370a543","tarball":"https://registry.npmjs.org/@codecai/web/-/web-0.4.1.tgz","fileCount":82,"integrity":"sha512-BUQBzVt4gUE1ekUxDeEdZS4e0FJDjCHe5EmYQ4LbOGMk6WCkcvGrhMw5kBwAMNAPRM0M9VI9MCcOAIs8XfoWXQ==","signatures":[{"sig":"MEUCIQDoMgrRkeLpu72hpAYJQXF9q3Ihk1eC6daqcbKSaVdFmQIgQktG/nznS9ARAPCpm6/Pao/gJIOkIBiVhDO8qaFzgNo=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":558093},"main":"./dist/cjs/index.cjs","type":"module","types":"./dist/types/index.d.ts","module":"./dist/esm/index.js","engines":{"node":">=18"},"exports":{".":{"types":"./dist/types/index.d.ts","import":"./dist/esm/index.js","default":"./dist/esm/index.js","require":"./dist/cjs/index.cjs"},"./map":{"types":"./dist/types/map.d.ts","import":"./dist/esm/map.js","require":"./dist/cjs/map.cjs"},"./stream":{"types":"./dist/types/stream.d.ts","import":"./dist/esm/stream.js","require":"./dist/cjs/stream.cjs"},"./discover":{"types":"./dist/types/discover.d.ts","import":"./dist/esm/discover.js","require":"./dist/cjs/discover.cjs"},"./tokenize":{"types":"./dist/types/tokenize.d.ts","import":"./dist/esm/tokenize.js","require":"./dist/cjs/tokenize.cjs"},"./detokenize":{"types":"./dist/types/detokenize.d.ts","import":"./dist/esm/detokenize.js","require":"./dist/cjs/detokenize.cjs"}},"gitHead":"53a71503f33fca91d353d7dfbeddfcf050c2d37a","scripts":{"test":"node --test --import tsx test/*.test.ts","build":"npm run build:esm && npm run build:cjs && npm run build:types","clean":"node -e \"require('fs').rmSync('dist',{recursive:true,force:true})\"","build:cjs":"tsc -p tsconfig.cjs.json && node -e \"require('fs').readdirSync('dist/cjs').filter(f=>f.endsWith('.js')).forEach(f=>require('fs').renameSync('dist/cjs/'+f,'dist/cjs/'+f.replace(/\\.js$/,'.cjs')))\"","build:esm":"tsc -p tsconfig.esm.json","build:types":"tsc -p tsconfig.types.json"},"_npmUser":{"name":"wdunn001","email":"wdunn001@gmail.com"},"repository":{"url":"git+https://github.com/wdunn001/Codec.git","type":"git","directory":"packages/web"},"_npmVersion":"10.8.2","description":"Isomorphic edge tokenizer + lazy detokenizer for the Codec binary transport protocol. Works in browsers, Node 18+, and edge runtimes.","directories":{},"sideEffects":false,"_nodeVersion":"20.20.2","dependencies":{"@msgpack/msgpack":"^3.0.0"},"_hasShrinkwrap":false,"devDependencies":{"tsx":"^4.19.2","typescript":"^5.7.3"},"_npmOperationalInternal":{"tmp":"tmp/web_0.4.1_1778954311942_0.04809185113088632","host":"s3://npm-registry-packages-npm-production"}},"0.5.0":{"name":"@codecai/web","version":"0.5.0","description":"Isomorphic edge tokenizer + lazy detokenizer for the Codec binary transport protocol. Works in browsers, Node 18+, and edge runtimes.","license":"MIT","type":"module","main":"./dist/cjs/index.cjs","module":"./dist/esm/index.js","types":"./dist/types/index.d.ts","exports":{".":{"types":"./dist/types/index.d.ts","import":"./dist/esm/index.js","require":"./dist/cjs/index.cjs","default":"./dist/esm/index.js"},"./detokenize":{"types":"./dist/types/detokenize.d.ts","import":"./dist/esm/detokenize.js","require":"./dist/cjs/detokenize.cjs"},"./tokenize":{"types":"./dist/types/tokenize.d.ts","import":"./dist/esm/tokenize.js","require":"./dist/cjs/tokenize.cjs"},"./map":{"types":"./dist/types/map.d.ts","import":"./dist/esm/map.js","require":"./dist/cjs/map.cjs"},"./discover":{"types":"./dist/types/discover.d.ts","import":"./dist/esm/discover.js","require":"./dist/cjs/discover.cjs"},"./stream":{"types":"./dist/types/stream.d.ts","import":"./dist/esm/stream.js","require":"./dist/cjs/stream.cjs"}},"sideEffects":false,"scripts":{"build":"npm run build:esm && npm run build:cjs && npm run build:types","build:esm":"tsc -p tsconfig.esm.json","build:cjs":"tsc -p tsconfig.cjs.json && node -e \"require('fs').readdirSync('dist/cjs').filter(f=>f.endsWith('.js')).forEach(f=>require('fs').renameSync('dist/cjs/'+f,'dist/cjs/'+f.replace(/\\.js$/,'.cjs')))\"","build:types":"tsc -p tsconfig.types.json","test":"node --test --import tsx test/*.test.ts","clean":"node -e \"require('fs').rmSync('dist',{recursive:true,force:true})\""},"dependencies":{"@msgpack/msgpack":"^3.0.0"},"devDependencies":{"tsx":"^4.19.2","typescript":"^5.7.3"},"engines":{"node":">=18"},"keywords":["codec","tokenizer","detokenizer","llm","binary","msgpack","protobuf","streaming","ai"],"repository":{"type":"git","url":"git+https://github.com/wdunn001/Codec.git","directory":"packages/web"},"bugs":{"url":"https://github.com/wdunn001/Codec/issues"},"homepage":"https://github.com/wdunn001/Codec#readme","_id":"@codecai/web@0.5.0","gitHead":"3acffb9c70bacc876e64ee42a63beab26cca5f26","_nodeVersion":"20.20.2","_npmVersion":"10.8.2","dist":{"integrity":"sha512-X1jc/Xcgi+6pUuCoKW5upMbT3qkVUkAYAgVFzMYKML+KVcLIgGsFDIR6/Pig1k3vmxrUnoTG/yoBS+feFpfPzQ==","shasum":"4cf13b75eb116574ec51bdcbecc241e1e1a9b531","tarball":"https://registry.npmjs.org/@codecai/web/-/web-0.5.0.tgz","fileCount":82,"unpackedSize":576385,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQCQ73l1vvSHah7Ycb6417GWEWhMeUPdjEr/d5RpQbKPrAIhAJrMh42cGoDiEQdx2VaqA/hkovbO1PW2MpROi6LNF4Nt"}]},"_npmUser":{"name":"wdunn001","email":"wdunn001@gmail.com"},"directories":{},"maintainers":[{"name":"wdunn001","email":"wdunn001@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/web_0.5.0_1779081177983_0.7319464739125274"},"_hasShrinkwrap":false}},"time":{"created":"2026-05-06T06:35:29.973Z","modified":"2026-05-18T05:12:58.244Z","0.1.0":"2026-05-06T06:35:30.190Z","0.2.0":"2026-05-06T07:22:52.329Z","0.3.0":"2026-05-06T12:49:05.588Z","0.4.0":"2026-05-08T13:51:08.162Z","0.4.1":"2026-05-16T17:58:32.094Z","0.5.0":"2026-05-18T05:12:58.124Z"},"bugs":{"url":"https://github.com/wdunn001/Codec/issues"},"license":"MIT","homepage":"https://github.com/wdunn001/Codec#readme","keywords":["codec","tokenizer","detokenizer","llm","binary","msgpack","protobuf","streaming","ai"],"repository":{"type":"git","url":"git+https://github.com/wdunn001/Codec.git","directory":"packages/web"},"description":"Isomorphic edge tokenizer + lazy detokenizer for the Codec binary transport protocol. Works in browsers, Node 18+, and edge runtimes.","maintainers":[{"name":"wdunn001","email":"wdunn001@gmail.com"}],"readme":"# @codecai/web\n\n**Isomorphic tokenizer + lazy detokenizer for the [Codec](https://github.com/wdunn001/Codec) binary transport protocol.**\n\nCodec ships token IDs over the wire instead of UTF-8 text. `@codecai/web` is the presentation layer:\n\n- **Detokenizer** (IDs → text) for rendering binary streams in the browser, lazily, only when a human needs to read them.\n- **BPETokenizer** (text → IDs) — pure-JS, exact, no wasm — so the browser can send token-ID prompts upstream and skip the JSON-text round trip entirely.\n- **Stream decoder** for both Codec wire modes (msgpack, protobuf) plus the future zstd/gzip-compressed variants (handled transparently by `fetch()`).\n- **`LatentStreamEncoder` / `LatentStreamDecoder`** (v0.3) for the latent modality — VAE latents on the wire across the seven [pipelines](https://github.com/wdunn001/Codec/blob/main/spec/PIPELINES.md) (`raw` / `int8` / `int4` / `int8-adaptive` / `int4-adaptive` / `delta+int8` / `delta+int4`). TypeScript twin of the Python reference encoder; works against [`codec-comfyui`](https://hub.docker.com/r/wdunn001/codec-comfyui) and [`codec-diffusers`](https://hub.docker.com/r/wdunn001/codec-diffusers).\n- **`tool_calling` block** on `TokenizerMap` — per-model tool-calling convention auto-derived from chat templates. Same field exposed in the Python / Rust / Java / .NET / C clients.\n- **`SafetyPolicyDescriptor` + `discoverSafetyPolicy`** (v0.4) — load the sanitized publishable safety-policy descriptor a server advertised via `safety_policy_id` + `safety_policy_hash` in `READY`. Fetches and verifies against `.well-known/codec/policies/<id>.json` or the content-addressed `.well-known/codec/policies/sha256/<hex>.json` sibling. Companion optional package [`@codecai/web-safety`](../web-safety) adds the client-side prefilter (secrets/PII/entropy) and a classifier registry (Prompt Guard 86M default, Llama Guard 3 1B opt-in).\n\nWorks in browsers, Node 18+, Cloudflare Workers, Deno, Bun. No Node-only imports. Only runtime dep is `@msgpack/msgpack` for stream decoding (~5 kB).\n\n## Why this exists\n\nReal numbers from `Codec/packages/bench`:\n\n| Configuration                              | B/token | vs JSON-SSE |\n|--------------------------------------------|--------:|------------:|\n| JSON-SSE (live Ollama qwen2.5)             |   186.4 |        1.0× |\n| Codec msgpack (identity)                   |    16.0 |        9.6× |\n| Codec protobuf (identity)                  |    10.9 |   **14.2×** |\n| Codec msgpack + `Content-Encoding: zstd`   |     3.4 |   **45.0×** |\n\nEnd-to-end agent round-trip (1024 tokens): **3.6× faster** with binary frames, because both the wire shrinks AND detokenize+tokenize gets eliminated.\n\n## Install\n\n```bash\nnpm install @codecai/web\n```\n\n## Quick start — decoding a stream\n\n```ts\nimport { loadMap, Detokenizer, decodeStream } from '@codecai/web';\n\n// 1. Load and pin the dialect map by hash. Cached forever after first fetch.\nconst map = await loadMap({\n  url:  'https://cdn.jsdelivr.net/gh/wdunn001/codec-maps/maps/qwen/qwen2.json',\n  hash: 'sha256:c73972f7a580…',\n});\n\n// 2. Stream from a Codec-compliant server (vLLM, SGLang).\nconst resp = await fetch('http://localhost:8000/v1/completions', {\n  method: 'POST',\n  headers: { 'Content-Type': 'application/json' },\n  body: JSON.stringify({\n    model: 'Qwen/Qwen2.5-7B-Instruct',\n    prompt: 'Explain entropy.',\n    stream_format: 'msgpack',     // ← the only required change\n    max_tokens: 256,\n  }),\n});\n\n// 3. Detokenize lazily — only when rendering for a human.\nconst detok = new Detokenizer(map);\nfor await (const frame of decodeStream(resp.body!, 'msgpack')) {\n  // frame.ids is the raw token output. Pass it forward unchanged for\n  // agent-to-agent. Render it for humans:\n  output.append(detok.render(frame.ids, { partial: !frame.done }));\n}\n```\n\n## Forwarding IDs to another model (agent-to-agent, same vocab)\n\nWhen the next consumer of this stream is another model on the same vocab — agent → agent, orchestrator → planner, model → tool that re-feeds the model — you do NOT need a `Detokenizer` at all. Forward `frame.ids` directly:\n\n```ts\n// No Detokenizer constructed: zero UTF-8 reassembly, zero BPE-merge work.\nfor await (const frame of decodeStream(resp.body!, 'msgpack')) {\n  forwardCodecFrame(nextAgent, frame.ids, frame.done);   // pass uint32[] straight on\n}\n```\n\nThis is the **hot-loop fast path** for agent mesh code. Skipping `detok.render(...)` saves ~10-20% client CPU on heavy reply streams (no string allocation, no partial-UTF-8 buffering, no metaspace decode). For cross-vocab handoff use `Translator` — that case still needs the byte-level path because the two vocabs disagree.\n\n## Quick start — encoding text (for the bidirectional path)\n\nWhen you want **zero text on the wire in either direction** — agent A's output IDs feeding straight into agent B's input — encode text to IDs in the browser before sending:\n\n```ts\nimport { BPETokenizer } from '@codecai/web';\n\nconst tok = new BPETokenizer(map);\nconst promptIds = tok.encode('Explain entropy.');   // pure-JS BPE, exact\n\nawait fetch('http://localhost:8000/v1/completions', {\n  method: 'POST',\n  headers: { 'Content-Type': 'application/json' },\n  body: JSON.stringify({\n    prompt: promptIds,            // OpenAI accepts int[] here\n    stream_format: 'msgpack',\n    max_tokens: 256,\n  }),\n});\n```\n\nFor **huge prompts** (>50K tokens, e.g. RAG with long context), the dedicated `/v1/completions/codec` endpoint accepts a binary msgpack request body too. Same wire output, smaller request:\n\n```ts\nimport { encode as msgpackEncode } from '@msgpack/msgpack';\n\nconst body = msgpackEncode({\n  prompt_ids: promptIds,\n  max_tokens: 256,\n  stream_format: 'msgpack',\n});\nawait fetch('http://localhost:8000/v1/completions/codec', {\n  method: 'POST',\n  headers: { 'Content-Type': 'application/x-msgpack' },\n  body,\n});\n```\n\n## Picking the right tokenizer\n\n`pickTokenizer(map)` returns the best implementation for the loaded map:\n\n```ts\nimport { pickTokenizer } from '@codecai/web';\n\nconst tok = pickTokenizer(map);  // → BPETokenizer if map has merges,\n                                  //   LongestMatchTokenizer otherwise\nconst ids = tok.encode(text);\n```\n\n| Class                    | When                                                    |\n|--------------------------|---------------------------------------------------------|\n| `BPETokenizer`           | Real model maps (have `merges` + `encoder`). Exact BPE. |\n| `LongestMatchTokenizer`  | Vocab-only / canonical-IR maps. Fallback.               |\n| `Detokenizer`            | Always. Decodes IDs → text via the map's encoder.       |\n\n`BPETokenizer` handles both byte-level (Llama-3, Qwen, Phi-3, DeepSeek-V3, Mistral-Nemo, Falcon, SmolLM2) and metaspace SentencePiece (Llama-2, Mistral-v3, Mixtral, Gemma, Codestral). Verified via round-trip tests against the real Qwen-2 152K-vocab tokenizer for ASCII, code, emoji, and CJK.\n\n## API\n\n| Export                      | Purpose                                                       |\n|-----------------------------|---------------------------------------------------------------|\n| `loadMap(opts)`             | Fetch + sha256-verify + cache a tokenizer map                 |\n| `discoverMap({ origin, id })` | Resolve a map via the `.well-known/codec/` convention       |\n| `discoverIndex({ origin })` | Fetch `.well-known/codec/index.json` (optional directory)     |\n| `discoverZstdDict({ origin, hash })` (v0.5) | Resolve a zstd dict at `.well-known/codec/dicts/<sha256-hex>.zstd`. Hash-pin-verified against the URL's path component; hard-fails on 404 / mismatch (no silent fallback). |\n| `MemoryMapCache`            | Default in-memory cache. Implement `MapCache` for IDB / KV    |\n| `validateMap(unknown)`      | Type-narrowing schema check                                   |\n| `Detokenizer`               | Stateful detokenizer: byte-level + metaspace + byte fallback + partial UTF-8 buffering |\n| `detokenize(map, ids)`      | One-shot helper for non-streaming use                         |\n| `BPETokenizer`              | Pure-JS BPE: byte-level and metaspace                         |\n| `LongestMatchTokenizer`     | Vocab-only longest-prefix-match (fallback for canonical-IR maps) |\n| `pickTokenizer(map)`        | Build the right tokenizer for the loaded map                  |\n| `tokenize(map, text)`       | One-shot helper                                               |\n| `decodeStream(body, fmt)`   | `ReadableStream<Uint8Array>` → `AsyncIterable<CodecFrame>`    |\n| `decodeMsgpackStream`       | msgpack-specific decoder                                      |\n| `decodeProtobufStream`      | protobuf-specific decoder                                     |\n| `ToolWatcher`               | Detect delimited regions (tool calls, reasoning blocks, vision spans) in a token-ID stream without decoding |\n| `Translator`                | Cross-vocab agent handoff: `ids_A → text → ids_B` with streaming-safe word-boundary buffering |\n\n## Detect tool calls without decoding\n\nMost chat-tuned models delimit tool calls with single-token specials (`<tool_call>` / `</tool_call>` for Qwen 2.5+, `<|python_tag|>` / `<|eom_id|>` for Llama 3.1+, `<think>` / `</think>` for DeepSeek-R1, etc.). Detecting *that* one happened is a uint32 compare — no detokenize, no string allocation.\n\n```ts\nimport { ToolWatcher } from '@codecai/web';\n\nconst watcher = new ToolWatcher(map, '<tool_call>', '</tool_call>');\n\nfor await (const frame of decodeStream(resp.body!)) {\n  for (const ev of watcher.feed(frame.ids)) {\n    if (ev.kind === 'passthrough') {\n      // Forward to the next agent verbatim. No decode.\n      forwardCodecFrame(nextAgent, ev.ids);\n    } else /* 'region' */ {\n      // Body of the tool call, markers excluded. Decode only when you\n      // actually need the JSON arguments.\n      const json = detok.render(ev.ids);\n      dispatchTool(JSON.parse(json));\n    }\n  }\n}\n```\n\nThe watcher is stateful: regions split between network frames buffer until the end marker arrives. `watcher.inside` reports whether one is currently in flight. The same primitive works for reasoning blocks, multimodal spans, code-interpreter regions — anything delimited by a (start, end) special pair.\n\n## Correctness notes\n\n- **Byte-level decode**: every vocab token is a sequence of GPT-2-encoded bytes. The Detokenizer reverses the byte→unicode table and accumulates bytes across tokens until they form a complete UTF-8 sequence. Tested against 3-byte (`€`) and 4-byte (`🚀`) sequences.\n- **Metaspace decode**: `▁` becomes space; SentencePiece byte-fallback IDs (`<0x00>`–`<0xFF>`) are decoded as raw bytes through the same UTF-8 buffer.\n- **Partial sequences across frames**: `Detokenizer` is stateful — call `render(ids, { partial: true })` while frames are streaming, then `render(ids, { partial: false })` (or omit `partial`) on the last frame so the buffer flushes. Use `reset()` between conversations.\n- **BPE merge ordering**: merges are applied greedily by priority, not left-to-right. Matches HuggingFace tokenizers reference behaviour. Test fixture verifies this explicitly.\n- **Hash verification** uses Web Crypto's `SubtleCrypto.digest('SHA-256', ...)` — available in every target runtime. A mismatch throws `TokenizerMapHashMismatchError`.\n\n## Map sources\n\n`loadMap` accepts any URL — the sha256 hash is what matters. For a curated set of pre-generated maps:\n\n```\nhttps://cdn.jsdelivr.net/gh/wdunn001/codec-maps/maps/<family>.json\n```\n\n14 families covering 70+ aliases — see [`codec-maps`](https://github.com/wdunn001/codec-maps) for the index.\n\nTo generate your own from a HuggingFace `tokenizer.json`:\n\n```bash\nnpx @codecai/maps-cli build my-org/my-model --id=my-org/my-model\nnpx @codecai/maps-cli hash my-org_my-model.json\n```\n\n### Self-hosted discovery via `.well-known/codec/`\n\nModel maintainers can publish their own maps at a known location on a domain they control, so clients only need to know the origin and the map ID — no out-of-band URL+hash exchange:\n\n```ts\nimport { discoverMap } from '@codecai/web';\n\nconst map = await discoverMap({\n  origin: 'https://qwen.io',\n  id:     'qwen/qwen2',\n});\n```\n\nThis fetches `https://qwen.io/.well-known/codec/maps/qwen/qwen2.json`. The document is either a tiny pointer (`{ id, url, hash }`) referencing the actual map on a CDN, or the full map served inline. Either way, hash verification still anchors the bytes. See [`spec/WELL_KNOWN_DISCOVERY.md`](https://github.com/wdunn001/Codec/blob/main/spec/WELL_KNOWN_DISCOVERY.md) for the convention, and [`@codecai/maps-cli`](https://www.npmjs.com/package/@codecai/maps-cli)'s `well-known` command to generate the publishing tree.\n\n## Compatibility\n\n| Runtime              | Status              |\n|----------------------|---------------------|\n| Browsers (modern)    | ✅ Chrome 123+ supports `Content-Encoding: zstd` natively |\n| Node.js 18+          | ✅                  |\n| Cloudflare Workers   | ✅                  |\n| Deno                 | ✅                  |\n| Bun                  | ✅                  |\n\n## License\n\nMIT. See [LICENSE](../../LICENSE) at the repo root.\n","readmeFilename":"README.md"}