{"_id":"bpe-lite","_rev":"10-c54377459b03a816a6d416f65cc4bc1e","name":"bpe-lite","dist-tags":{"latest":"0.5.2"},"versions":{"0.1.0":{"name":"bpe-lite","version":"0.1.0","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","_id":"bpe-lite@0.1.0","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"dist":{"shasum":"35fd8f8e886bbeb32a22588cc382216f1f459c5e","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.1.0.tgz","fileCount":9,"integrity":"sha512-A58Xbt39O8P4g94LCkeIIQouiLRldxoNeyD8DdIZc2X04dnAve/jz+3tHvmVtWazXp4prGquXjycuAT8x8asdA==","signatures":[{"sig":"MEQCIGv2WURoLSW/lF6E+CLnEl3XSt/FGO3HUi7OLgRE1KdaAiBnt7vICZS4Cji7jBhD00b2T2XuTXufF/MWVUPdvXsMuw==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":15308240},"main":"src/index.js","type":"commonjs","scripts":{"test":"node tests/run-tests.js","build":"node scripts/build-vocabs.js"},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"_npmVersion":"11.6.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","directories":{},"_nodeVersion":"24.13.0","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/bpe-lite_0.1.0_1773177971593_0.6497877476140561","host":"s3://npm-registry-packages-npm-production"}},"0.1.1":{"name":"bpe-lite","version":"0.1.1","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","_id":"bpe-lite@0.1.1","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"dist":{"shasum":"86e729459f899b921d685ec66942b40e43067146","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.1.1.tgz","fileCount":9,"integrity":"sha512-929jmbfJ2Qzbqb1IvNOpbnrQMYZBBJqjkE8GUvJYeEnDqO/Q1Ym8zt6El7nw/eFK12St970x44w4qURHRKVRFQ==","signatures":[{"sig":"MEUCIDBnS3Vj2rIzjcg53SykDLC4WxD5rTprltLPt67ODx9qAiEAyGrsbml3rS84IaFM7Ad8HNAsuGVOeKT8u8yqiTjIgKs=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":8514883},"main":"src/index.js","type":"commonjs","scripts":{"test":"node tests/run-tests.js","build":"node scripts/build-vocabs.js"},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"_npmVersion":"11.6.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","directories":{},"_nodeVersion":"24.13.0","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/bpe-lite_0.1.1_1773178176120_0.8270066131907197","host":"s3://npm-registry-packages-npm-production"}},"0.3.0":{"name":"bpe-lite","version":"0.3.0","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","_id":"bpe-lite@0.3.0","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"dist":{"shasum":"a72bdf317eee3c6b94bd4ce74153c43bcbe37cee","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.3.0.tgz","fileCount":12,"integrity":"sha512-eX5F1CkizMdrLmLqoZ+BiLIEYkXB6H6jqZQlE3lzI37l+tCChPc34UYY06cS1wOFuAP7+PG9RGkqsQJ/HnPpSQ==","signatures":[{"sig":"MEYCIQCMcsBgk5V+pjric7tWP68nk8YIzY0nn5kw5/FA71ik/wIhAPxENOEuZIOFiNeanyQNlFi7IlKA9aZNLhDC8F7GqTLZ","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":10247866},"main":"src/index.js","type":"commonjs","types":"index.d.ts","exports":{".":{"types":"./index.d.ts","import":"./src/index.mjs","require":"./src/index.js"}},"scripts":{"test":"node tests/run-tests.js","build":"node scripts/build-vocabs.js"},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"_npmVersion":"11.6.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","directories":{},"_nodeVersion":"24.13.0","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/bpe-lite_0.3.0_1773211952067_0.3100854347021724","host":"s3://npm-registry-packages-npm-production"}},"0.3.1":{"name":"bpe-lite","version":"0.3.1","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","_id":"bpe-lite@0.3.1","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"dist":{"shasum":"235e2d37fe5e90f2572d09fd490634a7e091cf57","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.3.1.tgz","fileCount":12,"integrity":"sha512-t72Vkxp/xMDRcUFzgtctIKGgIojXXXd3GqxLIV9i7qQ+A2ChvfiAoksdQj7fZPxLUHOR9rSAdF9o4Fa6Q3X0GA==","signatures":[{"sig":"MEUCIFLBMetXz30b/o3o1I0Nb60LDA9dp2ghFpdxqG02+vkAAiEAvSEBlQl3OG3TjCaSBEBr8TEjcOm6af6QSqnpGY5EL2I=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":10247883},"main":"src/index.js","type":"commonjs","types":"index.d.ts","exports":{".":{"types":"./index.d.ts","import":"./src/index.mjs","require":"./src/index.js"}},"scripts":{"test":"node tests/run-tests.js","build":"node scripts/build-vocabs.js"},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"_npmVersion":"11.6.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","directories":{},"_nodeVersion":"24.13.0","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/bpe-lite_0.3.1_1773212095140_0.7588210061040792","host":"s3://npm-registry-packages-npm-production"}},"0.4.0":{"name":"bpe-lite","version":"0.4.0","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","_id":"bpe-lite@0.4.0","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"dist":{"shasum":"78c21f257a6fb472224437408917ab7d5ec26f6d","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.4.0.tgz","fileCount":12,"integrity":"sha512-FqrbhXCjBUuaz8ATmZEJmCTe0erMDxyVu61o5k34Y9aqZtBt9gtevBfQd1CPy2ArlO/p8F9P6cL3RWXQ0Qt48A==","signatures":[{"sig":"MEYCIQCJQlcpAzRX8nXUrrN5fE5P8m/KCsKNH2+COo8th5v9uQIhALi2e6TRgjj1mxrCsUGDDyCx53am53e/ykbpHfbkEbMf","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":8079634},"main":"src/index.js","type":"commonjs","types":"index.d.ts","exports":{".":{"types":"./index.d.ts","import":"./src/index.mjs","require":"./src/index.js"}},"scripts":{"test":"node tests/run-tests.js","bench":"node --expose-gc scripts/bench.js","build":"node scripts/build-vocabs.js","compat":"node scripts/compat.js"},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"_npmVersion":"11.6.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","directories":{},"_nodeVersion":"24.13.0","_hasShrinkwrap":false,"devDependencies":{"js-tiktoken":"^1.0.21","ai-tokenizer":"^1.0.6"},"_npmOperationalInternal":{"tmp":"tmp/bpe-lite_0.4.0_1773259670851_0.4934761441709199","host":"s3://npm-registry-packages-npm-production"}},"0.4.1":{"name":"bpe-lite","version":"0.4.1","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","_id":"bpe-lite@0.4.1","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"dist":{"shasum":"cb054427c462831d93e8747c31224960e1fca9da","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.4.1.tgz","fileCount":12,"integrity":"sha512-uaGu/1iJ3fc4Lp5+Rj/tjRXU3O8F9it3oPGSxA/C52XG1Zk83GrxhGxPNDY4F/9WXS8pEZjFd3mxIJHuFPjzmw==","signatures":[{"sig":"MEUCIQDjGSHwtlO71tF5tL5nDHDEQd9hPAIqGenxOGSZlR0BqAIgO2xS9ops+U4tkRN0ruT0YwuzMnSsBRfFrGwUOZ+A4N0=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":8079766},"main":"src/index.js","type":"commonjs","types":"index.d.ts","exports":{".":{"types":"./index.d.ts","import":"./src/index.mjs","require":"./src/index.js"}},"scripts":{"test":"node tests/run-tests.js","bench":"node --expose-gc scripts/bench.js","build":"node scripts/build-vocabs.js","compat":"node scripts/compat.js"},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"_npmVersion":"11.6.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","directories":{},"_nodeVersion":"24.13.0","_hasShrinkwrap":false,"devDependencies":{"js-tiktoken":"^1.0.21","ai-tokenizer":"^1.0.6"},"_npmOperationalInternal":{"tmp":"tmp/bpe-lite_0.4.1_1773260275611_0.7827340722533773","host":"s3://npm-registry-packages-npm-production"}},"0.4.2":{"name":"bpe-lite","version":"0.4.2","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","_id":"bpe-lite@0.4.2","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"dist":{"shasum":"667cd534c1183a0bb7ab1e09a1996584b3178a7d","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.4.2.tgz","fileCount":12,"integrity":"sha512-U7yhkU3wdngw2tNiPz2yBtsvX5JITaS/l9GI/31caT7NT4Y8LPFsf7HR3gBz2n11h9JpZ9gtNgC8c2c+ZSQdvQ==","signatures":[{"sig":"MEYCIQCFds6l4WNnu5NshVPsg8Xle9BXotjd6a57a14ijiu8egIhAKF6OdzAOTpyNAdE99Xx2/iJ5OQDpoK/awERydIbyJ5C","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":8079668},"main":"src/index.js","type":"commonjs","types":"index.d.ts","exports":{".":{"types":"./index.d.ts","import":"./src/index.mjs","require":"./src/index.js"}},"scripts":{"test":"node tests/run-tests.js","bench":"node --expose-gc scripts/bench.js","build":"node scripts/build-vocabs.js","compat":"node scripts/compat.js"},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"_npmVersion":"11.6.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","directories":{},"_nodeVersion":"24.13.0","_hasShrinkwrap":false,"devDependencies":{"js-tiktoken":"^1.0.21","ai-tokenizer":"^1.0.6"},"_npmOperationalInternal":{"tmp":"tmp/bpe-lite_0.4.2_1773260643475_0.7205144874140343","host":"s3://npm-registry-packages-npm-production"}},"0.4.3":{"name":"bpe-lite","version":"0.4.3","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","_id":"bpe-lite@0.4.3","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"homepage":"https://github.com/jschoemaker/bpe-lite#readme","bugs":{"url":"https://github.com/jschoemaker/bpe-lite/issues"},"dist":{"shasum":"68d2b7d7b516c7f7b38da4022ea968c65ab50075","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.4.3.tgz","fileCount":12,"integrity":"sha512-dWi0sCfGpEZ/IMBrwBxzURYA9VmMOibpMlladb6FIOanqr8gM4nIbfDigHJHFj9fnm1VWiEyhk5HrVC0Jhxv3Q==","signatures":[{"sig":"MEQCIE0VNqIkhzR95lUwLI08lVgwXn5E5YsS4w60XYz5mARMAiAtBhN7WfEop8Pp8QbZvfks0KFQgk0ddDIRLJG1H/H7gw==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":8079763},"main":"src/index.js","type":"commonjs","types":"index.d.ts","exports":{".":{"types":"./index.d.ts","import":"./src/index.mjs","require":"./src/index.js"}},"gitHead":"f553c8d9f5131dfd4133a27c6d97108a9058ace0","scripts":{"test":"node tests/run-tests.js","bench":"node --expose-gc scripts/bench.js","build":"node scripts/build-vocabs.js","compat":"node scripts/compat.js"},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"repository":{"url":"git+https://github.com/jschoemaker/bpe-lite.git","type":"git"},"_npmVersion":"11.6.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","directories":{},"_nodeVersion":"24.13.0","_hasShrinkwrap":false,"devDependencies":{"js-tiktoken":"^1.0.21","ai-tokenizer":"^1.0.6"},"_npmOperationalInternal":{"tmp":"tmp/bpe-lite_0.4.3_1773334370561_0.4248021541343705","host":"s3://npm-registry-packages-npm-production"}},"0.5.1":{"name":"bpe-lite","version":"0.5.1","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","_id":"bpe-lite@0.5.1","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"homepage":"https://github.com/jschoemaker/bpe-lite#readme","bugs":{"url":"https://github.com/jschoemaker/bpe-lite/issues"},"dist":{"shasum":"a7dd826fa177895e033bf72dea4f05462a896fbf","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.5.1.tgz","fileCount":12,"integrity":"sha512-g7ZoQoJ3c0ruY8GDFGlTgpmTnLbE+1UOODnFD9KfuTyj2s53QjKX/ps+zz5dxtanhbx6aW+KBgBTkFdN5NJXgQ==","signatures":[{"sig":"MEUCIBNs6H7mrtTpJru4Brg5Ufq/HhzvmLEWe4Ni5cYKCmd5AiEAs0kEtE1qsiTvZ9LtljA93hgRHKWK3OUVq3eyAPBbPy4=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":7787570},"main":"src/index.js","type":"commonjs","types":"index.d.ts","exports":{".":{"types":"./index.d.ts","import":"./src/index.mjs","require":"./src/index.js"}},"gitHead":"bacec10b5b27b0de086e800ca748f4e7d491d57f","scripts":{"test":"node tests/run-tests.js","bench":"node --expose-gc scripts/bench.js","build":"node scripts/build-vocabs.js","compat":"node scripts/compat.js"},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"repository":{"url":"git+https://github.com/jschoemaker/bpe-lite.git","type":"git"},"_npmVersion":"11.6.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","directories":{},"_nodeVersion":"24.13.0","_hasShrinkwrap":false,"devDependencies":{"js-tiktoken":"^1.0.21","ai-tokenizer":"^1.0.6","gpt-tokenizer":"^3.4.0"},"_npmOperationalInternal":{"tmp":"tmp/bpe-lite_0.5.1_1773513519725_0.4018438996150644","host":"s3://npm-registry-packages-npm-production"}},"0.5.2":{"name":"bpe-lite","version":"0.5.2","description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","main":"src/index.js","types":"index.d.ts","type":"commonjs","exports":{".":{"types":"./index.d.ts","import":"./src/index.mjs","require":"./src/index.js"}},"scripts":{"build":"node scripts/build-vocabs.js","bench":"node --expose-gc scripts/bench.js","compat":"node scripts/compat.js","test":"node tests/run-tests.js"},"keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"license":"MIT","devDependencies":{"ai-tokenizer":"^1.0.6","gpt-tokenizer":"^3.4.0","js-tiktoken":"^1.0.21"},"repository":{"type":"git","url":"git+https://github.com/jschoemaker/bpe-lite.git"},"gitHead":"96df4cd69dbe5fcec32ad2d9310725d7a4d5fb1a","_id":"bpe-lite@0.5.2","bugs":{"url":"https://github.com/jschoemaker/bpe-lite/issues"},"homepage":"https://github.com/jschoemaker/bpe-lite#readme","_nodeVersion":"24.13.0","_npmVersion":"11.6.2","dist":{"integrity":"sha512-gyQ+nptbm7OIWuCBxtE/8a0f0oQk0OEe6rRCnBGvGtLgwZrKj8qUt7A+EENU9wOjZbDtD2W4rwFEkQ1LRxkVJQ==","shasum":"513ee3be57ac3d84d46046cc7c50c0b093ae4a27","tarball":"https://registry.npmjs.org/bpe-lite/-/bpe-lite-0.5.2.tgz","fileCount":12,"unpackedSize":7791905,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQDtQp7612uv+8lZ+oH7pODUsIGg0Ow70NG7OsC5I8lF+wIgFVHMpDmZJ9MDvoCSyQSK60mbUdSRelcpSkV2YNQN7wE="}]},"_npmUser":{"name":"jerown","email":"jeroenschoemaker1@gmail.com"},"directories":{},"maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/bpe-lite_0.5.2_1773952372291_0.38222150061973514"},"_hasShrinkwrap":false}},"time":{"created":"2026-03-10T21:26:11.592Z","modified":"2026-03-19T20:32:52.882Z","0.1.0":"2026-03-10T21:26:11.951Z","0.1.1":"2026-03-10T21:29:36.438Z","0.3.0":"2026-03-11T06:52:32.494Z","0.3.1":"2026-03-11T06:54:55.612Z","0.4.0":"2026-03-11T20:07:51.265Z","0.4.1":"2026-03-11T20:17:56.010Z","0.4.2":"2026-03-11T20:24:03.889Z","0.4.3":"2026-03-12T16:52:50.974Z","0.5.1":"2026-03-14T18:38:40.087Z","0.5.2":"2026-03-19T20:32:52.769Z"},"bugs":{"url":"https://github.com/jschoemaker/bpe-lite/issues"},"license":"MIT","homepage":"https://github.com/jschoemaker/bpe-lite#readme","keywords":["tokenizer","bpe","openai","anthropic","gemini","tokens","llm"],"repository":{"type":"git","url":"git+https://github.com/jschoemaker/bpe-lite.git"},"description":"Offline BPE tokenizer for OpenAI, Anthropic, and Gemini — zero dependencies","maintainers":[{"name":"jerown","email":"jeroenschoemaker1@gmail.com"}],"readme":"# bpe-lite\n\nOffline BPE tokenizer for OpenAI, Anthropic, and Gemini. Zero dependencies, no network calls at runtime. Works in any Node 18+ environment including Docker and edge runtimes. Ships CJS and ESM. TypeScript types included.\n\n```js\nconst { countTokens } = require('bpe-lite');\n\ncountTokens('Hello, world!', 'openai-o200k') // → 4  (GPT-4o, o1, o3, o4, GPT-4.1, GPT-5)\ncountTokens('Hello, world!', 'openai')       // → 4  (GPT-4, GPT-3.5)\ncountTokens('Hello, world!', 'anthropic')    // → 4\ncountTokens('Hello, world!', 'gemini')       // → 4\n```\n\n## Install\n\n```bash\nnpm install bpe-lite\n```\n\n## Usage\n\nBoth CommonJS and ESM are supported:\n\n```js\n// CommonJS\nconst { countTokens, encode, decode, isWithinTokenLimit, openai, openaiO200k, anthropic, gemini } = require('bpe-lite');\n\n// ESM\nimport { countTokens, encode, decode, isWithinTokenLimit, openai, openaiO200k, anthropic, gemini } from 'bpe-lite';\n```\n\n```js\n// Count tokens\ncountTokens('Your text here', 'openai-o200k'); // → number (GPT-4o, o1, o3, o4, GPT-4.1, GPT-5)\ncountTokens('Your text here', 'openai');       // → number (GPT-4, GPT-3.5)\n\n// Encode / decode\nconst ids = encode('Hello', 'openai-o200k');   // → [13225]\ndecode(ids, 'openai-o200k');                   // → 'Hello'\n\n// Check token limit — short-circuits early on long text, more efficient than encode()\n// Returns the token count if within the limit, false if exceeded\nisWithinTokenLimit('Hello, world!', 10, 'openai-o200k'); // → 4\nisWithinTokenLimit('Hello, world!', 3,  'openai-o200k'); // → false\n\n// Tokenizer instances — lazy-loaded and cached per provider\nconst tok = openaiO200k();\ntok.encode('Hello');          // → [13225]\ntok.decode([13225]);          // → 'Hello'\ntok.count('Hello, world!');   // → 4\n```\n\n## Providers\n\n| Provider | Vocab | Models | Accuracy |\n|----------|-------|--------|----------|\n| `openai-o200k` | o200k_base (200k) | GPT-4o, o1, o3, o4, GPT-4.1, GPT-5 | Exact — vocab sourced from OpenAI |\n| `openai` | cl100k_base (100k) | GPT-4, GPT-3.5 | Exact — vocab sourced from OpenAI |\n| `anthropic` | Xenova/claude-tokenizer (65k BPE) | Claude | See accuracy section below |\n| `gemini` | Gemma 3 SPM (262k) | Gemini | See accuracy section below |\n\nVocab files are bundled in the package — no network required at runtime or install time.\n\n## Accuracy — Anthropic\n\nAnthropic has not released the Claude tokenizer. bpe-lite uses [`Xenova/claude-tokenizer`](https://huggingface.co/Xenova/claude-tokenizer), a community reverse-engineering of the ~65k BPE vocabulary, with hand-tuned byte-level corrections applied on top.\n\n### Benchmark methodology\n\nTested against `claude-haiku-4-5-20251001` via the Anthropic `count_tokens` API on a 120-sample stratified corpus across 12 categories. The corpus deliberately over-represents difficult content (Arabic, symbols, emoji, numbers) to expose systematic failures — overall numbers are lower than you would see on typical prose-only workloads by design.\n\n### Overall results (120 samples, 114 eligible)\n\n| Metric | bpe-lite | ai-tokenizer |\n|--------|----------|--------------|\n| Within 5% | 46.5% | 18.4% |\n| Within 10% | 62.3% ±8.8% CI | 37.7% ±8.8% CI |\n| Mean abs error | 9.4% | 16.0% |\n| Median abs error | 5.7% | 13.6% |\n| Max abs error | 42.9% | 82.6% |\n\n### Per-category breakdown\n\n| Category | Within 10% | Mean error | Notes |\n|----------|-----------|------------|-------|\n| `code-js` | 100% | 4.2% | |\n| `english-prose` | 90% | 5.5% | |\n| `code-python` | 90% | 4.8% | |\n| `structured` | 90% | 3.6% | JSON, HTML, XML, Markdown, SQL |\n| `numbers` | 80% | 7.3% | |\n| `hex-binary` | 80% | 5.3% | |\n| `urls` | 80% | 3.6% | |\n| `cjk` | 40% | 8.8% | |\n| `short` | 30% | 6.8% | |\n| `emoji` | 20% | 17.7% | ZWJ sequences, flags, skin tones |\n| `symbols` | 10% | 17.6% | Cross-byte merges unreplicable |\n| `arabic` | 0% | 26.1% | Structural vocabulary gap — unfixable |\n\nFor prose, code, structured data, and URLs — the dominant content types in real-world prompts — bpe-lite is within 10% on 80–100% of samples. Arabic and symbol-cluster-heavy content cannot be accurately estimated without the actual Claude tokenizer.\n\n### Why bpe-lite outperforms ai-tokenizer on Claude\n\nai-tokenizer's `claude` encoding uses `\\p{N}+` (greedy, unlimited digit chunks). Current Claude models use `\\p{N}{1,3}` (1–3 digits). This causes 20–43% errors on anything involving numbers — including code, hex, and data. bpe-lite uses the correct pattern.\n\nai-tokenizer also does not have a Gemini encoding: all Gemini models are mapped to OpenAI's `o200k_base` vocabulary with a fudge multiplier. This is wrong by construction — see below.\n\n## Accuracy — Gemini\n\nbpe-lite implements the full Gemma 3 SentencePiece BPE algorithm using the actual Gemini vocabulary. On a 25-sample test against the Gemini API, bpe-lite scored 100% exact (no failures found; 25 samples is a limited basis — treat this as a lower bound, not a guarantee across all content types).\n\nai-tokenizer does not implement Gemini natively. Inspecting their bundled source (`dist/index.js`), every Gemini model is defined as `\"encoding\": \"o200k_base\"` with a `\"contentMultiplier\": 1.08` fudge factor — it runs the OpenAI vocabulary through a multiplier rather than using Gemini's actual tokenizer. bpe-lite uses the actual Gemma 3 vocabulary and SentencePiece algorithm.\n\n## Performance\n\nBenchmarked on Node v24 (win32/x64). Run `node --expose-gc scripts/bench.js` locally for numbers on your hardware.\n\n**Large text (~500 KB) — ops/s**\n\n| impl | cl100k | Anthropic | Gemini | note |\n|------|-------:|----------:|-------:|------|\n| bpe-lite | 291 | 289 | 998 | |\n| ai-tokenizer | 291 | 291 | 215 | Gemini column uses o200k — wrong algorithm |\n| js-tiktoken | 30 | — | — | WASM overhead |\n\nbpe-lite matches ai-tokenizer throughput for OpenAI and Anthropic large text. On Gemini, bpe-lite's SPM engine is ~4.6x faster — the ai-tokenizer column there is not a valid comparison since it uses a different algorithm.\n\n**Small text cold vs warm**\n\nbpe-lite maintains a per-instance chunk cache. For repeated text patterns (e.g. the same prompt template encoded thousands of times), cache hits eliminate BPE work entirely:\n\n| scenario | bpe-lite | ai-tokenizer |\n|----------|----------|--------------|\n| Small text, cold (novel input) | ~352k ops/s | ~453k ops/s |\n| Small text, warm (repeated input) | ~1.45M ops/s | ~453k ops/s |\n\nFor diverse, non-repeating inputs, ai-tokenizer is ~29% faster on very short strings. For any repeated-text workload, bpe-lite is ~3x faster.\n\n**Initialization**\n\nbpe-lite lazy-loads the gzipped vocab on first encode — one-time cost of ~235ms per provider per process. Negligible for any persistent process. Relevant only for cold serverless invocations that encode once and exit.\n\n## API\n\n### `countTokens(text, provider?)`\n\nReturns the number of tokens in `text`. Default provider: `'openai'`.\n\n### `encode(text, provider?)`\n\nReturns an array of token ids.\n\n### `decode(ids, provider?)`\n\nDecodes an array of token ids back to a string.\n\n### `isWithinTokenLimit(text, limit, provider?)`\n\nReturns the token count if `text` is within `limit` tokens, or `false` if exceeded. More efficient than `encode()` for long texts — short-circuits as soon as the limit is crossed.\n\n### Tokenizer instances\n\n`openai()`, `openaiO200k()`, `anthropic()`, `gemini()` each return a cached `Tokenizer` object with `.encode()`, `.decode()`, and `.count()` methods. Instances are created once per provider per process.\n\n## Why not tiktoken?\n\n`tiktoken` is accurate for OpenAI but requires Rust/WASM native bindings, which can break in Docker containers, edge runtimes, and serverless environments. `bpe-lite` is pure JavaScript — it runs anywhere Node 18+ runs, with no native compilation step.\n\n## License\n\nMIT\n","readmeFilename":"README.md"}