{"_id":"tokenx","_rev":"13-36bf84d321bf02ca60dfb6552da482f1","name":"tokenx","dist-tags":{"latest":"2.1.0"},"versions":{"1.0.0":{"name":"tokenx","version":"1.0.0","author":"","license":"ISC","_id":"tokenx@1.0.0","maintainers":[{"name":"johannschopplich","email":"mail@johannschopplich.com"}],"dist":{"shasum":"6902281a0b07f3ef2d48c61d230c35717fb205a6","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-1.0.0.tgz","fileCount":1,"integrity":"sha512-3cuy4Kz78IsaL6CvWFJwkWyvIZSEQGplZ84j/YUUKKIJfkzPUiimYJSYuuSbMWd0hdNyVnz+ovKWZzdgz53rvw==","signatures":[{"sig":"MEQCIEQuf2y76Pb/AnSiiRGvoC2M7hLlXuEKmx7rbiJ6hPRGAiAMgwd9S2l7qmLZnXookl8Ph3kOK+Uc6o8peBzi7uuHSg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":202},"main":"index.js","scripts":{"test":"echo \"Error: no test specified\" && exit 1"},"_npmUser":{"name":"johannschopplich","email":"mail@johannschopplich.com"},"_npmVersion":"10.5.0","directories":{},"_nodeVersion":"20.12.2","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/tokenx_1.0.0_1718083634904_0.9931924852501501","host":"s3://npm-registry-packages"}},"0.4.0":{"name":"tokenx","version":"0.4.0","keywords":["ai","gpt","token","tiktoken"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@0.4.0","maintainers":[{"name":"johannschopplich","email":"mail@johannschopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"ef9ed2e94b725ca50948b486c67afcbe97f25b45","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-0.4.0.tgz","fileCount":8,"integrity":"sha512-RT6a7tPHtMAGtyvNg1kISaqHqXNVbQi3yaCSoUdiswV+dRSieApKrqjh/x5nr7qZp6dzWFFzitC77YQ70obZJg==","signatures":[{"sig":"MEYCIQC5Q8QuoLesvXucgG3WpBfr9mtWZiyLjordeyucIEgyEwIhANpB+eKJ3KaJ7BdV1FRFWKiNY1SSmCBrITKfHtLGgYLj","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":20239},"main":"./dist/index.cjs","type":"module","types":"./dist/index.d.ts","module":"./dist/index.mjs","exports":{".":{"types":"./dist/index.d.mts","import":{"types":"./dist/index.d.mts","default":"./dist/index.mjs"},"default":"./dist/index.mjs","require":{"types":"./dist/index.d.cts","default":"./dist/index.cjs"}}},"gitHead":"5a64f2fe1870cf138d00f2db8c97b3d028fbd6db","scripts":{"dev":"unbuild --stub","lint":"eslint .","test":"vitest","build":"unbuild","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit","docs:generate":"tsx scripts/generateTable.ts"},"_npmUser":{"name":"johannschopplich","email":"mail@johannschopplich.com"},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"10.7.0","description":"GPT token estimation and context size utilities without a full tokenizer","directories":{},"sideEffects":false,"_nodeVersion":"20.15.0","_hasShrinkwrap":false,"packageManager":"pnpm@9.5.0","devDependencies":{"tsx":"^4.16.2","bumpp":"^9.4.1","eslint":"^9.6.0","vitest":"^1.6.0","unbuild":"^3.0.0-rc.6","typescript":"^5.5.3","@types/node":"^20.14.10","gpt-tokenizer":"^2.1.2","@antfu/eslint-config":"^2.21.3"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_0.4.0_1720434628047_0.3840354338047669","host":"s3://npm-registry-packages"}},"0.4.1":{"name":"tokenx","version":"0.4.1","keywords":["ai","gpt","token","tiktoken"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@0.4.1","maintainers":[{"name":"johannschopplich","email":"mail@johannschopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"34677c287bd294ea53fdc14ebcb6d10bbeca1801","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-0.4.1.tgz","fileCount":8,"integrity":"sha512-LCMniis0WsHel07xh3K9OIt5c9Xla1awtOoWBmUHZBQR7pvTvgGFuYpLiCZWohXPC1YuZORnN0+fCVYI/ie8Jg==","signatures":[{"sig":"MEUCIQCKvWZqrkpmkkT90LPUeUTeMQTOMkYYOOAsi6Zcr4UDLQIgN6/q0U4RGqzRDIiIh70J2RRQGy96LMQK4676AJ0lrB0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":19935},"main":"./dist/index.cjs","type":"module","types":"./dist/index.d.ts","module":"./dist/index.mjs","exports":{".":{"types":"./dist/index.d.mts","import":{"types":"./dist/index.d.mts","default":"./dist/index.mjs"},"default":"./dist/index.mjs","require":{"types":"./dist/index.d.cts","default":"./dist/index.cjs"}}},"gitHead":"fe622396e169d88644807a3f94452beebd15860e","scripts":{"dev":"unbuild --stub","lint":"eslint .","test":"vitest","build":"unbuild","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit","docs:generate":"tsx scripts/generateTable.ts"},"_npmUser":{"name":"johannschopplich","email":"mail@johannschopplich.com"},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"10.8.2","description":"GPT token estimation and context size utilities without a full tokenizer","directories":{},"sideEffects":false,"_nodeVersion":"20.18.1","_hasShrinkwrap":false,"packageManager":"pnpm@9.14.4","devDependencies":{"tsx":"^4.19.2","bumpp":"^9.8.1","eslint":"^9.15.0","vitest":"^2.1.6","unbuild":"^3.0.0-rc.11","typescript":"^5.7.2","@types/node":"^22.10.1","gpt-tokenizer":"^2.7.0","@antfu/eslint-config":"^3.11.2"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_0.4.1_1732880613395_0.0013941199083553624","host":"s3://npm-registry-packages"}},"1.0.1":{"name":"tokenx","version":"1.0.1","keywords":["ai","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@1.0.1","maintainers":[{"name":"johannschopplich","email":"mail@johannschopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"d1ae65bef2569cd8ff8a82049beec75bc4c14ad1","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-1.0.1.tgz","fileCount":5,"integrity":"sha512-MhOngUHRuVE0CHP4cNEZ/XpdXETFL65nJpEvoTW+VYPuXsT/MTeNj+UNnekNsnxecmj2DEvUYPebqz+CsPTUSg==","signatures":[{"sig":"MEQCIHYCi9riIritRznAwDbeNd9VGUsudFvu3v9WWh2wJHi1AiArhBRihTBuvpWI6HcdMW1jRi9MO1Io35bi9Dyw8JZzaw==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":10170},"type":"module","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"gitHead":"caa13a1c50e378c6c4c79574d14a6461a845afe6","scripts":{"lint":"eslint .","test":"vitest","build":"tsdown","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit","docs:generate":"tsx scripts/generateTable.ts"},"_npmUser":{"name":"johannschopplich","email":"mail@johannschopplich.com"},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"10.9.2","description":"Fast and lightweight token estimation for any LLM without requiring a full tokenizer","directories":{},"sideEffects":false,"_nodeVersion":"22.15.0","_hasShrinkwrap":false,"packageManager":"pnpm@10.11.0","devDependencies":{"tsx":"^4.19.4","bumpp":"^10.1.1","eslint":"^9.28.0","tsdown":"^0.12.6","vitest":"^3.2.0","typescript":"^5.8.3","@types/node":"^22.15.29","gpt-tokenizer":"^2.9.0","@antfu/eslint-config":"^4.13.2"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_1.0.1_1748875021170_0.1766664133112803","host":"s3://npm-registry-packages-npm-production"}},"1.1.0":{"name":"tokenx","version":"1.1.0","keywords":["ai","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@1.1.0","maintainers":[{"name":"johannschopplich","email":"mail@johannschopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"1367866b5e2822f2c3dbe96b66465d9bc721c640","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-1.1.0.tgz","fileCount":5,"integrity":"sha512-KCjtiC2niPwTSuz4ktM82Ki5bjqBwYpssiHDsGr5BpejN/B3ksacRvrsdoxljdMIh2nCX78alnDkeemBmYUmTA==","signatures":[{"sig":"MEUCICCU94xdS1PTbNDk9I+h9Qy9qnHAo3nvLQtn6U7rLGhHAiEA0FjgW4QVAgMp+GgHRRUSw/fghd/wZRH05XGxw7ra2sw=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":13624},"type":"module","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"gitHead":"42d0f1d53ad358e278e861a0ccbd47aab2c105f2","scripts":{"lint":"eslint .","test":"vitest","build":"tsdown","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit","docs:generate":"tsx scripts/generateTable.ts"},"_npmUser":{"name":"johannschopplich","actor":{"name":"johannschopplich","type":"user","email":"mail@johannschopplich.com"},"email":"mail@johannschopplich.com"},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"10.9.2","description":"Fast and lightweight token estimation for any LLM without requiring a full tokenizer","directories":{},"sideEffects":false,"_nodeVersion":"22.16.0","_hasShrinkwrap":false,"packageManager":"pnpm@10.11.0","devDependencies":{"tsx":"^4.19.4","bumpp":"^10.1.1","eslint":"^9.28.0","tsdown":"^0.12.6","vitest":"^3.2.0","typescript":"^5.8.3","@types/node":"^22.15.29","gpt-tokenizer":"^2.9.0","@antfu/eslint-config":"^4.13.2"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_1.1.0_1750682439356_0.23349010418990068","host":"s3://npm-registry-packages-npm-production"}},"1.2.0":{"name":"tokenx","version":"1.2.0","keywords":["ai","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@1.2.0","maintainers":[{"name":"johannschopplich","email":"johann@schopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"dedf3d2aba129670bb0cae5f0e81e3f13c97f4b5","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-1.2.0.tgz","fileCount":5,"integrity":"sha512-x4bRrL23b22H+EqW2pbhIkkt3ouj27ZGmAS1QoIqpocEO4m0sAl2H1M4L1UzKqleikY4U9lz/TbEw4jeG8tm2A==","signatures":[{"sig":"MEUCIA2TLaTaF4PGt+cEitjteEjiOOFeaQve+RynNu6yXfQUAiEA+e+B2TRkKFD00UzeNzD3A/nVbRLvrcImJSSUYnfI0kc=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/tokenx@1.2.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":16780},"type":"module","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"gitHead":"3ee5afa0f76a654cb9a6d873f6093cfa6d8e3151","scripts":{"lint":"eslint .","test":"vitest","build":"tsdown","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit","docs:generate":"tsx scripts/generateTable.ts"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:cabc91e4-f75e-4d80-a958-efb54847e3a5"}},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"11.6.2","description":"Fast and lightweight token estimation for any LLM without requiring a full tokenizer","directories":{},"sideEffects":false,"_nodeVersion":"24.10.0","_hasShrinkwrap":false,"packageManager":"pnpm@10.18.3","devDependencies":{"tsx":"^4.20.6","bumpp":"^10.3.1","eslint":"^9.37.0","tsdown":"^0.15.7","vitest":"^3.2.4","typescript":"^5.9.3","@types/node":"^24.7.2","gpt-tokenizer":"^3.2.0","@antfu/eslint-config":"^6.0.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_1.2.0_1760595032573_0.31162150800021915","host":"s3://npm-registry-packages-npm-production"}},"1.2.1":{"name":"tokenx","version":"1.2.1","keywords":["ai","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@1.2.1","maintainers":[{"name":"johannschopplich","email":"johann@schopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"38dce5b7d129d3ff239501f90b9a0187812e770f","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-1.2.1.tgz","fileCount":5,"integrity":"sha512-lVhFIhR2qh3uUyUA8Ype+HGzcokUJbHmRSN1TJKOe4Y26HkawQuLiGkUCkR5LD9dx+Rtp+njrwzPL8AHHYQSYA==","signatures":[{"sig":"MEYCIQCWefKgIXUN7EKTJqjaADz+zhVD+qgZ4vwVLPGbdfPo5gIhAModPOrKK7/3v6BxhenXRPCnIw1wAj3tG+Jgi3fNZoNz","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/tokenx@1.2.1","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":16805},"type":"module","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"gitHead":"a93a5da05b3d0bc03f19c8e1b465f5b139544c61","scripts":{"lint":"eslint .","test":"vitest","build":"tsdown","automd":"tsx scripts/generate-bench.ts && automd","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:cabc91e4-f75e-4d80-a958-efb54847e3a5"}},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"11.6.2","description":"Fast token estimation at 94% accuracy of a full tokenizer in a 2kB bundle","directories":{},"sideEffects":false,"_nodeVersion":"24.11.0","_hasShrinkwrap":false,"packageManager":"pnpm@10.21.0","devDependencies":{"tsx":"^4.20.6","bumpp":"^10.3.1","automd":"^0.4.2","eslint":"^9.39.1","tsdown":"^0.15.12","vitest":"^4.0.8","typescript":"^5.9.3","@types/node":"^24.10.0","gpt-tokenizer":"^3.4.0","@antfu/eslint-config":"^6.2.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_1.2.1_1762794014474_0.7009689809528041","host":"s3://npm-registry-packages-npm-production"}},"1.3.0":{"name":"tokenx","version":"1.3.0","keywords":["ai","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@1.3.0","maintainers":[{"name":"johannschopplich","email":"johann@schopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"d4373cc02de98cb5adeb1a48d52784893db3052b","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-1.3.0.tgz","fileCount":5,"integrity":"sha512-NLdXTEZkKiO0gZuLtMoZKjCXTREXeZZt8nnnNeyoXtNZAfG/GKGSbQtLU5STspc0rMSwcA+UJfWZkbNU01iKmQ==","signatures":[{"sig":"MEQCIBwWDe2nFP0ovYQl9mX2CumPehlNfYXL0NrPWcp0L75gAiA7L10VILmxzYDyOe9R+vn7dJ0tgXRDM6Jaz3TnsFfVEQ==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/tokenx@1.3.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":17028},"pnpm":{"onlyBuiltDependencies":["@parcel/watcher","esbuild"]},"type":"module","types":"./dist/index.d.mts","exports":{".":{"types":"./dist/index.d.mts","default":"./dist/index.mjs"}},"gitHead":"709fbad5daa23d1b6ddb31eb468d71c4f812a4d6","scripts":{"lint":"eslint .","test":"vitest","build":"tsdown","automd":"tsx scripts/generate-bench.ts && automd","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:cabc91e4-f75e-4d80-a958-efb54847e3a5"}},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"11.8.0","description":"Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle","directories":{},"sideEffects":false,"_nodeVersion":"24.12.0","_hasShrinkwrap":false,"packageManager":"pnpm@10.28.1","devDependencies":{"tsx":"^4.21.0","bumpp":"^10.4.0","automd":"^0.4.2","eslint":"^9.39.2","tsdown":"^0.19.0","vitest":"^4.0.17","typescript":"^5.9.3","@types/node":"^24.10.9","gpt-tokenizer":"^3.4.0","@antfu/eslint-config":"^7.2.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_1.3.0_1769096418712_0.7255624950258039","host":"s3://npm-registry-packages-npm-production"}},"1.4.0":{"name":"tokenx","version":"1.4.0","keywords":["ai","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@1.4.0","maintainers":[{"name":"johannschopplich","email":"johann@schopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"4333a4c1f1e1aa202495e08f09c53913ab052e42","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-1.4.0.tgz","fileCount":5,"integrity":"sha512-AphMpe4N2NlCagQnMOn+DMoaTJ4xFEWp0hfbDeCSuisZeCdBAa1z0r8OiHwofN5Igo9sQfXFd+NTVSVGEXu4cg==","signatures":[{"sig":"MEUCIBnmOkP9ldr2Soxu+Ikq2hBrHkB2bMyVshhhZunFAqSuAiEAvy5SxovmfJJB+hlWRFeQwxhR+dVbTvBRg9iwo1F3fNk=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"},{"sig":"MEQCIFQyUz6253E8yh8MPniYB4zivvlaxRBvDSJ/rK9wkuLqAiAsB7zhJxZ7fOnkOxcO1urllkEnAil2jwKM9cq3qJOhIg==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/tokenx@1.4.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":18917},"type":"module","types":"./dist/index.d.mts","exports":{".":{"types":"./dist/index.d.mts","default":"./dist/index.mjs"}},"gitHead":"a4bd39c0ff9ea2cae560dbb8c1fc374594c0da3c","scripts":{"lint":"eslint .","test":"vitest","build":"tsdown","automd":"node scripts/generate-bench.ts && automd","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:cabc91e4-f75e-4d80-a958-efb54847e3a5"}},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"12.0.1","description":"Fast token estimation at ~95% accuracy of a full tokenizer in a 2kB bundle","directories":{},"sideEffects":false,"_nodeVersion":"24.18.0","_hasShrinkwrap":false,"packageManager":"pnpm@11.17.0","devDependencies":{"bumpp":"^12.0.0","automd":"^0.4.3","eslint":"^10.8.0","tsdown":"^0.22.14","vitest":"^4.1.10","typescript":"^6.0.3","@types/node":"^26.1.1","gpt-tokenizer":"^3.4.0","@antfu/eslint-config":"^9.2.0","eslint-flat-config-utils":"^3.2.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_1.4.0_1785116797151_0.40092057617632193","host":"s3://npm-registry-packages-npm-production"}},"1.5.0":{"name":"tokenx","version":"1.5.0","keywords":["ai","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@1.5.0","maintainers":[{"name":"johannschopplich","email":"johann@schopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"ef618761bd381833e9285b13a42f6e87df20dd0e","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-1.5.0.tgz","fileCount":5,"integrity":"sha512-bPUbnWTMIurt+wuuu8M1aAKmmHoQQBsre9L0vbcRftO5xSXKZo5O9N3aXi09XkF5i8Wkjm3PX6LR8uQ+HkGz2w==","signatures":[{"sig":"MEQCIAEliGKeDn2Fow1cqTeW2G/LEMMs2J7/wl4gcvg4rwdPAiAWCdEEozZjLUQhY+n9ThWvXpTb8R6QuQEUB11lqVpyBw==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"},{"sig":"MEUCIQC2Obi9Z4r1o2VN9JhnkoXUlTfwixNfGqIKKesi21N7jQIgTWnPXdv7QOA6irSLvVJwoC+D6mh6AUxFw/XW2naKgco=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/tokenx@1.5.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":19282},"type":"module","types":"./dist/index.d.mts","exports":{".":{"types":"./dist/index.d.mts","default":"./dist/index.mjs"}},"gitHead":"725d3a46790ef07f53a046b86beb2a544588cdcf","scripts":{"lint":"eslint .","test":"vitest","build":"tsdown","automd":"node scripts/generate-bench.ts && automd","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:cabc91e4-f75e-4d80-a958-efb54847e3a5"}},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"12.0.1","description":"Fast token estimation at ~95% accuracy of a full tokenizer in a 2kB bundle","directories":{},"sideEffects":false,"_nodeVersion":"24.18.0","_hasShrinkwrap":false,"packageManager":"pnpm@11.17.0","devDependencies":{"bumpp":"^12.0.0","automd":"^0.4.3","eslint":"^10.8.0","tsdown":"^0.22.14","vitest":"^4.1.10","typescript":"^6.0.3","@types/node":"^26.1.1","gpt-tokenizer":"^3.4.0","@antfu/eslint-config":"^9.2.0","eslint-flat-config-utils":"^3.2.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_1.5.0_1785145238337_0.21081865649955844","host":"s3://npm-registry-packages-npm-production"}},"1.6.0":{"name":"tokenx","version":"1.6.0","keywords":["ai","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@1.6.0","maintainers":[{"name":"johannschopplich","email":"johann@schopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"8a5fde974ebd559d43081f143c7d70750b6dbea1","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-1.6.0.tgz","fileCount":5,"integrity":"sha512-CKTjk345ajvBAUp5xUI9a5KKN0zU0lBueVHQbCskH1Hp6WkUKsPW2qGCYNs0pxNyfzxfo+IIjdt2W4sMbw/qBw==","signatures":[{"sig":"MEYCIQDfNb6FLdpYJ6O/lUq1hTwO5nxMEdWWh3Yw5UaD7pmmxAIhAOMx9IuBnUZHqHgNELCG+4OYMBV4fizR5O4enBovk+ZA","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"},{"sig":"MEQCIDMlAT7Xnmzcqx67La69WqHpmGMv3kgECRQC8y3Oixh6AiB4Tv24vPP5mhTGcZmBvg6vGeZIesRkhUkWXHP1yEREGg==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/tokenx@1.6.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":19624},"type":"module","types":"./dist/index.d.mts","exports":{".":{"types":"./dist/index.d.mts","default":"./dist/index.mjs"}},"gitHead":"1f41011313af577a4636078da0b3c8d0e67c6523","scripts":{"lint":"eslint .","test":"vitest","build":"tsdown","automd":"node scripts/generate-bench.ts && automd","release":"bumpp","lint:fix":"eslint . --fix","test:types":"tsc --noEmit"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:cabc91e4-f75e-4d80-a958-efb54847e3a5"}},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"12.0.1","description":"Fast token estimation at ~96% accuracy of a full tokenizer in a 2kB bundle","directories":{},"sideEffects":false,"_nodeVersion":"24.18.0","_hasShrinkwrap":false,"packageManager":"pnpm@11.17.0","devDependencies":{"bumpp":"^12.0.0","automd":"^0.4.3","eslint":"^10.8.0","tsdown":"^0.22.14","vitest":"^4.1.10","typescript":"^6.0.3","@types/node":"^26.1.1","gpt-tokenizer":"^3.4.0","@antfu/eslint-config":"^9.2.0","eslint-flat-config-utils":"^3.2.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_1.6.0_1785174728149_0.8106018417896557","host":"s3://npm-registry-packages-npm-production"}},"2.0.0":{"name":"tokenx","version":"2.0.0","keywords":["ai","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","_id":"tokenx@2.0.0","maintainers":[{"name":"johannschopplich","email":"johann@schopplich.com"}],"homepage":"https://github.com/johannschopplich/tokenx#readme","bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"e59194ef04999b9c890e0ca4983e20fe502465fe","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-2.0.0.tgz","fileCount":5,"integrity":"sha512-kLZDSxJ/a8nvQEp0o6xfss4b7X8wa0YLdJfKD/TfzTCKiYR6nPckcAVdTymdo9f7ru9W/5gsWCk/BuPjpjUHwg==","signatures":[{"sig":"MEUCIQCGH0Ik/PAiOzEdgkbrwMd+wPhoZSe0SqoY72LfkruQ6QIgL5pafRmv2T5rUgPUHp8/l0dO0xQt/wcmNwnbE7ldg5g=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"},{"sig":"MEYCIQCJxGkw3WZopURIwZUlg+X1Ghpyd8M1C2cJYKawe5aWnAIhAP+IRfq8OPOwylX1Ufx9Vr9Zvyqi1gU6G0eEUJjQYZ0z","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/tokenx@2.0.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":21558},"type":"module","types":"./dist/index.d.mts","exports":{".":{"types":"./dist/index.d.mts","default":"./dist/index.mjs"}},"gitHead":"282a48f6f1e042274a05247e4a32e6cb8fedb88a","scripts":{"lint":"eslint .","test":"vitest","bench":"node scripts/generate-bench.ts && automd","build":"tsdown","release":"bumpp","lint:fix":"eslint . --fix","bench:png":"node scripts/render-bench.ts","test:types":"tsc --noEmit"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:cabc91e4-f75e-4d80-a958-efb54847e3a5"}},"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"12.0.1","description":"Fast token estimation with 95%+ average accuracy in a 2kB bundle","directories":{},"sideEffects":false,"_nodeVersion":"24.18.0","_hasShrinkwrap":false,"packageManager":"pnpm@11.17.0","devDependencies":{"ansis":"^4.3.1","bumpp":"^12.0.0","automd":"^0.4.3","eslint":"^10.8.0","tsdown":"^0.22.14","vitest":"^4.1.10","typescript":"^6.0.3","@types/node":"^26.1.1","gpt-tokenizer":"^3.4.0","@antfu/eslint-config":"^9.2.0","eslint-flat-config-utils":"^3.2.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenx_2.0.0_1785316603624_0.35477905413806154","host":"s3://npm-registry-packages-npm-production"}},"2.1.0":{"_id":"tokenx@2.1.0","bin":{"tokenx":"bin/tokenx.mjs"},"bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"dist":{"shasum":"1a4f2a30dedfdc52a12f6027c5ac014ac9de6e9e","tarball":"https://registry.npmjs.org/tokenx/-/tokenx-2.1.0.tgz","fileCount":7,"integrity":"sha512-iHhqfDuFFbMWHGfjEnVALgatV10KqcO5W5Fl99fYJmFjhE57JXPdnFrDlp6BTFIMmbbazsCg/lZYpJ2mPsPeQQ==","signatures":[{"sig":"MEQCIEN38Fe+mDcntzOq2dGEgCAPo2vg8uyKsgqE9zXA71lpAiAm2n7LpgpuT+md54Df769z/imL9ZCkRahHXBcQhlUEFg==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"},{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQCtfJTR4v/n88wMKOBAoyyNneKXPtUXqsWUYVXjFZZ8awIhANFvqUA65DIZe5UQO8edfbzHl3Gn6N+w4sKRAbF95AOf"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/tokenx@2.1.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":66954},"name":"tokenx","type":"module","types":"./dist/index.d.mts","author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"exports":{".":{"types":"./dist/index.d.mts","default":"./dist/index.mjs"}},"gitHead":"65bdb094ce945174d3d3ca2dbc9d1107ce2066ad","license":"MIT","scripts":{"dev":"node ./src/cli/entry.ts --help","lint":"eslint .","test":"vitest","bench":"node scripts/generate-bench.ts && automd","build":"tsdown","release":"bumpp","lint:fix":"eslint . --fix","bench:png":"node scripts/render-bench.ts","test:types":"tsc --noEmit"},"version":"2.1.0","_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:cabc91e4-f75e-4d80-a958-efb54847e3a5"}},"homepage":"https://github.com/johannschopplich/tokenx#readme","keywords":["ai","cli","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"_npmVersion":"12.0.2","description":"Fast token estimation with 95%+ average accuracy in a 2kB bundle","directories":{},"maintainers":[{"name":"johannschopplich","email":"johann@schopplich.com"}],"sideEffects":false,"_nodeVersion":"24.18.0","_hasShrinkwrap":false,"packageManager":"pnpm@11.20.0","devDependencies":{"ansis":"^4.3.1","bumpp":"^12.1.1","citty":"^0.2.2","automd":"^0.4.3","eslint":"^10.8.0","tsdown":"^0.22.14","vitest":"^4.1.10","typescript":"^6.0.3","@types/node":"^26.1.2","gpt-tokenizer":"^3.4.0","@antfu/eslint-config":"^9.2.0","eslint-flat-config-utils":"^3.2.0"},"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokenx_2.1.0_1785922992699_0.6943851584490222"}}},"time":{"created":"2024-06-11T05:27:14.903Z","modified":"2026-08-05T09:43:13.179Z","1.0.0":"2024-06-11T05:27:15.049Z","0.4.0":"2024-07-08T10:30:28.180Z","0.4.1":"2024-11-29T11:43:33.553Z","1.0.1":"2025-06-02T14:37:01.330Z","1.1.0":"2025-06-23T12:40:39.538Z","1.2.0":"2025-10-16T06:10:32.764Z","1.2.1":"2025-11-10T17:00:14.686Z","1.3.0":"2026-01-22T15:40:18.856Z","1.4.0":"2026-07-27T01:46:37.237Z","1.5.0":"2026-07-27T09:40:38.429Z","1.6.0":"2026-07-27T17:52:08.226Z","2.0.0":"2026-07-29T09:16:43.774Z","2.1.0":"2026-08-05T09:43:12.795Z"},"bugs":{"url":"https://github.com/johannschopplich/tokenx/issues"},"author":{"name":"Johann Schopplich","email":"hello@johannschopplich.com"},"license":"MIT","homepage":"https://github.com/johannschopplich/tokenx#readme","keywords":["ai","cli","llm","token","tokenizer","estimation","tiktoken","anthropic","openai"],"repository":{"url":"git+https://github.com/johannschopplich/tokenx.git","type":"git"},"description":"Fast token estimation with 95%+ average accuracy in a 2kB bundle","maintainers":[{"name":"johannschopplich","email":"johann@schopplich.com"}],"readme":"# tokenx\n\nFast and lightweight token count estimation without requiring a full tokenizer.\n\nEstimates are calibrated against OpenAI's `o200k_base` encoding – the tokenizer of all current GPT models. Counts for other LLM families will differ somewhat; the `defaultCharsPerToken` and `languageConfigs` options let you tune the heuristics for your model. For precise counts, use a full tokenizer like [`gpt-tokenizer`](https://github.com/niieani/gpt-tokenizer).\n\n## Features\n\n- ⚡ **95%+ average accuracy**, and no single sample below 90%\n- 📦 **Just 2kB** bundle size with zero dependencies\n- 🖥️ **[Bundled CLI](#cli)** – count, slice and split from the shell\n- 🌍 Multi-language support with configurable language rules\n- 🗣️ Built-in rules for accented scripts (German, French, Spanish, Slavic), Cyrillic, and Greek\n- 🀄 CJK (Chinese, Japanese, Korean) character handling\n- 😀 Emoji-aware pricing (emoji cost more tokens than their character count suggests)\n\n## Benchmarks\n\nThe following chart shows how close the estimates come to actual GPT token counts for different input texts:\n\n<!-- automd:file src=\"./docs/bench.md\" -->\n\nBars grow left when tokenx underestimates and right when it overestimates. The axis spans the ±10% per-sample deviation bound.\n\n```\n                                                          under ◂·▸ over\nTeam chat transcript (en)               293 →    285          ███│              -2.73%\nVite releases API response            8,075 →  8,551             │██████        +5.89%\ntokenx source code                    3,151 →  3,050          ███│              -3.21%\nVite plugin API docs (en)             6,901 →  7,155             │████          +3.68%\nCat article (ja)                     12,437 → 11,529      ███████│              -7.30%\nCat article (ko)                      7,117 →  6,841         ████│              -3.88%\nCat article (zh)                      9,057 →  8,828          ███│              -2.53%\nThe Great Gatsby by Fitzgerald (en)   4,391 →  4,479             │██            +2.00%\nDie Verwandlung by Kafka (de)         4,437 →  4,384            █│              -1.19%\n                                                       ─────────────────────\n                                                                        mean     3.60%\n```\n\n<!-- /automd -->\n\nAccuracy depends on the kind of text, not its length: a short excerpt deviates about as much as the full document it came from. A holdout corpus that no ratio was ever fitted against is held to a looser ±15% bound, so a retune cannot silently overfit the chart.\n\nThree cases are knowingly outside these bounds, all underestimates:\n\n- **High-entropy strings** – base64, hashes, digests: ≈-70%. Pricing them would cost every caller runtime for a case ordinary traffic rarely carries.\n- **Traditional and classical Chinese** – the hanzi rate is calibrated on contemporary simplified script: ≈-10% to -20%.\n- **Scripts without a built-in rule** – Arabic ≈-35%, Hindi ≈-30%, Hebrew ≈-45%, Thai ≈-60%; a custom language rule closes the gap.\n\n## Installation\n\n```bash\n# npm\nnpm install tokenx\n\n# pnpm\npnpm add tokenx\n\n# yarn\nyarn add tokenx\n```\n\n## CLI\n\nThe package ships a `tokenx` binary – no install needed via `npx`, or install it for the commands below.\n\n```bash\n# Count tokens in a file, a pipe, or several files at once\ntokenx README.md\ncat article.md | tokenx\ntokenx count src/*.ts\n\n# Fail a script when a prompt outgrows its budget (exit code 2)\ntokenx prompt.txt --limit 8000\n\n# Extract a token range, or chunk a document for RAG\ntokenx slice article.md --end 500\ntokenx split article.md --size 500 --overlap 50\n```\n\nOnly results go to stdout – counts as bare integers, chunks as a JSON array – so `$(tokenx count file.txt)` and `| jq` work unchanged. Notices and errors go to stderr. Run `tokenx --help` for every option.\n\n## Usage\n\n```ts\nimport { estimateTokenCount, isWithinTokenLimit, sliceByTokens, splitByTokens } from 'tokenx'\n\nconst text = 'Your text goes here.'\n\n// Estimate the number of tokens in the text\nconst estimatedTokens = estimateTokenCount(text)\nconsole.log(`Estimated token count: ${estimatedTokens}`)\n\n// Check if text is within a specific token limit\nconst tokenLimit = 1024\nconst withinLimit = isWithinTokenLimit(text, tokenLimit)\nconsole.log(`Is within token limit: ${withinLimit}`)\n\n// Slice text by token positions (like Array.slice)\nconst firstTokens = sliceByTokens(text, 0, 5)\nconsole.log(`First ~5 tokens: ${firstTokens}`)\n\n// Split text into token-based chunks\nconst chunks = splitByTokens(text, 100)\nconsole.log(`Split into ${chunks.length} chunks`)\n\n// Use custom options for different languages or models.\n// Custom language rules are checked before all built-in heuristics,\n// so they can also override the built-in CJK handling.\nconst customOptions = {\n  defaultCharsPerToken: 4, // More conservative estimation\n  languageConfigs: [\n    { pattern: /[\\u4E00-\\u9FFF]/, averageCharsPerToken: 2 }, // Custom Chinese rule\n  ]\n}\n\nconst customEstimate = estimateTokenCount(text, customOptions)\nconsole.log(`Custom estimate: ${customEstimate}`)\n```\n\n## API\n\n### `estimateTokenCount`\n\nEstimates the number of tokens in a given input string using heuristic rules that work across multiple languages and text types.\n\n**Usage:**\n\n```ts\nconst estimatedTokens = estimateTokenCount('Hello, world!')\n\n// With custom options\nconst customEstimate = estimateTokenCount('Bonjour le monde!', {\n  defaultCharsPerToken: 4,\n  languageConfigs: [\n    { pattern: /[éèêëàâîï]/i, averageCharsPerToken: 3 }\n  ]\n})\n```\n\n**Type Declaration:**\n\n```ts\nfunction estimateTokenCount(\n  text?: string,\n  options?: TokenEstimationOptions\n): number\n\ninterface TokenEstimationOptions {\n  /** Default average characters per token when no language-specific rule applies (default: 7). */\n  defaultCharsPerToken?: number\n  /** Custom language configurations to override defaults. */\n  languageConfigs?: LanguageConfig[]\n}\n\ninterface LanguageConfig {\n  /** Regular expression to detect the language. */\n  pattern: RegExp\n  /** Average number of characters per token for this language. */\n  averageCharsPerToken: number\n}\n```\n\n### `isWithinTokenLimit`\n\nChecks if the estimated token count of the input is within a specified token limit.\n\n**Usage:**\n\n```ts\nconst withinLimit = isWithinTokenLimit('Check this text against a limit', 100)\n// With custom options\nconst customCheck = isWithinTokenLimit('Text', 50, { defaultCharsPerToken: 3 })\n```\n\n**Type Declaration:**\n\n```ts\nfunction isWithinTokenLimit(\n  text: string,\n  tokenLimit: number,\n  options?: TokenEstimationOptions\n): boolean\n```\n\n### `sliceByTokens`\n\nExtracts a portion of text based on token positions, similar to `Array.prototype.slice()`. Supports both positive and negative indices.\n\n**Usage:**\n\n```ts\nconst text = 'Hello, world! This is a test sentence.'\n\nconst firstThree = sliceByTokens(text, 0, 3)\nconst fromSecond = sliceByTokens(text, 2)\nconst lastTwo = sliceByTokens(text, -2)\nconst middle = sliceByTokens(text, 1, -1)\n```\n\n**Type Declaration:**\n\n```ts\nfunction sliceByTokens(\n  text: string,\n  start?: number,\n  end?: number,\n  options?: TokenEstimationOptions\n): string\n```\n\n**Parameters:**\n\n- `text` - The input text to slice\n- `start` - The start token index (inclusive). If negative, treated as offset from end. Default: `0`\n- `end` - The end token index (exclusive). If negative, treated as offset from end. If omitted, slices to the end\n- `options` - Token estimation options (same as `estimateTokenCount`)\n\n**Returns:**\n\nThe sliced text portion corresponding to the specified token range.\n\n### `splitByTokens`\n\nSplits text into chunks based on token count. Useful for chunking documents for RAG, batch processing, or staying within context windows.\n\n`tokensPerChunk` is a target, not a hard maximum: a chunk closes once it reaches the target, so a single long segment can push a chunk beyond it. Chunks never break words apart. Segments split on whitespace and punctuation, so for CJK text – where a whole clause between two punctuation marks is a single segment – chunks can far exceed the target.\n\n**Usage:**\n\n```ts\nconst text = 'Long text that needs to be split into smaller chunks...'\n\n// Basic splitting\nconst chunks = splitByTokens(text, 100)\nconsole.log(`Split into ${chunks.length} chunks`)\n\n// With overlap for semantic continuity\nconst overlappedChunks = splitByTokens(text, 100, { overlap: 10 })\n\n// With custom options\nconst customChunks = splitByTokens(text, 50, {\n  defaultCharsPerToken: 4,\n  overlap: 5\n})\n```\n\n**Type Declaration:**\n\n```ts\ninterface SplitByTokensOptions extends TokenEstimationOptions {\n  /** Number of tokens to overlap between consecutive chunks (default: 0, clamped below `tokensPerChunk`). */\n  overlap?: number\n}\n\nfunction splitByTokens(\n  text: string,\n  tokensPerChunk: number,\n  options?: SplitByTokensOptions\n): string[]\n```\n\n**Parameters:**\n\n- `text` - The input text to split\n- `tokensPerChunk` - Target number of tokens per chunk\n- `options` - Token estimation options with optional overlap\n\n**Returns:**\n\nAn array of text chunks, each containing approximately `tokensPerChunk` tokens. With `overlap`, each chunk repeats the trailing tokens of the previous one; a final chunk consisting only of overlap content is never emitted.\n\n## License\n\n[MIT](./LICENSE) License © 2023-PRESENT [Johann Schopplich](https://github.com/johannschopplich)\n","readmeFilename":"README.md"}