{"_id":"@coderabbitai/openai-chat-tokens","_rev":"12-b3a5aca948850e045fa79818d6695d5b","name":"@coderabbitai/openai-chat-tokens","dist-tags":{"latest":"0.1.5"},"versions":{"0.1.0":{"name":"@coderabbitai/openai-chat-tokens","version":"0.1.0","license":"MIT","_id":"@coderabbitai/openai-chat-tokens@0.1.0","maintainers":[{"name":"coderabbitai","email":"gur@coderabbit.ai"}],"homepage":"https://github.com/coderabbitai/openai-chat-tokens","bugs":{"url":"https://github.com/coderabbitai/openai-chat-tokens/issues"},"dist":{"shasum":"2d6bac8d61deb5a6f0128983ff273afd08ab0db1","tarball":"https://registry.npmjs.org/@coderabbitai/openai-chat-tokens/-/openai-chat-tokens-0.1.0.tgz","fileCount":15,"integrity":"sha512-edcZxNIwOeafLPXfGPhNm7PH1tUnlBUSQ9L/eCJMri4w5a1DJrnvOKyiAj2TuXwKIqM6OdKz4MYmBPirRaW6gw==","signatures":[{"sig":"MEUCIGXisQDpTzk5+edT4HPPwsHpfcynKsF4uyAeR8Eta2/oAiEAvRpAwCPXFqjRqzRnO16mPTsrniR0IdlrjpuhuLDLG20=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@coderabbitai%2fopenai-chat-tokens@0.1.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":24051},"main":"dist/index.js","type":"module","_from":"file:coderabbitai-openai-chat-tokens-0.1.0.tgz","types":"dist/index.d.ts","module":"dist/index.js","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"scripts":{"docs":"typedoc","lint":"eslint . && prettier --check .","test":"vitest","build":"tsc","format":"prettier --write .","lint:fix":"eslint --fix .; prettier --list-different --write .","test:watch":"vitest --watch"},"_npmUser":{"name":"coderabbitai","email":"gur@coderabbit.ai"},"_resolved":"/tmp/cec087d61f1714eeb6de299f5804f610/coderabbitai-openai-chat-tokens-0.1.0.tgz","_integrity":"sha512-edcZxNIwOeafLPXfGPhNm7PH1tUnlBUSQ9L/eCJMri4w5a1DJrnvOKyiAj2TuXwKIqM6OdKz4MYmBPirRaW6gw==","repository":{"url":"git+https://github.com/coderabbitai/openai-chat-tokens.git","type":"git"},"_npmVersion":"10.5.0","description":"Estimate the number of tokens an OpenAI chat completion request will use","directories":{},"_nodeVersion":"21.7.1","dependencies":{"js-tiktoken":"^1.0.7"},"_hasShrinkwrap":false,"devDependencies":{"vite":"^5.2.6","eslint":"^8.57.0","openai":"^4.29.2","vitest":"^1.4.0","typedoc":"^0.25.12","prettier":"^3.1.0","typescript":"^5.4.3","eslint-config-prettier":"^9.1.0","@typescript-eslint/parser":"^7.4.0","@typescript-eslint/eslint-plugin":"^7.4.0"},"_npmOperationalInternal":{"tmp":"tmp/openai-chat-tokens_0.1.0_1711407967183_0.8812545292060996","host":"s3://npm-registry-packages"}},"0.1.1":{"name":"@coderabbitai/openai-chat-tokens","version":"0.1.1","license":"MIT","_id":"@coderabbitai/openai-chat-tokens@0.1.1","maintainers":[{"name":"coderabbitai","email":"gur@coderabbit.ai"}],"homepage":"https://github.com/coderabbitai/openai-chat-tokens","bugs":{"url":"https://github.com/coderabbitai/openai-chat-tokens/issues"},"dist":{"shasum":"dffb5615c528b2223577e18c5dfa3bda5c9e70f6","tarball":"https://registry.npmjs.org/@coderabbitai/openai-chat-tokens/-/openai-chat-tokens-0.1.1.tgz","fileCount":15,"integrity":"sha512-vF4wiEqRLB3CsV4Y1ee1rxErGZOcBs3nhO/FHuxewC30cEoZC0sRWbLL0ZP0YMCvy5YaxStkVnY/FqoYUGAI9g==","signatures":[{"sig":"MEYCIQCa3r1zD40OhBrXrWmoWA76+QK/DK+3jFwKsbr8RuZ4UgIhAIYx3kroanOhEAnzHUCH/Ys7HBwuFKfBOKIjIYZZtxeq","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@coderabbitai%2fopenai-chat-tokens@0.1.1","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":24051},"main":"dist/index.js","type":"module","_from":"file:coderabbitai-openai-chat-tokens-0.1.1.tgz","types":"dist/index.d.ts","module":"dist/index.js","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"scripts":{"docs":"typedoc","lint":"eslint . && prettier --check .","test":"vitest","build":"tsc","format":"prettier --write .","lint:fix":"eslint --fix .; prettier --list-different --write .","test:watch":"vitest --watch"},"_npmUser":{"name":"coderabbitai","email":"gur@coderabbit.ai"},"_resolved":"/tmp/0a568c398a3c4cbff347e40d8f28db1f/coderabbitai-openai-chat-tokens-0.1.1.tgz","_integrity":"sha512-vF4wiEqRLB3CsV4Y1ee1rxErGZOcBs3nhO/FHuxewC30cEoZC0sRWbLL0ZP0YMCvy5YaxStkVnY/FqoYUGAI9g==","repository":{"url":"git+https://github.com/coderabbitai/openai-chat-tokens.git","type":"git"},"_npmVersion":"10.5.0","description":"Estimate the number of tokens an OpenAI chat completion request will use","directories":{},"_nodeVersion":"21.7.3","dependencies":{"js-tiktoken":"^1.0.7"},"_hasShrinkwrap":false,"devDependencies":{"vite":"^5.2.6","eslint":"^8.57.0","openai":"^4.29.2","vitest":"^1.4.0","typedoc":"^0.25.12","prettier":"^3.1.0","typescript":"^5.4.3","eslint-config-prettier":"^9.1.0","@typescript-eslint/parser":"^7.4.0","@typescript-eslint/eslint-plugin":"^7.4.0"},"_npmOperationalInternal":{"tmp":"tmp/openai-chat-tokens_0.1.1_1714234353808_0.5537546196201062","host":"s3://npm-registry-packages"}},"0.1.2":{"name":"@coderabbitai/openai-chat-tokens","version":"0.1.2","license":"MIT","_id":"@coderabbitai/openai-chat-tokens@0.1.2","maintainers":[{"name":"coderabbitai","email":"gur@coderabbit.ai"}],"homepage":"https://github.com/coderabbitai/openai-chat-tokens","bugs":{"url":"https://github.com/coderabbitai/openai-chat-tokens/issues"},"dist":{"shasum":"6f8b483add8feed9fc6b1c1f7b7074087f21fdad","tarball":"https://registry.npmjs.org/@coderabbitai/openai-chat-tokens/-/openai-chat-tokens-0.1.2.tgz","fileCount":15,"integrity":"sha512-IjIDEK51xrauNqRSbTYsGwshM0kZzprg7ZwUvn/ELPI82/mZfpQxHqD1VuyVXrO5/OfNqVSIpqqyMdNoOGXsNw==","signatures":[{"sig":"MEUCIQC2dRfbxwBmjReQQVt9vWKKI+NkkqDNeeyNEG+h4hu06gIgc0gk8ivbJJMWGm6unKA+6L4o17ecXHsFo9avP2XH2Xs=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@coderabbitai%2fopenai-chat-tokens@0.1.2","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":24021},"main":"dist/index.js","type":"module","_from":"file:coderabbitai-openai-chat-tokens-0.1.2.tgz","types":"dist/index.d.ts","module":"dist/index.js","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"scripts":{"docs":"typedoc","lint":"eslint . && prettier --check .","test":"vitest","build":"tsc","format":"prettier --write .","lint:fix":"eslint --fix .; prettier --list-different --write .","test:watch":"vitest --watch"},"_npmUser":{"name":"coderabbitai","email":"gur@coderabbit.ai"},"_resolved":"/tmp/4192bb7aff32c897e3b29cc0789ab56a/coderabbitai-openai-chat-tokens-0.1.2.tgz","_integrity":"sha512-IjIDEK51xrauNqRSbTYsGwshM0kZzprg7ZwUvn/ELPI82/mZfpQxHqD1VuyVXrO5/OfNqVSIpqqyMdNoOGXsNw==","repository":{"url":"git+https://github.com/coderabbitai/openai-chat-tokens.git","type":"git"},"_npmVersion":"10.5.0","description":"Estimate the number of tokens an OpenAI chat completion request will use","directories":{},"_nodeVersion":"21.7.3","dependencies":{"js-tiktoken":"^1.0.7"},"_hasShrinkwrap":false,"devDependencies":{"vite":"^5.2.6","eslint":"^9.4.0","openai":"^4.29.2","vitest":"^1.4.0","typedoc":"^0.25.12","prettier":"^3.1.0","@eslint/js":"^9.4.0","typescript":"^5.4.3","typescript-eslint":"^7.12.0","eslint-config-prettier":"^9.1.0"},"_npmOperationalInternal":{"tmp":"tmp/openai-chat-tokens_0.1.2_1717440567996_0.4671831633365022","host":"s3://npm-registry-packages"}},"0.1.3":{"name":"@coderabbitai/openai-chat-tokens","version":"0.1.3","license":"MIT","_id":"@coderabbitai/openai-chat-tokens@0.1.3","maintainers":[{"name":"coderabbitai","email":"gur@coderabbit.ai"}],"homepage":"https://github.com/coderabbitai/openai-chat-tokens","bugs":{"url":"https://github.com/coderabbitai/openai-chat-tokens/issues"},"dist":{"shasum":"c570e51294d83fe28da579aab4f5209baf319d40","tarball":"https://registry.npmjs.org/@coderabbitai/openai-chat-tokens/-/openai-chat-tokens-0.1.3.tgz","fileCount":15,"integrity":"sha512-pmeVsHYgVyE6n8oqfcbk1zwXjyki7f2OUUlo1t3qwu8dQiRAn6agCbHgEnIq4hGH0vxOD9WzjXBScWrwfLJdBg==","signatures":[{"sig":"MEQCIGK8rcRsG0hkl9bFEJ80cY6mgM7P+jRJ2Gv+yo+ejyqwAiBnU5abBeTfaPnbwaD7v/swYTIaw9GQC+uvdtS0O1yNfw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@coderabbitai%2fopenai-chat-tokens@0.1.3","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":24020},"main":"dist/index.js","type":"module","_from":"file:coderabbitai-openai-chat-tokens-0.1.3.tgz","types":"dist/index.d.ts","module":"dist/index.js","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"scripts":{"docs":"typedoc","lint":"eslint . && prettier --check .","test":"vitest","build":"tsc","format":"prettier --write .","lint:fix":"eslint --fix .; prettier --list-different --write .","test:watch":"vitest --watch"},"_npmUser":{"name":"coderabbitai","email":"gur@coderabbit.ai"},"_resolved":"/tmp/d1e341f5fee2b9607d08a1892a7e9494/coderabbitai-openai-chat-tokens-0.1.3.tgz","_integrity":"sha512-pmeVsHYgVyE6n8oqfcbk1zwXjyki7f2OUUlo1t3qwu8dQiRAn6agCbHgEnIq4hGH0vxOD9WzjXBScWrwfLJdBg==","repository":{"url":"git+https://github.com/coderabbitai/openai-chat-tokens.git","type":"git"},"_npmVersion":"10.5.0","description":"Estimate the number of tokens an OpenAI chat completion request will use","directories":{},"_nodeVersion":"21.7.3","dependencies":{"js-tiktoken":"^1.0.7"},"_hasShrinkwrap":false,"devDependencies":{"vite":"^5.2.6","eslint":"^9.4.0","openai":"^4.29.2","vitest":"^1.4.0","typedoc":"^0.26.3","prettier":"^3.1.0","@eslint/js":"^9.4.0","typescript":"^5.4.3","typescript-eslint":"^7.12.0","eslint-config-prettier":"^9.1.0"},"_npmOperationalInternal":{"tmp":"tmp/openai-chat-tokens_0.1.3_1720030120763_0.5293333822499084","host":"s3://npm-registry-packages"}},"0.1.4":{"name":"@coderabbitai/openai-chat-tokens","version":"0.1.4","license":"MIT","_id":"@coderabbitai/openai-chat-tokens@0.1.4","maintainers":[{"name":"coderabbitai","email":"gur@coderabbit.ai"}],"homepage":"https://github.com/coderabbitai/openai-chat-tokens","bugs":{"url":"https://github.com/coderabbitai/openai-chat-tokens/issues"},"dist":{"shasum":"9c5f76de7031f52e13b1f32ec4b5fe9f9176c9df","tarball":"https://registry.npmjs.org/@coderabbitai/openai-chat-tokens/-/openai-chat-tokens-0.1.4.tgz","fileCount":15,"integrity":"sha512-ShWQsS8feYjXixMZf/EcEp8X8eVqWKInaDfbs1ikDdaE5yEx6rOMzVP27mS9SSAUTTspQdwDra8elcR4zR2v+A==","signatures":[{"sig":"MEUCIGOHUyE/bx0FLZ7FDBS+b7LnxkSBDGMVrJZSM/TnaltCAiEA0OSOUi6ZV4sv9uyJ8oYcrXLG5BKwVo5Z9CHyDKFyjD8=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@coderabbitai%2fopenai-chat-tokens@0.1.4","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":24021},"main":"dist/index.js","type":"module","_from":"file:coderabbitai-openai-chat-tokens-0.1.4.tgz","types":"dist/index.d.ts","module":"dist/index.js","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"scripts":{"docs":"typedoc","lint":"eslint . && prettier --check .","test":"vitest","build":"tsc","format":"prettier --write .","lint:fix":"eslint --fix .; prettier --list-different --write .","test:watch":"vitest --watch"},"_npmUser":{"name":"coderabbitai","email":"gur@coderabbit.ai"},"_resolved":"/tmp/3635771c8e6e767bb2d9c4b6326d4654/coderabbitai-openai-chat-tokens-0.1.4.tgz","_integrity":"sha512-ShWQsS8feYjXixMZf/EcEp8X8eVqWKInaDfbs1ikDdaE5yEx6rOMzVP27mS9SSAUTTspQdwDra8elcR4zR2v+A==","repository":{"url":"git+https://github.com/coderabbitai/openai-chat-tokens.git","type":"git"},"_npmVersion":"10.5.0","description":"Estimate the number of tokens an OpenAI chat completion request will use","directories":{},"_nodeVersion":"21.7.3","dependencies":{"js-tiktoken":"^1.0.12"},"_hasShrinkwrap":false,"devDependencies":{"vite":"^5.3.4","eslint":"^9.7.0","openai":"^4.52.7","vitest":"^2.0.3","typedoc":"^0.26.4","prettier":"^3.3.3","@eslint/js":"^9.7.0","typescript":"^5.5.3","typescript-eslint":"^7.16.1","eslint-config-prettier":"^9.1.0"},"_npmOperationalInternal":{"tmp":"tmp/openai-chat-tokens_0.1.4_1721327565634_0.641506179101853","host":"s3://npm-registry-packages"}},"0.1.5":{"name":"@coderabbitai/openai-chat-tokens","version":"0.1.5","license":"MIT","_id":"@coderabbitai/openai-chat-tokens@0.1.5","maintainers":[{"name":"coderabbitai","email":"gur@coderabbit.ai"}],"homepage":"https://github.com/coderabbitai/openai-chat-tokens","bugs":{"url":"https://github.com/coderabbitai/openai-chat-tokens/issues"},"dist":{"shasum":"c1e317cd8bdba58b7bea887c26d8d6f1f30c6a54","tarball":"https://registry.npmjs.org/@coderabbitai/openai-chat-tokens/-/openai-chat-tokens-0.1.5.tgz","fileCount":15,"integrity":"sha512-4JApy3q55fAeUC75xihifhuKkydFQO7dNsBtrQ84aLI4fIRV4aVph/GfdAp5ZxyQjTzo5Drmw3lntNgS6mfW5A==","signatures":[{"sig":"MEUCIAH/2QsCI/1yS685H2aXcB5fqjA73fqPsOe/bQvfcvqaAiEAs37JKVDGAaYSPk+zyJZSgaMTp40G9eMihxDt5f/li1I=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@coderabbitai%2fopenai-chat-tokens@0.1.5","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":24424},"main":"dist/index.js","type":"module","_from":"file:coderabbitai-openai-chat-tokens-0.1.5.tgz","types":"dist/index.d.ts","module":"dist/index.js","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"}},"scripts":{"docs":"typedoc","lint":"eslint . && prettier --check .","test":"vitest","build":"tsc","format":"prettier --write .","lint:fix":"eslint --fix .; prettier --list-different --write .","test:watch":"vitest --watch"},"_npmUser":{"name":"coderabbitai","email":"gur@coderabbit.ai"},"_resolved":"/tmp/b2267df09d26592e6d35a597d90a945b/coderabbitai-openai-chat-tokens-0.1.5.tgz","_integrity":"sha512-4JApy3q55fAeUC75xihifhuKkydFQO7dNsBtrQ84aLI4fIRV4aVph/GfdAp5ZxyQjTzo5Drmw3lntNgS6mfW5A==","repository":{"url":"git+https://github.com/coderabbitai/openai-chat-tokens.git","type":"git"},"_npmVersion":"10.5.0","description":"Estimate the number of tokens an OpenAI chat completion request will use","directories":{},"_nodeVersion":"21.7.3","dependencies":{"js-tiktoken":"^1.0.14"},"_hasShrinkwrap":false,"devDependencies":{"vite":"^5.4.2","eslint":"^9.9.1","openai":"^4.56.0","vitest":"^2.0.5","typedoc":"^0.26.6","prettier":"^3.3.3","@eslint/js":"^9.9.1","typescript":"^5.5.4","typescript-eslint":"^8.2.0","eslint-config-prettier":"^9.1.0"},"_npmOperationalInternal":{"tmp":"tmp/openai-chat-tokens_0.1.5_1724520676849_0.5297214243104387","host":"s3://npm-registry-packages"}}},"time":{"created":"2024-03-25T23:06:07.102Z","modified":"2026-09-11T23:18:57.722Z","0.1.0":"2024-03-25T23:06:07.329Z","0.1.1":"2024-04-27T16:12:33.997Z","0.1.2":"2024-06-03T18:49:28.160Z","0.1.3":"2024-07-03T18:08:41.010Z","0.1.4":"2024-07-18T18:32:45.804Z","0.1.5":"2024-08-24T17:31:17.033Z"},"bugs":{"url":"https://github.com/coderabbitai/openai-chat-tokens/issues"},"license":"MIT","homepage":"https://github.com/coderabbitai/openai-chat-tokens","repository":{"url":"git+https://github.com/coderabbitai/openai-chat-tokens.git","type":"git"},"description":"Estimate the number of tokens an OpenAI chat completion request will use","maintainers":[{"email":"hasit@coderabbit.ai","name":"hasit-coderabbit"},{"email":"gur@coderabbit.ai","name":"guritfaq"},{"email":"kartik@coderabbit.ai","name":"kartik-coderabbit"},{"email":"rohit@coderabbit.ai","name":"rohit121"}],"readme":"# `@coderabbitai/openai-chat-tokens`\n\n[![GitHub Pages](https://github.com/coderabbitai/openai-chat-tokens/actions/workflows/github-pages.yaml/badge.svg)](https://github.com/coderabbitai/openai-chat-tokens/actions/workflows/github-pages.yaml) [![Node.js CI](https://github.com/coderabbitai/openai-chat-tokens/actions/workflows/node.js.yaml/badge.svg)](https://github.com/coderabbitai/openai-chat-tokens/actions/workflows/node.js.yaml) [![GitHub Downloads](https://img.shields.io/github/downloads/coderabbitai/openai-chat-tokens/total?logo=github)](https://github.com/coderabbitai/openai-chat-tokens/releases) [![NPM Downloads](https://img.shields.io/npm/dt/%40coderabbitai/openai-chat-tokens?logo=npm)](https://www.npmjs.com/package/@coderabbitai/openai-chat-tokens)\n\nA TypeScript / JavaScript library for estimating the number of tokens an OpenAI chat completion request will use.\n\nEstimating token usage for chat completions isn't quite as easy as it sounds.\n\nFor regular chat messages, you need to consider how the messages are formatted by OpenAI when they're provided to the model, as they don't simply dump the JSON messages they receive via the API into the model.\n\nFor function calling, things are even more complex, as the OpenAPI-style function definitions get rewritten into TypeScript type definitions.\n\nThis library handles both of those cases, as well as a minor adjustment needed for handling the _results_ of function calling. [tiktoken](https://github.com/dqbd/tiktoken) is used to do the tokenization.\n\n## Usage\n\n```typescript\nimport { promptTokensEstimate } from \"openai-chat-tokens\"\n\nconst estimate = promptTokensEstimate({\n\tmessages: [\n\t\t{ role: \"system\", content: \"These aren't the droids you're looking for\" },\n\t\t{ role: \"user\", content: \"You can go about your business. Move along.\" },\n\t],\n\tfunctions: [\n\t\t{\n\t\t\tname: \"activate_hyperdrive\",\n\t\t\tdescription: \"Activate the hyperdrive\",\n\t\t\tparameters: {\n\t\t\t\ttype: \"object\",\n\t\t\t\tproperties: {\n\t\t\t\t\tdestination: { type: \"string\" },\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t],\n})\n```\n\n## Development and testing\n\nBuilt in TypeScript, tested with Vitest.\n\n```sh\npnpm install\npnpm test\n```\n\nWhen adding new test cases or debugging token count mismatches, it can be helpful to validate the estimated tokens in the tests against the live OpenAI API. To do this:\n\n1. Set up the `OPENAI_API_KEY` environment variable with a live API key\n2. Add `validate: true` to one of the test examples, or set `validateAll` to `true` in `token-counts.test.ts`, then run the tests\n\n## References\n\n1. \"Counting tokens for chat completions API calls\" in OpenAI's [\"How to count tokens with tiktoken\" notebook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb)\n2. A post about [counting function call tokens](https://community.openai.com/t/how-to-calculate-the-tokens-when-using-function-call/266573/23) on the OpenAI forum.\n","readmeFilename":"README.md"}