{"_id":"@anush008/tokenizers","_rev":"8-f8d3f5b799465d662ab5be6342e03625","name":"@anush008/tokenizers","dist-tags":{"next":"0.2.0","latest":"0.6.0"},"versions":{"0.0.0":{"name":"@anush008/tokenizers","version":"0.0.0","license":"MIT","_id":"@anush008/tokenizers@0.0.0","maintainers":[{"name":"anush008","email":"anushshetty90@gmail.com"}],"homepage":"https://github.com/Anush008/tokenizers#readme","bugs":{"url":"https://github.com/Anush008/tokenizers/issues"},"ava":{"timeout":"3m"},"dist":{"shasum":"7bcefa55755acba9a285d95d12b25b57427dda67","tarball":"https://registry.npmjs.org/@anush008/tokenizers/-/tokenizers-0.0.0.tgz","fileCount":20,"integrity":"sha512-IQD9wkVReKAhsEAbDjh/0KrBGTEXelqZLpOBRDaIRvlzZ9sjmUP+gKbpvzyJnei2JHQiE8JAgj7YcNloINbGBw==","signatures":[{"sig":"MEUCIQD7KAxu9l03HFpHlSdFT3GtQbbjAxkLVUljstVcC1K08QIgKDpuwhX8cc4VLfP0rbsto0n5g4eLxQ0kZsnBAOcADko=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@anush008%2ftokenizers@0.0.0","provenance":{"predicateType":"https://slsa.dev/provenance/v0.2"}},"unpackedSize":80251},"main":"index.js","napi":{"name":"tokenizers","triples":{"defaults":false,"additional":["x86_64-pc-windows-msvc","x86_64-unknown-linux-gnu","universal-apple-darwin"]}},"types":"index.d.ts","engines":{"node":">= 10"},"gitHead":"6257e92b2d77bec979e92533b8aa0ba2466d3c76","scripts":{"test":"ava","build":"napi build --platform --release","version":"napi version","artifacts":"napi artifacts","universal":"napi universal","build:debug":"napi build --platform","prepublishOnly":"napi prepublish -t npm"},"_npmUser":{"name":"anush008","email":"anushshetty90@gmail.com"},"deprecated":"This package has been archived. Please use the official and updated \"huggingface/tokenizers\" package.","repository":{"url":"git+https://github.com/Anush008/tokenizers.git","type":"git"},"_npmVersion":"9.6.7","description":"Multi-arch builds of HuggingFace tokenizers","directories":{},"_nodeVersion":"18.17.1","dependencies":{"@anush008/tokenizers-linux-x64-gnu":"0.0.0","@anush008/tokenizers-win32-x64-msvc":"0.0.0","@anush008/tokenizers-darwin-universal":"0.0.0"},"_hasShrinkwrap":false,"packageManager":"yarn@3.6.3","devDependencies":{"ava":"^5.1.1","@napi-rs/cli":"^2.16.3"},"optionalDependencies":{"@anush008/tokenizers-linux-x64-gnu":"0.0.0","@anush008/tokenizers-win32-x64-msvc":"0.0.0","@anush008/tokenizers-darwin-universal":"0.0.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.0.0_1694875319215_0.07177693662493301","host":"s3://npm-registry-packages"}},"0.2.0":{"name":"@anush008/tokenizers","version":"0.2.0","license":"MIT","_id":"@anush008/tokenizers@0.2.0","maintainers":[{"name":"anush008","email":"anushshetty90@gmail.com"}],"homepage":"https://github.com/Anush008/tokenizers#readme","bugs":{"url":"https://github.com/Anush008/tokenizers/issues"},"ava":{"timeout":"3m"},"dist":{"shasum":"09d6f658fa8ffc821d3b73f1b34e6f748aa88a83","tarball":"https://registry.npmjs.org/@anush008/tokenizers/-/tokenizers-0.2.0.tgz","fileCount":22,"integrity":"sha512-XvLLaPOhXZxxtqB7gVgd/eYfuAr+ewAl7SzsFEO9fXD2BoMKesF+puqKUlz8I6l4ZRmeI53xv9axQ9GXQ1/wCw==","signatures":[{"sig":"MEQCIDJyClohR82w1W3RzcAJjlYEGo1ed87nRvw2cI38pG9bAiAIa+zPpw2q0CA3chyCsJgq8b+23HM3hvn8VQOPp23BvA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@anush008%2ftokenizers@0.2.0","provenance":{"predicateType":"https://slsa.dev/provenance/v0.2"}},"unpackedSize":82769},"main":"index.js","napi":{"name":"tokenizers","triples":{"defaults":false,"additional":["x86_64-pc-windows-msvc","x86_64-unknown-linux-gnu","universal-apple-darwin"]}},"types":"index.d.ts","engines":{"node":">= 10"},"gitHead":"acc7067a415e1ec9d3b439ccf6ad3bd004db83c0","scripts":{"test":"ava","build":"napi build --platform --release","version":"napi version","artifacts":"napi artifacts","universal":"napi universal","build:debug":"napi build --platform","prepublishOnly":"napi prepublish -t npm"},"_npmUser":{"name":"anush008","email":"anushshetty90@gmail.com"},"deprecated":"This package has been archived. Please use the official and updated \"huggingface/tokenizers\" package.","repository":{"url":"git+https://github.com/Anush008/tokenizers.git","type":"git"},"_npmVersion":"9.6.7","description":"Multi-arch builds of HuggingFace tokenizers","directories":{},"_nodeVersion":"18.17.1","dependencies":{"@anush008/tokenizers-linux-x64-gnu":"0.2.0","@anush008/tokenizers-win32-x64-msvc":"0.2.0","@anush008/tokenizers-darwin-universal":"0.2.0"},"_hasShrinkwrap":false,"packageManager":"yarn@3.6.3","readmeFilename":"README.md","devDependencies":{"ava":"^5.1.1","@napi-rs/cli":"^2.16.3"},"optionalDependencies":{"@anush008/tokenizers-linux-x64-gnu":"0.2.0","@anush008/tokenizers-win32-x64-msvc":"0.2.0","@anush008/tokenizers-darwin-universal":"0.2.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.2.0_1694884037687_0.8176896945498098","host":"s3://npm-registry-packages"}},"0.5.0":{"name":"@anush008/tokenizers","version":"0.5.0","license":"MIT","_id":"@anush008/tokenizers@0.5.0","maintainers":[{"name":"anush008","email":"mail@anush.sh"}],"homepage":"https://github.com/Anush008/tokenizers#readme","bugs":{"url":"https://github.com/Anush008/tokenizers/issues"},"ava":{"timeout":"3m"},"dist":{"shasum":"b3bc2d841feb4dae617cf4092db6be4b187fc4a2","tarball":"https://registry.npmjs.org/@anush008/tokenizers/-/tokenizers-0.5.0.tgz","fileCount":22,"integrity":"sha512-FoXUACE5HcYKapGsWYl37pMCtsJxv0WHq12MTxEiMRfMygN3XlyrtZkN/eqjJ6PpId8GnSx0gksELMWYoTcwoQ==","signatures":[{"sig":"MEUCIQDkac8GkE6b5/jIP/ElKiZWVVsSyRNdlJjaDASA5/bt+AIgawdxW6lrfU9Hug7cVcYmqn3mlvywMS1IiK5A+Yo41W4=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@anush008%2ftokenizers@0.5.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":83391},"main":"index.js","napi":{"name":"tokenizers","triples":{"defaults":false,"additional":["x86_64-pc-windows-msvc","x86_64-unknown-linux-gnu","x86_64-unknown-linux-musl","aarch64-unknown-linux-gnu","aarch64-unknown-linux-musl","universal-apple-darwin"]}},"types":"index.d.ts","engines":{"node":">= 10"},"gitHead":"3b8663b947f6193ae9775401e9046d05c81d2dc5","scripts":{"test":"ava","build":"napi build --platform --release","version":"napi version","artifacts":"napi artifacts","universal":"napi universal","build:debug":"napi build --platform","prepublishOnly":"napi prepublish -t npm"},"_npmUser":{"name":"anush008","email":"mail@anush.sh"},"deprecated":"This package has been archived. Please use the official and updated \"huggingface/tokenizers\" package.","repository":{"url":"git+https://github.com/Anush008/tokenizers.git","type":"git"},"_npmVersion":"10.8.2","description":"Multi-arch builds of HuggingFace tokenizers","directories":{},"_nodeVersion":"18.20.8","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"packageManager":"yarn@3.6.3","devDependencies":{"ava":"^5.1.1","@napi-rs/cli":"^2.16.3"},"optionalDependencies":{"@anush008/tokenizers-linux-x64-gnu":"0.5.0","@anush008/tokenizers-linux-x64-musl":"0.5.0","@anush008/tokenizers-win32-x64-msvc":"0.5.0","@anush008/tokenizers-linux-arm64-gnu":"0.5.0","@anush008/tokenizers-darwin-universal":"0.5.0","@anush008/tokenizers-linux-arm64-musl":"0.5.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.5.0_1773469694703_0.15066655531799755","host":"s3://npm-registry-packages-npm-production"}},"0.6.0":{"name":"@anush008/tokenizers","version":"0.6.0","license":"MIT","_id":"@anush008/tokenizers@0.6.0","maintainers":[{"name":"anush008","email":"mail@anush.sh"}],"homepage":"https://github.com/Anush008/tokenizers#readme","bugs":{"url":"https://github.com/Anush008/tokenizers/issues"},"ava":{"timeout":"3m"},"dist":{"shasum":"eefab74f6de1e24d8b74faa5c38096d93679b595","tarball":"https://registry.npmjs.org/@anush008/tokenizers/-/tokenizers-0.6.0.tgz","fileCount":22,"integrity":"sha512-vbKFjlFM15Ljm6HMh2FSiyPq4236xHoe31XMveTMdwqowHf1/UCwySPT331wl/wRWpdHWdQ3eU5ENl5XbXnPQQ==","signatures":[{"sig":"MEQCIG7k/9qRlAc2gHfcAkA2Yc9sAGuIAn7wI5yvwCqShUHFAiBlTPVJ6IKBz9ErJjZXc6ZUR6Etb7Zj8FVKTgMwPTmUqQ==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@anush008%2ftokenizers@0.6.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":84629},"main":"index.js","napi":{"name":"tokenizers","triples":{"defaults":false,"additional":["x86_64-pc-windows-msvc","x86_64-unknown-linux-gnu","x86_64-unknown-linux-musl","aarch64-unknown-linux-gnu","aarch64-unknown-linux-musl","universal-apple-darwin"]}},"types":"index.d.ts","engines":{"node":">= 10"},"gitHead":"7e02f446be5a1d2443db82aa41c942f91209009e","scripts":{"test":"ava","build":"napi build --platform --release","version":"napi version","artifacts":"napi artifacts","universal":"napi universal","build:debug":"napi build --platform","prepublishOnly":"napi prepublish -t npm"},"_npmUser":{"name":"anush008","email":"mail@anush.sh"},"deprecated":"This package has been archived. Please use the official and updated \"huggingface/tokenizers\" package.","repository":{"url":"git+https://github.com/Anush008/tokenizers.git","type":"git"},"_npmVersion":"10.8.2","description":"Multi-arch builds of HuggingFace tokenizers","directories":{},"_nodeVersion":"18.20.8","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"packageManager":"yarn@3.6.3","devDependencies":{"ava":"^5.1.1","@napi-rs/cli":"^2.16.3"},"optionalDependencies":{"@anush008/tokenizers-linux-x64-gnu":"0.6.0","@anush008/tokenizers-linux-x64-musl":"0.6.0","@anush008/tokenizers-win32-x64-msvc":"0.6.0","@anush008/tokenizers-linux-arm64-gnu":"0.6.0","@anush008/tokenizers-darwin-universal":"0.6.0","@anush008/tokenizers-linux-arm64-musl":"0.6.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.6.0_1773905540093_0.34275462909145205","host":"s3://npm-registry-packages-npm-production"}}},"time":{"created":"2023-09-16T14:41:59.079Z","modified":"2026-09-09T04:16:42.195Z","0.0.0":"2023-09-16T14:41:59.510Z","0.2.0":"2023-09-16T17:07:17.896Z","0.5.0":"2026-03-14T06:28:14.840Z","0.6.0":"2026-03-19T07:32:20.221Z"},"bugs":{"url":"https://github.com/Anush008/tokenizers/issues"},"license":"MIT","homepage":"https://github.com/Anush008/tokenizers#readme","repository":{"url":"git+https://github.com/Anush008/tokenizers.git","type":"git"},"description":"Multi-arch builds of HuggingFace tokenizers","maintainers":[{"name":"anush008","email":"mail@anush.sh"}],"readme":"# @anush008/tokenizers\n\nThe official Node bindings are in a jinx with limited support for Node versions and architectures. This package offers multi-arch bindings for [@huggingface/tokenizers](https://github.com/huggingface/tokenizers) with Node v20.x supported. \n\n## Supports:\n> * Windows x86_64\n> * Linux x86_64 (glibc, musl/Alpine)\n> * Linux aarch64 (glibc, musl/Alpine)\n> * macOS x86_64\n> * macOS aarch64 (Apple Silicon)\n> * macOS Universal binary\n\n## Installation\n\n```bash\nnpm install @anush008/tokenizers\n```\n\n## Features\n\n - Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3\n   most common BPE versions).\n - Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes\n   less than 20 seconds to tokenize a GB of text on a server's CPU.\n - Easy to use, but also extremely versatile.\n - Designed for research and production.\n - Normalization comes with alignments tracking. It's always possible to get the part of the\n   original sentence that corresponds to a given token.\n - Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.\n\n\n## Basic example\n\n```ts\nimport { Tokenizer } from \"@anush008/tokenizers\";\n\nconst tokenizer = await Tokenizer.fromFile(\"tokenizer.json\");\nconst wpEncoded = await tokenizer.encode(\"Who is John?\");\n\nconsole.log(wpEncoded.getLength());\nconsole.log(wpEncoded.getTokens());\nconsole.log(wpEncoded.getIds());\nconsole.log(wpEncoded.getAttentionMask());\nconsole.log(wpEncoded.getOffsets());\nconsole.log(wpEncoded.getOverflowing());\nconsole.log(wpEncoded.getSpecialTokensMask());\nconsole.log(wpEncoded.getTypeIds());\nconsole.log(wpEncoded.getWordIds());\n```\n\n## License\n\n[MIT](LICENSE)\n","readmeFilename":"README.md"}