{"_id":"@huggingface/tokenizers","_rev":"11-fcb93da5a38a9533d144b6ea3c180e94","name":"@huggingface/tokenizers","dist-tags":{"latest":"0.2.0"},"versions":{"0.0.1":{"name":"@huggingface/tokenizers","version":"0.0.1","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.0.1","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"997ac91e86a1f4603fd0edac4c356515dd8d3a6d","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.0.1.tgz","fileCount":73,"integrity":"sha512-VtVgAoj4eEmjr3oURS5QtsOhSZHeC/qJQBcEQn6++jvVzKzdd4lYgKyms22cl1HXZRXNHi8mww+wWMJeMDdWyg==","signatures":[{"sig":"MEUCIQCU1gz/7nQKPO43BsBTDQg7niV+RS/ahSBKzoBn6AajsgIgbxA+TSS9JZhNqbDcQTTURmR2PKneh64fo60EJlBxiLE=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":290968},"main":"dist/tokenizers.min.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.min.mjs","browser":"dist/tokenizers.min.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.min.mjs","require":"./dist/tokenizers.min.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.min.mjs"},"default":"./dist/tokenizers.min.mjs"}},"gitHead":"40faad732aecc7ff3338f100fb7b482a068422c9","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write \"src/**/*.ts\" \"tests/**/*.ts\""},"_npmUser":{"name":"xenova","email":"admin@xenova.com"},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"9.8.1","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"22.17.0","_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.0.1_1762133427002_0.005974177314871376","host":"s3://npm-registry-packages-npm-production"}},"0.0.2":{"name":"@huggingface/tokenizers","version":"0.0.2","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.0.2","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"395e6e5b77b8df994ccd2487d874fb1f790bf111","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.0.2.tgz","fileCount":73,"integrity":"sha512-futFnxqdXEumr7EXCs0KQ/p0NwuBBMh9Qzg15IT42rGTf0k4fcVi0T/XwOJgUQEFzAktOOb2kH3jRN+w2YgvLQ==","signatures":[{"sig":"MEUCIEuX05C7pEeYfZmomm2nrWd/ndEdWEMpVbZHO3ZHDsn1AiEAjOtFZlp2ZQQTOyJzAcARsTotP7kZLLfoTdb+hyzitOc=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.0.2","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":290157},"main":"dist/tokenizers.min.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.min.mjs","browser":"dist/tokenizers.min.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.min.mjs","require":"./dist/tokenizers.min.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.min.mjs"},"default":"./dist/tokenizers.min.mjs"}},"gitHead":"4fa791290c0f9fb84de2b76ef030a8878f4aacd1","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"11.6.1","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"24.11.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.0.2_1762199091292_0.7654951859602439","host":"s3://npm-registry-packages-npm-production"}},"0.0.3":{"name":"@huggingface/tokenizers","version":"0.0.3","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.0.3","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"98692ba9271c9fe3f52e90b902ad47ce37474a2f","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.0.3.tgz","fileCount":73,"integrity":"sha512-9zTy3hprcoEMq6YtpYJCKYR++3W9MeHOcnJYPlPqwLPisgxA+aey437Qamlg7e5I8zzjnqW+nJGWfI9FUGcQTw==","signatures":[{"sig":"MEYCIQCNuqv3a/3h8XJYSnpW9Q71P2twiHt0NU4WqcbUKt5aYAIhAKb5Yg75RtLVIyMPifphw05GzeiyY6j0dXtMTf0rRsXI","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.0.3","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":290218},"main":"dist/tokenizers.min.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.min.mjs","browser":"dist/tokenizers.min.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.min.mjs","require":"./dist/tokenizers.min.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.min.mjs"},"default":"./dist/tokenizers.min.mjs"}},"gitHead":"098c42ff25d8bd84faa11882834e1149818fd014","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"11.6.1","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"24.11.0","dependencies":{"@huggingface/tokenizers":"^0.0.2"},"publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","tsc-alias":"^1.8.16","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.0.3_1762236972339_0.7113828041773358","host":"s3://npm-registry-packages-npm-production"}},"0.0.4":{"name":"@huggingface/tokenizers","version":"0.0.4","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.0.4","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"b58772cb15c111ba11022309689b2c4fba9a4271","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.0.4.tgz","fileCount":73,"integrity":"sha512-38DA47H077NItAgBnuhhkCnRr0TFBioI54Eo6eWVPbtL0cnPiy4ii/Xz/lCwlOak1sKbw45ErumBWF7ZM/h3Cw==","signatures":[{"sig":"MEYCIQDIy/LNLdOszZiwoblK7OkpJiI8hxedi4Hb/sdDdtQR1AIhAIqxjycDZgyGT0xlRRESv/9zO750+IJcJEi3rgGoZpbf","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.0.4","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":290153},"main":"dist/tokenizers.min.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.min.mjs","browser":"dist/tokenizers.min.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.min.mjs","require":"./dist/tokenizers.min.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.min.mjs"},"default":"./dist/tokenizers.min.mjs"}},"gitHead":"70aef0af4d480aae943aa78699a0f41faf2ef70c","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"11.6.1","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"24.11.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","tsc-alias":"^1.8.16","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.0.4_1762282909017_0.5978097632549901","host":"s3://npm-registry-packages-npm-production"}},"0.0.5":{"name":"@huggingface/tokenizers","version":"0.0.5","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.0.5","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"26d53261e592d1106a82398b4a6225040bab73c7","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.0.5.tgz","fileCount":78,"integrity":"sha512-Vlaf+fFA3v3kwbZzrfLkmyddYqbqTgFDvFLc/G0c0IMeXBFfd7sI0eNEyMmoEETkZfczb1eORW5kYu/jsgDu0A==","signatures":[{"sig":"MEQCIGyi1JbYT2hM/pCBz+MZIw5r4sbk0w0VUuyZ4vhzO48IAiBEhY14A7gImrZqyh4xRiq3OCBJVd5uVtgyEX+SwPLMxQ==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.0.5","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":293225},"main":"dist/tokenizers.min.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.min.mjs","browser":"dist/tokenizers.min.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.min.mjs","require":"./dist/tokenizers.min.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.min.mjs"},"default":"./dist/tokenizers.min.mjs"}},"gitHead":"717d6e3b45d2ea34fa52f8302d18cf59cb4953b3","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"11.6.1","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"24.11.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","tsc-alias":"^1.8.16","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.0.5_1762482520807_0.5314788581264154","host":"s3://npm-registry-packages-npm-production"}},"0.0.6":{"name":"@huggingface/tokenizers","version":"0.0.6","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.0.6","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"e24837731f287a8c39d6ecf8cc4cd7aa34e8d99b","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.0.6.tgz","fileCount":78,"integrity":"sha512-bnn3VWPiOwk613kFGHCCTc2TOVB8QCCsHTJGVlfyV5tQIDCn83dWrFkLB670Pr2xVs1e5ziSwlysX+tga+R3nQ==","signatures":[{"sig":"MEYCIQD9ycOt8+v/VWw0oEVSSc0ctKZAvPecY2fggCpfwwV1qQIhANNk1erGK5DRrGJ6AzR60JMkxf5MU+XJEpx0ohxTAcLx","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.0.6","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":292391},"main":"dist/tokenizers.min.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.min.mjs","browser":"dist/tokenizers.min.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.min.mjs","require":"./dist/tokenizers.min.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.min.mjs"},"default":"./dist/tokenizers.min.mjs"}},"gitHead":"addf69708f783bcaadec016d39fe498ddba3ee7b","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"11.6.1","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"24.11.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","tsc-alias":"^1.8.16","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.0.6_1763090401211_0.6441309719666413","host":"s3://npm-registry-packages-npm-production"}},"0.1.0":{"name":"@huggingface/tokenizers","version":"0.1.0","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.1.0","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"c7e4ad48ac031135373df890a970d238d13185ba","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.1.0.tgz","fileCount":79,"integrity":"sha512-gnOWcxTCyHG6ZHWbMay0H7bQTF6Xw5dqKg9juTy037gaGqNbZilEyE8/8VSAHBhvWFJKTFujetn8XbD+d2pzQQ==","signatures":[{"sig":"MEUCIFRI48D3pX1FyGDo2ZYdoO84hmFQ6744D3cYkrkIur/5AiEAyybYi4qy0KtiGu1uhtHqmxvlsj5M0UDWBSfPXEeHzEw=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.1.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":297645},"main":"dist/tokenizers.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.mjs","browser":"dist/tokenizers.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.mjs","require":"./dist/tokenizers.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.mjs"},"default":"./dist/tokenizers.mjs"}},"gitHead":"3ae4220676b915a2e0fc2d30d59f1d05cbb5507d","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"11.6.2","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"24.12.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","tsc-alias":"^1.8.16","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.1.0_1768935286707_0.25877080687343135","host":"s3://npm-registry-packages-npm-production"}},"0.1.1":{"name":"@huggingface/tokenizers","version":"0.1.1","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.1.1","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"a62f8ecfbbce3c95506e69d0c42ab4ea92a34742","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.1.1.tgz","fileCount":79,"integrity":"sha512-hNVdOq4kAONjz9FqL6agX1vfpJ57Z9cUDQSVSqRMLT1x31y84XbOXDw2WdxCUiseizA+LEHehwwLLkGsyu8iXQ==","signatures":[{"sig":"MEYCIQDd/TLsAej2J88V4KYVvxLmBRYPbKSb0OnyqOd11LSNMwIhAONBmkOHgxnB5PDIO545ymm8p7VPuVAhGsGSpiB02cxY","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.1.1","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":299469},"main":"dist/tokenizers.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.mjs","browser":"dist/tokenizers.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.mjs","require":"./dist/tokenizers.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.mjs"},"default":"./dist/tokenizers.mjs"}},"gitHead":"7969b0f987974611c8d643e014871dfc49693da6","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"11.6.2","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"24.13.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","tsc-alias":"^1.8.16","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.1.1_1769736017380_0.19868625758590586","host":"s3://npm-registry-packages-npm-production"}},"0.1.2":{"name":"@huggingface/tokenizers","version":"0.1.2","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.1.2","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"aaebb9d30e2ef39f21074563c3a9990d1f9db6f8","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.1.2.tgz","fileCount":79,"integrity":"sha512-6sPNgS2pHowVBXFBuaKKUP0e9+5K2vPyAlmi2bc5PofMkMyi6gttJsq5FKcIDpAjJIONV6Pom9DTjqKazsrdAw==","signatures":[{"sig":"MEUCIQDTFCMSORnVOPuXY2k/7jSjHu46lwXaSUWsuAvOptm1PgIgDmiguIjqXoPVh94h69i3D6XaaBzQpLEaQw1w/oFQxo8=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.1.2","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":300835},"main":"dist/tokenizers.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.mjs","browser":"dist/tokenizers.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.mjs","require":"./dist/tokenizers.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.mjs"},"default":"./dist/tokenizers.mjs"}},"gitHead":"488865017c65e9b84b84ede4d7fa89f348dc0fea","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"11.6.2","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"24.13.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","tsc-alias":"^1.8.16","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.1.2_1771707224963_0.30616602112319136","host":"s3://npm-registry-packages-npm-production"}},"0.1.3":{"name":"@huggingface/tokenizers","version":"0.1.3","author":{"name":"Hugging Face"},"license":"Apache-2.0","_id":"@huggingface/tokenizers@0.1.3","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierrot007@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"homepage":"https://github.com/huggingface/tokenizers.js#readme","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"d1bb2b25375e550c826e4c7151d5f764a14a6a69","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.1.3.tgz","fileCount":79,"integrity":"sha512-8rF/RRT10u+kn7YuUbUg0OF30K8rjTc78aHpxT+qJ1uWSqxT1MHi8+9ltwYfkFYJzT/oS+qw3JVfHtNMGAdqyA==","signatures":[{"sig":"MEUCIBgOysF7YioaJ9gWAcdwJbGh3Esy1dNfnYvGpacHIUvPAiEAj6fsdcNhAs36EdTvmk/UCkVbXZL+qZXWWTFo1IoIYeM=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.1.3","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":301035},"main":"dist/tokenizers.mjs","type":"module","types":"types/index.d.ts","module":"dist/tokenizers.mjs","browser":"dist/tokenizers.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.mjs","require":"./dist/tokenizers.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.mjs"},"default":"./dist/tokenizers.mjs"}},"gitHead":"d6fc380d3d4efec25eb1a132bb0cf4f32d618b6c","scripts":{"dev":"npm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"npm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"_npmVersion":"11.9.0","description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"_nodeVersion":"24.14.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","tsc-alias":"^1.8.16","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.1.3_1773865805571_0.8886577422549844","host":"s3://npm-registry-packages-npm-production"}},"0.2.0":{"_id":"@huggingface/tokenizers@0.2.0","bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"dist":{"shasum":"3913cd936b6e5cbc24e20d0c81dc3ec0f97b8eb0","tarball":"https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.2.0.tgz","fileCount":80,"integrity":"sha512-LidMHe1FpcSYH4vcSjXooda34pC0M8m1gmDOv7SQi6Iv+ib1zX7J3sHBCC9/3FPCkn2k1b55FneVqMRxrr99pg==","signatures":[{"sig":"MEYCIQCAIVics1gvelg33KGPhEGuHMm0fUOQJKagU3n1f5V4BAIhANjCcXTatPpAuURm3RgiC+AJxT97TpSdjTv5pF/anLA0","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"},{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIHmxh9QerKZ+7uCLE/5ktcDOX/qaGxSPO0Zi81lxITzZAiAPKE5UGWwClEG1v7VNff9LI+eVOJovwTalS4qgRb7vyg=="}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@huggingface%2ftokenizers@0.2.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":360962},"main":"dist/tokenizers.mjs","name":"@huggingface/tokenizers","type":"module","types":"types/index.d.ts","author":"Hugging Face","module":"dist/tokenizers.mjs","browser":"dist/tokenizers.mjs","exports":{".":{"node":{"import":"./dist/tokenizers.mjs","require":"./dist/tokenizers.cjs"},"types":"./types/index.d.ts","browser":{"import":"./dist/tokenizers.mjs"},"default":"./dist/tokenizers.mjs"}},"license":"Apache-2.0","scripts":{"dev":"pnpm run clean && node scripts/dev.mjs","lint":"eslint src --ext .ts,.tsx","test":"node --experimental-vm-modules --expose-gc --max-old-space-size=4096 node_modules/jest/bin/jest.js --verbose --logHeapUsage --maxWorkers=10%","build":"pnpm run clean && node scripts/build.mjs","clean":"rimraf dist types","format":"prettier --write .","format:check":"prettier --check ."},"version":"0.2.0","_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:d64155bf-4f96-4e21-9889-dd37695cec44"}},"homepage":"https://github.com/huggingface/tokenizers.js#readme","repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","directories":{},"maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierric.gh@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"_nodeVersion":"24.20.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^30.2.0","dotenv":"^17.2.3","eslint":"^9.37.0","rimraf":"^6.0.1","terser":"^5.40.0","esbuild":"^0.25.5","ts-jest":"^29.4.4","prettier":"^3.5.3","tsc-alias":"^1.8.16","typescript":"^5.4.0","@types/jest":"^30.0.0","@types/node":"^24.9.2","jest-environment-node":"^30.2.0","@typescript-eslint/parser":"^8.46.0","@typescript-eslint/eslint-plugin":"^8.46.0"},"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokenizers_0.2.0_1788751797476_0.7015223012448437"}}},"time":{"created":"2025-11-03T01:30:26.893Z","modified":"2026-09-07T03:29:57.860Z","0.0.1":"2025-11-03T01:30:27.189Z","0.0.2":"2025-11-03T19:44:51.561Z","0.0.3":"2025-11-04T06:16:12.547Z","0.0.4":"2025-11-04T19:01:49.234Z","0.0.5":"2025-11-07T02:28:41.025Z","0.0.6":"2025-11-14T03:20:01.423Z","0.1.0":"2026-01-20T18:54:46.911Z","0.1.1":"2026-01-30T01:20:17.529Z","0.1.2":"2026-02-21T20:53:45.124Z","0.1.3":"2026-03-18T20:30:05.853Z","0.2.0":"2026-09-07T03:29:57.557Z"},"bugs":{"url":"https://github.com/huggingface/tokenizers.js/issues"},"author":"Hugging Face","license":"Apache-2.0","homepage":"https://github.com/huggingface/tokenizers.js#readme","repository":{"url":"git+https://github.com/huggingface/tokenizers.js.git","type":"git"},"description":"🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers","maintainers":[{"name":"julien-c","email":"chaumond@gmail.com"},{"name":"pierric","email":"pierric.gh@gmail.com"},{"name":"coyotte508","email":"coyotte508@protonmail.com"},{"name":"xenova","email":"admin@xenova.com"},{"name":"gary149","email":"v@huggingface.co"}],"readme":"","readmeFilename":""}