{"_id":"@cyberlangke/tokkit-distilbert","name":"@cyberlangke/tokkit-distilbert","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-distilbert","version":"1.11.0","description":"DistilBERT tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"6c6190e95ac083fb23c0421a4f091f9166e309f7","_id":"@cyberlangke/tokkit-distilbert@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-jakOv20JWrRkURyTXu4HUCU7z3EQKvcrgt8pfuWHH/BRJxrdFBrnA/f2B/hr0f7BbZolLrZPPK6S5qnNMEUtJQ==","shasum":"de64139ceacacef918f09af9abb4b6f1b01dab4b","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-distilbert/-/tokkit-distilbert-1.11.0.tgz","fileCount":12,"unpackedSize":1840177,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQDCasVgSYkgR0CGo8qbye+xyMUMQtuo+/aVhM5PaMcxzAIhAIOWAQ1LXf/xViPChAY/w7c2xfitw1AquN9Y5rMMCeMd"}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-distilbert_1.11.0_1775017009489_0.34753245900209073"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-01T04:16:49.329Z","1.11.0":"2026-04-01T04:16:49.894Z","modified":"2026-04-01T04:16:50.206Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"DistilBERT tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-distilbert\n\nDistilBERT 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `distilgpt2`\n  - 覆盖 `distilbert/distilgpt2`\n\n说明：\n\n- 当前 `distilbert` 官方 `text-generation` 主线只有 `distilgpt2`。\n- 官方仓库直接公开 `tokenizer.json`、`tokenizer_config.json`、`vocab.json`、`merges.txt`。\n- 当前该模型的 tokenizer 已作为一个独立 family 收口到这个子包。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-distilbert\n```\n\n```ts\nimport { getEncoding } from \"@cyberlangke/tokkit-distilbert\"\n\nconst tokenizer = await getEncoding(\"distilbert/distilgpt2\")\n\nconsole.log(tokenizer.encode(\"Hello, distilgpt2!\"))\n```\n","readmeFilename":"README.md","_rev":"1-aa858416b45403bd021fea578767f689"}