{"_id":"@cyberlangke/tokkit-gsai-ml","name":"@cyberlangke/tokkit-gsai-ml","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-gsai-ml","version":"1.11.0","description":"GSAI-ML tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"SEE LICENSE IN LICENSE","gitHead":"871debd8e452cb6c1e9c884436c513987e78b173","_id":"@cyberlangke/tokkit-gsai-ml@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-Aa43b9Q0aXnd9c1EGfYZWSoXZen6bem2FGr3/q5D5ZfRdMxzBNeFbakMZ0q/Q2i+F7tXy+fxUZupj/DSHGNKGw==","shasum":"51ee7f8a276cc7eed9e18480c35a5ab438e4e21a","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-gsai-ml/-/tokkit-gsai-ml-1.11.0.tgz","fileCount":14,"unpackedSize":8173527,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIBj1akNEzcpvdnYtWC0d8VsMCSdwXJhUU/vBGpdEi782AiAJeqiYSd7uN4JtQAp553ELuXDo+6guVXsjWslJbqdp+A=="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-gsai-ml_1.11.0_1774942079424_0.3173594756310236"},"_hasShrinkwrap":false}},"time":{"created":"2026-03-31T07:27:59.323Z","1.11.0":"2026-03-31T07:27:59.776Z","modified":"2026-03-31T07:28:00.015Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"GSAI-ML tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"SEE LICENSE IN LICENSE","readme":"# @cyberlangke/tokkit-gsai-ml\n\nGSAI-ML 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `llada`\n  - 覆盖 `GSAI-ML/LLaDA-8B-Instruct`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `GSAI-ML/LLaDA-1.5`\n- `llada-base`\n  - 覆盖 `GSAI-ML/LLaDA-8B-Base`\n- `refusion`\n  - 覆盖 `GSAI-ML/ReFusion`\n\n说明：\n\n- `LLaDA-8B-Instruct` 与 `LLaDA-1.5` 的导出 `tokenizer.json` SHA256 一致，因此收口到同一个 `llada` family。\n- `LLaDA-8B-Base` 的 `tokenizer.json` SHA256 与 `llada` 不同，因此保留独立的 `llada-base` family。\n- `ReFusion` 官方仓库不直接公开 `tokenizer.json`，当前快照来自官方 `AutoTokenizer.from_pretrained(...).save_pretrained()` 基于仓库内 `vocab.json` / `merges.txt` / `added_tokens.json` 导出的标准 `tokenizer.json`。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-gsai-ml\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-gsai-ml\"\n\nconst llada = await getTokenizer(\"llada\")\nconst refusion = await getTokenizer(\"GSAI-ML/ReFusion\")\n\nconsole.log(llada.encode(\"Hello, GSAI!\"))\nconsole.log(refusion.encode(\"Hello, GSAI!\"))\n```\n","readmeFilename":"README.md","_rev":"1-5eb6c54afacbf7db593a40682c95eea1"}