{"_id":"@cyberlangke/tokkit-moonshotai","name":"@cyberlangke/tokkit-moonshotai","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-moonshotai","version":"1.11.0","description":"moonshotai tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"SEE LICENSE IN LICENSE","gitHead":"6d7b164cdcb511e7f5815e7278708c780b05d938","_id":"@cyberlangke/tokkit-moonshotai@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-RvoII5TjAvqmNejY4wEUzhld6ISBlgLz+fr6+3jl29Dt6+DcUcFeQCltzenL55wpZzw+9BYHn2O87tMYe1DnUA==","shasum":"1fd470d0b770886f7dd7ac325b86aa6206fb4ad6","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-moonshotai/-/tokkit-moonshotai-1.11.0.tgz","fileCount":17,"unpackedSize":17599922,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIH+EZ9IIXOE8NjyCNsMHkhDo6UEhp4JYSbcqy1rORJYlAiBTmOiehdJt3W5nGZ9B0Ik6QcJ7xHpETMBpao3c89/rbg=="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-moonshotai_1.11.0_1775230238333_0.7695979850513106"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-03T15:30:38.244Z","1.11.0":"2026-04-03T15:30:38.806Z","modified":"2026-04-03T15:30:39.074Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"moonshotai tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"SEE LICENSE IN LICENSE","readme":"# @cyberlangke/tokkit-moonshotai\n\nmoonshotai 官方文本 tokenizer 的 tokkit 子包。\n\n这是一个独立特殊协议子包，不包含在 `@cyberlangke/tokkit` 总包里。\n\n当前内置 family：\n\n- `kimi-k2`\n  - 覆盖 `moonshotai/Kimi-K2-Base`\n  - 覆盖 `moonshotai/Kimi-K2-Instruct`\n  - 覆盖 `moonshotai/Kimi-K2-Instruct-0905`\n  - 也聚合当前确认在 BPE / added token 行为上完全一致的：\n    - `moonshotai/Kimi-Linear-48B-A3B-Base`\n    - `moonshotai/Kimi-Linear-48B-A3B-Instruct`\n- `kimi-k2-thinking`\n  - 覆盖 `moonshotai/Kimi-K2-Thinking`\n  - 该线额外引入了 `<think>` / `</think>` added tokens，因此与普通 `kimi-k2` 分开\n- `moonlight`\n  - 覆盖 `moonshotai/Moonlight-16B-A3B`\n  - 也聚合当前确认完全复用同一 tokenizer 行为的 `moonshotai/Moonlight-16B-A3B-Instruct`\n  - 该线的 added tokens 少于 `kimi-k2`，因此单独保留 family\n- `kimi-dev`\n  - 覆盖 `moonshotai/Kimi-Dev-72B`\n\n说明：\n\n- `Kimi-K2-*`、`Moonlight-*`、`Kimi-Linear-*` 虽然共享同一份 `tiktoken.model`，但 `tokenizer_config.json` 里的 added token 集不同，不能粗暴压成一个 family。\n- `kimi-k2` 的模型之间当前只观察到 `chat_template` 差异；本项目不支持 chat template，因此仍可收口到同一个 family。\n- `moonshotai/Kimi-Dev-72B` 当前是独立 tokenizer，不命中现有仓库 family。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-moonshotai\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-moonshotai\"\n\nconst shared = await getTokenizer(\"moonshotai/Kimi-K2-Instruct-0905\")\nconst thinking = await getTokenizer(\"kimi-k2-thinking\")\n\nconsole.log(shared === thinking) // false\n```\n","readmeFilename":"README.md","_rev":"1-d894e038d1191d94ced346a9f63901bf"}