{"_id":"@cyberlangke/tokkit-llm360","name":"@cyberlangke/tokkit-llm360","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-llm360","version":"1.11.0","description":"LLM360 tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"npm run build --workspace @cyberlangke/tokkit-core && npm run build --workspace @cyberlangke/tokkit-snowflake && npm run build --workspace @cyberlangke/tokkit-xiaomi-mimo && tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0","@cyberlangke/tokkit-snowflake":"1.11.0","@cyberlangke/tokkit-xiaomi-mimo":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"6d7b164cdcb511e7f5815e7278708c780b05d938","_id":"@cyberlangke/tokkit-llm360@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-MJwAxbMeE9avvx41/ezP8ZUvQdPQ5Ule61PdbOt1hvnHoHqkn0+bIFPnV4y7Y6ci5On9JN6ch0A0h4Q2t0VEJQ==","shasum":"c57215bafccbf58de8a08d232764abb15b8ebc77","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-llm360/-/tokkit-llm360-1.11.0.tgz","fileCount":16,"unpackedSize":7957267,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIBQb4iJpPiChNzSwqzQnoIKOAlPygfujNL2FU7qSgJViAiAj9jEGjP91ovwd3XOeb2Io0iZb0NbksAN1mgZx+bM3NA=="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-llm360_1.11.0_1775230219821_0.9171677755310486"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-03T15:30:19.705Z","1.11.0":"2026-04-03T15:30:20.188Z","modified":"2026-04-03T15:30:20.401Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"LLM360 tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-llm360\n\nLLM360 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `snowflake-arctic-base`\n  - 复用现有 canonical family\n  - 覆盖 `LLM360/Amber`\n  - 也聚合 `LLM360/AmberChat`\n- `crystal`\n  - 覆盖 `LLM360/Crystal`\n  - 也聚合 `LLM360/CrystalChat`\n- `k2`\n  - 覆盖 `LLM360/K2`\n- `k2-chat`\n  - 覆盖 `LLM360/K2-Chat`\n- `mimo-7b-rl-0530`\n  - 复用现有 canonical family\n  - 覆盖 `LLM360/K2-Think`\n- `k2-think-v2`\n  - 覆盖 `LLM360/K2-Think-V2`\n\n当前不纳入：\n\n- `LLM360/AmberSafe`\n\n说明：\n\n- 当前纳入范围只包含 `LLM360` 官方组织下公开可下载、`apache-2.0`、非 gated、且落在纯文本 BPE 范围内的文本主线。\n- `Amber` / `AmberChat` 的 `tokenizer.json` SHA256 与 `Snowflake/snowflake-arctic-base` 一致，因此复用 `snowflake-arctic-base`。\n- `K2-Think` 的 `tokenizer.json` SHA256 与 `XiaomiMiMo/MiMo-7B-RL-0530` 一致，因此复用 `mimo-7b-rl-0530`。\n- `Crystal` / `CrystalChat`、`K2`、`K2-Chat`、`K2-Think-V2` 当前都需要独立 canonical family。\n- `AmberSafe` 当前没有 `tokenizer.json`，因此不纳入这一批。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-llm360\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-llm360\"\n\nconst amber = await getTokenizer(\"LLM360/Amber\")\nconst crystal = await getTokenizer(\"LLM360/Crystal\")\nconst k2Think = await getTokenizer(\"LLM360/K2-Think\")\n\nconsole.log(amber !== crystal)\nconsole.log(k2Think)\n```\n","readmeFilename":"README.md","_rev":"1-43a5a1f20ed60a31d789cc0142420020"}