{"_id":"@cyberlangke/tokkit-utter-project","name":"@cyberlangke/tokkit-utter-project","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-utter-project","version":"1.11.0","description":"utter-project tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"6d7b164cdcb511e7f5815e7278708c780b05d938","_id":"@cyberlangke/tokkit-utter-project@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-rM2PvQk7VN3IpeZpNyx9/IrT3LkO9yma8m7aD68DCubp0NSrLuEnUD+iuxflpHPzX6NiviINqxEDueB/FiTC1A==","shasum":"6c135c4f7feb7d240b0377608acaaba90093a878","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-utter-project/-/tokkit-utter-project-1.11.0.tgz","fileCount":16,"unpackedSize":15624370,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIAlCUNgwQKjp/JbcmtN2YtHMvV6AX/85GShtFraO/2AUAiEAkVmlKz1eIZZnzjrI42V6dRvu2vg7AILbWDTbk9y6Yzo="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-utter-project_1.11.0_1775230326653_0.22123181138947823"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-03T15:32:06.525Z","1.11.0":"2026-04-03T15:32:07.119Z","modified":"2026-04-03T15:32:07.320Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"utter-project tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-utter-project\n\nutter-project 官方 EuroLLM / EuroMoE 文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `eurollm-1.7b`\n  - 覆盖 `utter-project/EuroLLM-1.7B`\n- `eurollm-1.7b-instruct`\n  - 覆盖 `utter-project/EuroLLM-1.7B-Instruct`\n- `eurollm-2512`\n  - 覆盖 `utter-project/EuroLLM-9B-2512`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `utter-project/EuroLLM-22B-2512`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `utter-project/EuroMoE-2.6B-A0.6B-2512`\n- `eurollm-2512-instruct`\n  - 覆盖 `utter-project/EuroLLM-9B-Instruct-2512`\n  - 也聚合当前在仓库行为边界内可复用同一 tokenizer 的 `utter-project/EuroLLM-22B-Instruct-2512`\n  - 也聚合当前在仓库行为边界内可复用同一 tokenizer 的 `utter-project/EuroMoE-2.6B-A0.6B-Instruct-2512`\n\n说明：\n\n- 当前纳入范围只包含 `utter-project` 官方组织下公开可下载 `tokenizer.json` 且非 gated 的 EuroLLM / EuroMoE 文本主线。\n- `EuroLLM-9B`、`EuroLLM-9B-Instruct` 当前仍是 gated，暂不纳入。\n- `*-Preview` 当前按被 `2512` 线覆盖的历史预览版本处理，暂不纳入。\n- `EuroLLM-9B-2512`、`EuroLLM-22B-2512`、`EuroMoE-2.6B-A0.6B-2512` 的 `tokenizer.json` SHA256 一致，因此收口到同一个 `eurollm-2512` family。\n- `EuroLLM-9B-Instruct-2512`、`EuroLLM-22B-Instruct-2512` 与 `EuroMoE-2.6B-A0.6B-Instruct-2512` 在当前仓库行为边界内可复用同一 family，因此收口到 `eurollm-2512-instruct`。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-utter-project\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-utter-project\"\n\nconst base = await getTokenizer(\"utter-project/EuroLLM-22B-2512\")\nconst instruct = await getTokenizer(\"eurollm-2512-instruct\")\n\nconsole.log(base === instruct)\n```\n","readmeFilename":"README.md","_rev":"1-dc4013560307362caec1d3db75eeccd0"}