{"_id":"@cyberlangke/tokkit-aidc-ai","name":"@cyberlangke/tokkit-aidc-ai","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-aidc-ai","version":"1.11.0","description":"AIDC-AI tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"npm run build --workspace @cyberlangke/tokkit-core && npm run build --workspace @cyberlangke/tokkit-qwen && tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0","@cyberlangke/tokkit-qwen":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"6d7b164cdcb511e7f5815e7278708c780b05d938","_id":"@cyberlangke/tokkit-aidc-ai@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-z8FYO51ZTCMKFEZXrier5IiqbQbpAUywen+wuiwtxKrH0pJ66t2oUaGE9RUa7Ue1RzKi4n7kuWTftO03yj/Wfw==","shasum":"7ba5c0e0ef8920cb0c8eea25aada738a37d65592","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-aidc-ai/-/tokkit-aidc-ai-1.11.0.tgz","fileCount":12,"unpackedSize":5926761,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQDGQrAIVyj35wMXpYKxRl7nLy2L/WjyyT7s67z3wKOUbQIgDBFuK0N71666eLS0VQASXzK2J0qcOSLqye/E7pDsnOA="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-aidc-ai_1.11.0_1775230093714_0.9522880420378259"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-03T15:28:13.632Z","1.11.0":"2026-04-03T15:28:14.036Z","modified":"2026-04-03T15:28:14.576Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"AIDC-AI tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-aidc-ai\n\nAIDC-AI 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `marco-o1`\n  - 覆盖 `AIDC-AI/Marco-o1`\n\n当前复用的既有 family：\n\n- `qwen2`\n  - 覆盖 `AIDC-AI/Marco-LLM-ES`\n- `qwen2.5`\n  - 覆盖 `AIDC-AI/Marco-Nano-Instruct`\n  - 覆盖 `AIDC-AI/Marco-Mini-Global-Base`\n  - 覆盖 `AIDC-AI/Marco-Mini-Base`\n  - 覆盖 `AIDC-AI/Marco-Nano-Base`\n  - 覆盖 `AIDC-AI/Marco-LLM-SEA`\n  - 覆盖 `AIDC-AI/Marco-LLM-AR-V4`\n  - 覆盖 `AIDC-AI/Marco-LLM-AR-V2`\n\n说明：\n\n- 当前只纳入 `AIDC-AI` 官方组织下公开 `tokenizer.json`、非 gated、`license: apache-2.0` 且 `model.type = BPE` 的文本模型。\n- `Marco-Nano-Instruct`、`Marco-Mini-Global-Base`、`Marco-Mini-Base`、`Marco-Nano-Base`、`Marco-LLM-SEA`、`Marco-LLM-AR-V4`、`Marco-LLM-AR-V2` 的 `tokenizer.json` SHA256 一致，因此复用 `qwen2.5`。\n- `Marco-LLM-ES` 的 `tokenizer.json` SHA256 与 `qwen2` 一致，因此复用 `qwen2`。\n- `Marco-o1` 的 `tokenizer.json` SHA256 未命中仓库现有快照，因此单独提供 `marco-o1` family。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-aidc-ai\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-aidc-ai\"\n\nconst marcoO1 = await getTokenizer(\"AIDC-AI/Marco-o1\")\nconst marcoEs = await getTokenizer(\"AIDC-AI/Marco-LLM-ES\")\n\nconsole.log(marcoO1 !== marcoEs)\n```\n","readmeFilename":"README.md","_rev":"1-a6ff1f5fcf46ac6ceb77a446636075b4"}