{"_id":"@cyberlangke/tokkit-ai-sage","name":"@cyberlangke/tokkit-ai-sage","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-ai-sage","version":"1.11.0","description":"AI-Sage tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"MIT","gitHead":"b773a4621a1585e7ea80d2512918ff94a5110109","_id":"@cyberlangke/tokkit-ai-sage@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-kLoOzKr04K1dTcctEuHeHbC7HhgnPpSbtaeDPTaF/tG53M2oB37aG7A0JuGPXqQpY6zrDRjC/mibSshBKclLMQ==","shasum":"cd51c2f3abac0d20dc870c25b7e4d2881cf67b12","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-ai-sage/-/tokkit-ai-sage-1.11.0.tgz","fileCount":16,"unpackedSize":10537412,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIG/ELReg5odVXNrCxqTc5+vFfv4zM07Tw7/TZHrX8lUbAiBa4xqiYB2MQgq50ccTfMBTOntihJWfszCilsBIkQlm6A=="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-ai-sage_1.11.0_1775028100324_0.7643933882852754"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-01T07:21:40.176Z","1.11.0":"2026-04-01T07:21:40.749Z","modified":"2026-04-01T07:21:41.001Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"AI-Sage tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"MIT","readme":"# @cyberlangke/tokkit-ai-sage\n\nAI-Sage 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `gigachat-20b-base`\n  - 覆盖 `ai-sage/GigaChat-20B-A3B-base`\n- `gigachat-20b-instruct`\n  - 覆盖 `ai-sage/GigaChat-20B-A3B-instruct`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `ai-sage/GigaChat-20B-A3B-instruct-v1.5`\n- `gigachat3`\n  - 覆盖 `ai-sage/GigaChat3-10B-A1.8B-base`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `ai-sage/GigaChat3-10B-A1.8B`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `ai-sage/GigaChat3-702B-A36B-preview`\n- `gigachat3.1`\n  - 覆盖 `ai-sage/GigaChat3.1-10B-A1.8B`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `ai-sage/GigaChat3.1-702B-A36B`\n\n当前不纳入：\n\n- `ai-sage/*-GGUF`\n- `ai-sage/*-bf16`\n- `ai-sage/*-int8`\n\n说明：\n\n- `GigaChat-20B-A3B-instruct` 与 `GigaChat-20B-A3B-instruct-v1.5` 的 `tokenizer.json` SHA256 一致，因此收口到同一个 `gigachat-20b-instruct` family。\n- `GigaChat3-10B-A1.8B-base`、`GigaChat3-10B-A1.8B` 与 `GigaChat3-702B-A36B-preview` 的 `tokenizer.json` SHA256 一致，因此收口到同一个 `gigachat3` family。\n- `GigaChat3.1-10B-A1.8B` 与 `GigaChat3.1-702B-A36B` 的 `tokenizer.json` SHA256 一致，因此收口到同一个 `gigachat3.1` family。\n- `GGUF`、`bf16`、`int8` 当前只视为发布格式衍生物，不作为独立支持目标。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-ai-sage\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-ai-sage\"\n\nconst base = await getTokenizer(\"gigachat-20b-base\")\nconst latest = await getTokenizer(\"ai-sage/GigaChat3.1-702B-A36B\")\n\nconsole.log(base.encode(\"Hello, AI-Sage!\"))\nconsole.log(latest.encode(\"Hello, AI-Sage!\"))\n```\n","readmeFilename":"README.md","_rev":"1-75889862f2f7327bbd2a09cf3dcd6d2a"}