{"_id":"@cyberlangke/tokkit-skt","name":"@cyberlangke/tokkit-skt","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-skt","version":"1.11.0","description":"SKT tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"89c49d66409afcc48410be37ec76669b993439fe","_id":"@cyberlangke/tokkit-skt@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-QOiNPbO5BxO1lkudbuJqwr0GLrwAgo1Bdj6xa03fw9Nw1N8d/Hh6a9DZ+ec1PogSUrncjS1+NYcnCi3aRoAkgg==","shasum":"1e2b35e5a290629fb8695519abc0b6c0ef7f7ba1","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-skt/-/tokkit-skt-1.11.0.tgz","fileCount":14,"unpackedSize":7613853,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCICs3UKggjzqSt2+7601HGasPIE6SO6uxE+UR/nxutGgLAiEA4p2Yjns1oMm0zFlqUqwFKJUfORyXU9zhIkrK6qmDxUg="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-skt_1.11.0_1775011517428_0.5646520418200529"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-01T02:45:17.361Z","1.11.0":"2026-04-01T02:45:17.805Z","modified":"2026-04-01T02:45:18.051Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"SKT tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-skt\n\nSKT 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `ax-3.1`\n  - 覆盖 `skt/A.X-3.1`\n- `ax-light`\n  - 覆盖 `skt/A.X-3.1-Light`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `skt/A.X-4.0-Light`\n- `ax-k1`\n  - 覆盖 `skt/A.X-K1`\n\n当前不纳入：\n\n- `skt/A.X-4.0`\n- `skt/kogpt2-base-v2`\n- `skt/ko-gpt-trinity-1.2B-v0.5`\n\n说明：\n\n- `A.X-3.1-Light` 与 `A.X-4.0-Light` 的 `tokenizer.json` SHA256 一致，因此收口到同一个 `ax-light` family。\n- `A.X-3.1`、`A.X-K1` 的 `tokenizer.json` SHA256 都与 `ax-light` 不同，因此分别保留独立 family。\n- `A.X-4.0` 当前模型页许可证是 `other`，不进入当前 Apache-2.0 子包。\n- `kogpt2-base-v2` 与 `ko-gpt-trinity-1.2B-v0.5` 当前模型页许可证是 `cc-by-nc-sa-4.0`，不进入当前 Apache-2.0 子包。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-skt\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-skt\"\n\nconst base = await getTokenizer(\"ax-3.1\")\nconst light = await getTokenizer(\"skt/A.X-4.0-Light\")\n\nconsole.log(base.encode(\"Hello, SKT!\"))\nconsole.log(light.encode(\"Hello, SKT!\"))\n```\n","readmeFilename":"README.md","_rev":"1-7b5f3fda263abe01eb1fe29e38030e75"}