{"_id":"@cyberlangke/tokkit-snowflake","name":"@cyberlangke/tokkit-snowflake","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-snowflake","version":"1.11.0","description":"Snowflake tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"a1ee42794c134d748878aecc12f0567dbd3542f7","_id":"@cyberlangke/tokkit-snowflake@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-SpX+JAoxIC+CoEwJ/hQh6PjM+ewzUY/Doua+gPpN5rlHdpfBy7ofRi57WEfHGgzHEYC5xl1ii9R8NYrytKxV0A==","shasum":"be90d4936436818997ae21ca9b9fc1e78ba3f421","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-snowflake/-/tokkit-snowflake-1.11.0.tgz","fileCount":12,"unpackedSize":1724414,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQCP6g85ZY7pqNQCUu+jDyT2AaTxisG9JOUDo8RJj+UKTgIgB09ZpTt/NEVFMP4BxoxzLxX3miJMv7kBwKTbeSInI5c="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-snowflake_1.11.0_1775113733557_0.30925821488276783"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-02T07:08:53.408Z","1.11.0":"2026-04-02T07:08:53.900Z","modified":"2026-04-02T07:08:54.161Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"Snowflake tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-snowflake\n\nSnowflake 官方 Arctic 文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `snowflake-arctic-base`\n  - 覆盖 `Snowflake/snowflake-arctic-base`\n- `snowflake-arctic-instruct`\n  - 覆盖 `Snowflake/snowflake-arctic-instruct`\n\n当前不纳入：\n\n- `Snowflake/snowflake-arctic-instruct-vllm`\n\n说明：\n\n- 当前纳入范围只包含 `Snowflake` 官方组织下公开可下载 `tokenizer.json` 的 Arctic 文本主线。\n- `snowflake-arctic-base` 与 `snowflake-arctic-instruct` 的 `merges`、`normalizer`、`pre_tokenizer`、`decoder` 相同，但 `vocab` 与 `added_tokens` 不同，因此必须保留为两个独立 family。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-snowflake\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-snowflake\"\n\nconst base = await getTokenizer(\"snowflake-arctic-base\")\nconst instruct = await getTokenizer(\"Snowflake/snowflake-arctic-instruct\")\n\nconsole.log(base === instruct)\n```\n","readmeFilename":"README.md","_rev":"1-3c1db614d1ef843bd1f11d93bcdc1b59"}