{"_id":"@cyberlangke/tokkit-lightonai","name":"@cyberlangke/tokkit-lightonai","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-lightonai","version":"1.11.0","description":"LightOn model tokenizers for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"npm run build --workspace @cyberlangke/tokkit-core && npm run build --workspace @cyberlangke/tokkit-tiiuae && tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0","@cyberlangke/tokkit-tiiuae":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"SEE LICENSE IN LICENSE","gitHead":"6d7b164cdcb511e7f5815e7278708c780b05d938","_id":"@cyberlangke/tokkit-lightonai@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-xy6FHvv1zb/6waJKv09cv4dp9TrohS1O33YsKk1xJIEdgjxbf3kqNpmVfVWeROXvvGmM68yQCED3sQrMTqIQfg==","shasum":"f2b10b5a9ca2cee260701543605a965eff4c1a71","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-lightonai/-/tokkit-lightonai-1.11.0.tgz","fileCount":14,"unpackedSize":3074038,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQCASXN7k8qE6TR6FUCPyPkHEqsXiG7S5SmE/p8NhW3HOgIgCyEve5JQF9hZcI8T/nQXOdIWvNtF2ZxSQ9xHhwJxxgg="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-lightonai_1.11.0_1775230201631_0.6211742584630837"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-03T15:30:01.223Z","1.11.0":"2026-04-03T15:30:01.920Z","modified":"2026-04-03T15:30:02.160Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"LightOn model tokenizers for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"SEE LICENSE IN LICENSE","readme":"# @cyberlangke/tokkit-lightonai\n\nLightOn 官方文本模型的 tokkit 子包。\n\n当前纳入的官方主线模型：\n\n- `lightonai/pagnol-small`\n- `lightonai/pagnol-medium`\n- `lightonai/pagnol-large`\n- `lightonai/pagnol-xl`\n- `lightonai/alfred-40b-0723`\n- `lightonai/alfred-40b-1023`\n\n当前不纳入：\n\n- `lightonai/RITA_s`\n- `lightonai/RITA_m`\n- `lightonai/RITA_l`\n- `lightonai/RITA_xl`\n- `lightonai/FastVLM-*`\n\n说明：\n\n- `pagnol-small`、`pagnol-medium`、`pagnol-large` 共享同一 tokenizer，当前收口为 `pagnol` family。\n- `pagnol-xl` 使用独立 tokenizer，当前收口为 `pagnol-xl` family。\n- `alfred-40b-0723` 复用现有 `falcon-7b` family。\n- `alfred-40b-1023` 使用独立 tokenizer，当前收口为 `alfred-40b-1023` family。\n- `RITA_*` 当前是蛋白质序列模型，且模型页没有稳定的 MIT / Apache-2.0 许可证信号，不进入当前文本 BPE 主线。\n- `FastVLM-*` 是多模态路线，不进入当前纯文本 BPE 主线。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-lightonai\n```\n\n```ts\nimport { getEncoding } from \"@cyberlangke/tokkit-lightonai\"\n\nconst tokenizer = await getEncoding(\"lightonai/pagnol-small\")\n\nconsole.log(tokenizer.encode(\"Bonjour LightOn\"))\n```\n","readmeFilename":"README.md","_rev":"1-6a714f2ad5b69c7815e1874e784eb51e"}