{"_id":"@cyberlangke/tokkit-cerebras","name":"@cyberlangke/tokkit-cerebras","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-cerebras","version":"1.11.0","description":"Cerebras tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"1235320f110a411f3b06799059c8051280758341","_id":"@cyberlangke/tokkit-cerebras@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-KUXsBwn5djADlhqB8lvOV+BLeQegf/bPF0uNMHVYr5AVP3IyTdkI5RmvLWpNgN6/AeJ8beBlp4WNm551vgbL7w==","shasum":"1eef061a106330d83c37338abac7e5734fa7b6cc","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-cerebras/-/tokkit-cerebras-1.11.0.tgz","fileCount":12,"unpackedSize":1848862,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIBAWwykxJikH6HFLzDSDy7sK839+WgqH7gG+Y2xQEkzLAiEAw6fANlzfrwqkwAk5YG+nGqsD9lzU24XhQiXckiwXay0="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-cerebras_1.11.0_1775052631891_0.9712242100367972"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-01T14:10:31.820Z","1.11.0":"2026-04-01T14:10:32.134Z","modified":"2026-04-01T14:10:32.610Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"Cerebras tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-cerebras\n\nCerebras 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `cerebras-gpt`\n  - 覆盖 `cerebras/Cerebras-GPT-111M`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `cerebras/Cerebras-GPT-256M`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `cerebras/Cerebras-GPT-590M`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `cerebras/Cerebras-GPT-1.3B`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `cerebras/Cerebras-GPT-2.7B`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `cerebras/Cerebras-GPT-6.7B`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `cerebras/Cerebras-GPT-13B`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `cerebras/btlm-3b-8k-base`\n- `btlm-3b-8k-chat`\n  - 覆盖 `cerebras/btlm-3b-8k-chat`\n\n当前不纳入：\n\n- `cerebras/REAP-*`\n- `cerebras/Cerebras-LLaVA-*`\n- `cerebras/DocChat*`\n- `cerebras/Llama-*CBHybrid*`\n- `cerebras/Cerebras-GPT-Intermediate`\n\n说明：\n\n- `cerebras/Cerebras-GPT-*` 与 `cerebras/btlm-3b-8k-base` 的 `tokenizer.json` SHA256 一致，因此统一收口到 `cerebras-gpt`。\n- `cerebras/btlm-3b-8k-chat` 使用另一组 tokenizer，因此单独保留 `btlm-3b-8k-chat` family。\n- 为了兼容按模型线调用，`btlm-3b-8k` 作为 `cerebras-gpt` 的 family alias 保留。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-cerebras\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-cerebras\"\n\nconst gpt = await getTokenizer(\"cerebras/Cerebras-GPT-13B\")\nconst btlmBase = await getTokenizer(\"btlm-3b-8k\")\nconst btlmChat = await getTokenizer(\"cerebras/btlm-3b-8k-chat\")\n\nconsole.log(gpt === btlmBase)\nconsole.log(btlmChat.encode(\"Hello from Cerebras\"))\n```\n","readmeFilename":"README.md","_rev":"1-0b3fe9f976ac99832e2db7795f68986f"}