{"_id":"@cyberlangke/tokkit-state-spaces","name":"@cyberlangke/tokkit-state-spaces","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-state-spaces","version":"1.11.0","description":"state-spaces tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"b29408e64b90d369f1cf36235740f212c91a7e11","_id":"@cyberlangke/tokkit-state-spaces@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-6GESZkpCLOwnvqClrVRMUnfvJc/4gROyngr/MiSQ7RWM9tDtINimIJWepetLG+t6hp0ZkJq+fU5bS8KrphZbpQ==","shasum":"b024d215f3501efde4e8e08829ef200a048632ff","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-state-spaces/-/tokkit-state-spaces-1.11.0.tgz","fileCount":12,"unpackedSize":1833881,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQDv7WRegCY5Um5hZTuBD71zFt9ynk5hQt/OhTPk32NNiAIhAIeZevukIUs5HM5Nf1UBXLuqZUQOXRrE+HtAuHvSHNgh"}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-state-spaces_1.11.0_1775111067338_0.13025233632612876"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-02T06:24:27.219Z","1.11.0":"2026-04-02T06:24:27.605Z","modified":"2026-04-02T06:24:27.761Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"state-spaces tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-state-spaces\n\nstate-spaces 官方 Mamba 文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `mamba-130m`\n  - 覆盖 `state-spaces/mamba-130m-hf`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `state-spaces/mamba-370m-hf`\n- `mamba-790m`\n  - 覆盖 `state-spaces/mamba-790m-hf`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `state-spaces/mamba-1.4b-hf`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `state-spaces/mamba-2.8b-hf`\n\n当前不纳入：\n\n- `state-spaces/mamba2-*`\n- `state-spaces/transformerpp-2.7b`\n- `state-spaces/mamba2attn-2.7b`\n\n说明：\n\n- 当前纳入范围只包含官方 `*-hf` 仓库，因为这一组仓库直接公开了标准 `tokenizer.json`。\n- 对应的原始 `state-spaces/mamba-*` 基座模型卡明确声明 `license: apache-2.0`；`*-hf` 仓库 README 明确说明它们是同组织发布的 transformers-compatible 兼容仓库，checkpoint 保持不变，只额外推送了 `config.json` 与 tokenizer。\n- `state-spaces/mamba-130m-hf` 与 `state-spaces/mamba-370m-hf` 的 `tokenizer.json` SHA256 一致，因此收口到同一个 `mamba-130m` family。\n- `state-spaces/mamba-790m-hf`、`state-spaces/mamba-1.4b-hf`、`state-spaces/mamba-2.8b-hf` 的 `tokenizer.json` SHA256 一致，因此收口到同一个 `mamba-790m` family。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-state-spaces\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-state-spaces\"\n\nconst small = await getTokenizer(\"mamba-130m\")\nconst bigger = await getTokenizer(\"state-spaces/mamba-370m-hf\")\n\nconsole.log(small === bigger)\n```\n","readmeFilename":"README.md","_rev":"1-6e2565a297180b2ed5f72cce5d290b64"}