{"_id":"@cyberlangke/tokkit-ibm-research","name":"@cyberlangke/tokkit-ibm-research","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-ibm-research","version":"1.11.0","description":"IBM Research tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"2e0052c431e6edf13b7848cc247b7d7c731b4f83","_id":"@cyberlangke/tokkit-ibm-research@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-f72o/AsQWmL7NnAQjYda+9DysP94WcsaZ+43+QnEEv6d+atnp5Xka2xXC+K4nv09NN3fhaVXs2u1IAPsUZkfeQ==","shasum":"0ddae374015c9675441bc2cd70ef592c2cfdfaab","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-ibm-research/-/tokkit-ibm-research-1.11.0.tgz","fileCount":12,"unpackedSize":1833598,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIGYKA2iVOw0FUwXnqah4dD9Gh001zGobSB5wmSk0If3HAiAynzQrsEeiP9YBpkt3iypntuQv50Txh6i6nSxPfx7Law=="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-ibm-research_1.11.0_1774947144454_0.07666295243603627"},"_hasShrinkwrap":false}},"time":{"created":"2026-03-31T08:52:24.388Z","1.11.0":"2026-03-31T08:52:24.669Z","modified":"2026-03-31T08:52:24.833Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"IBM Research tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-ibm-research\n\nIBM Research 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `powerlm`\n  - 覆盖 `ibm-research/PowerLM-3b`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `ibm-research/PowerMoE-3b`\n- `molm`\n  - 覆盖 `ibm-research/MoLM-350M-4B`\n  - 也聚合当前确认完全复用同一 tokenizer 的：\n    - `ibm-research/MoLM-700M-4B`\n    - `ibm-research/MoLM-700M-8B`\n\n当前不纳入：\n\n- `ibm-research/gpt2-medium-multiexit`\n- `ibm-research/gpt-neo-125m-multiexit`\n- `ibm-research/mpt-7b-instruct2`\n- `ibm-research/merlinite-7b`\n- `GGUF` 等量化 / 导出对象\n\n说明：\n\n- `PowerLM-3b` 与 `PowerMoE-3b` 的 `tokenizer.json` SHA256 一致，因此收口到同一个 `powerlm` family。\n- `MoLM-350M-4B`、`MoLM-700M-4B` 与 `MoLM-700M-8B` 的 `tokenizer.json` SHA256 一致，因此收口到同一个 `molm` family。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-ibm-research\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-ibm-research\"\n\nconst powerlm = await getTokenizer(\"powerlm\")\nconst molm = await getTokenizer(\"ibm-research/MoLM-700M-8B\")\n\nconsole.log(powerlm.encode(\"Hello, IBM Research!\"))\nconsole.log(molm.encode(\"Hello, IBM Research!\"))\n```\n","readmeFilename":"README.md","_rev":"1-3dc5590eb23772df6949d14e9178c0ea"}