{"_id":"@cyberlangke/tokkit-baidu","name":"@cyberlangke/tokkit-baidu","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-baidu","version":"1.11.0","description":"baidu tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"npm run build --workspace @cyberlangke/tokkit-core && tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"6d7b164cdcb511e7f5815e7278708c780b05d938","_id":"@cyberlangke/tokkit-baidu@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-qChkddACgcxXEl88mk0MurCmmWffx5yrWBRGZ87VJvY6Gf3J5GKMjk/B/xox5GbKZXeRaGb0ggyXAijPJd2Pgg==","shasum":"d46535bd8976d186db2fb23b35f4fafc58e1bbcd","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-baidu/-/tokkit-baidu-1.11.0.tgz","fileCount":12,"unpackedSize":5660231,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQC/r5XqlOFFrtCEyNq9p4SRp//2ZYUy3d5dHxD9qGwTWwIgF+4J4/GbjZ+hhQ9W+wHZfwCBXgTe8ndple2ENNHxSqQ="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-baidu_1.11.0_1775230111986_0.30754515751268663"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-03T15:28:31.910Z","1.11.0":"2026-04-03T15:28:32.327Z","modified":"2026-04-03T15:28:32.517Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"baidu tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-baidu\n\nbaidu 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `ernie-4.5`\n  - 覆盖 `baidu/ERNIE-4.5-0.3B-PT`\n  - 覆盖 `baidu/ERNIE-4.5-0.3B-Base-PT`\n  - 覆盖 `baidu/ERNIE-4.5-21B-A3B-PT`\n  - 覆盖 `baidu/ERNIE-4.5-21B-A3B-Base-PT`\n  - 覆盖 `baidu/ERNIE-4.5-300B-A47B-PT`\n  - 覆盖 `baidu/ERNIE-4.5-300B-A47B-Base-PT`\n- `ernie-4.5-thinking`\n  - 覆盖 `baidu/ERNIE-4.5-21B-A3B-Thinking`\n\n说明：\n\n- 当前只纳入 `baidu` 官方组织下公开 `tokenizer.json`、`license: apache-2.0` 的官方 PT 主线。\n- 上述 `6` 个 `PT / Base-PT` 官方模型的 `tokenizer.json` SHA256 都是 `4e5b4d49475d27c7ef7a61eea2d675f9cc013e91e3ddb5e06d964f5027c1814a`，因此收口为同一个 `ernie-4.5` family。\n- `baidu/ERNIE-4.5-21B-A3B-Thinking` 的 `tokenizer.json` SHA256 是 `93a291ef6e5c89e6c0f2d5d004734ff429faa8ab925e39d71d2374b3d0daf49e`，当前收口为 `ernie-4.5-thinking` family。\n- `baidu/*Paddle*` 当前是导出格式变体，且没有当前流水线要求的标准 `tokenizer.json`，不进入当前文本 BPE 主线。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-baidu\n```\n\n```ts\nimport { getEncoding } from \"@cyberlangke/tokkit-baidu\"\n\nconst tokenizer = await getEncoding(\"baidu/ERNIE-4.5-21B-A3B-PT\")\nconst sameTokenizer = await getEncoding(\"ernie-4.5\")\n\nconsole.log(tokenizer === sameTokenizer)\n```\n","readmeFilename":"README.md","_rev":"1-e847207999efb5e8489fe7d15849d4ed"}