{"_id":"@cyberlangke/tokkit-datajuicer","name":"@cyberlangke/tokkit-datajuicer","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-datajuicer","version":"1.11.0","description":"datajuicer tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"npm run build --workspace @cyberlangke/tokkit-core && tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"Apache-2.0","gitHead":"6d7b164cdcb511e7f5815e7278708c780b05d938","_id":"@cyberlangke/tokkit-datajuicer@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-OwrrNX5yT2lIu0KyiMJbcWY793Sqi9P2QKnLivAuwEIOgc3RqibGaIZSEPWo99cBATGFDsxRMt7NYLsSskgtxA==","shasum":"5c7e915ae052c16b1e28d4d9c9ba4b14f84c771b","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-datajuicer/-/tokkit-datajuicer-1.11.0.tgz","fileCount":10,"unpackedSize":871538,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQC6mcxkQy2mvfF66r5SVg4l0hTLwlSyTF0x08jLQFGzEwIhAPGT8Pe9JCpMfWWylb5SDzgiwk3GnaXZ3nGeu9Cw9aGA"}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-datajuicer_1.11.0_1775230148719_0.4786283861590417"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-03T15:29:08.616Z","1.11.0":"2026-04-03T15:29:08.955Z","modified":"2026-04-03T15:29:09.193Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"datajuicer tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"Apache-2.0","readme":"# @cyberlangke/tokkit-datajuicer\n\ndatajuicer 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `dj-refine-1b`\n  - 覆盖 `datajuicer/LLaMA-1B-dj-refine-50B`\n  - 覆盖 `datajuicer/LLaMA-1B-dj-refine-100B`\n  - 覆盖 `datajuicer/LLaMA-1B-dj-refine-150B`\n  - 覆盖 `datajuicer/LLaMA-1B-dj-refine-150B-instruct-4.7B`\n\n说明：\n\n- 当前只纳入 `datajuicer` 官方组织下公开 `tokenizer.json`、`license: apache-2.0` 的官方 `1B-dj-refine` 主线。\n- 上述 `4` 个官方模型的 `tokenizer.json` SHA256 都是 `91bf98512883c20dcdac94cdbe33ecfc11765a00b99ca75c62adf63b488f2a17`，因此收口为同一个 `dj-refine-1b` family。\n- `datajuicer/LLaMA-7B-EN-Chat-40k` 的官方 README 明确写明 built upon `huggyllama/llama-7b`，不作为新的官方主线 family。\n- `datajuicer/LLaMA2-7B-ZH-Chat-52k` 的官方 README 明确写明 built upon `FlagAlpha/Atom-7B`，不作为新的官方主线 family。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-datajuicer\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-datajuicer\"\n\nconst tokenizer = await getTokenizer(\"datajuicer/LLaMA-1B-dj-refine-150B\")\nconst sameTokenizer = await getTokenizer(\"dj-refine-1b\")\n\nconsole.log(tokenizer === sameTokenizer)\n```\n","readmeFilename":"README.md","_rev":"1-dd857c49cdbb8ddec317df1c5dd1c609"}