{"_id":"@cyberlangke/tokkit-salesforce","name":"@cyberlangke/tokkit-salesforce","dist-tags":{"latest":"1.11.0"},"versions":{"1.11.0":{"name":"@cyberlangke/tokkit-salesforce","version":"1.11.0","description":"Salesforce tokenizer families for tokkit.","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch"},"dependencies":{"@cyberlangke/tokkit-core":"1.11.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"homepage":"https://github.com/1cyberlangke1/tokkit#readme","bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"author":{"name":"cyberlangke"},"license":"SEE LICENSE IN LICENSE","gitHead":"144b826b12439d8ffefafff109a34c1dd7e87baa","_id":"@cyberlangke/tokkit-salesforce@1.11.0","_nodeVersion":"24.14.0","_npmVersion":"11.9.0","dist":{"integrity":"sha512-Ylvi6JK/ljpcJysRGydZqp+sYSBaXrp8vMvgUCyJZUwynKqIYUk6n5LFKC+c+xp5UgpPtXacLq+1YWGwzsTNqA==","shasum":"2f54536e945aea6f689498e2b3ef56ea1ece8cdf","tarball":"https://registry.npmjs.org/@cyberlangke/tokkit-salesforce/-/tokkit-salesforce-1.11.0.tgz","fileCount":14,"unpackedSize":2775634,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQDQfLEndxaX2sWVYYJChNFiQ3euYhwrmEmPQNTpgXjjTQIgYMytxtbvCRn6ynOF7kqC2KalhuX5MvWkIQ8xRl7WWgc="}]},"_npmUser":{"name":"cyberlangke","email":"3531702083@qq.com"},"directories":{},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokkit-salesforce_1.11.0_1775048804896_0.7970105858474588"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-01T13:06:44.753Z","1.11.0":"2026-04-01T13:06:45.115Z","modified":"2026-04-01T13:06:45.534Z"},"maintainers":[{"name":"cyberlangke","email":"3531702083@qq.com"}],"description":"Salesforce tokenizer families for tokkit.","homepage":"https://github.com/1cyberlangke1/tokkit#readme","repository":{"type":"git","url":"git+https://github.com/1cyberlangke1/tokkit.git"},"author":{"name":"cyberlangke"},"bugs":{"url":"https://github.com/1cyberlangke1/tokkit/issues"},"license":"SEE LICENSE IN LICENSE","readme":"# @cyberlangke/tokkit-salesforce\n\nSalesforce 官方文本 tokenizer 的 tokkit 子包。\n\n当前内置 family：\n\n- `codegen`\n  - 覆盖 `Salesforce/codegen-350M-mono`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-350M-multi`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-2B-mono`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-2B-multi`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-6B-mono`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-6B-multi`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-16B-mono`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-16B-multi`\n- `codegen-nl`\n  - 覆盖 `Salesforce/codegen-350M-nl`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-2B-nl`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-6B-nl`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen-16B-nl`\n- `codegen2`\n  - 覆盖 `Salesforce/codegen2-1B_P`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen2-3_7B_P`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen2-7B_P`\n  - 也聚合当前确认完全复用同一 tokenizer 的 `Salesforce/codegen2-16B_P`\n\n当前不纳入：\n\n- `Salesforce/ctrl`\n- `Salesforce/xgen-*`\n- `Salesforce/codegen25-*`\n- `Salesforce/xLAM-*`\n- `Salesforce/*-gguf`\n- `Salesforce/codet5-*`\n\n说明：\n\n- `codegen` 当前至少分成 `mono/multi`、`nl` 两组 tokenizer，不能把整个 `codegen` 系列先验当作单一 family。\n- `codegen2` 与老 `codegen` 线的 tokenizer 也不同，因此单独保留 `codegen2` family。\n- 当前子包同时分发 `bsd-3-clause` 和 `apache-2.0` 资产，所以包级许可证使用 `SEE LICENSE IN LICENSE`。\n\n## 使用方法\n\n```bash\nnpm install @cyberlangke/tokkit-salesforce\n```\n\n```ts\nimport { getTokenizer } from \"@cyberlangke/tokkit-salesforce\"\n\nconst mono = await getTokenizer(\"Salesforce/codegen-6B-mono\")\nconst nl = await getTokenizer(\"codegen-nl\")\nconst codegen2 = await getTokenizer(\"Salesforce/codegen2-7B_P\")\n\nconsole.log(mono.encode(\"def hello_world():\\n    return 42\"))\nconsole.log(nl.encode(\"Write a summary about tokenization.\"))\nconsole.log(codegen2.encode(\"SELECT * FROM users WHERE id = 42;\"))\n```\n","readmeFilename":"README.md","_rev":"1-5d441a44c4d2146f3439f9651bd43e0b"}