{"_id":"@accessprotocol/tokenizers","_rev":"2-f58a4a3078759d8f96acd9e13bbbfb34","name":"@accessprotocol/tokenizers","dist-tags":{"latest":"0.3.3"},"versions":{"0.3.2":{"name":"@accessprotocol/tokenizers","version":"0.3.2","license":"MIT","_id":"@accessprotocol/tokenizers@0.3.2","maintainers":[{"name":"martincik","email":"ladislav.martincik@gmail.com"},{"name":"vl-dev","email":"vladislav@accessprotocol.co"},{"name":"saldev-access","email":"sal@accessprotocol.co"}],"homepage":"https://github.com/Access-Labs-Inc/tokenizers#readme","bugs":{"url":"https://github.com/Access-Labs-Inc/tokenizers/issues"},"ava":{"timeout":"3m"},"dist":{"shasum":"dd9a57edce18acf1043398d88f86e397a6c4b262","tarball":"https://registry.npmjs.org/@accessprotocol/tokenizers/-/tokenizers-0.3.2.tgz","fileCount":45,"integrity":"sha512-imr5uwb4bQfdIu6mJOM+zq8hflNcHrXquZSxIwovRJ0VtkKA6puDSH0X9vCJBpsLr6nMRTSLaZ7iTt13D+83GQ==","signatures":[{"sig":"MEYCIQDni7e7WisPfIBKCorgBll2tb4fj70iJtgk0eZTq3kajgIhALHmz5uCSmIVQECMktC240Kd1djHMn7501jyr4oKn8/O","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@accessprotocol%2ftokenizers@0.3.2","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":268778},"main":"index.js","napi":{"name":"tokenizers","triples":{"defaults":false,"additional":["x86_64-pc-windows-msvc","x86_64-unknown-linux-gnu","aarch64-unknown-linux-gnu","universal-apple-darwin"]}},"types":"index.d.ts","engines":{"node":">= 10"},"gitHead":"ead7a06d6eee5a0c036442fc4b2cea3705289d21","scripts":{"test":"ava","build":"napi build --platform --release","version":"napi version","artifacts":"napi artifacts","universal":"napi universal","build:debug":"napi build --platform","prepublishOnly":"napi prepublish -t npm"},"_npmUser":{"name":"martincik","email":"ladislav.martincik@gmail.com"},"repository":{"url":"git+https://github.com/Access-Labs-Inc/tokenizers.git","type":"git"},"_npmVersion":"10.8.2","description":"Multi-arch builds of HuggingFace tokenizers","directories":{},"_nodeVersion":"18.20.8","_hasShrinkwrap":false,"packageManager":"yarn@3.6.3","devDependencies":{"ava":"^5.1.1","@napi-rs/cli":"^2.16.3"},"optionalDependencies":{"@accessprotocol/tokenizers-linux-x64-gnu":"0.3.2","@accessprotocol/tokenizers-win32-x64-msvc":"0.3.2","@accessprotocol/tokenizers-linux-arm64-gnu":"0.3.2","@accessprotocol/tokenizers-darwin-universal":"0.3.2"},"_npmOperationalInternal":{"tmp":"tmp/tokenizers_0.3.2_1747914852341_0.6073512154662393","host":"s3://npm-registry-packages-npm-production"}},"0.3.3":{"name":"@accessprotocol/tokenizers","version":"0.3.3","main":"index.js","types":"index.d.ts","napi":{"name":"tokenizers","triples":{"defaults":false,"additional":["x86_64-pc-windows-msvc","x86_64-unknown-linux-gnu","aarch64-unknown-linux-gnu","universal-apple-darwin"]}},"license":"MIT","devDependencies":{"@napi-rs/cli":"^2.16.3","ava":"^5.1.1"},"ava":{"timeout":"3m"},"engines":{"node":">= 10"},"scripts":{"artifacts":"napi artifacts","build":"napi build --platform --release","build:debug":"napi build --platform","prepublishOnly":"napi prepublish -t npm","test":"ava","universal":"napi universal","version":"napi version"},"packageManager":"yarn@3.6.3","repository":{"type":"git","url":"git+https://github.com/Access-Labs-Inc/tokenizers.git"},"description":"Multi-arch builds of HuggingFace tokenizers","optionalDependencies":{"@accessprotocol/tokenizers-win32-x64-msvc":"0.3.3","@accessprotocol/tokenizers-linux-x64-gnu":"0.3.3","@accessprotocol/tokenizers-linux-arm64-gnu":"0.3.3","@accessprotocol/tokenizers-darwin-universal":"0.3.3"},"_id":"@accessprotocol/tokenizers@0.3.3","gitHead":"c2ad7c0df008162e5d1108d5f366cd1b224890a7","bugs":{"url":"https://github.com/Access-Labs-Inc/tokenizers/issues"},"homepage":"https://github.com/Access-Labs-Inc/tokenizers#readme","_nodeVersion":"18.20.8","_npmVersion":"10.8.2","dist":{"integrity":"sha512-s2YzdM/BbhVYbaYqdT40iT1o0rRsZb1ze6dSfoMj4dS2IwPmzumfpTByO3Ez/D6FkyNSzeRTAAFg1fneWQm/vQ==","shasum":"a1d0c3f830420289aa183f9400b1105546105888","tarball":"https://registry.npmjs.org/@accessprotocol/tokenizers/-/tokenizers-0.3.3.tgz","fileCount":45,"unpackedSize":268778,"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@accessprotocol%2ftokenizers@0.3.3","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQD4qaV/yEf8EGv+/3nnfDQf3fZyiBX/JnSg9UdqfXaFQwIgAySiVFFZ6vQlAeqB2jfhVEXM1fTsGgChxcUtDBo+zUU="}]},"_npmUser":{"name":"martincik","email":"ladislav.martincik@gmail.com"},"directories":{},"maintainers":[{"name":"martincik","email":"ladislav.martincik@gmail.com"},{"name":"vl-dev","email":"vladislav@accessprotocol.co"},{"name":"saldev-access","email":"sal@accessprotocol.co"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokenizers_0.3.3_1747916011483_0.1413345535792896"},"_hasShrinkwrap":false}},"time":{"created":"2025-05-22T11:54:12.200Z","modified":"2025-05-22T12:13:32.123Z","0.3.2":"2025-05-22T11:54:12.523Z","0.3.3":"2025-05-22T12:13:31.671Z"},"bugs":{"url":"https://github.com/Access-Labs-Inc/tokenizers/issues"},"license":"MIT","homepage":"https://github.com/Access-Labs-Inc/tokenizers#readme","repository":{"type":"git","url":"git+https://github.com/Access-Labs-Inc/tokenizers.git"},"description":"Multi-arch builds of HuggingFace tokenizers","maintainers":[{"name":"martincik","email":"ladislav.martincik@gmail.com"},{"name":"vl-dev","email":"vladislav@accessprotocol.co"},{"name":"saldev-access","email":"sal@accessprotocol.co"}],"readme":"# @accessprotocol/tokenizers\n\nThe official Node bindings are in a jinx with limited support for Node versions and architectures. This package offers multi-arch bindings for [@huggingface/tokenizers](https://github.com/huggingface/tokenizers) with Node v20.x supported. \n\n## Supports:\n> * Windows x86_64\n> * Linux x86_64\n> * Linux aarch64 (ARM64)\n> * MacOS aarch64/x86_64\n\n## Installation\n\n```bash\nnpm install @accessprotocol/tokenizers\n```\n\n## Features\n\n - Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3\n   most common BPE versions).\n - Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes\n   less than 20 seconds to tokenize a GB of text on a server's CPU.\n - Easy to use, but also extremely versatile.\n - Designed for research and production.\n - Normalization comes with alignments tracking. It's always possible to get the part of the\n   original sentence that corresponds to a given token.\n - Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.\n\n\n## Basic example\n\n```ts\nimport { Tokenizer } from \"@accessprotocol/tokenizers\";\n\nconst tokenizer = await Tokenizer.fromFile(\"tokenizer.json\");\nconst wpEncoded = await tokenizer.encode(\"Who is John?\");\n\nconsole.log(wpEncoded.getLength());\nconsole.log(wpEncoded.getTokens());\nconsole.log(wpEncoded.getIds());\nconsole.log(wpEncoded.getAttentionMask());\nconsole.log(wpEncoded.getOffsets());\nconsole.log(wpEncoded.getOverflowing());\nconsole.log(wpEncoded.getSpecialTokensMask());\nconsole.log(wpEncoded.getTypeIds());\nconsole.log(wpEncoded.getWordIds());\n```\n\n## Development\n\n### Prerequisites\n\n- [Rust](https://rustup.rs/) with `cargo`\n- [Node.js](https://nodejs.org/) >= 10\n- [Yarn](https://yarnpkg.com/)\n\n### Building for ARM64 Linux\n\nTo build for ARM64 Linux, you need to install the cross-compilation target:\n\n```bash\n# Install the ARM64 Linux target\nrustup target add aarch64-unknown-linux-gnu\n\n# Build for ARM64 Linux\nyarn build --target aarch64-unknown-linux-gnu\n```\n\n### Cross-compilation setup\n\nFor cross-compilation to work properly, you may need to install additional tools:\n\n**On Ubuntu/Debian:**\n```bash\nsudo apt-get install gcc-aarch64-linux-gnu\n```\n\n**On macOS:**\n```bash\nbrew install filosottile/musl-cross/musl-cross\n```\n\n## License\n\n[MIT](LICENSE)\n","readmeFilename":"README.md"}