# tokenizers

https://github.com/huggingface/tokenizers/

For some reason this library is not correctly published to NPM.

The latest version of `tokenizers` is currently `0.13.4-rc2` and was published
over a year ago.

https://www.npmjs.com/package/tokenizers?activeTab=versions

The `tokenizers-linux-x64-musl` library is also missing from NPM?

---

This code and binaries is copied from a recent fork:

- @tmfi/tokenizers v0.20.0-dev0
- @tmfi/tokenizers-tokenizers-darwin-universal v0.20.0-dev0
- @tmfi/tokenizers-tokenizers-linux-x64-gnu v0.20.0-dev0
- @tmfi/tokenizers-tokenizers-linux-x64-musl v0.20.0-dev0
