The SOTA tokenization library on all languages, all models, all hardwares.
Our goal with `tokenizers` is to develop and maintain the industry's standard tokenization engine, making it the defacto place for everyone to contribute to the whole ecosystem.
### Release candidate: v1.0.0.rc.0
As we are switching from 0.23 to v1.0.0, the current library does not ship all features. If you are worried, check the [v1 features](#v1-features) section for more details of what we are bringing back.
We will publish a blog about what breaking changes we introduced. We strived to keep them as small as possible.
# Installation
```bash
pip install --pre tokenizers
```
For Rust, add the crate to your `Cargo.toml`:
```toml
[dependencies]
tokenizers = "1.0.0"
```
Training (`train`) and downloading from the Hub (`http`) are opt-in features. The crate's
[Using the crate](tokenizers/README.md#using-the-crate) section lists every feature and how to
build a smaller binary.
# Usage
```python
>>> from tokenizers import Tokenizer
>>> tokenizer = Tokenizer.from_pretrained("meta-llama/Llama-3.1-8B") # or .from_file(path)
>>> tokenizer.encode("Hello, y'all! How are you π ?")
Encoding(ids=[128000, 9906, 11, 379, 65948, 0, 2650, 527, 499, 27623, 223, 949], type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
>>> tokenizer.tokenize("Hello")
['<|begin_of_text|>', 'Hello']
>>> tokenizer.decode([15339, 1917])
'hello world'
>>> tokenizer.decode_tokens(tokenizer.encode("Hello"))
['<|begin_of_text|>', 'Hello']
```
# Performances
For more details, checkout https://huggingface-tokenizers-v1.static.hf.space/index.html.
# Bindings
| | status |
|---|---|
| [Rust](tokenizers) | β
reference implementation |
| [Python](bindings/python) | β
|
| [Node.js](bindings/node) | β
|
| C / C++ / Java / Go/ etc | π§ planned |
| [Ruby](https://github.com/ankane/tokenizers-ruby) | community, external repo |
# v1.0.0 roadmap:
As we work toward v1, we are gonna bring back the entire python API that allows `transformers`' style of interacting with a tokenizer object:
## TODOS:
### Python Bindings
- Bring training back: this should be easy, but we want the perfs gains to come as well.
- Batch encoding API with zero-copy views on encoded ids
- dlpack API for torch / JAX / numpy interop.
- Composition: edit components of the pipeline (`tokenizer.normalizer = [...]`).
- `tokenizer.save` to persist a modified or trained tokenizer on disk.
- `role_to_token` override.
- Accept pair input for `encode` / `encode_batch`.
- Bring back offset output: this should be useful for people training.
- Windowing / stride in encode.
- `decode_stream`.
### Rust core
- `bitnorm`: more performance from optimized normalization, as this will bring breaking changes we want to make sure it gets to v1.
- Windowing (truncation with stride).
- Improve training performance
- Improve decode performance
- Offset tracking
- Padding: 2D flat buffer
- Wordcache: `shrn` vs emulated movemask
- Fix ModernBERT / OLMo load failure ("64-bit hash collision").
- Unroll more regex: we need to cover the most used ones, we might have missed one or two.
### Other
- C & C++ bindings for loading, encode and decode APIs
## After 1.0.0
- Improve / rewrite the `bitcannon` crate
- Java, Go, Swift... bindings, supported by the C bindings
- `tk-devices` β exploratory GPU encoding and batch decoding that keeps text and ids on the
device: upload the vocabulary once, compute output positions in parallel, gather the bytes on the
GPU. An optional component aimed at large batches, subject to prototyping and measurement.
# Crate details
## Crate size
## Sub-crates
tk-encode β the inference engine
The models (BPE, Unigram, WordPiece, WordLevel) and the full pipeline: `Normalizer`,
`PreTokenizer`, `Model`, `PostProcessor`, `Decoder`.
**Why separate:** you don't need all features to serve a model in production settings. Splitting it from training
means a serving binary never links a trainer, a corpus reader, legacy converters, etc.
bitcannon β bitstream pre-tokenization
Unicode tag classification plus pre-tokenization as a **bitstream program**. Follows *Interleaved Bitstream Execution for Multi-Pattern Regex Matching on GPUs*
(MICRO'25, [10.1145/3725843.3756052](https://doi.org/10.1145/3725843.3756052)) but we added our own algorithm to make sure we don't need a parabix engine.
Characters are first classified into "tags", which use a sparse table generated when compiling `bitcannon`. This means there are 0 depencencies to the heavy unicode tables.
Ships byte-exact grammars for gpt2 / ByteLevel, cl100k, o200k, tekken, deepseek and kimi-k2.
The classes or tags each have a high and low nibble (encoded on a u8), and carry refinement when needed. Using u8 tags allows us to leverage less bitsream, meaning have a simpler / faster pre tokenization.
**Why separate:** we just thought it might be useful to some people someday. As far as we know, this implementation is the fastest for every single pretokenizer regex on CPU.
tk-serialize β the reader
`from_json_file` turns a canonical `tokenizer.json` into a `PipelineTokenizer`.
**Why separate:** this just makes it optional for some usecases!
tk-convert β the upgrade pass
`canonicalize_file` rewrites legacy a `tokenizer.json` written by any older version of this library into
the canonical form the reader accepts. A pure JSONβJSON rewrite; it depends on nothing but
`std::path` and `serde_json`.
**Why separate:** every config ever published stays readable without the runtime carrying a decade
of compatibility branches. But, the legacy code amounted to quite a lot of the final crate size. Disabling it allows on device builds to be smaller.
tk-train β the training half
The `Trainer` trait, every concrete `*Trainer`, `TrainerWrapper` and the `Trainable` extension.
The `tokenizers` crate re-exports it behind the `train` feature, which is off by default. It
trains only the models whose feature is on (`bpe`, `unigram`, `wordpiece`, `wordlevel`). Add
`esaxx_fast` for the C++ suffix array behind Unigram training.
**Why separate:** training is a batch job on a workstation; inference is a hot loop in a server.
They have opposite constraints, so they get opposite dependency budgets.
bitmap_gen β dev-only table generator
`cargo run -p bitmap_gen` regenerates `bitcannon`'s committed classify tables from `unicode-properties`, emitting one `Atom` tag per codepoint.
**Why separate β this is what buys the zero Unicode dependency.** `unicode-properties` is a
dependency of *this* crate and of nothing else: the tables it produces are checked into
`bitcannon/src/classify/atom_tables.rs` as ordinary source, so the Unicode data is resolved once, at
development time, by a crate that is never linked into anything that ships and is never published.
No build script either β `bitcannon` compiles with no code generation step, and a release binary
contains the tags without containing a Unicode crate to derive them. The release workflow re-runs
the generator and fails if the committed table differs, so "baked" cannot silently mean "stale".
tokenizers β umbrella
A thin re-export so existing `tokenizers::β¦` paths keep working. Depend on this one unless you know
you want less.
## Hardware
**No SIMD is required.** Every kernel has a portable path that is always compiled and is the
byte-exact test oracle for its vectorised siblings, so correctness never depends on a kernel being
present β only throughput does.
Which ops are hardware-adapted
| op | aarch64 | x86_64 | wasm32 | portable fallback |
|---|---|---|---|---|
| Unicode atom classify | NEON (baseline) | AVX-512 VBMI β SSE4.1/SSSE3, runtime-detected | SIMD128 | `classify_scalar` |
| bitstream block build | NEON | SSE/AVX | β | `build_block_scalar` |
| literal & added-token scan | NEON | x86_64 | β | scalar |
| vocab bucket nibble match | NEON | β | β | scalar |
aarch64 selects at compile time (NEON is baseline). x86_64 dispatches at runtime via
`is_x86_feature_detected!`, so one binary covers every x86_64 CPU since 2008. wasm32 needs the
`simd128` target feature; without it, the scalar walk.
# Acknowledgements
This work stands on the shoulders of a lot of open source projects. Thanks to all of them for their great work.
[gigatoken](https://github.com/marcelroed/gigatoken),
[tiktoken](https://crates.io/crates/tiktoken-rs), [kitoken](https://crates.io/crates/kitoken),
[tokie](https://crates.io/crates/tokie), [fastokens](https://crates.io/crates/fastokens),
[wordchipper](https://crates.io/crates/wordchipper) and
[ai-tokenizer](https://www.npmjs.com/package/ai-tokenizer) each pushed on what a fast tokenizer can
be; we read that work, and several of the ideas above reached us because another project showed
they were worth trying.
Docs: [guide](https://huggingface.co/docs/tokenizers/index) Β·
[quicktour](https://huggingface.co/docs/tokenizers/quicktour) Β·
[docs.rs](https://docs.rs/tokenizers/)