🤗 Tokenizers provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

WWW: https://github.com/huggingface/tokenizers
