Tokenizer

BPE (Byte Pair Encoding) is a subword tokenization algorithm widely used in NLP. This implementation uses regex pre-tokenization and iteratively merges the most frequent byte-pair merges to create a vocabulary of subword units.