TL;DR. We were curious to explore the byte latent transformer and understand better how to train byte latent language models. We adapted Meta’s tokenizer-free Byte-Latent Transformer (BLT) and taught it new languages just by retraining only three small modules—an entropy model, a local encoder and a local decoder ending up training only 4% of total LLM parameters and achieving significant improvements in multi-lingual capabilities of the LLM showing that BLT has an interesting modular property to it that can be further explored in more applied domains.

Why tokenizer-free models?

You probably saw the ever-present issue with LLMs unable to count how many rs in the word strawberry - and the main issue is the way language models see textual information on word and subword level, but not on character or byte level. Existing mechanisms that encode textual information into numbers that are processed by neural networks are efficient, but their fixed vocabularies are optimized for particular data and writing systems. That can skew efficiency across languages and make character level understanding of the underlying information impossible. BLT language model, by operating on bytes, allows a more granular handling of character level information and operations like adjusting text, inserting characters into words, spelling and arithmetic become a lot more accurate.

How Byte Latent 🤖 sees the input?

Here’s the quick tour of how BLT actually reads text. Instead of staring at a fixed vocabulary, it listens to raw UTF-8 bytes and groups them on the fly into “patches,” which are the little chunks it thinks with. A tiny model sits beside it like a metronome, estimating how predictable the next byte will be. When predictability is high, a patch keeps growing; when it drops, BLT snips and starts a fresh patch. The graphic below along with the corresponding plot of entropy illustrates this approach to patching: the dotted line marks the threshold, and each time the entropy trace crosses it a new patch begins.

Example: Adaptive Segmentation

Because the byte sequence is segmented dynamically, BLT can adapt to the structure of the input. For example, when reading “Peter Piper” repeatedly, BLT begins to merge the repeated phrase into a single patch, while a conventional tokenizer continues to cut it into fixed tokens.

BLT's entropy based patching

Llama-3 tokenization

Methodology and Results

When we set out to teach BLT new languages, we aimed to change as little as possible. We left the core transformer untouched and gave each language its own small entropy model so patch boundaries could reflect the quirks of different scripts. We fine-tuned only the local encoder and decoder—together representing roughly 4% of the entire model—while keeping the heavy machinery frozen.

Crucially, since UTF-8 characters can span multiple bytes, we ensured every patch begins at a character boundary. This simple rule is essential when working with non-Latin scripts, preventing BLT from slicing characters in half.

For training, we used HuggingFace's excellent FineWeb-2 dataset for each language, then benchmarked progress using the Belebele benchmark—a multiple-choice reading comprehension test spanning 122 language variants.

The results were encouraging. Across languages, our tiny adapters outperformed the baseline using Meta's original entropy model and local modules—despite touching only a fraction of the parameters. The language-specific entropy models also made inference more efficient: for identical prompts, they created fewer patches than Llama-3 tokenization, resulting in shorter sequences, reduced memory usage, and snappier performance.

Because everything operates on bytes, BLT handles typos and irregular casing better than expected. You even get runtime control: raise the entropy threshold for longer context windows or lower costs; drop it when you need extra precision.

You even get a dial at inference time: raise the entropy threshold when you need longer context windows or lower cost; drop it when you want extra precision.

Discussion & Future Directions

Our exploration demonstrates that BLT's modularity works in practice: a single "brain" can pair with small, language-specific adapters to improve performance without retraining the entire model. This concept extends beyond multilingualism. We envision training domain-specific entropy models and encoders/decoders for programming languages, structured data, or even tone and style control—all while reusing the same latent transformer.

Also, we are hiring! You can learn more about Inflection AI and job opportunities at https://inflection.ai/about

References

[1] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. Advances in neural information processing systems. 2017;30.

[2] Webster JJ, Kit C. Tokenization as the initial phase in NLP. InCOLING 1992 volume 4: The 14th international conference on computational linguistics 1992.

[3] Land S, Bartolo M. Fishing for magikarp: Automatically detecting under-trained tokens in large language models. arXiv preprint arXiv:2405.05417. 2024 May 8.

[4] Kim S, Choi S, Jeong M. Efficient and effective vocabulary expansion towards multilingual large language models. arXiv preprint arXiv:2402.14714. 2024 Feb 22.

[5] Pagnoni A, Pasunuru R, Rodriguez P, Nguyen J, Muller B, Li M, Zhou C, Yu L, Weston J, Zettlemoyer L, Ghosh G. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv:2412.09871. 2024 Dec 13.

[6] Penedo G, Kydlíček H, Sabolčec V, Messmer B, Foroutan N, Kargaran AH, Raffel C, Jaggi M, Von Werra L, Wolf T. FineWeb2: One Pipeline to Scale Them All--Adapting Pre-Training Data Processing to Every Language. arXiv preprint arXiv:2506.20920. 2025 Jun 26.

[7] Bandarkar L, Liang D, Muller B, Artetxe M, Shukla SN, Husa D, Goyal N, Krishnan A, Zettlemoyer L, Khabsa M. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. arXiv preprint arXiv:2308.16884. 2023 Aug 31.