Retrofitting language models to operate over bytes

MainRecent progress in AI has been driven by end-to-end deep learning systems that learn representations directly from data. Large language models (LLMs) exemplify this trend, achieving strong capabilities by training on massive collections of text3,4. However, despite their apparent generality, contemporary LLMs are not fully end-to-end: before learning can begin, text must first be mapped to a sequence of discrete units called tokens. The choice of tokens, although sometimes overlooked, fundamentally shapes the representations that LLMs learn and the behaviours that they exhibit5,6,7,8,9.The vast majority of contemporary LLMs use words or parts of words as the tokens in a process known as subword tokenization1,10. This leads to many problems. LLMs that use subword tokenization suffer from limited character-level understanding11,12,13, which especially hinders performance with scientific data, such as code and biological sequences2,14,15,16,17; they are also implicitly biased towards generating particular responses based on how the prompt is tokenized18,19,20, they are restricted in the number of words they can incorporate in their vocabulary, which in practice leads to English-centricity6,21, and they potentially suboptimally allocate their compute2,22. These problems have motivated extensive research into alternatives to subword tokenization, most commonly by using the underlying UTF-8 bytes that the text is encoded as23 as the discrete units. Many earlier byte-level LLMs claim to outperform subword-level LLMs on the efficiency–performance Pareto frontier2,22,24,25,26,27. However, in practice, byte-level LLMs have not seen widespread adoption so far, and all leading LLMs still exclusively rely on subword tokenization.We hypothesize that the key reason for this mismatch between theory and practice is that existing approaches to byte-level language modelling focus predominantly on training a new byte-level model from a random initialization and comparing it against a subword-level LLM also trained from a random initialization. By contrast, the training of state-of-the-art subword-level LLMs is rapidly evolving, combining innovations in training data curation, model architecture and post-training. Keeping up with this pace is infeasible for byte-level LLM development without extensive investments.To resolve this mismatch, we introduce a general method for creating byte-level LLMs that achieve performance close to that of state-of-the-art subword-level LLMs across various tasks, while vastly surpassing earlier byte-level LLMs with a comparable parameter count. In contrast to earlier byte-level LLMs that focus predominantly on training from random initialization, we retrofitted an existing subword-level LLM to the byte level using less than 1% of a typical pretraining budget (49.1B tokens in total, where B represents a billion or 109). We refer to this process as byteification. Byteification establishes a connection between existing subword-level LLMs and byte-level LLMs. This let us train the byteified models Bolmo 7B and Bolmo 1B by starting from the existing fully open LLMs Olmo 3 7B28 and OLMo 2 1B29, Bwen 8B by byteifying Qwen3 8B Base30, and Blama 8B by byteifying Llama 3 8B31.Our architecture first aggregates the byte stream into a sequence of patches, where each patch itself consists of one or more bytes. The patches are then processed by a large transformer32 language model and finally depooled into bytes. Owing to its latent tokenization into byte patches, we refer to this style of architecture as a latent tokenizer language model (LTLM). Examples of LTLMs include the DTP24, BLT22 and H-Net2 models. However, in contrast to earlier work, we specifically designed our architecture to be well suited to byteification (section ‘Byteified language model architecture’). In particular, we resolved a mismatch between the expressivity of subword tokenization and the latent LTLM tokenization. Alongside an efficient two-stage training procedure (section ‘Byteification training procedure’), this allowed us to quickly recover and, in some cases, surpass the performance of the source subword-level LLM. We believe that byteifying provides a key missing research direction by enabling the creation of state-of-the-art byte-level LLMs without extensive investment. This approach is complementary to training from random initialization. Making it cheap to byteify any subword model can quickly unveil high-performing architectures that are promising candidates for training from random initialization as byte-level LLMs.Our byteified models outperformed, on average, all earlier publicly available byte-level LLMs of comparable size. For example, Bolmo 7B achieved a +16.5% absolute improvement in STEM tasks over BLT 7B, which was trained from random initialization. Bolmo 7B also greatly outperformed the source Olmo 3 on character understanding and was better in certain coding settings. Bwen 8B outperformed Bolmo 7B, exhibiting comparable trends and reaching performance close to and sometimes surpassing the source Qwen model. Furthermore, we found that the byteified models can arbitrarily be further sped up by training with higher ratios of bytes per patch, which is possible only to a limited extent in subword-level LLMs. Finally, we show that existing components in the source LLM ecosystem can be used to adapt a byteified model without any extra training cost; this could further accelerate research on byte-level LLMs.In aggregate, our results show that byte-level LLMs offer substantial promise as a foundation for future language models, with potential advantages for computational efficiency by reducing energy and deployment costs, for reducing biases introduced by English-centric subword tokenization and for enabling applications that require fine-grained textual understanding, particularly in scientific and technical domains.Byteified language model architectureFollowing the same overall structure as earlier LTLMs (‘Related work’ in Methods), the byteified models can be formalized as shown in Fig. 1. In a nutshell, a local encoder model operating over bytes first creates contextualized byte representations. A neural boundary predictor module then predicts, based on these representations, the byte positions at which to place a boundary. The boundary information is used to derive a sequence of patches: a patch is the contiguous sequence of bytes between consecutive boundaries (including the byte where the end boundary is placed). A pooling module then reduces the contextualized byte representations to a single representation per patch, which is passed through a deep global model. The patch representations contextualized through the global model are depooled back to the byte level and further contextualized through a local decoder model. Finally, the local decoder representations are used to predict the next byte and any other potential information. We chose the local models to be shallow but wide as we found that the embeddings of the original model typically have a high effective rank (Extended Data Fig. 1), and we chose matrix long short-term memory (mLSTM) layers33 to contextualize local information as they retained high performance at high throughput in our experiments (Extended Data Fig. 3). See ‘Byteified model architecture details’ in Methods for a detailed description of the architecture. Our primary departure from earlier architectures lies in the non-causal patch boundary prediction. The boundary predictor of earlier LTLMs uses only the past context to decide on whether to place a boundary (it is causal, referred to as incremental by Pagnoni et al.22). At a glance, this seems necessary: we are aiming to predict the next byte, so we must not leak any information about it. However, although subword-level LLMs use a causality constraint over the subword tokens, the subword tokens themselves do not depend exclusively on past context: subword tokenizers use information about future bytes to place token boundaries. To see this, let us interpret our subword tokenizer as a function that decides whether to place a token boundary after any byte: \({\mathcal{B}}(x):{\{0,1,\ldots ,255\}}^{n}\to {\{0,1\}}^{n}\), where 1 at the nth place denotes a token boundary predicted after the nth byte. Let us assume (1) a vocabulary of English words and subwords, (2) the example text ‘_Hello_Wor!’, which would typically be tokenized as {‘_Hello’, ‘_Wor’, ‘!’}, and (3) the position i = ∣‘_Hello_Wor’∣ − 1 = 9. \({\mathcal{B}}\) (‘_Hello_Wor!’)i = 1, as there is a boundary after ‘r’. However, in the text ‘_Hello_World!’, which would be tokenized as {‘_Hello’, ‘_World’, ‘!’}, we have \({\mathcal{B}}\) (‘_Hello_World!’)i = 0, despite ‘_Hello_Wor!’[:i + 1] = ‘_Hello_World!’[:i + 1] = ‘_Hello_Wor’. In other words, although the subword-level LLM uses only past subword tokens to predict the next subword token, the subword tokens themselves are created by taking future context into account. In this case, this means deciding that ‘_Wor’ should be a token in one case but not in the other, although the text up until that point is equivalent. Current LTLMs, in contrast, cannot take future context into account. This creates a mismatch between the expressivity of LTLM boundary predictors and subword tokenizers. We modified the boundary predictor to resolve this mismatch. In particular, although earlier boundary predictors can be written as $${\mathcal{B}}{(\hat{e})}_{t}:= \,f({\hat{e}}_{0},{\hat{e}}_{1},\ldots ,{\hat{e}}_{t}),$$with f a function mapping the byte representations \(\hat{e}\) to boundary scores by using context up to the current byte representation \({\hat{e}}_{t}\) (where t is the index of the current byte), we set our boundary predictor to $${{\mathcal{B}}}_{\mathrm{byteify}}{(\hat{e})}_{t}:= \,f({\hat{e}}_{0},{\hat{e}}_{1},\ldots ,{\hat{e}}_{t},{\hat{e}}_{t+1}).$$Fig. 1: Our byteified model architecture and boundary prediction procedure in