Tokenizers are a language policy

Every language model begins with a small, unglamorous component that decides how text is cut into pieces. The tokenizer is usually built once, early, and then frozen for the life of the model. It rarely appears in announcements and almost never in product descriptions. Yet it determines, before any learning happens, how much it costs to process a sentence in a given language, how much of a document fits in the model's context, and how easily the model can see the internal structure of words. For languages that were not well represented when the tokenizer was built, those consequences are substantial. We have come to think of tokenizers as a kind of language policy: a set of choices, made mostly implicitly, about whose language is treated as the default.

How subword tokenizers are built

Most modern models use subword tokenization, typically some variant of byte-pair encoding or a unigram language model. The idea is simple and effective. Start from characters or bytes, look at a large training corpus, and repeatedly merge the most frequent adjacent pieces into new vocabulary entries, until the vocabulary reaches a fixed size. Frequent words end up as single tokens. Rare words are spelled out from smaller pieces. The model then learns an embedding for each vocabulary entry.

The critical detail is that frequency is measured on the tokenizer's training corpus. If that corpus is mostly English, the vocabulary will be optimised for English. Common English words, prefixes and suffixes earn their own tokens. Words in other languages earn tokens only to the extent that they are frequent in that same mixture. A language that makes up a small share of the corpus gets a small share of the vocabulary, and its words are broken into more, shorter and less meaningful fragments.

Nothing about this is malicious. It is the natural outcome of optimising for the data at hand. But it is still a choice, and the choice is visible in the output.

What happens to Albanian

Albanian illustrates the problem well. Two features of the language interact badly with English-centric vocabularies.

The first is its characters. The letters ë and ç are among the most common in written Albanian, and neither is part of basic ASCII. In UTF-8 each takes two bytes. A byte-level tokenizer that has seen little Albanian may not have learned to merge those bytes into useful units, so a single letter can consume extra vocabulary capacity or be split in ways that obscure it. There is also a normalization subtlety: ë can be stored either as a single precomposed character or as a plain e followed by a combining diaeresis. The two look identical on screen but tokenize differently, and a pipeline that does not normalize consistently will fragment the language further and see the same word as two unrelated strings.

The second is morphology. Albanian marks definiteness, case and number with suffixes, so a single noun can appear in many forms. Qytet is a city, qyteti is the city, and further endings follow for other cases and for the plural. Verbs inflect richly as well. An English-optimised vocabulary will tend to contain a few frequent Albanian words whole and then carve up everything else at arbitrary boundaries, often not at the boundary between stem and suffix, where a linguist would cut. The model must then learn to reassemble meaning from pieces that do not correspond to anything meaningful.

The standard way to quantify this is fertility, the average number of tokens a tokenizer produces per word in a given language. Published analyses of widely used tokenizers have found that fertility varies considerably across languages, and that languages with diacritics, non-Latin scripts or rich morphology tend to fare worse than English. We are not going to quote a single ratio for Albanian here, because it depends heavily on the tokenizer, the text domain and how words are counted. But the direction is consistent and easy to reproduce.

Why fertility is not just an accounting detail

It would be tempting to treat higher fertility as a matter of price. Most commercial models bill by the token, so a language that needs more tokens to say the same thing costs more to use. That is real, and for individuals and small organisations it is not trivial. But the cost is only the most visible of three effects.

The second effect is on context. A model's context window is measured in tokens, not words. If Albanian text takes more tokens than the equivalent English text, then an Albanian user can fit less of a contract, a textbook or a conversation history into the same window. Long-document tasks, retrieval and multi-turn assistance all degrade in proportion. The Albanian user is, in effect, working with a smaller model.

The third effect is on quality. When words are split into fragments that do not align with their structure, the model has to spend capacity learning to compose them. Sequences are longer, so each prediction depends on more steps. And because rare fragments are shared across unrelated words, their embeddings carry a muddier signal. It is difficult to isolate the tokenizer's contribution to downstream quality, since it is entangled with the amount of training data, but there are good reasons to expect that poor segmentation makes an already data-poor language harder to learn.

A tokenizer decides, before training begins, whose language is cheap and whose is expensive.

The design space

Because these outcomes follow from choices, they can be changed. The main approaches each involve trade-offs.

Balanced sampling when training the tokenizer is the simplest intervention. Rather than learning the vocabulary from the raw distribution of the training corpus, one can upsample smaller languages so that they earn a fairer share of vocabulary entries. This improves fertility for those languages at a modest cost to the largest ones. The difficulty is choosing the balance, since a fixed vocabulary is a zero-sum resource.

Vocabulary extension is the approach most available to those adapting existing models. New tokens for a target language are added to a trained tokenizer, their embeddings are initialised sensibly, often from the embeddings of the pieces they replace, and the model is further trained on text in that language. This can reduce fertility substantially without retraining from scratch. The risks are that the new embeddings need enough data to become useful, and that careless extension can disturb behaviour in other languages.

Byte-level and character-level models avoid the problem of a learned vocabulary altogether by operating directly on bytes or characters. Every language is represented without preference, and there are no out-of-vocabulary words. The price is much longer sequences and therefore more computation, which newer architectures try to reduce by learning to group bytes dynamically. These models are an active research area and an appealing direction for languages like Albanian, though they are not yet the default in widely deployed systems.

Morphologically aware segmentation, which tries to cut words at linguistically meaningful boundaries, is a further option. It can help in languages with rich inflection, but it requires linguistic resources and tends to be brittle on informal text, names and loanwords.

What we measure

Internally, we treat tokenization as something to measure before any claim about Albanian quality is made. Our methods are straightforward, and we describe them here because we think they should be standard practice.

We compute fertility on parallel text, so that languages are compared on the same content rather than on different documents with different vocabulary. We measure it per domain, because news, legal text, informal chat and educational material behave differently, and informal Albanian written without diacritics is a distinct case of its own. We report tokens per character and per byte alongside tokens per word, since word counts depend on conventions. We check how often segment boundaries fall at the boundary between stem and suffix, using hand-annotated samples of inflected forms. We normalize Unicode consistently and test what happens when it is not normalized, because real users will not always send clean input. And we translate fertility into practical terms: the effective context length available to an Albanian user compared with an English one, and the relative cost of the same task in each language.

None of this is conceptually difficult. What matters is doing it routinely and treating the results as design constraints rather than trivia. A model's tokenizer is not neutral infrastructure. It encodes assumptions about which languages matter most, and those assumptions are paid for, token by token, by the people who speak the others. We would rather make that choice deliberately.