What it takes to measure an AI in Albanian

It is hard to improve what you cannot measure, and it is easy to fool yourself with measurements that do not mean what they appear to. For Albanian, both problems are acute. There are relatively few evaluation sets written natively in the language, and the most common shortcut, translating an established English benchmark, produces scores that are convenient to report and difficult to interpret. Before we can say anything reliable about how well a model works in Albanian, we need a way to measure it that we trust. This post explains why that is harder than it sounds and describes the approach we are taking to build an Albanian evaluation suite.

Why translated benchmarks mislead

Translation seems like the obvious solution. Take a well-studied English benchmark, translate it carefully, and compare scores across languages on the same questions. The appeal is real: it is fast, it allows direct comparison, and it inherits the credibility of the original. But it introduces at least three distortions, and they tend to compound.

The first is cultural context. English benchmarks are full of assumptions that do not travel. Questions about school grades, units of measurement, legal procedures, holidays, sports, food, public figures and everyday institutions often presuppose American or British life. Translated into Albanian, such a question does not become a question about Albanian life. It becomes a question about American life asked in Albanian. A model can answer it well while knowing very little about Prishtina, Tirana, Tetovo or Shkodër, and a model that knows a great deal about them gains no credit for it.

The second is translationese. Translated text has a recognisable character: sentence structures that follow the source, vocabulary chosen for fidelity rather than naturalness, idioms rendered literally or dropped. Native readers notice it immediately. A benchmark written in translationese measures how well a model handles that register, which may be closer to English than to the Albanian people actually write. It can flatter models that think in English and translate on the way out, and it can penalise answers that are more natural than the question.

The third is contamination. Popular English benchmarks have been online for years, along with their answers, discussions and translations. A model trained on web-scale data may have seen them, in one language or another, and a translated version tests recall of a familiar problem more than competence in Albanian. Because contamination is difficult to rule out, a score on a translated public benchmark carries an uncertainty that is rarely stated.

A translated benchmark tells you how a model handles English ideas in Albanian words.

None of this means translated benchmarks are useless. They can reveal gross failures and support some cross-lingual comparisons. But they cannot be the foundation of a claim that a model is good in Albanian.

Designing evaluations natively

The alternative is to build evaluations in Albanian from the start, written by people who speak the language natively, about things that matter to Albanian speakers. This is slower and more expensive, and it gives up easy comparability with English. We think it is the only way to get measurements that mean what they claim.

Native authorship is the core principle. Tasks are written in Albanian by native speakers, not translated, and reviewed by other native speakers for naturalness and correctness. Writers are briefed to draw on their own experience: the kinds of questions a student, a small business owner, a parent or a civil servant would actually ask. Where a task must be comparable with another language, we prefer to write both versions independently from a shared specification rather than translating one into the other.

Dialect coverage is the second. Albanian has two major dialect groups, Gheg and Tosk, and the standard language is based largely on Tosk. Much of what people actually type, especially in Kosovo, North Macedonia and northern Albania, reflects Gheg, and a great deal of informal writing everywhere omits the diacritics on ë and ç or mixes in words from English, German, Italian or other languages. An evaluation that includes only polished standard Albanian will overstate how well a model serves most real users. We design tasks that span these varieties and report results for each separately, so that a gain on standard text cannot hide a loss elsewhere.

What we want to measure

We think of an Albanian evaluation suite as covering several distinct capabilities, each of which can fail independently.

Language quality comes first: whether the model writes grammatical, idiomatic Albanian in an appropriate register, handles definite forms and case endings correctly, uses ë and ç properly, and avoids the calques and word order that mark text as translated. This is judged largely by human raters, because automatic metrics capture it poorly.

Regional knowledge is second. We want to know whether a model knows the geography, history, institutions, literature, public life and practical realities of Albanian-speaking places, and whether it says so honestly when it does not. Questions here are written to have verifiable answers, and we pay particular attention to confident errors, which are more harmful than acknowledged uncertainty.

Instruction following is third. A model that follows complex instructions reliably in English may lose track of constraints on length, format or tone in Albanian, or quietly answer in English. We test whether instructions given in Albanian are honoured as faithfully as the same kinds of instructions in English, including requests to answer in a particular variety.

Safety in Albanian is fourth, and it is often neglected. Safety training is concentrated in a few languages, and there is a well-documented concern in the research literature that protections can weaken in lower-resource languages. We evaluate whether a model declines harmful requests in Albanian as consistently as in English, and whether it avoids over-refusing ordinary requests simply because they are phrased in a less familiar language. Both failures matter.

Human raters, done carefully

Many of these judgements cannot be automated well, so human rating is central to the suite. It is also where evaluations most often go quietly wrong. We work with native speakers from Albania, Kosovo, North Macedonia and the diaspora, and we aim for panels that reflect that range rather than a single region standing in for all of them.

Raters receive written guidelines with worked examples, and we calibrate them on shared items before they rate independently. We measure agreement between raters and treat low agreement as information about the task, not noise to be averaged away: if careful native speakers disagree about whether an answer is correct, the question is probably ambiguous. Where possible, raters compare outputs side by side without knowing which system produced them. We also watch for the ways dialect can bias judgement, for example a rater marking a correct Gheg form as an error, and we adjust guidelines when we find them.

Automatic scoring still has a role. We use it for tasks with clear answers and to screen large volumes of output. When we use a model as a judge, we check its judgements against human ratings in Albanian specifically, since agreement in English tells us little about agreement in another language.

Where we are, and what we intend to share

Building this suite is ongoing work. We are writing and reviewing tasks, refining rater guidelines, and testing which parts of the design produce stable, informative results. We are deliberately not publishing scores yet, because a number from an unfinished instrument would invite more confidence than it deserves.

We are also thinking carefully about contamination from the start. An evaluation that is published in full will eventually be absorbed into training data and stop measuring what it was built to measure. Our current plan is to keep a held-out portion private and refresh it over time, while being open about how it was constructed.

What we do intend to share is the method: how tasks are specified and written, how dialect coverage is structured, how raters are recruited, briefed and calibrated, and how results are scored and reported. We hope that others working on Albanian, and on other languages in a similar position, will find it useful and tell us where it falls short. Good measurement in a language is a shared resource, and it is the precondition for everything else we want to build.