How we evaluate synthetic voices

Evaluating a speech model sounds straightforward: play the audio to people and ask how good it is. In practice, it is one of the hardest parts of the work. Voices that score well in a short listening test can fall apart on a long paragraph, on a phone number, or on a sentence that switches languages halfway through. A model can improve on average while getting noticeably worse at the things a particular group of users cares about. We have come to believe that how you evaluate a voice shapes what you end up building, so we try to be deliberate about it.

The limits of listening scores

The most familiar tool in speech evaluation is the mean opinion score: listeners rate samples on a five-point scale, and the ratings are averaged. It is useful, and we use variants of it. But it has well-known weaknesses that are easy to forget once a number is on a chart.

Ratings depend heavily on context. The same sample will be scored differently depending on what else a listener heard in the session, whether a real recording was included as a reference, and how the question was phrased. Scores from different studies are rarely comparable, and small differences between strong systems often sit within the noise. As systems approach the quality of natural recordings, listeners struggle to separate them on an absolute scale at all.

Listening tests also tend to use short, clean sentences, because long passages are tiring to rate. That biases evaluation toward exactly the conditions where modern models are strongest. Problems that emerge over minutes, such as drifting pitch, monotony or a voice subtly changing identity, rarely appear in a ten-second clip.

For these reasons we lean more on comparative tests, where listeners hear two versions of the same text and say which they prefer, or rate them side by side. Preference judgments are more sensitive to small differences and less affected by drift in how a listener uses a scale. They still do not tell you why one sample is better, which is why they are only one part of the picture.

Intelligibility and the ASR round-trip

Before a voice can be natural, it has to be correct. A useful automated check is to run synthesized audio through a speech recognition system and compare the transcript to the text the model was asked to say. Words that are dropped, repeated or garbled show up as errors. This is cheap enough to run on thousands of sentences, which makes it good at catching the rare but severe failures, such as skipped phrases or repeated loops, that token-based models occasionally produce.

The round-trip has limitations we try to keep in view. A recognizer has its own biases and blind spots, and it can be more forgiving or harsher than a human listener. For a lower-resource language like Albanian, recognition systems are themselves less mature, so a high error rate might reflect the recognizer rather than the voice. And a recognizer will often transcribe a mispronounced name correctly because it knows the word, hiding exactly the kind of error a native speaker would hear immediately. We treat the round-trip as a tripwire for gross failures, not as a measure of quality.

A voice that improves on average can still get worse at the things a particular group of users cares about.

Prosody and naturalness

Much of what makes speech sound human lives in prosody: where the emphasis falls, how pitch moves across a sentence, how long the pauses are between clauses. It is also where synthetic speech most often sounds subtly wrong. A model can pronounce every word correctly and still read a question as a statement, or give equal weight to every word in a sentence that has one word that matters.

Automated measures here are imperfect. We can compare pitch contours and durations against recordings of the same text, but there are many acceptable ways to read a sentence, and a model that departs from one reference may still be reading it well. So we pair these measures with targeted listening tasks: sentences designed so that the meaning depends on emphasis, lists that need the right phrasing, questions without question marks, and longer passages where listeners are asked specifically about monotony and pacing rather than overall quality.

Names, numbers and code-switching

Some of the most consequential errors come from a small set of difficult categories, and we evaluate them directly rather than hoping they show up in a general sample. Numbers are one: dates, times, prices, phone numbers, measurements and ordinals each have their own spoken conventions, and those conventions differ between languages. Abbreviations and symbols are another, along with addresses, email addresses and fragments of code.

Names are a category of their own. People notice immediately when their name, or the name of a place they know, is said wrongly, and names are exactly where both text normalization and learned pronunciation are weakest. We maintain lists of personal names, place names and brand names, with particular attention to Albanian names and names from the region, and we check them as a distinct set.

Code-switching is the third. Everyday speech in Albanian, as in many languages, mixes in English words and phrases, especially for technology. A good voice should say an English product name the way a bilingual speaker would inside an Albanian sentence, not in a jarring shift of accent and not flattened into Albanian phonology when that is not how people say it. There is rarely a single correct answer, which is part of why this needs human judgment.

Native-speaker panels

For the qualities that matter most, there is no substitute for native speakers. Non-native listeners, however attentive, miss the details that make a voice sound foreign to someone who grew up with the language: a stress pattern slightly off, a vowel from the wrong dialect, an intonation that marks a question in one region and not in another.

We're building panels of native Albanian speakers from different regions, including speakers of both Gheg and Tosk varieties, because a voice that sounds natural to one group can sound odd to the other. We ask panellists not just to rate but to describe what they hear, and those descriptions are often more useful than the ratings. A comment that a voice sounds like a news anchor reading a children's story tells us something no score can. We also try to pay attention to disagreement between listeners, which often signals a genuine ambiguity worth understanding rather than noise to be averaged away.

Regression suites and why one number misleads

Every change to a model, its data or its serving setup can quietly break something that used to work. To catch this, we're assembling a regression suite: a fixed collection of difficult inputs covering the categories above, long passages, edge cases that failed in the past, and sentences that exercise streaming boundaries. Each candidate model is run against the suite, with automated checks flagging changes and a subset reviewed by listeners. A new model is not an improvement until it has been shown not to regress on the things we already fixed.

All of this points to our central conviction about evaluation: a single number misleads. An average listening score collapses naturalness, intelligibility, prosody, pronunciation and robustness into one figure, and in doing so it lets a gain in one dimension hide a loss in another. It also averages over users, so improvements for the majority can mask regressions for a minority whose language or dialect is underrepresented in the test set. For a lab that cares specifically about serving Albanian speakers well, that failure mode is not hypothetical.

So we report evaluations as profiles rather than scores: a set of measures, each tied to a question we care about, examined together. It is slower and less satisfying than a single headline number. But it keeps us honest about what a model is actually good at, and it gives us a much clearer sense of where to work next.