Benchmarking LLMs in Low-Resource Languages — Lessons from Persian

A model that aces MMLU is impressive. But “aces MMLU” means aces it in English. Ask that same model to explain a Persian idiom, reason about Iranian civil law, or analyze a passage from Hafez — and the numbers collapse fast. That gap is the reason I built MELAC.

MELAC (Massive Evaluation of LLMs in Persian) started from a simple question: how well do modern LLMs actually perform in Persian, across the full range of tasks a Persian speaker would care about? Not translated English benchmarks. Not grammar quizzes. Real tasks — comprehension, legal reasoning, cultural knowledge, idiomatic understanding. After 41 models, 19 datasets, and six evaluation categories, the answer was more humbling than I expected. Here’s what I learned.

What “Low-Resource” Actually Means

The term “low-resource” is often misread as “not enough training data.” That’s part of it — but for Persian, a language with over 100 million speakers and a literary tradition stretching back over a thousand years, the resource gap is more nuanced than raw data volume. Persian is well-represented online. The problem runs deeper.

The real challenge is threefold. First, training data bias: most LLMs are overwhelmingly trained on English content, with other languages as an afterthought. Persian content in pretraining corpora is typically a small fraction — and often of lower quality, sourced from scraped web text with encoding inconsistencies, mixed-script artifacts, and transliteration noise that would never appear in professionally edited text.

Second, tokenizer inequity: Persian script is right-to-left with a character set that most tokenizers handle inefficiently. Persian words are routinely fragmented into far more subword tokens than their English equivalents. In practice, this means a Persian prompt consumes 2–4x the tokens of an equivalent English prompt — shrinking effective context windows, increasing inference costs, and degrading performance on tasks that depend on attending across long spans of text.

Third, and most importantly for this work: evaluation blindspots. The benchmarks we use to measure model quality were mostly designed for English. Translated versions introduce systematic artifacts — grammar that sounds unnatural to native speakers, cultural references that don’t map cleanly, concepts that have no clean equivalent. A model that scores 75% on a translated Persian MMLU might still fail completely on tasks any native speaker would consider basic. The benchmark is measuring the wrong thing.

How We Built MELAC

The core design decision for MELAC was to build native datasets, not translate existing ones. Every benchmark was created for Persian speakers, grounded in Iranian cultural and linguistic reality. This matters more than it sounds — it means the evaluation is actually measuring Persian competence, not translation fidelity.

We organized 19 datasets across six evaluation categories:

The linguistic and legal categories were the hardest to build — and, as it turned out, the most revealing. Persian idioms simply don’t translate. Iranian law is a mixture of civil code and Islamic jurisprudence that has no direct equivalent in Western legal systems; a model trained primarily on English legal texts has no scaffold to hang it on.

Forty-one models were evaluated in total: flagship closed-source models (GPT-4o, Gemini 2.0 Flash, Claude families), a range of open-source models of varying sizes, and — critically — several models that had been fine-tuned specifically on Persian data. That last group turned out to be the most interesting part of the study.

What the Numbers Revealed

The headline finding: on culturally-specific datasets, only one of the 41 models we evaluated exceeded 50% accuracy. That model was GPT-4o.

The Persian-specific fine-tuned models sitting at the bottom is not a formatting artifact or a quirk of dataset selection. PersianMind and Maral-7B were built specifically to perform in Persian — trained on Persian data, optimized for Persian tasks. And they were beaten by every major generalist model, often by 30 or more percentage points.

One more counterintuitive finding: there was no strong correlation between a model’s English MMLU performance and its Persian MELAC performance. Models that were close competitors in English showed wildly different trajectories in Persian. The quality and coverage of multilingual pretraining — not raw model capability — is what determines performance in low-resource settings.

What Surprised Me Most

I went into MELAC with several hypotheses. Most were confirmed. Two things genuinely surprised me.

The fine-tuned Persian models surprised me most. I expected them to underperform on general knowledge tasks — that’s the known trade-off when you fine-tune a smaller model on narrow domain data. What I didn’t expect was for them to lose on linguistic tasks too: grammar, morphology, idiomatic expressions — the exact categories where a model trained on Persian text should have a structural advantage. This points to a more fundamental problem than data volume. The Persian-specific training data was either too limited in linguistic diversity, too narrow in register and style, or both — and the smaller base model couldn’t compensate.

The cultural knowledge ceiling was steeper than I expected. For questions requiring knowledge of Iranian culture, history, or social context — not language, just knowledge — even the best models dropped significantly below their general performance. GPT-4o performs impressively overall, but Iranian cultural knowledge is genuinely underrepresented in its training data. What’s revealing is how it fails: the model isn’t confused about what’s being asked. It’s confident. It just has the wrong answer — which is the signature of missing pretraining knowledge, not a prompting problem.

Why This Matters Beyond Persian

Persian is one data point in a pattern that repeats across dozens of languages. The structural problems we encountered — tokenizer inequity, translated benchmark contamination, cultural knowledge gaps, fine-tuning not compensating for pretraining gaps — are not unique to Persian. They’re the default condition for most of the world’s languages.

There’s a deeper issue the field needs to confront: the evaluation problem precedes the training problem. Before we can claim that a model “works” in a language, we need evaluation frameworks that test the right things — natively, not via translation. MELAC-style efforts need to exist for every language that LLMs are being deployed in. Right now, for the vast majority of those languages, they don’t.

The current default in multilingual NLP is to translate English benchmarks and call it adequate evaluation. This is insufficient. A model can score 80% on a translated Persian MMLU and still be completely unable to reason about a common Persian idiom or interpret an Iranian legal document. We don’t actually know how well our best models perform in most of the world’s languages, because we haven’t built the evaluation infrastructure to measure it properly.

The evaluation gap is upstream of the training gap. You can’t close the training gap if you can’t measure it accurately. That’s the argument for MELAC, and for similar work in other languages — not just as academic benchmarks, but as prerequisites for responsible deployment.

The full MELAC paper, including methodology, dataset details, and complete results across all 41 models, is available on arXiv. If you work on multilingual NLP, LLM evaluation, or deploying models for non-English audiences, I’d genuinely like to hear your perspective. Find me on LinkedIn or GitHub.