Adding Bangla, Swahili and Hausa to a sovereign humanitarian assistant

Our humanitarian assistant answers questions about the normative canon of the sector — the Sphere Handbook, the Geneva Conventions, the 1951 Refugee Convention and their siblings — from a corpus we ingest and embed ourselves, behind an egress boundary where every model call is logged, redacted and attributable. The corpus is English. The people who need it often aren't English speakers.

So we decided to add three languages: Bangla, Swahili and Hausa. Together they cover humanitarian contexts from Cox's Bazar to the Sahel. Individually, they are three very different engineering problems — which we only know because we measured each one before building anything.

This is the story of those measurements: what they cost (single-digit euros), what they found, and the working method that made the architecture emerge from the data rather than from a preconceived opinion.

Four layers, four ways to fail

A language isn't "supported" because the model card says so. For a user writing in Bangla, four independent layers have to work:

1. Detect — know which language the message is in. Our existing heuristic, built for a Ukrainian/Russian/English world, classified all three new languages as English.
2. Retrieve — find the right English passages from a query written in another language. Cross-lingual retrieval is the embedding model's job.
3. Generate — answer in the user's language, grounded in English evidence, without reaching for a provider that breaks the sovereignty posture.
4. Redact — catch names, phone numbers and national IDs before anything crosses the perimeter. Fail closed, per language.

Each layer can fail on its own, but it happens that they fail differently per language.

Measure first, rent nothing

Before committing to any architecture (or any GPU), we ran two probes on a serverless, per-token EU inference tier. About 2,900 model calls, less than five euros, no instance ever provisioned.

The probes inherited the evaluation discipline we'd already paid tuition for on the English stack:

• Repeat everything. Our eval has a known noise floor (~4 points on a 100-point scale between identical runs). Any single-run comparison smaller than that is measuring the generator's mood. Every number below is a mean of three runs, with the spread reported.
• Score retrieval separately from generation. Doing it together, it prevents observability.
• Verify your translations. The eval questions were machine-translated, then round-trip-checked by an independent model with a third model judging semantic equivalence. Survival rates: Bangla 32/40, Swahili 25/40, Hausa 22/40 — and one Bangla flag was a silent measles–cholera substitution. That's a finding in itself: machine-translated content needs review, whatever you do about conversation.
• Distrust your judge. The Hausa results were re-judged in full by a second model from a different family. The ranking survived; the absolute numbers moved. Report both.

Finding 1 — Generation: one model is language-uniform

We gave three candidate models the correct English evidence and the question in each language, requiring the answer in that language. This isolates cross-lingual generation from retrieval entirely.

One model — Gemma — brackets its own English score in all three languages. One model fails Hausa outright. And the most instructive failure wasn't in the scores at all: mistral-small answered Bangla questions in English 32% of the time, despite explicit instructions. Its content was fine; its language compliance was not. An LLM judge missed this consistently — a five-line script checking the answer's Unicode script caught it every time. Cheap deterministic checks beat model judges wherever you can write one.

Finding 2 — Retrieval: the mirror image

Then the same questions went against the live corpus as queries, testing whether our embedding model (bge-m3) can retrieve English passages from non-English text. The English control reproduced our production baseline exactly — always validate the harness before believing it.

Bangla retrieves at English parity. Hausa collapses — and machine-translation noise was ruled out by re-scoring on only the verified-clean translations (same result). The embedding model simply hasn't seen enough Hausa, which matches what the African-language retrieval literature (AfriMTEB) reports for this model class.

Put the two findings side by side and the architecture designs itself:

We measured that pivot the same afternoon: translating Hausa queries to English with NLLB-200 (the 1.3B distilled model, on CPU, seconds per query) lifted retrieval from 0.37 MRR to 0.70 — above native Swahili — and cut never-retrieved questions from 15 to 2. A language we'd have called unservable in the morning had a proven end-to-end architecture by the evening, and the fix costs a CPU-seconds translation step on one language's queries.

Finding 3 — Redaction: the traps are all silent

The privacy layer was the biggest gap: the industry-standard PII framework has no models at all for these languages. We added a second analyzer service — transformer NER models trained on African-language and Bangla corpora — while leaving the existing, already-measured analyzer untouched. Two services, so that expanding coverage can never silently regress the thing you've already measured.

Measuring recall before deployment caught three defects, each of which would have been invisible in code review:

1. The redactor masked the word "beneficiary" and leaked the actual name. The Bangla model's tokenizer strips vowel marks; under the default span aggregation the name fragments were dropped in alignment. One config value (aggregation: max) fixed it — but only a recall measurement would ever have shown it.
2. The framework's context booster never fired. ID patterns scored just below the redaction threshold while sitting next to a perfect context word ("NID", "kitambulisho"), because the multilingual tokenizer provides no lemmas for the booster to match. The numbers looked plausible; they were systematically one notch too low.
3. "Common Article 3" became <PERSON_1>. In Hausa, the NER model tagged the citation opener of a Geneva Conventions question as a person — which would have quietly destroyed one of the most important queries in the corpus. Domain-vocabulary allowlists exist for exactly this.

The deployed system now measures 120/120 recall on our labelled corpus — and we publish that number with its caveat attached: the corpus is built from the same format rules the recognizers encode, so part of that score is tautological. When our Ukrainian/Russian redaction met a genuinely held-out test set, ~100% became 84.2%. That's the number we quote for those languages, and the three new languages don't get a public number until their held-out sets exist. A recall figure without its provenance is marketing, not evidence.

Finding 4 — Measurement pays for itself in bugs you weren't looking for

Making the lexical search layer Unicode-aware (it previously dropped every Bengali character), we found the tokenizer had never matched the Ukrainian letter "ї" — a one-character gap in a regex that had been silently fragmenting most Ukrainian words in keyword search since the feature shipped. Nobody noticed, because hybrid retrieval degraded gracefully instead of failing. That bug was found not by a bug report but by a probe refusing to produce a sensible number.

The same pattern recurred all day. Back-translation QA flagged 32 of 40 Hausa translations as broken — until we checked the checker and found the back-translation model was the one misreading Hausa ("toilet" became "distance from home"). Swap the reader, and 22 of 40 survive. Every measurement instrument is itself a thing to measure.

The self-hosting question, finally with a number

The last experiment of the session addressed a different standing question: what does self-hosting the production model actually cost in quality? We served the same model both ways — the hosted API at full precision, and a 4-bit quantized copy on our own GPU — and ran the identical evaluation, three times per arm.

Quantization to int4 costs about 6 points, outside the noise of both arms. That converts a fuzzy architectural debate into a decision rule: don't route quality-sensitive traffic to an int4 arm at price parity. The self-hosted route is sold on what it's actually for — a sovereignty requirement a client pays for — with its measured quality cost stated up front, and FP8 on production-class hardware queued as the parity test.

One more instrumentation lesson: the first run of this comparison reported the delta as inside the noise floor. Twelve answers had scored zero because the judge's API rate-limited mid-run — an infrastructure failure masquerading as a quality signal. Only per-record failure logging made it visible. Re-judge those twelve, and the verdict flips. If your eval can't tell you why a record scored zero, it will eventually lie to you.

What it cost, and what we still don't know

The entire investigation — two probes across four languages and three models, a retrieval pivot experiment, a deployed and verified redaction expansion, and a quantified self-hosting decision — cost single-digit euros in API tokens and zero GPU-hours rented. The expensive-looking question ("do we need a €1,000/month GPU?") was answered for the price of a coffee, and the answer was "not yet, and here is the number that will tell us when."

Just as important is what the evidence does not yet support, because publishing that list is what makes the rest credible:

• The 120/120 redaction recall is a wiring proof, not a production claim — held-out sets per language come first.
• Probe scores are not production scores; gold evidence makes generation easier than the live retrieval loop.
• Translations passed machine QA only; native-speaker review is pending for anything user-facing.
• Detection is measured on our evaluation register, not on colloquial chat, code-switching, or romanised Bangla.
• Every number above is a mean over repeated runs with its spread — and the Hausa generation numbers carry more uncertainty than the others, because even two judges are still just two judges.

The method is the takeaway. Nothing here required rare infrastructure or a research budget. It is based on architectural decisions made from measurements — and treating every surprising number as a question about the instrument before believing it as a fact about the world.

The full evidence pack behind this article — claim ledger, raw results, and the list of what each claim does and doesn't demonstrate — is part of our sovereign agent reference architecture.

As a bonus, we turned the same measure-first pipeline on the site itself: baena.ai is now machine-translated into Bangla, Swahili and Hausa — the same three languages this article is about — so this piece, and the rest of the site, are easier to reach for the people they're actually for.