Building the CharSoup Language Detector

Deduplication is performed at the Java training-pipeline stage using FNV-1a 64-bit hashing. Web-crawl data has lower duplication rates than Leipzig news corpora, so a single deduplication pass is sufficient.

Several languages have multiple ISO 639-3 codes that refer to the same language or are indistinguishable by character features. These are merged during both download and training:

The same merge map is maintained in three places and must be kept in sync:

Two data-quality issues were identified during development and addressed in CorpusReader.java:

Breton (bre) noise — approximately 5% of MADLAD Breton sentences were French blog posts, identifiable by lines containing three or more consecutive tildes (_~), a Common Crawl redaction marker. A filter discards any sentence containing this pattern.

Dhivehi (div) mixed-script headlines — MADLAD Dhivehi documents consistently begin with a Latin-script headline followed by Thaana-script body text, separated by a literal \n escape sequence. The pipeline splits on this separator and treats each segment as a distinct sentence, preventing the Latin headline from polluting the Thaana training signal. This raised 20-character accuracy from 32.9% to 95.5%.

Languages with fewer than 10,000 sentences after deduplication are excluded. This threshold ensures enough data for the model to learn useful distributions even after the mislabel-filtering step.

Some languages meet the 10,000-sentence minimum but are explicitly excluded. Exclusion decisions are made after a full training run by evaluating per-language F1 on the held-out test set and inspecting confusion patterns. A language is excluded if it falls into one or more of these categories:

Accuracy interference with a closely related language. When a language’s written form is nearly identical to a more widely-used language at the character n-gram level, including both causes the model to split probability mass between them. This depresses accuracy for the more widely-used language by an amount that exceeds any benefit from detecting the variant. The exclusion is a deliberate choice to serve the larger user population; it does not reflect on the importance or validity of the excluded language.

Own accuracy below a useful threshold. If the model cannot achieve reliable accuracy for a language — because its character profile overlaps too heavily with other included languages — then returning a prediction for it does more harm than good. A confidently wrong prediction is worse than no prediction.

Training corpus not representative of natural language use. Some languages have MADLAD corpus entries dominated by boilerplate, Lorem Ipsum placeholder text, or other non-natural content. A model trained on such data learns the boilerplate rather than the language, producing unreliable results in real-world documents.

These exclusions are applied by removing the language’s pool file before Pass 2 training. If future corpus improvements make a language reliably distinguishable, it can be reintroduced by adding its pool file back and retraining.

AdamW for 2 epochs followed by Hogwild! SGD for up to 3 more epochs, each with epoch-level resampling from the full training pool.

The Pass 1 model predicts each sentence in the entire training pool. Sentences where the prediction does not match the label are removed — unless the prediction falls within the same confusable language group (e.g., a sentence labeled msa predicted as ind is kept).

This filtering is applied once to the full pool, producing a pool_filtered/ directory that is used for Pass 2.

Same optimizer schedule and resampling strategy, but drawing from the filtered pool. This typically improves accuracy by 0.3–0.5 percentage points.

Copy the trained model into the resources directory:

Then update MODEL_RESOURCE in CharSoupLanguageDetector to point to the new file, and remove the old binary.

Results on the FLORES-200 dev set (204 test languages, 997 sentences each). All scores are macro-averaged F1. Raw eval output: flores-eval-20260320.txt.