Building the CharSoup Language Detector

How the tika-langdetect-charsoup model is trained, the decisions baked into it, and how it benchmarks against the alternatives. For architecture and API, see Language Detection.

Corpus paths below are written as <corpus> and <workdir>; substitute your own.

Training Corpus

The primary training data comes from Wikipedia database dumps (dumps.wikimedia.org). Wikipedia is preferred over web-crawl corpora for quality: articles are human-authored, editorial standards filter boilerplate and spam, and the sentence distribution reflects genuine prose rather than SEO content or duplicated web templates.

extract_wiki_sentences.py strips markup, splits into sentences, and writes one lineNum<TAB>sentence file per language directory. For 17 languages with thin Wikipedia coverage, extract_madlad_to_wiki.py writes a parallel sentences_madlad.txt from MADLAD-400 (Magnusson et al., 2023), applying the same quality filters and capping at 500,000 sentences per language. PrepareCorpus reads every *.txt in a language directory:

<corpus>/wikipedia-dumps/
    eng/sentences.txt
    deu/sentences.txt
    mya/
        sentences.txt          (Wikipedia)
        sentences_madlad.txt   (MADLAD supplement)
    ...

MADLAD-supplemented languages: mya, xho, nya, smo, sot, tet, orm, udm, tir, hil, ewe, tso, aka, tsn, ceb, mlg, che.

MADLAD documents encode paragraph boundaries as literal \n escape sequences; the extractor splits on those before processing.

Deduplication

Deduplication happens in the Java training pipeline using FNV-1a 64-bit hashing. One pass is enough — this corpus has a low duplication rate.

Language Code Merging

Several languages have multiple ISO 639-3 codes that refer to the same language or are indistinguishable by character features. These are merged during both download and training:

Merged From Merged To Note

azj (North Azerbaijani)

aze (Azerbaijani)

Code variant

cmn (Mandarin Chinese)

zho (Chinese)

Code variant

ekk (Standard Estonian)

est (Estonian)

Code variant

gug (Paraguayan Guaraní)

grn (Guaraní)

Code variant

lvs (Standard Latvian)

lav (Latvian)

Code variant

nor (Norwegian)

nob (Norwegian Bokmål)

Code variant

pes (Iranian Persian)

fas (Persian)

Code variant

plt (Plateau Malagasy)

mlg (Malagasy)

Code variant

quz (Cusco Quechua)

que (Quechua)

Code variant

swa (Swahili macrolanguage)

swh (Coastal Swahili)

Code variant

yid (Yiddish macrolanguage)

ydd (Eastern Yiddish)

Code variant

zsm (Standard Malay)

msa (Malay)

Code variant

zza (Zazaki)

diq (Dimli)

Code variant

The same merge map is copied into four places and must be kept in sync:

  • CorpusAliases.java — LANG_MERGE_MAP, the canonical definition

  • CommonTokenGenerator.java — LANG_MERGE_MAP, a hand-maintained copy in tika-eval-core (test-scope classes cannot be shared across modules)

  • download_madlad.py — LANG_MERGE_MAP

  • download_corpus.py — LANG_MERGE_MAP

The two Python copies currently omit zza → diq.

Corpus Cleaning

clean_madlad.py applies the generic filters before a sentence ever reaches the Java pipeline: length (50–500 characters), alphabetic ratio (≥ 0.5), a dirty-pattern regex (URLs, wiki markup, ISBN/ISSN, Lorem ipsum), a fastText top-10 contamination filter, and exact dedup against the language’s existing sentences.txt.

PrepareCorpus.java then enforces script consistency, but only for the languages listed in its SCRIPT_CONSISTENCY_LANGS map: a sentence is dropped unless at least SCRIPT_CONSISTENCY_THRESHOLD (0.80) of its letters belong to that language’s expected script. Two per-language carve-outs:

Dhivehi (div) — Maldivian prose naturally mixes Thaana with Arabic script for Islamic phrases, so 0.80 cuts too aggressively; the threshold is overridden to 0.50.

Japanese and Korean (jpn, kor) — legitimately mix some Latin, so instead of the script-consistency rule they are capped by MAX_LATIN_RATIO (0.50).

Language Exclusions

Languages with fewer than 10,000 sentences after deduplication are dropped — below that there is not enough data to survive mislabel filtering.

Some languages clear that bar and are still excluded, by removing the pool file before Pass 2. The decision is made after a full training run, from per-language F1 on the held-out test set and its confusion pattern, on any of three grounds:

  • Interference with a closely related language. When two written forms are nearly identical at the character n-gram level, including both splits probability mass and costs the more widely-used language more than the variant gains. This serves the larger user population; it says nothing about the importance of the excluded language.

  • Own accuracy too low to be useful. A confidently wrong prediction is worse than no prediction.

  • Corpus not representative. Where MADLAD entries are dominated by boilerplate or Lorem Ipsum, the model learns the boilerplate.

Any of these is reversible: restore the pool file and retrain if the corpus improves.

Data Splitting Strategy

Split Size Preprocessing

Test

10% per language, capped by --max-test (default 2,000)

Raw (no preprocessing)

Dev

10% per language, capped by --max-dev (default 2,000)

Preprocessed (NFC, lowercase, URL/email stripped)

Training pool

Remainder

Preprocessed, stored as per-language files

The two caps default differently depending on which tool does the prep: PrepareCorpus defaults both to 2,000, TrainLanguageModel’s own prep step defaults both to 20,000. The 20260320 build ran `PrepareCorpus separately, so 2,000 applied.

Each epoch draws a fresh sample from the pool: binary-search finds a flat cap C such that Σ min(n_i, C) ≈ 5,000,000, then each language contributes up to C sentences, globally shuffled. High-resource languages are capped per epoch; low-resource languages contribute all their data every epoch.

Training Pipeline

Pass 1: Initial Training

AdamW for 2 epochs followed by Hogwild! SGD for up to 6 more (8 epochs total, subject to early stopping), each with epoch-level resampling from the full training pool.

Mislabeled Sentence Filtering

The Pass 1 model predicts each sentence in the entire training pool. Sentences where the prediction does not match the label are removed — unless the prediction falls within the same confusable language group (e.g., a sentence labeled msa predicted as ind is kept).

This filtering is applied once to the full pool, producing a pool_filtered/ directory that is used for Pass 2.

Pass 2: Retraining on Filtered Data

Same optimizer schedule and resampling strategy, but drawing from the filtered pool. This typically improves accuracy by 0.3–0.5 percentage points.

Final Steps

  1. INT8 quantization — convert float32 weights to int8 with per-class scales

  2. Evaluate — test the quantized model on the raw test set (full pipeline)

  3. Export — write the LDM1 binary model file

Confusable Language Groups

Confusable groups are defined only for language pairs where the trained model demonstrably confuses them at meaningful rates on held-out data. Groups are not added speculatively; each entry is backed by observed confusion in evaluation.

Groups are defined in tika-langdetect-charsoup/src/main/resources/org/apache/tika/langdetect/charsoup/confusables.txt and must only contain codes that are actual output classes of the trained model (i.e., present in the training corpus). Dead codes add no benefit.

Current groups:

  • msa / ind — Malay and Indonesian share vocabulary and script so heavily that they cross-predict at roughly 9% at 500 characters.

  • xho / zul — Xhosa and Zulu are both Nguni Bantu languages written in the same Latin-based orthography with very similar character n-gram profiles.

  • bel / be-x-old — the two written standards for Belarusian; grouped to keep training-time contamination filtering from playing them off against each other.

  • yue / zho — Cantonese and Mandarin share Han script, so their character n-gram profiles overlap heavily. FLORES F1 for yue is near zero because FLORES carries only Mandarin text for that script.

These groups are used in:

  1. Training — group-aware mislabel filtering (a sentence labeled msa predicted as ind is not removed as mislabeled)

  2. Inference — probability mass within a group is collapsed to the highest-scoring member before returning a result

  3. Evaluation — within-group predictions count as correct in the group accuracy metric

Common Token Lists (tika-eval)

The same MADLAD corpus is used to generate common token frequency lists for tika-eval. The CommonTokenGenerator in tika-eval-core reads sentences_madlad.txt files and applies:

  • TikaEvalTokenizer in COMMON_TOKENS mode (NFKD normalization, minimum length 3, no numbers, no HTML terms)

  • The same language merge map and FNV-1a deduplication as the training pipeline

The 500k sentences stored per language provide stable frequency estimates for the top-30,000 tokens with a minimum document frequency of 10.

java -cp tika-eval/tika-eval-core/target/test-classes:\
tika-eval/tika-eval-core/target/classes:\
tika-langdetect/tika-langdetect-charsoup/target/classes:\
tika-langdetect/tika-langdetect-charsoup-core/target/classes \
    org.apache.tika.eval.core.tokens.tools.CommonTokenGenerator \
    <corpus>/madlad/data \
    <workdir>/common-tokens \
    30000 10 \
    --model tika-langdetect/tika-langdetect-charsoup-core/src/main/resources/org/apache/tika/langdetect/charsoup/langdetect-20260320.bin

Arguments: <corpusDir> <outputDir> [topN] [minDocFreq] [--model <modelFile>]

The --model flag restricts output to the languages that are actual trained output classes of the given CharSoup model. Without it, all non-excluded languages in the corpus directory are processed — which may include languages that did not survive the mislabel-filtering step and are not in the final model.

CommonTokenGenerator looks for sentences_madlad.txt files inside each language subdirectory.

Current Build: 20260320

langdetect-20260320.bin is the shipped model. Its output classes are listed in CharSoup Supported Languages.

20260320 Training Configuration

  • Corpus: Wikipedia dumps + MADLAD-400 supplements

  • Languages: 204

  • Pool cap: 500,000 sentences per language

  • Feature extractor: SaltedNgramFeatureExtractor — positional-salted character bigrams (BOW/EOW/FULL_WORD/MID), trigrams, 4-grams, CJK character unigrams, script block features (24 script categories, raw counts), short-word-anchored word bigrams (anchor = prev word ≤ 3 chars)

  • Hash buckets: 32,768

  • L2 normalization: enabled

  • Target epoch total: 5,000,000 sentences per epoch

  • Two-pass training: Pass 1 on full pool → mislabel filter → Pass 2

  • JVM: -Xmx8g

  • Model size: ~6.4 MB on disk (INT8 quantized); the ~8.1 MB in the eval log is the used-heap delta measured across the load call, not the retained size

20260320 Corpus Preparation

./mvnw clean compile test-compile \
  -pl tika-langdetect/tika-langdetect-charsoup-core,tika-langdetect/tika-langdetect-charsoup \
  -DskipTests -Dforbiddenapis.skip=true -Dcheckstyle.skip=true

./mvnw -pl tika-langdetect/tika-langdetect-charsoup exec:java \
  -Dexec.mainClass="org.apache.tika.langdetect.charsoup.tools.PrepareCorpus" \
  -Dexec.classpathScope=test \
  -Dexec.args="--corpus <corpus>/wikipedia-dumps \
               --output-dir <workdir>/model-20260320/preprocessed \
               --max-train 500000" \
  -DskipTests -Dforbiddenapis.skip=true -Dcheckstyle.skip=true

20260320 Training

./mvnw -pl tika-langdetect/tika-langdetect-charsoup exec:java \
  -Dexec.mainClass="org.apache.tika.langdetect.charsoup.tools.TrainLanguageModel" \
  -Dexec.classpathScope=test \
  -Dexec.args="--corpus <corpus>/wikipedia-dumps \
               --prep-dir <workdir>/model-20260320/preprocessed \
               --output <workdir>/model-20260320/langdetect-20260320.bin \
               --buckets 32768 \
               --4grams --salted --l2-norm --word-bigrams" \
  -DskipTests -Dforbiddenapis.skip=true -Dcheckstyle.skip=true \
  -Dexec.jvmArgs="-Xmx8g"

Copy the trained model into the resources directory:

cp <workdir>/model-20260320/langdetect-20260320.bin \
   tika-langdetect/tika-langdetect-charsoup-core/src/main/resources/\
org/apache/tika/langdetect/charsoup/langdetect-20260320.bin

Then update MODEL_RESOURCE in CharSoupLanguageDetector to point to the new file, and remove the old binary.

20260320 Evaluation: FLORES-200 Dev Set

Results on the FLORES-200 dev set (997 sentences per language, 203,381 test sentences in total). All scores are macro-averaged F1. Raw eval output: flores-eval-20260320.txt.

Coverage-adjusted accuracy

Each detector scored only on the languages it supports — sentences in languages a detector does not cover are skipped for that detector. This flatters narrow detectors, so read it alongside the breadth-weighted numbers below.

Length CharSoup (204 langs) OpenNLP (~105) Lingua (75) Optimaize (63)

@20 chars

82.51%

74.87%

76.35%

84.87%

@50 chars

94.44%

86.09%

90.99%

94.44%

@100 chars

96.98%

90.25%

95.43%

96.51%

@200 chars

97.45%

91.11%

96.23%

96.75%

full text

97.46%

91.12%

96.25%

96.76%

Breadth-weighted accuracy

All 203 FLORES languages, with unsupported languages scoring 0 — total useful output across all inputs.

Length CharSoup OpenNLP Lingua Optimaize

@20 chars

52.43%

42.05%

27.46%

26.76%

@100 chars

61.63%

50.68%

34.32%

30.43%

full text

61.93%

51.17%

34.61%

30.51%

Head-to-head on shared languages

CharSoup versus each alternative, restricted to the languages both support.

Length vs OpenNLP (105 shared) vs Lingua (71 shared) vs Optimaize (63 shared)

@20 chars

84.23% / 76.69%

85.28% / 77.78%

86.52% / 86.30%

@50 chars

95.73% / 87.27%

96.37% / 92.24%

96.92% / 95.40%

@100 chars

98.09% / 91.05%

98.51% / 96.58%

98.84% / 97.23%

@200 chars

98.51% / 91.77%

98.82% / 97.37%

99.13% / 97.34%

full text

98.51% / 91.78%

98.83% / 97.38%

99.14% / 97.34%

CharSoup leads at every length on every overlap. Optimaize is the closest competitor and covers 63 languages to CharSoup’s 204.

20260320 Resource Usage

Metric CharSoup OpenNLP Lingua (low accuracy) Optimaize

Languages supported

204

~105

75

63

Model heap (load-time used-heap delta)

~8.1 MB

~79.2 MB

~0.1 MB

~94.5 MB

Model file (disk)

6.4 MB

~22 MB

—

—

Throughput (@20)

~139K sent/s

~133K sent/s

~10K sent/s

~248K sent/s

Throughput (full)

~116K sent/s

—

—

—

Runtime dependencies

None

OpenNLP + model

Lingua jar + Kotlin

Optimaize jar
All detectors evaluated with 12 threads on the FLORES-200 dev set (203,381 sentences). Throughput is wall-clock sentences per second.

References

For the academic references behind the techniques used in training and inference, see the References section of the main language detection page.