Text Quality Scoring (Junk Detection)

Table of Contents

What it detects
How it works
Using the API
Thresholds and operating points
Limitations
Further reading

The tika-ml-junkdetect module provides a language-agnostic scorer that distinguishes clean natural-language text from garbled, corrupted, or mis-decoded content — without needing to know the language in advance.

What it detects

Mojibake — text decoded with the wrong character set (e.g., a Windows-1251 Russian document decoded as Windows-1252, producing Latin lookalike garbage)
Byte-level corruption — random or partially-overwritten byte sequences that produce structurally invalid UTF-8
Reversed or shuffled text — text that contains valid characters but in nonsensical order, as can occur in bidirectional rendering failures or corrupted OCR streams
OCR garbage — low-confidence OCR output full of symbol noise

It does not detect incorrect language (e.g., an English document mistakenly labeled as French) — use Language Detection for that.

How it works

The model is trained on clean Wikipedia and MADLAD-400 text. For each input it:

Buckets every codepoint bigram to its Unicode script (Latin, Cyrillic, Arabic, Han, …), folding COMMON/INHERITED glue into the adjacent script, and scores each against that script’s own codepoint-bigram table. Per-script scores are combined count-weighted, so mixed-script text is scored fairly rather than by picking one script.
Adds document-level features: Unicode block transitions, control-byte fraction, script transitions, letter-adjacent-to-mark ratio, replacement-character ratio, script density, coherence, and alternation.
Combines them with one global linear combiner into a single score.

The combined score is the primary output, and it is a logit, not a per-script z-score: positive means clean, negative means junk, and the natural threshold is 0. getPClean() is sigmoid(score).

See Building the Junk Detector for the feature list, calibration, and model format.

Using the API

The public interface is TextQualityDetector in tika-core. The implementation lives in tika-ml-junkdetect, which registers itself via the Java ServiceLoader mechanism.

Add the dependency to your project:

<dependency>
  <groupId>org.apache.tika</groupId>
  <artifactId>tika-ml-junkdetect</artifactId>
  <version>4.0.0</version>
</dependency>

Loading the detector

// Via ServiceLoader — picks up any registered TextQualityDetector implementation
TextQualityDetector detector = ServiceLoader.load(TextQualityDetector.class)
        .findFirst()
        .orElseThrow(() -> new IllegalStateException("No TextQualityDetector on classpath"));

// Or directly, when you know you want JunkDetector specifically
JunkDetector detector = JunkDetector.loadFromClasspath();

JunkDetector is immutable and thread-safe after construction. Load it once at application startup.

Scoring a string

TextQualityScore score = detector.score("The quick brown fox jumps over the lazy dog.");
System.out.println(score.getZScore());   // combiner logit; > 0 means clean
System.out.println(score.getPClean());   // sigmoid of the same value

Interpreting the score

Score	Interpretation
> 0	Clean. The natural decision boundary.
0 to −2	Leaning junk but not decisively. Noisy OCR, code-heavy text, or unusual domain language lands here.
< −2	A conservative junk threshold. Reasonable trigger for re-OCR or re-decoding.
< −5	Almost certainly garbled: wrong charset, reversed content, or heavy corruption.

Score

Interpretation

> 0

Clean. The natural decision boundary.

0 to −2

Leaning junk but not decisively. Noisy OCR, code-heavy text, or unusual domain language lands here.

< −2

A conservative junk threshold. Reasonable trigger for re-OCR or re-decoding.

< −5

Almost certainly garbled: wrong charset, reversed content, or heavy corruption.

The TextQualityScore also carries:

getPClean() — sigmoid(score). The combiner is a fitted logistic, so this is usable for ranking; do not read it as a precisely calibrated probability.
getCiLow() / getCiHigh() — the score plus/minus 1.96 × sigma / sqrt(bigrams) for the dominant script. Narrow on long texts, wide on short ones; use them for threshold decisions on short strings.
getDominantScript() — the Unicode script name that carried the scoring (e.g. "LATIN", "CYRILLIC", "HAN"), or "NONE" when no scoreable letter was found. When isUnknown() is true, no script in the input had a model and no score is available.

Comparing two candidates

The compare() method is the primary use case for charset detection: given the same raw bytes decoded two different ways, which decoding looks more like natural language?

The caller is responsible for decoding the raw bytes; the detector just compares the resulting strings. Each candidate is given a human-readable label (typically the charset name) that is echoed back in the result.

byte[] rawBytes = ...; // bytes from an unknown-encoding file

String ascp1252 = new String(rawBytes, Charset.forName("cp1252"));
String ascp1251 = new String(rawBytes, Charset.forName("cp1251"));

TextQualityComparison result = detector.compare("cp1252", ascp1252, "cp1251", ascp1251);

System.out.println(result.winner());  // the winning label: "cp1252" or "cp1251"
System.out.println(result.delta());   // score separation between the two

if ("cp1251".equals(result.winner()) && result.delta() > 1.0) {
    // cp1251 is confidently the better decoding
}

An isUnknown() candidate is treated as neutral (0), not -∞: a decoding whose script the model does not know must not lose to a garbled-but-recognisable one.

delta() is the absolute score difference between the two candidates. As a rough guide:

Delta	Confidence
< 0.5	Very uncertain — both decodings look similar to the model. Fall back to other heuristics.
0.5 – 1.0	Weak signal — winner is likely correct but not assured.
1.0 – 3.0	Useful signal. Trust the winner for most production purposes.
> 3.0	High confidence. One decoding is clearly more language-like.

Delta

Confidence

< 0.5

Very uncertain — both decodings look similar to the model. Fall back to other heuristics.

0.5 – 1.0

Weak signal — winner is likely correct but not assured.

1.0 – 3.0

Useful signal. Trust the winner for most production purposes.

> 3.0

High confidence. One decoding is clearly more language-like.

Listing known scripts

JunkDetector.knownScripts() returns the Set<String> of scripts with a trained table — 72 in the bundled model, covering the major living scripts plus a long tail of historic ones. It is on JunkDetector, not on the TextQualityDetector interface, so reach for it only when you have the concrete class. If none of an input’s scripts is in that set, score() returns a TextQualityScore with isUnknown() true and no usable score.

Thresholds and operating points

There is no universally correct threshold; it depends on your content and your tolerance for flagging good text as junk. Starting points:

Trigger re-OCR: score < −2.0.
Charset tiebreaking: take the higher-scoring candidate when delta() > 1.0; abstain below 0.5.
Training-data filtering: score < −1.5 to strip mojibake and bot noise from an NLP corpus.

Under ~50 UTF-8 bytes, threshold on getCiLow() rather than getZScore() — the interval widens substantially there.

Limitations

Script coverage — only scripts with a trained table can be scored; others return isUnknown().
Short text — unreliable below ~15 UTF-8 bytes; there are too few bigrams for a stable estimate.
Closely related charsets within one script — the LATIN table pools hundreds of languages, which dilutes the signal between neighbours like cp1252 and cp1257 on Lithuanian text. The winner is usually right, but delta() may be < 0.5.
Deliberately obfuscated text — content padded to look like natural language is not detected.