Class JunkDetector

java.lang.Object
org.apache.tika.ml.junkdetect.JunkDetector
All Implemented Interfaces:
TextQualityDetector
public final class JunkDetector extends Object implements TextQualityDetector
Language-agnostic text quality scorer. Discriminates clean UTF-8 text from mojibake, reversed text, wrong-codec decodings, and other corruption forms.

Scoring combines several features:

  1. Codepoint-bigram log-probability (z1) — every bigram is bucketed to its script (forEachScriptBigram(int[], JunkDetector.BigramSink)) and scored against that script's open-addressed bigram table; unseen pairs fall back to a unigram independence-assumption score α * (log P(cp_a) + log P(cp_b)). This is the only per-script feature.
  2. Unicode block-transition log-probability (z2) — a single global table over Character.UnicodeBlock values.
  3. Control-byte fraction (z3) — fraction of bytes in control ranges [0x01–0x08, 0x0B, 0x0C, 0x0E–0x1F, 0x7F].
  4. Script-transition log-probability (z4) — a single global table over raw Character.UnicodeScript values.
  5. Document-level quality features z5..z9 (letter-adjacent-to-mark, replacement ratio, script density, script coherence, script alternation).

Per-script z1 is calibrated (mu/sigma) on held-out dev text; z2..z9 are global. The z-scores are combined by a single global linear combiner: logit = w1*z1 + ... + w9*z9 + bias, fit on clean vs. corrupted windows. Natural junk threshold is 0 (positive logit = clean); use negative thresholds for conservative detection.

Model file format: a single binary spec (see load(InputStream) javadoc). No backwards-compat fallback to older formats — the loader rejects mismatched version bytes with a clear error. This is intentional: keeping parallel scoring paths is a known source of silent miscalibration bugs.

Instances are immutable and thread-safe after construction.

Typical usage: 

JunkDetector detector = JunkDetector.loadFromClasspath();
TextQualityScore score = detector.score("some text");
if (score.getZScore() < 0) { ... flag as junk ... }

// Arbitrate between two charset decodings
TextQualityComparison result = detector.compare("cp1252", ascp1252, "cp1251", ascp1251);
String winner = result.winner();  // returns "cp1252" or "cp1251"

  • Field Details

    • DEFAULT_MODEL_RESOURCE

      public static final String DEFAULT_MODEL_RESOURCE
      Classpath resource path for the bundled production model.
      See Also:

    • VERSION

      public static final int VERSION
      Sole supported file-format version. Mismatch is a hard error — prior versions live in git history and are not loadable by this build. We deliberately don't keep dual-version paths so it's impossible to confuse model versions.
      See Also:

    • COMMON_SCRIPT

      public static final String COMMON_SCRIPT
      Model script key for the pooled COMMON (digits/punctuation/symbols) table.
      See Also:

    • TOKEN_START

      public static final int TOKEN_START
      Sentinel "codepoints" (above the Unicode maximum U+10FFFF, so they cannot collide with real text) that wrap each letter token: TOKEN_START (^) before the first letter of a run and TOKEN_END ($) after the last. Emitted by forEachScriptBigram(int[], JunkDetector.BigramSink) so the bigram LM learns word-initial / word- final letter typicality, and so z1 never empties for text containing even a single letter.
      See Also:

    • TOKEN_END

      public static final int TOKEN_END
      See Also:

  • Method Details

    • loadFromClasspath

      public static JunkDetector loadFromClasspath() throws IOException
      Loads the bundled model from the classpath.
      Throws:
      IOException - if the model resource is missing or malformed

    • provider

      public static JunkDetector provider()
      ServiceLoader provider hook (Java 9+). Allows JunkDetector to be registered as a TextQualityDetector SPI implementation even though its construction goes through loadFromClasspath() rather than a public no-arg constructor.

    • loadFromPath

      public static JunkDetector loadFromPath(Path path) throws IOException
      Loads a model from the given file path. The file may be gzipped or raw.
      Throws:
      IOException

    • load

      public static JunkDetector load(InputStream rawIs) throws IOException
      Loads a model from an InputStream. Gzip-detection is automatic. Strictly requires the current file-format version (15) — older formats are rejected with a clear error rather than supported via a fallback path.

      File-format layout (gzipped): 

        [8 bytes]    magic "JUNKDET1" (ASCII)
  [1 byte]     version (= 15)
  [4 bytes]    num_scripts (int BE)
  [1 byte]     block_scheme_version  (must equal
               UnicodeBlockRanges.SCHEME_VERSION)
  // z4 — global script-transition section
  [1 byte]     num_script_buckets
  for each bucket:
    [2 bytes]      name length (ushort BE)
    [name bytes]   bucket name (UTF-8)
  [4 bytes]    scriptTrans_quant_min (float32 BE)
  [4 bytes]    scriptTrans_quant_max (float32 BE)
  [num_script_buckets² × 2 bytes]  script-transition table (z4, int16-quantized)
  [4 bytes]    mu4 (z4 calibration, float32 BE)
  [4 bytes]    sigma4
  // z2 — global block-transition section
  [4 bytes]    block_quant_min (float32 BE)
  [4 bytes]    block_quant_max (float32 BE)
  [block_N² × 2 bytes]  block-transition table (z2, int16-quantized)
  [4 bytes]    mu2 (z2 calibration)
  [4 bytes]    sigma2
  // global per-feature calibrations, {mu, sigma} float32 pairs
  [8 bytes]    z3 calibration (control-byte ratio)
  [8 bytes]    z5 calibration (letter-adjacent-to-mark)
  [8 bytes]    z6 calibration (replacement-char ratio)
  [8 bytes]    z9 calibration (script-alternation)
  // global combiner
  [1 byte]     num_features
  [(num_features+1) × 4 bytes]  combiner weights w1..wN and bias
  // per-script section
  for each script (sorted by name):
    [2 bytes]      name length
    [name bytes]   script name (UTF-8)
    [4 bytes]      mu1 (z1 calibration, codepoint-bigram mean log-prob)
    [4 bytes]      sigma1
    [variable]     bigram + unigram tables — exact layout in
                   BigramTables.writeTo(DataOutputStream): codepoint index, then the
                   sorted-occupied bigram keys (key[0] as int32 BE followed
                   by LEB128 varint deltas) and 8-bit quantized bigram and
                   unigram log-prob values
      Throws:
      IOException

    • score

      public TextQualityScore score(String text)
      Scores the given string for text quality.

      The text is split into contiguous runs of the same Unicode script. Each run is scored against its own script model. Logits are combined as a byte-count-weighted average, so mixed-script text (e.g. half LATIN, half HAN) is scored fairly without arbitrarily picking one script. COMMON, INHERITED, and UNKNOWN codepoints (spaces, punctuation, digits) are attached to the preceding script run.

      Specified by:
      score in interface TextQualityDetector
      Parameters:
      text - the string to score; must not be null
      Returns:
      a TextQualityScore; check TextQualityScore.isUnknown() if the input is empty or the script is not covered by the model

    • compare

      public TextQualityComparison compare(String labelA, String candidateA, String labelB, String candidateB)
      Compares two candidate strings and returns which is higher-quality (cleaner text).

      A common use case is charset-decoding arbitration: given raw bytes decoded via two different charsets, pass each decoded string here with a human-readable label (e.g. the charset name) and the detector will pick the one that looks more like natural language.

      Each candidate is scored independently via score(String). The candidate with the higher score wins.

      An UNKNOWN score (script not in model) is treated as neutral (0) rather than -∞. This prevents a garbled-but-recognisable decoding from beating a correct decoding whose script happens to be unknown to the model — for example, a pure-katakana zip entry name decoded as Shift-JIS (UNKNOWN) vs. the same bytes decoded as UTF-8 (garbled LATIN, negative z-score).

      Specified by:
      compare in interface TextQualityDetector
      Parameters:
      labelA - human-readable label for candidate A (e.g. "cp1252")
      candidateA - first candidate string
      labelB - human-readable label for candidate B (e.g. "cp1251")
      candidateB - second candidate string
      Returns:
      a TextQualityComparison with the winning label and confidence delta

    • knownScripts

      public Set<String> knownScripts()
      Returns the set of script names this model knows about.

    • getModelVersion

      public int getModelVersion()
      Returns the file-format version of the loaded model (always VERSION; mismatches are rejected at load time).

    • scoreWithFeatureComponents

      public JunkDetector.FeatureComponents scoreWithFeatureComponents(String text)
      Diagnostic — exposes the per-feature z-scores, the global combiner weights, and the dominant script. Same math as score(String).

    • computeZ5LetterAdjacentToMarkRatio

      public float computeZ5LetterAdjacentToMarkRatio(String text)
      z5: calibrated letter-adjacent-to-mark ratio. Delegates raw computation to TextQualityFeatures.letterAdjacentToMarkRatio(String) and applies the document-level (mu, sigma) calibration loaded from the model file. Returns 0 (neutral) when the model has no z5 calibration or when the raw value is NaN.

      Positive z5 = correct decoding of a precomposed-or-decomposed script (Vietnamese, Indic, Thai, Arabic). Negative z5 = mojibake of such content as Latin-1.

    • computeZ6ReplacementRatio

      public float computeZ6ReplacementRatio(String text)
      z6: calibrated replacement-character ratio. Direct decode-failure signal — fraction of codepoints that are U+FFFD. Higher raw value = more decode failure = junkier; but the calibration centers on the training distribution, so negative z6 = junkier than typical.

      Returns 0 (neutral) when no calibration available.

    • computeZ9AlternationRatio

      public float computeZ9AlternationRatio(String text)
      z9: calibrated script-alternation ratio. Catches the mojibake-of- Latin-as-CJK pattern where every accent becomes a singleton Han char scattered through Latin text (high alternation = max value). Length- and proportion-invariant by construction. Sign flipped so clean (low alternation) → positive z9 and mojibake (high alternation) → negative.

    • computeZ2BlockTransition

      public static float computeZ2BlockTransition(String text, float[] blockTable, float[] blockCal)
      Feature 2 — calibrated z-score for block-transition mean log-prob on one text window. Returns 0 if the window has fewer than two codepoints or if blockTable / blockCal are null.

      Block bucketing is via the JVM-independent UnicodeBlockRanges. Public so the trainer's classifier feature extractor calls into the exact same math used at inference time — single source of truth, no train/infer drift.

      Parameters:
      blockTable - (blockN)² × float log-prob table where blockN = UnicodeBlockRanges.bucketCount()

    • computeZ3ControlByte

      public static float computeZ3ControlByte(byte[] utf8, float[] controlCal)
      Feature 3 — calibrated z-score for control-byte fraction on the UTF-8 byte sequence of one text window. Stored score is -fraction so higher = cleaner (matching the direction convention of the other z-features).

      Public for train/infer math-sharing.

    • computeZ4ScriptTransition

      public static float computeZ4ScriptTransition(String text, float[] scriptTransTable, float[] scriptTransCal, Map<String,Integer> scriptBucketIndex, int numScriptBuckets)
      Feature 4 — calibrated z-score for global script-transition mean log-prob on one text window. Uses raw Character.UnicodeScript values (no model fallback) so HIRAGANA / KATAKANA / HAN remain distinct. Returns 0 if the window has fewer than two non-neutral codepoints or if the script-transition data isn't supplied.

      Public for train/infer math-sharing. Note: inference computes z4 once per document via computeScriptTransitionZ(String) (which uses the instance's loaded tables); this helper takes them as arguments so training can compute z4 before the model is finalised.

    • computeF1MeanLogP

      public static double computeF1MeanLogP(String text, BigramTables tables)
      Mean log-prob over the codepoint pairs in text using the given script's bigram tables.

      For each adjacent codepoint pair (a, b):

      1. Binary-search both codepoints in the script's codepoint index. If either is absent, the pair was never seen in training; emit α * (logP(a) + logP(b)) using each codepoint's unigram value (or BigramTables.unigramFallbackLogProb if the codepoint isn't even in the unigram index).
      2. Otherwise, look up the packed (idxA<<16)|idxB key in the open-addressing bigram table. Empty slot → unseen pair → unigram backoff (same formula). Match → dequantize the stored value.

      This is the single authoritative implementation of the bigram scoring math, shared by inference and training. Keeping one implementation eliminates the risk of train/infer drift in the F1 feature.

      Returns:
      mean log-prob, or Double.NaN if text has fewer than two codepoints or tables is null

    • computeF1MeanLogP

      public static double computeF1MeanLogP(int[] cps, BigramTables tables)
      Codepoint-array form of computeF1MeanLogP(String, BigramTables).

      Operates on a pre-decoded codepoint sequence rather than a String so callers (e.g. forEachScriptBigram(int[], JunkDetector.BigramSink) bucketing) can score arbitrary codepoint pairs without re-encoding. Same math — this is the single authoritative implementation; the String overload just decodes and delegates.

    • codepointToIndex

      public static int codepointToIndex(BigramTables tables, int cp)
      Binary-search a codepoint in the script's index.
      Returns:
      the dense index (≥ 0) if found, or -1 if the codepoint doesn't appear in any kept bigram for this script

    • packBigramKey

      public static int packBigramKey(int idxA, int idxB)
      Packed bigram key for indices (a, b) where each index fits in
      invalid reference 
      JunkDetectorTrainingConfig#KEY_INDEX_BITS
      bits. Asserts that indices are non-negative; that's the caller's contract.

    • bucketSumsAndCounts

      public static Map<String,double[]> bucketSumsAndCounts(int[] cps, Map<String, BigramTables> tablesByScript)
      Per-script {sumLogP, count} buckets for a codepoint sequence, scored with the given per-script tables. This is exactly the z1 input aggregate(String) computes, exposed so the trainer's calibration and the inference scorer cannot drift.

    • forEachScriptBigram

      public static void forEachScriptBigram(int[] cps, JunkDetector.BigramSink sink)