Class TextQualityFeatures

java.lang.Object
org.apache.tika.ml.junkdetect.TextQualityFeatures
public final class TextQualityFeatures extends Object
Candidate charset-agnostic features for use in the Phase-2 feature study (see plan wild-roaming-whale). All features are pure functions of a decoded String so they can be computed by the eval harness without touching the trained bigram model. Intended to migrate into JunkDetector's feature vector after the study identifies the non-redundant subset.

Also hosts the TextQualityFeatures.StripMode enum and strip(String, TextQualityFeatures.StripMode) entry point. The current production behaviour (JunkFilterEncodingDetector.stripCommonCodepoints) corresponds to TextQualityFeatures.StripMode.ALL_COMMON — it strips every COMMON / INHERITED / UNKNOWN codepoint before scoring. The user observation in the plan was that this is too aggressive: it removes pilcrows and other punctuation marks that are themselves mojibake fingerprints. The other modes let the eval measure that empirically.

  • Method Details

    • strip

      public static String strip(String text, TextQualityFeatures.StripMode mode)

    • alphabeticRatio

      public static double alphabeticRatio(String text)
      z6: fraction of codepoints that are letters (Character.isLetter(int)). Polish ciśnienia ≈ 1.0; ci¶nienia < 1.0 because is not a letter.

    • letterPairDensity

      public static double letterPairDensity(String text)
      z5: fraction of adjacent codepoint pairs where both codepoints are letters in the same script cluster. Script cluster groups HAN + HIRAGANA + KATAKANA + HANGUL + BOPOMOFO (CJK) into one cluster so Japanese mixed text and Korean Hanja text count as same-cluster pairs; all other scripts are their own cluster.

      Polish ciśnienia → 1.0 (every adjacent pair is two LATIN letters). ci¶nienia → 0.75 (the two pairs involving fail the both-letters test).

    • highByteEntropy

      public static double highByteEntropy(String text)
      z7: Shannon entropy (in bits) of the distribution of distinct codepoints in the high-byte range (cp >= 0x80). Clean text uses a small alphabet there; CJK-as-Latin mojibake (the ƅ storm) fans out across many distinct codepoints, raising entropy. Returns 0 if no high-byte codepoints are present.

    • replacementRatio

      public static double replacementRatio(String text)
      z6 raw input: fraction of codepoints that are "anomaly indicators" — codepoints that shouldn't appear in correctly-decoded natural text. Direct decode-failure signal — wrong encodings produce these in bulk (Java's CharsetDecoder emits U+FFFD per malformed byte; ISO-8859-X misreads windows-1252 high bytes as C1 controls; PDF cmap failures emit private-use codepoints; etc.).

      Anomaly set:

      • U+FFFD (REPLACEMENT CHARACTER) — the direct decode-failure marker
      • Anomalous Cc: 0x01-0x08, 0x0B, 0x0C, 0x0E-0x1F, 0x7F (matching z3's byte-level anomaly definition, at codepoint level)
      • C1 control codepoints: U+0080-U+009F — the ISO-8859-X-misdecodes-windows-1252 signal
      • Private use area: U+E000-U+F8FF, plus planes 15-16 PUA — the PDF cmap-failure signal

      Continuous (not a binary threshold) so the JunkDetector combiner LR can learn a proportional weight on it. Excluded: 0x00 (NUL — can occur legitimately in some text streams; matches z3's exclusion); 0x09/0x0A/0x0D/0x20 (legitimate whitespace); Cf format chars (ZWJ etc. have legitimate linguistic uses); Cn unassigned (rare in practice).

    • replacementCount

      public static int replacementCount(String text)
      Raw count of U+FFFD codepoints. Kept for diagnostics — the per-record eval TSV emits both the count (for easy spot-checking of "how bad was this decode") and the ratio (the trainable feature).

    • combiningMarkRatio

      public static double combiningMarkRatio(String text)
      z10: fraction of codepoints that are combining or spacing marks (Unicode general categories Mn, Mc, Me). Real Vietnamese / Indic / Thai / Arabic text uses combining marks heavily (Vietnamese ~30 %); mojibake from re-decoding precomposed scripts as Latin-1 has zero. Companion / corrective signal to alphabeticRatio(String), which is backwards on Vietnamese cohorts because marks aren't letters. Works on a single 5-codepoint word.

    • letterAdjacentToMarkRatio

      public static double letterAdjacentToMarkRatio(String text)
      z11: fraction of adjacent codepoint pairs where the first is a letter and the second is a combining or spacing mark. Bigram-shaped companion to combiningMarkRatio(String) — direct positive signal for "letter wearing decoration," which is what correct Vietnamese / Indic / Thai decoding produces and Latin-1 mojibake of those does not.

    • scriptDensity

      public static double scriptDensity(String text)
      Fraction of codepoints assigned to a "real" script (i.e. not in COMMON / INHERITED / UNKNOWN). Pure-whitespace, pure-digit, and pure-punctuation text score 0; mostly-letter text scores near 1.

      Used by JunkDetector's "no scoreable script" fallback classifier (the "NONE" model) to distinguish "real text in an unmodeled script" (high density, low fragmentation → modestly positive signal) from "all-whitespace / digit-only content" (zero density → strong negative signal in JunkDetector's bigram-based judgment, mild signal for general-purpose junk filtering).

      U+FFFD is excluded from both numerator and denominator: it is a decode-failure marker scored by the dedicated replacement-char feature (z6), so counting it here too would (a) double- count FFFD and (b) re-create the FFFD-drag when this feature dominates — a permissive wrong decode (few FFFD) would out-score a correct mixed- encoding decode (many FFFD from undecodable widget bytes). This measures the composition of the *decodable* content only.

    • scriptFragmentation

      public static double scriptFragmentation(String text)
      Fragmentation of script-bearing codepoints across distinct scripts: 1 - longest_same_script_run_length / total_script_codepoints. Coherent one-script text scores 0 (no fragmentation); script-salad mojibake (many tiny runs across multiple scripts) approaches 1.

      Combined with scriptDensity(String), distinguishes the four "no-scoreable-script" failure modes:

      • All-whitespace / pure-digit: density 0, fragmentation 0 (no scripted codepoints at all).
      • Real Gothic / unmodeled-but-coherent script: density 1, fragmentation 0 (one long run).
      • Script-salad mojibake: density > 0.5, fragmentation > 0.7 (many short runs across many scripts).
      • Real multilingual text (e.g. Japanese with romaji): density 1, fragmentation 0.3-0.5 (a handful of long runs).

      Returns 0 when text has no script-bearing codepoints (so the caller can rely on scriptDensity(String) to discriminate the "no-content" case separately).

    • scriptAlternationRatio

      public static double scriptAlternationRatio(String text)
      z9: script-alternation ratio — observed transitions over expected transitions under a random-shuffle null.

      Formally: for a sequence of n non-COMMON codepoints with script proportions p_1, ..., p_k, the expected number of (transition between different scripts) under random shuffling is (n - 1) * (1 - sum(p_i^2)) — the second factor is Gini-Simpson diversity (probability two random positions differ in script). This is the Wald-Wolfowitz runs-test statistic generalised to k categories.

      Returns observed_transitions / expected_transitions:

      • ≈ 1 — scripts randomly interleaved (the mojibake signature when accents are scattered through Latin text — each accent becomes a singleton Han run, looking random)
      • < 1 — clumped (normal: words/phrases stay in one script; English document with embedded Chinese phrase scores 0.05-0.3)
      • > 1 — more alternating than chance (pathological: "HLHLHL" patterns)

      Length- and proportion-invariant by construction. COMMON / INHERITED / UNKNOWN codepoints are ignored to keep whitespace and punctuation from dominating the signal in normal text.

      Returns 0 for single-script documents (no diversity possible).

    • perWordScriptPurity

      public static double perWordScriptPurity(String text)
      Candidate feature (not currently in the classifier): fraction of whitespace-delimited tokens whose letter codepoints all belong to the same script cluster. Mojibake often produces tokens with mixed-script letters (Latin + Cyrillic + Greek in one "word"). Tokens with zero letters are excluded from both numerator and denominator.