Class NaiveBayesBigramEncodingDetector

java.lang.Object
org.apache.tika.ml.chardetect.NaiveBayesBigramEncodingDetector
All Implemented Interfaces:
Serializable, SelfConfiguring, EncodingDetector
public class NaiveBayesBigramEncodingDetector extends Object implements EncodingDetector
Naive-Bayes byte-bigram charset classifier. Drop-in for ICU4J / juniversalchardet. No structural gates, no language-aware arbitration — just: count probe byte-bigrams, score each candidate charset by log-likelihood of the bigram sequence under its trained bigram distribution, return ranked candidates.

Model format and training described in TrainNaiveBayesBigram. Per-class data: top-K kept bigrams with log-probabilities + a log-probability floor for out-of-vocabulary bigrams (Laplace add-α smoothing).

In-memory layout — bigram-major: logP[bigram * numClasses + classIdx]. For the inner-loop hot path, all class log-probs for one bigram sit in numClasses * 4 bytes of consecutive memory — one or two L1 cache lines vs the 256 KB stride of a class-major layout.

See Also:

  • Field Details

    • MARGIN_THRESHOLD_NATS_PER_BIGRAM

      public static final double MARGIN_THRESHOLD_NATS_PER_BIGRAM
      Per-scored-bigram log-score margin (in nats) that defines "model is reliably right" vs "model is genuinely uncertain between candidates." Derived from a calibration run over the 158K-sample devtest split:
      • CORRECT picks have top-1-vs-top-2 margin median ≈ 1.5–2 nats/bg, with p10 ≥ 0.22 nats/bg in every length bucket.
      • WRONG picks have margin p90 < 0.10 nats/bg in every length bucket.

      A threshold of 0.20 cleanly separates the two regimes. Candidates within MARGIN_THRESHOLD_NATS_PER_BIGRAM of top-1's score (i.e., the model can't reliably tell them apart from top-1) are emitted into the candidate pool for downstream arbitration; candidates further away are dropped.

      Softmax-based confidence is deliberately not used here: softmax saturates to 1.0 on essentially every probe regardless of how uncertain the model actually is, so it cannot serve as a candidate-emission gate.

      See Also:

    • CAP_PER_BIGRAM_NATS

      public static final double CAP_PER_BIGRAM_NATS
      Per-distinct-bigram cap: top-scoring class's contribution is clipped to the best cross-cohort class's contribution + this many nats. Bounds both single-bigram corpus skew and the diffuse coverage asymmetry where broad-vocab cohorts (CJK, EBCDIC) collectively swamp narrow-vocab cohorts (LATIN) on rare-ASCII bigrams that fall to the unseen floor in the narrow cohort. See NaiveBayesBigramEncodingDetector.Cohort.
      See Also:

    • MIN_DISTINCT_FOR_CAP

      public static final int MIN_DISTINCT_FOR_CAP
      Minimum distinct bigrams required before the per-bigram cap applies. On short probes, each bigram carries proportionally more signal — clipping would destroy more discrimination than it saves.
      See Also:

    • MIN_DIVERSITY_RATIO

      public static final double MIN_DIVERSITY_RATIO
      Minimum distinct-bigram fraction of total-scored-bigrams. Below this, the input is treated as degenerate (looped / repeated / corrupt) and scoreClassesAndCount(byte[]) returns null so callers can fall back. Defends against pathological inputs like "thththth..." where one bigram appears hundreds of times.
      See Also:

    • MIN_BIGRAMS_FOR_DIVERSITY_GATE

      public static final int MIN_BIGRAMS_FOR_DIVERSITY_GATE
      Minimum scored bigrams required before the diversity gate applies. Short probes legitimately have lower diversity ratios (fewer total bigrams = fewer opportunities for distinct ones) and shouldn't be gated as degenerate. Above this floor, the ratio measurement is meaningful.
      See Also:

    • SUBLINEAR_COUNT

      public static final boolean SUBLINEAR_COUNT
      Sublinear count weighting ("count clipping"). A distinct bigram's raw repetition count n is replaced by 1 + ln(n) before it weights the per-class contribution, so a bigram repeated hundreds of times (e.g. a "--" separator run, observed 864× on one page) can no longer dominate the score by sheer volume.

      Length-dynamic by construction (no fixed cap) and class-agnostic: it bounds repetition, an axis orthogonal to the Type C cap (which bounds a single class's per-bigram cross-class advantage) and the Type A diversity gate (which abstains only on globally-degenerate input). Partial concentration — one bigram repeated heavily inside an otherwise diverse probe — falls through all three of those guards; this closes it.

      See Also:

  • Constructor Details

  • Method Details

    • detect

      public List<EncodingResult> detect(TikaInputStream tis, Metadata metadata, ParseContext parseContext) throws IOException
      Description copied from interface: EncodingDetector
      Detects the character encoding of the given text document.

      Returns an empty list if the encoding cannot be determined. Returns a ranked list of candidates in descending confidence order otherwise. The first entry is always the best guess.

      If the document input stream is not available, then the first argument may be null. Otherwise the detector may read bytes from the start of the stream to help in encoding detection. The given stream is guaranteed to support the mark feature and the detector is expected to mark the stream before reading any bytes from it, and to reset the stream before returning. The stream must not be closed by the detector.

      The given input metadata is only read, not modified, by the detector.

      Specified by:
      detect in interface EncodingDetector
      Parameters:
      tis - text document input stream, or null
      metadata - input metadata for the document
      parseContext - the parse context
      Returns:
      ranked list of encoding results, empty if unknown; never null
      Throws:
      IOException - if the document input stream could not be read

    • detect

      public List<EncodingResult> detect(byte[] probe)

    • scoreClasses

      public double[] scoreClasses(byte[] probe)
      Compute the raw per-class score vector for a probe, without top-K extraction or softmax. Returns null for null / tiny probes that can't be scored.

    • analyzeBigrams

      public List<NaiveBayesBigramEncodingDetector.BigramContrib> analyzeBigrams(byte[] probe, int classA, int classB)
      For each scored bigram in the probe (same skip rules as scoreClasses(byte[])), compute and return its dequantized contribution to two specified classes' scores. The list is in probe order, with duplicates allowed (a bigram that appears N times in the probe yields N entries).

    • scoreClassesAndCount

      public NaiveBayesBigramEncodingDetector.ScoreResult scoreClassesAndCount(byte[] probe)
      Like scoreClasses(byte[]) but also reports the number of bigrams that contributed to the dot product vs the total scored region. Used by offline calibration to bucket samples by "evidence available" rather than raw byte length.

    • getLabels

      public String[] getLabels()

    • getCharsets

      public Charset[] getCharsets()

    • getNumClasses

      public int getNumClasses()

    • getLabel

      public String getLabel(int idx)