Class CjkDecodeValidator

java.lang.Object
org.apache.tika.ml.chardetect.CjkDecodeValidator
public final class CjkDecodeValidator extends Object
Structural false-CJK veto: measures how badly a probe fails to decode under a legacy multi-byte CJK charset, robustly against embedded UTF-8.

A Latin/Cyrillic/garbage page mis-detected as a legacy CJK charset decodes with many malformed/unmappable sequences; real CJK decodes cleanly. Two corrections make the rate meaningful (see the findings doc):

  1. decode under the vendor superset (CharsetSupersets) so real vendor-extension chars aren't counted as failures;
  2. discount embedded UTF-8 — mixed-encoding pages (legacy CJK body + UTF-8 widgets) would otherwise inflate the rate. Post-discount, real CJK (pure or mixed) is ≤1.6% while genuine false-CJK stays ≥5.3%.

The discount is done by a UTF-8-aware single pass, NOT by physically stripping UTF-8 runs: a real legacy-CJK char can coincidentally match UTF-8 grammar (e.g. Shift_JIS kanji with lead 0xE0–0xEA), and physically removing it would misalign the stream and manufacture failures on genuine CJK. Instead we walk the bytes, skip positions that begin a valid UTF-8 sequence, and decode the legacy charset in place everywhere else — so real CJK is never misaligned and the rate errs toward not vetoing.

Does NOT catch the legal-but-wrong class (Latin bytes that form valid CJK at ~0 failure) — that's the typicality layer's job.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    MIN_HIGH_BYTES
    Minimum legacy (non-UTF-8) high bytes required before the rate is trusted.

  • Method Summary

    Modifier and Type
    Method
    Description
    static boolean
    appliesTo(String charsetName)
    True for the legacy multi-byte CJK charsets this veto applies to (the decode-failure signal is meaningful only for these; ISO-2022 is handled structurally and single-byte charsets don't apply).
    static double
    strippedFailureRate(byte[] bytes, Charset cjkCharset)
    Failure rate of bytes under cjkCharset's vendor superset, counting only legacy high bytes (embedded UTF-8 is skipped, not counted).

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • MIN_HIGH_BYTES

      public static final int MIN_HIGH_BYTES
      Minimum legacy (non-UTF-8) high bytes required before the rate is trusted.
      See Also:

  • Method Details

    • strippedFailureRate

      public static double strippedFailureRate(byte[] bytes, Charset cjkCharset)
      Failure rate of bytes under cjkCharset's vendor superset, counting only legacy high bytes (embedded UTF-8 is skipped, not counted).

      Special case: if every high byte is a valid UTF-8 sequence (i.e., nHigh == 0) and there are at least MIN_HIGH_BYTES UTF-8 multi-byte sequences, the probe is pure UTF-8 — no legacy CJK content at all. In that case 1.0 is returned to trigger the CJK veto. Real legacy CJK encodings (Shift_JIS, Big5, EUC-JP, GB18030 …) always have lead bytes in 0x81–0x9F or 0xF5–0xFF that are not valid UTF-8 starts, so nHigh > 0 for any genuine CJK document.

      Returns:
      failures / legacy-high-bytes, 1.0 when the probe is pure UTF-8 (nHigh==0, nUTF8seqs≥MIN_HIGH_BYTES), or -1.0 when there is too little evidence either way (legacy high bytes < MIN_HIGH_BYTES and not pure UTF-8)

    • appliesTo

      public static boolean appliesTo(String charsetName)
      True for the legacy multi-byte CJK charsets this veto applies to (the decode-failure signal is meaningful only for these; ISO-2022 is handled structurally and single-byte charsets don't apply).