Class HighByteLetterStats

java.lang.Object
org.apache.tika.detect.HighByteLetterStats
public final class HighByteLetterStats extends Object
High-byte decode-quality statistics shared by the charset detectors.

Used to disambiguate single-byte Latin charset siblings (windows-1252 vs IBM850 / x-MacRoman / ISO-8859-x), where a wrong decode maps high bytes to box-drawing / symbols while the right one maps them to accented letters. The cased-letter count reads that boundary; the byte-bigram typicality models cannot (both decodes look like typical Latin, and on COMMON-dominated docs the discriminating bytes are diluted to noise).

Latin-only. countCasedHighByteLetters(byte[], Charset) counts Lu/Ll/Lt, which also covers Cyrillic/Greek cased letters and would be polluted by a non-Latin SBCS; and it excludes Lo, so a CJK decode (every ideograph is Lo) cannot win on "letters". Callers must restrict the comparison to Latin SBCS candidates.

  • Method Details

    • countHighBytes

      public static int countHighBytes(byte[] probe)
      Count of bytes ≥ 0x80 in the probe.

    • countCasedHighByteLetters

      public static int countCasedHighByteLetters(byte[] probe, Charset cs)
      Decode probe under cs and count codepoints ≥ 0x80 that are Unicode cased letters (Lu/Ll/Lt). Excludes the ordinal / superscript indicators ª (U+00AA), º (U+00BA), ⁿ (U+207F): MacRoman's 0xBB/0xBC are ª/º while windows-1252's 0xBB is » (punctuation), so without the exclusion MacRoman's letter count would beat windows-1252's wherever » appears. Lo (CJK / other-letter) is excluded by counting cased categories only.