Class AdaptiveProbe

java.lang.Object
org.apache.tika.ml.chardetect.AdaptiveProbe
public final class AdaptiveProbe extends Object
Reads an encoding-detection probe sized by content, not raw bytes.

A fixed raw probe (e.g. 16 KB) starves the detectors when a page leads with a large <head>/inline script: after tag-stripping there's little body text, and the bytes that distinguish charsets sit past the window. This grows the read until ~contentTarget bytes of tag-stripped content are present, capped at rawCap raw bytes.

Text-rich pages stop early (~one chunk); markup-heavy pages read deeper, bounded by the cap. Multi-byte encodings (UTF-16/32) register no ASCII tags so they stop at contentTarget raw bytes.

  • Field Details

    • DEFAULT_CONTENT_TARGET

      public static final int DEFAULT_CONTENT_TARGET
      Default body-content target.
      See Also:

    • DEFAULT_RAW_CAP

      public static final int DEFAULT_RAW_CAP
      Default hard ceiling on raw bytes read.
      See Also:

  • Method Details

    • read

      public static byte[] read(TikaInputStream tis, int contentTarget, int rawCap) throws IOException
      Reads from tis (mark/reset preserved) until tag-stripped content reaches contentTarget, the raw read reaches rawCap, or EOF — whichever first. Returns the raw bytes read.
      Throws:
      IOException