Package org.apache.tika.ml.chardetect

Class MojibusterEncodingDetector

java.lang.Object

org.apache.tika.ml.chardetect.MojibusterEncodingDetector

All Implemented Interfaces:

Serializable, SelfConfiguring, EncodingDetector

public class MojibusterEncodingDetector
extends Object
implements EncodingDetector

Naive-Bayes pipeline detector: structural checks for wide Unicode + BOMs before falling through to the bigram NB classifier for everything else.

Order of operations:

1. UTF-32 codepoint validity via WideUnicodeDetector. 4-byte-aligned probes with valid Unicode codepoints in exactly one endian order are deterministically UTF-32.
2. UTF-16 column-asymmetry specialist. Stride-2 column histograms reliably distinguish UTF-16-LE / BE from other content — a question bigram NB fundamentally can't answer (LE and BE produce the same bigram multiset).
3. Naive-Bayes bigram classifier. Handles the single-byte and multi-byte CJK classes where byte-bigrams are the natural discriminative signal.

BOM detection is NOT handled here. The canonical location is org.apache.tika.detect.BOMDetector (tika-core), SPI-registered, runs first in DefaultEncodingDetector's chain and emits a DECLARATIVE candidate. This pipeline composes with that detector externally, not internally.

Each prefix layer short-circuits when it produces a confident candidate. Conservative: only return at a layer when that layer's structural check is clean.

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DEFAULT_MODEL_RESOURCE	Default NB bigram model on the classpath.

Constructor Summary

Constructors

Constructor	Description
MojibusterEncodingDetector()	Default SPI constructor: load the NB bigram model from the classpath at DEFAULT_MODEL_RESOURCE.
MojibusterEncodingDetector(Path nbModelPath)

Method Summary

Modifier and Type	Method	Description
List<EncodingResult>	detect(byte[] probe)	Byte-array entry point without metadata — same as passing null.
List<EncodingResult>	detect(byte[] probe, Metadata metadata)	Byte-array entry point with optional metadata.
List<EncodingResult>	detect(TikaInputStream tis, Metadata metadata, ParseContext parseContext)	Detects the character encoding of the given text document.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_MODEL_RESOURCE

public static final String DEFAULT_MODEL_RESOURCE

Default NB bigram model on the classpath.

See Also:

• Constant Field Values

Constructor Details

MojibusterEncodingDetector

public MojibusterEncodingDetector()
                           throws IOException

Default SPI constructor: load the NB bigram model from the classpath at DEFAULT_MODEL_RESOURCE. The UTF-16 specialist loads its own model the same way.

Throws:

IOException

MojibusterEncodingDetector

public MojibusterEncodingDetector(Path nbModelPath)
                           throws IOException

Throws:

IOException

Method Details

detect

public List<EncodingResult> detect(TikaInputStream tis,
 Metadata metadata,
 ParseContext parseContext)
                            throws IOException

Description copied from interface: EncodingDetector

Detects the character encoding of the given text document.

Returns an empty list if the encoding cannot be determined. Returns a ranked list of candidates in descending confidence order otherwise. The first entry is always the best guess.

If the document input stream is not available, then the first argument may be null. Otherwise the detector may read bytes from the start of the stream to help in encoding detection. The given stream is guaranteed to support the mark feature and the detector is expected to mark the stream before reading any bytes from it, and to reset the stream before returning. The stream must not be closed by the detector.

The given input metadata is only read, not modified, by the detector.

Specified by:

detect in interface EncodingDetector

Parameters:

tis - text document input stream, or null

metadata - input metadata for the document

parseContext - the parse context

Returns:

ranked list of encoding results, empty if unknown; never null

Throws:

IOException - if the document input stream could not be read

detect

public List<EncodingResult> detect(byte[] probe)

Byte-array entry point without metadata — same as passing null.

detect

public List<EncodingResult> detect(byte[] probe,
 Metadata metadata)

Byte-array entry point with optional metadata. If metadata's content-type suggests HTML/XML (or is absent), HTML is stripped before the NB stage — but never before the wide-Unicode structural checks, which need byte alignment intact.