Package org.apache.tika.ml.chardetect

Class NaiveBayesBigramEncodingDetector

java.lang.Object

org.apache.tika.ml.chardetect.NaiveBayesBigramEncodingDetector

All Implemented Interfaces:

Serializable, SelfConfiguring, EncodingDetector

public class NaiveBayesBigramEncodingDetector
extends Object
implements EncodingDetector

Naive-Bayes byte-bigram charset classifier. Drop-in for ICU4J / juniversalchardet. No structural gates, no language-aware arbitration — just: count probe byte-bigrams, score each candidate charset by log-likelihood of the bigram sequence under its trained bigram distribution, return ranked candidates.

Model format and training described in TrainNaiveBayesBigram. Per-class data: top-K kept bigrams with log-probabilities + a log-probability floor for out-of-vocabulary bigrams (Laplace add-α smoothing).

In-memory layout — bigram-major: logP[bigram * numClasses + classIdx]. For the inner-loop hot path, all class log-probs for one bigram sit in numClasses * 4 bytes of consecutive memory — one or two L1 cache lines vs the 256 KB stride of a class-major layout.

See Also:

• Serialized Form

Constructor Summary

Constructors

Constructor	Description
NaiveBayesBigramEncodingDetector(InputStream modelStream)
NaiveBayesBigramEncodingDetector(Path modelPath)

Method Summary

Modifier and Type	Method	Description
List<EncodingResult>	detect(byte[] probe)
List<EncodingResult>	detect(TikaInputStream tis, Metadata metadata, ParseContext parseContext)	Detects the character encoding of the given text document.
String	getLabel(int idx)
int	getNumClasses()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

NaiveBayesBigramEncodingDetector

public NaiveBayesBigramEncodingDetector(Path modelPath)
                                 throws IOException

Throws:

IOException

NaiveBayesBigramEncodingDetector

public NaiveBayesBigramEncodingDetector(InputStream modelStream)
                                 throws IOException

Throws:

IOException

Method Details

detect

public List<EncodingResult> detect(TikaInputStream tis,
 Metadata metadata,
 ParseContext parseContext)
                            throws IOException

Description copied from interface: EncodingDetector

Detects the character encoding of the given text document.

Returns an empty list if the encoding cannot be determined. Returns a ranked list of candidates in descending confidence order otherwise. The first entry is always the best guess.

If the document input stream is not available, then the first argument may be null. Otherwise the detector may read bytes from the start of the stream to help in encoding detection. The given stream is guaranteed to support the mark feature and the detector is expected to mark the stream before reading any bytes from it, and to reset the stream before returning. The stream must not be closed by the detector.

The given input metadata is only read, not modified, by the detector.

Specified by:

detect in interface EncodingDetector

Parameters:

tis - text document input stream, or null

metadata - input metadata for the document

parseContext - the parse context

Returns:

ranked list of encoding results, empty if unknown; never null

Throws:

IOException - if the document input stream could not be read

detect

public List<EncodingResult> detect(byte[] probe)

getNumClasses

public int getNumClasses()

getLabel

public String getLabel(int idx)