Class BigramTables

java.lang.Object

org.apache.tika.ml.junkdetect.BigramTables

public final class BigramTables
extends Object

Carrier for one script's codepoint-bigram tables.

Each script has its own pair of tables so that enlarging one script's training data does not perturb the others' z-scores.

Per-script layout:

• codepointIndex — sorted, ascending int[] of every codepoint that appears as either side of a kept bigram for this script. Codepoint → dense index is a binary search; index → codepoint is direct array access. Typical sizes: ~7K-15K for HAN, ~200-500 for most other scripts.
• bigramKeys / bigramValues — parallel arrays of the occupied entries only, sorted ascending by key for binary-search lookup. Each key is a 32-bit value (idxA << 16) | idxB. Indices are bounded at 16 bits (65535), comfortably above the largest per-script codepoint count we observe.
• unigramTable — byte[numCodepoints], quantized unigram log-probabilities indexed by the same codepoint→index map.
• bigramQuantMin/Max, unigramQuantMin/Max — per-quantization ranges; dequantize by min + (b/255) * (max - min).
• unigramFallbackLogProb — log-prob assigned when a codepoint is not in codepointIndex at all. Set to the script's most-pessimistic unigram value (its quantization min) so absent codepoints don't accidentally score above legitimately-rare ones.
• backoffAlpha — multiplier on the unigram-backoff independence sum.

Membership semantics: no Bloom filter. A pair is "seen" iff binary-search finds both codepoints in the index AND finds the packed key in bigramKeys. Lookups are therefore exact.

Instances are built by the trainer (TrainJunkModel, in the tika-ml-junkdetect-tools module) and read back via readFrom(DataInputStream).

Field Summary

Fields

Modifier and Type	Field	Description
static final int	EMPTY_KEY	Reserved value in bigramKeys marking an unoccupied slot.

Constructor Summary

Constructors

Constructor	Description
BigramTables(int[] codepointIndex, int[] bigramKeys, byte[] bigramValues, byte[] unigramTable, float bigramQuantMin, float bigramQuantMax, float unigramQuantMin, float unigramQuantMax, float unigramFallbackLogProb, float backoffAlpha)

Method Summary

Modifier and Type	Method	Description
int	bigramSlots()	Number of bigram-table slots (capacity).
int	codepointCount()	Number of codepoints in this script's index.
static BigramTables	readFrom(DataInputStream dis)	Inverse of writeTo(DataOutputStream).
String	statsString()	Returns a one-line summary for trainer progress output.
void	writeTo(DataOutputStream dos)	Serialises this script's bigram tables.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

EMPTY_KEY

public static final int EMPTY_KEY

Reserved value in bigramKeys marking an unoccupied slot.

See Also:

• Constant Field Values

Constructor Details

BigramTables

public BigramTables(int[] codepointIndex,
 int[] bigramKeys,
 byte[] bigramValues,
 byte[] unigramTable,
 float bigramQuantMin,
 float bigramQuantMax,
 float unigramQuantMin,
 float unigramQuantMax,
 float unigramFallbackLogProb,
 float backoffAlpha)

Method Details

writeTo

public void writeTo(DataOutputStream dos)
             throws IOException

Serialises this script's bigram tables. Read back via readFrom(DataInputStream).

Throws:

IOException

readFrom

public static BigramTables readFrom(DataInputStream dis)
                             throws IOException

Inverse of writeTo(DataOutputStream).

Throws:

IOException

statsString

public String statsString()

Returns a one-line summary for trainer progress output.

codepointCount

public int codepointCount()

Number of codepoints in this script's index. Diagnostic.

bigramSlots

public int bigramSlots()

Number of bigram-table slots (capacity). Diagnostic.