Class BoundaryBigramAudit

java.lang.Object
org.apache.tika.ml.junkdetect.tools.BoundaryBigramAudit
public final class BoundaryBigramAudit extends Object
For each *.train.gz file, classify every adjacent codepoint pair by its relation to the target script S (= file's script). Categories:
  • IN_S_INTERIOR — both codepoints are in S or in COMMON/INHERITED
  • S_BOUNDARY — exactly one codepoint is in S-or-COMMON, the other is a non-S script
  • FOREIGN_INTERIOR — both codepoints are in some non-S script (possibly different scripts). Under the proposed generalized boundary rule, these are the bigrams to drop from S's training.
  • ASCII_LETTER_RUN — special subcategory of foreign interior where both cps are ASCII A–Z/a–z; this is the English-run case.

Reports occurrence counts, distinct-pair counts, and singleton counts for each category, plus the implied model-size impact of dropping FOREIGN_INTERIOR (or just ASCII_LETTER_RUN) under min_count>=1 and min_count>=3.