Class DiagnoseDiscrimination

java.lang.Object
org.apache.tika.ml.chardetect.tools.DiagnoseDiscrimination
public final class DiagnoseDiscrimination extends Object
Diagnostic for the win-1252-vs-MacRoman / IBM850 / windows-1250 / windows-1257 / etc. confusion problem.

Reads <charset>.bin.gz files from a training directory, counts per-byte and per-bigram frequencies for the requested classes, then for each pair (A, B) prints the top-K bytes and bigrams by Kullback-Leibler-divergence contribution:

  KL_contrib(b) = (p_A(b) - p_B(b)) * log(p_A(b) / p_B(b))

The bytes / bigrams with the largest contributions are the ones whose frequencies differ most between A and B — i.e., the bytes / bigrams the NB classifier should rely on to tell A from B.

Annotates each output line with:

  • the byte's hex value
  • what Unicode character the byte (or bigram) decodes to under each charset — to see whether the discriminative bytes are letters, punctuation, smart quotes, controls, etc.
  • per-class probability of seeing this byte