Class RebalanceCharsetTraining

java.lang.Object
org.apache.tika.ml.chardetect.tools.RebalanceCharsetTraining
public final class RebalanceCharsetTraining extends Object
Rebalance the per-class training corpus so all Western European Latin SBCS classes have the same underlying source text — eliminating the "common ASCII bigrams pull toward MacRoman" bias surfaced by DiagnoseDiscrimination.

Reads windows-1252.bin.gz (which contains samples drawn from all 19 Western European languages), decodes each sample under windows-1252 to get the original text, then re-encodes that text under each target charset (MacRoman / IBM850 / IBM500 / IBM1047), writing new <target>.bin.gz files. Samples whose codepoints cannot be encoded under a target are skipped (handled via the encoder's REPORT action — keeps the per-class distribution clean rather than importing replacement chars that aren't part of the target's natural vocabulary).

Result: MacRoman / IBM850 / IBM500 / IBM1047 see the SAME source text as windows-1252, just encoded differently — so the only cross-class differences are byte-position differences, which is exactly what the model should be learning.

Usage:

  java RebalanceCharsetTraining \
      --train-dir <dir containing windows-1252.bin.gz> \
      --output-dir <dir to write re-encoded .bin.gz files> \
      [--targets x-MacRoman,IBM850,IBM500,IBM1047]