Class TrainJunkModel

java.lang.Object
org.apache.tika.ml.junkdetect.tools.TrainJunkModel
public class TrainJunkModel extends Object
Trains the junk detector model from per-script corpus files produced by BuildJunkTrainingData.

z1 (codepoint-bigram log-probability) is trained per script by bucketing every bigram to its script (JunkDetector.forEachScriptBigram(int[], JunkDetector.BigramSink)) and building a per-script sorted-occupied bigram table with unigram backoff. z2 (Unicode block-transition), z3 (control-byte fraction), and z4 (script-transition) are single global document-level features. All features are calibrated (mu/sigma) and combined by a single global contrastive combiner producing logit = w1*z1 + ... + w9*z9 + bias; positive values indicate clean text.

The output model is written in the JUNKDET1 gzipped binary format read by JunkDetector.load(InputStream) — see that method's javadoc for the byte layout. The file carries JunkDetector.VERSION; the loader rejects any other version.

  • Constructor Details

    • TrainJunkModel

      public TrainJunkModel()

  • Method Details

    • main

      public static void main(String[] args) throws IOException
      Throws:
      IOException

    • calibrateBucketScript

      public static float[] calibrateBucketScript(Path file, String script, Map<String, BigramTables> tables) throws IOException
      Bucket-model z1 calibration {mu, sigma} for one script, sampling windows from file and taking that script's bucket mean log-prob per window (the inference z1 input). Public for round-trip tests.
      Throws:
      IOException

    • trainBigramTablesForScript

      public static BigramTables trainBigramTablesForScript(Path trainFile, int minBigramCount, double loadFactor, int keyIndexBits) throws IOException
      Builds the BigramTables carrier for the dominant script bucket of one *.train.gz file. Tallies the file via the shared JunkDetector.forEachScriptBigram(int[], JunkDetector.BigramSink) enumeration (so trained tables fit scored bigrams), then builds tables from the bucket whose name matches the file's script (derived from the filename), falling back to the bucket with the most bigrams. Convenience for tests; the full pipeline tallies all files together in main(String[]).
      Parameters:
      trainFile - the per-script *.train.gz
      minBigramCount - drop pairs whose count is below this
      loadFactor - unused (retained for signature compatibility)
      keyIndexBits - bit-width per index in the packed key (each side of the pair must fit)
      Throws:
      IOException

    • buildBigramTablesFromCounts

      public static BigramTables buildBigramTablesFromCounts(HashMap<Long,long[]> pairCounts, HashMap<Integer,long[]> unigramCounts, long unigramTotal, int minBigramCount, double loadFactor, int keyIndexBits)
      Builds the BigramTables carrier from pre-tallied pair/unigram counts. Drops pairs below minBigramCount, assigns dense codepoint indices, and packs the sorted-occupied bigram table; unigram log-probs use unigramTotal as the denominator.

      loadFactor is retained for signature compatibility but unused: the sorted-occupied table (binary-search lookup) has no load factor.

    • saveModel

      public static void saveModel(TreeMap<String, BigramTables> f1Tables, TreeMap<String,float[]> f1Calibrations, float[] blockTable, float[] blockCal, float[] controlCal, float[] combinerWeights, List<String> scriptBuckets, float[] scriptTransTable, float[] scriptTransCal, Path output) throws IOException
      Writes a model file in the current binary format. Layout: gzip envelope around JUNKDET1 magic + JunkDetector.VERSION byte + global script-transition / block / control / z5 / z6 / z9 sections + the global combiner weight vector + per-script sections (z1 calibration + bigram tables). See JunkDetector.load(InputStream) for the load-side spec.
      Throws:
      IOException

    • quantizeDequantizeRoundTrip

      public static float[] quantizeDequantizeRoundTrip(float[] in)
      Quantize a float[] to int16 and dequantize back, returning a new float[] with the int16-precision values. Used at training time so downstream calibration (mu, sigma) is computed on values the inference path will actually see, eliminating train/infer drift. 65536 levels keep ~0.0002 nats/level resolution, essentially lossless for our [-15, -1] log-prob range.

    • quantizeToShorts

      public static TrainJunkModel.QuantizedShorts quantizeToShorts(float[] in)
      int16 (unsigned 0-65535) quantization of a float[]. Linear mapping [min, max] → [0, 65535].

    • quantizeFloats

      public static TrainJunkModel.QuantizedFloats quantizeFloats(float[] in)