Class JunkDetectorTrainingConfig

java.lang.Object
org.apache.tika.ml.junkdetect.tools.JunkDetectorTrainingConfig
public final class JunkDetectorTrainingConfig extends Object
Frozen set of training-time choices that together define a junk-detector model's identity. Any change to these values produces a meaningfully different model and must be reviewed in git.

Two principles drove making this a class rather than CLI flags:

  1. Reproducibility. When we look back at a model file six months later we want a single commit hash that says exactly what knobs produced it, not a half-remembered shell history.
  2. Drift prevention. CLI flags with defaults allow accidental deviation between developers ("did you remember to pass --min-target-script-frac 0.05?"). Constants in a tracked file remove that failure mode.

BuildJunkTrainingData and TrainJunkModel read the values here; both tools refuse to start if any CLI argument attempts to override a config-controlled parameter, surfacing the mistake at launch time rather than silently producing a non-canonical model.

The constants below reflect the choices that produced the current shipping model. Update them by editing this file and committing the change together with the new model output.

The class has no instance state; all values are exposed as public static final. This keeps callsites short and avoids the temptation of passing a runtime-mutable config around.

This is not part of the public model-loading API. The JunkDetector runtime is configuration- free; once a model file is built, all of its baked-in choices travel with the file's binary format.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final Set<String>
    DROP_SCRIPTS
    Script bucket names whose source data is too thin or too off-target to produce reliable per-script F1 calibration.
    static final long
    ENTROPY_SAMPLE_BYTES
    UTF-8 bytes loaded per script group for bigram entropy estimation, driving the entropy-proportional budget allocation. 200 KB is sufficient to characterise the bigram distribution of any single script.
    static final int
    KEY_INDEX_BITS
    Bit width of each codepoint's dense index within a script's F1 table.
    static final double
    MAX_PUNC_FRAC
    Maximum fraction of codepoints that may be ASCII punctuation/digits.
    static final int
    MIN_BIGRAM_COUNT
    Drop per-script F1 bigrams whose per-pair occurrence count (within that script's training data) is below this threshold.
    static final int
    MIN_BYTES_PER_SENTENCE
    Minimum UTF-8 byte length for a sentence to pass the quality filter.
    static final int
    MIN_DEV_SENTENCES
    Minimum number of sentences that must land in the dev split for a script to be included in the model.
    static final double
    MIN_TARGET_SCRIPT_FRAC
    Sentence-level filter: minimum fraction of non-COMMON/INHERITED codepoints that must belong to the script bucket's target script for a sentence to be accepted.
    static final double
    OA_LOAD_FACTOR
    Target load factor for the per-script open-addressing F1 hash table.
    static final long
    PER_LANGUAGE_CAP_BYTES
    Maximum UTF-8 bytes a single language may contribute to a multi-language script bucket.
    static final Map<String,Long>
    SCRIPT_BUDGET_OVERRIDES
    Per-script byte-budget overrides applied on top of the entropy- proportional allocation.
    static final int
    SCRIPT_SAMPLE_LINES
    Lines read per language to determine the language's dominant script.
    static final int
    SEED
    Random seed for sentence shuffling and other corpus-build randomness.
    static final long
    TOTAL_BUDGET_BYTES
    Total UTF-8 byte budget across all script groups.

  • Method Summary

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • TOTAL_BUDGET_BYTES

      public static final long TOTAL_BUDGET_BYTES
      Total UTF-8 byte budget across all script groups. Divided proportionally by per-script bigram entropy after the sampling phase.
      See Also:

    • PER_LANGUAGE_CAP_BYTES

      public static final long PER_LANGUAGE_CAP_BYTES
      Maximum UTF-8 bytes a single language may contribute to a multi-language script bucket. Prevents one large source (e.g. zho with 8 GB of MADLAD) from dominating a multi-language script model. Buckets with only one language ignore this cap and may consume their full budget. See BuildJunkTrainingData Phase 4.
      See Also:

    • MIN_TARGET_SCRIPT_FRAC

      public static final double MIN_TARGET_SCRIPT_FRAC
      Sentence-level filter: minimum fraction of non-COMMON/INHERITED codepoints that must belong to the script bucket's target script for a sentence to be accepted. Set low so legitimate mixed-script content (Japanese kanji + kana, Korean with hanja annotations, Chinese with English citations, etc.) is preserved, but enough to reject lines that are essentially off-target (e.g. an English article about Gothic in the GOTHIC bucket).
      See Also:

    • MIN_BYTES_PER_SENTENCE

      public static final int MIN_BYTES_PER_SENTENCE
      Minimum UTF-8 byte length for a sentence to pass the quality filter.
      See Also:

    • MAX_PUNC_FRAC

      public static final double MAX_PUNC_FRAC
      Maximum fraction of codepoints that may be ASCII punctuation/digits.
      See Also:

    • MIN_DEV_SENTENCES

      public static final int MIN_DEV_SENTENCES
      Minimum number of sentences that must land in the dev split for a script to be included in the model. Scripts below this floor have insufficient data to reliably estimate calibration statistics, which inflates FPR. With DEV_FRAC = 0.10 this corresponds to a total-sentence floor of 500 / 0.10 = 5000 per script.
      See Also:

    • SCRIPT_SAMPLE_LINES

      public static final int SCRIPT_SAMPLE_LINES
      Lines read per language to determine the language's dominant script.
      See Also:

    • ENTROPY_SAMPLE_BYTES

      public static final long ENTROPY_SAMPLE_BYTES
      UTF-8 bytes loaded per script group for bigram entropy estimation, driving the entropy-proportional budget allocation. 200 KB is sufficient to characterise the bigram distribution of any single script.
      See Also:

    • SEED

      public static final int SEED
      Random seed for sentence shuffling and other corpus-build randomness.
      See Also:

    • DROP_SCRIPTS

      public static final Set<String> DROP_SCRIPTS
      Script bucket names whose source data is too thin or too off-target to produce reliable per-script F1 calibration. Excluded from the model entirely; the JunkDetector.score(String) routing falls back to "unknown script" behavior for these scripts.

      The current selection is based on a corpus audit that found these scripts either had thin native source data (e.g. THAANA: 216 train sentences from Maldivian), or had sources dominated by off-target content (e.g. GOTHIC: 40% of lines are <5% Gothic — the Wikipedia "gothic" directory is English text about Gothic).

      Three further scripts (CANADIAN_ABORIGINAL, CHEROKEE, TIFINAGH) are not listed here because the MIN_TARGET_SCRIPT_FRAC filter implicitly removes them — their MADLAD sources contain effectively no native-script content at the 5% threshold. Listing them here is unnecessary and would obscure the data-quality finding.

    • SCRIPT_BUDGET_OVERRIDES

      public static final Map<String,Long> SCRIPT_BUDGET_OVERRIDES
      Per-script byte-budget overrides applied on top of the entropy- proportional allocation. Empty in the current configuration.

      With per-script tables a HAN=60MB experiment leaves other scripts untouched, but the HAN gain itself was negligible (Cohen's d moved 7.26 → 7.35) — the per-script HAN model is already near its data-saturation point with ~18 MB of training data. Override left empty until a more decisive HAN-coverage experiment is designed.

    • MIN_BIGRAM_COUNT

      public static final int MIN_BIGRAM_COUNT
      Drop per-script F1 bigrams whose per-pair occurrence count (within that script's training data) is below this threshold. Set to 3 on evidence that singleton and doubleton pairs are overwhelmingly OCR artifacts and proper-noun noise that inflate the clean-side score distribution tail without contributing signal.

      Set to 1 to disable the filter (every observed pair retained).

      See Also:

    • OA_LOAD_FACTOR

      public static final double OA_LOAD_FACTOR
      Target load factor for the per-script open-addressing F1 hash table. Table capacity is sized as the smallest power of two larger than keptPairs / loadFactor, giving an average of 1 / (1 - loadFactor) probes per lookup. 0.5 → ~2 probes; modestly wasteful in space but very cheap to probe.
      See Also:

    • KEY_INDEX_BITS

      public static final int KEY_INDEX_BITS
      Bit width of each codepoint's dense index within a script's F1 table. Each bigram is packed as (idxA << KEY_INDEX_BITS) | idxB, so each side must fit in this many bits. 16 bits supports up to 65535 distinct codepoints per script, which is comfortably above the largest per-script count we have measured (HAN is the worst case at ~15K kept codepoints).
      See Also: