Package org.apache.tika.ml.junkdetect.tools
package org.apache.tika.ml.junkdetect.tools
-
ClassesClassDescriptionFor each
*.train.gzfile, classify every adjacent codepoint pair by its relation to the target script S (= file's script).Builds per-script positive training data for the junk detector from MADLAD-400 and Wikipedia sentence files.Diagnostic: replicate JunkDetector.buildScriptRuns exactly on a fixture and print every run.Frozen set of training-time choices that together define a junk-detector model's identity.For each*.train.gzfile in a directory, compute per-line statistics of "target-script fraction" — i.e. the fraction of codepoints in each line that belong to the script the file is supposed to represent.Codepoint-level script census of one or more text files.Trains the junk detector model from per-script corpus files produced byBuildJunkTrainingData.Return type ofTrainJunkModel.quantizeFloats(float[]).