Class LineScriptFractions

java.lang.Object

org.apache.tika.ml.junkdetect.tools.LineScriptFractions

public final class LineScriptFractions
extends Object

For each *.train.gz file in a directory, compute per-line statistics of "target-script fraction" — i.e. the fraction of codepoints in each line that belong to the script the file is supposed to represent.

Reports a histogram across the buckets [0, 5, 10, 20, 30, 50, 70, 90, 100]% so we can pick a per-script keep threshold (e.g. "drop lines with <20% HAN codepoints"). Also reports what fraction of total bytes / lines would be dropped at each threshold.

Each {script}.train.gz maps to a Character.UnicodeScript; the file basename is uppercased. Special-case handling routes a few project-internal script names (e.g. HAN includes HALF_FULL ideographic forms) when desired.

Usage:

  java LineScriptFractions <dataDir> [thresholds]

Method Summary

Modifier and Type	Method	Description
static void	main(String[] args)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException