Class ScriptCensus

java.lang.Object

org.apache.tika.ml.junkdetect.tools.ScriptCensus

public final class ScriptCensus
extends Object

Codepoint-level script census of one or more text files. For each input file, reports the percentage of codepoints in each Character.UnicodeScript, optionally per-line script-mix histograms.

Useful to verify whether BuildJunkTrainingData is bucketing languages correctly: e.g. Japanese is usually a mix of HIRAGANA, KATAKANA and HAN; if jpn ends up in han.train.gz we want to know what fraction of its codepoints are actually Han ideographs vs. kana.

Usage:

  java ScriptCensus <file> [file ...]   # supports .gz and plain text

Method Summary

Modifier and Type	Method	Description
static void	main(String[] args)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException