Package org.apache.tika.eval.app.tools
package org.apache.tika.eval.app.tools
-
ClassesClassDescriptionUtility class that runs TopCommonTokenCounter against a directory of table files (named {lang}_table.gz or leipzip-like afr_...-sentences.txt) and outputs common tokens files for each input table file in the output directory.COPIED VERBATIM FROM LUCENE This class forces a composite reader (eg a
MultiReaderorDirectoryReader) to emulate aLeafReader.Utility class that reads in a UTF-8 input file with one document per row and outputs the 20000 tokens with the highest document frequencies.