org.apache.tika.eval.app.tools

Class TopCommonTokenCounter

java.lang.Object

org.apache.tika.eval.app.tools.TopCommonTokenCounter

public class TopCommonTokenCounter
extends Object

Utility class that reads in a UTF-8 input file with one document per row and outputs the 20000 tokens with the highest document frequencies.

The CommmonTokensAnalyzer intentionally drops tokens shorter than 4 characters, but includes bigrams for cjk.

It also has a include list for __email__ and __url__ and a skip list for common html markup terms.

Constructor Summary

Constructors

Constructor and Description
TopCommonTokenCounter()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TopCommonTokenCounter

public TopCommonTokenCounter()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception