Package org.apache.tika.eval.core.tokens

package org.apache.tika.eval.core.tokens

Class	Description
AnalyzerManager	Manages tokenization for tika-eval.
CommonTokenCountManager	Lazily loads the per-language "common tokens" Bloom filters (see CommonTokensBloom) from the classpath (/common_tokens_bloom/<langCode>) or from a configured directory.
CommonTokenResult
CommonTokensBloom	Shared serialization, sizing and hashing for the per-language "common tokens" Bloom filters.
ContrastStatistics
LangModel	The set of "common tokens" for a single language, backed by a Bloom filter.
TikaEvalTokenizer	Tokenizer for tika-eval text analysis.
TikaEvalTokenizer.Mode	Tokenization mode.
TokenContraster	Computes some corpus contrast statistics.
TokenCountPriorityQueue	Bounded min-heap that keeps the top-N TokenIntPairs by value.
TokenCounts
TokenIntPair
TokenStatistics