Class CommonTokenCountManager

java.lang.Object

org.apache.tika.eval.core.tokens.CommonTokenCountManager

public class CommonTokenCountManager
extends Object

Lazily loads the per-language "common tokens" Bloom filters (see CommonTokensBloom) from the classpath (/common_tokens_bloom/<langCode>) or from a configured directory.

Constructor Summary

Constructors

Constructor	Description
CommonTokenCountManager()
CommonTokenCountManager(Path commonTokensDir, String defaultLangCode)

Method Summary

Modifier and Type	Method	Description
void	close()
Set<String>	getLangs()
org.apache.commons.lang3.tuple.Pair<String, LangModel>	getLangTokens(String lang)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

CommonTokenCountManager

public CommonTokenCountManager()

CommonTokenCountManager

public CommonTokenCountManager(Path commonTokensDir,
 String defaultLangCode)

Method Details

getLangs

public Set<String> getLangs()

getLangTokens

public org.apache.commons.lang3.tuple.Pair<String, LangModel> getLangTokens(String lang)

Parameters:

lang - requested language code

Returns:

pair of the actual language code used and the common-tokens model for it

close

public void close()
           throws IOException

Throws:

IOException