Class LangModel

java.lang.Object

org.apache.tika.eval.core.tokens.LangModel

public class LangModel
extends Object

The set of "common tokens" for a single language, backed by a Bloom filter.

Membership is approximate: contains(String) may occasionally return true for a token that is not actually common (a false positive), but never returns false for one that is. See CommonTokensBloom for the rationale and false-positive rate.

Field Summary

Fields

Modifier and Type	Field	Description
static final LangModel	EMPTY_MODEL	Model used when a language has no common-tokens resource; nothing is "common".

Constructor Summary

Constructors

Constructor	Description
LangModel(org.apache.commons.collections4.bloomfilter.BloomFilter filter)

Method Summary

Modifier and Type	Method	Description
boolean	contains(String token)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

EMPTY_MODEL

public static final LangModel EMPTY_MODEL

Model used when a language has no common-tokens resource; nothing is "common".

Constructor Details

LangModel

public LangModel(org.apache.commons.collections4.bloomfilter.BloomFilter filter)

Method Details

contains

public boolean contains(String token)

Returns:

true if token is (probably) one of the common tokens for this language; false means it is definitely not.