Class CommonTokensBloom

java.lang.Object
org.apache.tika.eval.core.tokens.CommonTokensBloom
public final class CommonTokensBloom extends Object
Shared serialization, sizing and hashing for the per-language "common tokens" Bloom filters.

The common-token lists are only ever queried for membership ("is this token one of the N most common tokens in language X?"), so they are shipped as one Bloom filter per language instead of the raw token lists. This is dramatically smaller (the filter is sized by the token count, not the token text) at the cost of a small, bounded false-positive rate: a token that is not common may occasionally be reported as common. There are never false negatives. For the OOV/common-token ratio this feeds, that bias is negligible.

Both the runtime loader (CommonTokenCountManager) and the offline generator use the methods here so the hashing and on-disk format can never drift apart. The format is: 

  int  magic   (0x54424631, "TBF1")
  int  k       (number of hash functions)
  int  m       (number of bits)
  int  nLongs  (length of the bit-map array)
  long[nLongs] bit map (big-endian)

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final double
    DEFAULT_FPP
    Target false-positive rate for the generated filters.

  • Method Summary

    Modifier and Type
    Method
    Description
    static org.apache.commons.collections4.bloomfilter.Hasher
    hasher(String token)
    Deterministic hasher for a token, used identically at generation and query time.
    static boolean
    mightContain(org.apache.commons.collections4.bloomfilter.BloomFilter filter, String token)
     
    static org.apache.commons.collections4.bloomfilter.BloomFilter
    read(InputStream is)
    Reads a filter written by write(OutputStream, BloomFilter).
    static org.apache.commons.collections4.bloomfilter.Shape
    shapeFor(int numTokens)
     
    static void
    write(OutputStream os, org.apache.commons.collections4.bloomfilter.BloomFilter filter)
    Writes filter to os in the format documented on this class.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • DEFAULT_FPP

      public static final double DEFAULT_FPP
      Target false-positive rate for the generated filters. See the class comment for why a non-zero rate is acceptable here. Changing this requires regenerating the shipped common_tokens_bloom/ resources.

      0.0001 (1 in 10,000) keeps the OOV metric close to a literal lookup list: the candidate filter already removes numeric/short/HTML false positives, and this rate makes a residual collision on a genuine word ~10x rarer, which matters for repetitive junk docs where a single dominant token can swing occurrence-weighted OOV. Costs ~2 MB over 0.001.

      See Also:

  • Method Details

    • shapeFor

      public static org.apache.commons.collections4.bloomfilter.Shape shapeFor(int numTokens)
      Parameters:
      numTokens - number of tokens that will be added to the filter
      Returns:
      a Shape sized for numTokens at DEFAULT_FPP

    • hasher

      public static org.apache.commons.collections4.bloomfilter.Hasher hasher(String token)
      Deterministic hasher for a token, used identically at generation and query time.

    • mightContain

      public static boolean mightContain(org.apache.commons.collections4.bloomfilter.BloomFilter filter, String token)
      Returns:
      true if token might be in filter (subject to the filter's false-positive rate); false means it is definitely not present.

    • write

      public static void write(OutputStream os, org.apache.commons.collections4.bloomfilter.BloomFilter filter) throws IOException
      Writes filter to os in the format documented on this class. Does not close the stream.
      Throws:
      IOException

    • read

      public static org.apache.commons.collections4.bloomfilter.BloomFilter read(InputStream is) throws IOException
      Reads a filter written by write(OutputStream, BloomFilter). Does not close the stream.
      Throws:
      IOException