Package org.apache.tika.langdetect.charsoup

Class ShortTextFeatureExtractor

java.lang.Object
org.apache.tika.langdetect.charsoup.ShortTextFeatureExtractor
All Implemented Interfaces:
FeatureExtractor
public class ShortTextFeatureExtractor extends Object implements FeatureExtractor
Production feature extractor for the CharSoup short-text language detection model.

Hardcoded to the configuration used to train the short-text model (bigrams + trigrams + word unigrams + 4-grams, no suffixes or prefixes):

  • Character bigrams with word-boundary sentinels (non-CJK)
  • Character trigrams including boundary trigrams
  • Character 4-grams including boundary 4-grams
  • Whole-word unigrams (2–30 codepoints, non-CJK)
  • CJK/kana character unigrams
All features share a single flat hash space.

Both training (Phase2Trainer) and inference (CharSoupLanguageDetector) must use this class for the short-text model to ensure feature consistency.

  • Field Details

    • FEATURE_FLAGS

      public static final int FEATURE_FLAGS
      Bitmask of CharSoupModel.FLAG_* constants that exactly describes the features this extractor emits.
      See Also:

    • FEATURE_FLAGS_LEGACY

      public static final int FEATURE_FLAGS_LEGACY
      See Also:

  • Constructor Details

    • ShortTextFeatureExtractor

      public ShortTextFeatureExtractor(int numBuckets)

    • ShortTextFeatureExtractor

      public ShortTextFeatureExtractor(int numBuckets, boolean useScriptBlocks)

  • Method Details

    • extract

      public int[] extract(String rawText)
      Description copied from interface: FeatureExtractor
      Full preprocessing + feature extraction pipeline.
      Specified by:
      extract in interface FeatureExtractor
      Parameters:
      rawText - raw input text (may be null)
      Returns:
      int array of size FeatureExtractor.getNumBuckets() with feature counts

    • extract

      public void extract(String rawText, int[] counts)
      Description copied from interface: FeatureExtractor
      Extract into caller-supplied buffer (zeroed first).
      Specified by:
      extract in interface FeatureExtractor
      Parameters:
      rawText - raw input text (may be null)
      counts - pre-allocated int array of size FeatureExtractor.getNumBuckets() (will be zeroed)

    • extractFromPreprocessed

      public int[] extractFromPreprocessed(String text)
      Description copied from interface: FeatureExtractor
      Extract from already-preprocessed text.
      Specified by:
      extractFromPreprocessed in interface FeatureExtractor
      Parameters:
      text - text already passed through CharSoupFeatureExtractor.preprocess(String)
      Returns:
      int array of size FeatureExtractor.getNumBuckets() with feature counts

    • extractFromPreprocessed

      public void extractFromPreprocessed(String text, int[] counts, boolean clear)
      Description copied from interface: FeatureExtractor
      Extract from already-preprocessed text into a caller-supplied buffer.
      Specified by:
      extractFromPreprocessed in interface FeatureExtractor
      Parameters:
      text - text already passed through CharSoupFeatureExtractor.preprocess(String)
      counts - pre-allocated int array of size FeatureExtractor.getNumBuckets()
      clear - if true, zero the array before extracting; if false, accumulate on top of existing counts

    • extractAndCount

      public int extractAndCount(String rawText, int[] counts)
      Description copied from interface: FeatureExtractor
      Extract features into counts and return the total n-gram emission count.

      The count is the raw number of individual n-gram tokens processed before bucket hashing. It is a script-neutral measure of how much signal the input carries: whitespace-only input yields 0; ~200 chars of typical Latin or CJK prose yields roughly 400. This is the right threshold variable for length-gated confusables because it is insensitive to padding spaces or punctuation-heavy inputs, and it naturally accounts for the higher feature density of CJK text vs. Latin text.

      The default implementation sums the feature vector after extraction, which is correct because every emission does counts[bucket]++; the sum therefore equals the total emission count regardless of hash collisions.

      Specified by:
      extractAndCount in interface FeatureExtractor
      Parameters:
      rawText - raw input text (may be null)
      counts - pre-allocated int array of size FeatureExtractor.getNumBuckets() (will be zeroed)
      Returns:
      total n-gram emission count (≥ 0)

    • getNumBuckets

      public int getNumBuckets()
      Specified by:
      getNumBuckets in interface FeatureExtractor
      Returns:
      number of hash buckets (feature vector size)

    • getFeatureFlags

      public int getFeatureFlags()
      Description copied from interface: FeatureExtractor
      Returns the bitmask of CharSoupModel FLAG_* constants that describes which feature types this extractor emits.

      This must match the featureFlags stored in any CharSoupModel used with this extractor. A mismatch means the model was trained with a different feature set and will produce garbage scores.

      Specified by:
      getFeatureFlags in interface FeatureExtractor
      Returns:
      bitmask of active feature flags