Class TikaLanguageDetector


  • public class TikaLanguageDetector
    extends LanguageDetector
    This is Tika's original legacy, homegrown language detector. As it is currently implemented, it computes vector distance of trigrams between input string and language models.

    Because it works only on trigrams, it is not suitable for short texts.

    There are better performing language detectors. This module is still here in the hopes that we'll get around to improving it, because it is elegant and could be fairly trivially improved.

    • Constructor Detail

      • TikaLanguageDetector

        public TikaLanguageDetector()
    • Method Detail

      • loadModels

        public LanguageDetector loadModels()
                                    throws IOException
        Description copied from class: LanguageDetector
        Load (or re-load) all available language models. This must be called after any settings that would impact the models being loaded (e.g. mixed language/short text), but before any of the document processing routines (below) are called. Note that it only needs to be called once.
        Specified by:
        loadModels in class LanguageDetector
        Returns:
        this
        Throws:
        IOException
      • loadModels

        public LanguageDetector loadModels​(Set<String> languages)
                                    throws IOException
        Description copied from class: LanguageDetector
        Load (or re-load) the models specified in . These use the ISO 639-1 names, with an optional "-" for more specific specification (e.g. "zh-CN" for Chinese in China).
        Specified by:
        loadModels in class LanguageDetector
        Parameters:
        languages - list of target languages.
        Returns:
        this
        Throws:
        IOException
      • hasModel

        public boolean hasModel​(String language)
        Description copied from class: LanguageDetector
        Provide information about whether a model exists for a specific language.
        Specified by:
        hasModel in class LanguageDetector
        Parameters:
        language - ISO 639-1 name for language
        Returns:
        true if a model for this language exists.
      • reset

        public void reset()
        Description copied from class: LanguageDetector
        Reset statistics about the current document being processed
        Specified by:
        reset in class LanguageDetector
      • addText

        public void addText​(char[] cbuf,
                            int off,
                            int len)
        Description copied from class: LanguageDetector
        Add statistics about this text for the current document. Note that we assume an implicit word break exists before/after each of these runs of text.
        Specified by:
        addText in class LanguageDetector
        Parameters:
        cbuf - Character buffer
        off - Offset into cbuf to first character in the run of text
        len - Number of characters in the run of text.
      • detectAll

        public List<LanguageResult> detectAll()
        Description copied from class: LanguageDetector
        Detect languages based on previously submitted text (via addText calls).
        Specified by:
        detectAll in class LanguageDetector
        Returns:
        list of all possible languages with at least medium confidence, sorted by confidence from highest to lowest. There will always be at least one result, which might have a confidence of NONE.