Package org.apache.tika.inference

Class AbstractEmbeddingFilter

java.lang.Object
org.apache.tika.metadata.filter.MetadataFilter
org.apache.tika.inference.AbstractEmbeddingFilter
All Implemented Interfaces:
Closeable, Serializable, AutoCloseable, SelfConfiguring
Direct Known Subclasses:
OpenAIEmbeddingFilter
@Deprecated public abstract class AbstractEmbeddingFilter extends MetadataFilter implements SelfConfiguring
Deprecated.
since 4.1.0, removed in 4.2.0: configure the endpoint once under "engines" and bind it with a TEXT "inference" binding, which embeds the whole document tree in batched requests.
Base class for metadata filters that chunk text content and call a remote embeddings endpoint to produce vectors for each chunk.

The pipeline is:

  1. Read source text from contentField in metadata
  2. Chunk it with MarkdownChunker
  3. Call
    invalid reference 
    #embed(List, InferenceConfig)
    to get vectors
  4. Serialize chunks + vectors as JSON into outputField

The MarkdownChunker requires markdown-formatted content to split at semantic boundaries (headings, paragraphs, etc.). The content handler type must be set to MARKDOWN in the configuration. If the TikaCoreProperties.TIKA_CONTENT_HANDLER_TYPE metadata field indicates a different handler type, a warning is logged.

Subclasses implement embed(java.util.List<org.apache.tika.inference.Chunk>, org.apache.tika.inference.InferenceConfig, org.apache.tika.parser.ParseContext) for their specific API format.

Thread safety: instances are safe for concurrent MetadataFilter.filter(java.util.List<org.apache.tika.metadata.Metadata>, org.apache.tika.parser.ParseContext) calls once fully constructed. Setters must not be called concurrently with MetadataFilter.filter(java.util.List<org.apache.tika.metadata.Metadata>, org.apache.tika.parser.ParseContext).

See Also:

  • Constructor Details

    • AbstractEmbeddingFilter

      protected AbstractEmbeddingFilter()
      Deprecated.

    • AbstractEmbeddingFilter

      protected AbstractEmbeddingFilter(InferenceConfig config)
      Deprecated.

  • Method Details

    • embed

      protected abstract void embed(List<Chunk> chunks, InferenceConfig config, ParseContext parseContext) throws IOException, TikaException
      Deprecated.
      Call the embeddings endpoint to fill in vectors on each chunk. Implementations should set Chunk.setVector(float[]) on each chunk in the list.
      Parameters:
      chunks - the text chunks to embed
      config - the resolved config for this call
      parseContext - the task's ParseContext -- pass to TikaHttpClient calls to bound the request by the task's remaining budget and checkpoint while waiting
      Throws:
      IOException - on HTTP errors
      TikaException - on API-level errors

    • getComponentName

      protected abstract String getComponentName()
      Deprecated.
      The @TikaComponent name this filter's per-request JSON config is keyed by in parse-context (e.g. {"openai-embedding-filter": {...}}).

    • doFilter

      protected void doFilter(List<Metadata> metadataList, ParseContext parseContext) throws TikaException
      Deprecated.
      Description copied from class: MetadataFilter
      Applies the filter in place. A filter that needs to write a reserved Tika-native (tk:) key by name (rather than via its Property) should call Metadata.addTrusted(java.lang.String, java.lang.String) / Metadata.setTrusted(java.lang.String, java.lang.String) explicitly at that write.
      Specified by:
      doFilter in class MetadataFilter
      Parameters:
      metadataList - the list to filter (must be mutable)
      parseContext - per-request context
      Throws:
      TikaException - if filtering fails

    • getDefaultConfig

      public InferenceConfig getDefaultConfig()
      Deprecated.

    • getBaseUrl

      public String getBaseUrl()
      Deprecated.

    • setBaseUrl

      public void setBaseUrl(String baseUrl) throws TikaConfigException
      Deprecated.
      Throws:
      TikaConfigException

    • getModel

      public String getModel()
      Deprecated.

    • setModel

      public void setModel(String model)
      Deprecated.

    • getApiKey

      public String getApiKey()
      Deprecated.

    • setApiKey

      public void setApiKey(String apiKey) throws TikaConfigException
      Deprecated.
      Throws:
      TikaConfigException

    • getTimeoutMillis

      public long getTimeoutMillis()
      Deprecated.

    • setTimeoutMillis

      public void setTimeoutMillis(long timeoutMillis)
      Deprecated.

    • getMaxChunkChars

      public int getMaxChunkChars()
      Deprecated.

    • setMaxChunkChars

      public void setMaxChunkChars(int maxChunkChars)
      Deprecated.

    • getOverlapChars

      public int getOverlapChars()
      Deprecated.

    • setOverlapChars

      public void setOverlapChars(int overlapChars)
      Deprecated.

    • getContentField

      public String getContentField()
      Deprecated.

    • setContentField

      public void setContentField(String contentField)
      Deprecated.

    • getOutputField

      public String getOutputField()
      Deprecated.

    • setOutputField

      public void setOutputField(String outputField)
      Deprecated.

    • isSkipEmbedding

      public boolean isSkipEmbedding()
      Deprecated.

    • setSkipEmbedding

      public void setSkipEmbedding(boolean skipEmbedding)
      Deprecated.

    • isClearContentAfterChunking

      public boolean isClearContentAfterChunking()
      Deprecated.

    • setClearContentAfterChunking

      public void setClearContentAfterChunking(boolean clearContentAfterChunking)
      Deprecated.

    • getMaxBatchSize

      public int getMaxBatchSize()
      Deprecated.

    • setMaxBatchSize

      public void setMaxBatchSize(int maxBatchSize)
      Deprecated.

    • getMaxChunks

      public int getMaxChunks()
      Deprecated.

    • setMaxChunks

      public void setMaxChunks(int maxChunks)
      Deprecated.