Package org.apache.tika.inference

Class MarkdownChunker

java.lang.Object
org.apache.tika.inference.MarkdownChunker
All Implemented Interfaces:
Initializable, TextChunker
public class MarkdownChunker extends Object implements TextChunker, Initializable
Splits markdown text into chunks that respect structural boundaries.

The chunker first splits on markdown heading boundaries (# ...). If a section exceeds the maximum chunk size, it is further split on paragraph boundaries (double newlines). As a last resort, oversized paragraphs are split at the character limit.

Consecutive chunks can overlap by a configurable number of characters to avoid losing context at boundaries.

  • Constructor Details

    • MarkdownChunker

      public MarkdownChunker()
      The binding form: {"markdown-chunker": {"maxChunkChars": .., "overlapChars": ..}}.

    • MarkdownChunker

      public MarkdownChunker(int maxChunkChars, int overlapChars)

  • Method Details

    • initialize

      public void initialize() throws TikaConfigException
      Description copied from interface: Initializable
      Called after all properties have been set to allow for validation and initialization that depends on multiple properties.
      Specified by:
      initialize in interface Initializable
      Throws:
      TikaConfigException - if there is a problem with the configuration

    • getMaxChunkChars

      public int getMaxChunkChars()

    • setMaxChunkChars

      public void setMaxChunkChars(int maxChunkChars)

    • getOverlapChars

      public int getOverlapChars()

    • setOverlapChars

      public void setOverlapChars(int overlapChars)

    • spans

      public List<int[]> spans(String text)
      Description copied from interface: TextChunker
      Half-open [start, end) character ranges of the chunks, in document order.
      Specified by:
      spans in interface TextChunker

    • chunk

      public List<Chunk> chunk(String text)
      Chunk the given markdown text.
      Parameters:
      text - the full markdown content
      Returns:
      ordered list of chunks with offsets relative to text