Tika Pipes

This section covers Tika Pipes for scalable, fault-tolerant document processing.

Overview

Tika Pipes provides a framework for fault-tolerant, scalable document processing. Each document is parsed in a forked JVM — a child process the parent starts, feeds over a socket and restarts when it dies, called a fork throughout these docs — with configurable timeouts and memory limits, so a single malformed file cannot crash or hang your application. numClients sets how many forks run at once.

While Tika Pipes has a programmatic Java API, it is best used through:

  • tika-app — batch processing from the command line

  • tika-server — REST API with pipes-based robustness built in

  • tika-grpc — gRPC API with pipes-based robustness built in. More exposed by default than tika-server; run only on a trusted network (see Security).

See Robustness for details on how Tika Pipes protects against problematic files.

The four roles

A pipeline is assembled from four kinds of pluggable component. Each is named in the JSON config and documented on its own page:

  • Iterator — enumerates the documents to process (directory walk, S3 listing, CSV, JDBC, Kafka)

  • Fetcher — retrieves one document’s bytes from its source (filesystem, S3, HTTP, GCS, Azure)

  • Emitter — writes the parsed result to a destination (filesystem, OpenSearch, Elasticsearch, Solr, S3, Kafka)

  • Reporter — records per-document processing status

Topics

  • Getting Started — complete working example with tika-app

  • Pipeline Configuration — numClients, timeouts, JVM args, parse modes, emit batching

  • Fetchers — all available document sources (filesystem, S3, HTTP, GCS, Azure, etc.)

  • Emitters — all available output destinations (filesystem, ES, OpenSearch, Solr, S3, Kafka, etc.)

  • Iterators — document enumeration (directory walk, S3 listing, CSV, JDBC, Kafka, etc.)

  • Reporters — track per-document processing status

  • Parse Modes — control how documents are parsed and emitted (RMETA, CONCATENATE, CONTENT_ONLY, NO_PARSE, UNPACK)

  • Extracting Embedded Bytes — extract raw bytes from embedded documents

  • Timeouts — the timeout model for handling long-running and hung parsers

  • Plugins — the PF4J-packaged fetchers, emitters and iterators, and how they are loaded

Advanced Topics