Pipes Configuration

numClients

CPU-derived

Number of parallel forked JVMs. Each processes one document at a time. Defaults to max(1, mincores - 2) / 2, 4 — roughly half the host’s cores, capped at 4. The batch CLIs (tika-app -i/-o, tika-async-cli) override that to 2 when neither -n nor a config value is given — see Tika Pipes Processing. See Forked-JVM CPU and Heap Sizing for guidance on choosing this value relative to host CPU count.

forkedJvmArgs

[]

JVM arguments for forked processes (e.g., ["-Xmx512m", "-Xms256m"]). Unless you supply your own, Tika auto-injects -XX:ActiveProcessorCount to right-size each fork’s GC and JIT thread pools, and -XX:MaxRAMPercentage to keep the forks' combined heap ceiling under the host’s memory; see Forked-JVM CPU and Heap Sizing. This is also where the fork’s system properties go, e.g. -Dtika.pipes.cacheMemoryBudgetBytes below — setting them on the parent JVM has no effect on the forks.

javaPath

java

Path to the Java executable for forked processes.

maxFilesProcessedPerProcess

10000

Restart forked processes after this many files. Prevents slow-building memory leaks in parsing libraries.

tempDirectory

system default

Directory for temporary files. Each fork gets a subdirectory here, and the fork’s whole java.io.tmpdir points at it, so anything the fork spools while parsing, unpacked embedded files and JVM crash logs all land inside. It does not cover the host side: tika-server spools over-threshold request bodies into its own input temp directory, and PipesForkParser into the calling JVM’s java.io.tmpdir. Consider a RAM-backed filesystem (e.g., /dev/shm) for better performance — but see the caveat below.

socketTimeoutMillis

60000

Maximum time (ms) to wait for data from a forked process. If no heartbeat or result is received within this window, the parse is considered hung. Also serves as the fork’s idle-shutdown timer: a fork that receives no work for this long exits and is restarted transparently on next use.

heartbeatIntervalMillis

1000

Interval (ms) between heartbeats sent from the forked process. Must be significantly less than socketTimeoutMillis.

startupTimeoutMillis

60000

Socket read timeout (ms) applied to a freshly-connected fork until it completes its READY handshake, after which socketTimeoutMillis takes over. Raise it if plugin loading in the fork is slow. It does not bound how long the parent waits for the fork to connect in the first place — that wait is fixed at 60 s.

maxTotalTaskTimeoutMillis

3600000

Ceiling for request-supplied timeout limits: a per-request timeout-limits override may lower its timeouts freely but can never raise totalTaskTimeoutMillis or progressTimeoutMillis above this value (values over the cap are clamped with a warning). Limits set in the server’s own parse-context are trusted and not subject to this cap.

maxWaitForClientMillis

60000

Maximum time (ms) to wait for an available forked process when all are busy.

parseMode

RMETA

How embedded documents are handled: RMETA (recursive metadata list), CONCATENATE, CONTENT_ONLY, NO_PARSE, UNPACK. See Parse Modes.

onParseException

EMIT

What to do when a parse fails: EMIT (emit error metadata) or SKIP (silently skip). Set this per tuple, not here — the effective value is the one on each FetchEmitTuple, and the iterators set EMIT. Nothing reads the value from this section.

stopOnlyOnFatal

false

When false, stop the pipeline on configuration errors (missing fetcher/emitter). When true, only stop on fatal initialization failures. Use true for server mode, false for batch mode.

numEmitters

1

Number of emitter threads.

queueSize

10000

Size of the fetch/emit tuple queue.

emitWithinMillis

10000

Flush the emit batch if nothing has been emitted within this many milliseconds, even if the batch is not full.

emitMaxEstimatedBytes

100000

Flush the emit batch when the estimated size reaches this many bytes.

emitIntermediateResults

false

When false, only successfully-parsed tuples reach the emitter — files that crash, time out, or otherwise fail are dropped from the output. When true, every tuple is emitted, including failures (the metadata carries the exception). Turn this on if you need a complete record of what was attempted (audit, retry logic, chaos-monkey tests).

maxIpcPayloadBytes

104857600 (100 MB)

Maximum size in bytes of a single IPC message between the client and the fork. This limit is bidirectional: it applies both to parse results returned from the fork (FINISHED) and to requests sent from the client (NEW_REQUEST). Raising it lets very large documents pass over IPC; set the forked JVM -Xmx to at least approximately 3× this value to keep heap usage under control. Setting it too small (below the size of a typical FetchEmitTuple) will cause requests to be rejected silently as UNSPECIFIED_CRASH. The minimum accepted value is the serialized size of a PAYLOAD_LIMIT_EXCEEDED response (a few dozen bytes); values below that are rejected at config load time.

maxInlineBytes

10485760 (10 MB)

Largest document carried inline to the fork instead of being written to a file first. A host that already holds the content (tika-server’s /tika, /rmeta, /meta, /detect, /unpack; PipesForkParser with a non-file-backed stream) sends anything at or below this size inside the request and touches no disk at all; anything larger is written out once, to tika-server’s dedicated input temp directory (the \_\_tika-server fetcher’s basePath) or, for PipesForkParser, to the JVM’s java.io.tmpdir. A stream already backed by a file always keeps its file, whatever the size. Set to 0 to spool every non-empty body (a zero-length body still rides inline, since the test is size ⇐ maxInlineBytes). The cost of raising it is heap, not disk: the parent holds the payload and the frame containing a copy of it, and the child holds it again, so budget roughly 2 * maxInlineBytes * concurrent-requests in the parent. Must leave room for the rest of the request inside maxIpcPayloadBytes; values that do not are rejected at config load time.

type

DYNAMIC

One of DYNAMIC, EMIT_ALL, PASSBACK_ALL. DYNAMIC switches per-extract based on size (see thresholdBytes). EMIT_ALL always emits from the forked process. PASSBACK_ALL always passes extracts back to the parent for emission.

thresholdBytes

100000

Only used when type is DYNAMIC. Extracts larger than this are emitted directly from the forked PipesServer; smaller ones are passed back to the parent. Setting thresholdBytes with type EMIT_ALL or PASSBACK_ALL is a config error.

configStoreType

"memory"

Backend for storing fetcher/emitter configurations. "memory" (default) is in-process; "ignite" uses Apache Ignite for shared state across nodes.

configStoreParams

"{}"

JSON object (as a string) with backend-specific parameters. Structure depends on configStoreType.

useSharedServer

false

When true, multiple clients share a single forked JVM instead of each having its own. Reduces memory overhead but sacrifices isolation — one crash affects all in-flight requests. Not recommended for production.