Migrating Tika Server to 4.x

Overview

Tika Server 4.x introduces pipes-based parsing for the main content-extraction endpoints (/tika, /rmeta, /unpack, /meta), which provides process isolation for those operations. This improves stability and resource management but introduces some breaking changes.

New /tika Endpoint Structure

The /tika endpoint has been simplified with path-based routing:

Method Path Config? Output

PUT

/tika

-

raw Markdown

PUT

/tika/text

-

raw text (body only)

PUT

/tika/html

-

raw HTML

PUT

/tika/xml

-

raw XML

PUT

/tika/md

-

raw Markdown

PUT

/tika/json

-

JSON (default handler: markdown)

PUT

/tika/json/{handler}

-

JSON with specified handler (text, txt, html, xml, body, markdown, md, ignore)

POST

/tika/config

YES

raw Markdown (multipart with optional config)

POST

/tika/config/\{text,html,xml,md\}

YES

raw output in the named format

POST

/tika/config/json

YES

JSON (default handler: markdown)

POST

/tika/config/json/{handler}

YES

JSON with specified handler

Using PUT endpoints (simple)

# Get plain text
curl -T document.pdf http://localhost:9998/tika/text

# Get JSON with metadata and text
curl -T document.pdf http://localhost:9998/tika/json

# Get JSON with HTML content
curl -T document.pdf http://localhost:9998/tika/json/html

Using POST endpoints (with configuration)

POST endpoints accept multipart requests with a file part and optional config part:

# Parse with custom PDF parser settings (requires allowPerRequestConfig)
curl -X POST http://localhost:9998/tika/config/json \
  -F "file=@document.pdf" \
  -F "config={\"pdf-parser\":{\"ocr\":{\"strategy\":\"no_ocr\"}}};type=application/json"

Breaking Changes

Removed Endpoints

/tika/main and /tika/form/main (Boilerpipe)

The Boilerpipe content extraction endpoints have been removed. These endpoints used BoilerpipeContentHandler which is not compatible with pipes-based parsing.

Migration: Use /tika/text for plain text extraction.

/tika/form, /tika/form/*

All /form endpoints have been removed. Use the simplified endpoint structure above.

Migration: Use PUT endpoints for simple requests, POST multipart for requests with configuration.

/tika/config, /tika/form/config

The 3.x forms of these endpoints are gone. The /tika/config* paths in the table above are their replacements, and they work differently: per-request configuration now travels as a config part in a multipart POST, and requires allowPerRequestConfig.

Migration: POST /tika/config for raw output, POST /tika/config/json for JSON, each with a config part in the multipart request.

/translate/*

The translation endpoints have been removed, along with tika-server’s dependency on tika-translate.

These endpoints were never usable as shipped. Every request had to name a fully-qualified translator implementation class in the URL path, and each of the bundled implementations wraps a commercial API that requires credentials the server has no way to supply — there is no configuration surface for them and none was ever wired up. With the default distribution, every request returned an error.

Migration: Call the translation service you use directly, or invoke tika-translate from your own code. Tika’s translation support is unaffected; only the server endpoints are gone.

Renamed Endpoints

/detect/stream is now /detect

/language/stream and /language/string are now both /language

The /stream and /string suffixes never described anything a caller could choose between. Every one of these endpoints has always taken the request body and @Consumes("/"); the two /language paths differed only in whether JAX-RS bound the body to a String or an InputStream, and both then ran the identical detection. The suffixes existed to disambiguate Java method signatures, not to offer an option.

Collapsing them also removes the /detect/stream vs /detectors near-collision. /detectors is unchanged — it lists the server’s configured detectors and does not detect anything.

Migration: drop the suffix. PUT /detect/stream becomes PUT /detect; PUT /language/stream and PUT /language/string both become PUT /language. Request bodies and headers are unchanged, but behavior is not:

  • /detect now runs in the fork pool (detection opens containers over caller-supplied bytes), so it can return 429 or 503 with Retry-After, and 413, like the parsing endpoints. A failure reading the body is now a 500; 3.x returned 200 with application/octet-stream as if detection had succeeded.

  • /language caps detection input at the first 100,000 characters and uses the default LanguageDetector on the classpath; 3.x pinned it to Optimaize.

Handler-Type Changes on /tika

/tika/text is body-only again

3.x served this from a BodyContentHandler. 4.0.0 prereleases ran it over the whole XHTML document, so output began with the document title as bare text. This is restored to 3.x behaviour: /tika/text returns body content only.

Migration: if you relied on the prerelease behaviour, PUT /tika/json/text still runs the whole-document handler and returns the content in a JSON envelope.

/tika/json and /tika/config/json default to markdown

Previously they hardcoded plain text. With no handler named in the path they now use the server default, which is markdown — matching /rmeta. Name one explicitly to pin it: /tika/json/text, /tika/json/html, /tika/json/xml, /tika/json/body, /tika/json/md.

Raw /tika and POST /tika/config default to Markdown

3.x — and the 4.0 prereleases — returned XHTML from the bare PUT /tika endpoint (and XHTML from the bare POST /tika/config). Both now return Markdown by default, matching /tika/json and /rmeta. Use /tika/xml for XHTML, /tika/html for HTML, or /tika/text for body-only text (and the /tika/config/{xml,html,text} POST variants for the configured equivalents).

Unrecognized handler names are rejected

/rmeta/txet used to fall back to the default handler and return output that looked correct. Any unrecognized handler name in the path is now a 400 naming the valid types (text, txt, html, xml, body, markdown, md, ignore). This applies to /tika/json/{handler} and the /rmeta/{handler} family.

New: POST /tika/config/json/{handler}

The PUT family had /tika/json/{handler} but the multipart POST family had only fixed paths, so a POST caller wanting text-in-JSON had nowhere to go — /tika/config/text returns raw text with no metadata envelope. Added for parity. Requires allowPerRequestConfig.

Error Response Bodies Are Now JSON

In 3.x, error responses from /tika, /rmeta, and /unpack returned a plain-text body such as "Parse failed: TIMEOUT". In 4.x these endpoints return a JSON body with at least a status field:

{"status": "TIMEOUT"}

A message field is included when one is available and may contain a server-side stack trace, e.g. {"status": "TIMEOUT", "message": "Task timed out after 60000ms"}.

The HTTP status codes are also more precise:

  • UNSPECIFIED_CRASH, TIMEOUT, and OOM return 503 — transient process failures, not a 500 server misconfiguration.

  • Pool saturation (CLIENT_UNAVAILABLE_WITHIN_MS) returns 429 with a Retry-After header, not 503 or 200 — see Backpressure.

  • An unknown or reserved fetcher/emitter (FETCHER_NOT_FOUND, EMITTER_NOT_FOUND) returns 400, not 500 — the request is permanently malformed, so retrying will not help.

  • Two distinct limits return 413. A request body over maxRequestSizeBytes is rejected by a request filter with a plain-text 413 — both a declared Content-Length over the limit and a chunked body that turns out to be too large. Separately, a parse result too large for the pipes IPC channel (pipes.maxIpcPayloadBytes) returns 413 with the JSON status body {"status":"PAYLOAD_LIMIT_EXCEEDED"}.

  • /pipes and /async now signal the outcome through the HTTP status (the same 429/503/400/413 mappings, plus Retry-After) instead of always returning 200 with the failure only in the body.

The raw /tika, /tika/text, /tika/html, /tika/xml, and /tika/md endpoints return 422 with the extracted content as the body when a container-level exception occurs during a partial parse; the exception itself is not appended to the raw body — use /rmeta for the structured exception.

Migration: Clients that parse plain-text error bodies must switch to JSON. Clients that branch only on HTTP status code should account for the 429/400/413 mappings above and stop treating UNSPECIFIED_CRASH as 500.

/pipes and /async Wire Formats

/pipes response body. /pipes now returns the same JSON body as /tika, /rmeta, and /unpack — {"status":"<RESULT_STATUS>","message":"…​"} — where status is the PipesResult status enum (EMIT_SUCCESS, EMIT_SUCCESS_PARSE_EXCEPTION, PARSE_EXCEPTION_NO_EMIT, TIMEOUT, OOM, FETCHER_NOT_FOUND, …​) and message carries the exception when there is one. It previously used a /pipes-only shape ({"status":"ok"|"process_crash"|"application_error", …​} with type, parse_exception, and a stringified emitted). A parse that threw is now reported by its status enum, not as "ok".

/pipes also rejects with 400: a malformed request body (the reason is in the response), and any emit strategy other than EMIT_ALL — the /pipes response carries only status and message, so a passback strategy would silently discard the parsed data. Use /rmeta if you want the data passed back.

/async request body. POST /async now requires an object envelope, {"tuples":[ …​ ]}, instead of a bare JSON array; the envelope leaves room for future batch-level fields. A bare array (or any body without a tuples array) is rejected with 400, as is a tuple naming a fetcher or emitter the server does not have — validated at POST time, before anything is queued. A batch larger than the queue’s total capacity is also a 400 telling the caller to split it: retrying cannot help. 429 with Retry-After is reserved for transient fullness, where the same batch can succeed later.

FetchEmitTuple wire format. The per-tuple parse-context key is parse-context; 3.x used parseContext. An unknown field anywhere in a tuple is rejected with a 400 naming the field and listing the known ones, instead of being silently ignored.

/meta Is Now Pipes-Backed

/meta previously parsed in-process, in the request-handling JVM, with no crash isolation and its own ad hoc error handling (500 for most parse failures, 400 for a field that couldn’t be extracted from an incompletely-parsed document). It now shares the same pipes-backed PipesParser as /tika, /rmeta, and /unpack (see Endpoints and Forked-Process Groups), with the same crash isolation and the same per-document exception handling as those endpoints (see Error Responses):

  • /meta, /meta/form, /meta/config now return 200 OK with the exception embedded in tk:exception:container-exception, instead of 500.

  • /meta/{field} now returns 422 Unprocessable Entity for a genuine parse exception, instead of 500 or 400.

Migration: clients that treated any non-200 from /meta as "parse failed" should check the new status codes above. Clients that inspected the response body for error text should check tk:exception:container-exception (full-object endpoints) or the 422 body (/meta/{field}).

The default representation is now JSON, not CSV. A /meta request without an Accept header returned CSV in 3.x; it now returns JSON. CSV is still available with Accept: text/csv.

Two changes to the returned metadata come with this, neither of which produces an error:

  • /meta no longer returns a language field. Language detection previously ran inline on this endpoint via a dedicated content handler that buffered text solely to detect the language, which meant holding the document text twice to populate one field. That handler was removed. /meta deliberately parses with the ignore content handler, so there is no text for a language detector to work from.

    Migration: configure a language-detection metadata filter (charsoup-metadata-filter, optimaize-metadata-filter, or open-nlp-metadata-filter) and use /rmeta or /tika/json, which capture content. The detected value arrives as tk:detected-language, with tk:detected-language-confidence. Note that these filters read tk:content, so they are no-ops on /meta and on any endpoint configured with the ignore handler.

  • /meta suppresses embedded parsing without stamping an exception flag. It skips embedded documents with a document selector, so — unlike the 4.0 prereleases, which set an embedded depth limit of 0 and recorded tk:exception:embedded-depth-limit-reached on every container — no tk:exception:* flag is set merely for suppressing embedded content.

Accept Header Routing Removed

The /tika endpoint no longer routes on the Accept header. Name the format in the path instead — /tika/text, /tika/html, /tika/json, and the rest of the table above.

Removed Features

  • Fetcher-based streaming - The InputStreamFactory pattern for fetching documents via HTTP headers (fetcherName, fetchKey) has been removed. All documents are now processed through the pipes infrastructure: a body at or below maxInlineBytes (default 10 MB) travels inside the request to the forked worker and touches no disk; a larger one is spooled to a temp file the worker reads.

Command-Line Flags

  • The no-op -a/--pluginsConfig flag has been removed; delete it from launch scripts, where it now fails option parsing.

  • --help now exits 0 (it previously exited non-zero).

/pipes and /async Require allowPipes; Per-Request Config Requires allowPerRequestConfig

This replaces the enableUnsecureFeatures flag that alpha-1 briefly used, and before that, enabling these capabilities simply by listing endpoints under server.endpoints. enableUnsecureFeatures no longer exists: a config that still carries it fails to start with an "Unrecognized field" error naming the key, rather than silently ignoring it. The single flag has been split into two, so that granting batch/fetcher access and granting per-request parser configuration are separate decisions:

Was Now

enableUnsecureFeatures: true (to use /pipes or /async)

allowPipes: true

enableUnsecureFeatures: true (to send per-request config)

allowPerRequestConfig: true

The capabilities are two default-false flags in the server section:

  • allowPipes gates the /pipes and /async endpoints, which drive process-isolated batch parsing through your fetchers and emitters. Selecting either without allowPipes causes the server to refuse to start with a clear error.

  • allowPerRequestConfig gates per-request parser configuration: the /config family of endpoints and the multipart config part. When off, such requests are rejected with 403.

/status is no longer gated: it exposes only aggregate counters, so it is enabled simply by listing status under endpoints.

The endpoints allowlist now also gates SPI-provided resources: a discovered resource whose root path matches a named endpoint binds only when that endpoint is enabled (e.g. the application/rdf+xml XMP resource serves /meta, so omitting meta removes it too). An SPI resource with a custom root path still loads unconditionally — installing the jar is the opt-in.

Migration: if your config selects pipes or async, add "allowPipes": true; if you rely on per-request config, add "allowPerRequestConfig": true:

{
  "server": {
    "allowPipes": true,
    "allowPerRequestConfig": true,
    "endpoints": ["tika", "rmeta", "pipes", "async", "status"]
  }
}

Configuration Changes

Removed Configuration Options

The following TikaServerConfig options have been removed:

  • returnStackTrace - Removed as misleading. It gated only the error-response body, while exception detail — including messages carrying file paths and document fragments — continued to travel in tk:exception:* metadata on successful parses and inside /unpack zips. Setting it to false did not keep stack traces inside the server, which is what the name implied. Exception detail is now always returned; filter Tika’s output with a MetadataFilter before forwarding it somewhere less trusted.

  • taskTimeoutMillis - Was the total time allowed per task before the forked process was killed; that is now parse-context.timeout-limits.totalTaskTimeoutMillis (same meaning, new home). The default changed: 3.x taskTimeoutMillis defaulted to 300000 (5 minutes), while totalTaskTimeoutMillis defaults to 3600000 (1 hour) — an upgrader who never set the 3.x value gets a much larger worst-case per-document bound. Set totalTaskTimeoutMillis: 300000 to keep the 3.x behavior. 4.x also adds a second, independent axis with no pre-4.0 equivalent — progressTimeoutMillis, a stall detector that kills the task only after a period of genuine silence, not merely after totalTaskTimeoutMillis of wall-clock time — worth setting deliberately rather than leaving at its default. See Timeouts.

  • taskPulseMillis - No longer needed

  • minimumTimeoutMillis - No longer needed

  • logLevel - Renamed to requestLogLevel, with a narrower meaning: it only sets the level at which each request URI is logged (empty, the default, disables request logging), and no longer changes any other log level. A config still carrying logLevel fails to start.

  • digest, digestMarkLimit - Moved to the parse-context digester configuration; configure digests there instead of in the server section.

  • idBase - Renamed to id.

  • preventStopMethod, maxFiles, javaPath, maxRestarts, numRestarts, forkedStatusFile, maxForkedStartupMillis, tempFilePrefix, noFork - Vestigial options from the pre-4.0 spawn-child server model, which no longer exists (the -noFork CLI flag is gone too). Delete them from your config; leaving any in place now fails startup, because unrecognized config keys are rejected.

  • port is now a single integer. The 3.x multi-instance port ranges and lists (-p 9995-9998, -p 9995,9997) are no longer supported; run one server per port.

Configuration via HTTP Headers Removed

4.x configures parsing through the tika-config file, not through request headers. The remaining per-request configuration headers have been removed and are now silently ignored if sent — the request succeeds, the header has no effect:

Removed header Replacement

writeLimit

parse-context.output-limits.writeLimit

throwOnWriteLimitReached

parse-context.output-limits.throwOnWriteLimit

maxEmbeddedResources, maxEmbeddedCount

parse-context.embedded-limits.maxCount

meta_* (arbitrary metadata injection)

no replacement — see below

X-Tika-Handler

Use an explicit handler path (/tika/text, /rmeta/xml, /tika/json/html, …​)

The X-Tika-OCR* and X-Tika-PDF* header families were removed earlier in the 4.x line. Parser configuration is now supplied as JSON.

{
  "parse-context": {
    "output-limits": {
      "writeLimit": 100000,
      "throwOnWriteLimit": false
    },
    "embedded-limits": {
      "maxCount": 10
    }
  }
}

Migration: move these settings into your tika-config. For per-request values, POST to a /config endpoint (/tika/config, /tika/config/json, /rmeta/config, /meta/config) with the JSON above as the multipart config part; this requires allowPerRequestConfig=true.

Two capabilities are genuinely gone, not relocated.

Per-request output bounds without allowPerRequestConfig. With the headers removed and per-request config off by default, a caller can no longer bound the output of a single request; the limits are whatever the operator configured. If you relied on clients setting their own writeLimit, either enable allowPerRequestConfig or set a server-wide limit.

Client-supplied metadata (meta_*). Headers prefixed meta_ were copied into the returned metadata under the remainder of the header name, with no key restrictions — so a request could also overwrite keys Tika itself populates. There is no replacement on the push endpoints; attach provenance on your side of the call, or use /pipes, where the FetchEmitTuple carries metadata.

Transport headers are unaffected: Content-Disposition, Content-Type and Content-Length still describe the payload and still influence detection.

The way Content-Type influences detection changed. In 3.x, parsing ran in-process and a request Content-Type acted as a hard override that forced the type. In 4.x, parsing runs in a forked worker and the header is carried across as a soft hint: detection keeps it only when it equals or specializes the type detected from the content, and otherwise ignores it (TIKA-4825). A 3.x client that forced an unrelated type onto arbitrary bytes (for example text/plain) will now see that type ignored in favor of content-based detection. Supply the correct Content-Type (or a filename) to refine within the detected hierarchy.

Pipes Configuration (for /pipes and /async)

No pipes or fetcher configuration is required to start the server: the default-on endpoints (/tika, /rmeta, /meta, /unpack, /detect) run on the built-in \_\_bytes fetcher for content carried inside the request, an internally-configured spool fetcher for anything larger, and a default pipes config, and the server starts with no config file at all. You need a pipes section and your own file-system-fetcher only when you enable allowPipes and want /pipes//async to fetch documents from a directory you control:

{
  "fetchers": {
    "file-system-fetcher": {
      "file-system-fetcher": {
        "basePath": "/path/to/your/input"
      }
    }
  },
  "parse-context": {
    "timeout-limits": {
      "progressTimeoutMillis": 30000
    }
  },
  "pipes": {
    "numClients": 2
  },
  "plugin-roots": "path/to/plugins"
}

The 3.x pipes keys staleFetcherTimeoutSeconds and staleFetcherDelaySeconds are gone. A pipes config still carrying either fails startup: unknown pipes keys are rejected, not ignored.

Set basePath to a directory that contains only the documents you intend the server to read. It is the filesystem sandbox: the fetcher rejects any fetch key that resolves outside it, including absolute paths and ../ traversal, and re-checks after resolving symlinks.

Setting allowAbsolutePaths instead of basePath turns that sandbox off entirely — fetch keys are then used as raw absolute paths, so any caller who can reach /pipes can read any file the server process can read. The matching emitter setting is worse: it grants arbitrary file write. allowAbsolutePaths is not a relaxation of basePath; it is what you get when there is no basePath at all, and it is a no-op when basePath is set. Use it only if you genuinely intend an unsandboxed fetcher and have restricted access to the server by other means.

numClients is not boilerplate to copy unchanged from this example. In 3.x, /tika, /rmeta, and /unpack parsed in-process, in the request-handling JVM — no forked processes, no fixed concurrency limit. In 4.x, these same default-on endpoints always start numClients forked JVMs and share a fixed pool of that many concurrent forks, so this single number sets both your server’s concurrency ceiling and its CPU footprint. Neither mis-sizing raises an error naming numClients as the cause. See Backpressure for what each looks like from the client’s side, and Endpoints and Forked-Process Groups for which endpoints share which pool.

maxRequestSizeBytes now defaults to 1 GiB

The request-body cap was unlimited by default in the prereleases; it now defaults to 1 GiB. Bodies above maxInlineBytes are spooled to disk, so this also bounds how much one request can write, and a body over the limit is rejected with 413.

Migration: if you legitimately post bodies larger than 1 GiB, set "maxRequestSizeBytes" in the server section above your largest payload (or -1 to restore no limit).

http-fetcher verifies TLS by default

The http-fetcher now verifies TLS certificates by default (verifySsl defaults to true). A pipeline that fetched from a self-signed or internal host and previously succeeded will start failing at the TLS handshake.

Migration: trust the host’s CA (preferred), or set verifySsl: false on the fetcher if you accept the risk.

New Features

Process Isolation

All parsing now occurs in isolated child processes, providing:

  • Protection against parser crashes affecting the server

  • Memory isolation (OOM in parser doesn’t crash server)

  • Configurable timeouts at the pipes level

This isolation carries a throughput and memory cost relative to a single in-JVM parser, and the size of that cost depends on your document mix and the pool configuration (per-client versus shared-server, numClients, per-fork heap). See Performance and Isolation Trade-offs for the comparison and tuning guidance.

Backpressure: 429 Separates "Busy" From "Broken"

In 3.x a saturated server had no way to say so. /tika, /rmeta, and /unpack parsed in the request-handling JVM, so load simply piled onto request threads, and a client could not tell overload apart from a document that broke the parse.

Because parsing now runs in a fixed pool of forks, 4.x reports the two conditions differently:

  • The pool was full429, status CLIENT_UNAVAILABLE_WITHIN_MS. Nothing failed; the same request can succeed later, and Retry-After says how long the server thinks that is.

  • A fork died on this document503, status OOM, TIMEOUT, or UNSPECIFIED_CRASH. The replacement fork is up within seconds (Retry-After: 5), but resubmitting the same bytes will likely reproduce the failure.

  • The request is malformed400. Retrying never helps.

A client, load balancer, or queue consumer can now back off on the first, quarantine or route elsewhere on the second, and drop the third — instead of hammering a busy server or failing blind on a single opaque status.

The cost is a sizing decision 3.x did not ask of you: the pool is exactly numClients forks wide. 429 under normal load is what "too small for this request volume" looks like; forks starving each other of CPU is what "too large for this host" looks like. Neither raises an error naming numClients, so size it deliberately — see Forked-JVM CPU and Heap Sizing.

Performance Optimizations

  • TCP_NODELAY enabled for reduced latency on small requests

  • Configurable temp directory for RAM disk optimization (pipes.tempDirectory)

Advanced: Shared Server Mode

For memory-constrained environments, an experimental shared server mode is available. Instead of running N separate server processes (one per client), all clients share a single server process.

This mode sacrifices reliability for reduced memory usage. One crash, OOM, or timeout affects all in-flight requests. Only use if you fully understand the tradeoffs.

{
  "pipes": {
    "numClients": 4,
    "useSharedServer": true,
    "forkedJvmArgs": ["-Xmx4g"]
  }
}

See Shared Server Mode for details.