unpack-config: Extracting Embedded Document Bytes

When processing container files (ZIP, DOCX, PDF with attachments, etc.), you may want to extract the raw bytes of embedded documents in addition to parsing them. The unpack-config component (Java: UnpackConfig) controls how embedded bytes are extracted and emitted.

Quick Start

To turn on byte extraction for every document the pipeline processes, set parseMode to UNPACK in the pipes section of your tika-config.json. That’s the minimum configuration — extraction defaults are fine for most cases.

{
  "pipes": {
    "parseMode": "UNPACK"
  }
}

To tune extraction (size limits, naming, ZIP output, etc.), add an unpack-config block under the top-level parse-context section. All the options listed below live inside that block:

{
  "pipes": {
    "parseMode": "UNPACK"
  },
  "parse-context": {
    "unpack-config": {
      "maxUnpackBytes": 104857600,
      "zipEmbeddedFiles": true
    }
  }
}

This extracts both metadata (like RMETA mode) and embedded document bytes.

You can also set UnpackConfig programmatically per request from Java code by calling parseContext.set(UnpackConfig.class, …​) on the ParseContext attached to your FetchEmitTuple. The JSON parse-context section above is the declarative equivalent.

Configuration Options

All options below are fields of the unpack-config block — nest them inside parse-context.unpack-config as shown in the Quick Start.

Property Type Default Description

emitter

String

(from FetchEmitTuple)

Emitter name for embedded bytes. Falls back to the FetchEmitTuple’s emitterId.

maxUnpackBytes

long

10 GiB

Maximum total bytes to extract per file. Set to -1 for unlimited (not recommended). 0 is not unlimited — it means zero bytes, so the first embedded file’s extraction is immediately capped.

includeOriginal

boolean

false

Include the container document itself in the output.

zipEmbeddedFiles

boolean

false

Collect all embedded files into a single ZIP archive, emitted at the container’s emit key plus -embedded.zip.

includeMetadataInZip

boolean

false

Include .metadata.json files for each embedded document in the ZIP.

zeroPadName

int

0

Zero-pad embedded IDs in output names (e.g., 8 produces 00000001).

suffixStrategy

NONE, EXISTING, DETECTED

NONE

How to determine file extensions for extracted files. See Suffix Strategies.

embeddedIdPrefix

String

"-"

Separator between emitKeyBase and the embedded ID. Read only when keyBaseStrategy=CUSTOM; the DEFAULT strategy uses a fixed -embed/ separator instead.

keyBaseStrategy

DEFAULT, CUSTOM

DEFAULT

Strategy for generating emit keys. See Key Base Strategies.

emitKeyBase

String

""

Custom base path when keyBaseStrategy=CUSTOM.

outputFormat

REGULAR, FRICTIONLESS

REGULAR

Output format for the ZIP archive. See Frictionless Data Package Output.

outputMode

ZIPPED, DIRECTORY

ZIPPED

ZIPPED packages everything into one archive; DIRECTORY emits each extracted file to the emitter as its own item.

includeFullMetadata

boolean

false

Include a metadata.json file with full RMETA-style metadata for all extracted files. Frictionless output only.

Examples

ZIP Output with Metadata

Collect all embedded files into a ZIP with metadata:

{
  "pipes": {
    "parseMode": "UNPACK"
  },
  "parse-context": {
    "unpack-config": {
      "zipEmbeddedFiles": true,
      "includeMetadataInZip": true,
      "includeOriginal": true
    }
  }
}

Custom Naming

embeddedIdPrefix only applies to keyBaseStrategy=CUSTOM, so set both:

{
  "pipes": {
    "parseMode": "UNPACK"
  },
  "parse-context": {
    "unpack-config": {
      "zeroPadName": 8,
      "suffixStrategy": "DETECTED",
      "keyBaseStrategy": "CUSTOM",
      "emitKeyBase": "document",
      "embeddedIdPrefix": "-embed-"
    }
  }
}

Produces names like document-embed-00000001.pdf. Under the DEFAULT strategy the same zeroPadName/suffixStrategy settings would produce <containerKey>-embed/00000001.pdf.

Suffix Strategies

NONE

No file extension added to extracted files.

EXISTING

Use the file extension from the embedded document’s resource name.

DETECTED

Use the file extension based on the detected MIME type.

Key Base Strategies

DEFAULT

Output key is {containerKey}-embed/{id}{suffix}. The -embed/ separator is fixed; embeddedIdPrefix is not consulted.

CUSTOM

Output key is {emitKeyBase}{embeddedIdPrefix}{id}{suffix}.

Safety Limits

maxUnpackBytes bounds zip bombs and other files that expand to enormous sizes. The 10 GB default suits most corpora; lower it for untrusted input.

Hitting the limit is silent apart from a log line. The embedded file being written is truncated at the remaining budget, each later embedded file is skipped, and each case logs a WARN — but nothing is stamped on the metadata and the result status is unchanged (PARSE_SUCCESS / EMIT_SUCCESS). Watch the log, not the status, for truncation.

maxUnpackBytes: -1 (or any negative value) disables the limit — not recommended for untrusted input. 0 is not "unlimited": it caps extraction at zero bytes.

Frictionless Data Package Output

The UNPACK mode can output files in Frictionless Data Package format, a standard for packaging data files with their metadata. This format includes a datapackage.json manifest with file checksums and MIME types, making it easy to verify and process extracted files.

Enabling Frictionless Output

Set outputFormat to FRICTIONLESS in your unpack-config:

{
  "pipes": {
    "parseMode": "UNPACK"
  },
  "parse-context": {
    "unpack-config": {
      "outputFormat": "FRICTIONLESS",
      "includeFullMetadata": true
    }
  }
}

Output Structure

When using Frictionless output format, the ZIP archive contains:

output.zip
├── datapackage.json      # Manifest with file list, SHA256 hashes, mimetypes
├── metadata.json         # Full RMETA metadata (if includeFullMetadata=true)
└── unpacked/
    ├── 00000001.pdf
    ├── 00000002.png
    └── ...

The datapackage.json file contains:

  • List of all extracted files as "resources"

  • SHA256 hash for each file

  • MIME type for each file

  • File size in bytes

CLI Usage

Extract files in Frictionless format using the CLI. The -Z flag turns on recursive unpack (the Pipes-mode counterpart of standard-mode -z), and -i/-o are the Pipes input/output directories:

java -jar tika-app.jar -Z --unpack-format=FRICTIONLESS -i /path/to/input -o /path/to/output
-i expects a directory of containers to unpack, not a single file. For one-off unpacking of a single document, see the standard-mode -z/--extract flag — though as of 4.x that path also routes through the Pipes machinery and expects an input directory.

Code Examples

For working code examples, see:

  • tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/UnpackModeTest.java

  • tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/TikaPipesTest.java

These test files demonstrate all configuration options with assertions.