File System Plugin

The File System plugin (tika-pipes-file-system) is the most common starting point for Tika Pipes. It provides all four interfaces — fetcher, emitter, iterator, and reporter — backed by the local (or mounted) filesystem.

Interface Component name Class

Fetcher

file-system-fetcher

FileSystemFetcher

Emitter

file-system-emitter

FileSystemEmitter

Iterator

file-system-pipes-iterator

FileSystemPipesIterator

Reporter

file-system-reporter

FileSystemStatusReporter

Complete Pipeline Example

The canonical filesystem-to-filesystem integration-test config. FETCHER_BASE_PATH, EMITTER_BASE_PATH and PLUGINS_PATHS are placeholders the test harness substitutes with real paths; EMIT_INTERMEDIATE_RESULTS with the boolean true or false. See Pipes Configuration for what each setting does.

{
  "content-handler-factory": {
    "basic-content-handler-factory": {
      "type": "TEXT",
      "writeLimit": -1,
      "throwOnWriteLimitReached": true
    }
  },
  "fetchers": {
    "fsf": {
      "file-system-fetcher": {
        "basePath": "FETCHER_BASE_PATH",
        "extractFileSystemMetadata": false
      }
    }
  },
  "emitters": {
    "fse": {
      "file-system-emitter": {
        "basePath": "EMITTER_BASE_PATH",
        "fileExtension": "json",
        "onExists": "EXCEPTION"
      }
    }
  },
  "pipes-iterator": {
    "file-system-pipes-iterator": {
      "basePath": "FETCHER_BASE_PATH",
      "countTotal": true,
      "fetcherId": "fsf",
      "emitterId": "fse"
    }
  },
  "pipes": {
    "parseMode": "RMETA",
    "onParseException": "EMIT",
    "numClients": 4,
    "emitIntermediateResults": "EMIT_INTERMEDIATE_RESULTS",
    "forkedJvmArgs": ["-Xmx512m"],
    "emitStrategy": {
      "type": "DYNAMIC",
      "thresholdBytes": 1000000
    }
  },
  "auto-detect-parser": {
    "throwOnZeroBytes": false
  },
  "parse-context": {
    "mock-digester-factory": {},
    "timeout-limits": {
      "progressTimeoutMillis": 5000
    }
  },
  "plugin-roots": "PLUGINS_PATHS"
}

View source on GitHub

File System Fetcher (file-system-fetcher)

Reads files from a local or mounted filesystem. Fetch keys are resolved relative to basePath.

{
  "fetchers": {
    "fsf": {
      "file-system-fetcher": {
        "basePath": "/data/input",
        "extractFileSystemMetadata": true
      }
    }
  }
}

The outer key (fsf) is the fetcher ID — referenced by pipesIterator.fetcherId elsewhere in the config.

Configuration

Field Default Description

basePath

none

Base directory for fetch operations. Fetch keys are resolved relative to this path and must stay inside it. Omitting it requires allowAbsolutePaths: true and disables containment entirely — see Security Notes.

extractFileSystemMetadata

false

When true, attach file size, created, and modified timestamps to the metadata of each fetched document.

allowAbsolutePaths

false

Permission to run without a basePath. It is not a relaxation of basePath — see Security Notes.

File System Emitter (file-system-emitter)

Writes parsed results as files under basePath. The relative output path is derived from the emit key of each FetchEmitTuple.

{
  "emitters": {
    "fse": {
      "file-system-emitter": {
        "basePath": "/data/output",
        "fileExtension": "json",
        "onExists": "EXCEPTION",
        "prettyPrint": false
      }
    }
  }
}

Configuration

Field Default Description

basePath

none

Base output directory. Emit keys are resolved relative to this path and must stay inside it. Omitting it requires allowAbsolutePaths: true and disables containment entirely — see Security Notes.

allowAbsolutePaths

false

Permission to run without a basePath. It is not a relaxation of basePath — see Security Notes.

fileExtension

none

Extension appended to each output file; omitted entirely when unset. Applied on both output paths, so under CONTENT_ONLY set it to match the handler type (txt, html, md, xml). Without it the emit key keeps the input’s own name, and output collides with input when -i and -o are the same directory.

onExists

EXCEPTION

Behavior when the output file already exists: SKIP (do nothing), REPLACE (overwrite), EXCEPTION (fail loudly).

prettyPrint

false

Pretty-print JSON output. Has no effect in CONTENT_ONLY mode (raw bytes are written).

File System Iterator (file-system-pipes-iterator)

Recursively walks a directory tree, emitting one FetchEmitTuple per file found.

{
  "pipes-iterator": {
    "file-system-pipes-iterator": {
      "basePath": "/data/input",
      "countTotal": true,
      "fetcherId": "fsf",
      "emitterId": "fse"
    }
  }
}

Configuration

In addition to the required fetcherId / emitterId (see Wiring an Iterator):

Field Default Description

basePath

required

Root directory to walk.

countTotal

true

If true, a daemon thread walks the tree in parallel with processing to count files. Enables progress reporting at the cost of an extra scan over the tree.

Notes

  • Walk order is filesystem-dependent and not guaranteed stable across runs.

  • The relative path of each file (from basePath) becomes the fetch key, and by default also the emit key.

  • Symbolic links are not followed: the walk runs without FOLLOW_LINKS, so a symlinked directory is emitted as a single entry rather than descended into.

File System Reporter (file-system-reporter)

Maintains a JSON status file that summarizes pipeline progress. The reporter writes the file periodically on a background thread; per-record report() calls only update in-memory counters.

{
  "pipes-reporters": {
    "file-system-reporter": {
      "statusFile": "/var/log/tika/status.json",
      "reportUpdateMs": 1000
    }
  }
}

pipes-reporters accepts multiple reporters keyed by component name — see Pipes Reporters for how multiple reporters compose.

Configuration

Field Default Description

statusFile

required

Path of the JSON status file. Absolute paths are written as given; relative paths resolve against the JVM’s working directory at startup. Missing parent directories are created at startup. Always include a parent component (e.g., ./status.json rather than bare status.json) — a path with no parent throws a NullPointerException during that step. The file is created on first write and overwritten in place.

reportUpdateMs

0

Interval in milliseconds between status-file writes. Typical values: 1000 for a low-overhead heartbeat, 100 for near-real-time updates. Always set it: the value is used as the writer thread’s sleep, so leaving it unset rewrites the file in a tight loop.

Status file schema

The reporter serializes an AsyncStatus object to JSON, containing:

  • started — ISO-8601 timestamp of when the reporter was constructed.

  • lastUpdate — ISO-8601 timestamp of the most recent write.

  • asyncStatus — current pipeline phase (STARTED, COMPLETED, CRASHED).

  • statusCounts — map of PipesResult.RESULT_STATUS to count (e.g., PARSE_SUCCESS, PARSE_SUCCESS_WITH_EXCEPTION, TIMEOUT, OOM, EMIT_SUCCESS, EMIT_EXCEPTION).

  • totalCountResult — total documents discovered by the iterator and whether the enumeration is complete.

  • crashMessage — empty string under normal operation; populated with a stack trace on fatal pipeline failure.

The file is rewritten in full on each tick, not appended.

The write is not atomic — the reporter opens the target path with Files.newBufferedWriter, truncates, and streams the JSON. A watcher reading concurrently with a write can observe a truncated or partial document. Have the watcher treat a parse error as "stale read, try again on the next poll" rather than as a real error.

Live status for watching applications

The reporter is designed to support external "watchers" — UIs, dashboards, or monitoring scripts that poll the status file to display pipeline progress. To use it that way, set reportUpdateMs to match your desired refresh rate:

"reportUpdateMs": 250

The watcher polls statusFile on its own interval and reads the most recent snapshot. Each tick rewrites the file in full, so successive snapshots are always coherent — but because the write is not atomic, a watcher reading mid-write can see a truncated document. Tolerate JSON parse errors as transient and retry on the next poll (see the NOTE under Status file schema).

This pattern is used by tika-gui-v2 to drive its progress UI: the GUI starts a pipeline subprocess, points the reporter at a temp file, and polls that file every few hundred milliseconds.

Tradeoffs:

  • Smaller reportUpdateMs values mean more disk writes. On a fast SSD this is negligible, but on a slow disk (or NFS) the writer thread can become a bottleneck.

  • The reporter thread sleeps between writes, so the worst-case staleness of the file is reportUpdateMs milliseconds plus serialization time.

  • Per-record report() calls are cheap (counter increment only). The cost of "watching" is bounded by the periodic write, not by document throughput.

Security Notes

  • basePath is the sandbox boundary, and it is the only one. With basePath set, the fetcher and emitter reject any key that resolves outside it, including absolute paths and ../ traversal. allowAbsolutePaths has no effect in this state.

  • Without basePath there is no containment at all. The key is used as a raw absolute path, and the containment checks are skipped entirely. allowAbsolutePaths=true is how you assert that you intend this; it is a switch between two states, not a dial that loosens basePath. For the fetcher this means any file the process can read; for the emitter, any file it can write. Use it only when fetch/emit keys come from a fully trusted source and access to the service is restricted by other means.

  • Symlink containment differs between fetcher and emitter. The fetcher re-checks with toRealPath(), so a symlink under basePath pointing outside it is rejected. The emitter does not: it checks only the normalized path, so a symlink already present under its basePath can be written through. Do not rely on symlinks being contained on the emit side.

  • Output directories are created automatically. The emitter creates intermediate directories as needed. Make sure the process’s umask is appropriate for the data being written.