Elasticsearch Plugin

The Elasticsearch plugin (tika-pipes-es) provides an emitter (writes parsed docs to an Elasticsearch index) and a reporter (writes per-document processing status to Elasticsearch).

It mirrors the OpenSearch plugin in structure. The field names differ — esUrl instead of openSearchUrl — and ES adds an apiKey field for ApiKey-based auth in addition to basic auth.

Interface Component name Class

Emitter

es-emitter

ESEmitter

Reporter

es-pipes-reporter

ESPipesReporter

Authentication

Two auth modes are supported, in this priority order:

  1. ApiKey — set the top-level apiKey field to the Base64-encoded id:api_key string Elasticsearch generates. Sent as Authorization: ApiKey <value>.

  2. Basic — leave apiKey null/empty and set userName + password inside httpClientConfig. Sent as Authorization: Basic …​.

The emitter overrides toString() to redact the apiKey value, so it does not leak into logs.

Shared HTTP Client Settings

Both the emitter and the reporter take an optional nested httpClientConfig block. Omit it entirely and every field below keeps its default.

Field Default Description

userName / password

none

Basic-auth credentials. Used only when apiKey is unset.

authScheme

basic

Credential scheme. basic or ntlm.

connectionTimeoutMillis

120000

HTTP connect timeout, ms. Values of 0 or less are ignored and the default stands.

socketTimeoutMillis

120000

HTTP socket read timeout, ms. Values of 0 or less are ignored and the default stands.

proxyHost / proxyPort

none / 0

Outbound HTTP proxy. proxyPort is read only when proxyHost is set.

verifySsl

true

Verify server certificates and hostnames against the JVM trust store.

An https:// esUrl is therefore encrypted and authenticated. If the cluster uses a private CA, add it to the JVM truststore. To disable verification — self-signed certificates are the usual reason — set it on the emitter and the reporter separately:

"httpClientConfig": { "verifySsl": false }

With verifySsl: false the client accepts any certificate from any host and skips the hostname check, so anything on the network path can read and alter what is indexed. It logs a WARN saying so at startup. Prefer trusting the CA.

Elasticsearch Emitter (es-emitter)

Writes parsed documents to an Elasticsearch index.

{
  "emitters": {
    "ese": {
      "es-emitter": {
        "esUrl": "https://es.example.com:9200/tika-docs",
        "idField": "doc_id",
        "attachmentStrategy": "PARENT_CHILD",
        "updateStrategy": "OVERWRITE",
        "commitWithin": 1000,
        "embeddedFileFieldName": "embedded",
        "apiKey": "REDACTED_BASE64_ID_AND_KEY",
        "httpClientConfig": {
          "connectionTimeoutMillis": 10000,
          "socketTimeoutMillis": 60000
        }
      }
    }
  }
}

Configuration

Field Default Description

esUrl

required

Full URL of the target Elasticsearch index, e.g., https://es.example.com:9200/tika-docs.

idField

required

Validated non-blank, but not otherwise read: the _id is the emit key (attachments get the emit key plus a random suffix).

attachmentStrategy

SEPARATE_DOCUMENTS

How attached/embedded documents are indexed. SEPARATE_DOCUMENTS — each attachment becomes its own top-level document; PARENT_CHILD — attachments are nested under the parent in a parent/child relation. Absent behaves as SEPARATE_DOCUMENTS.

updateStrategy

UPSERT

How existing documents are handled. OVERWRITE — replaces an existing document at _id; UPSERT — merges into an existing document. Absent behaves as UPSERT.

commitWithin

unused

Accepted by the config parser but never read. Control refresh visibility on the Elasticsearch side instead.

embeddedFileFieldName

none

Name of the field used to hold embedded-file content (used by PARENT_CHILD).

apiKey

none

Base64-encoded id:api_key. See Authentication.

httpClientConfig

none

See Shared HTTP Client Settings.

Elasticsearch Reporter (es-pipes-reporter)

Writes per-document processing status records to an Elasticsearch index. Useful for building dashboards over pipeline activity.

{
  "pipes-reporters": {
    "es-pipes-reporter": {
      "esUrl": "https://es.example.com:9200/tika-status",
      "includes": ["PARSE_SUCCESS", "PARSE_EXCEPTION_NO_EMIT", "OOM", "TIMEOUT"],
      "keyPrefix": "tika_",
      "includeRouting": true,
      "apiKey": "REDACTED_BASE64_ID_AND_KEY",
      "httpClientConfig": {
        "connectionTimeoutMillis": 10000,
        "socketTimeoutMillis": 60000
      }
    }
  }
}

pipes-reporters accepts multiple reporters keyed by component name — see Pipes Reporters for how multiple reporters compose.

Configuration

Field Default Description

esUrl

required

Full URL of the status index, e.g., https://es.example.com:9200/tika-status.

includes

none

Set of RESULT_STATUS names to include (e.g., PARSE_SUCCESS, PARSE_SUCCESS_WITH_EXCEPTION). Names are resolved with RESULT_STATUS.valueOf, so an unrecognised name fails at startup. If unset, all are reported.

excludes

none

Set of RESULT_STATUS names to skip. Mutually exclusive with includes; setting both is a config error.

keyPrefix

none

Prefix prepended to status field names in the emitted documents.

includeRouting

false

If true, include ES routing info in each status record.

apiKey

none

Base64-encoded id:api_key. See Authentication.

httpClientConfig

none

See Shared HTTP Client Settings.

Complete Pipeline Example

A filesystem iterator/fetcher feeding the Elasticsearch emitter and reporter — the usual shape for ingesting a directory into ES.

{
  "content-handler-factory": {
    "basic-content-handler-factory": {
      "type": "TEXT",
      "writeLimit": -1,
      "throwOnWriteLimitReached": true
    }
  },
  "fetchers": {
    "fsf": {
      "file-system-fetcher": {
        "basePath": "/data/input",
        "extractFileSystemMetadata": false
      }
    }
  },
  "emitters": {
    "ese": {
      "es-emitter": {
        "esUrl": "https://es.example.com:9200/tika-docs",
        "idField": "doc_id",
        "attachmentStrategy": "PARENT_CHILD",
        "updateStrategy": "OVERWRITE",
        "commitWithin": 1000,
        "embeddedFileFieldName": "embedded",
        "apiKey": "REDACTED_BASE64_ID_AND_KEY",
        "httpClientConfig": {
          "connectionTimeoutMillis": 10000,
          "socketTimeoutMillis": 60000
        }
      }
    }
  },
  "pipes-iterator": {
    "file-system-pipes-iterator": {
      "basePath": "/data/input",
      "countTotal": true,
      "fetcherId": "fsf",
      "emitterId": "ese"
    }
  },
  "pipes-reporters": {
    "es-pipes-reporter": {
      "esUrl": "https://es.example.com:9200/tika-status",
      "includes": ["PARSE_SUCCESS", "PARSE_EXCEPTION_NO_EMIT", "OOM", "TIMEOUT"],
      "keyPrefix": "tika_",
      "includeRouting": true,
      "apiKey": "REDACTED_BASE64_ID_AND_KEY",
      "httpClientConfig": {
        "connectionTimeoutMillis": 10000,
        "socketTimeoutMillis": 60000
      }
    }
  },
  "pipes": {
    "parseMode": "RMETA",
    "onParseException": "EMIT",
    "numClients": 4
  }
}

Notes

  • The ES plugin’s HTTP client is REST-based; it does not depend on the Elasticsearch transport client.

  • For OpenSearch deployments, use the parallel OpenSearch plugin instead — the field names differ (openSearchUrl vs. esUrl).

  • Don’t check real credentials into source control — the apiKey and password values in the examples above are placeholders.