Amazon S3 Plugin

The Amazon S3 plugin (tika-pipes-s3) provides fetcher, emitter, and iterator interfaces for objects in S3 (or any S3-compatible service such as MinIO).

Interface Component name Class

Fetcher

s3-fetcher

S3Fetcher

Emitter

s3-emitter

S3Emitter

Iterator

s3-pipes-iterator

S3PipesIterator

Credentials

All three components share the same credentialsProvider selector:

  • profile — reads credentials from the local AWS profile named by profile (e.g., default).

  • instance — uses the IAM role of the EC2 instance the host runs on, read from the instance metadata service. No additional fields needed.

  • web_identity — assumes a role with a web identity token, as EKS IAM roles for service accounts (IRSA) provide. Reads the role from AWS_ROLE_ARN and the token from AWS_WEB_IDENTITY_TOKEN_FILE, which EKS sets on the pod. No additional fields needed.

  • key_secret — reads accessKey and secretKey from the config. Avoid checking these into source control; prefer environment-variable substitution or one of the other providers.

All three components reject any other value when they are created. If credentialsProvider is unset, the emitter’s validate() rejects it, but the fetcher and iterator fall back to instance.

ECS task roles and EKS Pod Identity are not supported — neither instance nor web_identity picks them up.

S3 Fetcher (s3-fetcher)

Reads objects from an S3 bucket. The fetch key is the S3 key under prefix (if set).

{
  "fetchers": {
    "s3f": {
      "s3-fetcher": {
        "bucket": "my-tika-input",
        "region": "us-east-1",
        "prefix": "incoming/",
        "credentialsProvider": "profile",
        "profile": "default",
        "extractUserMetadata": true,
        "spoolToTemp": true
      }
    }
  }
}

Configuration

Field Default Description

bucket

required

S3 bucket name.

region

required

AWS region (e.g., us-east-1).

prefix

none

Optional key prefix, prepended verbatim to the fetch key — include the trailing / yourself. Leave it unset when the fetch keys already carry the prefix (see [prefix-warning]).

credentialsProvider

instance

One of profile, instance, web_identity, key_secret. See Credentials.

profile / accessKey / secretKey

conditional

Required by the matching credentialsProvider. ${env:AWS_SECRET_ACCESS_KEY} keeps the secret out of the file (Secrets from the environment).

spoolToTemp

true

If true, the fetched object is spooled to a temp file before being parsed.

extractUserMetadata

true

If true, S3 user-metadata is copied into the parsed Metadata.

maxConnections

0

Maximum HTTP connections in the S3 client pool. 0 or less falls back to the SDK’s own default (50).

maxLength

-1

Maximum object size, in bytes. -1 means no limit.

endpointConfigurationService

none

Custom S3 endpoint, for S3-compatible services such as MinIO or LocalStack.

pathStyleAccessEnabled

false

Force path-style URLs (e.g., https://endpoint/bucket/key). Required by some S3-compatible services.

throttleSeconds

none

Retry/back-off array. Its length is the number of attempts, and entry i is the seconds to sleep before retry i. Absent means a single attempt with no retry.

S3 Emitter (s3-emitter)

Writes parsed results back to an S3 bucket. The emit key (relative to prefix) is derived from the FetchEmitTuple.

{
  "emitters": {
    "s3e": {
      "s3-emitter": {
        "bucket": "my-tika-output",
        "region": "us-east-1",
        "prefix": "results/",
        "fileExtension": "json",
        "credentialsProvider": "profile",
        "profile": "default"
      }
    }
  }
}

Configuration

Field Default Description

bucket

required

Destination S3 bucket name (validated non-blank).

region

required

AWS region (validated non-blank).

credentialsProvider

required

One of profile, instance, web_identity, key_secret (validated). See Credentials.

profile / accessKey / secretKey

conditional

Required by the matching credentialsProvider (validated).

prefix

none

Optional key prefix. A trailing / is stripped automatically.

fileExtension

json

Extension appended to each emitted key. Set it to "" to append nothing.

spoolToTemp

true

If true, output is spooled to a temp file before being uploaded; if false, it is buffered in memory.

maxConnections

50

Maximum HTTP connections in the S3 client pool. The value goes straight to the underlying connection manager, which rejects 0 and negatives.

endpointConfigurationService

none

Custom S3 endpoint, for S3-compatible services.

pathStyleAccessEnabled

false

Force path-style URLs.

S3 Iterator (s3-pipes-iterator)

Lists objects under a bucket/prefix and emits one FetchEmitTuple per object found.

{
  "pipes-iterator": {
    "s3-pipes-iterator": {
      "bucket": "my-tika-input",
      "region": "us-east-1",
      "prefix": "incoming/",
      "credentialsProvider": "profile",
      "profile": "default",
      "fetcherId": "s3f",
      "emitterId": "s3e"
    }
  }
}

Configuration

In addition to the required fetcherId / emitterId (see Wiring an Iterator):

Field Default Description

bucket

required

S3 bucket to enumerate.

region

required

AWS region.

prefix

""

Key prefix to scope the listing.

credentialsProvider

instance

One of profile, instance, web_identity, key_secret. See Credentials.

profile / accessKey / secretKey / endpointConfigurationService

conditional

Auth fields, mirroring the fetcher and emitter.

fileNamePattern

none

Optional regex, applied with find() to the last path segment of the key only. It is a substring search, so anchor it (^…​$) if you want a whole-name match.

maxConnections

50

Maximum HTTP connections in the S3 client pool.

pathStyleAccessEnabled

false

Force path-style URLs.

Complete Pipeline Example

Fetcher, emitter, and iterator wired to list s3://my-tika-input/incoming/ and write results to s3://my-tika-output/results/.

{
  "content-handler-factory": {
    "basic-content-handler-factory": {
      "type": "TEXT",
      "writeLimit": -1,
      "throwOnWriteLimitReached": true
    }
  },
  "fetchers": {
    "s3f": {
      "s3-fetcher": {
        "bucket": "my-tika-input",
        "region": "us-east-1",
        "credentialsProvider": "profile",
        "profile": "default",
        "extractUserMetadata": true
      }
    }
  },
  "emitters": {
    "s3e": {
      "s3-emitter": {
        "bucket": "my-tika-output",
        "region": "us-east-1",
        "prefix": "results/",
        "fileExtension": "json",
        "credentialsProvider": "profile",
        "profile": "default"
      }
    }
  },
  "pipes-iterator": {
    "s3-pipes-iterator": {
      "bucket": "my-tika-input",
      "region": "us-east-1",
      "prefix": "incoming/",
      "credentialsProvider": "profile",
      "profile": "default",
      "fetcherId": "s3f",
      "emitterId": "s3e"
    }
  },
  "pipes": {
    "parseMode": "RMETA",
    "onParseException": "EMIT",
    "numClients": 4
  }
}

The iterator’s prefix and the fetcher’s prefix do not compose — they stack. The iterator emits the full S3 key it listed (incoming/report.pdf), and the fetcher then prepends its own prefix to that, asking for incoming/incoming/report.pdf — every fetch misses.

Set the prefix on exactly one side: on the iterator (and leave the fetcher’s unset), as the example above does, or on the fetcher when fetch keys come from somewhere that emits bare names. The emitter’s prefix is independent of both and is safe to set alongside either.

Notes

  • The fetcher, emitter, and iterator each maintain their own S3 client. Auth and endpoint settings need to be configured per component, not globally.

  • The S3 SDK enforces TLS 1.2+ by default; in-flight encryption is on. For at-rest encryption, configure bucket-level SSE on the AWS side.

  • When using endpointConfigurationService against MinIO or LocalStack, you almost always need pathStyleAccessEnabled: true.