HTTP Plugin

The HTTP plugin (tika-pipes-http) provides a fetcher that downloads documents over HTTP(S). It is fetcher-only — pair it with another emitter and iterator.

Interface Component name Class

Fetcher

http-fetcher

HttpFetcher

HTTP Fetcher (http-fetcher)

Fetches document bytes from an HTTP(S) URL. The fetch key is the URL.

{
  "fetchers": {
    "httpf": {
      "http-fetcher": {
        "userName": "tika",
        "password": "REDACTED",
        "authScheme": "basic",
        "userAgent": "tika-pipes/1.0",
        "maxConnections": 2000,
        "maxConnectionsPerRoute": 1000,
        "connectTimeoutMillis": 30000,
        "socketTimeoutMillis": 60000,
        "requestTimeoutMillis": 60000,
        "overallTimeoutMillis": 120000,
        "maxRedirects": 5,
        "maxSpoolSize": -1,
        "httpHeaders": ["Accept: application/octet-stream"]
      }
    }
  }
}

Configuration

Field Default Description

userName / password

none

Credentials. Applied only when authScheme is also set — without it they are silently ignored.

ntDomain

none

NT domain for NTLM auth. Same authScheme precondition.

authScheme

none

basic or ntlm. Acts as the switch that enables userName/password/ntDomain.

proxyHost / proxyPort

none

Outbound HTTP proxy.

userAgent

none

User-Agent header sent on each request.

maxConnections

2000

HTTP connection-pool size.

maxConnectionsPerRoute

1000

Per-route connection-pool size.

connectTimeoutMillis

120000

TCP connect timeout.

socketTimeoutMillis

120000

Socket read timeout.

requestTimeoutMillis

120000

Connection-manager request timeout.

overallTimeoutMillis

120000

Hard cap on total time for a single fetch operation.

maxRedirects

0

Maximum number of redirects to follow. 0 means follow none.

verifySsl

true

Verify server certificates and hostnames. Set false to accept any certificate — see Security Notes.

maxSpoolSize

-1

Maximum bytes to spool locally before failing. -1 means no limit.

maxErrMsgSize

10000000

Maximum bytes of error response body to capture into the exception.

httpHeaders

empty

Extra HTTP headers, formatted as "Header: value" strings (list).

httpRequestHeaders

empty

Structured per-request headers as a Header → [values] map. Used when a header has multiple values.

jwtIssuer / jwtSubject / jwtExpiresInSeconds

none

JWT claims, for endpoints that accept JWT-bearer auth.

jwtSecret

none

HMAC secret for symmetric-key JWT signing.

jwtPrivateKeyBase64

none

Base64-encoded private key for asymmetric (RSA/ECDSA) JWT signing. Mutually exclusive with jwtSecret.

Security Notes

This fetcher makes the server issue HTTP requests to a URL supplied as the fetch key. That is a server-side request forgery primitive by design, and it is not constrained by this plugin. Treat the source of fetch keys as fully trusted, and restrict access to any endpoint that can reach it (/pipes, /async).

  • The fetch key is used as the URL with no validation. It is passed straight to new HttpGet(fetchKey). There is no scheme allowlist, no host denylist, and no check against loopback, link-local, or RFC1918 addresses. A fetch key of http://169.254.169.254/…​; reaches a cloud metadata endpoint like any other URL. The resolved address is recorded in metadata after the fetch, not consulted before it. Schemes other than http/https fail only because no other scheme is registered in the connection manager — that is a side effect of the transport setup, not a check.

  • TLS certificates are verified by default (verifySsl: true). Not every HTTP-based plugin does this — check the plugin page before assuming.

    To opt out — self-signed internal certificates are the usual reason — set verifySsl to false on the fetcher:

    {
  "fetchers": {
    "my-http-fetcher": {
      "http-fetcher": { "verifySsl": false }
    }
  }
}

    That installs an accept-everything trust strategy and NoopHostnameVerifier: any certificate from any host is accepted, so anything on the network path can read and alter what is fetched. Prefer adding your CA to the JVM truststore over turning this off.

  • There is no redirect host allowlist. Redirects are followed up to maxRedirects regardless of the target host — a redirect can send the fetch to any host the server can reach, including loopback, link-local, or RFC1918 addresses. Set maxRedirects to 0 (the default) to follow no redirects at all.

Notes

  • Both basic auth and JWT auth may be configured at the same time, but only one will apply per request (JWT takes precedence when present).

  • For zero-redirect crawling, leave maxRedirects at 0. The fetcher returns the redirect response as-is so the caller can decide what to do.

  • overallTimeoutMillis is enforced by the fetcher itself, not the HTTP client — it covers slow drains and zombie connections that the lower-level timeouts may miss.

  • For Atlassian Cloud endpoints that require an Atlassian Connect JWT, use the dedicated Atlassian JWT fetcher instead — it has the correct claim layout baked in.