Package org.apache.tika.pipes.core

Class PipesConfig

java.lang.Object
org.apache.tika.pipes.core.PipesConfig
public class PipesConfig extends Object

  • Field Details

    • DEFAULT_MAX_IPC_PAYLOAD_BYTES

      public static final int DEFAULT_MAX_IPC_PAYLOAD_BYTES
      See Also:

    • DEFAULT_MAX_INLINE_BYTES

      public static final int DEFAULT_MAX_INLINE_BYTES
      Largest request body carried inline to the forked worker rather than spooled to disk.

      Sized for the common case -- most documents are far smaller -- because the cost is heap, not disk: the parent holds the payload and the Smile frame containing a copy of it, and the child holds it again. Budget roughly 2 * maxInlineBytes * concurrent-requests in the parent before raising this.

      See Also:

    • MAX_AUTO_NUM_CLIENTS

      public static final int MAX_AUTO_NUM_CLIENTS
      Past this, worker count becomes a memory decision, and memory is not visible here.
      See Also:

    • DEFAULT_MAX_FILES_PROCESSED_PER_PROCESS

      public static final int DEFAULT_MAX_FILES_PROCESSED_PER_PROCESS
      See Also:

    • DEFAULT_MAX_WAIT_FOR_CLIENT_MILLIS

      public static final long DEFAULT_MAX_WAIT_FOR_CLIENT_MILLIS
      See Also:

    • DEFAULT_SOCKET_TIMEOUT_MILLIS

      public static final long DEFAULT_SOCKET_TIMEOUT_MILLIS
      See Also:

    • DEFAULT_STARTUP_TIMEOUT_MILLIS

      public static final long DEFAULT_STARTUP_TIMEOUT_MILLIS
      See Also:

    • DEFAULT_HEARTBEAT_INTERVAL_MILLIS

      public static final long DEFAULT_HEARTBEAT_INTERVAL_MILLIS
      See Also:

    • DEFAULT_USE_SHARED_SERVER

      public static final boolean DEFAULT_USE_SHARED_SERVER
      See Also:

    • DEFAULT_MAX_TOTAL_TASK_TIMEOUT_MILLIS

      public static final long DEFAULT_MAX_TOTAL_TASK_TIMEOUT_MILLIS
      See Also:

    • DEFAULT_EMIT_WITHIN_MILLIS

      public static final long DEFAULT_EMIT_WITHIN_MILLIS
      See Also:

    • DEFAULT_EMIT_MAX_ESTIMATED_BYTES

      public static final long DEFAULT_EMIT_MAX_ESTIMATED_BYTES
      See Also:

    • DEFAULT_QUEUE_SIZE

      public static final int DEFAULT_QUEUE_SIZE
      See Also:

    • DEFAULT_NUM_EMITTERS

      public static final int DEFAULT_NUM_EMITTERS
      See Also:

  • Constructor Details

    • PipesConfig

      public PipesConfig()

  • Method Details

    • defaultNumClients

      public static int defaultNumClients()
      Worker count when the operator has not chosen one. CPU-derived, so the default satisfies Tika's own sizing rule on any host; a fixed 4 needs 10 cores and would warn about itself on smaller ones. Memory cannot participate -- no Java SE API exposes container memory -- so each fork checks its own heap at startup instead.

    • load

      public static PipesConfig load(TikaJsonConfig tikaJsonConfig) throws IOException, TikaConfigException
      Loads PipesConfig from the "pipes" section of the JSON configuration.

      This configuration is used by both PipesServer (forking process) and AsyncProcessor (async processing). Some fields are specific to each:

      • PipesServer uses: numClients, socketTimeoutMillis, directEmitThresholdBytes, etc.
      • AsyncProcessor uses: emitWithinMillis, queueSize, numEmitters, etc.
      Unused fields in each context are simply ignored.
      Parameters:
      tikaJsonConfig - the JSON configuration to load from
      Returns:
      the loaded PipesConfig, or a new default instance if not found in config
      Throws:
      IOException - if deserialization fails
      TikaConfigException - if configuration is invalid

    • getDefaultTimeoutLimits

      public TimeoutLimits getDefaultTimeoutLimits()
      The config-level parse-context.timeout-limits defaults -- what the forked server enforces when a request carries no TimeoutLimits of its own.

    • getSocketTimeoutMillis

      public long getSocketTimeoutMillis()

    • setSocketTimeoutMillis

      public void setSocketTimeoutMillis(long socketTimeoutMillis)
      Socket timeout in milliseconds for reading from the forked process. If no data is received within this time, the connection is considered timed out. This is distinct from the parse/processing timeout, which lives on TimeoutLimits under parse-context.timeout-limits.
      Parameters:
      socketTimeoutMillis -

    • getMaxTotalTaskTimeoutMillis

      public long getMaxTotalTaskTimeoutMillis()

    • setMaxTotalTaskTimeoutMillis

      public void setMaxTotalTaskTimeoutMillis(long maxTotalTaskTimeoutMillis)

    • getStartupTimeoutMillis

      public long getStartupTimeoutMillis()

    • setStartupTimeoutMillis

      public void setStartupTimeoutMillis(long startupTimeoutMillis)
      Timeout in milliseconds for the forked server to start up and send its READY handshake. Distinct from getSocketTimeoutMillis(): cold-starting the forked JVM (loading config, parsers and plugins) can take far longer than a normal per-read timeout, so the handshake gets its own generous budget. Once the server is ready, reads switch to socketTimeoutMillis.
      Parameters:
      startupTimeoutMillis -

    • getHeartbeatIntervalMillis

      public long getHeartbeatIntervalMillis()

    • setHeartbeatIntervalMillis

      public void setHeartbeatIntervalMillis(long heartbeatIntervalMillis)
      Interval in milliseconds between heartbeat messages sent from server to client. Should be significantly less than socketTimeoutMillis to ensure the client doesn't timeout. WARNING: Setting this >= socketTimeoutMillis will cause socket timeouts during normal processing. This only exists for testing. We encourage you never to use it.
      Parameters:
      heartbeatIntervalMillis -

    • getNumClients

      public int getNumClients()

    • setNumClients

      public void setNumClients(int numClients)

    • setForkedJvmArgs

      public void setForkedJvmArgs(ArrayList<String> jvmArgs)

    • getForkedJvmArgs

      public ArrayList<String> getForkedJvmArgs()

    • getMaxFilesProcessedPerProcess

      public int getMaxFilesProcessedPerProcess()
      Restart the forked PipesServer after it has processed this many files to avoid slow-building memory leaks.
      Returns:

    • setMaxFilesProcessedPerProcess

      public void setMaxFilesProcessedPerProcess(int maxFilesProcessedPerProcess)

    • getJavaPath

      public String getJavaPath()

    • setJavaPath

      public void setJavaPath(String javaPath)

    • getEmitStrategy

      public EmitStrategyConfig getEmitStrategy()
      Get the emit strategy configuration.
      Returns:
      the emit strategy configuration

    • setEmitStrategy

      public void setEmitStrategy(EmitStrategyConfig emitStrategy)
      Set the emit strategy configuration.
      Parameters:
      emitStrategy - the emit strategy configuration

    • getMaxWaitForClientMillis

      public long getMaxWaitForClientMillis()

    • setMaxWaitForClientMillis

      public void setMaxWaitForClientMillis(long maxWaitForClientMillis)

    • getEmitWithinMillis

      public long getEmitWithinMillis()

    • setEmitWithinMillis

      public void setEmitWithinMillis(long emitWithinMillis)
      If nothing has been emitted in this amount of time and the getEmitMaxEstimatedBytes() has not been reached yet, emit what's in the emit queue.
      Parameters:
      emitWithinMillis - time in milliseconds

    • getEmitMaxEstimatedBytes

      public long getEmitMaxEstimatedBytes()
      When the emit queue hits this estimated size (sum of estimated extract sizes), emit the batch.
      Returns:
      the maximum estimated bytes before emitting

    • setEmitMaxEstimatedBytes

      public void setEmitMaxEstimatedBytes(long emitMaxEstimatedBytes)

    • getQueueSize

      public int getQueueSize()
      FetchEmitTuple queue size
      Returns:
      the queue size

    • setQueueSize

      public void setQueueSize(int queueSize)

    • getNumEmitters

      public int getNumEmitters()
      Number of emitters
      Returns:
      the number of emitters

    • setNumEmitters

      public void setNumEmitters(int numEmitters)

    • isEmitIntermediateResults

      public boolean isEmitIntermediateResults()

    • setEmitIntermediateResults

      public void setEmitIntermediateResults(boolean emitIntermediateResults)

    • isStopOnlyOnFatal

      public boolean isStopOnlyOnFatal()
      When true, only stop processing on fatal errors (FAILED_TO_INITIALIZE). When false (default), also stop on initialization failures (FETCHER_INITIALIZATION_EXCEPTION, EMITTER_INITIALIZATION_EXCEPTION, CLIENT_UNAVAILABLE_WITHIN_MS) and not-found errors (FETCHER_NOT_FOUND, EMITTER_NOT_FOUND).

      Use true for server mode (tika-server /pipes, /async) where different requests may use different fetchers/emitters - a bad request shouldn't kill the server. Use false (default) for CLI batch mode where all tasks typically use the same fetcher/emitter configuration - no point continuing if configuration is wrong.

      Returns:
      true if only fatal errors should stop processing

    • setStopOnlyOnFatal

      public void setStopOnlyOnFatal(boolean stopOnlyOnFatal)

    • getParseMode

      public ParseMode getParseMode()
      Gets the default parse mode for how embedded documents are handled.
      Returns:
      the default parse mode

    • setParseMode

      public void setParseMode(ParseMode parseMode)
      Sets the default parse mode for how embedded documents are handled. This can be overridden per-file via ParseContext.
      Parameters:
      parseMode - the parse mode (RMETA or CONCATENATE)

    • setParseMode

      public void setParseMode(String parseMode)
      Sets the default parse mode from a string.
      Parameters:
      parseMode - the parse mode name (rmeta or concatenate)

    • getOnParseException

      public FetchEmitTuple.ON_PARSE_EXCEPTION getOnParseException()
      Gets the default behavior when a parse exception occurs.
      Returns:
      the parse exception behavior

    • setOnParseException

      public void setOnParseException(FetchEmitTuple.ON_PARSE_EXCEPTION onParseException)
      Sets the default behavior when a parse exception occurs.
      Parameters:
      onParseException - the parse exception behavior

    • getConfigStoreType

      public String getConfigStoreType()

    • setConfigStoreType

      public void setConfigStoreType(String configStoreType)

    • getConfigStoreParams

      public String getConfigStoreParams()

    • setConfigStoreParams

      public void setConfigStoreParams(String configStoreParams)

    • getTempDirectory

      public String getTempDirectory()
      Gets the directory for temporary files during pipes-based parsing.
      Returns:
      the temp directory path, or null to use system default

    • createTempDirectory

      public Path createTempDirectory(String prefix) throws IOException
      Creates a temp directory under getTempDirectory(), or under the system default when unset. Callers must not use Files.createTempDirectory directly or the configured directory is silently ignored.
      Throws:
      IOException

    • setTempDirectory

      public void setTempDirectory(String tempDirectory)
      Sets the directory for temporary files during pipes-based parsing. If not set, the system default temp directory will be used. Consider using a RAM-backed filesystem (e.g., /dev/shm or /tmpfs) for better performance.
      Parameters:
      tempDirectory - the temp directory path, or null to use system default

    • isUseSharedServer

      public boolean isUseSharedServer()
      Returns whether shared server mode is enabled.
      Returns:
      true if shared server mode is enabled
      See Also:

    • setUseSharedServer

      public void setUseSharedServer(boolean useSharedServer)
      Sets whether to use shared server mode.

      When true, multiple PipesClients connect to a single shared PipesServer process instead of each client having its own dedicated server. This reduces memory overhead but sacrifices isolation: one crash affects all in-flight requests.

      Not recommended for production. See the Tika Pipes documentation for limitations and guidance.

      Parameters:
      useSharedServer - true to enable shared server mode, false for per-client mode (default)

    • getMaxIpcPayloadBytes

      public int getMaxIpcPayloadBytes()
      Returns the maximum IPC payload size in bytes. Configurable via maxIpcPayloadBytes in the pipes section of tika-config.json.
      Returns:
      the maximum IPC payload size in bytes (default 100 MB)

    • getMaxInlineBytes

      public int getMaxInlineBytes()
      Returns:
      largest request body sent inline instead of spooled; see DEFAULT_MAX_INLINE_BYTES

    • setMaxInlineBytes

      public void setMaxInlineBytes(int maxInlineBytes)
      Sets the inline-payload threshold. Must stay under maxIpcPayloadBytes: the payload travels inside the NEW_REQUEST frame, so a threshold above that limit would let the parent build requests the child refuses, surfacing as an undiagnosable crash rather than a clean fallback to spooling. The pair is checked in checkPayloadLimits(), not here, so the two fields may be set in either order.
      Throws:
      IllegalArgumentException - if negative

    • setMaxIpcPayloadBytes

      public void setMaxIpcPayloadBytes(int maxIpcPayloadBytes)
      Sets the maximum IPC payload size in bytes. This limit is bidirectional: it controls both the largest result the client will accept back from the forked server (the FINISHED payload) and the largest request the server will accept from the client (the NEW_REQUEST payload). Lowering this value below the size of a typical FetchEmitTuple will cause requests to be rejected on the server side and reported as undiagnosable UNSPECIFIED_CRASH errors.

      The value must be at least ServerProtocolIO.MIN_FALLBACK_PAYLOAD_BYTES so that the server can always write a PAYLOAD_LIMIT_EXCEEDED response that the client will accept.

      Parameters:
      maxIpcPayloadBytes - payload limit in bytes (must be ≥ ServerProtocolIO.MIN_FALLBACK_PAYLOAD_BYTES)
      Throws:
      IllegalArgumentException - if the value is below the minimum

    • checkPayloadLimits

      public void checkPayloadLimits()
      Checks that maxInlineBytes leaves headroom for the rest of the tuple (metadata, parseContext) inside maxIpcPayloadBytes. Runs automatically after Jackson deserialization; call it directly after configuring an instance through setters.
      Throws:
      IllegalArgumentException - if the pair is inconsistent