Package org.apache.tika.pipes.pipesiterator

Class PipesIterator

java.lang.Object

org.apache.tika.config.ConfigBase

org.apache.tika.pipes.pipesiterator.PipesIterator

All Implemented Interfaces:

Iterable<FetchEmitTuple>, Callable<Integer>, Initializable

Direct Known Subclasses:

AZBlobPipesIterator, CSVPipesIterator, FileListPipesIterator, FileSystemPipesIterator, GCSPipesIterator, JDBCPipesIterator, JsonPipesIterator, KafkaPipesIterator, S3PipesIterator, SolrPipesIterator

public abstract class PipesIterator
extends ConfigBase
implements Callable<Integer>, Iterable<FetchEmitTuple>, Initializable

Abstract class that handles the testing for timeouts/thread safety issues. Concrete classes implement the blocking enqueue(). If there's an exception in the enqueuing thread, this will throw a RuntimeException. It will throw an IllegalStateException if next() is called after hasNext() has returned false.

Field Summary

Fields

Modifier and Type	Field	Description
static final FetchEmitTuple	COMPLETED_SEMAPHORE
static final long	DEFAULT_MAX_WAIT_MS
static final int	DEFAULT_QUEUE_SIZE

Constructor Summary

Constructors

Constructor	Description
PipesIterator()

Method Summary

Modifier and Type	Method	Description
static PipesIterator	build(Path tikaConfigFile)
Integer	call()
void	checkInitialization(InitializableProblemHandler problemHandler)
protected abstract void	enqueue()
String	getEmitterName()
String	getFetcherName()
protected HandlerConfig	getHandlerConfig()
FetchEmitTuple.ON_PARSE_EXCEPTION	getOnParseException()
void	initialize(Map<String,Param> params)
Iterator<FetchEmitTuple>	iterator()
void	setEmitterName(String emitterName)
void	setFetcherName(String fetcherName)
void	setHandlerType(String handlerType)
void	setMaxEmbeddedResources(int maxEmbeddedResources)
void	setMaxWaitMs(long maxWaitMs)
void	setOnParseException(String onParseException)
void	setOnParseException(FetchEmitTuple.ON_PARSE_EXCEPTION onParseException)
void	setParseMode(String parseModeString)
void	setParseMode(HandlerConfig.PARSE_MODE parseMode)
void	setQueueSize(int queueSize)
void	setThrowOnWriteLimitReached(boolean throwOnWriteLimitReached)
void	setWriteLimit(int writeLimit)
protected void	tryToAdd(FetchEmitTuple p)

Methods inherited from class org.apache.tika.config.ConfigBase

buildComposite, buildComposite, buildSingle, buildSingle, configure, handleSettings

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Details

DEFAULT_MAX_WAIT_MS

public static final long DEFAULT_MAX_WAIT_MS

See Also:

• Constant Field Values

DEFAULT_QUEUE_SIZE

public static final int DEFAULT_QUEUE_SIZE

See Also:

• Constant Field Values

COMPLETED_SEMAPHORE

public static final FetchEmitTuple COMPLETED_SEMAPHORE

Constructor Details

PipesIterator

public PipesIterator()

Method Details

build

public static PipesIterator build(Path tikaConfigFile)
                           throws IOException,
TikaConfigException

Throws:

IOException

TikaConfigException

getFetcherName

public String getFetcherName()

setFetcherName

@Field
public void setFetcherName(String fetcherName)

getEmitterName

public String getEmitterName()

setEmitterName

@Field
public void setEmitterName(String emitterName)

setMaxWaitMs

@Field
public void setMaxWaitMs(long maxWaitMs)

setQueueSize

@Field
public void setQueueSize(int queueSize)

getOnParseException

public FetchEmitTuple.ON_PARSE_EXCEPTION getOnParseException()

setOnParseException

@Field
public void setOnParseException(String onParseException)
                         throws TikaConfigException

Throws:

TikaConfigException

setOnParseException

public void setOnParseException(FetchEmitTuple.ON_PARSE_EXCEPTION onParseException)

setHandlerType

@Field
public void setHandlerType(String handlerType)

setWriteLimit

@Field
public void setWriteLimit(int writeLimit)

setThrowOnWriteLimitReached

@Field
public void setThrowOnWriteLimitReached(boolean throwOnWriteLimitReached)

setMaxEmbeddedResources

@Field
public void setMaxEmbeddedResources(int maxEmbeddedResources)

setParseMode

@Field
public void setParseMode(String parseModeString)

setParseMode

public void setParseMode(HandlerConfig.PARSE_MODE parseMode)

call

public Integer call()
             throws Exception

Specified by:

call in interface Callable<Integer>

Throws:

Exception

getHandlerConfig

protected HandlerConfig getHandlerConfig()

enqueue

protected abstract void enqueue()
                         throws IOException,
TimeoutException,
InterruptedException

Throws:

IOException

TimeoutException

InterruptedException

tryToAdd

protected void tryToAdd(FetchEmitTuple p)
                 throws InterruptedException,
TimeoutException

Throws:

InterruptedException

TimeoutException

initialize

public void initialize(Map<String,Param> params)
                throws TikaConfigException

Specified by:

initialize in interface Initializable

Parameters:

params - params to use for initialization

Throws:

TikaConfigException

checkInitialization

public void checkInitialization(InitializableProblemHandler problemHandler)
                         throws TikaConfigException

Specified by:

checkInitialization in interface Initializable

Parameters:

problemHandler - if there is a problem and no custom initializableProblemHandler has been configured via Initializable parameters, this is called to respond.

Throws:

TikaConfigException

iterator

public Iterator<FetchEmitTuple> iterator()

Specified by:

iterator in interface Iterable<FetchEmitTuple>