Pipes Iterators
A pipes iterator enumerates the documents to be processed. It emits one FetchEmitTuple per document; the pipeline then hands each tuple to a fork, which calls the bound fetcher (to get the bytes) and the parser, and routes the result to the bound emitter.
The Iterator Contract
A PipesIterator produces a stream of FetchEmitTuple records. Each tuple carries:
-
the fetch key — passed to the fetcher to retrieve the document bytes
-
the emit key — passed to the emitter to decide where to write results
-
an optional
idand arbitrary metadata fields
The iterator runs on its own thread; the pipeline reads tuples as fast as the forks can keep up.
Wiring an Iterator Into a Pipeline
The iterator lives under the singular top-level pipes-iterator key. The inner map key is the iterator’s component name.
Every iterator config takes two required flat fields alongside its own options: fetcherId and emitterId, the IDs of the fetcher and emitter bound to each emitted tuple. The exception is the JSON iterator, where each line names its own fetcher and emitter. The per-plugin pages list only the options specific to that iterator.
{
"fetchers": { "fsf": { "file-system-fetcher": { "basePath": "/data/in" } } },
"emitters": { "fse": { "file-system-emitter": { "basePath": "/data/out" } } },
"pipes-iterator": {
"file-system-pipes-iterator": {
"basePath": "/data/in",
"fetcherId": "fsf",
"emitterId": "fse"
}
}
}Only one iterator is active per pipeline. To process multiple sources in parallel, run multiple pipelines.
Available Iterators
| Plugin | Component name | Notes |
|---|---|---|
|
Recursively walks a directory tree. |
|
|
Lists S3 objects under a prefix. |
|
|
Lists GCS objects under a prefix. |
|
|
Lists blobs under a prefix. |
|
|
Queries a Solr collection (useful for re-parsing). |
|
|
Walks rows from a SELECT query. |
|
|
Consumes fetch-request messages from a topic. |
|
|
Reads work items from a CSV file. |
|
|
Reads work items from a JSON-lines file. |
For the full plugin / interface matrix, see Plugins.