Inference: engines and bindings

A map of names you choose to one engine each: an endpoint or a local binary with its settings. An engine knows nothing about documents. Two names may point at the same endpoint with different models. Secrets and paths live here and only here.

openai-embedding-engine posts to /v1/embeddings in the OpenAI shape and sends up to maxBatchSize images per request. Options: baseUrl, model, apiKey, timeoutMillis (120000), maxBatchSize (32), maxRetries (4), embeddingsPath, apiKeyHeaderName, apiKeyPrefix, requestParameters, imageInput, mediaInput (see Media).

A list of bindings. Each names an engine, says what it is fed, and what to ask:

A binding that names an engine not in engines, a task that cannot use its engine (embed on a chat engine), or an unknown key fails config load.

Parsers do not call engines. Every image, audio or video document Tika parses is offered to one dispatcher after its own parser has run, whatever container it came from and however deep. The dispatcher keeps the units of the whole document tree until the top-level parse ends, then sends each binding’s units to its engine in requests of up to maxBatchSize units: a docx with twelve pictures is one embeddings request, not twelve, when the engine takes twelve at once; if one image in a batch is rejected, the rest are retried one by one. Results land on the document a unit belongs to, as it appears in the output. In the recursive outputs (/rmeta, -J, pipes RMETA), where every document has its own metadata object, a picture in the body of a docx or an email puts its vector on the docx or the email, with an embedded locator naming the picture, whether the docx is the file itself or inside a zip; an attachment keeps its own. In the single-object outputs (/tika as JSON, -j, pipes CONCATENATE) there is one object, so everything lands on it: an attachment’s pages and pictures put their vectors on the top-level document too, each with an embedded locator whose id path and name say which part they came from. The chunks are the same either way; only the object they sit on differs. A failed request marks the top-level document with a warning and never fails the parse.

The flush is the end of the top-level walk, still inside parse(): every byte has been read and the last document closed, the engines are called under the parse’s timeout, and the units, held as files the dispatcher owns and bounded by each binding’s maxChunks and maxBytes, are deleted when it ends. The TEXT stage alone runs after parse() returns, over the finished metadata list (see Text). A chunker runs where its input is: a media chunker at the flush, on the file; the text chunker in the TEXT stage, on the text.

A PAGES binding is fed by the parsers that render pages. Today that is the PDF parser, and only when its own config releases pages:

pages.inference lists what a rendering parser releases to inference: TEXT, the default (text bindings follow), and PAGES. With PAGES, and a PAGES binding that runs for the request, the PDF renders every page once and hands the same image to the OCR engine, when text asks for it, and to the bindings: a scanned page costs one render whether one consumer or two see it. Each page is one unit. Its vector lands on the PDF’s own tk:chunks with a paginated locator naming the page, and maxChunks is the number of pages one document sends. Nothing is rendered for inference unless a PAGES binding runs, so the block can stay on a config whose bindings are on standby.

The render is the pages.render render: dpi, imageType, imageFormat, imageQuality and a maxWidth/maxHeight box shape it, and the defaults (300 dpi grayscale PNG) suit Tesseract, not an embedding endpoint. Set them to what the engine wants; when OCR and page vectors run together the one setting serves both. maxImagePixels and the minimum apply as for OCR; a page skipped for its size is not offered. See Pages.

Page renders emitted as embedded documents (pages.emit) are not images to an IMAGES binding: pages reach inference through PAGES only, so no page is embedded twice.

Per request, like every pages setting:

A TEXT binding embeds the extracted text of every document in the tree, the container and its attachments alike:

It runs after the parse, over the finished metadata list and before the metadata filters (so a filter shapes what it wrote and cannot starve it): every document’s tk:content is chunked, the chunks of the whole tree are sent in maxBatchSize requests in document order (a request may hold a container’s last chunks and an attachment’s first; the endpoint treats inputs independently and answers by index), and each chunk lands on the document its text came from, with a text locator and the binding as its producer. A rejected request is retried one document at a time. maxChunks caps the chunks of the tree, maxBytes the text of one document, and _mime-include/_mime-exclude narrow by document type. Use a MARKDOWN content handler for the chunker to see headings.

A document that released other inputs to inference is not embedded twice: a PDF parsed with pages.inference: ["PAGES"] records tk:inference-released: PAGES and the TEXT stage skips it; list ["TEXT", "PAGES"] to get both. This stage runs over the metadata list, so it covers -J, /rmeta and pipes, and /tika as JSON, whose list is the one object with the concatenated text; the plain-text /tika output carries no metadata and so no vectors. It is not a metadata filter: a per-request metadata-filters list does not touch it, and the per-request inference block below is the way to switch it off.

The 4.0 openai-embedding-filter and jina-embedding-filter keep working and embed one document per request; the binding batches the tree and shares the engine with the other bindings.

A MEDIA binding embeds audio and video. The engine sees one channel, so the binding says which: modality is visual (the picture, as short silent clips) or audio (the sound, as mono Opus). Two bindings on one engine index both channels of a video:

Tika cuts the file into segments with ffmpeg (on the PATH) on one grid per document, set in the parse context:

Those are the defaults: 25 s windows, each starting 20 s after the last, so any event of 5 s or less is whole in at least one segment, and no audio item reaches the 30 s cap the hosted omni engines enforce (see Jina omni); maxSegments caps a document, the rest of the file is not embedded. A block with overlap outside [0, seconds), or maxSegments of 0 or below -1, fails at config load and per request. The segmenter defines the unit and every binding on the document sees the same segments, so each segment’s chunks share the segment’s id: a chunk carries a temporal locator (start_ms, end_ms on the container’s presentation timeline), its producer (the binding), its modality (visual or audio) and a correlator, the segment’s id. Group by correlator to get a segment with both channels:

An audio file has no picture and a silent video has no sound: the binding for the missing channel skips the document; cover art in an audio file is not a picture. The binding’s maxChunks (480 unless set) also caps segments per document, and since a request may send its own media block, it is the ceiling an operator relies on: the smaller of the two wins. maxBytes bounds one cut segment (the cuts are small: 360p H.264 and 32 kbps Opus); the file’s own size is not gated. The engine must accept media, which the OpenAI shape has no form for, so the vendor’s form is declared: mediaInput: "object" sends {"audio": data-uri} and {"video": data-uri} items as Jina names them; without it a MEDIA binding on the engine fails at load. Each segment is one request input; a request holds up to maxBatchSize of them (the engine default is 32; the cuts are 100-500 KB each as base64, so set it lower for a small request body). A request the engine rejects is retried one segment at a time from the same cut files; nothing is cut twice. A file ffprobe cannot read is reported once as the binding’s warning and the next file runs.

ffmpeg and ffprobe are looked up on the PATH once, at config load: a MEDIA binding without them logs one warning and stands by, and each audio or video document is then skipped with a warning in its metadata rather than failed, the way OCR behaves without tesseract. The apache/tika:<version>-full image ships them; the minimal and grpc images do not.

ffmpeg runs under the parse’s time budget: each probe and cut asks for its own cap (60 s and 5 min) and gets the smaller of that and what remains of the request’s total timeout, with the wait counted as progress. When the budget runs out mid-file the task writes the segments it has embedded, stops, and reports the timeout once as the binding’s warning on the root document; every later MEDIA binding’s first probe fails fast the same way.

enabled: false runs nothing for this request; bindings names the ones that run, every enabled binding when absent. The recognizers have the same switch, {"text-recognizers": {"enabled": false}}; together they are "everything off for one request" (see Recipes). Names only, so it is wire-safe and works as a preset. A name not in the configured list fails the request. Engines cannot be defined or changed per request.

The 4.0 openai-image-embedding-parser is deprecated since 4.1.0 and removed in 4.2.0. It still works as a text-recognizers entry, with a WARN at startup, and embeds one image per request. The same endpoint as an engine plus a binding on IMAGES embeds a document’s images in requests of the engine’s batch size and puts the vectors where they belong; move the baseUrl, model and apiKey to the engine and drop the parser entry.