Regression evaluation with tika-app + tika-eval

The flow is three stages:

Both extracts include SHA-256 digests so tika-eval can align embedded documents across runs even when filenames differ.

This produces two distribution zips:

Running the bare tika-app-.jar or tika-eval-app-.jar (without the unzipped lib/ siblings) will fail at runtime with ClassNotFoundException — the jars are intentionally not fat jars.

Put the input files somewhere on disk. For this example we’ll use <corpus>/. Tika’s pipes iterator walks the tree recursively; files at any depth are processed.

When you pass -i <input> and -o <output> on the command line, tika-app auto-builds the fetcher / emitter / pipes-iterator for the file system. The config only needs the things you want to OVERRIDE from defaults — typically just the digester and (for this kind of A/B test) the field under test.

The two configs should differ in exactly one thing — the field under test. Everything else (digester, parsers, fork count) should match.

Flags:

Each run forks JVMs and produces one .json per input file under the output directory. Run them sequentially or in parallel — they’re independent.

Progress is logged to stderr. On a typical workstation, extraction throughput is in the hundreds-of-files-per-minute range, depending on parser cost and fork count.

The Compare subcommand keyword is optional — the CLI infers it from the -a / -b flags. The -r flag both runs the Report stage and creates a .tgz archive of the resulting reports directory (<reportsDir>.tgz) for easy archiving.

Options:

The reports directory contains subdirectories:

Open the .xlsx files directly, or query the H2 database for custom counts and joins (see Querying the H2 database directly).

Compare the SPI default detector chain (A) against an explicit [BOM, HTML, Mojibuster, JunkFilter] chain (B) over a 29 000-file Common Crawl HTML sample at <corpus>/cc-html-sample.

For a 29 K-file HTML sample on a typical workstation (8 forks, i9-ish CPU): each extraction completes in roughly 45-60 seconds; the Compare step finishes in 2-5 minutes depending on extract size. . Open reports/mimes/mime_diffs_A_to_B.xlsx to see the headline MIME-detection differences. The encoding-detector chain change shows up in reports/charset/ — charset_coverage.xlsx counts the A→B flips, and the distribution reports break down tk:detected-encoding per run. It also shows up in mimes/mime_diffs_A_to_B_details.xlsx, because the recorded MIME string carries the charset= parameter.

Keep the writeup next to the reports directory it describes, so the numbers and the narrative travel together.

tika-eval stores everything in an H2 database — the -d <name> passed to Compare produces <name>.mv.db. The generated reports surface only pre-canned views; for custom counts and joins, connect to the H2 file and run SQL.

The database is created with no username and no password (H2Util opens it with DriverManager.getConnection(url) and no credentials), so -user sa fails with "Wrong user name or password" — use empty credentials. Append ;IFEXISTS=TRUE (so a typo opens nothing rather than creating a new empty db) and ;ACCESS_MODE_DATA=r for a read-only open. H2 is single-writer: query only after the Compare/Profile/Report run has released the file lock. Use the h2-*.jar shipped with tika-eval-app.

Pass SQL to -sql as a single line (a multi-line argument leaves the Shell waiting on stdin).

Key tables: profiles_a/profiles_b (one row per extracted file: file_name, mime_id, length, …), contents_a/contents_b (text profile: oov, languageness, num_tokens, lang_id_1, num_replacement (U+FFFD count), num_non_ascii, …), content_comparisons (dice_coefficient, overlap), mimes, containers. A and B are paired by id — the same row id is the same file in both runs (this is how the built-in reports join: join profiles_b pb on pa.id = pb.id). Always join on id.