Data Extraction [Experiment Part II]
3 archival document images · 5 models · 3 prompt conditions · 10 runs for each condition
Structured data extraction against a 9-field schema (name, titles, dates, place of stay, family, origin) on three pages chosen to span content density: 18 travelers (page 28, an Ebrei page), 37 travelers (page 36, the densest), and 10 travelers (page 39). Each (model, page) combination is run under three conditions — naive (prompt only; N), guided (with expert-authored domain knowledge; A), and guided + transcription (guidance plus a pre-computed transcription; B) — for 10 runs each.
Setup & Reference
-
Side-by-side viewer of the three archival document images used in the experiment, with the correct transcription. Note: this is not the text the models received. Press relevant button to switch between pages 28, 36, and 39.
-
The three transcriptions as they stood when the phase-B responses were collected, verbatim, with an account of how they differ from the correct transcription.
-
Scoring rules: name matching, character accuracy for text fields, exact-match for booleans and integer counts, and the full abbreviation equivalence table (e.g.
q.m. / qm / q(uonda)m → quondam). -
The domain-knowledge block added to the system prompt for guided-condition runs — including explanatinons of Venetian archival conventions, the M.V. calendar, named-vs-unnamed traveler rules, family vocabulary, and abbreviations.
-
The authoritative per-traveler structured output for for all three pages in the 9-field schema against which model outputs are evaluated.
Method & Scoring how the evaluation works and why
-
What the method measures and how it scores model outputs in four different ways (count, align, measure transcription, and characterize what is left over), in explanation of how the method addresses the two separate questions: how many travelers were found, and how faithfully their names were read.
-
Comparison of the different methods explored (Ratcliff-Obershelp, CER, normalized edit distance, token F1 and Jaro-Winkler) to provide evidence for the choice of using Ratcliff to decide identity and CER to measure transcription.
-
What the name score is, and where a model puts the right value in the wrong field (a title or place of origin written into the name, which is correct in its own column and breaks the match from the other).
-
What normalization and the equivalence rules are actually worth, per page and model in each condition.
-
Raw results from each model and condition used to define both normalization rules and equivalence criteria.
Overall Results
-
F1 by model and condition, per-field accuracy, per-image F1 variation across runs, and notes on collection caveats. Includes how F1 is calculated for this experiment.
-
Same model, same three page images: direct-count (Experiment Part 1 Q13 naive + guided) vs structured emission (data extraction N / A / B). Isolates the task-framing effect on counting and explains why the two ground truths differ (mentions vs unique rows).
-
Underlying per-image, per-run precision, recall, F1, name CER, and per-field accuracy for every (model, condition, run, page) combination.
Per-page Analyses click a button to view that page
-
Compare (interactive)Toggle the best run for any (model, condition) on/off and compare per-field, row-by-row against ground truth. Cells colored green (match) / red (mismatch).
-
ResultsGround truth plus per-model traveler list for each condition.
-
Per-field ResultsSide-by-side per-field comparison (name, dates, place of stay, family, origin) of every model and phase against ground truth.
Page 28 = AVS IS 758-781 1766 02 (Ebrei page, 18 travelers) · Page 36 = AVS IS 758-781 1770 05 28 p1 (37 travelers, densest) · Page 39 = AVS IS 758-781 1770 08 (10 travelers)