← Analysis Reports

Page generated: 2026-08-27
Data (model runs) generated: 2026-07-23
Scoring inputs: ground truth 36c390d2, evaluate.py 5c52b890, rules 355f2e77

The evaluation method

What it measures, why it is built this way, and the same model runs scored four ways so the differences can be inspected rather than taken on trust. Every figure on this page is computed by build_eval_method.py from the run checkpoints and the ground-truth CSVs.

1. What we are asking of the sources

The Venice 39 material is a weekly register of foreign visitors: each page lists arrivals in the left column and departures in the right, one line per named traveler, with title, dates, lodging and place of origin. Two different historical questions are being put to it, and this experiment is the second half of a two-part study that separates them.

1. How many people are recorded here? A count of unique named travelers per page. This is the question the question-answering half of the study asks directly, and it is answerable without reading a single name correctly.
2. Who were they? The names as written, transcribed faithfully enough to identify individuals, link them across records and search for them.

These are different capabilities and a model can have one without the other. An evaluation that returns a single number cannot tell you which you have. That is the whole design constraint.

2. What the method does

Step 1 — count. Count the distinct named travelers the model reports on a page and compare with the ground-truth count for that page. Duplicated rows collapse to one traveler, because the question is how many people are recorded, not how many rows were emitted. No name matching is involved; a model that invents every name still counts correctly if it reports the right number of people.

Step 2 — align. Pair each model row with the ground-truth traveler it is closest to, order-blind, using character similarity on the name. Before comparing, added titles, articles and places of origin are removed from the model's name field, and the best available reading wins — so Il Barone Bassa is compared as Bassa and Israel Baysan di Verona as Israel Baysan. The unreduced name is always among the candidates, so a reduction can only reveal a name that was buried, never lower a score.

Step 3 — measure transcription. Over the aligned pairs, report character error rate on the name itself: CER = levenshtein(gt, model) / len(gt), micro-averaged, plus the share of names reproduced exactly. The alignment used for this measure has no similarity floor, so a badly read name stays in the pool instead of being dropped for being badly read.

Step 4 — characterise what is left over. Rows with no ground-truth counterpart are classified rather than lumped together: a traveler listed twice, two travelers merged into one row, a fragment of a name, a real traveler misread past recognition, or an invention with no counterpart on the page.

Detection precision and recall are still reported, at a 0.70 similarity threshold, but they are one output among several rather than the headline.

3. The same runs, scored four ways

F1 under each, pooled across the three pages and 10 runs:

Reading across a row shows how much of a model's score is an artefact of the instrument. What no column shows is whether the model could read the names — all four are counts of rows that cleared a bar.

4. What the adopted method reports

Counting and transcription, per page, so the counts are whole numbers against a whole-number ground truth:

Page 28 — 18 unique named travelers

Page 36 — 37 unique named travelers

Page 39 — 10 unique named travelers

Plus the composition of the surplus — rows left over once every ground-truth traveler has its closest candidate:

5. Why this rather than the alternatives

A single number answers neither question. F1 counts rows that cleared a similarity bar. GPT in the naive condition scores 0.471, which reads as a failure to find the travelers. It is not: on page 39 it reports 9–10 unique named travelers against a ground truth of 10, the same count Gemini 3.1 reports, and then misreads them at 0.57 character error against Gemini 3.1's 0.12. Equally good at the first question, very unequal at the second. So detection and transcription are reported as two numbers, F1 and name CER.

The gate and the transcription measure do different jobs, so they are different measures. The gate decides which traveler a row is about; it is thresholded and then discarded, so a pair at 0.71 and one at 1.00 are identical to F1. Transcription asks how faithfully the name was read, and wants a graded, interpretable, composable number. Ratcliff-Obershelp serves the first and CER the second. Trying to make one measure do both is what produced the original design, where the name's reading accuracy appeared in no published figure at all.

CER is the right instrument for transcription and the wrong one for identity. Its reference-length denominator is a virtue when measuring error per reference character and a defect when deciding identity: appended content is penalised as heavily as wrong content, so a model that writes the traveler's origin or title into the name field is recorded as having found nobody. The cer gate column shows the cost. On this corpus the choice is also less consequential than it looks — Ratcliff-Obershelp, CER and edit distance normalized by the longer string agree to two decimals on equal-length substitutions, which is what most misreadings here are; they disagree on 77 occurrences out of 9,586. Ratcliff was kept because it is already in place and nothing in the evidence justified moving the gate.

The name reductions are what the adopted column adds over original. Titles, articles and appended places of origin are removed from the model's name before comparing, and the best reading wins — so a reduction can only reveal a name that was buried, never lower a score. That matters because the ground truth itself contains Ernesto di Bargh and De Flamarin, where the connective is part of the name, and Salamon Cormos, whose recorded origin is Cormos. An unconditional strip would damage all three.

Order-preserving alignment costs more than it returns. Requiring the model's row order to follow the ground truth's forfeits matches when a model reads across the two columns instead of finishing the left one first, or groups a family together. Both are legitimate readings of a page the prompt never asked to be traversed in a particular order, and the cost falls almost entirely on one model, which would read as an extraction difference rather than a traversal choice.

The transcription measure is not conditioned on the gate. Reporting CER only over rows that cleared the threshold would measure the names a model read well, by construction. The gate is set permissively so that marginal readings stay in the pool; the two numbers must therefore be read together, since tightening the gate would raise F1's selectivity and flatter CER at the same time. GPT's naive character error is 0.44 measured this way.

6. What this method does not do

• The alignment used for the transcription measure is bookkeeping, not a claim of identity. Pairing a model's row with a ground-truth traveler at 0.30 similarity does not assert they are the same person; it places the row where its character error can be counted. Detection is answered by the counts and the thresholded precision, never by the size of the assignment.
• CER is a corpus statistic. Over hundreds of names per cell it is stable; on a single short name it is very noisy, which is why no per-name threshold should be read as a confident judgment about that name.
• It measures names. The other eight fields are scored separately and their normalization rules are still under review.
• Nothing here evaluates whether a traveler was correctly placed in the arrival or the departure column.

Generated by build_eval_method.py. Scoring rules: eval_guidelines.html. Worked cases of the name measure: scoring_name_match.html.