Page generated: 2026-08-27
Source updated: 2026-08-22
Scoring inputs: ground truth 36c390d2, evaluate.py 5c52b890, rules 355f2e77
This document defines how model responses are scored against ground truth in the
Venice 39 data-extraction experiment (per-traveler tabular extraction). It is
not shown to models during collection — it records the answer-key logic
applied afterward. The implementation is evaluate.py.
The equivalence and ignore rules are not listed here. They live in
EQUIVALENCES.md, which rules.py reads at run time and
evaluate.py applies — so the rules are data with one home, and adding a row
there changes the scoring. This page describes the mechanism; that file holds
the content. Where an earlier version of this page reproduced the equivalence
tables, it now points at them, because two copies of a rule set is how they drift.
This is a different task and scoring scheme from the question-answering
experiment (venice-39-guided / -naive), which scores 22 fixed questions. The two
eval_guidelines pages do not cross-apply. The shared parts are the character-
accuracy method and the expand-both-sides equivalence philosophy.
Each model response is a list of traveler records (9 fields each). Scoring has two
stages:
1. Traveler identification — match model rows to ground-truth (GT) rows by
name, then compute precision / recall / F1. This is the headline metric.
2. Per-field accuracy — for each matched row, score each of the other
fields. Reported separately (per field), computed only on matched rows.
F1 measures who is on the page; per-field accuracy measures what the page says
about them. They are independent: a model can score F1 = 1.00 while its
place_of_stay accuracy is 0.80.
Each model row is greedily paired with at most one GT row by name similarity:
collapsed, punctuation stripped. A ? (illegible-glyph marker) is a
wildcard: it matches any single character at no cost. See "Reading what the
verifier could not" below. The quondam abbreviation variants
(q.m / q.m. / qm / q / q(uonda)m / q(uondam) / quondam) are first
collapsed to a single short canonical, so e.g. Leon q.m. Iacob Tedesco and
Leon qm Iacob Tedesco match fully. The short canonical is length-neutral, so
this equivalence doesn't shift unrelated borderline matches across the threshold.
difflib.SequenceMatcher (Ratcliff-Obershelp)
character ratio and (b) exact token-set overlap |A∩B| / max(|A|,|B|), taken
over every reading of the model name the §1 ignore rules allow — added titles,
articles and places of origin are stepped over, and the best reading wins.
Two measures, two questions. This similarity is the gate: it decides which
ground-truth traveler a row is about, and the value is discarded once the
threshold is applied, so a pair at 0.71 and one at 1.00 are identical to F1. It is
set permissively on purpose. How faithfully the name was read is a separate
question, reported separately as name CER — character edits over reference
characters, micro-averaged as sum(edits) / sum(reference length) so it composes
across pages and runs. Neither number substitutes for the other: a model can find
every traveler and misread all of them.
The two are not independent. Tightening the gate raises F1's selectivity and
flatters CER, since only clean readings remain in the pool; loosening it does the
reverse. So CER must always be read against the gate it was computed under.
Reading what the verifier could not. A ? in the ground truth is the verifier
saying "I cannot read this glyph" — one ? per unreadable character, the same
convention the collection prompt gave the models. It is scored as a wildcard,
in the gate and in name CER alike: a model that reads a letter where the verifier
could not may be reading the hand better than the verifier did, and the answer key
has no standing to call it wrong. Three GT names carry one — Francesco Kin?gsech,
Niccolò ?lelingen, Federico ?gorosd.
Three qualifications:
? as an uncertainty hedge on names
the register records in full (Pavel (?), Regina (Vedova?)). That is a model
declining to read, not a verifier declining to read, and it is compared literally.
A symmetric rule would let a model buy characters back by hedging.
? asserts a glyph is present. A model
that reads no glyph at all — Kingsech against Kin?gsech — is disagreeing
about how many characters are there rather than out-reading the verifier, and
still pays for the deletion.
?,
or read a letter and still landed close, the pair already cleared 0.70 and only
its CER changes. But a model that attempted the glyph and was otherwise imperfect
could fall below the gate on the strength of that one character, and the
traveler then counted as missed. Under the wildcard, 24 image-scorings recover a
match — Niccolò ?lelingen 11, Federico ?gorosd 9, Francesco Kin?gsech 4 —
raising F1 in 21 of 150 runs and lowering it in none. The effect is monotone by
construction: resolving ? can only raise a similarity, so no pairing is broken.
Those gains are concentrated in phase N, where models had no guidance telling
them to mark an unreadable glyph and so attempted a reading instead. Five of the
15 published (model, phase) cells move, by +0.0015 to +0.0158 F1.
Recovered matches also raise CER in 19 runs, because a row that was below the
gate was there for being badly read, and it now joins the pool it was excluded
from. Every CER increase is a run that gained a match; no existing pair scored
worse. This is the gate/CER coupling described above, and the reason the two
numbers must be read together.
This was compared literally until 2026-08-22. analyze_name_precision.py had
implemented the wildcard from the start, so the CER in the scoring-method page and
the CER in scores_summary.csv disagreed on these three names. Both now read the
semantics from evaluate.resolve_illegible.
bearing the same name**; no name repeats in any of the three GT files, so it
changed the pairing in none of the 450 image scorings (3 pages × 150 runs). This
is not a claim about the dates: the task is one row per unique traveler carrying
both dates when the source records both, so page 28's Mandolin Todros arrives
9 February and departs 13 February in a single row.
name is used only as the matching key — it is not itself scored as a field.
Precision / recall / F1 (per image):
Aggregation: F1 is micro-averaged — matched / n_model / n_gt totals are
pooled across all successful images and runs in a (model, phase) cell, then F1 is
computed from the pooled rates. Failed runs are excluded.
Each field returns a score in [0, 1], or is unscorable — excluded from both
numerator and denominator. The blank-GT rules are not uniform, so precisely:
blank there may mean the register does not record it, or that the verifier did
not transcribe it, and the two cannot be told apart.
title → 0.0, not unscorable. The register
records titles only for aristocracy, clergy, physicians and military, so a
blank title is a deliberate omission and a supplied one is wrong (PI's ruling).
| Field(s) | Method |
|---|---|
with_spouse, with_siblings | Exact Y/N match → 1.0 / 0.0 |
children_count | Exact integer match → 1.0 / 0.0 |
title | Character accuracy after base normalization. No courtesy-form stripping: a courtesy form the register never uses as a title (Signor, Mister) is a wrong answer, not something to normalize away. |
arrival_date, departure_date | Character accuracy after base normalization + day leading-zero strip (February 09 1769 == February 9 1769; GT is non-padded). Genuinely different or differently-formatted dates (day-first, month abbreviations, raw Italian like 9bre) still differ under char_accuracy — those are model deviations from the requested Month D YYYY, not accepted equivalents. |
place_of_stay, place_of_origin | Character accuracy after place normalization (§5) |
Character accuracy = 1 − levenshtein(gt, model) / len(gt), clamped to
[0, 1]. A one-character slip in a 16-character string scores ≈ 0.94, not 0. Both
GT and model are normalized to a canonical form before the edit-distance
comparison, so a correctly-written abbreviation matches the expanded GT exactly.
Applied to every text field (title, dates, and — as the first layer — places):
1. Straighten the typographic apostrophe: \u2019 → '. A rendering variant of
the same character, so all\u2019Ospizio and all'Ospizio compare equal.
Accents are not touched — Abb\u00e9 is a different word from Abbe, not a
different rendering of it.
2. Lowercase; collapse whitespace.
3. Strip trailing punctuation ,.;:][ from each token. Apostrophes and backticks
are not stripped.
4. Strip a trailing ? from GT tokens (illegible marker) before the
equivalence lookup, so a marked token still finds its dictionary entry. This is
a lookup aid, not the name wildcard of §2 — it applies to the non-name text
fields and removes the marker rather than matching through it.
5. Apply the single-token equivalences for that field from
EQUIVALENCES.md §2, token by token (expand-both-sides).
6. Apply the multi-token equivalences from §2 as a sequence replacement.
7. Apply the ignore rules for that field from §1, honouring their position and
side. Not applied to name — see §2 above.
Steps 1-4 match venice-39-guided's token preprocessing so the two projects' text
scores stay comparable; steps 5-7 are this project's rule file.
place_of_stay, place_of_origin)
Place fields get base normalization plus Venetian place-name handling, built
from a corpus scan of GT + model outputs:
in (the document reads "in casa di…"; GT records
"Casa di…", so the in is a connective, not content).
Sant'Angelo → san angelo).
Genuine misreadings of the handwriting (e.g. Feltrina→Pestiora,
Zobenigo→Colenigo) are left intact and scored by character accuracy — the
normalization never rescues them.
Held in EQUIVALENCES.md, not here. That file is the input to
the scorer: rules.py parses it at import and evaluate.py applies it, so a row
added there changes how runs are scored.
leading /
trailing / anywhere) and a side (model / both).
an abbreviation expansion; a multi-token row is replaced as a sequence.
rule should cover.
because they are order-dependent rather than lists of forms.
Only rows marked settled are applied. OPEN rows are parsed and checked but
change no score, so a rule can be drafted and previewed before it is switched on.
Governing principles, unchanged: expand-both-sides, so GT and model normalize
to the same canonical and a correctly-written abbreviation earns full credit while
an unrecognized misreading still falls to character accuracy; and every form must
have a witness in the transcription or the ground truth, since this evaluator is
scoped to this corpus rather than built for reuse.
A model-side ignore is only valid when the token never appears in the GT for
that field — otherwise it splits pairs that are identical. Verified across all 220
GT values in the four text fields: no current rule does this.
gt/ (gt_page_NN.csv).per-page results / compare pages.