← Analysis Reports

Page generated: 2026-08-27
Data (model runs) generated: 2026-07-23
Scoring inputs: ground truth 36c390d2, evaluate.py 5c52b890, rules 355f2e77

Name matching and field conflation

How a traveler name is scored, why two different similarity measures give different answers, and where models put the right value in the wrong field. Every figure on this page is computed from the run checkpoints and the ground-truth CSVs by build_scoring_doc.py; none of it is typed by hand.

1. What the name score is, and what it is not

The name is the matching key, not a scored field. It never appears in SCORED_FIELDS, so its accuracy contributes to no published figure directly. It enters the results in exactly one way:

1. A similarity is computed for each candidate (model row, GT row) pair.
2. Pairs below 0.7 are rejected; the rest are eligible to match.
3. Rows are paired, and then the similarity value is discarded.
4. Matched rows are counted: precision = matched / rows produced, recall = matched / GT travelers, F1 = 2PR/(P+R).

So a pair at 0.71 and a pair at 1.00 are indistinguishable in every published number, and a pair at 0.69 is counted twice against the model — once as a false positive and once as a missed traveler.

2. The two similarity measures

Both run on the reduced readings of the model name, not the raw string. evaluate.name_variants generates every reading the EQUIVALENCES.md §1 rules allow — the untouched name, the name with a leading article and title removed, the name with an appended place of origin removed — and the best-scoring one wins. The unreduced form is always among the candidates, so a reduction can reveal a name that was buried but never lower a score. The §1 removed column in the tables below reports which reduction won.

Gate (evaluate.name_match, applying name_similarity to each reading) — the larger of two candidates:

• difflib.SequenceMatcher.ratio(), which is Ratcliff-Obershelp: 2·M / T, where T is the combined length of both strings and M counts characters matched by taking the longest contiguous block and recursing either side of it. It is not an edit distance: characters stranded across an anchor block cannot be matched.
• token-set overlap, |A∩B| / max(|A|,|B|), requiring whole identical tokens.

CER-based (analyze_name_precision.pair_sim) — 1 − CER, where CER = levenshtein(gt, model) / len(gt), a true minimum edit distance normalized by the reference length. This is the measure already used for the eight scored fields, via evaluate.char_accuracy.

They agree on plain substitutions and diverge on length: the combined-length denominator absorbs appended text, the reference-length denominator does not.

3. Worked cases

gate, no §1 is the similarity on the untouched string — what this page reported before the §1 rules were wired into the scorer. gate is what evaluate.match_rows actually applies. The difference between the two columns is the rules doing their job. title score and origin score are the scorer's own per-field results for the same row.

Conflation no longer costs a match by itself: wherever §1 removed is non-empty, the added title, article or origin was taken off before the comparison, and what is left is the reading of the name. Two things the reduction does not fix are visible below — a short name misread is still rejected on its own merits, and a higher gate score lets some genuinely wrong readings clear 0.7 that previously fell short.

Conflation removed, and the row is still lost — to the reading, not the title

Page 39 — GT traveler Baissen. GPT reports title correctly as Barone and also prefixes it to the name. §1 takes the title back off, so the comparison is Bassa against Baissen: three wrong characters in a seven-character name. The row is rejected on the reading alone. This case used to be presented as conflation breaking the match; the title was never the reason it would fail today.

Place of origin conflated into the name, and now absorbed

Page 28 — GT traveler Israel Bassan. GPT fills place_of_origin correctly and also appends it to the name. The raw string falls below threshold; with the appended origin removed the same row matches. Nothing about the model's reading changed — only whether the scorer could see it.

GT is the bare surname, so the raw string is mostly title

Page 36 — GT traveler Tornielli. GT records only the surname, so before §1 almost every row here was rejected for carrying a title the model also filed correctly. With the title removed all but the worst misreading now clear the threshold — including Barone Donizelli, where the surname is simply wrong. That is the cost side of the reduction: it raises every score, not only the deserving ones.

Three travelers in one row

Page 36 — GT traveler Francesco Mazuchelli. The document writes the three brothers on one line and GT expands them to three rows. A model that transcribes the line faithfully produces one row. §1 does not help here, and it hurts once: stripping Conte lifts Conte Francesco Alberti over the threshold, matching a traveler it is not.

Control: a long name absorbs the same number of errors

Page 39 — GT traveler Giovanni Battista Maffoni. Nothing is conflated here, so §1 removed is empty throughout and the two gate columns agree. Included to show what the same character error rate does to a long reference.

Several travelers in one row

The register writes family groups on a single line (f.lli = fratelli, brothers) and the ground truth expands them to one row per named traveler, which is what the prompt asks for. A model that transcribes the line as written produces one row where GT has two or three, so it is charged one false positive plus a missed traveler for each name it did in fact read.

The definite article halves a short title score

Not conflation, but the same arithmetic. title is scored by character accuracy against a short reference, so writing the article the document uses costs half the score:

4. Reference length decides how much error is survivable

CER divides by the length of the GT name, so the same number of wrong characters produces opposite verdicts depending on how long the name is:

The verdict column is the gate's, not CER's. An earlier version of this table compared 1 − CER against the gate's 0.70 threshold, which is the wrong measure for that cutoff — the gate is Ratcliff-Obershelp. Both are shown so the divergence is visible.

The shortest names on a page are the easiest to read and the hardest to match.

5. Rows returned, before any scoring

How many per-traveler rows each model actually returned for a page, as the lowest–highest range across its 10 runs, with the median and its distance from the ground-truth count beneath. Matching is ignored here: a model returning 20 rows for an 18-row page is reported as 20 even if only 10 of those rows correspond to a real traveler. This separates over- and under-generation from reading accuracy — a model can return exactly the right number of rows and still have read none of them correctly.

rows returned is the raw count of per-traveler rows the model produced, before any scoring — range across the 10 runs, with the median and its distance from the ground-truth count. matched is the mean number of those rows that paired with a real GT traveler, and unmatched the remainder. name CER is the character error rate of the traveler names on the matched rows, micro-averaged as total edits over total reference length so it composes across pages. Detection and transcription are separate questions, reported separately: a model can return the right number of rows, match all of them, and still have misread the names.

6. How often the right value lands in the wrong field

Counted over all rows a model produced, matched or not, and over the scorer's own pairing (evaluate.match_rows) rather than a separate one. also correct in its own field = the model filled that field correctly as well, so it duplicated the value rather than misfiling it. row lost = the name match failed, costing a false positive and a missed traveler.

Read row lost as a residue, not as the cost of conflation: §1 already removed the added title or origin before the match was attempted, so these are rows that failed on the reading of the name even after the reduction. The column stays because it is the honest denominator — a matched-only count cannot see the rows that never made it.

The title in name and origin in name counts are still worth reading on their own: they measure how often a model puts a value somewhere the schema did not ask for, which is a question about instruction-following rather than about transcription, and the §1 rules deliberately make it cost nothing.

Generated by build_scoring_doc.py from the phase N/A/B checkpoints. See eval_guidelines.html for the full scoring rules.