← Question Answering

Evaluation Guidelines

This document defines the rules for scoring model responses against ground truth. It is not shown to models during testing. Rules here encode answer-key logic and acceptable equivalences.

1. Pre-processing (applied to both GT and model response before comparison)

These transformations are applied before any comparison is made:

1. Trim whitespace: strip leading and trailing whitespace from the full response and from each individual token.
2. Strip trailing punctuation from words: remove trailing commas, periods, semicolons, colons, and closing brackets from individual words (e.g., moglie, → moglie). Do not strip punctuation from within words (e.g., 1767/8 stays as 1767/8).
3. Strip ? from GT values: the ? character in ground truth indicates annotator uncertainty about a reading, not a character in the document. Remove trailing ? from GT tokens before comparison (e.g., GT Caltran? → Caltran). Model responses are compared as transcribed.
4. Case-insensitive comparison: all text comparisons are case-insensitive (e.g., fratelli = Fratelli = FRATELLI).
5. Apply equivalence dictionary: before comparison, normalize tokens using the active equivalences in Section 5.1. Abbreviated forms are expanded to their canonical full forms on both GT and model response sides before any comparison.

2. Illegible Responses

The response token illegible is only valid for boolean questions (Q1, Q3, Q4, Q6, Q21). It means the model attempted to read the relevant part of the document and could not determine whether the answer is Y or N.

This is distinct from not-applicable cases (blank responses), which indicate the question does not apply given a prior answer (e.g., Q2 blank when Q1=N).

• If GT has a definite value and model returns illegible: score as illegibility claim — recorded separately.
• If GT has a definite value and model returns a wrong value: score as incorrect.
• If model returns blank where a value is expected: treat as incorrect (blank indicates a missing or skipped response).

For all other question types, the model is expected to provide its best attempt. Illegibility in those questions manifests naturally:

• Integer questions: a partially illegible document produces an undercount relative to GT.
• String/text questions: character-level errors appear in the comparison against GT.

Two scores are reported for boolean questions:

1. Conditional accuracy: correct / (correct + incorrect), excluding illegibility claims. Measures how accurate the model is when it commits to an answer.
2. Combined score: correct / (correct + incorrect + illegibility claims). Treats illegibility as a failure to answer. Measures overall performance including coverage.

Both are reported per question and per model. The illegibility claim rate (illegibility claims / total questions) is reported as a third metric.

3. Scoring by Question Type

3.1 Boolean questions (Q1, Q3, Q4, Q6, Q21)

Permitted responses: Y, N, illegible, or blank (only when the question is conditional and the prerequisite is not met — Q21 when Q19=0)

• Exact match after pre-processing: correct.
• Wrong value: incorrect.
• illegible response: illegibility claim — reported separately; excluded from conditional accuracy; counted as failure in combined score (see Section 2).
• Blank when GT is also blank (Q21 when Q19=0): excluded from the base — the question was not asked, so it neither earns nor costs anything.
• Non-blank when GT is blank (Q21 when Q19=0): incorrect.

3.2 Integer questions (Q2, Q7, Q8, Q9, Q11, Q13, Q14, Q15, Q16, Q18, Q19)

Permitted responses: non-negative integer, or blank (Q2 only, when Q1=N)

• Exact integer match: correct.
• Any other integer: incorrect (undercounts due to illegibility are scored as incorrect; the degree of error is visible in the difference from GT).
• Blank when GT is also blank (Q2 when Q1=N): excluded from the base, as for Q21 above.
• A year given when GT has no date (Q2 when Q1=N): incorrect. Changed 2026-08-16 — this previously returned the same excluded-from-base result as a blank, which made Q2 unscorable on the 11 no-date pages and left a fabricated year costing nothing. Asserting a year for a page that has none is a wrong answer, not a non-answer.

3.3 Paired integer questions (Q17, Q20)

Format: N | N (left column count | right column count)

• Both values must match GT exactly: correct.
• Either value wrong: incorrect.

3.4 String question (Q5 — signature location)

Permitted responses: two-word string, or blank (when Q4=N)

• Evaluated character by character against GT after pre-processing. Score = character accuracy (matching characters / GT character count, using edit distance).
• Blank when GT is also blank (Q4=N): correct.
• Blank when GT has a value: incorrect.

3.5 Text array questions (Q10, Q12, Q22)

Format: pipe-separated list of entries, one per document instance, in document order (top to bottom).

Pre-processing for these fields:

• Split on | to produce a list of entries.
• For Q10 and Q12: take only the first two words of each entry (both GT and model response). The question asks for exactly two words.
• Apply pre-processing rules from Section 1 to each word.
• Apply dictionary equivalences (Section 5) to each word.

Entry-level scoring: Each matched pair of (GT entry, model entry) is scored character by character. Entry score = character accuracy (matching characters / GT character count, using edit distance).

Question-level score — F1 (all three questions): The question-level score is an F1 combining recall and precision, so that both missed entries and hallucinated entries reduce the score.

• Recall = sum of character accuracies for GT entries against their matches / N_GT. Unmatched GT entries (model omitted them) score 0.
• Precision = sum of character accuracies for model entries against their matches / N_model. Unmatched model entries (hallucinations with no GT counterpart) score 0.
• Score = F1 = 2 × Precision × Recall / (Precision + Recall)
• If model returns blank (and GT is non-empty): Precision = Recall = 0, score = 0.

Q10 and Q12 — order-independent optimal matching: Entries may appear in any order in the model response. The optimal one-to-one matching between GT entries and model entries is found using the Hungarian algorithm, maximising total character accuracy. Take only the first two words of each entry before character-level comparison.

Q22 — order-dependent sequence alignment: Bracket annotation entries appear in document order (top to bottom) and the model is expected to follow that order. Matching uses global sequence alignment (Needleman-Wunsch at the entry level): the similarity score between a GT entry and a model entry is their character accuracy; the gap penalty is 0. The alignment that maximises total character accuracy while respecting sequence order is used. A missed annotation and a hallucinated annotation each introduce a gap in the alignment, scoring 0 for that position, without displacing the scoring of surrounding entries.

4. Scoring Summary

5. Equivalence Dictionary

Used in pre-processing step 5 for text array questions (Q10, Q12, Q22). Both GT tokens and model response tokens are normalized to the canonical form before comparison. All comparisons are already case-insensitive (see Section 1.4), so case variants are not listed here.

Governing principle: Transcriptions must reflect what is written on the page. Modernized standard spellings or expanded full forms of clipped place names are not accepted equivalents. However, abbreviations are treated differently: a model may return either the abbreviated form as written or the correctly expanded full form, and both are accepted. Before scoring, both GT and model response tokens are normalized to the expanded canonical form. This means character-level scoring (Q10, Q12, Q22) always compares expanded forms on both sides.

Abbreviations — scoring rule: For any abbreviated token (including superscript-marked abbreviations indicating gender and number, e.g. singular masculine, singular feminine, plural masculine, plural feminine), the model response is correct if it matches the abbreviated form as written or the correctly expanded full form with matching gender and number. A wrong gender or number (e.g. returning the masculine form when the page has the feminine) is scored as incorrect. Normalization expands both sides to the canonical full form before character comparison.

Populated by systematic scan of all text_after_con, text_after_e, and bracket_annotation_text values across the 39-document corpus. The equivalence dictionary is expected to grow iteratively: after each evaluation run, entries with unexpectedly low character accuracy should be reviewed. Cases where a model correctly returned an abbreviated form that appears in expanded form in the GT (or vice versa) — for example, d.co / domestico, fig.lo / figliuolo — should be added to Section 5.1 as confirmed pairs.

5.1 Active equivalences

Both sides are normalized to the canonical (expanded) form before comparison.

Abbreviations

figliuolo family — gender/number equivalences

GT always uses the archaic spelling with u (figliuolo, figliuola, etc.). Models may return the modern Italian spelling (no u), a period variant (no u, stem figliola), or a period abbreviation. All forms within the same gender/number class are accepted; cross-class forms are not.

Character corruptions observed in model responses — figlo, figolo, figlivoli, figle, figla — are not accepted equivalences and are scored by character accuracy.

domestico family — gender/number equivalences

GT always uses the full form. Models may return a period abbreviation. Full forms (domestico, domestici, domestica, domestiche) require no entry. Abbreviated forms are listed below.

Truncated forms domesti and domest (with or without trailing period) are not accepted equivalences. These occur when the model fails to recognize that the word is hyphenated across a line break (domest- / ico) and reads only the first half. They are ambiguous as to gender/number and are scored by character accuracy.

5.2 Forms documented but NOT accepted as equivalences

The following pairs were identified during corpus scan. They are recorded here to explain scoring decisions, not to define accepted equivalences.

Clipped place names

Clipped place names are transcriptions of what is written. The clipped form is the correct answer; the full modern form is not accepted.

Spelling variants

Each form below appears on specific pages. They are distinct transcriptions of what is written, not interchangeable. A model should write what appears on the page being analyzed.

5.3 Uncertain Readings (GT ? words)

These words carry ? in the GT, indicating annotator uncertainty. The ? is stripped in pre-processing (Section 1.3). The “probable referent” column shows the likely modern or standard form; per the governing principle, these are not accepted as equivalents.

6. Note on the ANSWER RULES across the Naive and Guided conditions

The ANSWER RULES differ between the Naive and Guided conditions in how models should handle unreadable characters in "transcription" questions (Q2 year, and Q10/Q12/Q22 word lists). The guided prompt explicitly enables a ?-for-illegible convention; the naive prompt instructs the model to "give its best attempt." For Q10/Q12/Q22, character-accuracy scoring treats ? exactly like any other character. Only GPT used ? on Q2 (16 of 390 guided responses). The partial-credit contributes approximately 1.9 percentage points to GPT's guided Q2 score (61.0%) out of the +9.6-point improvement over its naive Q2 score (51.4%).

7. What This Document Does Not Cover

• The model prompt and question list: see questions.md
• Ground truth data: see ground_truth.md
• Per-run results and consistency analysis: see analysis scripts