← Analysis Reports

Page generated: 2026-08-27
Data (model runs) generated: 2026-07-23
Scoring inputs: ground truth 36c390d2, evaluate.py 5c52b890, rules 355f2e77

Venice 39 — Data Extraction Experiment — Summary

3-image dataset, 5 models × 3 phases × 10 runs.

• N (Naive): image only, minimal prompt (no expert guidance, no transcription).
• A (Guided): image + expert guidance about Venetian archival conventions.
• B (Guided + Transcription): image + guidance + full transcription.

How scoring works

Scoring is defined in eval_guidelines.html and implemented in evaluate.py; the equivalence and ignore rules it applies are in EQUIVALENCES.md, read at run time. This page reports results and does not restate the rules — a second copy is how the description and the scoring come apart.

In brief: model rows are paired with ground-truth travelers by name similarity, and F1 counts those pairings — it answers which travelers were found. Name CER is reported beside it and answers how faithfully those names were read: character edits over reference characters, micro-averaged. The two are separate questions and neither substitutes for the other. Per-field accuracy below is computed only on matched rows.

1. F1 by model × phase

1.1 Rows returned vs travelers matched

rows returned is the raw count of per-traveler rows the model produced, before any scoring — range across the 10 runs, with the median and its distance from the ground-truth count. matched is the mean number of those rows that paired with a real GT traveler, and unmatched the remainder. name CER is the character error rate of the traveler names on the matched rows, micro-averaged as total edits over total reference length so it composes across pages. Detection and transcription are separate questions, reported separately: a model can return the right number of rows, match all of them, and still have misread the names.

1.2 What the unmatched rows were

A row that matched no ground-truth traveler is one of: double-listed — the traveler is real but another row already claimed it; under-segmented — two or more travelers in one row, so the others are counted as missed rather than expanded and credited; fragment — part of a name; bad read — a real traveler misread past the matching threshold, recognizable at 0.55 similarity or better; fabricated — nothing on the page it corresponds to. Diagnostic thresholds only: these change no score, and the columns sum to the unmatched figure above.

2. Per-field accuracy by model × phase

Computed only on rows that matched a GT traveler (so a model that misses half the travelers does not get a free pass on the half it never produced — those are captured by F1 in section 1).

Scoring per field on a matched row:

• Text fields (title, arrival_date, departure_date, place_of_stay, place_of_origin) — character accuracy = 1 − levenshtein(gt, model) / len(gt) after normalization. A 1-character misspelling on a 16-char string scores ~0.94 rather than 0. Normalization lowercases, collapses whitespace, strips trailing punctuation (and a trailing ? on GT illegible markers), and applies an equivalence dictionary (expand-both-sides, so a correctly-written abbreviation matches the expanded GT).
• Place fields (place_of_stay, place_of_origin) get extra Venetian place normalization built from a corpus scan: a leading connective in and commas are dropped; the saint prefix San/Santo/Santa (and abbreviations S./St./Sta/S.ta…) collapse to one canonical, while the plural Santi is kept distinct; Cà/Ca'/Ca collapse together but not to Casa; loc.→locanda, osp.→ospizio, m.→maria, and the quondam group (q.m/qm/q(uonda)m…). Genuine misreadings (e.g. Feltrina→Pestiora) are left intact and scored by character accuracy. See evaluate.py norm_place / ABBREVIATION_EXPANSIONS / PLACE_EXPANSIONS.
• Date fields (arrival_date, departure_date) additionally strip a leading zero from the day (February 09 1769 == February 9 1769; GT is non-padded). Differently-formatted dates (day-first, month abbreviations) still differ — those are deviations from the requested Month D YYYY, not equivalents.
• Binary fields (with_spouse, with_siblings, children_count) — 1.0 if equal, 0.0 if not.
• Unscorable cells (e.g. GT title is empty) are excluded from both numerator and denominator — they neither help nor hurt.

This matches venice-39-guided/-naive's text-question scoring so numbers across the three projects are directly comparable.

Phase N — Naive (image only, no expert guidance, no transcription)

Phase A — Guided (image + expert guidance (no transcription))

Phase B — Guided + Transcription (image + expert guidance + transcription)

3. Per-image F1 by model × phase (variation across 10 runs)

Split by phase to mirror section 2. Each cell is the mean F1 across the runs for that (model, phase, page); where the runs varied, the min–max range and σ (std) follow beneath in grey. A bare number means every run scored identically (σ = 0). Cells aggregate 10 runs except where a † marks fewer successful runs.

Phase N — Naive (image only, no expert guidance, no transcription)

Phase A — Guided (image + expert guidance (no transcription))

Phase B — Guided + Transcription (image + expert guidance + transcription)

4. Per-image difficulty (averaged across models, phases, runs)

5. Notes and caveats

• Gemini 3.1 thinking budget: Gemini 3.1 preview is a thinking model with unbounded default thinking, which caused all 10 Phase A page-36 runs to hit MAX_TOKENS before any JSON was emitted. Resolved by setting thinkingLevel=LOW and max_output_tokens=32768 (collect_responses_gemini31.py:96-99); all current scores reflect successful runs. (thinkingBudget is silently ignored by Gemini 3.1 preview — thinkingLevel is the lever that works.)
• Claude extended thinking requires temperature=1 (Anthropic API constraint). All other models run at temperature=0.
• GPT 5.2 needs response_format=json_object enforcement to avoid degenerate-loop generation at temperature=0. The "image" field was also removed from the response schema after empirical evidence that GPT-5.2 hallucinates hex into a literally-named "image" field. See top of collect_responses.py.
• Claude wraps responses in markdown code fences despite the prompt saying not to. A narrow Claude-only fence-strip is applied before parsing — the only tolerance in the otherwise strict-json.loads pipeline.
• No ground-truth contamination: collection scripts never read files from gt/. The GT enters the pipeline only at evaluation. Format coincidence (e.g. q.m) comes from the prompt's expert-guidance section, not from the GT.
• Image normalization: all images are auto-oriented (EXIF rotation baked into pixels, tag stripped) before sending. 18 of the 39 venice-39-guided images had orientation=6 EXIF tags; we verified all current providers auto-orient as of 2026-05-06, but normalization is kept as defensive infrastructure.

6. Prompt provenance of the scored checkpoints

The expert guidance was revised once, on 2026-06-24. The earlier version was a condensed rewrite of the scholar's text (4,708 chars, preserved as Claude-modifiedguidance.md); the current EXPERT_GUIDANCE constant is the scholar's wording verbatim, aligned with venice-39-guided for cross-experiment comparison (8,259 chars). Which text produced which checkpoints:

Phase N predates the revision and was deliberately not re-run: the naive prompt contains no expert guidance, so the revision does not apply to it. This is also why archive_adapted_EG/ — the superseded run under the condensed guidance, collected 2026-05-06/07 — holds only checkpoints_phase_a and checkpoints_phase_b.

Two caveats on the strength of this record:

• It is inferred, not recorded. Checkpoints are bare traveler lists with no prompt field, and this repo's git history begins 2026-07-14, after both collections. The evidence is (a) file mtimes, with the archive's summary files stamped 17:15 on 2026-06-24 and the first realigned checkpoint at 17:16, and (b) the fact that only collect_responses.py defines EXPERT_GUIDANCE — the gemini31 and claude-thinking collectors import build_prompt from it, so no model can have used different text.
• Phase N's field rules are unverified. Phase N was collected under whatever NAIVE_FIELD_RULES said on 2026-05-07. If the June realignment touched the field-rules blocks and not only the guidance, N would differ from A/B on the field spec. The TODO(next collection) note in collect_responses.py implies the field rules have been stable since collection, but this cannot be confirmed from git.

Runs collected from 2026-08-19 onward record this themselves: record_provenance writes checkpoints_phase_*/prompt_provenance.json with per-component SHA-256 hashes of the guidance, field rules, assembled scaffold, and (Phase B) transcription corpus, plus the git commit and model id — and fingerprint_summary prints the same identity into phase_*.log. Runs skipped on resume are intentionally left unstamped rather than credited to the current prompt. The 150 checkpoints scored above have no sidecar; this section is their record.