← Data Extraction

Gemini 3.1 — Named Travelers Across Task Framings

Figures verified: 2026-08-21
Data (model runs) generated: 2026-07-23
Scoring inputs at verification: ground truth 87535c64, evaluate.py 19dedec2, rules 575192c4

Same model, same three page images, three task framings. The numbers below are mean counts across 10 t = 0 runs. This isolates a behaviour that's easy to miss when only one experiment is in view: Gemini 3.1's recall of who is on the page depends sharply on whether you ask for a count or ask it to enumerate.

1. The numbers

Each column is coloured against the ground truth for its own task: the Q&A columns against Q&A mentions, the data-extraction columns against DE unique rows. This matters on page 28, where the two ground truths differ — the 19.0 that Gemini 3.1 emits in phases N and A equals the Q&A mention count but overcounts the DE schema by one row, so it is marked as a miss. Section 3 explains why.

2. What the table says

The same model produces two systematically different counts depending on output shape. On page 36, asked directly, Gemini 3.1 says 28 travelers are named. Asked to extract them, it emits a list of 36 traveler records — off by one against the data-extraction GT of 37, but closing all but one of the nine-traveler shortfall the direct count shows. Structured emission isn't perfect (we'll see a duplication error on page 28 below), but it's dramatically closer than counting in a vacuum.

This isolates task framing as the variable. Enumerating forces the model to "look" once per row; a direct count appears to be pulled from a small-number prior rather than from an exhaustive scan. The gap widens with page size — page 39 (10 mentions) is off by 1, page 28 (19 mentions) by 1.5–2, page 36 (37 mentions) by 8.5–9. The direct count never exceeds 29 on any run of any page, while the page-36 enumeration reaches 37.

Guidance helps a little; transcription helps a lot. Adding expert domain guidance nudges the Q&A count up by half a traveler on pages 28 and 36 (17.0 → 17.5 and 28.0 → 28.5), and in the data-extraction setting it closes the page-36 gap from three records to one (34 → 36). But it doesn't change the model's strategy: the direct count still undershoots by the same margin, and the page-28 duplication described below is present in every run of both phases N and A.

Supplying the transcription changes the outcome on all three pages at once. Page 36 goes to an exact 37.0 with a mean F1 of 1.000; page 39 stays exact; and page 28 falls from 19.0 to 18.2, because 8 of the 10 phase-B runs collapse the duplicate row the other phases always emit. Phase B is the only condition whose mean row count lands within 0.2 of the DE ground truth on all three pages — exactly on 36 and 39, and on 28 in eight runs out of ten.

3. Why the two ground truths differ on page 28

Q13 asks "How many travelers are mentioned by name?" — not "how many unique travelers." Each GT applies a different counting convention, each internally consistent with the task it serves:

• Q&A GT counts mentions. On page 28, Mandolin Todros appears in the left column as an arrival and again in the right column as a departure (bracketed with Regina Todros). That's two mentions of the same person, so the Q&A GT total is 19.
• Data-extraction GT counts unique traveler records. The data-extraction schema asks for one row per traveler with separate arrival_date and departure_date fields, so Mandolin Todros contributes one row capturing both events. The DE GT for page 28 is 18.

The extra record in phases N and A is not faithful — it is a duplication error

In phases N and A, Gemini 3.1 emits 19 records on page 28 in all 10 runs. That number happens to equal the Q&A mention count, but the model isn't answering the Q&A question — it's filling out the data-extraction schema, which asks for one row per traveler. Inspecting the output (see Per-field results — Page 28) confirms it: in phase A, row 1 is "Mandolin Todros" carrying only the arrival date, and row 16 is a second "Mandolin Todros" carrying only the departure date, instead of the two being collapsed into one row with both fields filled. That is the duplication the schema explicitly forbids. It costs a one-row precision dent against the DE GT and holds page-28 F1 at 0.973 in both phases.

Phase N produces the same structural error but does not recognise the repeated name: its extra row reads Mandolin Vodevs (6 runs) or Mandolin Vodos (4 runs) rather than a second clean copy, so the model there is not knowingly listing one person twice — it has read the second occurrence as a different person.

The transcription resolves it in 8 of 10 runs

In phase B the same row comes back merged: row 1 is "Mandolin Todros" with both arrival_date and departure_date populated, and no second Todros row appears. Two of the ten runs still emit the duplicate, which is exactly what the 18.2 mean records (eight runs at 18, two at 19). Page-28 F1 rises from 0.973 to 0.995. So the duplication is not a fixed property of the model's schema-filling — it is a reading failure, and it goes away once the model no longer has to recover the second mention from the image.

The same row shows the same ordering in its dates

This page is about how many travelers are found, but the Todros row happens to make the N → A → B progression visible in a second field. Page 28 is headed 20 February 1768 M.V. — more veneto, the Venetian calendar whose year began on 1 March, so February 1768 M.V. is February 1769 in standard reckoning. The ground truth records the converted year: Mandolin Todros arrives February 9 1769 and departs February 13 1769. Across all 10 runs, every date value Gemini 3.1 writes on a Todros row:

Three different failures, in the same order as the row counts. Naive ignores the requested Month DD YYYY format and supplies no year, so there is nothing to be right or wrong about. Guided adopts the format and then copies the year off the page as written, leaving the more veneto conversion undone in seven runs out of ten — and inconsistently, since the other three convert it. With the transcription, every value carries the converted year.

One caveat against reading this as "transcription fixes dates": it fixes the year, not the whole field. Phase B writes February 12 for a departure the ground truth records as the 13th, so the day is off by one in the runs that merge the row. Phase A gets that day right more often than B does. The year convention is what moves cleanly across the three framings; the day-level reading does not.

Data sources: Q&A answers from venice-39-guided / venice-39-naive checkpoints (files eval_gemini31_t0_run01.json – run10.json, Q13), and Q&A mention totals from named_travelers_total in venice-39-guided/ground_truth.md; data-extraction n_model, f1 and the unmatched-row names from venice-39-data-extraction/scores_phase_{n,a,b}/score_gemini31_phase{n,a,b}_run*.json, with row positions read from the corresponding checkpoints_phase_* files. All numbers are means across 10 runs at temperature 0; ranges are given in parentheses where runs disagreed. This page is hand-authored — it has no generator, so it does not update when the scorer changes. Every figure above was re-verified against the current scores on 2026-08-21.