Condition: Naive vs Guided — both conditions, scored identically
Figures shown: mean accuracy, run-to-run std dev and illegibility rate across 10 runs at each model's base temperature, naive and guided side by side
Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9
Why “tbase” and not “t0”: this is each model at its own established base temperature, which is t0 for four models but t1 for Claude (thinking) — extended thinking requires temperature 1, so that model has no t0 runs at all. Calling the page t0 was wrong for one of the five columns. Resolved per model: Claude t0, GPT t0, Gemini 2.5 t0, Gemini 3.1 t0, Claude (thinking) t1. 10 runs per model per condition.
Colour: naive cells use the blue ramp and guided cells the green ramp, consistent with the other comparison pages; the shade tracks the value in that cell. Δ = guided − naive.
| Model | Naive | Guided | Δ |
|---|---|---|---|
| Claude | 56.6% | 55.4% | -1.2% |
| GPT | 47.7% | 51.0% | +3.2% |
| Gemini 2.5 | 64.5% | 68.3% | +3.8% |
| Gemini 3.1 | 70.6% | 71.7% | +1.2% |
| Claude (thinking) | 63.5% | 66.8% | +3.3% |
Mean accuracy pooled over all pages and runs. Higher is better, so a positive Δ is an improvement.
| Q# | Description | Claude | GPT | Gemini 2.5 | Gemini 3.1 | Claude (thinking) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | ||
| Feature identification | ||||||||||||||||
| Q1 Bool | Has date? | 92.3% | 100.0% | +7.7% | 77.4% | 97.7% | +20.3% | 97.4% | 97.4% | ±0.0% | 100.0% | 100.0% | ±0.0% | 97.2% | 100.0% | +2.8% |
| Q3 Bool | Two columns? | 92.3% | 92.2% | ±0.0% | 97.2% | 97.4% | +0.3% | 92.8% | 97.9% | +5.1% | 95.6% | 100.0% | +4.4% | 94.1% | 99.2% | +5.1% |
| Q4 Bool | Signed? | 97.4% | 92.7% | -4.7% | 92.8% | 100.0% | +7.2% | 83.1% | 97.4% | +14.3% | 100.0% | 100.0% | ±0.0% | 95.1% | 100.0% | +4.9% |
| Q6 Bool | Top-left mark? | 74.4% | 77.7% | +3.4% | 80.3% | 83.8% | +3.6% | 90.0% | 91.0% | +1.0% | 87.2% | 84.1% | -3.1% | 70.3% | 79.0% | +8.7% |
| Q21 Bool | Bracket annotation? | 97.1% | 97.1% | ±0.0% | 81.2% | 77.1% | -4.1% | 89.9% | 89.9% | ±0.0% | 94.4% | 95.5% | +1.1% | 96.6% | 97.7% | +1.1% |
| Spatial location | ||||||||||||||||
| Q5 Str | Signature location | 97.4% | 92.7% | -4.7% | 92.8% | 100.0% | +7.2% | 84.4% | 100.0% | +15.6% | 100.0% | 100.0% | ±0.0% | 95.1% | 100.0% | +4.9% |
| Counting | ||||||||||||||||
| Q7 Int | Archivist marks | 55.1% | 67.6% | +12.5% | 46.7% | 52.3% | +5.6% | 51.5% | 71.6% | +20.0% | 62.6% | 72.3% | +9.7% | 56.4% | 65.4% | +9.0% |
| Q8 Int | Quondam count | 48.7% | 49.5% | +0.8% | 24.9% | 25.9% | +1.0% | 44.6% | 46.8% | +2.2% | 67.2% | 65.9% | -1.3% | 58.5% | 60.3% | +1.8% |
| Q9 Int | Con count | 47.4% | 45.3% | -2.1% | 23.1% | 22.6% | -0.5% | 64.4% | 71.8% | +7.5% | 69.5% | 69.0% | -0.5% | 69.7% | 71.5% | +1.8% |
| Q11 Int | E conjunction count | 22.6% | 24.4% | +1.8% | 28.5% | 32.6% | +4.1% | 17.7% | 32.0% | +14.3% | 45.4% | 47.2% | +1.8% | 39.2% | 45.6% | +6.4% |
| Q13 Int | Named travelers | 13.6% | 4.7% | -8.9% | 10.0% | 15.4% | +5.4% | 40.5% | 37.2% | -3.3% | 20.0% | 31.3% | +11.3% | 44.9% | 50.3% | +5.4% |
| Q14 Int | Unnamed travelers | 35.4% | 35.2% | ±0.0% | 15.6% | 16.2% | +0.5% | 48.2% | 53.7% | +5.5% | 47.9% | 53.1% | +5.1% | 47.2% | 50.8% | +3.6% |
| Q15 Int | Places of origin | 30.3% | 37.6% | +7.3% | 2.6% | 5.9% | +3.3% | 33.6% | 33.3% | -0.3% | 34.4% | 22.1% | -12.3% | 31.0% | 34.4% | +3.3% |
| Q16 Int | Unique places of stay | 37.9% | 18.4% | -19.6% | 29.7% | 32.1% | +2.3% | 45.4% | 51.7% | +6.3% | 52.6% | 50.0% | -2.6% | 47.7% | 49.0% | +1.3% |
| Q18 Int | Stay w/ person name | 42.8% | 33.2% | -9.7% | 12.8% | 18.2% | +5.4% | 81.3% | 86.6% | +5.3% | 62.6% | 67.4% | +4.9% | 60.0% | 46.2% | -13.8% |
| Q19 Int | Bracket total | 32.3% | 26.2% | -6.1% | 59.5% | 60.8% | +1.3% | 53.3% | 40.3% | -13.0% | 72.8% | 77.4% | +4.6% | 39.5% | 50.0% | +10.5% |
| Transcription | ||||||||||||||||
| Q10 Text | Words after con | 86.0% | 81.4% | -4.6% | 60.9% | 62.6% | +1.6% | 92.2% | 92.0% | ±0.0% | 91.5% | 91.0% | -0.5% | 88.8% | 88.0% | -0.8% |
| Q12 Text | Words after e | 58.6% | 54.7% | -3.9% | 41.1% | 44.3% | +3.2% | 51.8% | 55.3% | +3.5% | 68.7% | 70.4% | +1.8% | 62.5% | 62.3% | ±0.0% |
| Q22 Text | Annotation text | 83.2% | 85.7% | +2.5% | 63.4% | 60.0% | -3.4% | 80.4% | 83.8% | +3.4% | 90.8% | 90.1% | -0.7% | 83.6% | 84.5% | +0.9% |
| Counting + Spatial location | ||||||||||||||||
| Q17 Pair | Stay per column | 31.0% | 31.6% | +0.6% | 22.3% | 15.4% | -6.9% | 60.3% | 64.1% | +3.8% | 50.0% | 47.2% | -2.8% | 34.4% | 40.3% | +5.9% |
| Q20 Pair | Brackets per column | 25.6% | 23.8% | -1.8% | 52.1% | 47.7% | -4.4% | 51.0% | 39.5% | -11.5% | 72.8% | 77.4% | +4.6% | 35.6% | 42.8% | +7.2% |
| Transcription + Rule-based resolution | ||||||||||||||||
| Q2 Int | Year? | 45.2% | 50.4% | +5.2% | 39.1% | 59.1% | +20.0% | 67.9% | 71.9% | +4.1% | 67.5% | 67.9% | +0.4% | 48.1% | 50.7% | +2.7% |
Standard deviation of per-run mean accuracy across the 10 runs — how much the model's score wobbles from run to run. Lower is better, so the Δ colouring is inverted here: a fall in std dev is an improvement and renders green. Shade saturates at 0.07.
| Q# | Description | Claude | GPT | Gemini 2.5 | Gemini 3.1 | Claude (thinking) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | ||
| Feature identification | ||||||||||||||||
| Q1 Bool | Has date? | 2.1% | 0.0% | -2.1% | 1.1% | 0.8% | -0.3% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 1.5% | 0.0% | -1.5% |
| Q3 Bool | Two columns? | 0.0% | 0.2% | +0.2% | 1.5% | 0.0% | -1.5% | 1.1% | 1.1% | ±0.0% | 1.2% | 0.0% | -1.2% | 1.2% | 1.2% | ±0.0% |
| Q4 Bool | Signed? | 0.0% | 1.0% | +1.0% | 1.6% | 0.0% | -1.6% | 2.2% | 0.0% | -2.1% | 0.0% | 0.0% | ±0.0% | 1.5% | 0.0% | -1.5% |
| Q6 Bool | Top-left mark? | 0.0% | 1.5% | +1.5% | 4.4% | 3.2% | -1.2% | 1.9% | 1.3% | -0.6% | 0.0% | 1.1% | +1.1% | 2.5% | 5.0% | +2.5% |
| Q21 Bool | Bracket annotation? | 0.0% | 0.1% | ±0.0% | 3.2% | 2.8% | -0.3% | 1.9% | 1.1% | -0.8% | 0.0% | 1.4% | +1.4% | 1.1% | 1.2% | ±0.0% |
| Spatial location | ||||||||||||||||
| Q5 Str | Signature location | 0.0% | 1.0% | +1.0% | 1.6% | 0.0% | -1.6% | 1.9% | 0.0% | -1.9% | 0.0% | 0.0% | ±0.0% | 1.5% | 0.0% | -1.5% |
| Counting | ||||||||||||||||
| Q7 Int | Archivist marks | 2.2% | 1.8% | -0.4% | 4.0% | 5.7% | +1.7% | 4.3% | 2.2% | -2.1% | 1.8% | 3.4% | +1.6% | 3.6% | 3.0% | -0.6% |
| Q8 Int | Quondam count | 1.7% | 2.7% | +1.0% | 1.7% | 0.8% | -0.9% | 4.4% | 4.2% | ±0.0% | 2.0% | 4.0% | +2.0% | 6.6% | 3.7% | -2.9% |
| Q9 Int | Con count | 3.0% | 1.6% | -1.4% | 4.5% | 3.4% | -1.1% | 3.5% | 3.0% | -0.6% | 1.5% | 4.3% | +2.8% | 2.4% | 4.3% | +1.9% |
| Q11 Int | E conjunction count | 3.6% | 3.1% | -0.5% | 3.1% | 3.0% | ±0.0% | 4.9% | 3.0% | -1.9% | 3.0% | 3.0% | ±0.0% | 5.5% | 5.1% | -0.4% |
| Q13 Int | Named travelers | 3.6% | 2.7% | -1.0% | 1.5% | 5.8% | +4.3% | 3.8% | 3.7% | ±0.0% | 2.0% | 3.2% | +1.1% | 6.1% | 7.8% | +1.7% |
| Q14 Int | Unnamed travelers | 4.8% | 2.8% | -2.0% | 3.9% | 4.5% | +0.6% | 6.8% | 3.5% | -3.3% | 1.2% | 6.7% | +5.5% | 7.2% | 4.7% | -2.5% |
| Q15 Int | Places of origin | 3.2% | 4.6% | +1.5% | 2.4% | 3.0% | +0.6% | 5.0% | 5.9% | +0.9% | 3.7% | 4.4% | +0.7% | 2.8% | 5.2% | +2.3% |
| Q16 Int | Unique places of stay | 4.0% | 3.3% | -0.7% | 5.7% | 5.7% | ±0.0% | 3.6% | 3.6% | ±0.0% | 2.5% | 5.4% | +2.9% | 5.4% | 6.8% | +1.3% |
| Q18 Int | Stay w/ person name | 5.7% | 3.5% | -2.1% | 2.1% | 6.4% | +4.4% | 2.7% | 3.9% | +1.2% | 1.8% | 4.7% | +2.9% | 3.5% | 3.4% | ±0.0% |
| Q19 Int | Bracket total | 3.0% | 3.9% | +0.9% | 3.2% | 7.6% | +4.4% | 3.2% | 5.0% | +1.9% | 3.2% | 4.0% | +0.7% | 5.7% | 6.4% | +0.7% |
| Transcription | ||||||||||||||||
| Q10 Text | Words after con | 0.7% | 0.4% | -0.3% | 1.6% | 2.4% | +0.8% | 0.9% | 0.5% | -0.4% | 0.5% | 0.8% | +0.4% | 1.1% | 0.7% | -0.4% |
| Q12 Text | Words after e | 2.1% | 2.2% | ±0.0% | 2.3% | 2.6% | +0.3% | 2.9% | 2.2% | -0.6% | 1.3% | 2.5% | +1.3% | 2.6% | 2.8% | +0.3% |
| Q22 Text | Annotation text | 0.9% | 0.6% | -0.3% | 2.7% | 1.3% | -1.5% | 1.9% | 1.1% | -0.8% | 1.0% | 2.4% | +1.4% | 1.6% | 1.6% | ±0.0% |
| Counting + Spatial location | ||||||||||||||||
| Q17 Pair | Stay per column | 2.8% | 3.0% | ±0.0% | 5.0% | 3.8% | -1.2% | 4.7% | 4.9% | ±0.0% | 2.8% | 3.2% | +0.5% | 4.6% | 6.9% | +2.4% |
| Q20 Pair | Brackets per column | 2.1% | 3.3% | +1.2% | 2.7% | 4.9% | +2.1% | 2.8% | 5.7% | +2.9% | 3.2% | 4.0% | +0.7% | 5.7% | 4.5% | -1.2% |
| Transcription + Rule-based resolution | ||||||||||||||||
| Q2 Int | Year? | 1.2% | 1.2% | ±0.0% | 2.0% | 1.6% | -0.4% | 0.0% | 1.1% | +1.1% | 1.1% | 0.0% | -1.1% | 1.9% | 1.5% | -0.4% |
Share of answers where the model claimed the page was illegible rather than answering. Boolean questions only. Lower is better, so the Δ colouring is inverted.
| Q# | Description | Claude | GPT | Gemini 2.5 | Gemini 3.1 | Claude (thinking) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | ||
| Feature identification | ||||||||||||||||
| Q1 Bool | Has date? | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% |
| Q3 Bool | Two columns? | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% |
| Q4 Bool | Signed? | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 1.3% | 2.6% | +1.3% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% |
| Q6 Bool | Top-left mark? | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% |
| Q21 Bool | Bracket annotation? | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% |