Condition: Naive vs Guided — both conditions, scored identically
Figures shown: mean accuracy across all 10 runs, naive and guided side by side
Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9
Claude (thinking) runs at t1 (temperature=1); all other models at t0. Δ = guided − naive. Green = improvement, red = regression.
Question-weighted: every score from all 22 questions × all images × all 10 runs pooled into a single mean. Compare with the category-weighted figures further down, which run higher.
| Model | Naive | Guided | Δ |
|---|---|---|---|
| Claude | 56.6% | 55.4% | -1.2% |
| GPT | 47.7% | 51.0% | +3.2% |
| Gemini 2.5 | 64.5% | 68.3% | +3.8% |
| Gemini 3.1 | 70.6% | 71.7% | +1.2% |
| Claude (thinking) | 63.5% | 66.8% | +3.3% |
Char accuracy for Q5 (string), Q10, Q12, Q22 (text arrays).
| Q# | Description | Claude | GPT | Gemini 2.5 | Gemini 3.1 | Claude (thinking) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | ||
| Feature identification | ||||||||||||||||
| Category mean | 90.7% | 92.0% | +1.3% | 85.8% | 91.2% | +5.4% | 90.7% | 94.7% | +4.1% | 95.5% | 95.9% | +0.5% | 90.7% | 95.2% | +4.5% | |
| Q1 Bool | Has date? | 92.3% | 100.0% | +7.7% | 77.4% | 97.7% | +20.3% | 97.4% | 97.4% | ±0.0% | 100.0% | 100.0% | ±0.0% | 97.2% | 100.0% | +2.8% |
| Q3 Bool | Two columns? | 92.3% | 92.2% | ±0.0% | 97.2% | 97.4% | +0.3% | 92.8% | 97.9% | +5.1% | 95.6% | 100.0% | +4.4% | 94.1% | 99.2% | +5.1% |
| Q4 Bool | Signed? | 97.4% | 92.7% | -4.7% | 92.8% | 100.0% | +7.2% | 83.1% | 97.4% | +14.3% | 100.0% | 100.0% | ±0.0% | 95.1% | 100.0% | +4.9% |
| Q6 Bool | Top-left mark? | 74.4% | 77.7% | +3.4% | 80.3% | 83.8% | +3.6% | 90.0% | 91.0% | +1.0% | 87.2% | 84.1% | -3.1% | 70.3% | 79.0% | +8.7% |
| Q21 Bool | Bracket annotation? | 97.1% | 97.1% | ±0.0% | 81.2% | 77.1% | -4.1% | 89.9% | 89.9% | ±0.0% | 94.4% | 95.5% | +1.1% | 96.6% | 97.7% | +1.1% |
| Spatial location | ||||||||||||||||
| Category mean | 97.4% | 92.7% | -4.7% | 92.8% | 100.0% | +7.2% | 84.4% | 100.0% | +15.6% | 100.0% | 100.0% | ±0.0% | 95.1% | 100.0% | +4.9% | |
| Q5 Str | Signature location | 97.4% | 92.7% | -4.7% | 92.8% | 100.0% | +7.2% | 84.4% | 100.0% | +15.6% | 100.0% | 100.0% | ±0.0% | 95.1% | 100.0% | +4.9% |
| Counting | ||||||||||||||||
| Category mean | 36.6% | 34.2% | -2.4% | 25.3% | 28.2% | +2.8% | 48.1% | 52.5% | +4.5% | 53.5% | 55.6% | +2.1% | 49.4% | 52.3% | +2.9% | |
| Q7 Int | Archivist marks | 55.1% | 67.6% | +12.5% | 46.7% | 52.3% | +5.6% | 51.5% | 71.6% | +20.0% | 62.6% | 72.3% | +9.7% | 56.4% | 65.4% | +9.0% |
| Q8 Int | Quondam count | 48.7% | 49.5% | +0.8% | 24.9% | 25.9% | +1.0% | 44.6% | 46.8% | +2.2% | 67.2% | 65.9% | -1.3% | 58.5% | 60.3% | +1.8% |
| Q9 Int | Con count | 47.4% | 45.3% | -2.1% | 23.1% | 22.6% | -0.5% | 64.4% | 71.8% | +7.5% | 69.5% | 69.0% | -0.5% | 69.7% | 71.5% | +1.8% |
| Q11 Int | E conjunction count | 22.6% | 24.4% | +1.8% | 28.5% | 32.6% | +4.1% | 17.7% | 32.0% | +14.3% | 45.4% | 47.2% | +1.8% | 39.2% | 45.6% | +6.4% |
| Q13 Int | Named travelers | 13.6% | 4.7% | -8.9% | 10.0% | 15.4% | +5.4% | 40.5% | 37.2% | -3.3% | 20.0% | 31.3% | +11.3% | 44.9% | 50.3% | +5.4% |
| Q14 Int | Unnamed travelers | 35.4% | 35.2% | ±0.0% | 15.6% | 16.2% | +0.5% | 48.2% | 53.7% | +5.5% | 47.9% | 53.1% | +5.1% | 47.2% | 50.8% | +3.6% |
| Q15 Int | Places of origin | 30.3% | 37.6% | +7.3% | 2.6% | 5.9% | +3.3% | 33.6% | 33.3% | -0.3% | 34.4% | 22.1% | -12.3% | 31.0% | 34.4% | +3.3% |
| Q16 Int | Unique places of stay | 37.9% | 18.4% | -19.6% | 29.7% | 32.1% | +2.3% | 45.4% | 51.7% | +6.3% | 52.6% | 50.0% | -2.6% | 47.7% | 49.0% | +1.3% |
| Q18 Int | Stay w/ person name | 42.8% | 33.2% | -9.7% | 12.8% | 18.2% | +5.4% | 81.3% | 86.6% | +5.3% | 62.6% | 67.4% | +4.9% | 60.0% | 46.2% | -13.8% |
| Q19 Int | Bracket total | 32.3% | 26.2% | -6.1% | 59.5% | 60.8% | +1.3% | 53.3% | 40.3% | -13.0% | 72.8% | 77.4% | +4.6% | 39.5% | 50.0% | +10.5% |
| Transcription | ||||||||||||||||
| Category mean | 75.9% | 73.9% | -2.0% | 55.2% | 55.6% | +0.5% | 74.8% | 77.0% | +2.2% | 83.7% | 83.9% | ±0.0% | 78.3% | 78.3% | ±0.0% | |
| Q10 Text | Words after con | 86.0% | 81.4% | -4.6% | 60.9% | 62.6% | +1.6% | 92.2% | 92.0% | ±0.0% | 91.5% | 91.0% | -0.5% | 88.8% | 88.0% | -0.8% |
| Q12 Text | Words after e | 58.6% | 54.7% | -3.9% | 41.1% | 44.3% | +3.2% | 51.8% | 55.3% | +3.5% | 68.7% | 70.4% | +1.8% | 62.5% | 62.3% | ±0.0% |
| Q22 Text | Annotation text | 83.2% | 85.7% | +2.5% | 63.4% | 60.0% | -3.4% | 80.4% | 83.8% | +3.4% | 90.8% | 90.1% | -0.7% | 83.6% | 84.5% | +0.9% |
| Counting + Spatial location | ||||||||||||||||
| Category mean | 28.3% | 27.7% | -0.6% | 37.2% | 31.5% | -5.6% | 55.6% | 51.8% | -3.8% | 61.4% | 62.3% | +0.9% | 35.0% | 41.5% | +6.5% | |
| Q17 Pair | Stay per column | 31.0% | 31.6% | +0.6% | 22.3% | 15.4% | -6.9% | 60.3% | 64.1% | +3.8% | 50.0% | 47.2% | -2.8% | 34.4% | 40.3% | +5.9% |
| Q20 Pair | Brackets per column | 25.6% | 23.8% | -1.8% | 52.1% | 47.7% | -4.4% | 51.0% | 39.5% | -11.5% | 72.8% | 77.4% | +4.6% | 35.6% | 42.8% | +7.2% |
| Transcription + Rule-based resolution | ||||||||||||||||
| Category mean | 45.2% | 50.4% | +5.2% | 39.1% | 59.1% | +20.0% | 67.9% | 71.9% | +4.1% | 67.5% | 67.9% | +0.4% | 48.1% | 50.7% | +2.7% | |
| Q2 Int | Year? | 45.2% | 50.4% | +5.2% | 39.1% | 59.1% | +20.0% | 67.9% | 71.9% | +4.1% | 67.5% | 67.9% | +0.4% | 48.1% | 50.7% | +2.7% |
Mean accuracy across all questions in each category. Δ = guided − naive.
| Category | Questions | Claude | GPT | Gemini 2.5 | Gemini 3.1 | Claude (thinking) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | ||
| Feature identification | Q1, Q3, Q4, Q6, Q21 | 90.7% | 92.0% | +1.3% | 85.8% | 91.2% | +5.4% | 90.7% | 94.7% | +4.1% | 95.5% | 95.9% | +0.5% | 90.7% | 95.2% | +4.5% |
| Spatial location | Q5 | 97.4% | 92.7% | -4.7% | 92.8% | 100.0% | +7.2% | 84.4% | 100.0% | +15.6% | 100.0% | 100.0% | ±0.0% | 95.1% | 100.0% | +4.9% |
| Counting | Q7, Q8, Q9, Q11, Q13, Q14, Q15, Q16, Q18, Q19 | 36.6% | 34.2% | -2.4% | 25.3% | 28.2% | +2.8% | 48.1% | 52.5% | +4.5% | 53.5% | 55.6% | +2.1% | 49.4% | 52.3% | +2.9% |
| Transcription | Q10, Q12, Q22 | 75.9% | 73.9% | -2.0% | 55.2% | 55.6% | +0.5% | 74.8% | 77.0% | +2.2% | 83.7% | 83.9% | ±0.0% | 78.3% | 78.3% | ±0.0% |
| Counting + Spatial location | Q17, Q20 | 28.3% | 27.7% | -0.6% | 37.2% | 31.5% | -5.6% | 55.6% | 51.8% | -3.8% | 61.4% | 62.3% | +0.9% | 35.0% | 41.5% | +6.5% |
| Transcription + Rule-based resolution | Q2 | 45.2% | 50.4% | +5.2% | 39.1% | 59.1% | +20.0% | 67.9% | 71.9% | +4.1% | 67.5% | 67.9% | +0.4% | 48.1% | 50.7% | +2.7% |
Mean of the six category means above, so each category carries equal weight regardless of how many questions it contains — Spatial location is one question (Q5), Counting is ten. This runs higher than the question-weighted figures in the first table because the single-question categories are among the highest-scoring.
| Model | Naive | Guided | Δ |
|---|---|---|---|
| Claude | 62.4% | 61.8% | -0.5% |
| GPT | 55.9% | 60.9% | +5.0% |
| Gemini 2.5 | 70.2% | 74.7% | +4.4% |
| Gemini 3.1 | 76.9% | 77.6% | +0.7% |
| Claude (thinking) | 66.1% | 69.7% | +3.6% |
| Model | Category | Q# | Naive | Guided | Δ |
|---|---|---|---|---|---|
| GPT | Feature identification | Q1 | 77.4% | 97.7% | +20.3% |
| Gemini 2.5 | Counting | Q7 | 51.5% | 71.6% | +20.0% |
| GPT | Transcription + Rule-based resolution | Q2 | 39.1% | 59.1% | +20.0% |
| Gemini 2.5 | Spatial location | Q5 | 84.4% | 100.0% | +15.6% |
| Gemini 2.5 | Counting | Q11 | 17.7% | 32.0% | +14.3% |
| Gemini 2.5 | Feature identification | Q4 | 83.1% | 97.4% | +14.3% |
| Claude | Counting | Q7 | 55.1% | 67.6% | +12.5% |
| Gemini 3.1 | Counting | Q13 | 20.0% | 31.3% | +11.3% |
| Model | Category | Q# | Naive | Guided | Δ |
|---|---|---|---|---|---|
| Claude | Counting | Q16 | 37.9% | 18.4% | -19.6% |
| Claude (thinking) | Counting | Q18 | 60.0% | 46.2% | -13.8% |
| Gemini 2.5 | Counting | Q19 | 53.3% | 40.3% | -13.0% |
| Gemini 3.1 | Counting | Q15 | 34.4% | 22.1% | -12.3% |
| Gemini 2.5 | Counting + Spatial location | Q20 | 51.0% | 39.5% | -11.5% |
| Claude | Counting | Q18 | 42.8% | 33.2% | -9.7% |
| Claude | Counting | Q13 | 13.6% | 4.7% | -8.9% |
| GPT | Counting + Spatial location | Q17 | 22.3% | 15.4% | -6.9% |