Condition: Naive vs Guided — both conditions, scored identically
Figures shown: mean accuracy across all 10 runs, naive and guided side by side
Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9
Per-Question Difficulty: Naive vs Guided
Claude (thinking) runs at t1 (temperature=1); all other models at t0. Δ = guided − naive. Green = improvement, red = regression.
Accuracy by Question Category
Mean of the per-question accuracies in each category. Categories differ in size (Spatial location is one question, Counting is ten), so these are NOT weighted equally against each other.
| Category | Questions | Claude | GPT | Gemini 2.5 | Gemini 3.1 | Claude (thinking) |
|---|
| Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ |
|---|
| Feature identification | Q1, Q3, Q4, Q6, Q21 | 90.7% | 92.0% | +1.3% | 85.8% | 91.2% | +5.4% | 90.7% | 94.7% | +4.1% | 95.5% | 95.9% | +0.5% | 90.7% | 95.2% | +4.5% |
| Spatial location | Q5 | 97.4% | 92.7% | -4.7% | 92.8% | 100.0% | +7.2% | 84.4% | 100.0% | +15.6% | 100.0% | 100.0% | ±0.0% | 95.1% | 100.0% | +4.9% |
| Counting | Q7, Q8, Q9, Q11, Q13, Q14, Q15, Q16, Q18, Q19 | 36.6% | 34.2% | -2.4% | 25.3% | 28.2% | +2.8% | 48.1% | 52.5% | +4.5% | 53.5% | 55.6% | +2.1% | 49.4% | 52.3% | +2.9% |
| Transcription | Q10, Q12, Q22 | 75.9% | 73.9% | -2.0% | 55.2% | 55.6% | +0.5% | 74.8% | 77.0% | +2.2% | 83.7% | 83.9% | ±0.0% | 78.3% | 78.3% | ±0.0% |
| Counting + Spatial location | Q17, Q20 | 28.3% | 27.7% | -0.6% | 37.2% | 31.5% | -5.6% | 55.6% | 51.8% | -3.8% | 61.4% | 62.3% | +0.9% | 35.0% | 41.5% | +6.5% |
| Transcription + Rule-based resolution | Q2 | 45.2% | 50.4% | +5.2% | 39.1% | 59.1% | +20.0% | 67.9% | 71.9% | +4.1% | 67.5% | 67.9% | +0.4% | 48.1% | 50.7% | +2.7% |
Difficulty Ranking (hardest → easiest)
Questions sorted by mean accuracy across all five models in the GUIDED condition. The Mean columns average the five models; Δ is guided − naive of those means.
| Rank | Q# | Description | Category | Mean naive | Mean guided | Δ |
|---|
| 1 | Q15 Int | Places of origin | Counting | 26.4% | 26.6% | +0.3% |
| 2 | Q13 Int | Named travelers | Counting | 25.8% | 27.8% | +2.0% |
| 3 | Q11 Int | E conjunction count | Counting | 30.7% | 36.4% | +5.7% |
| 4 | Q17 Pair | Stay per column | Counting + Spatial location | 39.6% | 39.7% | ±0.0% |
| 5 | Q16 Int | Unique places of stay | Counting | 42.7% | 40.2% | -2.4% |
| 6 | Q14 Int | Unnamed travelers | Counting | 38.9% | 41.8% | +2.9% |
| 7 | Q20 Pair | Brackets per column | Counting + Spatial location | 47.4% | 46.3% | -1.2% |
| 8 | Q8 Int | Quondam count | Counting | 48.8% | 49.7% | +0.9% |
| 9 | Q18 Int | Stay w/ person name | Counting | 51.9% | 50.3% | -1.6% |
| 10 | Q19 Int | Bracket total | Counting | 51.5% | 50.9% | -0.6% |
| 11 | Q9 Int | Con count | Counting | 54.8% | 56.0% | +1.2% |
| 12 | Q12 Text | Words after e | Transcription | 56.5% | 57.4% | +0.9% |
| 13 | Q2 Int | Year? | Transcription + Rule-based resolution | 53.5% | 60.0% | +6.5% |
| 14 | Q7 Int | Archivist marks | Counting | 54.5% | 65.8% | +11.4% |
| 15 | Q22 Text | Annotation text | Transcription | 80.3% | 80.8% | +0.5% |
| 16 | Q10 Text | Words after con | Transcription | 83.9% | 83.0% | -0.9% |
| 17 | Q6 Bool | Top-left mark? | Feature identification | 80.4% | 83.1% | +2.7% |
| 18 | Q21 Bool | Bracket annotation? | Feature identification | 91.9% | 91.5% | -0.4% |
| 19 | Q3 Bool | Two columns? | Feature identification | 94.4% | 97.4% | +3.0% |
| 20 | Q4 Bool | Signed? | Feature identification | 93.7% | 98.0% | +4.3% |
| 21 | Q5 Str | Signature location | Spatial location | 93.9% | 98.5% | +4.6% |
| 22 | Q1 Bool | Has date? | Feature identification | 92.9% | 99.0% | +6.1% |
Per-Question Detail
| Q# | Description | Claude | GPT | Gemini 2.5 | Gemini 3.1 | Claude (thinking) |
|---|
| Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ |
|---|
| Q1 Bool | Has date? | 92.3% | 100.0% | +7.7% | 77.4% | 97.7% | +20.3% | 97.4% | 97.4% | ±0.0% | 100.0% | 100.0% | ±0.0% | 97.2% | 100.0% | +2.8% |
| Q3 Bool | Two columns? | 92.3% | 92.2% | ±0.0% | 97.2% | 97.4% | +0.3% | 92.8% | 97.9% | +5.1% | 95.6% | 100.0% | +4.4% | 94.1% | 99.2% | +5.1% |
| Q4 Bool | Signed? | 97.4% | 92.7% | -4.7% | 92.8% | 100.0% | +7.2% | 83.1% | 97.4% | +14.3% | 100.0% | 100.0% | ±0.0% | 95.1% | 100.0% | +4.9% |
| Q6 Bool | Top-left mark? | 74.4% | 77.7% | +3.4% | 80.3% | 83.8% | +3.6% | 90.0% | 91.0% | +1.0% | 87.2% | 84.1% | -3.1% | 70.3% | 79.0% | +8.7% |
| Q21 Bool | Bracket annotation? | 97.1% | 97.1% | ±0.0% | 81.2% | 77.1% | -4.1% | 89.9% | 89.9% | ±0.0% | 94.4% | 95.5% | +1.1% | 96.6% | 97.7% | +1.1% |
| Q5 Str | Signature location | 97.4% | 92.7% | -4.7% | 92.8% | 100.0% | +7.2% | 84.4% | 100.0% | +15.6% | 100.0% | 100.0% | ±0.0% | 95.1% | 100.0% | +4.9% |
| Q7 Int | Archivist marks | 55.1% | 67.6% | +12.5% | 46.7% | 52.3% | +5.6% | 51.5% | 71.6% | +20.0% | 62.6% | 72.3% | +9.7% | 56.4% | 65.4% | +9.0% |
| Q8 Int | Quondam count | 48.7% | 49.5% | +0.8% | 24.9% | 25.9% | +1.0% | 44.6% | 46.8% | +2.2% | 67.2% | 65.9% | -1.3% | 58.5% | 60.3% | +1.8% |
| Q9 Int | Con count | 47.4% | 45.3% | -2.1% | 23.1% | 22.6% | -0.5% | 64.4% | 71.8% | +7.5% | 69.5% | 69.0% | -0.5% | 69.7% | 71.5% | +1.8% |
| Q11 Int | E conjunction count | 22.6% | 24.4% | +1.8% | 28.5% | 32.6% | +4.1% | 17.7% | 32.0% | +14.3% | 45.4% | 47.2% | +1.8% | 39.2% | 45.6% | +6.4% |
| Q13 Int | Named travelers | 13.6% | 4.7% | -8.9% | 10.0% | 15.4% | +5.4% | 40.5% | 37.2% | -3.3% | 20.0% | 31.3% | +11.3% | 44.9% | 50.3% | +5.4% |
| Q14 Int | Unnamed travelers | 35.4% | 35.2% | ±0.0% | 15.6% | 16.2% | +0.5% | 48.2% | 53.7% | +5.5% | 47.9% | 53.1% | +5.1% | 47.2% | 50.8% | +3.6% |
| Q15 Int | Places of origin | 30.3% | 37.6% | +7.3% | 2.6% | 5.9% | +3.3% | 33.6% | 33.3% | -0.3% | 34.4% | 22.1% | -12.3% | 31.0% | 34.4% | +3.3% |
| Q16 Int | Unique places of stay | 37.9% | 18.4% | -19.6% | 29.7% | 32.1% | +2.3% | 45.4% | 51.7% | +6.3% | 52.6% | 50.0% | -2.6% | 47.7% | 49.0% | +1.3% |
| Q18 Int | Stay w/ person name | 42.8% | 33.2% | -9.7% | 12.8% | 18.2% | +5.4% | 81.3% | 86.6% | +5.3% | 62.6% | 67.4% | +4.9% | 60.0% | 46.2% | -13.8% |
| Q19 Int | Bracket total | 32.3% | 26.2% | -6.1% | 59.5% | 60.8% | +1.3% | 53.3% | 40.3% | -13.0% | 72.8% | 77.4% | +4.6% | 39.5% | 50.0% | +10.5% |
| Q10 Text | Words after con | 86.0% | 81.4% | -4.6% | 60.9% | 62.6% | +1.6% | 92.2% | 92.0% | ±0.0% | 91.5% | 91.0% | -0.5% | 88.8% | 88.0% | -0.8% |
| Q12 Text | Words after e | 58.6% | 54.7% | -3.9% | 41.1% | 44.3% | +3.2% | 51.8% | 55.3% | +3.5% | 68.7% | 70.4% | +1.8% | 62.5% | 62.3% | ±0.0% |
| Q22 Text | Annotation text | 83.2% | 85.7% | +2.5% | 63.4% | 60.0% | -3.4% | 80.4% | 83.8% | +3.4% | 90.8% | 90.1% | -0.7% | 83.6% | 84.5% | +0.9% |
| Q17 Pair | Stay per column | 31.0% | 31.6% | +0.6% | 22.3% | 15.4% | -6.9% | 60.3% | 64.1% | +3.8% | 50.0% | 47.2% | -2.8% | 34.4% | 40.3% | +5.9% |
| Q20 Pair | Brackets per column | 25.6% | 23.8% | -1.8% | 52.1% | 47.7% | -4.4% | 51.0% | 39.5% | -11.5% | 72.8% | 77.4% | +4.6% | 35.6% | 42.8% | +7.2% |
| Q2 Int | Year? | 45.2% | 50.4% | +5.2% | 39.1% | 59.1% | +20.0% | 67.9% | 71.9% | +4.1% | 67.5% | 67.9% | +0.4% | 48.1% | 50.7% | +2.7% |
Hardest and Easiest per Model (guided)
Ranked within the guided condition; the naive figure and Δ show whether guidance helped that question.
Claude
| | Q# | Description | Naive | Guided | Δ |
|---|
| hardest | Q13 | Named travelers | 13.6% | 4.7% | -8.9% |
| hardest | Q16 | Unique places of stay | 37.9% | 18.4% | -19.6% |
| hardest | Q20 | Brackets per column | 25.6% | 23.8% | -1.8% |
| hardest | Q11 | E conjunction count | 22.6% | 24.4% | +1.8% |
| hardest | Q19 | Bracket total | 32.3% | 26.2% | -6.1% |
| easiest | Q1 | Has date? | 92.3% | 100.0% | +7.7% |
| easiest | Q21 | Bracket annotation? | 97.1% | 97.1% | ±0.0% |
| easiest | Q5 | Signature location | 97.4% | 92.7% | -4.7% |
| easiest | Q4 | Signed? | 97.4% | 92.7% | -4.7% |
| easiest | Q3 | Two columns? | 92.3% | 92.2% | ±0.0% |
GPT
| | Q# | Description | Naive | Guided | Δ |
|---|
| hardest | Q15 | Places of origin | 2.6% | 5.9% | +3.3% |
| hardest | Q13 | Named travelers | 10.0% | 15.4% | +5.4% |
| hardest | Q17 | Stay per column | 22.3% | 15.4% | -6.9% |
| hardest | Q14 | Unnamed travelers | 15.6% | 16.2% | +0.5% |
| hardest | Q18 | Stay w/ person name | 12.8% | 18.2% | +5.4% |
| easiest | Q5 | Signature location | 92.8% | 100.0% | +7.2% |
| easiest | Q4 | Signed? | 92.8% | 100.0% | +7.2% |
| easiest | Q1 | Has date? | 77.4% | 97.7% | +20.3% |
| easiest | Q3 | Two columns? | 97.2% | 97.4% | +0.3% |
| easiest | Q6 | Top-left mark? | 80.3% | 83.8% | +3.6% |
Gemini 2.5
| | Q# | Description | Naive | Guided | Δ |
|---|
| hardest | Q11 | E conjunction count | 17.7% | 32.0% | +14.3% |
| hardest | Q15 | Places of origin | 33.6% | 33.3% | -0.3% |
| hardest | Q13 | Named travelers | 40.5% | 37.2% | -3.3% |
| hardest | Q20 | Brackets per column | 51.0% | 39.5% | -11.5% |
| hardest | Q19 | Bracket total | 53.3% | 40.3% | -13.0% |
| easiest | Q5 | Signature location | 84.4% | 100.0% | +15.6% |
| easiest | Q3 | Two columns? | 92.8% | 97.9% | +5.1% |
| easiest | Q4 | Signed? | 83.1% | 97.4% | +14.3% |
| easiest | Q1 | Has date? | 97.4% | 97.4% | ±0.0% |
| easiest | Q10 | Words after con | 92.2% | 92.0% | ±0.0% |
Gemini 3.1
| | Q# | Description | Naive | Guided | Δ |
|---|
| hardest | Q15 | Places of origin | 34.4% | 22.1% | -12.3% |
| hardest | Q13 | Named travelers | 20.0% | 31.3% | +11.3% |
| hardest | Q11 | E conjunction count | 45.4% | 47.2% | +1.8% |
| hardest | Q17 | Stay per column | 50.0% | 47.2% | -2.8% |
| hardest | Q16 | Unique places of stay | 52.6% | 50.0% | -2.6% |
| easiest | Q5 | Signature location | 100.0% | 100.0% | ±0.0% |
| easiest | Q4 | Signed? | 100.0% | 100.0% | ±0.0% |
| easiest | Q3 | Two columns? | 95.6% | 100.0% | +4.4% |
| easiest | Q1 | Has date? | 100.0% | 100.0% | ±0.0% |
| easiest | Q21 | Bracket annotation? | 94.4% | 95.5% | +1.1% |
Claude (thinking)
| | Q# | Description | Naive | Guided | Δ |
|---|
| hardest | Q15 | Places of origin | 31.0% | 34.4% | +3.3% |
| hardest | Q17 | Stay per column | 34.4% | 40.3% | +5.9% |
| hardest | Q20 | Brackets per column | 35.6% | 42.8% | +7.2% |
| hardest | Q11 | E conjunction count | 39.2% | 45.6% | +6.4% |
| hardest | Q18 | Stay w/ person name | 60.0% | 46.2% | -13.8% |
| easiest | Q5 | Signature location | 95.1% | 100.0% | +4.9% |
| easiest | Q4 | Signed? | 95.1% | 100.0% | +4.9% |
| easiest | Q1 | Has date? | 97.2% | 100.0% | +2.8% |
| easiest | Q3 | Two columns? | 94.1% | 99.2% | +5.1% |
| easiest | Q21 | Bracket annotation? | 96.6% | 97.7% | +1.1% |