Condition: Naive vs Guided — both conditions, scored identically
Figures shown: mean accuracy across all 10 runs, both conditions
Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9
Both models run at their fixed temperatures (Gemini 3.1 t0, Claude thinking t1=1). Bold = highest score in each row. Green columns = Gemini 3.1, blue = Claude thinking.
| Gemini 3.1 — Naive | Claude thinking — Naive | Gemini 3.1 — Guided | Claude thinking — Guided | |
|---|---|---|---|---|
| Overall | 70.6% | 63.5% | 71.7% | 66.8% |
| Feature identification | 95.5% | 90.7% | 95.9% | 95.2% |
| Spatial location | 100.0% | 95.1% | 100.0% | 100.0% |
| Counting | 53.5% | 49.4% | 55.6% | 52.3% |
| Transcription | 83.7% | 78.3% | 83.9% | 78.3% |
| Counting + Spatial location | 61.4% | 35.0% | 62.3% | 41.5% |
| Transcription + Rule-based resolution | 67.5% | 48.1% | 67.9% | 50.7% |
How much does expert guidance improve or hurt each model?
| Gemini 3.1 Δ | Claude thinking Δ | |
|---|---|---|
| Overall | +1.2% | +3.3% |
| Feature identification | +0.5% | +4.5% |
| Spatial location | ±0.0% | +4.9% |
| Counting | +2.1% | +2.9% |
| Transcription | ±0.0% | ±0.0% |
| Counting + Spatial location | +0.9% | +6.5% |
| Transcription + Rule-based resolution | +0.4% | +2.7% |
| Q# | Description | Gemini 3.1 — Naive | Claude thinking — Naive | Gemini 3.1 — Guided | Claude thinking — Guided | G3.1 Δ guided−naive |
CT Δ guided−naive |
|---|---|---|---|---|---|---|---|
| Feature identification | |||||||
| Category mean | 95.5% | 90.7% | 95.9% | 95.2% | +0.5% | +4.5% | |
| Q1 Bool | Has date? | 100.0% | 97.2% | 100.0% | 100.0% | ±0.0% | +2.8% |
| Q3 Bool | Two columns? | 95.6% | 94.1% | 100.0% | 99.2% | +4.4% | +5.1% |
| Q4 Bool | Signed? | 100.0% | 95.1% | 100.0% | 100.0% | ±0.0% | +4.9% |
| Q6 Bool | Top-left mark? | 87.2% | 70.3% | 84.1% | 79.0% | -3.1% | +8.7% |
| Q21 Bool | Bracket annotation? | 94.4% | 96.6% | 95.5% | 97.7% | +1.1% | +1.1% |
| Spatial location | |||||||
| Category mean | 100.0% | 95.1% | 100.0% | 100.0% | ±0.0% | +4.9% | |
| Q5 Str | Signature location | 100.0% | 95.1% | 100.0% | 100.0% | ±0.0% | +4.9% |
| Counting | |||||||
| Category mean | 53.5% | 49.4% | 55.6% | 52.3% | +2.1% | +2.9% | |
| Q7 Int | Archivist marks | 62.6% | 56.4% | 72.3% | 65.4% | +9.7% | +9.0% |
| Q8 Int | Quondam count | 67.2% | 58.5% | 65.9% | 60.3% | -1.3% | +1.8% |
| Q9 Int | Con count | 69.5% | 69.7% | 69.0% | 71.5% | -0.5% | +1.8% |
| Q11 Int | E conjunction count | 45.4% | 39.2% | 47.2% | 45.6% | +1.8% | +6.4% |
| Q13 Int | Named travelers | 20.0% | 44.9% | 31.3% | 50.3% | +11.3% | +5.4% |
| Q14 Int | Unnamed travelers | 47.9% | 47.2% | 53.1% | 50.8% | +5.1% | +3.6% |
| Q15 Int | Places of origin | 34.4% | 31.0% | 22.1% | 34.4% | -12.3% | +3.3% |
| Q16 Int | Unique places of stay | 52.6% | 47.7% | 50.0% | 49.0% | -2.6% | +1.3% |
| Q18 Int | Stay w/ person name | 62.6% | 60.0% | 67.4% | 46.2% | +4.9% | -13.8% |
| Q19 Int | Bracket total | 72.8% | 39.5% | 77.4% | 50.0% | +4.6% | +10.5% |
| Transcription | |||||||
| Category mean | 83.7% | 78.3% | 83.9% | 78.3% | ±0.0% | ±0.0% | |
| Q10 Text | Words after con | 91.5% | 88.8% | 91.0% | 88.0% | -0.5% | -0.8% |
| Q12 Text | Words after e | 68.7% | 62.5% | 70.4% | 62.3% | +1.8% | ±0.0% |
| Q22 Text | Annotation text | 90.8% | 83.6% | 90.1% | 84.5% | -0.7% | +0.9% |
| Counting + Spatial location | |||||||
| Category mean | 61.4% | 35.0% | 62.3% | 41.5% | +0.9% | +6.5% | |
| Q17 Pair | Stay per column | 50.0% | 34.4% | 47.2% | 40.3% | -2.8% | +5.9% |
| Q20 Pair | Brackets per column | 72.8% | 35.6% | 77.4% | 42.8% | +4.6% | +7.2% |
| Transcription + Rule-based resolution | |||||||
| Category mean | 67.5% | 48.1% | 67.9% | 50.7% | +0.4% | +2.7% | |
| Q2 Int | Year? | 67.5% | 48.1% | 67.9% | 50.7% | +0.4% | +2.7% |
Per question, which thinking model scores higher? Shows winner margin for naive and guided conditions separately.
| Q# | Description | Naive winner | Naive margin | Guided winner | Guided margin | Agreement |
|---|---|---|---|---|---|---|
| Q1 | Has date? | Gemini 3.1 | 2.8% | Gemini 3.1 | 0.0% | ✓ |
| Q2 | Year? | Gemini 3.1 | 19.4% | Gemini 3.1 | 17.1% | ✓ |
| Q3 | Two columns? | Gemini 3.1 | 1.5% | Gemini 3.1 | 0.8% | ✓ |
| Q4 | Signed? | Gemini 3.1 | 4.9% | Gemini 3.1 | 0.0% | ✓ |
| Q5 | Signature location | Gemini 3.1 | 4.9% | Gemini 3.1 | 0.0% | ✓ |
| Q6 | Top-left mark? | Gemini 3.1 | 16.9% | Gemini 3.1 | 5.1% | ✓ |
| Q7 | Archivist marks | Gemini 3.1 | 6.2% | Gemini 3.1 | 6.9% | ✓ |
| Q8 | Quondam count | Gemini 3.1 | 8.7% | Gemini 3.1 | 5.6% | ✓ |
| Q9 | Con count | Claude thinking | 0.3% | Claude thinking | 2.6% | ✓ |
| Q10 | Words after con | Gemini 3.1 | 2.8% | Gemini 3.1 | 3.0% | ✓ |
| Q11 | E conjunction count | Gemini 3.1 | 6.2% | Gemini 3.1 | 1.5% | ✓ |
| Q12 | Words after e | Gemini 3.1 | 6.2% | Gemini 3.1 | 8.1% | ✓ |
| Q13 | Named travelers | Claude thinking | 24.9% | Claude thinking | 19.0% | ✓ |
| Q14 | Unnamed travelers | Gemini 3.1 | 0.8% | Gemini 3.1 | 2.3% | ✓ |
| Q15 | Places of origin | Gemini 3.1 | 3.3% | Claude thinking | 12.3% | ✗ flip |
| Q16 | Unique places of stay | Gemini 3.1 | 4.9% | Gemini 3.1 | 1.0% | ✓ |
| Q17 | Stay per column | Gemini 3.1 | 15.6% | Gemini 3.1 | 6.9% | ✓ |
| Q18 | Stay w/ person name | Gemini 3.1 | 2.6% | Gemini 3.1 | 21.3% | ✓ |
| Q19 | Bracket total | Gemini 3.1 | 33.3% | Gemini 3.1 | 27.4% | ✓ |
| Q20 | Brackets per column | Gemini 3.1 | 37.2% | Gemini 3.1 | 34.6% | ✓ |
| Q21 | Bracket annotation? | Claude thinking | 2.1% | Claude thinking | 2.2% | ✓ |
| Q22 | Annotation text | Gemini 3.1 | 7.2% | Gemini 3.1 | 5.7% | ✓ |