Condition: Naive vs Guided — both conditions, scored identically
Figures shown: mean accuracy across 10 runs per temperature; t0 vs varied within each condition, naive and guided in separate tables
Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9
Δ = varied − t0, within each condition. The two tables are the two prompt conditions; they are kept separate because the comparison of interest here is temperature, not naive vs guided.
Claude, GPT, Gemini 2.5 have both temperature conditions. Gemini 3.1 shows t0 only — its temperature is not settable, so a varied condition does not exist by design (this is an exception, not missing data). Claude (thinking) shows t1 only — extended thinking requires temperature 1, so it has neither t0 nor varied. 10 runs per model per temperature.
| Q# | Description | Claude | GPT | Gemini 2.5 | Gemini 3.1 | Claude (thinking) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| t0 | varied | Δ | t0 | varied | Δ | t0 | varied | Δ | t0 | t1 | ||
| Feature identification | ||||||||||||
| Q1 Bool | Has date? | 92.3% | 91.5% | -0.8% | 77.4% | 76.7% | -0.8% | 97.4% | 96.4% | -1.0% | 100.0% | 97.2% |
| Q3 Bool | Two columns? | 92.3% | 92.3% | ±0.0% | 97.2% | 97.2% | ±0.0% | 92.8% | 93.8% | +1.0% | 95.6% | 94.1% |
| Q4 Bool | Signed? | 97.4% | 97.4% | ±0.0% | 92.8% | 92.3% | -0.5% | 83.1% | 80.3% | -2.8% | 100.0% | 95.1% |
| Q6 Bool | Top-left mark? | 74.4% | 74.8% | +0.4% | 80.3% | 79.0% | -1.3% | 90.0% | 89.2% | -0.8% | 87.2% | 70.3% |
| Q21 Bool | Bracket annotation? | 97.1% | 96.6% | -0.6% | 81.2% | 79.2% | -2.0% | 89.9% | 89.7% | -0.2% | 94.4% | 96.6% |
| Spatial location | ||||||||||||
| Q5 Str | Signature location | 97.4% | 97.4% | ±0.0% | 92.8% | 92.3% | -0.5% | 84.4% | 82.1% | -2.3% | 100.0% | 95.1% |
| Counting | ||||||||||||
| Q7 Int | Archivist marks | 55.1% | 55.5% | +0.4% | 46.7% | 45.6% | -1.0% | 51.5% | 51.0% | -0.5% | 62.6% | 56.4% |
| Q8 Int | Quondam count | 48.7% | 48.3% | -0.4% | 24.9% | 25.1% | +0.3% | 44.6% | 43.6% | -1.0% | 67.2% | 58.5% |
| Q9 Int | Con count | 47.4% | 44.5% | -3.0% | 23.1% | 20.8% | -2.3% | 64.4% | 65.9% | +1.5% | 69.5% | 69.7% |
| Q11 Int | E conjunction count | 22.6% | 22.6% | ±0.0% | 28.5% | 26.7% | -1.8% | 17.7% | 24.9% | +7.2% | 45.4% | 39.2% |
| Q13 Int | Named travelers | 13.6% | 14.7% | +1.1% | 10.0% | 14.6% | +4.6% | 40.5% | 36.4% | -4.1% | 20.0% | 44.9% |
| Q14 Int | Unnamed travelers | 35.4% | 33.4% | -2.0% | 15.6% | 17.9% | +2.3% | 48.2% | 47.4% | -0.8% | 47.9% | 47.2% |
| Q15 Int | Places of origin | 30.3% | 29.8% | -0.4% | 2.6% | 6.7% | +4.1% | 33.6% | 33.6% | ±0.0% | 34.4% | 31.0% |
| Q16 Int | Unique places of stay | 37.9% | 42.2% | +4.2% | 29.7% | 22.3% | -7.4% | 45.4% | 44.9% | -0.5% | 52.6% | 47.7% |
| Q18 Int | Stay w/ person name | 42.8% | 45.5% | +2.7% | 12.8% | 13.6% | +0.8% | 81.3% | 71.3% | -10.0% | 62.6% | 60.0% |
| Q19 Int | Bracket total | 32.3% | 33.9% | +1.6% | 59.5% | 55.1% | -4.4% | 53.3% | 50.3% | -3.1% | 72.8% | 39.5% |
| Transcription | ||||||||||||
| Q10 Text | Words after con | 86.0% | 85.3% | -0.7% | 60.9% | 58.1% | -2.8% | 92.2% | 91.2% | -1.0% | 91.5% | 88.8% |
| Q12 Text | Words after e | 58.6% | 58.8% | +0.2% | 41.1% | 40.7% | -0.4% | 51.8% | 52.2% | +0.3% | 68.7% | 62.5% |
| Q22 Text | Annotation text | 83.2% | 82.4% | -0.8% | 63.4% | 60.2% | -3.2% | 80.4% | 79.0% | -1.4% | 90.8% | 83.6% |
| Counting + Spatial location | ||||||||||||
| Q17 Pair | Stay per column | 31.0% | 31.6% | +0.6% | 22.3% | 17.7% | -4.6% | 60.3% | 54.6% | -5.6% | 50.0% | 34.4% |
| Q20 Pair | Brackets per column | 25.6% | 24.9% | -0.7% | 52.1% | 47.7% | -4.4% | 51.0% | 47.2% | -3.8% | 72.8% | 35.6% |
| Transcription + Rule-based resolution | ||||||||||||
| Q2 Int | Year? | 45.2% | 44.9% | -0.3% | 39.1% | 36.1% | -3.0% | 67.9% | 69.9% | +2.0% | 67.5% | 48.1% |
| Overall | 56.6% | 56.7% | ±0.0% | 47.7% | 46.5% | -1.3% | 64.5% | 63.3% | -1.3% | 70.6% | 63.5% | |
| Q# | Description | Claude | GPT | Gemini 2.5 | Gemini 3.1 | Claude (thinking) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| t0 | varied | Δ | t0 | varied | Δ | t0 | varied | Δ | t0 | t1 | ||
| Feature identification | ||||||||||||
| Q1 Bool | Has date? | 100.0% | 100.0% | ±0.0% | 97.7% | 97.2% | -0.5% | 97.4% | 97.2% | -0.2% | 100.0% | 100.0% |
| Q3 Bool | Two columns? | 92.2% | 92.2% | ±0.0% | 97.4% | 97.4% | ±0.0% | 97.9% | 97.9% | ±0.0% | 100.0% | 99.2% |
| Q4 Bool | Signed? | 92.7% | 94.8% | +2.1% | 100.0% | 99.7% | -0.3% | 97.4% | 97.7% | +0.3% | 100.0% | 100.0% |
| Q6 Bool | Top-left mark? | 77.7% | 74.6% | -3.1% | 83.8% | 76.9% | -6.9% | 91.0% | 89.5% | -1.5% | 84.1% | 79.0% |
| Q21 Bool | Bracket annotation? | 97.1% | 97.1% | ±0.0% | 77.1% | 74.2% | -2.9% | 89.9% | 90.6% | +0.7% | 95.5% | 97.7% |
| Spatial location | ||||||||||||
| Q5 Str | Signature location | 92.7% | 94.8% | +2.1% | 100.0% | 99.7% | -0.3% | 100.0% | 99.5% | -0.5% | 100.0% | 100.0% |
| Counting | ||||||||||||
| Q7 Int | Archivist marks | 67.6% | 63.2% | -4.4% | 52.3% | 48.2% | -4.1% | 71.6% | 72.0% | +0.4% | 72.3% | 65.4% |
| Q8 Int | Quondam count | 49.5% | 49.5% | ±0.0% | 25.9% | 25.6% | -0.3% | 46.8% | 44.5% | -2.3% | 65.9% | 60.3% |
| Q9 Int | Con count | 45.3% | 43.5% | -1.8% | 22.6% | 21.0% | -1.5% | 71.8% | 73.3% | +1.4% | 69.0% | 71.5% |
| Q11 Int | E conjunction count | 24.4% | 26.2% | +1.8% | 32.6% | 29.5% | -3.1% | 32.0% | 37.8% | +5.7% | 47.2% | 45.6% |
| Q13 Int | Named travelers | 4.7% | 10.1% | +5.4% | 15.4% | 11.5% | -3.8% | 37.2% | 39.8% | +2.6% | 31.3% | 50.3% |
| Q14 Int | Unnamed travelers | 35.2% | 35.5% | +0.3% | 16.2% | 16.9% | +0.8% | 53.7% | 54.0% | +0.2% | 53.1% | 50.8% |
| Q15 Int | Places of origin | 37.6% | 31.3% | -6.2% | 5.9% | 11.3% | +5.4% | 33.3% | 31.1% | -2.2% | 22.1% | 34.4% |
| Q16 Int | Unique places of stay | 18.4% | 22.0% | +3.6% | 32.1% | 26.7% | -5.4% | 51.7% | 47.3% | -4.4% | 50.0% | 49.0% |
| Q18 Int | Stay w/ person name | 33.2% | 31.3% | -1.8% | 18.2% | 16.9% | -1.3% | 86.6% | 78.4% | -8.2% | 67.4% | 46.2% |
| Q19 Int | Bracket total | 26.2% | 35.2% | +9.1% | 60.8% | 56.4% | -4.4% | 40.3% | 44.5% | +4.2% | 77.4% | 50.0% |
| Transcription | ||||||||||||
| Q10 Text | Words after con | 81.4% | 81.2% | -0.2% | 62.6% | 60.3% | -2.2% | 92.0% | 91.2% | -0.8% | 91.0% | 88.0% |
| Q12 Text | Words after e | 54.7% | 56.9% | +2.1% | 44.3% | 40.3% | -4.1% | 55.3% | 54.8% | -0.6% | 70.4% | 62.3% |
| Q22 Text | Annotation text | 85.7% | 85.3% | -0.4% | 60.0% | 58.4% | -1.6% | 83.8% | 82.2% | -1.6% | 90.1% | 84.5% |
| Counting + Spatial location | ||||||||||||
| Q17 Pair | Stay per column | 31.6% | 28.2% | -3.4% | 15.4% | 14.4% | -1.0% | 64.1% | 59.9% | -4.2% | 47.2% | 40.3% |
| Q20 Pair | Brackets per column | 23.8% | 29.0% | +5.2% | 47.7% | 48.5% | +0.8% | 39.5% | 42.7% | +3.1% | 77.4% | 42.8% |
| Transcription + Rule-based resolution | ||||||||||||
| Q2 Int | Year? | 50.4% | 50.7% | +0.4% | 59.1% | 55.9% | -3.2% | 71.9% | 71.2% | -0.8% | 67.9% | 50.7% |
| Overall | 55.4% | 55.9% | +0.5% | 51.0% | 49.2% | -1.8% | 68.3% | 68.0% | -0.4% | 71.7% | 66.8% | |