Condition: Naive vs Guided — both conditions, scored identically
Figures shown: agreement classified per page x question from each model's mean across 10 runs, naive and guided side by side
Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9
Claude (thinking) runs at t1 (temperature=1); all other models at t0. Δ = guided − naive. Green = improvement, red = regression.
Agreement describes the ENSEMBLE of all 5 models on one page × question, using each model's mean score across its 10 runs. All correct / All wrong = every model scored above 0.9 / below 0.1. One wrong = all but one correct. One right = exactly one correct and every other model wrong. Mixed = anything else.
Each class as a percentage of the page×question cells scored for that question — N is shown per condition because it varies by question (a question can be N/A on some pages), so the denominators are not all 39. Columns are agreement classes, NOT models — agreement is not a per-model quantity.
| Q# | Description | N nv | N gd | All correct | All wrong | One wrong | One right | Mixed | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | Naive | Guided | Δ | ||||
| Q1 | Has date? | 39 | 39 | 74.4% | 94.9% | +20.5% | 0.0% | 0.0% | ±0.0% | 15.4% | 5.1% | -10.3% | 0.0% | 0.0% | ±0.0% | 10.3% | 0.0% | -10.3% |
| Q2 | Year? | 32 | 28 | 28.1% | 42.9% | +14.7% | 37.5% | 17.9% | -19.6% | 12.5% | 7.1% | -5.4% | 6.2% | 10.7% | +4.5% | 15.6% | 21.4% | +5.8% |
| Q3 | Two columns? | 39 | 39 | 87.2% | 92.3% | +5.1% | 0.0% | 0.0% | ±0.0% | 5.1% | 2.6% | -2.6% | 0.0% | 0.0% | ±0.0% | 7.7% | 5.1% | -2.6% |
| Q4 | Signed? | 39 | 39 | 71.8% | 89.7% | +17.9% | 0.0% | 0.0% | ±0.0% | 17.9% | 10.3% | -7.7% | 0.0% | 0.0% | ±0.0% | 10.3% | 0.0% | -10.3% |
| Q5 | Signature location | 39 | 39 | 71.8% | 92.3% | +20.5% | 0.0% | 0.0% | ±0.0% | 20.5% | 7.7% | -12.8% | 0.0% | 0.0% | ±0.0% | 7.7% | 0.0% | -7.7% |
| Q6 | Top-left mark? | 39 | 39 | 46.2% | 51.3% | +5.1% | 2.6% | 2.6% | ±0.0% | 15.4% | 20.5% | +5.1% | 0.0% | 2.6% | +2.6% | 35.9% | 23.1% | -12.8% |
| Q7 | Archivist marks | 39 | 39 | 10.3% | 30.8% | +20.5% | 7.7% | 5.1% | -2.6% | 28.2% | 12.8% | -15.4% | 5.1% | 0.0% | -5.1% | 48.7% | 51.3% | +2.6% |
| Q8 | Quondam count | 39 | 39 | 12.8% | 20.5% | +7.7% | 2.6% | 2.6% | ±0.0% | 12.8% | 7.7% | -5.1% | 2.6% | 0.0% | -2.6% | 69.2% | 69.2% | ±0.0% |
| Q9 | Con count | 39 | 39 | 7.7% | 10.3% | +2.6% | 0.0% | 0.0% | ±0.0% | 7.7% | 17.9% | +10.3% | 5.1% | 0.0% | -5.1% | 79.5% | 71.8% | -7.7% |
| Q10 | Words after con | 39 | 39 | 7.7% | 5.1% | -2.6% | 0.0% | 0.0% | ±0.0% | 30.8% | 25.6% | -5.1% | 0.0% | 0.0% | ±0.0% | 61.5% | 69.2% | +7.7% |
| Q11 | E conjunction count | 39 | 39 | 2.6% | 10.3% | +7.7% | 5.1% | 5.1% | ±0.0% | 7.7% | 2.6% | -5.1% | 0.0% | 2.6% | +2.6% | 84.6% | 79.5% | -5.1% |
| Q12 | Words after e | 39 | 39 | 2.6% | 10.3% | +7.7% | 0.0% | 0.0% | ±0.0% | 12.8% | 5.1% | -7.7% | 0.0% | 0.0% | ±0.0% | 84.6% | 84.6% | ±0.0% |
| Q13 | Named travelers | 39 | 39 | 0.0% | 0.0% | ±0.0% | 5.1% | 0.0% | -5.1% | 0.0% | 0.0% | ±0.0% | 0.0% | 2.6% | +2.6% | 94.9% | 97.4% | +2.6% |
| Q14 | Unnamed travelers | 39 | 39 | 5.1% | 2.6% | -2.6% | 5.1% | 2.6% | -2.6% | 5.1% | 12.8% | +7.7% | 2.6% | 5.1% | +2.6% | 82.1% | 76.9% | -5.1% |
| Q15 | Places of origin | 39 | 39 | 0.0% | 2.6% | +2.6% | 5.1% | 10.3% | +5.1% | 2.6% | 0.0% | -2.6% | 0.0% | 0.0% | ±0.0% | 92.3% | 87.2% | -5.1% |
| Q16 | Unique places of stay | 39 | 39 | 7.7% | 7.7% | ±0.0% | 2.6% | 2.6% | ±0.0% | 2.6% | 2.6% | ±0.0% | 0.0% | 0.0% | ±0.0% | 87.2% | 87.2% | ±0.0% |
| Q17 | Stay per column | 39 | 39 | 10.3% | 10.3% | ±0.0% | 7.7% | 5.1% | -2.6% | 0.0% | 7.7% | +7.7% | 5.1% | 0.0% | -5.1% | 76.9% | 76.9% | ±0.0% |
| Q18 | Stay w/ person name | 39 | 39 | 2.6% | 2.6% | ±0.0% | 0.0% | 0.0% | ±0.0% | 23.1% | 12.8% | -10.3% | 5.1% | 0.0% | -5.1% | 69.2% | 84.6% | +15.4% |
| Q19 | Bracket total | 39 | 39 | 2.6% | 7.7% | +5.1% | 0.0% | 0.0% | ±0.0% | 17.9% | 10.3% | -7.7% | 2.6% | 0.0% | -2.6% | 76.9% | 82.1% | +5.1% |
| Q20 | Brackets per column | 39 | 39 | 2.6% | 7.7% | +5.1% | 0.0% | 0.0% | ±0.0% | 12.8% | 10.3% | -2.6% | 7.7% | 0.0% | -7.7% | 76.9% | 82.1% | +5.1% |
| Q21 | Bracket annotation? | 37 | 36 | 67.6% | 66.7% | -0.9% | 8.1% | 5.6% | -2.6% | 24.3% | 27.8% | +3.5% | 0.0% | 0.0% | ±0.0% | 0.0% | 0.0% | ±0.0% |
| Q22 | Annotation text | 39 | 39 | 10.3% | 12.8% | +2.6% | 0.0% | 0.0% | ±0.0% | 20.5% | 15.4% | -5.1% | 2.6% | 0.0% | -2.6% | 66.7% | 71.8% | +5.1% |
Cases where every model returned the SAME answer and all scored near zero — they agree with each other but not with ground truth, which is a strong signal of an annotation error. This is a set of cases, not a rate, so there is no Δ to take: what matters is whether a case appears in both conditions (robust signal) or only one (condition-specific).
In both conditions: 8 | naive only: 3 | guided only: 0
| Where | Page | Q# | Description | Ground truth | Model consensus |
|---|---|---|---|---|---|
| both | 4 · AVS IS 758-781 1767 02 01 p3.jpg | Q21 | Bracket annotation? | N | y |
| both | 22 · AVS IS 758-781 1768 01 02.jpg | Q2 | Year? | 1769 | 1768 |
| both | 23 · AVS IS 758-781 1768 01 09.jpg | Q2 | Year? | 1769 | 1768 |
| both | 24 · AVS IS 758-781 1768 01 16 p1.jpg | Q2 | Year? | 1769 | 1768 |
| both | 25 · AVS IS 758-781 1768 01 16 p2.jpg | Q7 | Archivist marks | 1 | 0 |
| both | 26 · AVS IS 758-781 1768 01 16 p3.jpg | Q2 | Year? | 1769 | 1768 |
| both | 6 · AVS IS 758-781 1768 03 26.jpg | Q8 | Quondam count | 2 | 0 |
| both | 34 · AVS IS 758-781 1770 05 17 p1.jpg | Q6 | Top-left mark? | N | y |
| naive only | 28 · AVS IS 758-781 1766 02.jpg | Q2 | Year? | 1769 | 1768 |
| naive only | 4 · AVS IS 758-781 1767 02 01 p3.jpg | Q2 | Year? | 1768 | 1767 |
| naive only | 22 · AVS IS 758-781 1768 01 02.jpg | Q7 | Archivist marks | 1 | 0 |
Uniquely right: this model correct while every other model was wrong — unique strength. Uniquely wrong: this model wrong while every other was correct — unique weakness. Both as a percentage of the cells that model was scored on.
| Model | Uniquely right | Uniquely wrong | ||||
|---|---|---|---|---|---|---|
| Naive | Guided | Δ | Naive | Guided | Δ | |
| Claude | 0.9% | 1.8% | +0.8% | 3.8% | 7.2% | +3.4% |
| GPT | 0.7% | 1.3% | +0.6% | 7.4% | 8.1% | +0.7% |
| Gemini 2.5 | 4.7% | 7.4% | +2.6% | 2.7% | 2.1% | -0.6% |
| Gemini 3.1 | 8.4% | 6.4% | -2.0% | 1.9% | 1.1% | -0.8% |
| Claude (thinking) | 1.2% | 1.5% | +0.4% | 0.4% | 0.0% | -0.4% |