Condition: Naive vs Guided — both conditions, scored identically
Figures shown: mean accuracy across 10 runs (t0; Claude thinking t1), naive and guided side by side

Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9

Q20 and Q22 Performance by Bracket Count: Naive vs Guided

Claude (thinking) runs at t1 (temperature=1); all other models at t0. Δ = guided − naive. Green = improvement, red = regression.

Pages: 39  |  Runs: 10 per model per condition (t0; Claude (thinking) at t1). Q20 = brackets per column, exact match. Q22 = annotation text, character accuracy. The 0-bracket group is included — those 5 pages score well because answering zero is correct. Note: the retired single-condition pages averaged 20 runs (10 t0 + 10 t_varied); this page is a 10-run mean, so figures differ.

Q20 Accuracy by Bracket Count

BracketsPagesClaudeGPTGemini 2.5Gemini 3.1Claude (thinking)
NaiveGuidedΔNaiveGuidedΔNaiveGuidedΔNaiveGuidedΔNaiveGuidedΔ
0580.0%80.0%±0.0%100.0%98.0%-2.0%24.0%24.0%±0.0%60.0%68.0%+8.0%76.0%84.0%+8.0%
1633.3%48.3%+15.0%81.7%76.7%-5.0%65.0%63.3%-1.7%80.0%73.3%-6.7%50.0%65.0%+15.0%
2728.6%27.1%-1.4%82.9%70.0%-12.9%75.7%47.1%-28.6%97.1%100.0%+2.9%51.4%67.1%+15.7%
347.5%10.5%+3.0%82.5%45.0%-37.5%42.5%37.5%-5.0%100.0%100.0%±0.0%27.5%32.5%+5.0%
442.5%0.0%-2.5%10.0%0.0%-10.0%42.5%32.5%-10.0%75.0%75.0%±0.0%27.5%20.0%-7.5%
520.0%0.0%±0.0%10.0%10.0%±0.0%50.0%20.0%-30.0%90.0%100.0%+10.0%20.0%20.0%±0.0%
6532.0%0.0%-32.0%12.0%30.0%+18.0%48.0%26.0%-22.0%56.0%74.0%+18.0%8.0%22.0%+14.0%
720.0%0.0%±0.0%5.0%30.0%+25.0%60.0%55.6%-4.4%50.0%50.0%±0.0%15.0%10.0%-5.0%
810.0%0.0%±0.0%0.0%0.0%±0.0%0.0%0.0%±0.0%100.0%80.0%-20.0%0.0%0.0%±0.0%
1010.0%0.0%±0.0%0.0%10.0%+10.0%60.0%40.0%-20.0%20.0%50.0%+30.0%10.0%0.0%-10.0%
1110.0%0.0%±0.0%0.0%0.0%±0.0%40.0%77.8%+37.8%0.0%30.0%+30.0%0.0%0.0%±0.0%
1310.0%0.0%±0.0%0.0%0.0%±0.0%50.0%40.0%-10.0%0.0%10.0%+10.0%10.0%10.0%±0.0%
All pages3925.6%23.8%-1.8%52.1%47.7%-4.4%51.0%39.5%-11.5%72.8%77.4%+4.6%35.6%42.8%+7.2%

Q22 Accuracy by Bracket Count

BracketsPagesClaudeGPTGemini 2.5Gemini 3.1Claude (thinking)
NaiveGuidedΔNaiveGuidedΔNaiveGuidedΔNaiveGuidedΔNaiveGuidedΔ
0580.0%80.0%±0.0%100.0%100.0%±0.0%42.0%52.0%+10.0%64.0%68.0%+4.0%80.0%84.0%+4.0%
1683.3%83.9%+0.6%50.9%40.0%-10.9%97.9%100.0%+2.1%100.0%88.3%-11.7%82.2%85.4%+3.2%
2780.3%90.6%+10.3%54.7%56.8%+2.0%90.4%92.0%+1.6%97.3%98.4%+1.1%85.3%90.8%+5.5%
3494.9%95.9%+1.1%61.8%59.8%-2.0%84.8%91.1%+6.3%99.7%100.0%+0.3%96.5%91.8%-4.7%
4486.7%87.6%+0.8%59.2%48.5%-10.8%70.2%80.3%+10.0%95.0%95.5%+0.5%86.2%85.2%-1.0%
5275.8%87.7%+11.9%69.2%67.9%-1.3%81.1%77.3%-3.8%93.3%92.1%-1.2%78.9%81.0%+2.1%
6587.0%86.3%-0.8%64.2%63.5%-0.6%83.1%82.7%-0.4%87.6%88.8%+1.2%84.0%82.3%-1.6%
7268.7%73.2%+4.5%66.0%53.2%-12.7%85.2%90.4%+5.1%88.9%88.9%±0.0%76.8%78.1%+1.3%
8173.9%66.7%-7.3%19.0%24.0%+5.0%75.2%69.8%-5.5%93.4%94.0%+0.7%65.2%50.6%-14.6%
10194.4%84.9%-9.6%54.9%62.1%+7.2%87.1%80.3%-6.8%82.2%89.4%+7.2%91.8%88.5%-3.2%
11182.1%86.2%+4.2%82.7%83.3%+0.6%90.4%91.1%+0.7%96.3%92.5%-3.8%73.1%69.4%-3.8%
13181.4%80.7%-0.7%52.7%41.0%-11.7%84.8%89.1%+4.3%87.5%93.4%+5.9%77.4%80.9%+3.5%
All pages3983.2%85.7%+2.5%63.4%60.0%-3.4%80.4%83.8%+3.4%90.8%90.1%-0.7%83.6%84.5%+0.9%

Per-Page Q20 Scores (sorted by bracket count)

PageBracketsClaudeGPTGemini 2.5Gemini 3.1Claude (thinking)
NaiveGuidedΔNaiveGuidedΔNaiveGuidedΔNaiveGuidedΔNaiveGuidedΔ
4 · AVS IS 758-781 1767 02 01 p3.jpg00.0%0.0%±0.0%100.0%100.0%±0.0%0.0%0.0%±0.0%0.0%40.0%+40.0%0.0%20.0%+20.0%
5 · AVS IS 758-781 1767 02 01 p4.jpg0100.0%100.0%±0.0%100.0%100.0%±0.0%0.0%0.0%±0.0%100.0%100.0%±0.0%80.0%100.0%+20.0%
14 · AVS IS 758-781 1768 05 09 p1.jpg0100.0%100.0%±0.0%100.0%100.0%±0.0%0.0%0.0%±0.0%0.0%0.0%±0.0%100.0%100.0%±0.0%
15 · AVS IS 758-781 1768 05 09 p3.jpg0100.0%100.0%±0.0%100.0%100.0%±0.0%70.0%20.0%-50.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
29 · AVS IS 758-781 1769 05 p1.jpg0100.0%100.0%±0.0%100.0%90.0%-10.0%50.0%100.0%+50.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
28 · AVS IS 758-781 1766 02.jpg10.0%0.0%±0.0%100.0%100.0%±0.0%60.0%80.0%+20.0%20.0%20.0%±0.0%0.0%0.0%±0.0%
3 · AVS IS 758-781 1767 02 01 p2.jpg1100.0%100.0%±0.0%10.0%30.0%+20.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
26 · AVS IS 758-781 1768 01 16 p3.jpg10.0%50.0%+50.0%80.0%30.0%-50.0%0.0%0.0%±0.0%60.0%40.0%-20.0%0.0%60.0%+60.0%
13 · AVS IS 758-781 1768 05 09 p2.jpg1100.0%100.0%±0.0%100.0%100.0%±0.0%50.0%100.0%+50.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
17 · AVS IS 758-781 1768 08 01.jpg10.0%0.0%±0.0%100.0%100.0%±0.0%80.0%0.0%-80.0%100.0%80.0%-20.0%30.0%30.0%±0.0%
39 · AVS IS 758-781 1770 08.jpg10.0%40.0%+40.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%70.0%100.0%+30.0%
22 · AVS IS 758-781 1768 01 02.jpg20.0%0.0%±0.0%100.0%80.0%-20.0%0.0%20.0%+20.0%100.0%100.0%±0.0%0.0%10.0%+10.0%
25 · AVS IS 758-781 1768 01 16 p2.jpg20.0%0.0%±0.0%100.0%90.0%-10.0%80.0%40.0%-40.0%100.0%100.0%±0.0%30.0%80.0%+50.0%
7 · AVS IS 758-781 1768 05 02.jpg2100.0%90.0%-10.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
9 · AVS IS 758-781 1768 05 08 p2.jpg20.0%0.0%±0.0%100.0%50.0%-50.0%100.0%100.0%±0.0%100.0%100.0%±0.0%30.0%20.0%-10.0%
18 · AVS IS 758-781 1768 09.jpg2100.0%100.0%±0.0%80.0%70.0%-10.0%100.0%0.0%-100.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
32 · AVS IS 758-781 1770 05 13.jpg20.0%0.0%±0.0%100.0%100.0%±0.0%100.0%70.0%-30.0%80.0%100.0%+20.0%40.0%90.0%+50.0%
35 · AVS IS 758-781 1770 05 17 p2.jpg20.0%0.0%±0.0%0.0%0.0%±0.0%50.0%0.0%-50.0%100.0%100.0%±0.0%60.0%70.0%+10.0%
16 · AVS IS 758-781 1768 05 16.jpg30.0%0.0%±0.0%70.0%0.0%-70.0%70.0%0.0%-70.0%100.0%100.0%±0.0%40.0%40.0%±0.0%
19 · AVS IS 758-781 1768 11 07.jpg30.0%0.0%±0.0%80.0%60.0%-20.0%0.0%30.0%+30.0%100.0%100.0%±0.0%0.0%0.0%±0.0%
30 · AVS IS 758-781 1769 05 p2.jpg330.0%50.0%+20.0%100.0%70.0%-30.0%100.0%100.0%±0.0%100.0%100.0%±0.0%70.0%90.0%+20.0%
33 · AVS IS 758-781 1770 05.jpg30.0%0.0%±0.0%80.0%50.0%-30.0%0.0%20.0%+20.0%100.0%100.0%±0.0%0.0%0.0%±0.0%
2 · AVS IS 758-781 1767 02 01 p1.jpg40.0%0.0%±0.0%0.0%0.0%±0.0%0.0%30.0%+30.0%10.0%90.0%+80.0%60.0%0.0%-60.0%
24 · AVS IS 758-781 1768 01 16 p1.jpg40.0%0.0%±0.0%0.0%0.0%±0.0%0.0%0.0%±0.0%100.0%100.0%±0.0%0.0%10.0%+10.0%
6 · AVS IS 758-781 1768 03 26.jpg410.0%0.0%-10.0%40.0%0.0%-40.0%70.0%0.0%-70.0%90.0%10.0%-80.0%0.0%0.0%±0.0%
31 · AVS IS 758-781 1770 03.jpg40.0%0.0%±0.0%0.0%0.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%50.0%70.0%+20.0%
1 · AVS IS 758-781 1767 07 01.jpg50.0%0.0%±0.0%0.0%0.0%±0.0%0.0%0.0%±0.0%80.0%100.0%+20.0%10.0%30.0%+20.0%
20 · AVS IS 758-781 1768 11 28.jpg50.0%0.0%±0.0%20.0%20.0%±0.0%100.0%40.0%-60.0%100.0%100.0%±0.0%30.0%10.0%-20.0%
23 · AVS IS 758-781 1768 01 09.jpg60.0%0.0%±0.0%40.0%0.0%-40.0%0.0%0.0%±0.0%0.0%60.0%+60.0%0.0%20.0%+20.0%
12 · AVS IS 758-781 1768 05 08 p5.jpg60.0%0.0%±0.0%0.0%40.0%+40.0%100.0%100.0%±0.0%40.0%100.0%+60.0%0.0%10.0%+10.0%
8 · AVS IS 758-781 1768 05 08.jpg60.0%0.0%±0.0%0.0%60.0%+60.0%40.0%0.0%-40.0%40.0%70.0%+30.0%0.0%10.0%+10.0%
21 · AVS IS 758-781 1768 05.jpg6100.0%0.0%-100.0%20.0%30.0%+10.0%100.0%30.0%-70.0%100.0%90.0%-10.0%40.0%40.0%±0.0%
34 · AVS IS 758-781 1770 05 17 p1.jpg660.0%0.0%-60.0%0.0%20.0%+20.0%0.0%0.0%±0.0%100.0%50.0%-50.0%0.0%30.0%+30.0%
27 · AVS IS 758-781 1768 02 20.jpg70.0%0.0%±0.0%0.0%50.0%+50.0%20.0%0.0%-20.0%100.0%100.0%±0.0%0.0%0.0%±0.0%
38 · AVS IS 758-781 1770 05 28 p3.jpg70.0%0.0%±0.0%10.0%10.0%±0.0%100.0%100.0%±0.0%0.0%0.0%±0.0%30.0%20.0%-10.0%
36 · AVS IS 758-781 1770 05 28 p1.jpg80.0%0.0%±0.0%0.0%0.0%±0.0%0.0%0.0%±0.0%100.0%80.0%-20.0%0.0%0.0%±0.0%
10 · AVS IS 758-781 1768 05 08 p3.jpg100.0%0.0%±0.0%0.0%10.0%+10.0%60.0%40.0%-20.0%20.0%50.0%+30.0%10.0%0.0%-10.0%
11 · AVS IS 758-781 1768 05 08 p4.jpg110.0%0.0%±0.0%0.0%0.0%±0.0%40.0%77.8%+37.8%0.0%30.0%+30.0%0.0%0.0%±0.0%
37 · AVS IS 758-781 1770 05 28 p2.jpg130.0%0.0%±0.0%0.0%0.0%±0.0%50.0%40.0%-10.0%0.0%10.0%+10.0%10.0%10.0%±0.0%

Per-Page Q22 Scores (sorted by bracket count)

PageBracketsClaudeGPTGemini 2.5Gemini 3.1Claude (thinking)
NaiveGuidedΔNaiveGuidedΔNaiveGuidedΔNaiveGuidedΔNaiveGuidedΔ
4 · AVS IS 758-781 1767 02 01 p3.jpg00.0%0.0%±0.0%100.0%100.0%±0.0%0.0%0.0%±0.0%0.0%40.0%+40.0%0.0%20.0%+20.0%
5 · AVS IS 758-781 1767 02 01 p4.jpg0100.0%100.0%±0.0%100.0%100.0%±0.0%0.0%60.0%+60.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
14 · AVS IS 758-781 1768 05 09 p1.jpg0100.0%100.0%±0.0%100.0%100.0%±0.0%30.0%0.0%-30.0%20.0%0.0%-20.0%100.0%100.0%±0.0%
15 · AVS IS 758-781 1768 05 09 p3.jpg0100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
29 · AVS IS 758-781 1769 05 p1.jpg0100.0%100.0%±0.0%100.0%100.0%±0.0%80.0%100.0%+20.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
28 · AVS IS 758-781 1766 02.jpg10.0%3.3%+3.3%0.0%0.0%±0.0%100.0%100.0%±0.0%100.0%80.0%-20.0%3.3%16.7%+13.3%
3 · AVS IS 758-781 1767 02 01 p2.jpg1100.0%100.0%±0.0%45.2%20.0%-25.2%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%98.6%-1.4%
26 · AVS IS 758-781 1768 01 16 p3.jpg1100.0%100.0%±0.0%40.0%20.0%-20.0%100.0%100.0%±0.0%100.0%50.0%-50.0%90.0%100.0%+10.0%
13 · AVS IS 758-781 1768 05 09 p2.jpg1100.0%100.0%±0.0%20.0%0.0%-20.0%87.1%100.0%+12.9%100.0%100.0%±0.0%100.0%97.1%-2.9%
17 · AVS IS 758-781 1768 08 01.jpg1100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
39 · AVS IS 758-781 1770 08.jpg1100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%
22 · AVS IS 758-781 1768 01 02.jpg277.4%78.2%+0.9%0.0%0.0%±0.0%76.8%91.2%+14.4%100.0%98.2%-1.8%75.3%82.9%+7.6%
25 · AVS IS 758-781 1768 01 16 p2.jpg221.8%100.0%+78.2%71.9%71.9%±0.0%75.3%73.4%-1.9%100.0%100.0%±0.0%64.0%96.2%+32.2%
7 · AVS IS 758-781 1768 05 02.jpg288.2%85.1%-3.1%53.6%71.6%+18.0%86.4%86.0%-0.4%97.1%97.1%±0.0%88.2%87.3%-0.9%
9 · AVS IS 758-781 1768 05 08 p2.jpg274.8%70.7%-4.1%72.3%64.3%-8.0%94.5%93.2%-1.4%93.2%93.2%±0.0%77.7%81.4%+3.6%
18 · AVS IS 758-781 1768 09.jpg2100.0%100.0%±0.0%85.5%70.7%-14.8%99.5%100.0%+0.5%100.0%100.0%±0.0%100.0%99.8%-0.2%
32 · AVS IS 758-781 1770 05 13.jpg2100.0%100.0%±0.0%100.0%100.0%±0.0%100.0%100.0%±0.0%90.6%100.0%+9.4%100.0%100.0%±0.0%
35 · AVS IS 758-781 1770 05 17 p2.jpg2100.0%100.0%±0.0%0.0%18.8%+18.8%100.0%100.0%±0.0%100.0%100.0%±0.0%91.8%87.9%-3.8%
16 · AVS IS 758-781 1768 05 16.jpg392.9%92.2%-0.8%0.0%0.0%±0.0%94.7%100.0%+5.3%100.0%100.0%±0.0%98.2%97.6%-0.6%
19 · AVS IS 758-781 1768 11 07.jpg394.8%92.3%-2.5%84.3%84.5%±0.0%58.5%74.8%+16.3%100.0%100.0%±0.0%94.5%86.0%-8.5%
30 · AVS IS 758-781 1769 05 p2.jpg391.7%100.0%+8.3%83.8%82.3%-1.5%97.9%99.4%+1.5%100.0%100.0%±0.0%98.3%98.8%+0.5%
33 · AVS IS 758-781 1770 05.jpg3100.0%100.0%±0.0%79.0%72.3%-6.7%88.1%90.4%+2.2%98.9%100.0%+1.1%95.0%84.7%-10.3%
2 · AVS IS 758-781 1767 02 01 p1.jpg497.0%87.8%-9.3%33.7%0.0%-33.7%53.8%78.7%+25.0%99.8%98.4%-1.4%81.5%79.2%-2.4%
24 · AVS IS 758-781 1768 01 16 p1.jpg490.2%91.3%+1.1%39.5%37.4%-2.2%67.6%73.6%+6.0%94.1%94.1%±0.0%88.1%82.4%-5.7%
6 · AVS IS 758-781 1768 03 26.jpg478.5%90.1%+11.7%83.3%84.9%+1.6%75.4%76.1%+0.7%100.0%100.0%±0.0%90.6%93.3%+2.7%
31 · AVS IS 758-781 1770 03.jpg481.2%81.2%±0.0%80.6%71.7%-8.9%84.1%92.5%+8.4%86.0%89.5%+3.5%84.4%85.8%+1.4%
1 · AVS IS 758-781 1767 07 01.jpg573.2%81.8%+8.6%67.9%69.2%+1.3%70.0%76.2%+6.2%86.6%84.2%-2.4%68.9%77.3%+8.4%
20 · AVS IS 758-781 1768 11 28.jpg578.4%93.7%+15.3%70.6%66.7%-3.9%92.1%78.3%-13.8%100.0%100.0%±0.0%89.0%84.7%-4.3%
23 · AVS IS 758-781 1768 01 09.jpg671.8%70.3%-1.5%63.8%63.5%-0.3%84.6%88.2%+3.6%68.3%76.8%+8.6%72.3%75.9%+3.7%
12 · AVS IS 758-781 1768 05 08 p5.jpg693.0%91.8%-1.2%83.4%85.0%+1.6%97.4%96.3%-1.1%87.9%97.3%+9.4%80.3%82.7%+2.4%
8 · AVS IS 758-781 1768 05 08.jpg695.0%91.4%-3.6%42.8%39.9%-2.9%56.4%54.8%-1.6%94.2%86.4%-7.8%92.4%86.0%-6.4%
21 · AVS IS 758-781 1768 05.jpg691.7%91.0%-0.7%81.8%76.3%-5.6%91.6%89.4%-2.2%92.3%92.3%±0.0%86.4%83.6%-2.8%
34 · AVS IS 758-781 1770 05 17 p1.jpg683.6%86.8%+3.2%48.9%52.9%+4.0%85.6%84.9%-0.8%95.2%90.9%-4.3%88.5%83.5%-5.0%
27 · AVS IS 758-781 1768 02 20.jpg761.5%79.3%+17.8%58.8%39.6%-19.2%81.5%81.6%±0.0%95.6%95.8%±0.0%75.0%73.9%-1.1%
38 · AVS IS 758-781 1770 05 28 p3.jpg776.0%67.8%-8.1%73.1%66.8%-6.3%88.9%97.4%+8.5%82.1%82.0%±0.0%78.7%82.4%+3.7%
36 · AVS IS 758-781 1770 05 28 p1.jpg873.9%66.7%-7.3%19.0%24.0%+5.0%75.2%69.8%-5.5%93.4%94.0%+0.7%65.2%50.6%-14.6%
10 · AVS IS 758-781 1768 05 08 p3.jpg1094.4%84.9%-9.6%54.9%62.1%+7.2%87.1%80.3%-6.8%82.2%89.4%+7.2%91.8%88.5%-3.2%
11 · AVS IS 758-781 1768 05 08 p4.jpg1182.1%86.2%+4.2%82.7%83.3%+0.6%90.4%91.1%+0.7%96.3%92.5%-3.8%73.1%69.4%-3.8%
37 · AVS IS 758-781 1770 05 28 p2.jpg1381.4%80.7%-0.7%52.7%41.0%-11.7%84.8%89.1%+4.3%87.5%93.4%+5.9%77.4%80.9%+3.5%