Condition: Naive vs Guided — both conditions, scored identically
Figures shown: mean signed error (model − ground truth) per page across 10 runs, naive and guided overlaid per model
Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9
One point per page: x = mean(predicted left − GT left), y = mean(predicted right − GT right) for Q20. The origin is a perfect answer; up-and-right means over-counting both columns. Only the 34 pages that have brackets are plotted (Q20 is N/A on the other 5). Naive is the blue ramp, guided the green ramp, and within each ramp the shade tracks the ground-truth bracket count.
The legend is shared across all five panels — clicking Guided · 3 hides that series for every model at once. Use the buttons to switch a whole condition on or off. Axes are locked to the same range and 1:1 aspect in every panel, so panels are directly comparable.
10 runs per model per condition (t0, or t1 for Claude (thinking)); t_varied is not read. The retired single-condition pages pooled 20 runs, so figures here differ by design — and the naive one EXCLUDED Claude (thinking) entirely, because a t0+t_varied basis has no place for a t1-only model. On this basis every model contributes 10 runs, so Claude (thinking) now appears in both conditions.