Naive vs Guided Question Answering
39 archival document images · 5 models · 10 runs each
Two experimental conditions: Naive — models prompted with the questions only; Guided — same questions with an expert-authored domain-knowledge block prepended. Pills indicate which condition each report covers.
The Experiment Design: Questions, Evaluation Guidelines, and Sourcesapplies to both experiments
-
The 22 questions asked of each page image — categories, data types, and answer formats.
-
The domain-knowledge block injected into the system prompt for guided-condition runs.
-
Scoring rules, pre-processing, illegibility handling, and equivalence dictionary.
-
Thumbnail grid of all 39 pages, sortable by page ID, structural complexity, visual density, overall complexity and density, or mean accuracy.
Cross-experiment Comparisons naive vs guided
-
All 5 models compared across naive and guided conditions; per-question accuracy and deltas.
-
Per-image combined mean accuracy (across all 5 models) for naive and guided side by side, with a sortable bar chart and a per-model heatmap.
-
Per-page signed error from GT for all 10 counting questions, with naive and guided overlaid for direct comparison.
-
The underlying per-run integers behind the chart above — every individual run's raw answer (10 per model) for all 10 counting questions, naive and guided in separate tables. Raw responses, not accuracy.
-
Per-question comparison of original vs revised expert guidance across all 5 models.
-
Which questions are hardest and easiest across models, naive and guided side by side with deltas.
-
Q20 and Q22 accuracy grouped by bracket count, naive and guided side by side with deltas.
-
Cross-model agreement per question, possible ground-truth errors, and per-model outliers — naive and guided side by side.
-
t0 vs varied temperature within each condition — naive and guided in separate tables. Gemini 3.1 is t0-only and Claude (thinking) t1-only by design.
-
Q20 signed error per page, naive (blue) and guided (green) overlaid across five models. Shade tracks bracket count; condition and bracket count can each be filtered.
-
Per-question accuracy, run-to-run consistency and illegibility rates, naive and guided side by side. Each model at its base temperature — t0, except Claude (thinking) at t1.
-
Does the count a model gives for Q11 match the number of values it lists in Q12? Raw per-run counts, not accuracy — naive and guided compared. Guidance roughly halved self-contradiction (1.4% → 0.6%).
-
Does Q1 “is there a date” agree with whether Q2 supplies a year? Every page × model × run, with the raw Q2 answer and its accuracy, naive and guided compared. Self-contradiction is rare, but on the 11 pages ground truth records no date the models volunteered a year in 22% of naive answers — and Q2 is unscored there.
-
Head-to-head between the two thinking models across naive and guided conditions.
Per-experiment Analyses click a pill to view that condition
-
Page ViewerPer-image view of model answers and accuracy across all questions.
-
Transcription ViewerBrowse model transcriptions alongside ground truth per image.
Guided-only Analyses guided condition
-
Structural complexity (text density, traveler counts, brackets) vs model accuracy. Correlations and per-page ranking.
-
Box plots of raw model response values per page — actual counts, not accuracy. Toggle models on/off.
-
Comparative token usage across models — per-image averages, grand totals, and cost efficiency.