Page generated: 2026-08-27
Data (model runs) generated: 2026-07-23
Scoring inputs: ground truth 36c390d2, evaluate.py 5c52b890, rules 355f2e77

Raw vs normalized similarity

Per page and model, the mean over that cell's 10 runs of the similarity between model output and ground truth, scored twice: on the values as recorded, and after normalization plus the EQUIVALENCES.md rules.

Row pairing is identical in both columns — established once by the current matcher — so the gap is what normalization and the rules are worth to transcription scoring alone. Their effect on detection is F1, reported in summary.html.

name is 1 − CER on the traveler name. fields is the micro-averaged score over the eight scored fields. Both use character accuracy for text and exact equality for the Y/N and count fields; the only difference between the two columns is normalization.

Phase N — image only

page	model	runs	name raw	name norm	Δ	fields raw	fields norm	Δ
28	claude	10	0.813	0.865	+0.052	0.793	0.719	-0.074
28	claude_thinking	10	0.793	0.836	+0.042	0.825	0.749	-0.076
28	gemini	10	0.853	0.887	+0.034	0.797	0.736	-0.062
28	gemini31	10	0.871	0.902	+0.030	0.847	0.824	-0.023
28	gpt	10	0.262	0.773	+0.512	0.751	0.671	-0.080
36	claude	10	0.705	0.803	+0.098	0.713	0.677	-0.036
36	claude_thinking	10	0.753	0.793	+0.040	0.766	0.668	-0.098
36	gemini	10	0.787	0.829	+0.042	0.728	0.656	-0.072
36	gemini31	10	0.881	0.885	+0.004	0.804	0.760	-0.043
36	gpt	10	0.579	0.772	+0.193	0.626	0.576	-0.050
39	claude	10	0.844	0.852	+0.007	0.769	0.693	-0.076
39	claude_thinking	10	0.810	0.823	+0.013	0.847	0.715	-0.132
39	gemini	10	0.744	0.754	+0.010	0.757	0.643	-0.114
39	gemini31	10	0.885	0.897	+0.012	0.810	0.725	-0.085
39	gpt	10	0.619	0.723	+0.104	0.730	0.632	-0.099

Phase A — image + guidance

page	model	runs	name raw	name norm	Δ	fields raw	fields norm	Δ
28	claude	10	0.806	0.854	+0.047	0.836	0.749	-0.088
28	claude_thinking	10	0.809	0.833	+0.024	0.931	0.764	-0.167
28	gemini	10	0.873	0.917	+0.045	0.936	0.844	-0.093
28	gemini31	10	0.920	0.929	+0.008	0.945	0.839	-0.106
28	gpt	10	0.678	0.852	+0.174	0.882	0.692	-0.190
36	claude	10	0.742	0.806	+0.064	0.756	0.678	-0.078
36	claude_thinking	10	0.778	0.797	+0.020	0.860	0.739	-0.121
36	gemini	10	0.809	0.811	+0.002	0.818	0.770	-0.048
36	gemini31	10	0.831	0.851	+0.020	0.865	0.770	-0.096
36	gpt	10	0.682	0.767	+0.085	0.779	0.646	-0.132
39	claude	10	0.847	0.855	+0.007	0.910	0.778	-0.132
39	claude_thinking	10	0.811	0.822	+0.011	0.907	0.753	-0.154
39	gemini	10	0.819	0.827	+0.008	0.872	0.730	-0.142
39	gemini31	10	0.875	0.888	+0.013	0.911	0.802	-0.110
39	gpt	10	0.678	0.752	+0.074	0.796	0.659	-0.137

Phase B — image + guidance + transcription

page	model	runs	name raw	name norm	Δ	fields raw	fields norm	Δ
28	claude	10	0.971	0.974	+0.003	0.969	0.912	-0.057
28	claude_thinking	10	0.962	0.982	+0.021	0.972	0.915	-0.057
28	gemini	10	0.966	0.995	+0.029	0.959	0.917	-0.042
28	gemini31	10	0.924	0.995	+0.071	0.962	0.914	-0.048
28	gpt	10	0.903	0.971	+0.068	0.935	0.912	-0.022
36	claude	10	0.999	0.999	+0.000	0.957	0.909	-0.047
36	claude_thinking	10	0.998	0.998	+0.000	0.965	0.967	+0.002
36	gemini	10	0.997	0.997	+0.000	0.982	0.975	-0.007
36	gemini31	10	0.997	0.997	+0.000	0.964	0.960	-0.004
36	gpt	10	0.997	0.997	+0.000	0.949	0.948	-0.001
39	claude	10	0.982	0.994	+0.012	0.951	0.938	-0.013
39	claude_thinking	10	0.998	0.999	+0.001	0.949	0.946	-0.002
39	gemini	10	1.000	1.000	+0.000	0.915	0.885	-0.030
39	gemini31	10	1.000	1.000	+0.000	0.949	0.925	-0.024
39	gpt	10	1.000	1.000	+0.000	0.942	0.950	+0.008