Condition: Naive vs Guided — both conditions, scored identically
Figures shown: every individual run's raw Q11 integer and Q12 value count (10 per model per condition) — not accuracy

Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9

Q11 count vs number of Q12 values: Naive vs Guided

Q11 asks HOW MANY instances of “e” as a conjunction are on the page; Q12 asks for the text after each one, as a pipe-separated list. The two answers state the same count twice, so a mismatch is a model contradicting itself within one response. This is not an accuracy measure — nothing is scored against ground truth, so the Δ columns compare self-contradiction RATES between the two prompt conditions, not correctness. A Q12 value count drops empty pipe segments, so a legitimately blank Q12 (when Q11 = 0) counts as 0 rather than 1. 10 runs per model per condition; t_varied is not read.

1. Self-Contradiction Rate per Model

Disagreements as a share of the answers where Q11 parsed as an integer. A NEGATIVE Δ means guidance reduced self-contradiction.

2. Direction of the Contradiction

Whether the count claimed in Q11 EXCEEDS the number of values actually listed in Q12, or falls short of it. On the naive page these ran almost entirely one way; this shows whether guidance changes that.

3a. Every Disagreement — Naive (27)

3b. Every Disagreement — Guided (12)