Condition: Guided — models given the questions plus expert domain guidance
Figures shown: mean token usage per image across all 10 runs, plus grand totals
Page generated: 2026-08-16
Data (model runs) generated: 2026-04-11
Scoring inputs: ground truth cc2ed8f5, evaluate.py 845b70e9
Token counts as reported by each model's API. Claude (thinking) output includes thinking tokens.
Mean input and output tokens per image, across all runs in the primary temp config.
| Model | Temp | Images | Avg Input | Avg Output | Avg Total | Input:Output Ratio |
|---|---|---|---|---|---|---|
| Claude | t0 | 386 | 5,823 | 419 | 6,242 | 13.9:1 |
| Claude | t_varied | 386 | 5,822 | 417 | 6,239 | 14.0:1 |
| GPT | t0 | 390 | 5,635 | 445 | 6,080 | 12.7:1 |
| GPT | t_varied | 390 | 5,635 | 453 | 6,088 | 12.4:1 |
| Gemini 2.5 | t0 | 387 | 4,346 | 534 | 4,880 | 8.1:1 |
| Gemini 2.5 | t_varied | 389 | 4,346 | 575 | 4,921 | 7.6:1 |
| Gemini 3.1 | t0 | 390 | 5,162 | 484 | 5,646 | 10.7:1 |
| Claude (thinking) | t1 | 390 | 5,852 | 4,244 | 10,096 | 1.4:1 |
| Model | Runs | Total Input | Total Output | Grand Total | % of All Tokens |
|---|---|---|---|---|---|
| Claude | 20 | 4,494,954 | 322,731 | 4,817,685 | 24.7% |
| GPT | 20 | 4,395,560 | 350,068 | 4,745,628 | 24.3% |
| Gemini 2.5 | 20 | 3,372,496 | 430,343 | 3,802,839 | 19.5% |
| Gemini 3.1 | 10 | 2,013,350 | 188,591 | 2,201,941 | 11.3% |
| Claude (thinking) | 10 | 2,282,090 | 1,655,226 | 3,937,316 | 20.2% |
| TOTAL | 16,558,450 | 2,946,959 | 19,505,409 | 100% |
Total output tokens per run (39 images). Input tokens are near-constant per model (same prompt + images). Claude (thinking) shown on a separate scale.
| Run | Claude | GPT | Gemini 2.5 | Gemini 3.1 |
|---|---|---|---|---|
| 1 | 16,398 | 17,598 | 21,041 | 19,802 |
| 2 | 16,402 | 17,606 | 20,627 | 18,723 |
| 3 | 16,443 | 16,781 | 20,358 | 18,319 |
| 4 | 16,304 | 17,293 | 21,049 | 18,577 |
| 5 | 15,927 | 17,222 | 19,909 | 18,965 |
| 6 | 16,299 | 17,307 | 21,058 | 18,103 |
| 7 | 15,765 | 17,851 | 20,823 | 19,004 |
| 8 | 15,873 | 17,223 | 20,400 | 19,206 |
| 9 | 16,416 | 17,569 | 21,134 | 18,779 |
| 10 | 16,025 | 16,931 | 20,179 | 19,113 |
| Mean | 16,185 | 17,338 | 20,658 | 18,859 |
| Run | Output Tokens | vs Mean |
|---|---|---|
| 1 | 166,105 | +582 |
| 2 | 164,904 | −619 |
| 3 | 166,163 | +640 |
| 4 | 169,930 | +4,407 |
| 5 | 156,645 | −8,878 |
| 6 | 176,148 | +10,625 |
| 7 | 158,291 | −7,232 |
| 8 | 165,289 | −234 |
| 9 | 165,426 | −97 |
| 10 | 166,325 | +802 |
| Mean | 165,523 |
Total tokens per run (input + output) compared to mean accuracy at t0. Lower tokens per accuracy point = more efficient.
| Model | Tokens/Run | Accuracy | Tokens per 1% Accuracy |
|---|---|---|---|
| Claude | 240,934 | 55.5% | 4,341 |
| GPT | 237,116 | 51.5% | 4,604 |
| Gemini 2.5 | 188,848 | 68.1% | 2,773 |
| Gemini 3.1 | 220,194 | 71.5% | 3,080 |
| Claude (thinking) | 393,732 | 66.4% | 5,930 |