← Deep Source Analysis with LLMs

1.4 Expert-Directed Analysis of Deep Sources

This experiment is to measure how the models improve (or not) with expert guidance. This experiment was run manaually on three document images across three frontier models as of September 2025, Claude Opus, GPT5, and Gemini 2.5 Pro. The scale of this analysis made it possible to provide accurate transcriptions as ground truth and carefully evaluate the model outputs including the “thinking” “reasoning” steps.

Question Answering - Tasks 1 and 2

The questions are meant to be straightforward and easily measurable. There are binary questions, counting questions, structural questions, and two that require OCR.

TASK 1: The guidance comes in the form of 1. Contextual information about the sources from a scholar’s point of view. (See )

TASK 2: In addition to the contextual information, the models are given full transcriptions of the documents.

Task 1 Instructions

For the attached document images, provide answers to the following questions in a tabular format. Create a separate table for each image. Include the confidence value (as percentage measure) of the answer for each question.

Task 1 Performance Evaluation

To evaluate Task 1.1 performance, we compared the model responses to the ground truth answers for each of the three documents using an evaluation methodology that we arrived at by first allowing the three models to define their own methodology, reviewing and revising. See the methodology described here: task-1-task2-comparison-methodology.md

Factual Accuracy

• Gemini 2.5 Pro: Best at basic document features (dates, columns, signatures)
• Claude Opus 4: Moderate accuracy, inconsistent with complex counts
• GPT-5: Good on structural elements, struggles with detailed content

Text Recognition

• GPT-5: Acknowledges illegibility but provides fewer answers
• Gemini 2.5 Pro: Confident responses but many inaccuracies in text extraction
• Claude Opus 4: Mixed performance, some good transcriptions but many errors

Analytical Capability

• All models struggled with distinguishing word usage contexts (“con” vs “e”)
• Gemini 2.5 Pro: Most consistent attempts at analysis
• GPT-5: Often acknowledged difficulty rather than guessing

Completeness vs Accuracy Trade-off

• Gemini 2.5 Pro: High completion rate but lower accuracy
• GPT-5: Lower completion rate and introduces confusing qualifications of certainty
• Claude Opus 4: Moderate completion with variable accuracy

Task 2 Instructions

For the attached document images and transcriptions, provide answers to the following questions in a tabular format. Create a separate table for each image. Include the confidence value (as percentage measure)of the answer for each question.

Task 2 Performance Evaluation

See the full results at task2-comparison.md. The same evaluation methodology was used as for Task 1.1.

Model Rankings

1. Tie: Gemini 2.5 Pro & GPT-5: 53/63 (84.1%)
2. Claude Opus: 34/63 (54.0%)

Category Performance Analysis by Question Type

Binary Questions (Y/N)

• Claude Opus: 13/15 (86.7%)
• Gemini 2.5 Pro: 15/15 (100%)
• GPT-5: 15/15 (100%)

Factual Extraction (Counts, Values)

• Claude Opus: 11/30 (36.7%)
• Gemini 2.5 Pro: 25/30 (83.3%)
• GPT-5: 24/30 (80.0%)

Complex Text Analysis (Word following, annotations)

• Claude Opus: 10/18 (55.6%)
• Gemini 2.5 Pro: 13/18 (72.2%)
• GPT-5: 14/18 (77.8%)

Key Findings

1. Overall Performance: Gemini 2.5 Pro and GPT-5 had identical overall scores (84.1%). Claude Opus (54.0%).

2. Strengths by Model:
   • Gemini 2.5 Pro: Perfect on binary questions, strong factual extraction
   • GPT-5: Perfect on binary questions, best at complex text analysis
   • Claude Opus: Good at binary questions but struggled with counting and extraction tasks
3. Common Error Patterns:
   • All models had difficulty with counting circular marks
   • Column-specific counting proved challenging
   • Complex annotations with curly brackets caused errors
4. Document-Specific Performance:
   • Document 1 (August 20, 1770): GPT-5 performed best (90.5%)
   • Document 2 (May 28, 1770): Tie between Gemini and GPT-5 (76.2%)
   • Document 3 (February 20, 1768): Gemini 2.5 Pro performed best (90.5%)
5. Consistency: Gemini 2.5 Pro showed the most consistent performance across documents (85.7%, 76.2%, 90.5%), while Claude Opus showed significant variability (42.9%, 42.9%, 76.2%).

Task 3 Named Travelers

This third task is to extract named travelers and their associated fields from the same three documents. The models were given the same contextual information as in Task 1.1 for Task 3.1, and the transcriptions as in Task 2 for Task 3.2. The evaluation methodology is described here: task3-comparison-methodology.md

Task 3.1 Instructions

Attached are three document images. Provide a table of output per image of all the named travelers (named foreign visitors) and the following information for each: Traveler Name, Traveler title, Date of arrival, Date of departure, Place of stay, Traveling with spouse (yes or no)?, Traveling with sibling/s (yes or no)?, Traveling with children? (count), Place of origin.

Performance Metrics

Key Findings

1. Best Overall Performance: Gemini Pro 2.5 significantly outperformed the other models with 20% strict recall and 17.1% field accuracy

2. Document Difficulty: May 28, 1770 proved most challenging for all models, with Claude and GPT5 unable to extract any matching travelers

3. Name Transcription: Gemini Pro 2.5 achieved the highest name transcription accuracy at 80.7%, suggesting better OCR capabilities

4. Common Errors:
   • All models struggled with handwritten text legibility
   • Place of origin often confused or incorrectly transcribed
   • Date fields frequently missing or incorrect
   • False positives (fabricated travelers) appeared in all model outputs
5. Model-Specific Observations:
   • Claude Opus: Conservative approach, many “not clear” entries
   • OpenAI GPT5: Acknowledged illegibility but attempted some extraction
   • Gemini Pro 2.5: Most aggressive extraction, highest accuracy but also more false positives

Task 3.2 Instructions

Attached are three document images and their transcriptions. Provide a table of output per image of all the named travelers (named foreign visitors) and the following information for each: Traveler Name, Traveler title, Date of arrival, Date of departure, Place of stay, Traveling with spouse (yes or no)?, Traveling with sibling/s (yes or no)?, Traveling with children? (count), Place of origin.

Performance Metrics

See full results at task3.2-comparison.md

traveler detection rate (98.5% recall)

• Highest name transcription accuracy (99.8%)
• Tied for lowest false positives (1)

Most Accurate Fields: GPT-5

• Slightly highest field accuracy (81.5%)
• But missed more travelers than Claude (92.3% vs 98.5% recall)

Most False Positives: Gemini 2.5 Pro

• Generated 5 false positive travelers
• Lowest field accuracy (80.4%)
• But showed some ability to identify probable matches (relaxed recall 93.1% vs strict 92.3%)

Performance Patterns:

• All models achieved >99% name transcription accuracy as expected since the transcriptions were provided.
• Field accuracy (80-82%) was similar across models, suggesting systematic challenges with certain field types
• The main differentiator was traveler detection completeness.

Propel 1.4 — Scale Experiment Results (39 images, 5 models, 10 runs)

Full automated evaluation across 39 archival document images using 5 frontier models with expert guidance.

Analysis Reports

• Naive vs Guided Comparison — All 5 models: guided vs naive accuracy, Δ per question
• Gemini 3.1 vs Claude Thinking — Head-to-head thinking models, naive & guided
• Evaluation Report (t0) — Full per-question accuracy table
• Agreement Analysis — Cross-model agreement patterns
• Difficulty Analysis — Which questions are hardest
• Question Breakdown — Per-question detail
• Run Chart — Accuracy across 10 runs
• Temperature Analysis — t0 vs t_varied comparison
• Text Questions — OCR / character accuracy deep dive
• Bracket Analysis — Curly bracket question analysis
• Bracket Errors — Bracket error patterns

Token usage by task and model