benchmark playgroup

UK Charity Document Extraction — Multi-Model Benchmark

Why

A playgroup needed a fair comparison of models on the same real charity PDFs, with the same fields and the same score.

Problem

UK charity annual reports are PDFs. Eight financial fields have to come out correctly, and models differ in accuracy, failed runs, time, and cost.

Built

A scored extractor across OpenRouter, the Doubleword Batch API, and V7 Go, plus a public playground of every run.

Result

122 scored runs. The best models reach about 0.95 F1. Provider averages and failures are on the leaderboard below.

Results snapshot: 2026-09-18 · 122 scored runs · playground 2026-09-18T0710Z (25 archived versions from git history)

Who is this for?

• Playgroup attendees — start with QUICKSTART.md, then run extraction and scoring locally.
• Curious explorers — read the key findings below and open the latest playground or browse all snapshots (no code required).
• Contributors — see the repo file map for scripts, registries, and workflow.
• Doubleword / V7 teams — provider tables and timing notes mirror python score.py output in the repo README.

Key findings

Provider aggregates match python score.py over every data/*_dev_extracted__*.tsv file on 2026-09-18: 122 scored runs (46 OpenRouter, 43 Doubleword, 33 V7 Go). Registries: 39 OpenRouter keys, 36 Doubleword models, 32 V7 keys.

Doubleword’s average F1 among active models (0.827) edges OpenRouter (0.788), and Doubleword retains the highest single-model F1 (dw-deepseek-v4-pro-0813, 0.947). gemini-3-pro ranks 2nd at 0.946; new entry dw-deepseek-v4.1-flash joins at rank 10 (F1 0.941, $0.032/run — highly cost-efficient). V7 averages 0.833 F1 with no failed runs in this snapshot. V7 cost stays at zero until price_in / price_out are set in config_models_v7.py.

Top 5 models by F1

Takeaways

• Doubleword holds 5 of the global top 10. Retains the global leader (dw-deepseek-v4-pro-0813, 0.947); gemini-3-pro is 2nd at 0.946. dw-qwen3.8-27b is rank 6 at F1 0.943.
• 41 of 43 Doubleword models produced usable results (F1 range 0.827–0.947 for standard text LLMs).
• Free-tier models universally failed — the zero-score runs are free-tier or had context/format issues.
• Precision is consistently high (0.96–0.97); recall differentiates leaders.
• Hardest fields: income_annually_in_british_pounds and spending_annually_in_british_pounds.

Interactive playground

The hosted playground is a static snapshot generated by python playground.py in the repo. Eight tabs: Rankings, Field Heatmap, Document Analysis, Error Breakdown, Deep Dive, Recommendations, Provider Analysis, and Project Evolution.

Open latest playground Browse all snapshots

Playground version history

Each row is a frozen copy of which-models-extracted-playground.html from git, named with its commit time in UTC. Use this to see how the benchmark grew from 40 OpenRouter-only models (Feb 2026) to 122 scored runs across three providers (Sep 2026).

View all 25 snapshots with search and filters →

Dataset & fields

Small export from the Kleister Charity dataset (dev-0): 11 UK charity financial PDFs (up to 200 pages each) with pre-extracted OCR text. Data is UK open data — see Kleister license.

Workflow (local)

Step-by-step video guide for Doubleword batch runs: extractor.py --all-doubleword.

# Smoke test
python llm_openrouter.py

# Extract (auto-detects OpenRouter / Doubleword / V7 from registries)
python extractor.py gemini-2.0-flash
python extractor.py --all-doubleword
python extractor.py --all-v7

# Score and regenerate playground
python score.py
python playground.py

Repo essentials

• extractor.py — unified runner with checkpoint/resume for async backends
• score.py — F1 / precision / recall leaderboard
• playground.py — generates which-models-extracted-playground.html
• config_models_*.py — OpenRouter, Doubleword, and V7 model registries
• data/extraction_stats.csv — per-run stats (time, cost, field hits)
• docs/v7-go.md — focused V7 Go setup guide