benchmark playgroup

UK Charity Document Extraction — Multi-Model Benchmark

Extract structured fields from UK charity financial PDFs using LLMs via OpenRouter, the Doubleword Batch API, or V7 Go, then score and rank results across every run in data/.

repo: playgroup_202602_docextract dataset: Kleister Charity (11 PDFs) fields: 8

Open latest playground All 21 snapshots View on GitHub QUICKSTART

Results snapshot: 2026-08-22 · 111 scored runs · playground 2026-08-22T0830Z (21 archived versions from git history)

Who is this for?

Playgroup attendees — start with QUICKSTART.md, then run extraction and scoring locally.
Curious explorers — read the key findings below and open the latest playground or browse all snapshots (no code required).
Contributors — see the repo file map for scripts, registries, and workflow.
Doubleword / V7 teams — provider tables and timing notes mirror python score.py output in the repo README.

Key findings

Provider aggregates match python score.py over every data/*_dev_extracted__*.tsv file on 2026-08-22: 111 scored runs (40 OpenRouter, 38 Doubleword, 33 V7 Go). Registries: 33 OpenRouter keys, 31 Doubleword models, 32 V7 keys.

Provider	Models	Active	Failed	Avg F1	Best F1	Best model	Avg fields	Avg time	Avg cost
Doubleword	38	36	2	0.812	0.943	dw-qwen3.8-27b	63.5/85 (75%)	763s	$0.036
OpenRouter	40	27	13	0.753	0.946	gemini-3-pro	56.0/85 (66%)	1314s	$0.092
V7 Go	33	33	0	0.833	0.852	gpt4-1	62.8/85 (74%)	124s	$0.000

Doubleword’s average F1 among active models (0.812) edges OpenRouter (0.753); OpenRouter still holds the highest single-model F1 (gemini-3-pro, 0.946). dw-qwen3.8-27b ranks 2nd at 0.943. V7 averages 0.833 F1 with no failed runs in this snapshot. V7 cost stays at zero until price_in / price_out are set in config_models_v7.py.

Top 5 models by F1

Rank	Model	Provider	F1	Precision	Recall	Fields found
1	gemini-3-pro	OpenRouter	0.946	0.975	0.918	78/85
2	dw-qwen3.8-27b	Doubleword	0.943	0.987	0.903	77/85
3=	dw-qwen3-5-397b-a17b-dottxt	Doubleword	0.942	0.975	0.911	77/85
3=	dw-kimi-k3	Doubleword	0.942	0.975	0.911	77/85
5	qwen3-235b	OpenRouter	0.937	0.975	0.902	77/85

Takeaways

Doubleword holds 8 of the global top 10. Best DW models outperform all OpenRouter models except gemini-3-pro.
36 of 38 Doubleword models produced usable results (F1 0.812–0.943 for standard text LLMs).
Free-tier models universally failed — all 14 zero-score models are free-tier or had context/format issues.
Precision is consistently high (0.96–0.97); recall differentiates leaders.
Hardest fields: income_annually_in_british_pounds and spending_annually_in_british_pounds.

Interactive playground

The hosted playground is a static snapshot generated by python playground.py in the repo. Eight tabs: Rankings, Field Heatmap, Document Analysis, Error Breakdown, Deep Dive, Recommendations, Provider Analysis, and Project Evolution.

Two scoring views: Rankings and Provider Analysis use semantic F1 from score.py; heatmaps and error tabs use exact-match vs expected TSV. F1 can exceed exact-match when values are close but not identical — that is expected.

Open latest (2026-08-22T0830Z) Browse all snapshots

Playground version history

Each row is a frozen copy of which-models-extracted-playground.html from git, named with its commit time in UTC. Use this to see how the benchmark grew from 40 OpenRouter-only models (Feb 2026) to 111 scored runs across three providers (Aug 2026).

Snapshot (UTC)	Models	F1 runs	Highlights
2026-08-22T0830Z	111	111	Latest — dw-deepseek-ocr-2 OCR results	Open
2026-08-21T2219Z	111	111	dw-qwen3.8-27b, dw-muse-glimmer-30b	Open
2026-08-21T2145Z	109	109	Tab sync fix across all dashboards	Open
2026-08-09T0746Z	109	109	Aug 2026 benchmark refresh	Open
2026-07-29T1317Z	107	107	34 Doubleword models	Open
2026-04-11T1748Z	85	85	V7 Go provider added	Open
2026-03-07T2116Z	47	47	Provider Analysis tab	Open
2026-03-07T2104Z	47	47	F1 scoring & provider summary	Open
2026-02-28T1312Z	40	—	First leaderboard (exact-match tabs)	Open

View all 21 snapshots with search and filters →

Dataset & fields

Small export from the Kleister Charity dataset (dev-0): 11 UK charity financial PDFs (up to 200 pages each) with pre-extracted OCR text. Data is UK open data — see Kleister license.

charity_number charity_name report_date income_annually_in_british_pounds spending_annually_in_british_pounds address__postcode address__post_town address__street_line

Workflow (local)

Step-by-step video guide for Doubleword batch runs: extractor.py --all-doubleword.

# Smoke test
python llm_openrouter.py

# Extract (auto-detects OpenRouter / Doubleword / V7 from registries)
python extractor.py gemini-2.0-flash
python extractor.py --all-doubleword
python extractor.py --all-v7

# Score and regenerate playground
python score.py
python playground.py

Repo essentials

extractor.py — unified runner with checkpoint/resume for async backends
score.py — F1 / precision / recall leaderboard
playground.py — generates which-models-extracted-playground.html
config_models_*.py — OpenRouter, Doubleword, and V7 model registries
data/extraction_stats.csv — per-run stats (time, cost, field hits)
docs/v7-go.md — focused V7 Go setup guide

github.com/neomatrix369/playgroup_202602_docextract Back to home