neomatrix369 github.io

guide doubleword

Batch extraction with --all-doubleword

Run every Doubleword model in the registry against the Kleister Charity dev set (11 UK financial PDFs, 8 fields). The extractor submits async batch jobs, polls for completion, checkpoints progress, and writes one TSV per model under data/.

command: extractor.py provider: Doubleword Batch API ~38 models
QUICKSTART Playground snapshots Source repo

Demo recording

Terminal walkthrough of a full --all-doubleword run — run plan, batch submit, polling, and per-model status.

Screen recording · python extractor.py --all-doubleword

Before you run

1. Environment

2. Smoke test (optional)

python llm_openrouter.py   # confirms API keys & JSON parsing

The command

# All Doubleword registry models (default 1h completion window)
python extractor.py --all-doubleword

# Cheaper / slower — 24h batch window
python extractor.py --all-doubleword --completion-window 24h

# One model only
python extractor.py dw-qwen3.8-27b

What happens

Each model runs sequentially. Already-completed outputs are skipped (idempotent). Ctrl-C preserves checkpoints — re-run the same command to resume.

  1. 1 Run plan — lists Doubleword models to run, skip (done), or resume from checkpoint.
  2. 2 Batch submit — OCR text for 11 documents sent to the Doubleword Batch API per model.
  3. 3 Poll & checkpoint — async status updates; progress saved to data/.doubleword_checkpoints.json.
  4. 4 TSV output — structured fields written to data/playgroup_dev_extracted__doubleword__<model>.tsv.
  5. 5 Summary — completed / skipped / interrupted / failed counts per model.
Partial failures: rejected rows are logged and stored in data/.doubleword_failed_rows.json. Re-run with python extractor.py --retry-failed (or add model names) to merge fixes back into existing TSVs.

Output files

FilePurpose
data/playgroup_dev_extracted__doubleword__*.tsvPer-model extraction results
data/extraction_stats.csvTime, cost, field hits per run
data/.doubleword_checkpoints.jsonResume state for in-flight batches
data/.doubleword_failed_rows.jsonRow indices for --retry-failed
data/.doubleword_unavailable_models.jsonModels that failed on submit (skipped next run)

Related commands

Other providers

python extractor.py --all-openrouter   # OpenRouter models (sync)
python extractor.py --all-v7           # V7 Go agents
python extractor.py                    # everything in all registries

After extraction

python score.py          # F1 leaderboard across all TSVs
python playground.py     # regenerate which-models-extracted-playground.html

View results on this site via the playground archive or the project findings page.

Flags reference

FlagEffect
--all-doublewordRun only config_models_doubleword.py keys
--completion-window 1h|24hDoubleword batch SLA (default 1h)
--retry-failedRe-submit failed rows and merge into existing TSVs
<model>Run one or more named models (any provider)
Full README workflow Back to playground archive