wholeboard.bench¶
The benchmark: reference prompts run end to end, to see whether a change to a model, a prompt or a block helps.
python -m wholeboard.bench # every prompt in bench/prompts.json, with the AI in Settings
python -m wholeboard.bench --only blink555,can_node
python -m wholeboard.bench --worker openai:qwen3-coder --base-url http://localhost:8080/v1 --expert claude-opus-5-5
python -m wholeboard.bench --table # only rewrite bench/RESULTS.md from bench/results/
Each prompt runs unattended (the designer is told not to ask questions, and any question it asks anyway gets
"use your judgement"). For each run it records: completed or not, AI rebuilds, ERC errors, DRC errors, unrouted
connections, simulation failures, the board cost from the quote, tokens and API cost, and the time taken. A run
is saved to bench/results/