1 Commits

Author SHA1 Message Date
cuihuan
2d57732d40 benchmarks: add reproducible coding capability-vs-cost chart (Part 3.3)
Plot every model with both a SWE-bench Verified score and a price on the
shared coding-agent session, so the value frontier is visible at a glance:
open-weight DeepSeek V4 Pro / Kimi K2.6 reach ~80% (level with Gemini 3.1
Pro) at ~1/10-1/25 the cost, while the 95% ceiling (Fable 5) costs ~46x the
cheapest 80% model.

- scripts/make_coding_chart.py renders assets/coding-value.png. The cost
  axis reuses the unit-tested cost_calc engine (no drift from the published
  tables); the capability axis is the dated swe_bench_verified figure. PIL is
  lazy-imported so the pure data layer stays importable on the CI runner.
- scripts/test_make_coding_chart.py unit-tests build_coding_data and
  value_headline (filtering, sort order, cost math, peer/value logic).
- Embed the chart + caption in BENCHMARKS.md and BENCHMARKS.zh-CN.md 3.3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 22:48:03 +08:00