umpirix/migrations/0012_benchmark_execution_trace.sql
caelum0x cececb2772 feat(benchmark): add execution tracing + fault injection to Benchmark Submissions
Builds the execution half of the Geek Room "Build a Benchmarking Platform
for AI Agents, Not Just Models" theme: every submission now carries a real
execution trace (steps, outbound/tool-call count, latency p50/p95, tokens +
cost when the endpoint reports them, error/timeout counts) and an opt-in,
deterministically seeded fault-injection stress test (injected timeouts,
5xx responses, malformed answers at configurable rates) that reports
robustness: accuracy under faults vs. the clean run.

- benchmark-scoring.ts: instrumentRunner wraps any Runner with tracing;
  endpointRunner gains an optional onOutcome hook + parses real
  usage/cost_usd from the submitter's response body; mulberry32-seeded
  fault injection with disjoint timeout/server_error/malformed buckets.
- benchmark-submissions.server.ts: wires tracing into both runner kinds,
  runs a second faulted pass when requested, persists {clean trace,
  robustness} + the fault-injection config on two new additive, nullable
  columns (migrations/0012_benchmark_execution_trace.sql).
- API route + leaderboard feed: pass faultInjection through, return
  trace/robustness/faultInjection on every submission.
- UI: /submit-model gets a "Stress-test with fault injection" checkbox and
  Execution trace / Robustness under faults cards; /leaderboard's gold-suite
  tab gets p50 ms / Outbound calls / Errors / Robustness columns for
  community submissions.

Verified end to end: a real HTTP submission against a running dev server
(fault injection on, real trace + robustness persisted and read back from
the public leaderboard feed) and endpointRunner against a real local HTTP
agent (93 real round trips; real measured latency/tokens/cost; a 60%-fault
pass provably short-circuiting before the real network call) — see
hackathons/geekroom-2026/verify-execution-trace.mjs and
HACKATHON-SUBMISSION.md's "Real execution-trace + fault-injection run".
Re-recorded demo.mp4/poster.png (real Playwright capture, real ffmpeg
conversion) to show the new flow. npm test (138/1 skip/0 fail), npm run
gold (61/0), npm run typecheck (clean) all unchanged.
2026-11-10 18:11:57 +03:00

25 lines
1.4 KiB
SQL

-- Execution-trace + fault-injection half of "Benchmark Submissions" (Geek
-- Room hackathon theme: "Build a Benchmarking Platform for AI Agents, Not
-- Just Models" asks for execution-step/tool-call/token/cost/time tracing
-- plus fault injection, in addition to the accuracy/calibration scoring
-- 0011_external_benchmarks.sql already shipped).
--
-- Both columns are additive and nullable: existing rows (scored before this
-- shipped) keep both null and render with no trace/robustness columns, the
-- same "Not yet measured" pattern the leaderboard already uses elsewhere —
-- no backfill needed, no existing query breaks.
--
-- execution_trace: {"clean": SubmissionTrace, "robustness": RobustnessReport | null}
-- (src/lib/jev/benchmark-scoring.ts) — the clean run's per-case trace
-- (steps, outbound calls, latency p50/p95, tokens, cost, errors) plus, when
-- fault injection was requested, the faulted run's own trace and the
-- faulted-vs-clean accuracy comparison.
-- fault_injection: the FaultInjectionConfig actually used (seed + rates),
-- when the caller opted in; null otherwise. Duplicated out of
-- execution_trace.robustness.config so the leaderboard query can select it
-- without parsing the larger trace blob.
alter table benchmark_submissions
add column if not exists execution_trace jsonb,
add column if not exists fault_injection jsonb;