Builds the execution half of the Geek Room "Build a Benchmarking Platform
for AI Agents, Not Just Models" theme: every submission now carries a real
execution trace (steps, outbound/tool-call count, latency p50/p95, tokens +
cost when the endpoint reports them, error/timeout counts) and an opt-in,
deterministically seeded fault-injection stress test (injected timeouts,
5xx responses, malformed answers at configurable rates) that reports
robustness: accuracy under faults vs. the clean run.
- benchmark-scoring.ts: instrumentRunner wraps any Runner with tracing;
endpointRunner gains an optional onOutcome hook + parses real
usage/cost_usd from the submitter's response body; mulberry32-seeded
fault injection with disjoint timeout/server_error/malformed buckets.
- benchmark-submissions.server.ts: wires tracing into both runner kinds,
runs a second faulted pass when requested, persists {clean trace,
robustness} + the fault-injection config on two new additive, nullable
columns (migrations/0012_benchmark_execution_trace.sql).
- API route + leaderboard feed: pass faultInjection through, return
trace/robustness/faultInjection on every submission.
- UI: /submit-model gets a "Stress-test with fault injection" checkbox and
Execution trace / Robustness under faults cards; /leaderboard's gold-suite
tab gets p50 ms / Outbound calls / Errors / Robustness columns for
community submissions.
Verified end to end: a real HTTP submission against a running dev server
(fault injection on, real trace + robustness persisted and read back from
the public leaderboard feed) and endpointRunner against a real local HTTP
agent (93 real round trips; real measured latency/tokens/cost; a 60%-fault
pass provably short-circuiting before the real network call) — see
hackathons/geekroom-2026/verify-execution-trace.mjs and
HACKATHON-SUBMISSION.md's "Real execution-trace + fault-injection run".
Re-recorded demo.mp4/poster.png (real Playwright capture, real ffmpeg
conversion) to show the new flow. npm test (138/1 skip/0 fail), npm run
gold (61/0), npm run typecheck (clean) all unchanged.
25 lines
1.4 KiB
SQL
25 lines
1.4 KiB
SQL
-- Execution-trace + fault-injection half of "Benchmark Submissions" (Geek
|
|
-- Room hackathon theme: "Build a Benchmarking Platform for AI Agents, Not
|
|
-- Just Models" asks for execution-step/tool-call/token/cost/time tracing
|
|
-- plus fault injection, in addition to the accuracy/calibration scoring
|
|
-- 0011_external_benchmarks.sql already shipped).
|
|
--
|
|
-- Both columns are additive and nullable: existing rows (scored before this
|
|
-- shipped) keep both null and render with no trace/robustness columns, the
|
|
-- same "Not yet measured" pattern the leaderboard already uses elsewhere —
|
|
-- no backfill needed, no existing query breaks.
|
|
--
|
|
-- execution_trace: {"clean": SubmissionTrace, "robustness": RobustnessReport | null}
|
|
-- (src/lib/jev/benchmark-scoring.ts) — the clean run's per-case trace
|
|
-- (steps, outbound calls, latency p50/p95, tokens, cost, errors) plus, when
|
|
-- fault injection was requested, the faulted run's own trace and the
|
|
-- faulted-vs-clean accuracy comparison.
|
|
-- fault_injection: the FaultInjectionConfig actually used (seed + rates),
|
|
-- when the caller opted in; null otherwise. Duplicated out of
|
|
-- execution_trace.robustness.config so the leaderboard query can select it
|
|
-- without parsing the larger trace blob.
|
|
|
|
alter table benchmark_submissions
|
|
add column if not exists execution_trace jsonb,
|
|
add column if not exists fault_injection jsonb;
|