- Rebrand every visitor-visible "OpenJev" string in src/routes, src/components,
src/lib/content (+ model chip labels in src/lib/jev/models.ts and the served
SKILL.md); env-var aliases (OPENJEV_*) and wire aliases untouched.
- Quickstart is now true for the product that ships: homepage, docs hub,
pricing, API reference and self-hosting show `npm run dev` on DEV_SITE_URL
(localhost:8080) with no Bearer header; the ots_live_ key step and header
only render when VITE_AUTH_ENABLED=true. Docker/serve keep :3000, labelled.
- Shareable playground: request JSON travels as base64url in the URL
(`/playground?r=…` server-visible, `/playground#r=…` client-only), loads on
mount, runs itself once, and a "Copy link" button sits next to Run.
- New GET /api/v1/og/decision?r=… renders a 1200x630 PNG of the top answers
and probabilities (hand-laid SVG + @resvg/resvg-wasm + embedded IBM Plex
Mono, OFL); shared links emit og:image/twitter:image pointing at it.
- Embedding engine re-probes a missing model dir on the next request, so
`npm run embed-download` after boot takes effect without a restart; README
says so.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- README Status/SDKs/Layout/Rename/Roadmap: the Python SDKs are live on
PyPI under their OpenJev-era names (openjev-client 0.7.1,
openjev-adapters 0.2.2, otsone 0.3.0) — `pip install openjev-client
otsone openjev-adapters` works now; umpirix-* names publishing soon.
- Say plainly that @umpirix/* and @openjev/* are not on npm; JS SDKs
install from source.
- sdk/python, adapters/python, otsone, sdk/js, sdk/ai-sdk READMEs: drop
"not published yet" for the live PyPI packages, replace @openjev/* npm
names with @umpirix/*, clone URLs -> caelum0x/umpirix.
- Verified against registry.npmjs.org / pypi.org and a fresh-venv
install + import on 2026-11-12.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Add GOVERNANCE.md (BDFL-style, referenced from CONTRIBUTING.md).
- Scribase family branding: "Umpirix — an open-source project by Scribase"
header line in README (linking scribase.com and scribase.com/decisions),
a "Run it on Scribase" section next to the self-host instructions in
README and SELF-HOSTING.md (self-hosting stays fully supported, nothing
gated), the same line appended to every package's npm/PyPI description
(root, sdk/js, sdk/js-lite, sdk/ai-sdk, sdk/scribase-decisions, skills,
adapters/python, sdk/python, otsone), and the site's per-page footer
brand line (10 routes) + roadmap prose, which still said the pre-rebrand
"OpenJev · Apache-2.0" - now "Umpirix · Apache-2.0 · scribase.com".
- Fix two real gaps in the MCP server found in a fresh clone: serverInfo
still said name "openjev" / title "OpenJev typed decisions"
(src/lib/jev/mcp.ts) and the stdio bridge's log prefix was still
"[openjev mcp]" (scripts/mcp-stdio.mjs). Both now say Umpirix. Verified
live over stdio (plain node; Docker/colima unavailable in this
environment) that `initialize` reports serverInfo.name "umpirix" /
title "Umpirix typed decisions", and `tools/list` returns the real tool
set: choice, score, noul, decide (no "cascade").
- npm run typecheck / npm test (138/138) / npm run gold (61/61) / npm run
lint (0 errors, pre-existing warnings only) all green after this change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Root LICENSE: replace the short-form notice with the full canonical
Apache-2.0 text (fixes GitHub's "Other" license detection); update the
OpenJev -> Umpirix copyright line in LICENSE/NOTICE, otsone/NOTICE and
sdk/python/LICENSE+NOTICE (missed by the rebrand pass).
- sdk/ai-sdk: was MIT with no LICENSE/NOTICE and no upstream fork trail
(unlike sdk/js, sdk/python, adapters/python, skills, which are real
TypeSafe AI SDK forks and correctly keep MIT per their NOTICE). It is
original Umpirix code, so switch it to Apache-2.0 with its own
LICENSE/NOTICE, matching the root.
- README Licensing section: list sdk/ai-sdk under Apache-2.0 and
sdk/scribase-decisions under MIT, and spell out why the TypeSafe-forked
directories stay MIT (their MIT terms require preserving the notice).
- Audited caelum0x/openjev-private (read-only clone): it's an older,
mostly-superseded snapshot of this repo. Migrated the 3 self-contained,
secret-free example scripts it had that this repo didn't (raw-HTTP, no
SDK dependency): content-moderation, lead-scoring, support-ticket-routing
(Python + Node.js, each with a sample JSONL), plus
scripts/reproduce-leaderboard.mjs (wired to a new `npm run
bench:reproduce`) for leaderboard reproducibility. Left behind in
private: src/lib/seo/retired.ts (a reversed decision - this repo now has
live /compare pages) and the hosted-SaaS pieces already superseded by
this repo's own, newer versions of the same files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Owner rename (GitHub repo already caelum0x/openjev -> caelum0x/umpirix,
origin updated): rebrands every user-facing surface while keeping full
backward compatibility.
- README.md: title, badges, clone/MCP commands, SDK table, model-family
table and a new "Renamed from OpenJev" section (mirroring the existing
"Renamed from opentypesafe" section) documenting every old->new mapping.
Never claims Umpirix is or was made by TypeSafe; "Jev-compatible" kept
only where it describes wire compatibility.
- CLI (scripts/cli.mjs): primary bin is now `umpirix`; `openjev` and
`opentypesafe` still work and print a one-line rename notice (detected
via the invoked bin's basename). package.json's `name` is now "umpirix"
and all three bin names are installed.
- Site/app title: the single SITE_NAME constant (src/lib/seo/site.ts) is
now "Umpirix", which cascades into every page's <title> and JSON-LD; the
nav sidebar brand text (app-shell.tsx) and OG site title (og/site.json)
updated too.
- SDK package names: @openjev/sdk -> @umpirix/sdk, @openjev/lite ->
@umpirix/lite, @openjev/ai-sdk -> @umpirix/ai-sdk (sdk/js, sdk/js-lite,
sdk/ai-sdk, plus the dependent sdk/scribase-decisions package and its
source re-export). All import sites, peer/dev dependency keys, jsr.json
and doc comments updated to match. Verified free on npm (404) as of
2026-11-11. sdk/js (193 tests) and sdk/js-lite (99 tests) both typecheck
and test clean after the rename; `npm pack --dry-run` confirmed real,
correctly-named tarballs for both.
- PyPI: openjev-client -> umpirix-client (sdk/python), openjev-adapters ->
umpirix-adapters (adapters/python); both plain names confirmed free on
PyPI (404) as of 2026-11-11. The importable module stays `openjev` /
`openjev_adapters` for compatibility. Found and fixed a real regression
from the distribution rename: both packages' `_version.py` read
`importlib.metadata.version()` by the old hardcoded distribution name,
which would have raised PackageNotFoundError post-rename — now tries the
new name first, falls back through the old names. `uv sync` + `uv build`
+ `uvx twine check` pass clean for both under the new names (681 + 452
tests pass respectively); `adapters/python/pyproject.toml`'s
`[tool.uv.sources]` key updated to match so the local editable override
still resolves. No PyPI token exists on this machine, so nothing was
published — exact commands are in PUBLISHING.md for whoever holds the
credentials.
- PUBLISHING.md: renamed package table, documented the verified-clean local
builds above, and the owner-gated npm (@umpirix org doesn't exist yet,
needs browser 2FA) / PyPI (token removed) bootstrap steps.
- Root `npm run typecheck` / `npm test` (138 pass) / `npm run gold` (61
pass) all clean after every change above.
- hackathons/geekroom-2026/demo-umpirix.mp4: the existing real
narrated Bifrost demo capture (119.8s, Deepgram aura-2-asteria-en
narration, -14.6 LUFS measured) copied to the name this task's submission
expects, plus its real raw captures (media/raw, media/terminal) and a
real SRT built from the narration's own word-level timestamps
(media/demo-umpirix.srt) for caption generation. Known gap, not hidden:
the capture's on-screen app UI still reads "OpenJev" (pre-rebrand,
pre-redeploy) — a fresh re-record after this rebrand ships to the running
app is a real follow-up, noted in plans/openjev.md.
Also pushed a companion one-line copy fix to caelum0x/scribase (commit
a8c2f72, not this repo): scribase.com/decisions now says "Built on
Umpirix" instead of "Built on OpenJev", deployed live (verified 200,
"Built on Umpirix" present).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Real Bifrost (github.com/maximhq/bifrost) integration for the Geek Room
hackathon's Bifrost-by-Maxim-AI track, replacing zero prior usage:
- BENCH_LLM_GATEWAY=bifrost routes every LLM-backed call (leaderboard's
llm row, the auto cascade, and the agent loop's judge stage) through
Bifrost's universal OpenAI-compatible endpoint instead of the provider
directly, via one choke point (callBifrostStructured in llm.server.ts).
- Per-call telemetry read straight from Bifrost's own response
(extra_fields), not recomputed: SystemOneBifrostTelemetry type, a new
Gateway badge/column on /leaderboard (LeaderboardRow.gateway).
- Real run against a local Bifrost v2.5.1 gateway + Anthropic: 93/93 gold
calls (accuracy 0.9794), fault-injected re-run (67/93 calls, robustness
0.621), and a real Bifrost-side fallback demo (5/5 calls served by the
fallback model, routing_info.is_fallback: true). Full numbers in
bifrost-run-results.json; reproduce with verify-bifrost.mjs. Committed
leaderboard.json's llm row now carries this real Bifrost-routed run.
- README gets a "How Bifrost powers Umpirix" section; HACKATHON-SUBMISSION.md
gets a dedicated Bifrost section + honest-limits bullet.
- Pitch deck: new slide "How Bifrost powers Umpirix", title renamed to
Umpirix — Benchmark Arena for AI Agents (the owner's submitted project
title), now 14 slides; re-verified against the actual committed PDF
(not just the Marp PNG preview) and fixed one real overflow found that
way. PPTX/PDF rebuilt.
- demo.mp4 rebuilt as a 42.9s real video (real Playwright screen recording
of the Gateway badge + code-rendered stat cards, no AI footage); prior
cuts kept as demo-v2-120s-narrated.mp4 / demo-v1-33s-silent.mp4.
Repo/package name intentionally left as openjev (not renamed) per the
owner's direction; Umpirix is the submission's project title.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Scribase-named re-export of @openjev/sdk's client, question helpers
(choice/score/noul) and error types (ScribaseDecisionsClient alias for
OpenJevClient), for the new Scribase Decisions product page/docs. No new
wire behavior: same Jev-compatible /v1/systemone contract. Builds clean
(tsdown) and `npm pack --dry-run` verified; not published (depends on the
still-unpublished @openjev/sdk — see plans/openjev.md for the exact publish
commands once the @scribase npm org exists and @openjev/sdk ships).
Also cross-links the Python SDK README to scribase.com/decisions without
republishing to PyPI.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The classify CLI's lexical engine had zero lexicon coverage for "feature"
and "praise" (only "bug" was covered) and none at all for generic
sentiment (positive/negative/neutral), so bare-label runs of the CLI's
own README example either tied on everything or confidently mislabeled a
praise/feature row as a bug whenever a bug-shaped word like "crash"
appeared. Add two small, additive synonym groups to the existing lexicon.
Measured on a new 65-row synthetic gold set (examples/classify-eval.jsonl,
run via `npm run classify-eval`, scripts/classify-eval.mjs): 32.3% -> 49.2%
accuracy counting ties as wrong, 67.7% -> 71.1% on rows the engine
actually committed to; the sentiment set alone went 5.0% -> 35.0%. The
product-wide gold regression suite (`npm run gold`) still passes 61/61 —
every change is a new synonym entry, not an edit to an existing one.
README now documents the measured numbers, the --threshold tradeoff
(0.55 raises decided-row accuracy to 83.8% at the cost of more abstains),
and the real remaining limit: a bag-of-words engine can't fully resolve
a praise sentence that mentions "crash" from an actual bug report.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- examples/triage-tickets.py no longer passes a placeholder key or says the SDK requires one
- sdk/js-lite README imported from @caelum/typesafe, which does not resolve; use @openjev/lite
- POST /v1/mcp answers a non-JSON body with a JSON-RPC -32700 error instead of the REST envelope
New subcommand that imports the engine straight from src/ with Node type
stripping: no server, no key, no model download. Reads one text per line on
stdin, or a .txt, .csv (--column) or .jsonl (--field) file; prints a table or
JSON lines, or writes .csv / .jsonl with --out. Each row gets a label, a
probability and the engine that answered (embedding when the gte-small model
is installed, else lexical). --threshold blanks low-probability rows for
review, and near-ties are always left blank. Labels accept an optional
description (name:description).
Exit codes: 2 for usage errors, 1 for missing files, columns or bad JSON.
POST /v1/mcp speaks the Model Context Protocol over stateless Streamable
HTTP (initialize, tools/list, tools/call, ping). Tool calls go through the
same guard, quota checks and executeSystemOne path as POST /v1/systemone.
`openjev mcp` is a stdio bridge that forwards to a running server at
OPENJEV_BASE_URL.
Adds the /integrations/mcp page with Claude Code and Cursor config, a README
section, and moves the roadmap item to shipped.
Found by running typesafe-sdk 0.7.5 and @typesafe-ai/sdk 0.7.0 against a
local server and fixed:
- zstd (and gzip/deflate/br) request bodies are decoded; the SDKs compress
bodies of 1 KiB or more, which used to fail with a 500
- GET /v1/models also returns models: [{ name, description, release_date }]
- every systemone, batch and models response carries x-typesafe-request-id
- a base URL ending in /v1 (or /api/v1) resolves to the same handlers
- request errors carry a detail list; a non-JSON body is a 400, not a 500
- multipart (image asset) requests get a clear 415
Adds examples/official-sdk.py and .mjs, a /docs/typesafe-sdk page, a README
block and a COMPARISON.md section, limited to what was observed to pass.
A self-hosted server needs no key, but all three SDKs threw before the first
request. The key is now optional: without one no Authorization header is sent.
A key that is present is validated as before.
Release prep: openjev-client 0.7.2, @openjev/sdk 0.6.1, @openjev/lite 0.2.1.
Adoption-ranked work from plans/research/ai-and-active-next-2026-11-11.md
section 2 (openjev_next list), skipping items that need training/eval or a
live deploy this pass:
- docs/COMPARISON.md: honest OpenJev vs TypeSafe Jev vs Kev vs OpenAI
Decisions API table, sourced from already-verified facts in
plans/openjev.md's 2026-11-06 research entry, with our own real
otsone-mini/otsone-base/system-one numbers and their caveats stated
(item #4, "public comparison writeup").
- examples/triage-tickets.py: a real runnable Python SDK example (Choice +
Score + Noul ticket triage), verified end to end against a mocked
TypeSafe2/httpx2 transport; examples/README.md indexes both examples.
Fixes a real gap found while writing it: TypeSafeClient() always
requires a non-empty api_key even against a keyless self-hosted server
(server-side "unknown keys pass through") -- worked around in the
example with a documented placeholder, not changed in the SDK itself.
- sdk/js/package.json: @openjev/sdk had no `keywords` field at all (its
sibling @openjev/lite and both Python packages do) -- real npm-search
discoverability gap on the flagship package, fixed; publint still
passes clean.
Not done this pass (needs model eval/training or a live deploy, out of
scope under the no-training/no-deploy constraints): Kev-style versioned
eval-suite naming (#1), growing gold.json with a real holdout split (#2),
a hosted try-it-now demo widget (#3).
ModernBERT-base's GeGLU-gated MLP lowers to an ONNX Split node using the
opset-18-only `num_outputs` attribute; torch's dynamo exporter's opset-17
downgrade pass can't rewrite it, so onnxruntime rejected the exported graph
as invalid during temperature calibration. otsone-mini (plain-GELU MiniLM)
never hits this Split pattern, so the bug was invisible until otsone-base
was actually trained for the first time. Fix: export at opset 18 directly.
Ran the fixed pipeline end to end (otsone distill --backbone otsone-base):
a full 14-epoch run stalled twice under ~500-900 load average from other
concurrent agents on this shared Mac (one killed after 23min/0 epochs, one
suspended by the OS post-training at 35min before export), so this records
a labeled reduced 3-epoch run that completed clean in <6min once load eased.
Real held-out gold-set result: 12/14 cases, 14/18 checks, accuracy=0.7778,
ece=0.1596 - slightly behind otsone-mini-run1-nogold's 14-epoch 0.8333/0.1255,
honestly attributable to the 4.7x-shorter training budget on a 6.6x-larger
backbone, not the architecture. Full writeup + model_card/rl_agent_config
(no weights) in otsone/docs/results/otsone-base-reduced-3ep/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Independent judge re-audit re-rendered every slide to PNG and the actual
committed PDF (not just the Marp preview) and found the prior pass's
"zero overflow, verified visually" claim was still false for 2 of 11
slides: the architecture slide was missing its last 3 bullets (SSRF
guard, Real UI, Real API) and the proof slide was missing its entire
execution-trace + fault-injection real-numbers paragraph, both silently
clipped below the 1280x720 frame.
Fixed by splitting the two overloaded slides into four (13 slides total
now, was 11) instead of shrinking fonts further, renumbering every
internal "see slide N" cross-reference including the judging-criteria
map, and merging/trimming one more bullet that was still touching the
bottom edge after the split. Rebuilt PITCH-DECK.pptx/.pdf from the fixed
source and re-verified by rendering the actual committed PDF to PNG and
inspecting pixel crops of every slide that had sat near the frame edge.
npm run typecheck / npm test (138/1 skip/0 fail) / npm run gold (61/0)
unaffected, re-run clean.
The release table and PyPI setup note claimed openjev-adapters was
"already a real PyPI project at 0.2.1." Verified live: pypi.org/pypi/
openjev-adapters/json returns 404 (checked hyphen/underscore/no-separator
variants), and the repo has zero git tags and zero GitHub Releases ever
pushed — the 0.2.1 in the table is only the local pyproject.toml version,
not a published state. Both openjev-adapters and openjev-client are
unclaimed on PyPI and need the same pending-publisher bootstrap as npm,
not an easier path for adapters.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
otsone shipped 3 real changes since 0.2.0 (eval-external feature, CLI/find_gold bug
fixes, Content-Length DoS cap in the laya sidecar) with no version bump and no
changelog entry - bumped to 0.3.0 and documented them in docs/changelog.md,
matching the format the other packages' release tooling already expects.
Added PUBLISHING.md: exact one-command release recipe for all 5 packages (tag-push
for the 3 with CI, direct npm/uv publish for sdk/js-lite and otsone which have none
by design), plus the real owner-gated registry/account setup each needs before the
first tag. Verified every package actually builds and packages clean right now:
npm pack --dry-run for sdk/js and sdk/js-lite, uv build + twine check (PASSED) for
sdk/python, adapters/python, and otsone.
Also fixed a real latent bug found while verifying sdk/js-lite: its
package-lock.json root metadata was still "@caelum/typesafe"@0.1.0, never updated
through the OpenJev rebrand (package.json has been @openjev/lite since the initial
commit) - npm install resynced it for real.
Rendered all 11 PITCH-DECK slides to PNG and found 8 of 11 silently
truncating real content off the bottom of the frame (no visual cue,
content just cut) -- fixed with a tighter style: block and by trimming
one slide's duplicate repo-link paragraph, not by dropping any claim.
Added a PDF export alongside the existing PPTX.
Upgraded demo.mp4 from a 33s silent capture to a 120s narrated cut built
entirely from real material: the same real Playwright-captured UI frames
(re-timed, never fabricated), a real Deepgram Aura-2 narration track with
nova-3 round-trip-verified word timing driving burned-in captions, and a
few plain typographic title/outro cards (build-demo-video.py). Mastered
to -14.6 LUFS / -2.3 dBTP. Original capture kept as
demo-v1-33s-silent.mp4 for provenance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
npm ci --omit=dev alone never omitted onnxruntime-node because it's an
optionalDependency, so every build (embed on or off) shipped the ~69MB
Linux-pruned native binary even when the model weights (and therefore
the embed engine) aren't present. Pass --omit=optional too when the
build-time OPENJEV_EMBED arg is off.
Also regenerates package-lock.json: it had drifted out of sync with
package.json (missing nested @tailwindcss/oxide-wasm32-wasi deps,
stale onnxruntime-node placement), which broke `npm ci` outright on a
clean checkout independent of this fix -- found while verifying this
change actually builds.
Verified locally: `docker build --build-arg OPENJEV_EMBED=0` now
installs node_modules at 263MB vs 410MB before (-147MB, onnxruntime-node
plus other optional-only deps not needed at runtime); container boots
and GET /api/v1/health returns 200 {"status":"ok"}. `OPENJEV_EMBED=1`
build and boot verified unaffected (same health response, onnxruntime
loads as before). Box cutover is a separate owner/infra step, not done
here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
HACKATHON-SUBMISSION.md and PITCH-DECK.md both referenced only the
2026-11-07 commit (c70f3d0) that built the original Benchmark Submissions
feature; add the 2026-11-10 commit (cececb2) that added execution tracing
+ fault injection, and rebuild PITCH-DECK.pptx from the updated source.
Builds the execution half of the Geek Room "Build a Benchmarking Platform
for AI Agents, Not Just Models" theme: every submission now carries a real
execution trace (steps, outbound/tool-call count, latency p50/p95, tokens +
cost when the endpoint reports them, error/timeout counts) and an opt-in,
deterministically seeded fault-injection stress test (injected timeouts,
5xx responses, malformed answers at configurable rates) that reports
robustness: accuracy under faults vs. the clean run.
- benchmark-scoring.ts: instrumentRunner wraps any Runner with tracing;
endpointRunner gains an optional onOutcome hook + parses real
usage/cost_usd from the submitter's response body; mulberry32-seeded
fault injection with disjoint timeout/server_error/malformed buckets.
- benchmark-submissions.server.ts: wires tracing into both runner kinds,
runs a second faulted pass when requested, persists {clean trace,
robustness} + the fault-injection config on two new additive, nullable
columns (migrations/0012_benchmark_execution_trace.sql).
- API route + leaderboard feed: pass faultInjection through, return
trace/robustness/faultInjection on every submission.
- UI: /submit-model gets a "Stress-test with fault injection" checkbox and
Execution trace / Robustness under faults cards; /leaderboard's gold-suite
tab gets p50 ms / Outbound calls / Errors / Robustness columns for
community submissions.
Verified end to end: a real HTTP submission against a running dev server
(fault injection on, real trace + robustness persisted and read back from
the public leaderboard feed) and endpointRunner against a real local HTTP
agent (93 real round trips; real measured latency/tokens/cost; a 60%-fault
pass provably short-circuiting before the real network call) — see
hackathons/geekroom-2026/verify-execution-trace.mjs and
HACKATHON-SUBMISSION.md's "Real execution-trace + fault-injection run".
Re-recorded demo.mp4/poster.png (real Playwright capture, real ffmpeg
conversion) to show the new flow. npm test (138/1 skip/0 fail), npm run
gold (61/0), npm run typecheck (clean) all unchanged.
Corrects outbound-guard.server.test.ts claim from 6 to the verified real
count (5 tests), and adds an explicit "built 2026-11-07, inside the
official Nov 1-15 build window" note next to the "not a repo created for
the submission" line in both the pitch deck and the submission writeup,
pre-empting a literal Rule-1 misread flagged by an external judge pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Re-verified the real hackculture.io page directly (no Discord gate needed,
a stale assumption from an earlier pass): official theme text, Bifrost/
Maxim AI track (cash prize, not Tavily credits), and the real submission
deadline (Nov 15 2026 11:59 PM IST = 18:29 UTC). Updated
HACKATHON-SUBMISSION.md with the exact official theme wording and an
honest gap map against its fuller spec (execution-trace metrics and
fault-injection scenarios are roadmap, not built). Fixed a real "10 slides"
vs. actual 11-slide PPTX export drift in both PITCH-DECK.md and
HACKATHON-SUBMISSION.md, and rebuilt PITCH-DECK.pptx from the corrected
source (verified: 11 slide parts, valid zip, 3.1MB). Added a README section
stating the hackathon fit so it isn't only documented in the hackathon
subfolder.
PITCH-DECK.md already existed but was plain prose with no Marp front
matter and no documented export path. Add marp/theme/paginate front
matter, a required "how AI is used" slide, a judging-criteria map, a
team slide, and the one-command PPTX/PDF export (verified working via
npx @marp-team/marp-cli). Commit the built PITCH-DECK.pptx (11 slides)
so a judge can open the file directly without running the command.
Also fix HACKATHON-SUBMISSION.md's stale "source of truth" link, which
still pointed at the dev-tools/openjev clone removed on 2026-11-09.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
importlib.metadata.version("openjev") raises PackageNotFoundError now that the
real distribution name is "openjev-client" (renamed in a0c6263). The test
imported the stdlib helper directly instead of going through the SDK's own
__version__ (openjev._version, which already has the openjev-client/openjev
fallback), so any fresh install installed under the real "openjev-client"
distribution name fails this test with zero relation to the behavior being
tested (the user-agent header format itself is unaffected -- it is still
"openjev/<version>", matching SDK_NAME).
Verified live: before the fix, `pytest tests/test_clients.py` failed 2/111
(sync + async) with PackageNotFoundError; after, 111/111 pass. Full suite:
681 passed, 54 skipped (unchanged skip count, no regressions). ruff and
pyrefly both clean on the touched file.
do_POST trusted the client-supplied Content-Length header with no upper
bound and no sign check, then called self.rfile.read(length) directly --
the exact DoS class already fixed in the Rust engine's HTTP server
(a6c662f, engine/rust/src/http.rs MAX_BODY_BYTES) and never ported to
this Python sidecar. A single unauthenticated request claiming a huge
Content-Length (observed: 5GB, 0 bytes actually sent) made the handler
thread block/allocate against that claimed size; a negative value
(e.g. "-1") would have made read(-1) block waiting for EOF instead of
failing fast.
Add the same 1MB cap (matching the Rust engine's MAX_BODY_BYTES) and
reject negative/non-numeric Content-Length with 400, oversized with 413,
before touching rfile. Verified live against a running instance (stubbed
engine, real HTTP round trips): 5GB claim -> 413 immediately, "-1" -> 400
immediately, a normal small request -> 200 as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The root .gitignore's bare `dist` pattern matches at any depth, which
was also silently excluding sdk/js/test/dist/ -- a real, hand-written
test suite (cjs.dist.ts/esm.dist.ts/types.dist-d.ts/helpers.ts) that
verifies the published package's actual compiled exports, not build
output. `git log --all` shows zero commits ever touched this path: it
has been untracked since the repo's initial commit, so a fresh clone
(CI, another machine, a contributor) silently never had these tests.
Un-ignore sdk/js/test/dist/ explicitly while leaving the real build
output (sdk/js/dist/) ignored. Also fixed the EXPECTED_VALUE_EXPORTS
list in helpers.ts, which had drifted from src/index.ts's deprecated
OpenTypeSafeClient/OpenTypeSafeError aliases and was failing both dist
tests the moment they actually ran.
This also resolves the "biome: command not found" report from the
2026-11-09 npm publish --dry-run attempt: @biomejs/biome was already a
correct devDependency, the real missing step was `npm install` (never
run that session). Verified live: fresh npm install + npm run check
(lint/typecheck/test:coverage/build/test:dist/check:package) + npm
publish --dry-run all pass clean end to end now.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The plain `openjev` PyPI name belongs to an unrelated project, and the
two fallback names from the earlier plan (`openjev-sdk`, `pyjev`) are
also taken (verified live, both 200 not 404). `openjev-client` is
confirmed free. Renamed the distribution in pyproject.toml, fixed a
real runtime bug the rename exposed (_version.py read
importlib.metadata.version("openjev"), which now raises
PackageNotFoundError since the installed distribution is
"openjev-client"; added a fallback), refreshed uv.lock, rebuilt the
wheel to confirm it produces openjev_client-0.7.1-*.whl with correct
metadata, and re-ran the existing test suite clean (8 passed/2
skipped on the touched subset). Import path stays `openjev`. Updated
both READMEs with the real registry names (npm: @openjev/sdk,
@openjev/lite, both verified unclaimed; PyPI: openjev-client) and the
real publish blocker (npm whoami -> 401, owner auth needed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds 4 unambiguous new labeled cases (shipping-delay-clear,
double-charge-billing, newsletter-unsubscribe, bug-intermittent-workaround)
covering paraphrases of existing question types. Re-ran `otsone eval`
live against both checkpoints: otsone-mini (never trained on gold) scores
12/14 (0.8889 acc, ece 0.1253, up from 8/10/0.857/0.16) with all 4 new
cases passing; otsone-mini-v2 (gold folded into training on the old 10)
also scores 14/14 on the new cases, a genuine held-out signal since those
4 examples did not exist when v2 was trained. This is the real first step
toward a genuine train/holdout split identified as the actual blocker on
every eval number this pipeline produces, independent of GPU/EVREN access.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Strengthens the Benchmark Submissions code-forward evidence with real
artifacts instead of pitch-deck claims alone, per owner direction:
- hackathons/geekroom-2026/demo.mp4 (+poster.png, raw demo-raw.webm) — a
real Playwright-driven Chromium screen recording against an actual
running `npm run dev`: create a real API key, submit two real answer
sources through the live /submit-model form, scroll to the real
/leaderboard rows showing the computed scores. No staged footage.
- A second, genuinely different example submission: "uniform-baseline-v1"
(generate-baseline-submission.ts), a content-blind uninformed baseline
that never imports the scoring engine and never reads gold-suite expected
answers (no test-set leakage, not a disguised system-one replay). Scored
35.0% accuracy / 13.1% strict pass / 0.027 ECE vs system-one-replay's
85.6% / 100.0% / 0.096 — both now real "Community submitted" rows on
/leaderboard, proving the platform scores more than one self-referential
submission.
- generate-system-one-replay.ts saves the previously ad-hoc 93-case
system-one replay payload as a reusable fixture for the demo and judges.
No changes to benchmark-scoring.ts / benchmark-submissions.server.ts /
outbound-guard.server.ts. Verified: npm run typecheck clean, npm test
138/138 passing (1 pre-existing skip, unchanged), npm run gold 61/61
unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Judge-readable Markdown slide deck covering the problem, what Benchmark
Submissions does, real architecture, verified proof numbers, honest
scope/limitations, and next steps — closes the last open item from the
HACKATHON-SUBMISSION.md writeup before the Nov 15 deadline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Geek Room hackathon ("a benchmarking platform for AI agents"): a real,
platform-integrated feature replacing the earlier, too-small otsone
eval-external CLI addition. A third party can now submit their own
typed-decision model (live HTTP endpoint or precomputed answers JSON)
through the real console UI or API, get scored against the same 93-case
gold suite by the same calibration engine every OpenJev model uses
(no second, divergent scoring implementation), and show up ranked
alongside OpenJev's own models on the real /leaderboard page.
- migrations/0011_external_benchmarks.sql: benchmark_submissions +
benchmark_results tables.
- src/lib/jev/benchmark-scoring.ts: pure scoring logic (answers-payload
parsing/validation, HTTP + answers-replay runners, per-question-type
breakdown) reusing evalGoldWith/buildReliability directly.
- src/lib/jev/benchmark-submissions.server.ts: DB-backed wrapper
(create/get/list/public-leaderboard-feed), same console auth +
checkQuota pattern as every other /api/v1/* route.
- src/lib/jev/outbound-guard.server.ts: SSRF guard extracted from
webhooks.server.ts (one implementation, not two) — validates +
DNS-resolves submitted endpoint URLs before any outbound call.
- src/routes/api/v1/benchmark-submissions.ts (+ .leaderboard.ts): real
API routes, auth + quota enforced.
- src/routes/submit-model.tsx: real submission form + history UI.
- src/routes/leaderboard.tsx + src/lib/jev/leaderboard.ts: external
submissions render as "Community submitted" rows in the same gold-suite
table, fetched client-side from the public leaderboard feed.
- Tests: benchmark-scoring.test.ts (9, incl. an end-to-end check that
replaying system-one's own answers through this path reproduces
evalGold() byte-for-byte) + outbound-guard.server.test.ts (6).
Verified end to end locally: npm run typecheck clean, npm test 138/138
(1 pre-existing skip), npm run gold unchanged (61/61), and a real
browser session (gstack browse) submitting a 93-case answers payload
through the actual /submit-model form — result card showed real
computed scores (accuracy 85.6%, ECE 0.096, matching evalGold() exactly)
and the submission appeared live on /leaderboard's gold-suite tab.
distill.py's find_gold() (eval_gold.py's was already fixed) assumed
otsone/ sits two directories above a sibling openjev/ checkout; the
real layout has otsone/ as a direct child of openjev/, so gold.json
lives one level higher with no extra "openjev" segment. Add that
candidate so `otsone distill` auto-folds engine/gold.json without
needing an explicit --gold flag.
torch>=2.2's dynamo-based ONNX exporter (torch.onnx.export) requires
onnxscript; without it export crashes with ModuleNotFoundError after
a full training run has already completed. Add onnxscript>=0.1 to the
onnx/train extras.
Verified live: `otsone distill --backbone otsone-mini --out ...` (real
CLI, uv venv, no workaround) -> model_card.json shows
gold_targets_folded=14 (886 examples = 872 synthetic + 14 gold,
matching 2026-11-06 session numbers), onnxscript installed cleanly via
`uv pip install -e ".[train]"`, ONNX export succeeded. `otsone eval
--model ...` -> 10/10 cases, 14/14 checks, accuracy=1.0, ece=0.0005.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mirrors dev-tools/openjev commit c1bd918. New `otsone eval-external` CLI
command and laya_mlx.eval_external module let a third party point OpenJev's
existing gold-case calibration report (accuracy, ECE/adaptive-ECE/MCE/Brier/
NLL, per-type reliability) at their OWN typed-decision model or API over
HTTP, or replay precomputed predictions from an answers file -- with zero
changes to the underlying eval_gold/calibration scoring math. Built for the
Geek Room Global AI Hackathon's "benchmarking platform for AI agents" theme.
Also backports two small, already-verified fixes needed for this command to
work out of the box in this mirror: the early-dispatch fix in cli.py so
REMAINDER-based subcommands (eval/distill/eval-external) correctly receive a
leading `--flag`, and the missing repo.parent/engine/gold.json candidate in
eval_gold.find_gold's auto-detection (this mirror's engine/gold.json sits one
level above otsone/, same real layout as the primary dev-tools/openjev repo).
22 new tests in tests/test_eval_external.py, all passing; zero regressions
(49 relevant tests across test_eval_external.py/test_otsone.py/
test_calibration.py pass against this mirror's own otsone copy).
New `otsone eval-external` CLI command and laya_mlx.eval_external module let
any third party point OpenJev's existing gold-case calibration report
(accuracy, ECE/adaptive-ECE/MCE/Brier/NLL, per-type reliability) at their OWN
typed-decision model or API, not just OpenJev's own ONNX checkpoints -- the
real gap between an internal eval script and a benchmarking platform for AI
agents (targets the Geek Room Global AI Hackathon's published theme of the
same name). HttpEngine posts {state, questions} to a live endpoint;
AnswersFileEngine replays precomputed predictions offline. Zero changes to
the underlying eval_gold/calibration scoring math. 22 new tests, all passing;
zero regressions in the existing otsone suite (216 tests).
The native Rust HTTP server trusted the client-supplied Content-Length
header with no upper bound, looping to read the full claimed body size
with no cap. A tiny request claiming a huge Content-Length let a single
unauthenticated connection force unbounded memory allocation (DoS).
Reject any Content-Length over the existing 1MB request-size guard
before entering the body-accumulation loop.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
isValidWebhookUrl only checked for a well-formed http(s) URL with a host,
so a console user (or ots_live_ API key) could register/test a webhook
pointed at loopback, RFC1918/link-local ranges, or the cloud metadata
address (169.254.169.254), making the server fetch it on their behalf.
Reject known-bad literals at validation time, and add a DNS-resolution
guard (assertPublicAddress) before every delivery so a hostname that
resolves to a private address — including internal docker-compose
service names like laya-onnx/db — is blocked too.
Mirrors commit 77f4cd3 in openjev-public to keep source and mirror in sync.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The native Rust HTTP server trusted the client-supplied Content-Length
header with no upper bound, looping to read the full claimed body size
with no cap. A tiny request claiming a huge Content-Length let a single
unauthenticated connection force unbounded memory allocation (DoS).
Reject any Content-Length over the existing 1MB request-size guard
before entering the body-accumulation loop.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
clientIp() trusted CF-Connecting-IP / X-Real-IP / X-Forwarded-For
unconditionally, but the stock docker-compose self-host path exposes the
Node process directly with no reverse proxy overwriting those headers.
Any caller could rotate X-Forwarded-For per request to mint a fresh
per-IP bucket and fully bypass the demo rate limiter (and the hosted
anonymous systemone/batch limiter).
Gate header trust behind a new trustProxyHeaders() check: trusted when
OPENJEV_HOSTED=1 (hosted deployment sits behind Cloudflare) or when a
self-hoster opts in explicitly via OPENJEV_TRUST_PROXY_HEADERS=1 after
confirming their own proxy sets these headers. Untrusted callers share
one "unknown" bucket, so the global window still backstops traffic.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
isValidWebhookUrl only checked for a well-formed http(s) URL with a host,
so a console user (or ots_live_ API key) could register/test a webhook
pointed at loopback, RFC1918/link-local ranges, or the cloud metadata
address (169.254.169.254), making the server fetch it on their behalf.
Reject known-bad literals at validation time, and add a DNS-resolution
guard (assertPublicAddress) before every delivery so a hostname that
resolves to a private address — including internal docker-compose
service names like laya-onnx/db — is blocked too.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
clientIp() trusted CF-Connecting-IP / X-Real-IP / X-Forwarded-For
unconditionally, but the stock docker-compose self-host path exposes the
Node process directly with no reverse proxy overwriting those headers.
Any caller could rotate X-Forwarded-For per request to mint a fresh
per-IP bucket and fully bypass the demo rate limiter (and the hosted
anonymous systemone/batch limiter).
Gate header trust behind a new trustProxyHeaders() check: trusted when
OPENJEV_HOSTED=1 (hosted deployment sits behind Cloudflare) or when a
self-hoster opts in explicitly via OPENJEV_TRUST_PROXY_HEADERS=1 after
confirming their own proxy sets these headers. Untrusted callers share
one "unknown" bucket, so the global window still backstops traffic.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The commented-out durable-Postgres block that SELF-HOSTING.md tells operators
to uncomment shipped a default openjev/openjev/openjev user/password/db and
published 5432:5432 to the host. An operator who followed the docs verbatim
got Postgres (API key hashes, webhook secrets, session data) reachable by
anyone on the host's network with a trivially guessable credential.
Now requires POSTGRES_PASSWORD to be set (fails fast via compose's :? syntax
if missing) and no longer publishes the Postgres port to the host — db stays
reachable only on the internal compose network.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The commented-out durable-Postgres block that SELF-HOSTING.md tells operators
to uncomment shipped a default openjev/openjev/openjev user/password/db and
published 5432:5432 to the host. An operator who followed the docs verbatim
got Postgres (API key hashes, webhook secrets, session data) reachable by
anyone on the host's network with a trivially guessable credential.
Now requires POSTGRES_PASSWORD to be set (fails fast via compose's :? syntax
if missing) and no longer publishes the Postgres port to the host — db stays
reachable only on the internal compose network.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Captures two real, published Jev-like training recipes (Together AI hosted
fine-tune ~$17, Unsloth local/Colab LoRA FastDecisionModel) as reference
material for anyone extending otsone-1's own training pipeline, plus a
comparison of how each differs from otsone's from-scratch small-encoder
approach.