umpirix/docs/COMPARISON.md
caelum0x 26becee66a feat(api): official TypeSafe SDKs work unchanged against OpenJev
Found by running typesafe-sdk 0.7.5 and @typesafe-ai/sdk 0.7.0 against a
local server and fixed:
- zstd (and gzip/deflate/br) request bodies are decoded; the SDKs compress
  bodies of 1 KiB or more, which used to fail with a 500
- GET /v1/models also returns models: [{ name, description, release_date }]
- every systemone, batch and models response carries x-typesafe-request-id
- a base URL ending in /v1 (or /api/v1) resolves to the same handlers
- request errors carry a detail list; a non-JSON body is a 400, not a 500
- multipart (image asset) requests get a clear 415

Adds examples/official-sdk.py and .mjs, a /docs/typesafe-sdk page, a README
block and a COMPARISON.md section, limited to what was observed to pass.
2026-11-11 05:41:06 +03:00

7.3 KiB
Raw Permalink Blame History

OpenJev vs. the rest of the decision-model category

The "decision layer" category (a typed classifier that stands in for a full LLM call on routing/triage-shaped questions) is new: TypeSafe released the hosted Jev API on 2026-09-15, the open-weights Laya model followed three days later, and OpenAI announced a Decisions API (model name "Luna") at the same DevDay, moving to public beta on 2026-10-06. This page compares what is actually known and measured about each, including our own numbers, with sources — nothing here is estimated or rounded in our favor.

The four options

Access License Pricing What it is
TypeSafe Jev Hosted API (api.typesafe.ai) Proprietary Paid The original hosted System One /v1/systemone service this whole category is wire-compatible with.
OpenAI Decisions API Limited preview → public beta 2026-10-06 Proprietary OpenAI pricing GPT-6 "Luna"-backed; not yet broadly available or independently benchmarked as of this writing.
Kev (jaredpalmer/kev) Self-hosted, open weights Apache-2.0 (repo + Qwen3.5/3.8 base weights) Free A decoder-LLM (Qwen3.5-0.8B/4B/9B-Base, Qwen3.8-27B) + rank-16 LoRA + a pointer head that scores each option's token against a readout token. Needs a GPU.
OpenJev (us) Self-hosted, open source Apache-2.0 (core); MIT (SDKs, derived from TypeSafe's) Free A sentence-embedding classifier (system-one, CPU, no GPU) plus an optional encoder-marker model family (otsone-mini/otsone-base) we train ourselves.

Drop-in compatibility with the official TypeSafe SDKs

Works with unmodified typesafe-sdk / @typesafe-ai/sdk How it was checked
TypeSafe Jev Yes (it is the API the SDKs were written for) —
OpenJev (us) Yes for choice / score / noul, multi-question requests, models.list(), request ids, error classes and zstd request bodies, with TYPESAFE_BASE_URL set to the server (with or without /v1). No for image assets (415) and box questions (400). Run on 2026-11-11 with typesafe-sdk 0.7.5 and @typesafe-ai/sdk 0.7.0 against a local server: examples/official-sdk.py, examples/official-sdk.mjs
Kev, OpenAI Decisions API Not checked by us —

Measured accuracy, with sources

Model Benchmark Accuracy Calibration Source
TypeSafe Jev decision-v4 dev set (1,200 q, in-distribution) 0.845 Brier-type score ≈0.70 at 5% error budget Kev's own published model cards (kev-4b.md), read directly from the repo
TypeSafe Jev held-out new-source set 0.857 — same
Kev-4B decision-v4 dev set 0.854 — same
Kev-8B decision-v4 dev set 0.863 — same
Kev-27B held-out new-source set 0.851 Brier 0.52–0.69 at 5% error budget (weaker than Jev) same
Kev-27B MMLU-Pro (general knowledge, not decision-specific) 0.675 — same
TypeSafe Jev MMLU-Pro 0.840 — same
OpenAI Decisions API — no public numbers found — still in limited/public beta as of this writing
OpenJev otsone-mini (ours, MiniLM-L6, CPU-only) our 14-case gold suite, genuinely held out (--no-gold, not trained on) 0.8333 (12/14 cases, 15/18 checks) ECE 0.1255 otsone/models/otsone-mini-run1-nogold/model_card.json, reproducible via otsone distill --backbone otsone-mini --no-gold && otsone eval
OpenJev otsone-base (ours, ModernBERT-base, CPU-only, reduced 3-epoch run) same 14-case gold suite 0.7778 (12/14, 14/18) ECE 0.1596 otsone/docs/results/otsone-base-reduced-3ep/ — explicitly a reduced-budget run (shared-machine CPU contention cut it short of the full 14-epoch recipe), not a final number
OpenJev system-one (default engine, embedding) Banking77 test, 3,080 rows / 77 intents 0.624 ECE 0.035 src/lib/content/leaderboard.json, rendered at /leaderboard, reproducible with npm run leaderboard
OpenJev system-one CLINC150 plus test, 4,500 rows / 150 intents 0.674 ECE 0.055 same
OpenJev system-one AG News test, 7,600 rows / 4 topics 0.750 ECE 0.053 same

Honest read

  • Jev vs. Kev is the real headline, not ours: a fully open, Apache-2.0, from-scratch effort (Kev) already matches real Jev's accuracy on some slices (0.854–0.863 vs. 0.845) while trailing on calibration and raw knowledge. That is evidence the decision-model category does not require a closed model to compete on accuracy — it is not evidence about otsone-mini specifically, which has not been run against the same decision-v4 benchmark Kev and Jev share.
  • Our numbers and Kev/Jev's numbers are not on the same benchmark. otsone-mini's 0.8333/0.1255 is measured on our own 14-case gold suite, not decision-v4 — the two are not directly comparable, and our gold suite is small enough that a handful of cases move the score several points. Treat it as a regression check on our own pipeline, not a claim that otsone-mini beats or loses to Jev/Kev on any shared ground.
  • otsone-base's number here is explicitly partial. The 0.7778/0.1596 row is a reduced 3-epoch run that finished end-to-end after two full 14-epoch attempts died to shared-machine CPU contention, not disk or code — see plans/openjev.md's 2026-11-11 entry for the full account. A genuinely comparable 14-epoch otsone-base number does not exist yet.
  • What we do have that a bare accuracy number does not show: an ood/abstain score on every answer, a shadow-mode comparator against your production LLM, per-label calibration fit on your own data with a held-out check, drift alerts, and signed audit reports — see the trust layer in the main README. None of the four rows above measure that; it is a different, complementary axis (what happens when the model is wrong or the input is out of distribution) that this comparison does not attempt to score.
  • The real next step toward a comparable number is adopting Kev's own versioned, named eval-suite convention (breadth-v1, transfer-v4, hard-v1) and growing our gold suite with a real train/holdout split — flagged in plans/openjev.md and not yet done.

Where these numbers come from

Kev's model cards and license were read directly from github.com/jaredpalmer/kev (not trusted from a search snippet) in a prior research pass — see plans/openjev.md's 2026-11-06 entry for the full verification trail. The category-timeline claims (Jev's 2026-09-15 launch, OpenAI's Decisions API beta date) are sourced from DeepInfra: Decision Models and Jev vs OpenAI Decisions API. Our own numbers are regenerated by running the commands cited in the table above against this repository; they are not copied from a press release.

OpenJev is an independent, unaffiliated project; "Jev" and "System One" are TypeSafe AI's names, used here only to describe wire compatibility and for benchmark comparison — see Origins & licenses in the main README.