Found by running typesafe-sdk 0.7.5 and @typesafe-ai/sdk 0.7.0 against a
local server and fixed:
- zstd (and gzip/deflate/br) request bodies are decoded; the SDKs compress
bodies of 1 KiB or more, which used to fail with a 500
- GET /v1/models also returns models: [{ name, description, release_date }]
- every systemone, batch and models response carries x-typesafe-request-id
- a base URL ending in /v1 (or /api/v1) resolves to the same handlers
- request errors carry a detail list; a non-JSON body is a 400, not a 500
- multipart (image asset) requests get a clear 415
Adds examples/official-sdk.py and .mjs, a /docs/typesafe-sdk page, a README
block and a COMPARISON.md section, limited to what was observed to pass.
7.3 KiB
OpenJev vs. the rest of the decision-model category
The "decision layer" category (a typed classifier that stands in for a full LLM call on routing/triage-shaped questions) is new: TypeSafe released the hosted Jev API on 2026-09-15, the open-weights Laya model followed three days later, and OpenAI announced a Decisions API (model name "Luna") at the same DevDay, moving to public beta on 2026-10-06. This page compares what is actually known and measured about each, including our own numbers, with sources — nothing here is estimated or rounded in our favor.
The four options
| Access | License | Pricing | What it is | |
|---|---|---|---|---|
| TypeSafe Jev | Hosted API (api.typesafe.ai) |
Proprietary | Paid | The original hosted System One /v1/systemone service this whole category is wire-compatible with. |
| OpenAI Decisions API | Limited preview → public beta 2026-10-06 | Proprietary | OpenAI pricing | GPT-6 "Luna"-backed; not yet broadly available or independently benchmarked as of this writing. |
Kev (jaredpalmer/kev) |
Self-hosted, open weights | Apache-2.0 (repo + Qwen3.5/3.8 base weights) | Free | A decoder-LLM (Qwen3.5-0.8B/4B/9B-Base, Qwen3.8-27B) + rank-16 LoRA + a pointer head that scores each option's token against a readout token. Needs a GPU. |
| OpenJev (us) | Self-hosted, open source | Apache-2.0 (core); MIT (SDKs, derived from TypeSafe's) | Free | A sentence-embedding classifier (system-one, CPU, no GPU) plus an optional encoder-marker model family (otsone-mini/otsone-base) we train ourselves. |
Drop-in compatibility with the official TypeSafe SDKs
Works with unmodified typesafe-sdk / @typesafe-ai/sdk |
How it was checked | |
|---|---|---|
| TypeSafe Jev | Yes (it is the API the SDKs were written for) | — |
| OpenJev (us) | Yes for choice / score / noul, multi-question requests, models.list(), request ids, error classes and zstd request bodies, with TYPESAFE_BASE_URL set to the server (with or without /v1). No for image assets (415) and box questions (400). |
Run on 2026-11-11 with typesafe-sdk 0.7.5 and @typesafe-ai/sdk 0.7.0 against a local server: examples/official-sdk.py, examples/official-sdk.mjs |
| Kev, OpenAI Decisions API | Not checked by us | — |
Measured accuracy, with sources
| Model | Benchmark | Accuracy | Calibration | Source |
|---|---|---|---|---|
| TypeSafe Jev | decision-v4 dev set (1,200 q, in-distribution) |
0.845 | Brier-type score ≈0.70 at 5% error budget | Kev's own published model cards (kev-4b.md), read directly from the repo |
| TypeSafe Jev | held-out new-source set | 0.857 | — | same |
| Kev-4B | decision-v4 dev set |
0.854 | — | same |
| Kev-8B | decision-v4 dev set |
0.863 | — | same |
| Kev-27B | held-out new-source set | 0.851 | Brier 0.52–0.69 at 5% error budget (weaker than Jev) | same |
| Kev-27B | MMLU-Pro (general knowledge, not decision-specific) | 0.675 | — | same |
| TypeSafe Jev | MMLU-Pro | 0.840 | — | same |
| OpenAI Decisions API | — | no public numbers found | — | still in limited/public beta as of this writing |
OpenJev otsone-mini (ours, MiniLM-L6, CPU-only) |
our 14-case gold suite, genuinely held out (--no-gold, not trained on) |
0.8333 (12/14 cases, 15/18 checks) | ECE 0.1255 | otsone/models/otsone-mini-run1-nogold/model_card.json, reproducible via otsone distill --backbone otsone-mini --no-gold && otsone eval |
OpenJev otsone-base (ours, ModernBERT-base, CPU-only, reduced 3-epoch run) |
same 14-case gold suite | 0.7778 (12/14, 14/18) | ECE 0.1596 | otsone/docs/results/otsone-base-reduced-3ep/ — explicitly a reduced-budget run (shared-machine CPU contention cut it short of the full 14-epoch recipe), not a final number |
OpenJev system-one (default engine, embedding) |
Banking77 test, 3,080 rows / 77 intents | 0.624 | ECE 0.035 | src/lib/content/leaderboard.json, rendered at /leaderboard, reproducible with npm run leaderboard |
OpenJev system-one |
CLINC150 plus test, 4,500 rows / 150 intents | 0.674 | ECE 0.055 | same |
OpenJev system-one |
AG News test, 7,600 rows / 4 topics | 0.750 | ECE 0.053 | same |
Honest read
- Jev vs. Kev is the real headline, not ours: a fully open, Apache-2.0, from-scratch
effort (Kev) already matches real Jev's accuracy on some slices (0.854–0.863 vs. 0.845)
while trailing on calibration and raw knowledge. That is evidence the decision-model
category does not require a closed model to compete on accuracy — it is not evidence
about
otsone-minispecifically, which has not been run against the samedecision-v4benchmark Kev and Jev share. - Our numbers and Kev/Jev's numbers are not on the same benchmark.
otsone-mini's 0.8333/0.1255 is measured on our own 14-case gold suite, notdecision-v4— the two are not directly comparable, and our gold suite is small enough that a handful of cases move the score several points. Treat it as a regression check on our own pipeline, not a claim thatotsone-minibeats or loses to Jev/Kev on any shared ground. otsone-base's number here is explicitly partial. The 0.7778/0.1596 row is a reduced 3-epoch run that finished end-to-end after two full 14-epoch attempts died to shared-machine CPU contention, not disk or code — seeplans/openjev.md's 2026-11-11 entry for the full account. A genuinely comparable 14-epochotsone-basenumber does not exist yet.- What we do have that a bare accuracy number does not show: an
ood/abstain score on every answer, a shadow-mode comparator against your production LLM, per-label calibration fit on your own data with a held-out check, drift alerts, and signed audit reports — see the trust layer in the main README. None of the four rows above measure that; it is a different, complementary axis (what happens when the model is wrong or the input is out of distribution) that this comparison does not attempt to score. - The real next step toward a comparable number is adopting Kev's own versioned, named
eval-suite convention (
breadth-v1,transfer-v4,hard-v1) and growing our gold suite with a real train/holdout split — flagged inplans/openjev.mdand not yet done.
Where these numbers come from
Kev's model cards and license were read directly from github.com/jaredpalmer/kev (not
trusted from a search snippet) in a prior research pass — see plans/openjev.md's
2026-11-06 entry for the full verification trail. The category-timeline claims (Jev's
2026-09-15 launch, OpenAI's Decisions API beta date) are sourced from
DeepInfra: Decision Models
and Jev vs OpenAI Decisions API.
Our own numbers are regenerated by running the commands cited in the table above against
this repository; they are not copied from a press release.
OpenJev is an independent, unaffiliated project; "Jev" and "System One" are TypeSafe AI's names, used here only to describe wire compatibility and for benchmark comparison — see Origins & licenses in the main README.