- README Status/SDKs/Layout/Rename/Roadmap: the Python SDKs are live on PyPI under their OpenJev-era names (openjev-client 0.7.1, openjev-adapters 0.2.2, otsone 0.3.0) — `pip install openjev-client otsone openjev-adapters` works now; umpirix-* names publishing soon. - Say plainly that @umpirix/* and @openjev/* are not on npm; JS SDKs install from source. - sdk/python, adapters/python, otsone, sdk/js, sdk/ai-sdk READMEs: drop "not published yet" for the live PyPI packages, replace @openjev/* npm names with @umpirix/*, clone URLs -> caelum0x/umpirix. - Verified against registry.npmjs.org / pypi.org and a fresh-venv install + import on 2026-11-12. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
235 lines
10 KiB
Markdown
235 lines
10 KiB
Markdown
# openjev-adapters
|
|
|
|
A drop-in replacement for `typesafe_sdk`'s `system_one` evaluation API, backed by LLM
|
|
APIs. It runs typed System One decisions (producing `Choice` / `Score` / `Noul`
|
|
answers) through OpenAI, Anthropic, or Gemini — no server required.
|
|
|
|
Part of the **OpenJev** ecosystem. It stays wire-compatible with both the hosted
|
|
TypeSafe API (`https://api.typesafe.ai/v1`) and a self-hosted OpenJev deployment,
|
|
which share the `/v1/systemone` contract: the request/response shapes (`model` +
|
|
`state` + `questions` as a **dict**) are unchanged, so it remains a straight drop-in for
|
|
`typesafe_sdk`.
|
|
|
|
Useful for comparing OpenJev against an LLM on cost/speed/intelligence, or for running
|
|
System One decisions entirely against your own models.
|
|
|
|
> Based on [`@typesafe-ai/system-one-adapter-python`](https://github.com/typesafe-ai/system-one-adapter-python)
|
|
> (MIT). See [NOTICE](NOTICE) for changes.
|
|
|
|
## Install
|
|
|
|
The provider SDKs are optional extras — install the one(s) you use:
|
|
|
|
> Live on PyPI as `openjev-adapters` (importable module `openjev_adapters`). The project
|
|
> was renamed to Umpirix; the `umpirix-adapters` package name is publishing soon and
|
|
> `openjev-adapters` will keep installing. To work from source instead:
|
|
> `pip install './umpirix/adapters/python[openai]'`.
|
|
|
|
```bash
|
|
pip install 'openjev-adapters[openai]' # OpenAI-compatible providers
|
|
pip install 'openjev-adapters[anthropic]' # native Anthropic
|
|
pip install 'openjev-adapters[gemini]' # native Gemini
|
|
```
|
|
|
|
## Usage
|
|
|
|
Unlike `TypeSafeClient`, the client is configured with how the LLM should answer, and
|
|
each call names a `provider` alongside the `model`:
|
|
|
|
```python
|
|
from openjev_adapters import SystemOneAdapterClient, Noul, Score, Choice
|
|
|
|
client = SystemOneAdapterClient(
|
|
structured_outputs=True, # use the provider's native structured-output mode
|
|
llm_answer_mode="probabilities", # or "discrete"
|
|
normalize_probabilities=True,
|
|
)
|
|
|
|
response = client.system_one(
|
|
state="This book was a delight to read.",
|
|
questions={"positive": Noul(instructions="The book review is positive.")},
|
|
provider="openai", # "openai", "anthropic", or "gemini"
|
|
model="gpt-4o-mini",
|
|
)
|
|
```
|
|
|
|
`provider` and `model` may also be set on the constructor as defaults. `provider`
|
|
is required unless `model` is a provider instance (e.g. a custom OpenAI-compatible
|
|
endpoint or a Gemini client):
|
|
|
|
```python
|
|
from openjev_adapters.providers.openai import OpenAIProvider
|
|
|
|
client.system_one(state, questions, model=OpenAIProvider("grok-4", base_url="https://api.x.ai/v1"))
|
|
```
|
|
|
|
Use clients as context managers (`with` / `async with`), or call `close()` /
|
|
`await aclose()` after all evaluations finish. The adapter closes providers it
|
|
creates; provider instances passed as `model` remain caller-owned.
|
|
|
|
### Endpoint configuration
|
|
|
|
The adapter is wire-compatible with the shared `/v1/systemone` contract, so the same
|
|
code can target the hosted TypeSafe API or a self-hosted OpenJev deployment. The
|
|
base URL and credential are resolved from the environment, with an explicit argument
|
|
always taking precedence:
|
|
|
|
| Setting | Precedence |
|
|
| --- | --- |
|
|
| Base URL | argument → `OPENJEV_BASE_URL` → `TYPESAFE_BASE_URL` → `http://localhost:8080/v1` |
|
|
| API key | argument → `OPENJEV_API_KEY` → `TYPESAFE_API_KEY` |
|
|
|
|
```python
|
|
from openjev_adapters import resolve_base_url, resolve_api_key
|
|
from openjev_adapters.providers import build_endpoint_provider
|
|
|
|
# Defaults to localhost; set OPENJEV_BASE_URL to point at any openjev host.
|
|
resolve_base_url() # "http://localhost:8080/v1" unless overridden
|
|
|
|
# Point at a remote openjev host with: export OPENJEV_BASE_URL=http://your-host/v1
|
|
provider = build_endpoint_provider("gpt-4o-mini") # OpenAI-compatible, env-resolved
|
|
client.system_one(state, questions, model=provider)
|
|
```
|
|
|
|
`build_endpoint_provider` (and `build_async_endpoint_provider`) build an
|
|
OpenAI-compatible provider aimed at the resolved endpoint; pass `base_url=` / `api_key=`
|
|
to override the environment explicitly. `resolve_base_url` / `resolve_api_key` expose the
|
|
same resolution for custom wiring.
|
|
|
|
OpenAI's endpoint uses the Responses API, with strict JSON Schema for structured
|
|
output and JSON mode for prompted output. Custom endpoints (including
|
|
`OPENAI_BASE_URL`) default to Chat Completions. Pass `api="responses"` or
|
|
`api="chat_completions"` to `OpenAIProvider` / `AsyncOpenAIProvider` to select
|
|
explicitly, for example when using an OpenAI proxy. Responses are requested with
|
|
`store=False`; corrective retries send the conversation history with each request.
|
|
|
|
Gemini uses the Interactions API. Structured mode sets `response_format` to JSON
|
|
Schema; prompted mode leaves the schema in the system prompt. Requests use
|
|
`store=False` and send the full conversation on corrective retries. Credentials
|
|
come from `GEMINI_API_KEY` or `GOOGLE_API_KEY`, or `api_key=` on
|
|
`GeminiProvider` / `AsyncGeminiProvider`.
|
|
|
|
For larger Anthropic evaluations, configure the output token limit on the provider
|
|
(default: 4,096 tokens):
|
|
|
|
```python
|
|
from openjev_adapters.providers.anthropic import AnthropicProvider
|
|
|
|
client.system_one(state, questions, model=AnthropicProvider("claude-haiku-4-5", max_tokens=8192))
|
|
```
|
|
|
|
`AsyncAnthropicProvider` accepts the same option. A response that reaches the limit
|
|
raises `typesafe_sdk.TypeSafeError` with instructions to increase `max_tokens` or
|
|
request fewer questions; it does not consume malformed-output retries.
|
|
|
|
Provider-declared refusals and incomplete generations, including Anthropic context
|
|
overflow, also raise `TypeSafeError` without corrective retries. The error names the
|
|
reason and preserves the full provider response in `error.debug["llm_attempts"]`.
|
|
|
|
### Response
|
|
|
|
The response is a `typesafe_sdk.SystemOneResponse` subclass — same `answers` and typed
|
|
views — with two additions:
|
|
|
|
- `response.usage` adds `input_tokens_total` / `output_tokens_total` (across retries),
|
|
`n_retries`, `n_retries_malformed_structure`, and `latency`.
|
|
- `response.debug` holds `llm_attempts`, `retry_reasons`, and probability-normalization
|
|
diagnostics.
|
|
|
|
OpenAI-compatible endpoints may omit token usage. Unreported counts are `None`;
|
|
the evaluation still succeeds. A cumulative token count is `None` if any attempt
|
|
omitted that count. Reported counts, including zero, are preserved.
|
|
|
|
`llm_attempts` records every provider call in order, including transient failures and
|
|
malformed responses. Each entry contains a snapshot of `messages`,
|
|
`model_request_parameters` (`schema` and `structured`), `llm_response`, and `debug_info`
|
|
with the model, provider, and any error. The built-in providers also include the exact
|
|
SDK `request` arguments, the full provider response in `llm_response`, and the API and
|
|
finish reason in `debug_info`. Custom providers return their text and token counts in
|
|
`llm_response`. Calls that fail before returning a model response leave it as `None`.
|
|
Terminal `TypeSafeError` exceptions expose the same attempt history in `error.debug`.
|
|
|
|
To replay an attempt through the same configured provider (use `await` for async):
|
|
|
|
```python
|
|
from openjev_adapters.providers import Message
|
|
|
|
attempt = response.debug["llm_attempts"][0]
|
|
result = provider.request(
|
|
[Message(**message) for message in attempt["messages"]],
|
|
**attempt["model_request_parameters"],
|
|
)
|
|
```
|
|
|
|
It is a Pydantic model like every SDK response in `typesafe-sdk>=0.7.0`, so use
|
|
`model_dump()` for a dictionary or `model_dump_json()` for JSON:
|
|
|
|
```python
|
|
print(response.model_dump_json())
|
|
```
|
|
|
|
### Async
|
|
|
|
`AsyncSystemOneAdapterClient` mirrors the sync client with `await client.system_one(...)`
|
|
and `async with`.
|
|
|
|
## Options
|
|
|
|
| Option | Meaning |
|
|
| --- | --- |
|
|
| `structured_outputs` | Use the provider's native structured output, else prompt for JSON and validate client-side (works with any chat model). |
|
|
| `llm_answer_mode` | `"probabilities"` (per-label distribution) or `"discrete"` (one value per question). |
|
|
| `normalize_probabilities` | Rescale invalid LLM probability distributions to sum to 1. |
|
|
| `n_retry_malformed_structure` | Corrective retries when the model's output fails schema validation. |
|
|
| `retry` | `typesafe_sdk.RetryPolicy` for transient provider failures. |
|
|
|
|
The transient retry count and time budget apply separately to each provider request.
|
|
Corrective requests share the evaluation's `n_retry_malformed_structure` allowance
|
|
and preserve the earlier responses and correction messages.
|
|
|
|
## Instructor bridge: confidence for `response_model=`
|
|
|
|
If you already describe decisions as Pydantic models for
|
|
[Instructor](https://python.useinstructor.com/), `openjev_adapters.instructor` answers the
|
|
same model through an OpenJev server and hands back the confidence behind every field.
|
|
Nothing is generated, so the model always validates.
|
|
|
|
```python
|
|
from openjev_adapters.instructor import from_openjev
|
|
client = from_openjev() # reads OPENJEV_BASE_URL / OPENJEV_API_KEY
|
|
ticket, report = client.chat.completions.create_with_confidence(response_model=Ticket, messages=messages)
|
|
```
|
|
|
|
```python
|
|
from typing import Literal
|
|
from pydantic import BaseModel, Field
|
|
|
|
class Ticket(BaseModel):
|
|
category: Literal["refund", "cancel", "billing"] = Field(description="Main request")
|
|
urgent: bool
|
|
severity: int = Field(ge=0, le=3, json_schema_extra={"levels": ["none", "low", "high", "critical"]})
|
|
topic: Literal["shipping", "account"] | None = None # may abstain -> None
|
|
|
|
report.confidence["category"] # confidence of the chosen label (fitted to your labels with calibration_id)
|
|
report.probabilities["category"] # full distribution over the options
|
|
report.abstained # fields that said "I don't know", e.g. ("topic",)
|
|
report.escalated # fields the `auto` model re-asked to an LLM
|
|
```
|
|
|
|
| Pydantic field | Question | Value |
|
|
| --- | --- | --- |
|
|
| `Literal[...]`, `str`/`int` `Enum` | `Choice` | the chosen option |
|
|
| `bool` | `Noul` | `True` when P(yes) >= 0.5 |
|
|
| `int` with `ge`/`gt` and `le`/`lt` (2 to 10 levels) | `Score` | the most likely level |
|
|
| `X \| None` | same, with abstention on | `None` when the answer abstains |
|
|
|
|
Other fields must have a default and are left at it; a required free-text field raises
|
|
`UnsupportedFieldError` so you can keep extracting it with Instructor. A required field
|
|
that abstains raises `AbstainedError` (with `.report`) rather than guessing.
|
|
|
|
Options pass straight to `/v1/systemone`: `model="auto"` (local first, LLM only when unsure),
|
|
`calibration_id="cal_..."` (a map fitted on your labels), `abstain_below=0.5`,
|
|
`escalate_below`, `timeout`, `retry`. `client.create(...)` returns only the model;
|
|
`from_openjev_async()` gives the same API with `await`. System messages are treated as
|
|
LLM instructions and are not part of the judged text.
|