Raven-1/tests/test_providers_codex_client_version.py
Dizhan Xue a2d391ea4c
fix(providers): list every Codex model and name the newest release on each turn (#995)
## Summary

Fixes #986

The Codex model list (`https://chatgpt.com/backend-api/codex/models`)
was requested with `client_version=0.0.0`. The backend filters the
catalogue by each model's own minimum client version, so every model
gated above `0.0.0` was dropped before Raven saw it: `gpt-6.1-sol`,
`gpt-6-sol` and `gpt-6-luna` were missing from both consumers of
`account_models()`:

- **Add model**: `model.fetch_models` -> `test_provider` ->
`_probe_codex_catalog` (`raven/config/update_providers.py`)
- **Chat model picker**: `model.options` -> `_runtime_models`
(`raven/rpc/methods/model.py`)

The first commit raised the value to the release `0.163.0`. The
follow-up commit replaces that with the split below: the **catalogue**
asks as a discovery sentinel so it sees the whole roster, and
**generation** names the newest published Codex CLI release, resolved at
runtime.

### Evidence

From #986 (live account, read-only GET, 2026-10-16):

| `client_version` | visible models |
|---|---|
| `0.0.0` | 4: `gpt-6-astra`, `gpt-5.6-sol`, `gpt-5.6-terra`,
`gpt-5.6-luna` |
| `0.163.0` | all 7 (adds `gpt-6.1-sol`, `gpt-6-sol`, `gpt-6-luna`) |
| `99.0.0` | the same 7 |
| omitted | HTTP 400 |

Bisected: `gpt-6-sol` / `gpt-6-luna` appear from `0.155.0`,
`gpt-6.1-sol` from `0.159.0`.

Public reports:

- `99.0.0` returned HTTP 200 for two distinct Codex OAuth accounts on
2026-10-08: https://github.com/NousResearch/hermes-agent/issues/135024
(that report shows the `gpt-6-astra` entry; the full 7-model result at
`99.0.0` is #986's own measurement above).
- opencodex discovers the roster with
`CODEX_ROSTER_DISCOVERY_CLIENT_VERSION = "99.0.0"` ("asks as a client
newer than any release so it sees the whole roster"):
`src/codex/model-entitlements.ts:807` in
https://app.unpkg.com/@bitkyc08/opencodex@2.75.0/files/src/codex/model-entitlements.ts
- `0.0.0` returns a frozen legacy list, and any version at or above the
newest gate returns the full catalogue:
https://github.com/NousResearch/hermes-agent/issues/119412
- `gpt-6.1-sol` declares `minimal_client_version: 0.153.0` but is served
only from `0.159.0` (so the declared value cannot be used as the version
to send): https://github.com/farion1231/cc-switch/issues/7797

## Changes

### Catalogue: discovery sentinel (`raven/providers/codex_catalog.py`)

`CLIENT_VERSION = "0.163.0"` -> `DISCOVERY_CLIENT_VERSION = "99.0.0"`.
Any real release, including the newest, hides a model gated above it
until the next release ships. The sentinel lists the account's whole
roster. The comment keeps the #986 gate history (`0.155.0` / `0.159.0`),
cites the public measurements, and records the risk: if the backend ever
starts refusing versions newer than any release, this has to go back to
a real one.

### Generation: newest published release
(`raven/providers/codex_client_version.py`, new)

`resolve()` returns the release to send:

1. in-process cache, 15 minutes (monotonic, the same style as
`codex_catalog`);
2. npm `https://registry.npmjs.org/-/package/@openai/codex/dist-tags`,
the `latest` tag only (currently `0.163.0`; the `alpha` tag, currently
`0.164.0-alpha.3`, is never read);
3. the last good value, persisted as `codex-client-version.json` under
`get_cache_dir()` (atomic replace, the same convention as
`served_models.py`), checked again on read;
4. `FALLBACK_VERSION = "0.163.0"`: a real release, not below the known
generation-side gate `0.159.0`, to be bumped with Codex CLI releases.

Every candidate must be a plain `X.Y.Z` (so no prerelease, no empty
value, no `v` prefix) and no older than `FALLBACK_VERSION`. So `0.0.0`
cannot be returned, and neither can a rolled-back tag older than the
bundled release. Nothing raises to the caller.

### Generation request carries the version
(`raven/providers/openai_codex_provider.py`)

`_build_headers` adds `version: <resolved release>`, and `chat()`
resolves it via `asyncio.to_thread` (an npm round trip at most once per
cache window). `originator: raven` and `User-Agent` are unchanged.

The transport was confirmed from the official source before wiring it
in. At tag `rust-v0.163.0`:

- the built-in OpenAI provider sets `http_headers: {"version":
env!("CARGO_PKG_VERSION")}`: [`codex-rs/model-provider-info/src/lib.rs`
L526-L543](https://github.com/openai/codex/blob/rust-v0.163.0/codex-rs/model-provider-info/src/lib.rs#L526-L543);
the workspace version at that tag is `0.163.0` ([`codex-rs/Cargo.toml`
L161-L162](https://github.com/openai/codex/blob/rust-v0.163.0/codex-rs/Cargo.toml#L161-L162));
- `to_api_provider` points ChatGPT-login auth at
`https://chatgpt.com/backend-api/codex` and copies those headers into
the API provider:
[L427-L475](https://github.com/openai/codex/blob/rust-v0.163.0/codex-rs/model-provider-info/src/lib.rs#L427-L475);
- every request starts from the provider's headers
([`codex-rs/codex-client/src/provider.rs`
L79-L89](https://github.com/openai/codex/blob/rust-v0.163.0/codex-rs/codex-client/src/provider.rs#L79-L89)),
including `POST /responses`
([`codex-rs/codex-api/src/endpoint/responses.rs`](https://github.com/openai/codex/blob/rust-v0.163.0/codex-rs/codex-api/src/endpoint/responses.rs)).

cc-switch independently sends the same `version` header on its Codex
OAuth generation path: [`src-tauri/src/proxy/providers/claude.rs`
L932-L933](77c6dc793d/src-tauri/src/proxy/providers/claude.rs (L932-L933)).

### Tests

- `tests/test_providers_codex_catalog.py`: the pinned value is now
`"99.0.0"`. The gate test serves a mock catalogue that filters by client
version and now includes a model gated above the newest release
(`0.170.0`); it checks that the sentinel lists it, that `visibility:
"hide"` entries are still dropped, and that `0.0.0` is not sent.
- `tests/test_providers_codex_client_version.py` (new, 17 cases):
`latest` is used and persisted; one fetch per cache window; prerelease,
empty, missing, `v`-prefixed, two-part, `0.0.0` and older-than-bundled
tags all fall back; offline falls back to the persisted value; a
persisted `0.0.0` is rejected; offline with nothing persisted gives
`FALLBACK_VERSION`; malformed JSON and a 429 fall back quietly;
`FALLBACK_VERSION` is a plain release `>= 0.159.0`.
- `tests/test_openai_codex_provider.py`: `_build_headers` sends
`version`; a turn sends npm's `latest`; a turn with npm unreachable
sends `FALLBACK_VERSION`.
- `tests/conftest.py`: an autouse fixture keeps the npm lookup off the
network (as `_no_openrouter_network` does for OpenRouter), so other
provider tests do not depend on what npm published that day.

## Test plan

- [x] Related tests (`test_providers_codex_client_version`,
`test_providers_codex_catalog`, `test_openai_codex_provider`,
`test_config_update_providers`, `test_rpc_model*`,
`test_providers_first_byte_budget`, `test_providers_chatgpt_token`,
`test_cli_provider_commands`): **586 passed**, with the host's `*_PROXY`
variables unset (with them set, 5 proxy-redaction tests in
`test_config_update_providers.py` fail identically on `main`).
- [x] Full suite (excluding `tests/integration`): 32427 passed, 101
failed. All 101 also fail with this branch's changes stashed (same IDs;
none in providers). They come from this host's environment, for example
a host default model leaking into playbook and sub-agent tests.
- [x] Mutation check: with the catalogue back on a real release and the
`version` header removed, the 5 new or changed assertions fail.
- [x] `ruff check` / `ruff format --check` on the changed files.

## Product trade-off

Because the catalogue now asks as `99.0.0`, it can list a model whose
gate is **above npm's `latest`**. Generation sends the real `latest`, so
such a model may be listed in Add model and the picker and then get a
400 on `/responses` until the next Codex CLI release ships. We chose to
show the account's whole roster over hiding models the account is
offered. The alternative, asking the catalogue as the resolved release,
would keep the two in step but recreate #986 whenever a gate moves ahead
of a release.

## Review follow-up

Two review findings are addressed in the latest commit:

- **Issue references in source (Blocking, inline):** `AGENTS.md` 1.1
forbids comments that reference an issue, and the changed module,
comment and test docstrings cited #986. The citations are removed; the
measured behaviour each one accompanied stays where it was.
- **An unusable cache aborted the turn (P2):** `_remember` resolved
`cache_path()` outside its `try`, and that path went through
`get_cache_dir()`, which creates the directory on the way out; `chat()`
awaited `resolve()` outside its request error handling. A file where the
cache directory belongs (or a refused `mkdir`) therefore raised after a
valid release had been fetched and stopped generation. Path resolution
is now inside the guard, every persistence failure is swallowed, and the
path is built from `raven_home()` directly so resolving it no longer has
a filesystem side effect. Covered by three tests: an unwritable cache
keeps the fetched release, an unwritable cache falls back instead of
raising, and a turn still goes out with the fetched release; all three
fail against the previously reviewed head.

## Known unverified (Open in #986)

- The catalogue sentinel is based on public measurements (#986,
hermes-agent#135024) and community use (opencodex), not on an
OpenAI-documented contract.
- **Not verified with a real account:** signing in to Codex and running
a turn on one of the new models (`gpt-6.1-sol` / `gpt-6-sol` /
`gpt-6-luna`). The `version` header matches what the official CLI sends,
but `originator` is still `raven` rather than `codex_cli_rs`. Whether
the backend accepts a turn from this client on these models still needs
one signed-in run before #986 can be considered fully resolved.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-18 21:23:42 +08:00

149 lines
5.4 KiB
Python

"""Unit tests for the Codex CLI release a generation request names."""
from __future__ import annotations
import json
from typing import Any
import httpx
import pytest
from raven.providers import codex_client_version
#: Captured at import, before the suite's autouse fixture replaces it.
_REAL_FETCH = codex_client_version._fetch_latest
def _serve(monkeypatch: pytest.MonkeyPatch, payload: Any = None, *, error: Exception | None = None) -> list[str]:
"""Put back the real fetch the suite stubs out, over a mocked transport."""
seen: list[str] = []
def handler(request: httpx.Request) -> httpx.Response:
seen.append(str(request.url))
if error is not None:
raise error
return httpx.Response(200, json=payload)
def fake_get(url, **kwargs):
with httpx.Client(transport=httpx.MockTransport(handler)) as client:
return client.get(url, **{k: v for k, v in kwargs.items() if k != "timeout"})
monkeypatch.setattr(httpx, "get", fake_get)
monkeypatch.setattr(codex_client_version, "_fetch_latest", _REAL_FETCH)
return seen
def _remembered() -> object:
try:
return json.loads(codex_client_version.cache_path().read_text(encoding="utf-8")).get("version")
except OSError:
return None
def test_the_latest_tag_is_the_answer(monkeypatch: pytest.MonkeyPatch) -> None:
seen = _serve(monkeypatch, {"latest": "0.170.0", "alpha": "0.171.0-alpha.1"})
assert codex_client_version.resolve() == "0.170.0"
assert seen == [codex_client_version.DIST_TAGS_URL]
assert _remembered() == "0.170.0"
def test_the_answer_is_not_asked_for_again_within_the_window(monkeypatch: pytest.MonkeyPatch) -> None:
seen = _serve(monkeypatch, {"latest": "0.170.0"})
codex_client_version.resolve()
codex_client_version.resolve()
assert len(seen) == 1
@pytest.mark.parametrize(
"latest",
["0.164.0-alpha.3", "", None, "v0.170.0", "0.170", "0.0.0", "0.150.0"],
ids=["prerelease", "empty", "missing", "prefixed", "two-part", "zero", "older-than-ours"],
)
def test_a_tag_that_is_not_a_usable_release_falls_back(monkeypatch: pytest.MonkeyPatch, latest: object) -> None:
"""A prerelease is not a client anyone runs, and nothing older than the bundled
release may be sent -- `0.0.0` least of all."""
_serve(monkeypatch, {"latest": latest, "alpha": "0.164.0-alpha.3"})
assert codex_client_version.resolve() == codex_client_version.FALLBACK_VERSION
assert _remembered() is None
def test_offline_answers_the_release_last_seen(monkeypatch: pytest.MonkeyPatch) -> None:
_serve(monkeypatch, {"latest": "0.170.0"})
codex_client_version.resolve()
codex_client_version.reset_cache()
_serve(monkeypatch, error=httpx.ConnectError("offline"))
assert codex_client_version.resolve() == "0.170.0"
def test_a_remembered_release_is_checked_like_a_fetched_one(monkeypatch: pytest.MonkeyPatch) -> None:
path = codex_client_version.cache_path()
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps({"version": "0.0.0"}), encoding="utf-8")
_serve(monkeypatch, error=httpx.ConnectError("offline"))
assert codex_client_version.resolve() == codex_client_version.FALLBACK_VERSION
def test_offline_with_nothing_remembered_answers_the_bundled_release(monkeypatch: pytest.MonkeyPatch) -> None:
_serve(monkeypatch, error=httpx.ConnectError("offline"))
assert codex_client_version.resolve() == codex_client_version.FALLBACK_VERSION
@pytest.mark.parametrize(
"payload",
[["0.170.0"], "0.170.0", {"next": "0.170.0"}],
ids=["list", "string", "no-latest"],
)
def test_an_unexpected_answer_falls_back_quietly(monkeypatch: pytest.MonkeyPatch, payload: object) -> None:
_serve(monkeypatch, payload)
assert codex_client_version.resolve() == codex_client_version.FALLBACK_VERSION
def test_an_error_status_falls_back_quietly(monkeypatch: pytest.MonkeyPatch) -> None:
def fake_get(url, **kwargs):
return httpx.Response(429, request=httpx.Request("GET", url))
monkeypatch.setattr(httpx, "get", fake_get)
monkeypatch.setattr(codex_client_version, "_fetch_latest", _REAL_FETCH)
assert codex_client_version.resolve() == codex_client_version.FALLBACK_VERSION
def test_the_bundled_release_is_a_plain_release_past_the_known_gate() -> None:
"""`gpt-6.1-sol` is served from 0.159.0; the floor must not sit below it."""
parts = tuple(int(p) for p in codex_client_version.FALLBACK_VERSION.split("."))
assert len(parts) == 3
assert parts >= (0, 159, 0)
def _block_cache_dir(monkeypatch: pytest.MonkeyPatch, tmp_path) -> None:
"""A cache that cannot be prepared: a regular file where the directory goes."""
(tmp_path / "cache").write_text("not a directory", encoding="utf-8")
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
def test_an_unwritable_cache_keeps_the_release_it_fetched(monkeypatch: pytest.MonkeyPatch, tmp_path) -> None:
"""Persisting is best-effort; the answer this process has is not."""
_serve(monkeypatch, {"latest": "0.170.0"})
_block_cache_dir(monkeypatch, tmp_path)
assert codex_client_version.resolve() == "0.170.0"
def test_an_unwritable_cache_falls_back_instead_of_raising(monkeypatch: pytest.MonkeyPatch, tmp_path) -> None:
_serve(monkeypatch, error=httpx.ConnectError("offline"))
_block_cache_dir(monkeypatch, tmp_path)
assert codex_client_version.resolve() == codex_client_version.FALLBACK_VERSION