Raven-1/tests/_tls.py
Dizhan Xue 9c893bb65e
fix(*): trust the system certificate store and stop unverified Codex retries (#878)
## Summary

Behind a TLS-inspecting corporate proxy, or against a company-hosted
gateway signed by a private CA, `raven onboard` failed with
`CERTIFICATE_VERIFY_FAILED` even with the corporate root installed in
the operating system (#637). httpx, LiteLLM and the OpenAI SDK verify
against certifi's bundled roots unless told otherwise, so the
operating-system store was never consulted on the paths that reach a
model. aiohttp, urllib and websockets read OpenSSL's defaults instead,
so one process trusted two different sets of roots depending on which
client made the call.

What changes:

- `raven/security/tls.py` (new): `use_system_ca()` injects truststore,
so every TLS context built afterwards is verified by the operating
system (CryptoAPI on Windows, the Security framework on macOS, OpenSSL's
default paths on Linux). A bundle a caller loads into a context is still
honoured: on Linux it joins the store, on macOS its roots become extra
anchors, on Windows it is tried when the system roots fail. So every
root trusted before stays trusted. `RAVEN_NO_SYSTEM_CA=1` opts out,
parsed like `RAVEN_NO_UPDATE_CHECK`.
- `raven/cli/entry.py` (new) is now the entry of both the console script
and `python -m raven`, the form the web supervisor starts its gateway
with. It switches trust before importing the CLI, because the CLI's
imports reach aiohttp (through the sub-agent backends), and aiohttp
builds its default context when it is imported. `raven/cli/commands.py`
no longer starts the CLI when run as a script, which skipped the switch;
pointing it at the entry would not have helped, since by then its own
imports have built aiohttp's context.
- The provider probe names a certificate that fails verification
`certificate_untrusted` instead of `network_error`. It checks for that
ahead of the proxy branch: through a proxy that answered the tunnel, a
refused certificate used to be reported as `proxy_unreachable`. The
Codex check (its backend's list of the account's models), MiniMax's
token renewal (which reported `oauth_token_missing`) and the sub-agent
key prompt's probe name it too. LiteLLM's Copilot token exchange and its
ChatGPT renewal drop the handshake error before Raven sees it, so those
still read as the credential.
- The three surfaces that read the probe say so. `raven onboard` and
`raven provider test` no longer send the reader to check the network,
proxy or VPN; their hints name the three causes: a CA to add to the
system certificate store; macOS and Windows refusing a certificate by
their own rules even when `SSL_CERT_FILE` trusts its CA, which
`RAVEN_NO_SYSTEM_CA=1` turns off; and an expired certificate, or one for
another host, which is the server's to fix. `raven onboard` offers the
menu a network error gets (re-entering the key cannot fix a
certificate); the web Settings page has its own line, in English and
Chinese, in `i18n/messages.json` and the TUI's catalogue generated from
it.
- The RPC contract's description of the model catalogue's `status`
(`rpc-schema/openrpc.json` and both generated clients) now lists the
probe's real words. It named `unauthorized`, which the probe never
returns (a 401 or 403 is `invalid_key`), and missed `proxy_unreachable`
as well as the new status.
- The Codex provider no longer retries a request whose certificate
failed verification with verification turned off (`verify=False`). The
second attempt carried the ChatGPT OAuth bearer token and the whole
conversation to whoever presented the certificate (a captive portal, a
hostile network, a proxy that re-signs TLS), and on a network like the
one in #637 it is what would keep Codex working unverified while every
other provider fails. The retry dates from the initial import;
`_request_codex` loses the `verify` parameter that existed only to carry
it.
- `raven upgrade` and the page's Update verify against the same store as
the release lookup. The upgrade helper runs on the base interpreter,
outside the tool environment that holds truststore, and fetched the
release lists with urllib's default context; uv fetched the rest with
its bundled roots. When the spawning process itself verifies against the
system store, it now copies truststore into a fresh temporary directory
and hands the helper its path, as it hands over the marker path; one
that opted out, or never went through `cli.entry`, hands over nothing
and needs no truststore to say so. The helper loads the copy before its
first download and deletes it, and gives `UV_SYSTEM_CERTS` (and
`UV_NATIVE_TLS`, which older uv reads) to uv's own calls only, so the
Raven it relaunches inherits neither. A copy, because
`inject_into_ssl()` imports requests and urllib3 wherever it finds them,
and the tool environment's belong to what uv is about to rewrite.
- Quick Start, in both languages, gains a section on corporate proxies
and private CAs: WSL2 does not see the Windows store, and the EverOS
memory server and Node-based agents run as separate programs that need
`SSL_CERT_FILE` and `NODE_EXTRA_CA_CERTS` in the shell profile.
`SSL_CERT_FILE` must name a complete bundle that exists, since httpx
refuses to start with a missing one:
`/etc/ssl/certs/ca-certificates.crt` on Debian, Ubuntu and WSL, and one
built from `/etc/ssl/cert.pem` on macOS. A note says macOS and Windows
apply their own certificate policy and names `RAVEN_NO_SYSTEM_CA=1` as
the way back.

The LiteLLM warning in the issue ("Failed to fetch remote model cost map
... CERTIFICATE_VERIFY_FAILED", with `SSL_CERT_FILE` exported) is not
LiteLLM building its own store. That fetch is a plain `httpx.get`, which
reads `SSL_CERT_FILE`, and the variable replaces the whole trusted set:
with a corporate-only PEM the gateway verifies while
raw.githubusercontent.com, signed by a public root, no longer does.
Measured with a private CA: corporate-only PEM, warning; certifi plus
the corporate root, remote fetch succeeds.

Not covered, by construction: the EverOS server is a separate `everos`
executable and Node-based agents are not Python, so an injection in this
process cannot reach them. The new docs section gives them their
environment variables.

`uv.lock`: adding truststore re-resolved the lock (uv 0.11.25, revision
3 kept). Besides the truststore entries, five packages' dependency edges
lose a marker: matrix-nio, h2, aiohttp-socks, aiologic and culsans. Each
dropped marker is implied by the only path that reaches its package
(matrix-nio, and through it h2 and aiohttp-socks, only where
`sys_platform != 'win32'`; culsans and aiologic only below Python 3.13),
so the resolved set is unchanged on every platform. uv 0.12.23 produces
the same edges and also rewrites the lock to revision 5, so it was not
used; both versions accept the committed lock under `uv lock --locked`.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Red first, on the base: the end-to-end `raven provider test custom`
against a gateway whose root only the stand-in system store holds exits
1 with `network_error ... unable to get local issuer certificate` under
both launch forms, and its opted-out twin prints "Check network /
firewall / VPN settings"; the probe test returns `network_error`, and
through a CONNECT proxy `proxy_unreachable`; the onboarding menu offers
`rekey` for a certificate failure; the web note falls to
`probe_unknown`; and the Codex test's endpoint receives `Bearer
tok-secret`, with the captured log line `SSL certificate verification
failed for Codex API; retrying with verify=False`.

Mutations, each applied and then restored byte for byte:

- the entry no longer switching trust: both end-to-end trust cases fail;
- the switch moved after the CLI import: only
`test_trust_is_settled_before_the_cli_builds_a_context` fails;
- the classifier accepting any `ssl.SSLError`: the plain-HTTP case
fails;
- the certificate check placed after the proxy branch: the proxy case
fails.

Review round two, red first at the round-one head: against an endpoint
signed by a root no store holds, the Codex check returned
`network_error`, MiniMax's token renewal `oauth_token_missing` and the
sub-agent key probe `network_error`; both hint tests failed on the
missing opt-out; the guard scan found `raven/cli/commands.py`; and the
upgrade tests found nothing handed over. Round-two mutations, each
applied and then restored byte for byte:

- the classifier matching `CERTIFICATE_VERIFY_FAILED` in the message:
the new macOS and Windows cases fail (282 of 282 stayed green before
them);
- 17 single-construct mutants of the upgrade hand-over (not switching,
not putting the copy on the path, keeping it, hiding a failed load,
leaving the path in the environment, not telling uv, overwriting a uv
setting the user made, dropping the older uv name, telling uv without a
copy, uv not given its environment, passing on an inherited path,
importing truststore instead of reading whether this process switched,
handing over without that check, handing over always, handing over no
copy, and keeping a copy no helper started, at each spawn site): each
fails at least one test;
- the copy failure's warning and its cleanup: each fails the
copy-failure test.

Commands and results, at the head after rebasing onto `main`. The
commits `main` gained since round one touch none of this branch's lines:
`git range-diff` shows every commit unchanged, and the aggregate
patch-id is the same before and after.

- The touched Python test files (`tests/test_security_tls.py`,
`tests/test_cli_system_ca.py`, `tests/test_config_update_providers.py`,
`tests/test_cli_onboard_commands.py`,
`tests/test_cli_provider_commands.py`,
`tests/test_cli_subagent_setup.py`, `tests/test_updates_upgrade.py`,
`tests/test_openai_codex_provider.py`), the Codex provider's neighbours
(`tests/test_providers_first_byte_budget.py`,
`tests/test_provider_protocol.py`) and the i18n tests, in full: 1010
passed.
- Full suite with branch coverage, `pytest -q -p no:randomly --cov=raven
--cov=raven_everos --cov-branch` on the all-extras environment with the
box proxy variables unset: 2 failed, 27750 passed, 119 skipped. Both
failures fail the same way on a clean tree of the new merge base under
the same interpreter, so neither is introduced; they are local to this
box (an `npm` on its PATH, and running as root):
-
`tests/test_install_script.py::test_resolve_node_dir_answers_each_case_it_exists_for`
-
`tests/test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing`
- `scripts/coverage_gate.py diff --base-ref origin/main --threshold 90`
over that run: 100.00% (57/57 executable changed lines).
- `tests/integration/test_cli_upgrade_real_uv.py`, which CI runs only in
the Windows self-upgrade job (`--noconftest`): 3 passed locally. Its
fixture calls the handoff without `cli.entry`; the `system store`
variant switches first, as `cli.entry` does, so that job runs the whole
hand-over on Windows, and the fixture prints what it handed over so
neither variant passes without its branch running.
- `ruff check`, `ruff format --check`, `ty check` over the changed
modules, `lint-imports` (10 contracts kept), `deptry raven`: clean.
- Repository gates over `origin/main...HEAD`:
`check_source_language.py`, `check_large_files.py` and
`check_commit_messages.py` exit 0.
- Generated files on the rebased tree, where `main` had also changed the
RPC schema, the message catalogue and their generated copies:
`gen-i18n.mjs --check` and `gen-rpc-types.mjs --check` (ui-tui) and
`gen-rpc-client.mjs --check` (ui-web) exit 0.
- Round two changes no file under `ui-web`, `ui-tui` or `rpc-schema`;
these were measured at round one:
- ui-web, `node node_modules/vitest/vitest.mjs run
src/features/settings/SetupBodies.test.tsx
scripts/gates/i18n-keys.test.mjs`: 33 passed; `scripts/gates`: 46 files,
205 passed; `tsc --noEmit` clean, its `--listFiles` resolving the edited
files inside this tree; ESLint clean on the three files.
- ui-tui, the steps of TUI checks: ESLint 0 errors; `lint:rpc` and
`lint:i18n` pass; `tsc --noEmit` clean; the build passes; `vitest run`
has 9 failures in 4 files (renderer and typing-burst tests), the same 9
that fail on a clean worktree of the merge base on this box, so none is
introduced.
- RPC contract: `node scripts/gen-rpc-client.mjs --check` (ui-web) and
`node scripts/gen-rpc-types.mjs --check` (ui-tui) exit 0, each generated
file differing from the base by the one description line; the 9 Python
test files that read the schema, 798 passed.
- Docs: `mkdocs build --strict` clean, with the new paragraphs on the
English and Chinese pages; `tests/test_docs_site.py` and
`tests/test_living_docs.py` pass in the full suite.

The end-to-end cases run on Linux only. They stand the system store in
with OpenSSL's directory store through `SSL_CERT_DIR`, and name a bundle
of public roots in `SSL_CERT_FILE` so that httpx cannot pass by reading
`SSL_CERT_DIR` itself. Verification on macOS and Windows was read from
truststore 0.10.4's source, not run. The upgrade helper's tests share
the limit: on Linux truststore and the standard library read the same
store, so they show the hand-over (the copy loads on the base
interpreter under `-I` before the first download, and uv alone is told),
not the Keychain difference it exists for.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- Every Raven process started as `raven` or `python -m raven` now trusts
the operating system's roots in addition to what each library bundles.
For those processes `SSL_CERT_FILE` adds to the trusted set instead of
replacing it; someone using it to narrow trust to one private root keeps
that root but also trusts the system store.
- On macOS and Windows the system verifier applies its own certificate
policy as well. A private certificate that OpenSSL accepted from
`SSL_CERT_FILE` can be refused there if it breaks that policy;
`RAVEN_NO_SYSTEM_CA=1` restores OpenSSL's verification.
- On macOS and Windows truststore also verifies the client's chain on
server-side contexts, refusing a client that sends none. Raven builds no
server-side TLS context: it serves plain HTTP on localhost.
- Upgrades verify their downloads the way the release lookup does: on
macOS that includes Apple's certificate policy, which the helper's
OpenSSL did not apply. uv is told to use the system store only for the
upgrade's own calls, and a `UV_SYSTEM_CERTS` or `UV_NATIVE_TLS` the user
set is kept.
- Behind a proxy that re-signs TLS, Codex requests now fail with the
certificate error instead of continuing unverified. Trusting the proxy's
root, in the system store or in a complete bundle named by
`SSL_CERT_FILE`, is the remedy.
- Startup: about 100 ms warm and 270 ms cold, measured on Linux, mostly
truststore importing urllib3 and requests to patch their references.
- Rollback: `RAVEN_NO_SYSTEM_CA=1` restores the previous trust per
process, and an upgrade started under it hands its helper and uv
nothing; the Codex change has no switch, by design. Reverting the squash
commit removes both.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

Fixes #637

---------

Co-authored-by: Claude (claude-opus-5-5[1m]) <noreply@anthropic.com>
2026-10-10 10:40:12 +08:00

205 lines
7.7 KiB
Python

"""Throwaway TLS material and HTTPS endpoints for tests that need a real handshake.
Every key is minted for the test that asks for it, so none lives in the
repository, and every endpoint listens on 127.0.0.1 only.
"""
from __future__ import annotations
import contextlib
import datetime
import hashlib
import http.server
import ipaddress
import json
import socket
import ssl
import threading
from collections.abc import Iterator
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.x509.oid import ExtendedKeyUsageOID, NameOID
# Already in OpenSSL's canonical form (lowercase, single spaces), so the body of
# its DER encoding is exactly what OpenSSL hashes into the file name a
# directory store finds the root under.
_ROOT_NAME = "raven test root"
@dataclass(frozen=True)
class PrivateCA:
"""A root no public bundle carries, and a certificate it issued for 127.0.0.1."""
root: Path
server: Path
store: Path
def private_ca(directory: Path) -> PrivateCA:
"""Mint the root, the server certificate, and a directory store holding the root.
``server`` holds the certificate and its key, the shape ``load_cert_chain``
takes. ``store`` is laid out the way OpenSSL reads a ``capath`` such as
``/etc/ssl/certs``.
"""
directory.mkdir(parents=True, exist_ok=True)
now = datetime.datetime.now(datetime.timezone.utc)
root_key = ec.generate_private_key(ec.SECP256R1())
root_name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, _ROOT_NAME)])
root = (
x509.CertificateBuilder()
.subject_name(root_name)
.issuer_name(root_name)
.public_key(root_key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(now - datetime.timedelta(hours=1))
.not_valid_after(now + datetime.timedelta(hours=1))
.add_extension(x509.BasicConstraints(ca=True, path_length=0), critical=True)
.add_extension(_key_usage(key_cert_sign=True, crl_sign=True), critical=True)
.add_extension(x509.SubjectKeyIdentifier.from_public_key(root_key.public_key()), critical=False)
.sign(root_key, hashes.SHA256())
)
server_key = ec.generate_private_key(ec.SECP256R1())
server = (
x509.CertificateBuilder()
.subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "127.0.0.1")]))
.issuer_name(root_name)
.public_key(server_key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(now - datetime.timedelta(hours=1))
.not_valid_after(now + datetime.timedelta(hours=1))
.add_extension(x509.BasicConstraints(ca=False, path_length=None), critical=True)
.add_extension(_key_usage(digital_signature=True), critical=True)
.add_extension(x509.ExtendedKeyUsage([ExtendedKeyUsageOID.SERVER_AUTH]), critical=False)
.add_extension(x509.SubjectAlternativeName([x509.IPAddress(ipaddress.ip_address("127.0.0.1"))]), critical=False)
.add_extension(x509.AuthorityKeyIdentifier.from_issuer_public_key(root_key.public_key()), critical=False)
.sign(root_key, hashes.SHA256())
)
root_pem = root.public_bytes(serialization.Encoding.PEM)
ca = PrivateCA(root=directory / "root.pem", server=directory / "server.pem", store=directory / "store")
ca.root.write_bytes(root_pem)
ca.server.write_bytes(
server.public_bytes(serialization.Encoding.PEM)
+ server_key.private_bytes(
serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption()
)
)
ca.store.mkdir()
(ca.store / f"{_subject_hash(root)}.0").write_bytes(root_pem)
return ca
class OpenAIModels(http.server.BaseHTTPRequestHandler):
"""Answers ``GET /v1/models`` the way an OpenAI-compatible gateway does."""
def do_GET(self) -> None:
body = json.dumps({"object": "list", "data": [{"id": "corp-model", "object": "model"}]}).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args: object) -> None:
pass
@contextmanager
def https_endpoint(server: Path, handler: type[http.server.BaseHTTPRequestHandler]) -> Iterator[str]:
"""Serve ``handler`` over HTTPS on 127.0.0.1 and yield its origin."""
httpd = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain(server)
httpd.socket = context.wrap_socket(httpd.socket, server_side=True)
thread = threading.Thread(target=httpd.serve_forever, daemon=True)
thread.start()
try:
yield f"https://127.0.0.1:{httpd.server_address[1]}"
finally:
httpd.shutdown()
httpd.server_close()
thread.join()
@contextmanager
def connect_proxy() -> Iterator[str]:
"""A forward proxy on 127.0.0.1 that tunnels ``CONNECT`` and nothing else; yields its URL."""
listener = socket.create_server(("127.0.0.1", 0))
listener.settimeout(0.2)
stopped = threading.Event()
def pipe(source: socket.socket, sink: socket.socket) -> None:
with contextlib.suppress(OSError):
while chunk := source.recv(65536):
sink.sendall(chunk)
for end in (source, sink):
with contextlib.suppress(OSError):
end.shutdown(socket.SHUT_RDWR)
def tunnel(client: socket.socket) -> None:
with client:
head = b""
while b"\r\n\r\n" not in head:
chunk = client.recv(4096)
if not chunk:
return
head += chunk
host, _, port = head.split(b" ", 2)[1].decode().rpartition(":")
with socket.create_connection((host, int(port))) as upstream:
client.sendall(b"HTTP/1.1 200 Connection established\r\n\r\n")
back = threading.Thread(target=pipe, args=(upstream, client), daemon=True)
back.start()
pipe(client, upstream)
back.join()
def accept() -> None:
while not stopped.is_set():
try:
client, _ = listener.accept()
except TimeoutError:
continue
except OSError:
return
threading.Thread(target=tunnel, args=(client,), daemon=True).start()
acceptor = threading.Thread(target=accept, daemon=True)
acceptor.start()
try:
yield f"http://127.0.0.1:{listener.getsockname()[1]}"
finally:
stopped.set()
acceptor.join()
listener.close()
def _key_usage(**granted: bool) -> x509.KeyUsage:
usages = (
"digital_signature",
"content_commitment",
"key_encipherment",
"data_encipherment",
"key_agreement",
"key_cert_sign",
"crl_sign",
"encipher_only",
"decipher_only",
)
return x509.KeyUsage(**{usage: granted.get(usage, False) for usage in usages})
def _subject_hash(cert: x509.Certificate) -> str:
"""OpenSSL's ``X509_NAME_hash``: the first four bytes of a SHA-1, little-endian.
It hashes the name's canonical encoding without the outer SEQUENCE header,
which for a name already in canonical form is the DER body as written.
"""
der = cert.subject.public_bytes()
length = der[1]
body = der[2:] if length < 0x80 else der[2 + (length & 0x7F) :]
return f"{int.from_bytes(hashlib.sha1(body).digest()[:4], 'little'):08x}"