## Summary Behind a TLS-inspecting corporate proxy, or against a company-hosted gateway signed by a private CA, `raven onboard` failed with `CERTIFICATE_VERIFY_FAILED` even with the corporate root installed in the operating system (#637). httpx, LiteLLM and the OpenAI SDK verify against certifi's bundled roots unless told otherwise, so the operating-system store was never consulted on the paths that reach a model. aiohttp, urllib and websockets read OpenSSL's defaults instead, so one process trusted two different sets of roots depending on which client made the call. What changes: - `raven/security/tls.py` (new): `use_system_ca()` injects truststore, so every TLS context built afterwards is verified by the operating system (CryptoAPI on Windows, the Security framework on macOS, OpenSSL's default paths on Linux). A bundle a caller loads into a context is still honoured: on Linux it joins the store, on macOS its roots become extra anchors, on Windows it is tried when the system roots fail. So every root trusted before stays trusted. `RAVEN_NO_SYSTEM_CA=1` opts out, parsed like `RAVEN_NO_UPDATE_CHECK`. - `raven/cli/entry.py` (new) is now the entry of both the console script and `python -m raven`, the form the web supervisor starts its gateway with. It switches trust before importing the CLI, because the CLI's imports reach aiohttp (through the sub-agent backends), and aiohttp builds its default context when it is imported. `raven/cli/commands.py` no longer starts the CLI when run as a script, which skipped the switch; pointing it at the entry would not have helped, since by then its own imports have built aiohttp's context. - The provider probe names a certificate that fails verification `certificate_untrusted` instead of `network_error`. It checks for that ahead of the proxy branch: through a proxy that answered the tunnel, a refused certificate used to be reported as `proxy_unreachable`. The Codex check (its backend's list of the account's models), MiniMax's token renewal (which reported `oauth_token_missing`) and the sub-agent key prompt's probe name it too. LiteLLM's Copilot token exchange and its ChatGPT renewal drop the handshake error before Raven sees it, so those still read as the credential. - The three surfaces that read the probe say so. `raven onboard` and `raven provider test` no longer send the reader to check the network, proxy or VPN; their hints name the three causes: a CA to add to the system certificate store; macOS and Windows refusing a certificate by their own rules even when `SSL_CERT_FILE` trusts its CA, which `RAVEN_NO_SYSTEM_CA=1` turns off; and an expired certificate, or one for another host, which is the server's to fix. `raven onboard` offers the menu a network error gets (re-entering the key cannot fix a certificate); the web Settings page has its own line, in English and Chinese, in `i18n/messages.json` and the TUI's catalogue generated from it. - The RPC contract's description of the model catalogue's `status` (`rpc-schema/openrpc.json` and both generated clients) now lists the probe's real words. It named `unauthorized`, which the probe never returns (a 401 or 403 is `invalid_key`), and missed `proxy_unreachable` as well as the new status. - The Codex provider no longer retries a request whose certificate failed verification with verification turned off (`verify=False`). The second attempt carried the ChatGPT OAuth bearer token and the whole conversation to whoever presented the certificate (a captive portal, a hostile network, a proxy that re-signs TLS), and on a network like the one in #637 it is what would keep Codex working unverified while every other provider fails. The retry dates from the initial import; `_request_codex` loses the `verify` parameter that existed only to carry it. - `raven upgrade` and the page's Update verify against the same store as the release lookup. The upgrade helper runs on the base interpreter, outside the tool environment that holds truststore, and fetched the release lists with urllib's default context; uv fetched the rest with its bundled roots. When the spawning process itself verifies against the system store, it now copies truststore into a fresh temporary directory and hands the helper its path, as it hands over the marker path; one that opted out, or never went through `cli.entry`, hands over nothing and needs no truststore to say so. The helper loads the copy before its first download and deletes it, and gives `UV_SYSTEM_CERTS` (and `UV_NATIVE_TLS`, which older uv reads) to uv's own calls only, so the Raven it relaunches inherits neither. A copy, because `inject_into_ssl()` imports requests and urllib3 wherever it finds them, and the tool environment's belong to what uv is about to rewrite. - Quick Start, in both languages, gains a section on corporate proxies and private CAs: WSL2 does not see the Windows store, and the EverOS memory server and Node-based agents run as separate programs that need `SSL_CERT_FILE` and `NODE_EXTRA_CA_CERTS` in the shell profile. `SSL_CERT_FILE` must name a complete bundle that exists, since httpx refuses to start with a missing one: `/etc/ssl/certs/ca-certificates.crt` on Debian, Ubuntu and WSL, and one built from `/etc/ssl/cert.pem` on macOS. A note says macOS and Windows apply their own certificate policy and names `RAVEN_NO_SYSTEM_CA=1` as the way back. The LiteLLM warning in the issue ("Failed to fetch remote model cost map ... CERTIFICATE_VERIFY_FAILED", with `SSL_CERT_FILE` exported) is not LiteLLM building its own store. That fetch is a plain `httpx.get`, which reads `SSL_CERT_FILE`, and the variable replaces the whole trusted set: with a corporate-only PEM the gateway verifies while raw.githubusercontent.com, signed by a public root, no longer does. Measured with a private CA: corporate-only PEM, warning; certifi plus the corporate root, remote fetch succeeds. Not covered, by construction: the EverOS server is a separate `everos` executable and Node-based agents are not Python, so an injection in this process cannot reach them. The new docs section gives them their environment variables. `uv.lock`: adding truststore re-resolved the lock (uv 0.11.25, revision 3 kept). Besides the truststore entries, five packages' dependency edges lose a marker: matrix-nio, h2, aiohttp-socks, aiologic and culsans. Each dropped marker is implied by the only path that reaches its package (matrix-nio, and through it h2 and aiohttp-socks, only where `sys_platform != 'win32'`; culsans and aiologic only below Python 3.13), so the resolved set is unchanged on every platform. uv 0.12.23 produces the same edges and also rewrites the lock to revision 5, so it was not used; both versions accept the committed lock under `uv lock --locked`. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Red first, on the base: the end-to-end `raven provider test custom` against a gateway whose root only the stand-in system store holds exits 1 with `network_error ... unable to get local issuer certificate` under both launch forms, and its opted-out twin prints "Check network / firewall / VPN settings"; the probe test returns `network_error`, and through a CONNECT proxy `proxy_unreachable`; the onboarding menu offers `rekey` for a certificate failure; the web note falls to `probe_unknown`; and the Codex test's endpoint receives `Bearer tok-secret`, with the captured log line `SSL certificate verification failed for Codex API; retrying with verify=False`. Mutations, each applied and then restored byte for byte: - the entry no longer switching trust: both end-to-end trust cases fail; - the switch moved after the CLI import: only `test_trust_is_settled_before_the_cli_builds_a_context` fails; - the classifier accepting any `ssl.SSLError`: the plain-HTTP case fails; - the certificate check placed after the proxy branch: the proxy case fails. Review round two, red first at the round-one head: against an endpoint signed by a root no store holds, the Codex check returned `network_error`, MiniMax's token renewal `oauth_token_missing` and the sub-agent key probe `network_error`; both hint tests failed on the missing opt-out; the guard scan found `raven/cli/commands.py`; and the upgrade tests found nothing handed over. Round-two mutations, each applied and then restored byte for byte: - the classifier matching `CERTIFICATE_VERIFY_FAILED` in the message: the new macOS and Windows cases fail (282 of 282 stayed green before them); - 17 single-construct mutants of the upgrade hand-over (not switching, not putting the copy on the path, keeping it, hiding a failed load, leaving the path in the environment, not telling uv, overwriting a uv setting the user made, dropping the older uv name, telling uv without a copy, uv not given its environment, passing on an inherited path, importing truststore instead of reading whether this process switched, handing over without that check, handing over always, handing over no copy, and keeping a copy no helper started, at each spawn site): each fails at least one test; - the copy failure's warning and its cleanup: each fails the copy-failure test. Commands and results, at the head after rebasing onto `main`. The commits `main` gained since round one touch none of this branch's lines: `git range-diff` shows every commit unchanged, and the aggregate patch-id is the same before and after. - The touched Python test files (`tests/test_security_tls.py`, `tests/test_cli_system_ca.py`, `tests/test_config_update_providers.py`, `tests/test_cli_onboard_commands.py`, `tests/test_cli_provider_commands.py`, `tests/test_cli_subagent_setup.py`, `tests/test_updates_upgrade.py`, `tests/test_openai_codex_provider.py`), the Codex provider's neighbours (`tests/test_providers_first_byte_budget.py`, `tests/test_provider_protocol.py`) and the i18n tests, in full: 1010 passed. - Full suite with branch coverage, `pytest -q -p no:randomly --cov=raven --cov=raven_everos --cov-branch` on the all-extras environment with the box proxy variables unset: 2 failed, 27750 passed, 119 skipped. Both failures fail the same way on a clean tree of the new merge base under the same interpreter, so neither is introduced; they are local to this box (an `npm` on its PATH, and running as root): - `tests/test_install_script.py::test_resolve_node_dir_answers_each_case_it_exists_for` - `tests/test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing` - `scripts/coverage_gate.py diff --base-ref origin/main --threshold 90` over that run: 100.00% (57/57 executable changed lines). - `tests/integration/test_cli_upgrade_real_uv.py`, which CI runs only in the Windows self-upgrade job (`--noconftest`): 3 passed locally. Its fixture calls the handoff without `cli.entry`; the `system store` variant switches first, as `cli.entry` does, so that job runs the whole hand-over on Windows, and the fixture prints what it handed over so neither variant passes without its branch running. - `ruff check`, `ruff format --check`, `ty check` over the changed modules, `lint-imports` (10 contracts kept), `deptry raven`: clean. - Repository gates over `origin/main...HEAD`: `check_source_language.py`, `check_large_files.py` and `check_commit_messages.py` exit 0. - Generated files on the rebased tree, where `main` had also changed the RPC schema, the message catalogue and their generated copies: `gen-i18n.mjs --check` and `gen-rpc-types.mjs --check` (ui-tui) and `gen-rpc-client.mjs --check` (ui-web) exit 0. - Round two changes no file under `ui-web`, `ui-tui` or `rpc-schema`; these were measured at round one: - ui-web, `node node_modules/vitest/vitest.mjs run src/features/settings/SetupBodies.test.tsx scripts/gates/i18n-keys.test.mjs`: 33 passed; `scripts/gates`: 46 files, 205 passed; `tsc --noEmit` clean, its `--listFiles` resolving the edited files inside this tree; ESLint clean on the three files. - ui-tui, the steps of TUI checks: ESLint 0 errors; `lint:rpc` and `lint:i18n` pass; `tsc --noEmit` clean; the build passes; `vitest run` has 9 failures in 4 files (renderer and typing-burst tests), the same 9 that fail on a clean worktree of the merge base on this box, so none is introduced. - RPC contract: `node scripts/gen-rpc-client.mjs --check` (ui-web) and `node scripts/gen-rpc-types.mjs --check` (ui-tui) exit 0, each generated file differing from the base by the one description line; the 9 Python test files that read the schema, 798 passed. - Docs: `mkdocs build --strict` clean, with the new paragraphs on the English and Chinese pages; `tests/test_docs_site.py` and `tests/test_living_docs.py` pass in the full suite. The end-to-end cases run on Linux only. They stand the system store in with OpenSSL's directory store through `SSL_CERT_DIR`, and name a bundle of public roots in `SSL_CERT_FILE` so that httpx cannot pass by reading `SSL_CERT_DIR` itself. Verification on macOS and Windows was read from truststore 0.10.4's source, not run. The upgrade helper's tests share the limit: on Linux truststore and the standard library read the same store, so they show the hand-over (the copy loads on the base interpreter under `-I` before the first download, and uv alone is told), not the Keychain difference it exists for. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - Every Raven process started as `raven` or `python -m raven` now trusts the operating system's roots in addition to what each library bundles. For those processes `SSL_CERT_FILE` adds to the trusted set instead of replacing it; someone using it to narrow trust to one private root keeps that root but also trusts the system store. - On macOS and Windows the system verifier applies its own certificate policy as well. A private certificate that OpenSSL accepted from `SSL_CERT_FILE` can be refused there if it breaks that policy; `RAVEN_NO_SYSTEM_CA=1` restores OpenSSL's verification. - On macOS and Windows truststore also verifies the client's chain on server-side contexts, refusing a client that sends none. Raven builds no server-side TLS context: it serves plain HTTP on localhost. - Upgrades verify their downloads the way the release lookup does: on macOS that includes Apple's certificate policy, which the helper's OpenSSL did not apply. uv is told to use the system store only for the upgrade's own calls, and a `UV_SYSTEM_CERTS` or `UV_NATIVE_TLS` the user set is kept. - Behind a proxy that re-signs TLS, Codex requests now fail with the certificate error instead of continuing unverified. Trusting the proxy's root, in the system store or in a complete bundle named by `SSL_CERT_FILE`, is the remedy. - Startup: about 100 ms warm and 270 ms cold, measured on Linux, mostly truststore importing urllib3 and requests to patch their references. - Rollback: `RAVEN_NO_SYSTEM_CA=1` restores the previous trust per process, and an upgrade started under it hands its helper and uv nothing; the Codex change has no switch, by design. Reverting the squash commit removes both. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues Fixes #637 --------- Co-authored-by: Claude (claude-opus-5-5[1m]) <noreply@anthropic.com>
205 lines
7.7 KiB
Python
205 lines
7.7 KiB
Python
"""Throwaway TLS material and HTTPS endpoints for tests that need a real handshake.
|
|
|
|
Every key is minted for the test that asks for it, so none lives in the
|
|
repository, and every endpoint listens on 127.0.0.1 only.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import contextlib
|
|
import datetime
|
|
import hashlib
|
|
import http.server
|
|
import ipaddress
|
|
import json
|
|
import socket
|
|
import ssl
|
|
import threading
|
|
from collections.abc import Iterator
|
|
from contextlib import contextmanager
|
|
from dataclasses import dataclass
|
|
from pathlib import Path
|
|
|
|
from cryptography import x509
|
|
from cryptography.hazmat.primitives import hashes, serialization
|
|
from cryptography.hazmat.primitives.asymmetric import ec
|
|
from cryptography.x509.oid import ExtendedKeyUsageOID, NameOID
|
|
|
|
# Already in OpenSSL's canonical form (lowercase, single spaces), so the body of
|
|
# its DER encoding is exactly what OpenSSL hashes into the file name a
|
|
# directory store finds the root under.
|
|
_ROOT_NAME = "raven test root"
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class PrivateCA:
|
|
"""A root no public bundle carries, and a certificate it issued for 127.0.0.1."""
|
|
|
|
root: Path
|
|
server: Path
|
|
store: Path
|
|
|
|
|
|
def private_ca(directory: Path) -> PrivateCA:
|
|
"""Mint the root, the server certificate, and a directory store holding the root.
|
|
|
|
``server`` holds the certificate and its key, the shape ``load_cert_chain``
|
|
takes. ``store`` is laid out the way OpenSSL reads a ``capath`` such as
|
|
``/etc/ssl/certs``.
|
|
"""
|
|
directory.mkdir(parents=True, exist_ok=True)
|
|
now = datetime.datetime.now(datetime.timezone.utc)
|
|
root_key = ec.generate_private_key(ec.SECP256R1())
|
|
root_name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, _ROOT_NAME)])
|
|
root = (
|
|
x509.CertificateBuilder()
|
|
.subject_name(root_name)
|
|
.issuer_name(root_name)
|
|
.public_key(root_key.public_key())
|
|
.serial_number(x509.random_serial_number())
|
|
.not_valid_before(now - datetime.timedelta(hours=1))
|
|
.not_valid_after(now + datetime.timedelta(hours=1))
|
|
.add_extension(x509.BasicConstraints(ca=True, path_length=0), critical=True)
|
|
.add_extension(_key_usage(key_cert_sign=True, crl_sign=True), critical=True)
|
|
.add_extension(x509.SubjectKeyIdentifier.from_public_key(root_key.public_key()), critical=False)
|
|
.sign(root_key, hashes.SHA256())
|
|
)
|
|
server_key = ec.generate_private_key(ec.SECP256R1())
|
|
server = (
|
|
x509.CertificateBuilder()
|
|
.subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "127.0.0.1")]))
|
|
.issuer_name(root_name)
|
|
.public_key(server_key.public_key())
|
|
.serial_number(x509.random_serial_number())
|
|
.not_valid_before(now - datetime.timedelta(hours=1))
|
|
.not_valid_after(now + datetime.timedelta(hours=1))
|
|
.add_extension(x509.BasicConstraints(ca=False, path_length=None), critical=True)
|
|
.add_extension(_key_usage(digital_signature=True), critical=True)
|
|
.add_extension(x509.ExtendedKeyUsage([ExtendedKeyUsageOID.SERVER_AUTH]), critical=False)
|
|
.add_extension(x509.SubjectAlternativeName([x509.IPAddress(ipaddress.ip_address("127.0.0.1"))]), critical=False)
|
|
.add_extension(x509.AuthorityKeyIdentifier.from_issuer_public_key(root_key.public_key()), critical=False)
|
|
.sign(root_key, hashes.SHA256())
|
|
)
|
|
root_pem = root.public_bytes(serialization.Encoding.PEM)
|
|
ca = PrivateCA(root=directory / "root.pem", server=directory / "server.pem", store=directory / "store")
|
|
ca.root.write_bytes(root_pem)
|
|
ca.server.write_bytes(
|
|
server.public_bytes(serialization.Encoding.PEM)
|
|
+ server_key.private_bytes(
|
|
serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption()
|
|
)
|
|
)
|
|
ca.store.mkdir()
|
|
(ca.store / f"{_subject_hash(root)}.0").write_bytes(root_pem)
|
|
return ca
|
|
|
|
|
|
class OpenAIModels(http.server.BaseHTTPRequestHandler):
|
|
"""Answers ``GET /v1/models`` the way an OpenAI-compatible gateway does."""
|
|
|
|
def do_GET(self) -> None:
|
|
body = json.dumps({"object": "list", "data": [{"id": "corp-model", "object": "model"}]}).encode()
|
|
self.send_response(200)
|
|
self.send_header("Content-Type", "application/json")
|
|
self.send_header("Content-Length", str(len(body)))
|
|
self.end_headers()
|
|
self.wfile.write(body)
|
|
|
|
def log_message(self, *args: object) -> None:
|
|
pass
|
|
|
|
|
|
@contextmanager
|
|
def https_endpoint(server: Path, handler: type[http.server.BaseHTTPRequestHandler]) -> Iterator[str]:
|
|
"""Serve ``handler`` over HTTPS on 127.0.0.1 and yield its origin."""
|
|
httpd = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
|
|
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
|
context.load_cert_chain(server)
|
|
httpd.socket = context.wrap_socket(httpd.socket, server_side=True)
|
|
thread = threading.Thread(target=httpd.serve_forever, daemon=True)
|
|
thread.start()
|
|
try:
|
|
yield f"https://127.0.0.1:{httpd.server_address[1]}"
|
|
finally:
|
|
httpd.shutdown()
|
|
httpd.server_close()
|
|
thread.join()
|
|
|
|
|
|
@contextmanager
|
|
def connect_proxy() -> Iterator[str]:
|
|
"""A forward proxy on 127.0.0.1 that tunnels ``CONNECT`` and nothing else; yields its URL."""
|
|
listener = socket.create_server(("127.0.0.1", 0))
|
|
listener.settimeout(0.2)
|
|
stopped = threading.Event()
|
|
|
|
def pipe(source: socket.socket, sink: socket.socket) -> None:
|
|
with contextlib.suppress(OSError):
|
|
while chunk := source.recv(65536):
|
|
sink.sendall(chunk)
|
|
for end in (source, sink):
|
|
with contextlib.suppress(OSError):
|
|
end.shutdown(socket.SHUT_RDWR)
|
|
|
|
def tunnel(client: socket.socket) -> None:
|
|
with client:
|
|
head = b""
|
|
while b"\r\n\r\n" not in head:
|
|
chunk = client.recv(4096)
|
|
if not chunk:
|
|
return
|
|
head += chunk
|
|
host, _, port = head.split(b" ", 2)[1].decode().rpartition(":")
|
|
with socket.create_connection((host, int(port))) as upstream:
|
|
client.sendall(b"HTTP/1.1 200 Connection established\r\n\r\n")
|
|
back = threading.Thread(target=pipe, args=(upstream, client), daemon=True)
|
|
back.start()
|
|
pipe(client, upstream)
|
|
back.join()
|
|
|
|
def accept() -> None:
|
|
while not stopped.is_set():
|
|
try:
|
|
client, _ = listener.accept()
|
|
except TimeoutError:
|
|
continue
|
|
except OSError:
|
|
return
|
|
threading.Thread(target=tunnel, args=(client,), daemon=True).start()
|
|
|
|
acceptor = threading.Thread(target=accept, daemon=True)
|
|
acceptor.start()
|
|
try:
|
|
yield f"http://127.0.0.1:{listener.getsockname()[1]}"
|
|
finally:
|
|
stopped.set()
|
|
acceptor.join()
|
|
listener.close()
|
|
|
|
|
|
def _key_usage(**granted: bool) -> x509.KeyUsage:
|
|
usages = (
|
|
"digital_signature",
|
|
"content_commitment",
|
|
"key_encipherment",
|
|
"data_encipherment",
|
|
"key_agreement",
|
|
"key_cert_sign",
|
|
"crl_sign",
|
|
"encipher_only",
|
|
"decipher_only",
|
|
)
|
|
return x509.KeyUsage(**{usage: granted.get(usage, False) for usage in usages})
|
|
|
|
|
|
def _subject_hash(cert: x509.Certificate) -> str:
|
|
"""OpenSSL's ``X509_NAME_hash``: the first four bytes of a SHA-1, little-endian.
|
|
|
|
It hashes the name's canonical encoding without the outer SEQUENCE header,
|
|
which for a name already in canonical form is the DER body as written.
|
|
"""
|
|
der = cert.subject.public_bytes()
|
|
length = der[1]
|
|
body = der[2:] if length < 0x80 else der[2 + (length & 0x7F) :]
|
|
return f"{int.from_bytes(hashlib.sha1(body).digest()[:4], 'little'):08x}"
|