Raven-1/tests/test_cli_system_ca.py
Dizhan Xue 916761ed4c
perf(*): trim the resident memory of the gateway and the web supervisor (#954)
## Summary

A gateway, and the supervisor `raven web` leaves running beside it, held
memory from startup on for things they never use or may never be asked
to use. Two of them are no longer kept, and three are loaded on first
use. On a key-less gateway with a warm home and no terminal, as `raven
web` runs it, PSS 90 seconds after start goes from 257.2 MiB to 211.8
MiB, and the supervisor's from 77.3 MiB to 35.2 MiB. Nothing on the
wire, on disk or in the page changes shape.

- **The supervisor no longer imports the command table** (77.3 to 35.2
MiB). It only restarts its gateway, but it reached its loop through
typer and the table. `raven/cli/entry.py` now hands the argv that `raven
web` and `system.upgrade` start it with, `web --supervise --port N` with
N in ASCII digits, to `serve_commands.supervise_from_argv`, which calls
`_web` with the arguments the `web` command passes; a refusal ends the
process with the code the command would have exited with. Any other argv
is parsed by the table as before. Two process-wide settings stay ahead
of the fast path in the entry point: the TLS switch, as before, and,
moved there from the table's import, the switch of a Windows process's
stdout and stderr to UTF-8 with `PYTHONIOENCODING` set for its children.
The table's own setup, the saved language and the config migration
notices, is skipped with it: the supervisor prints no translated text
and never reads the config.
- **Two tokenizer copies left by litellm's import are freed** (25.3
MiB). `litellm.utils` keeps the dict it parses from the bundled
Anthropic tokenizer beside the string it re-serializes from it, and only
the string is read: the Claude tokenizer is built from it. cl100k's
registered encoding keeps a Python dict of its ranks beside the copy
tiktoken's core encodes with; tiktoken reads the Python one only to
pickle an encoding outside the registry, since a registered one pickles
by name. `import_litellm` now drops the dict, only while the string is
there, and the Python ranks. A failure is logged at debug level and
leaves the import standing.
- **A process whose terminal cannot be asked for its colours finds the
console's host modules on its first console command** (6.2 MiB).
Registering the console feature imported every CLI module to find the
ones that host console commands. `cli_socket.register_cli` now also
accepts a function, run once on the first `console_hosts()`. That covers
a process with no terminal, as the gateway `raven web` detaches has
none, and Windows, SSH, screen or tmux, and CI, which
`_theme._can_probe` rules out. Where the terminal can be asked, as from
`raven tui`, the host modules are still found at registration:
`raven.cli._styles` asks the terminal for its colours as it is imported,
and that question has to come before the Ink screen starts.
- **The RPC models build each validator on first use** (6.0 MiB).
`_Strict`, the base every model in `raven/rpc/models.py` derives from,
sets `defer_build=True`. The reason is a comment rather than the
docstring, because pydantic copies the docstring into the JSON schema.
- **Loading the deck engine's plugin no longer loads python-pptx** (7.9
MiB). `plugins-dist/ppt-engine/raven_ppt/services/state.py` imported the
template service, and with it python-pptx, so a gateway with the plugin
installed held it even with the plugin's config slice off, which is the
default. The service is now imported where a deck is read. With the
slice enabled, python-pptx is still loaded by the check that decks can
be made.

The upgrade regression tests also use the native launcher filename
(`raven.exe` on Windows, `raven` elsewhere), so the complete
relaunch-argv and supervisor-handoff assertions run on both platforms.
This follow-up changes test expectations only.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [x] Other

## Verification

The original author's Linux performance and suite measurements below
were recorded before the Windows test-fixture follow-up and rebase. The
seven original patches remain equivalent after rebasing; the Windows
measurements below are from the final tree. No new memory or
live-provider measurements are claimed.

- **Memory:** an isolated gateway with a key-less config and
`memory.backend` null, started without a terminal on an exported copy of
the base commit and of this head, each on its own home warmed by an
earlier start, base and head runs alternating; PSS from
`/proc/self/smaps_rollup` 90 seconds after start. The base read 257.2
MiB and this head 211.8 MiB, each within 0.1 MiB from run to run. The
figure on each change above is what taking that change back out of this
head, alone, puts back; the four add up to the difference. The
supervisor's figures are its PSS under `raven web` started on each tree.
- **Behaviour, base against this head on the same trees:**
- Token ids and counts over the repo's docs, the `raven/agent` sources
and the message catalogs, through raven's two estimators, cl100k
directly and `litellm.token_counter` for Claude, GPT, GLM, DeepSeek and
Gemini model names, with decoding, copying and pickling the encoding:
identical.
- The console: the same host modules, found at registration on the base
and on the first lookup here. On a pseudo-terminal the colour query is
written during registration on both.
- Every model in `raven/rpc/models.py`: validation and serialization
schemas, fields, and the outcome of validating the same probe payloads,
identical. On the base all but eight were built at import, here none.
Sixteen threads using every model for the first time at once: no error,
one set of answers.
- The deck engine's plugin imports without python-pptx here; on the base
it loads it.
- `raven web` end to end: it starts and serves the page; after a SIGKILL
of the gateway the supervisor brings it back on the same port with
`RAVEN_SERVE_PORT_STRICT=1`; `raven web --stop` leaves no process and no
`web.json`; `web.log` carries the same lines, and the gateway's
environment is the same key by key.
- A streamed turn and a tool-calling turn on glm-5.3-flash through a
live config's LiteLLMProvider: the same reply, the same tool call, the
same prompt usage, and the same Raven estimate.
- **New tests, against the base code:** of the 33 new or changed cases,
29 fail there. Four pass there by design: the token counts, a near-miss
argv that goes to the table, a process whose terminal can be asked
finding the host modules at registration, and the streams left alone off
Windows. All 33 pass here.
- **Mutation:** each construct was removed or bent one at a time, with
the backup outside the tree, and each mutant failed the tests written
for it:
- the supervisor: no fast path, `isdigit` as the port check, extra
arguments accepted, a refusal swallowed, every refusal ending with code
1, the port ignored;
- the Windows streams: no switch, the switch after the fast path, no
`PYTHONIOENCODING`, the switch off Windows too, strict errors, no guard
around the switch;
- the tokenizer copies: none freed, the dict dropped without its string,
the string dropped too, the ranks kept (in this process and in a fresh
one), no guard around the freeing;
- the console: the finder run on every lookup, a reset keeping a finder
that never ran, lazy everywhere, eager everywhere (in this process and
in a fresh one), registration importing one of the finder's modules;
- the validators built at import; the template service imported with the
state module.
- **Directly affected and neighbouring files:** `uv run --frozen
--python 3.12 --all-extras pytest -q -p no:cacheprovider
--idle-ceiling-strict` over the seven changed test files and
`tests/test_ppt_engine_state.py`, `tests/test_ppt_engine_prepare.py`,
`tests/test_rpc_console.py`, `tests/test_cli_tui_commands.py`,
`tests/test_cli_gateway_commands.py`,
`tests/test_rpc_contract_shapes.py`, `tests/test_cli_smoke.py`,
`tests/test_updates_upgrade.py` and `tests/test_security_tls.py`. Result
at this head: 1522 passed.
- **Full suite at this head:** `uv run --frozen --python 3.12
--all-extras pytest -q -p no:cacheprovider` (repo addopts: `-n 4`,
integration and e2e deselected; proxy variables unset). Result: 3
failed, 31663 passed, 164 skipped, 2 xfailed. The three are this box's
environment and fail the same way on an exported copy of the base
commit:
`tests/test_install_script.py::test_resolve_node_dir_answers_each_case_it_exists_for`,
`tests/test_knowledge_parser_excel.py::test_a_legacy_workbook_is_converted_and_read`
(no LibreOffice) and
`tests/test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing`
(run as root).
- **Exported trees:** the same suite on exported copies of the base
commit and of this head, compared case by case: every case they share
has the same outcome, apart from two timing-sensitive cases that failed
in the base run and pass in this head's run and on repeated runs of both
trees:
`tests/test_subagent_acp.py::test_an_interrupted_first_turn_still_binds_its_session`
and
`tests/test_subagent_cli_remote.py::test_a_cancel_ends_the_agent_s_whole_process_tree_on_the_machine`.
- **Diff coverage:** `uv run --frozen --python 3.12 python
scripts/coverage_gate.py diff --base-ref origin/main --threshold 90`
over the neighbouring run's coverage. Result: 100.00%, 50 of 50
executable changed lines. The gate measures `raven/`;
`tests/test_ppt_engine_*.py` executes both imports moved in the deck
engine.
- **Lint and gates:** `make lint-python lint-imports lint-deps
lint-types check-large-files check-source-language`, and `commitlint
--from origin/main --to HEAD --config commitlint.config.cjs` plus
`scripts/check_commit_messages.py origin/main..HEAD`. Result: all exit
0; import contracts 10 kept, 0 broken.

**Native Windows follow-up (2026-10-15):** Windows 10.0.26300, Python
3.13.12, `PYTHONUTF8=1`, isolated Raven/pytest homes and a dedicated
worktree. Setup: `uv sync --frozen --python 3.13 --all-packages --extra
dev`. Imports were verified against the tested worktree. The same frozen
sync was repeated after the rebase picked up the memory-plugin
dependency update.

- Before the fix, `uv run --frozen --no-sync pytest
tests/test_rpc_system.py -n 0 --tb=short` returned **3 failed, 40
passed**. The gateway-hosted handoff, standalone handoff and dev-flag
regression each expected `raven` where native Windows correctly selected
`raven.exe`.
- After the fix, `uv run --frozen --no-sync pytest
tests/test_rpc_system.py
tests/test_cli_serve_commands.py::TestTheSupervisorStartsWithoutTheCommandTable
tests/test_cli_system_ca.py -n 0 --idle-ceiling-strict --tb=short`
returned **66 passed, 4 skipped**.
- On the final rebased tree, the following command returned **631
passed, 4 skipped**, exit 0 with the idle ceiling enforced. The four
skips require the Linux certificate store; none are counted as Windows
passes. JUnit output paths are omitted from these commands.

```powershell
uv run --frozen --no-sync pytest tests/test_rpc_system.py tests/test_cli_serve_commands.py::TestTheSupervisorStartsWithoutTheCommandTable tests/test_cli_system_ca.py tests/test_litellm_setup.py tests/test_ppt_engine_plugin.py::test_loading_the_plugin_leaves_python_pptx_for_the_first_deck tests/test_ppt_engine_state.py tests/test_rpc_cli_dispatch.py tests/test_rpc_schema_match.py -n 0 --idle-ceiling-strict --tb=short
```

- `uv run --frozen --no-sync ruff check raven evolver agents
plugins-dist tests scripts docs-site` and the matching `ruff format
--check`: passed (2,337 files formatted).
- `uv run --frozen --no-sync lint-imports`: 10 contracts kept. `uv run
--frozen --no-sync ty check raven/cli/_console_feature.py
raven/cli/commands.py raven/cli/entry.py raven/cli/serve_commands.py
raven/providers/litellm_setup.py raven/rpc/cli_socket.py
raven/rpc/models.py
plugins-dist/ppt-engine/raven_ppt/services/state.py`: passed.
- `uv run --frozen --no-sync pre-commit run --from-ref origin/main
--to-ref HEAD`, `node node_modules/@commitlint/cli/cli.js --from
origin/main --to HEAD --config commitlint.config.cjs`, `uv run --frozen
--no-sync python scripts/check_commit_messages.py origin/main..HEAD`,
`uv run --frozen --no-sync python scripts/check_large_files.py
origin/main...HEAD`, `uv run --frozen --no-sync python
scripts/check_source_language.py origin/main...HEAD`, and `git diff
--check origin/main...HEAD`: passed.

The test correction retains the complete executable path, argv,
credentials, delayed shutdown and supervisor-handoff assertions, without
adding a skip. Upgrade spawning and the supervisor loop are substituted
in these tests. No real upgrade or resident restart, live provider call,
Windows memory measurement, full Windows suite or new Linux/macOS run
was performed by this follow-up.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

No user-facing docs change, because no command, flag, setting or file
format does.

## Risk

User-visible behaviour changes:

- **The first console command in a process whose terminal cannot be
asked** imports the finder's modules then, once, on the event loop and
under the console's dispatch lock, instead of at startup: about 0.1 to
0.2 seconds on this box. A CLI module that fails to import then fails
each console command, where before it failed the process's startup.
- **Each RPC model builds its validator the first time it is used**, so
a method's first call pays for its own models, and a model whose fields
pydantic cannot build a validator for fails there rather than when
`raven.rpc.models` is imported.
- **An unhandled exception in the supervisor** reaches `web.log` as
Python's plain traceback rather than typer's formatted one.
- **The deck engine's template service is imported at the first deck
read** instead of when the plugin loads. A python-pptx that cannot be
imported therefore no longer stops the plugin from loading: with its
config slice enabled, the check that decks can be made withholds the
deck tools, and the turn hook's note on deck state, which logs an
unreadable state at debug level and goes on, meets the import error
there.
- **Code in the same process that reads the private `_mergeable_ranks`
off the registered cl100k encoding**, as the extension example in
tiktoken's README does, gets an `AttributeError` once raven has imported
litellm. In raven, its plugins and the locked dependencies, only
tiktoken's educational module does.

Left as today:

- **Any other supervisor argv**, for example one typed by hand with
`--port=N` or the flags reordered, goes through the command table and
keeps today's memory.
- **Freeing the tokenizer copies relies on litellm's and tiktoken's
internals.** The tests read each copy where it lives, check in a fresh
interpreter that litellm's own import registers the encoding whose ranks
are freed, and pin the token counts, so a release that moves a copy,
starts reading one or stops registering the encoding fails them.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

Rollback is a revert. Nothing written to disk or sent over the wire
changes.

## Related Issues

N/A
2026-10-15 11:24:20 +08:00

186 lines
6.7 KiB
Python

"""The CLI verifies TLS against the operating system's certificate store.
A company gateway, or a proxy that inspects TLS, presents a certificate signed by
a root that IT installs in the operating system; no bundle a Python package
ships carries it. Which store a client trusts is settled before the first client
is built, so these launch raven the two ways it is launched -- the console
script, and ``python -m raven`` as the web supervisor starts its gateway -- each
in a fresh interpreter.
OpenSSL's directory store, relocated by ``SSL_CERT_DIR``, stands in for the
operating system's: it is the store Linux verifies against. The shell also names
a bundle of public roots in ``SSL_CERT_FILE``, so a client that reads those
variables itself, as httpx does, cannot pass by taking ``SSL_CERT_DIR`` on its
own.
"""
from __future__ import annotations
import os
import subprocess
import sys
from pathlib import Path
import certifi
import pytest
from raven.config.update_providers import set_provider_fields
from tests._tls import OpenAIModels, https_endpoint, private_ca
linux_only = pytest.mark.skipif(sys.platform != "linux", reason="the stand-in store is the one Linux verifies against")
_CONSOLE_SCRIPT = (
"import sys; from importlib.metadata import entry_points; "
"sys.argv[0] = 'raven'; sys.exit(entry_points(group='console_scripts')['raven'].load()())"
)
LAUNCHES = {
"console script": [sys.executable, "-c", _CONSOLE_SCRIPT],
"python -m raven": [sys.executable, "-P", "-m", "raven"],
}
def _isolated_env(tmp_path: Path) -> dict[str, str]:
"""The test's own environment, minus anything that would point raven elsewhere."""
env = {
key: value
for key, value in os.environ.items()
if not key.lower().endswith("_proxy") and not key.startswith(("SSL_CERT_", "RAVEN_"))
}
env.update(HOME=str(tmp_path / "user"), RAVEN_HOME=str(tmp_path / "home"), RAVEN_NO_UPDATE_CHECK="1")
return env
@pytest.fixture
def corporate(tmp_path):
"""The environment of a shell whose gateway only the operating system vouches for."""
ca = private_ca(tmp_path / "pki")
with https_endpoint(ca.server, OpenAIModels) as origin:
set_provider_fields(
"custom", {"api_key": "k", "api_base": f"{origin}/v1"}, config_path=tmp_path / "home" / "config.json"
)
yield {**_isolated_env(tmp_path), "SSL_CERT_FILE": certifi.where(), "SSL_CERT_DIR": str(ca.store)}
def _provider_test(launch: list[str], env: dict[str, str], cwd: Path) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[*launch, "provider", "test", "custom", "--timeout", "5"],
env=env,
cwd=cwd,
capture_output=True,
text=True,
timeout=120,
)
@pytest.mark.slow
@linux_only
@pytest.mark.parametrize("launch", LAUNCHES.values(), ids=LAUNCHES.keys())
def test_a_root_only_the_system_store_holds_is_trusted(corporate, launch, tmp_path) -> None:
done = _provider_test(launch, corporate, tmp_path)
assert done.returncode == 0, done.stdout + done.stderr
assert "custom OK" in done.stdout
@pytest.mark.slow
@linux_only
@pytest.mark.parametrize("launch", LAUNCHES.values(), ids=LAUNCHES.keys())
def test_opting_out_leaves_the_probe_on_its_bundled_roots(corporate, launch, tmp_path) -> None:
done = _provider_test(launch, {**corporate, "RAVEN_NO_SYSTEM_CA": "1"}, tmp_path)
assert done.returncode == 1, done.stdout + done.stderr
assert "certificate_untrusted" in done.stdout
@pytest.mark.slow
def test_trust_is_settled_before_the_cli_builds_a_context(tmp_path) -> None:
"""aiohttp makes its default context when it is imported, and importing the
CLI reaches it, so a switch made after that import would leave every aiohttp
session on the context built before it. The probe reads that context back.
"""
probe = (
"import sys\n"
"sys.argv = ['raven', '--help']\n"
"from raven.cli import entry\n"
"try:\n"
" entry.run()\n"
"except SystemExit:\n"
" pass\n"
"import aiohttp.connector, truststore\n"
"print('built after the switch:', isinstance(aiohttp.connector._SSL_CONTEXT_VERIFIED, truststore.SSLContext))\n"
)
done = subprocess.run(
[sys.executable, "-P", "-c", probe],
env=_isolated_env(tmp_path),
cwd=tmp_path,
capture_output=True,
text=True,
timeout=120,
)
assert "built after the switch: True" in done.stdout, done.stdout[-2000:] + done.stderr[-2000:]
def test_the_entry_settles_trust_then_hands_the_process_to_the_cli(monkeypatch) -> None:
"""The in-process half of the entry's contract: the switch, then the CLI.
The subprocess tests above show what that order buys; this one runs the
entry in the test process, where its own lines can be measured.
"""
from raven.cli import commands, entry
from raven.security import tls
calls: list[str] = []
monkeypatch.setattr(tls, "use_system_ca", lambda: calls.append("trust"))
monkeypatch.setattr(commands, "run", lambda: calls.append("cli"))
entry.run()
assert calls == ["trust", "cli"]
@pytest.mark.parametrize(
("argv", "taken"),
[
(["raven", "web", "--supervise", "--port", "18999"], ["trust", "supervisor"]),
(["raven", "web", "--supervise", "--port", "x"], ["trust", "cli"]),
],
ids=["the supervisor", "a near miss"],
)
def test_the_supervisor_skips_the_cli_but_not_the_switch(monkeypatch, argv: list[str], taken: list[str]) -> None:
"""`raven web` starts its supervisor through the entry too. That launch is taken
without the command table, after the switch like every other."""
from raven.cli import commands, entry, serve_commands
from raven.security import tls
calls: list[str] = []
monkeypatch.setattr(tls, "use_system_ca", lambda: calls.append("trust"))
monkeypatch.setattr(commands, "run", lambda: calls.append("cli"))
monkeypatch.setattr(serve_commands, "_web", lambda *_a, **_kw: calls.append("supervisor"))
monkeypatch.setattr(sys, "argv", argv)
entry.run()
assert calls == taken
def test_no_cli_module_is_a_third_way_to_start_raven() -> None:
"""The two launches above are the only ones: a CLI module run as a script skips the switch.
Pointing such a guard at ``cli.entry.run`` instead would not help either. By the
time a module's guard runs, its own imports have built aiohttp's context.
"""
import ast
import raven.cli
package = Path(raven.cli.__file__).parent
guarded = sorted(
path.relative_to(package).as_posix()
for path in package.rglob("*.py")
for node in ast.parse(path.read_text(encoding="utf-8")).body
if isinstance(node, ast.If) and ast.unparse(node.test).replace("'", '"') == '__name__ == "__main__"'
)
assert guarded == []