Raven-1/tests/test_subagent_probe.py
Dizhan Xue 14edd7bdb2
feat(*): run the host as an agent manager that hands work, coding included, to sub-agents (#1010)
## Summary

Raven's host acts as the user's agent manager: it delegates production
work, including small edits, to the agent whose ownership or description
fits the task -- or, on a turn bound to a Worker Table, to the worker
whose brief covers it -- keeps track of the task and its instance,
routes follow-ups back to that instance, and gathers the results for the
user. Its own tools support briefing, checking and coordination.

- Add the resident `multi-agent-coding` skill for choosing coding
agents, keeping one instance per task, passing results by file,
isolating concurrent edits in worktrees, and handing over work when an
agent exhausts its quota.
- Keep the new host role authoritative in both the DAG skill's resident
description and its full guide. A single task uses `spawn`, or a
one-node graph when only the DAG dispatch path is available.
- Rename the generic sub-agent to `Raven-General`, with `Raven` and
`raven` retained as compatibility aliases. Instance records, handle
locks, live activity, cancellation, preferences and RPC lookups use the
same canonical identity. Historical spawn and DAG readers normalize
names before liveness checks and serialization, so tasks and transcripts
join the current instance roster. Concurrent alias calls preserve both
turns instead of overwriting one conversation snapshot. The instance
registry reads former-name rows under the current name and, on its next
write, saves every loaded row under it; transcripts, direct chats and
stored graphs stay as written.
- Resolve built-in configuration aliases before merging fields:
`Raven-General` wins over `Raven`, then `raven`, regardless of file
order. Discard older rows wholesale with a warning, preventing stale
model, permission or transport settings from leaking into the selected
row.
- Resume the most recently written message state and instance log across
former names. Saved messages take precedence over an empty chat
directory; equal timestamps prefer the current name and then the most
recent former name. Existing files stay in place. Legacy spawn records,
completed DAG manifests and unfinished graphs remain readable through
the current name.
- Emit the agent roster once at the end of `spawn`'s description. The
registry places `run_subagent_dag` and then `spawn` last, including
after late MCP or session tools. The graph tool renders the roster
itself when `spawn` is unavailable. With a Worker Table bound, dispatch
guidance selects worker labels from the live enum; the roster explains
underlying capabilities. The tool descriptions and resident DAG guide
follow the same rule.
- On a turn bound to a Worker Table, such as a Persona session, `spawn`
and the DAG nodes offer worker labels alone, and a generated Persona
never includes the generic agent. The host role on such a turn routes
production work to the worker whose brief covers it, by label, and puts
work no worker covers to the user, who can add a worker or give the
conversation another Harness, instead of doing it itself or naming an
agent outside the enum. Ordinary turns render byte for byte as before. A
Harness that `load_playbook` binds mid-turn takes effect after the
identity was assembled, so that call's result carries the same routing
as this system's trusted note, after the untrusted fence. The coding
skill picks worker labels on such a turn, for its first pick and for a
quota takeover.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Measured at this head on Linux/Python 3.12.13.

- `UV_NO_SYNC=1 uv run --no-sync pytest -q -n 12`: **33,767 passed, 178
skipped, 2 xfailed; 9 failed and 1 error.** Eight of the failures are
bound to this box and fail the same way on current main, measured the
same day: five provider tests that read the box's proxy environment, a
Node.js path case, a LibreOffice conversion, and a fixture that root can
read anyway. The ninth, with its teardown error, is the `remote_runs`
race tracked in #990; that test passed 5 of 5 runs alone, and its file
58 of 58.
- 13 new regression tests for Worker Table routing: 12 fail against the
previous head's source and all pass now; the thirteenth checks that a
Harness loaded mid-turn leaves the next turn's role plain, and passes on
both by design. The end-to-end case drives a real Persona-bound turn:
both enums offer `research-a` alone, dispatching `Raven-General` is
refused, and the role must name neither that agent nor the roster
fallback.
- Nine single-construct mutants, each applied to an isolated copy of the
tree, each fail at least one new test: the role ignoring the table, the
default path not reading the turn's table, the uncovered-work rule
removed, the generic fallback kept in the worker routing, the coding
skill reverted, and the mid-turn note never added, added on every call,
added without a dispatch path, or phrased apart from the role.
- The host identity on ordinary turns is byte-identical to the previous
head in all 16 rendered shapes: a launched sub-agent process or not,
with or without ownership lines, and every combination of the two
dispatch tools.
- Instance registry probe, run with this head and with main on one
synthetic file: reads leave the file byte-identical; one unrelated
`upsert_spawn` saves every former-name row as `Raven-General`, keeping
the more recently updated of two colliding spellings and the later one
in the file on a tie; main then finds none of those rows under `Raven`
or `raven`.
- `uv run --no-sync python scripts/coverage_gate.py --coverage-json
<coverage.json> diff --base-ref origin/main --threshold 90`, over a
coverage run of 225 test files (8,952 passed): **99.40%**, 167/168
executable changed lines covered.
- `UV_NO_SYNC=1 make lint-types lint-imports check-large-files
check-source-language COMMIT_RANGE=origin/main...HEAD`: passed; all 10
import contracts kept.
- `UV_NO_SYNC=1 make check-commits`: both commit-message validators
passed; each also refused a dangling control commit with a malformed
subject.
- `ruff check` and `ruff format --check` over the branch's Python files
passed. `node ui-tui/scripts/gen-i18n.mjs --check` passed after the
rebase onto #1015, which also changed the i18n catalogue.
- `uv run --no-sync pre-commit run --from-ref origin/main --to-ref
HEAD`: every hook passed except the whole-TUI Prettier hook, which
reports only `newInstancePicker.tsx` and `subagentsHub.tsx`; both are
byte-identical to current main.
- Live-model replay and native Windows/macOS testing were not repeated
for these fixes.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- The host delegates work it previously performed itself, including
small scripts. Task-to-instance routing remains prompt-based; a task can
still stay with the host after it has already answered a follow-up
itself.
- In a conversation bound to a Persona or another Harness with workers,
work no worker's brief covers now waits for the user to add a worker or
change the conversation's Harness, rather than being carried out by the
host.
- Old agent spellings remain accepted. The instance registry's first
write after the upgrade, of any row for any session, saves every
former-name row in `subagent_instances.json` as `Raven-General`, and
keeps only the more recently updated row where two spellings of one
instance collide. Builds from before the rename cannot look those rows
up, so reverting the squash commit alone does not restore their instance
bindings: that needs the registry file as it stood before that first
write. Renaming rows back by hand cannot recover a colliding row the
write dropped. Transcripts, direct chats and stored graphs are not
rewritten.
- When multiple generic-agent configuration spellings coexist, only the
newest row contributes overrides. Fields supplied exclusively by an
older row are intentionally discarded.
- Moving the dispatch tools changes the tool-array cache prefix once
after upgrade.
- A plugin participant that removes `spawn` through `select_tools` can
still leave the graph tool pointing at an absent roster, because that
selection occurs after the registry's offer check. No bundled
participant does this.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-10-20 17:56:05 +08:00

2164 lines
92 KiB
Python

"""Free availability probe for third-party subagents (cli PATH + openai /models)."""
from __future__ import annotations
import asyncio
import json
import os
import socket
from contextlib import closing
from pathlib import Path
import pytest
from aiohttp import web
import raven.agent.subagent.backends.env as env_mod
import raven.agent.subagent.probe as probe_mod
from raven.agent.subagent import instances as instances_mod
from raven.agent.subagent.probe import probe_all, probe_one, run_test
from raven.config.schema import (
Config,
ThirdPartyAcpSubagentConfig,
ThirdPartyCliSubagentConfig,
ThirdPartyOpenAISubagentConfig,
)
from tests._windows_dirs import scratch_root, windows_removal
def _free_port() -> int:
with closing(socket.socket()) as s:
s.bind(("127.0.0.1", 0))
return s.getsockname()[1]
@pytest.fixture(autouse=True)
def _isolated_instance_registry(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""No test here may touch the real ~/.raven/subagent_instances.json.
Same reasoning as the fixture of this name in
`tests/test_subagent_third_party.py`: a stateful create commits a handle
binding through the process-wide `get_registry()` singleton, so without
this every run would accumulate junk rows in the user's real file.
"""
monkeypatch.setattr(
instances_mod, "_registry", instances_mod.InstanceRegistry(path=tmp_path / "_autouse_inst.json")
)
def _cli(command: str, name: str = "agent") -> ThirdPartyCliSubagentConfig:
return ThirdPartyCliSubagentConfig(name=name, command=command)
def _openai(base_url: str, model: str = "m1", api_key: str = "k") -> ThirdPartyOpenAISubagentConfig:
return ThirdPartyOpenAISubagentConfig(name="api", base_url=base_url, model=model, api_key=api_key)
# --- cli probe -----------------------------------------------------------
async def test_cli_probe_reports_the_resolved_absolute_path(tmp_path: Path) -> None:
exe = tmp_path / "faux-agent"
exe.write_text("#!/bin/sh\nexit 0\n")
exe.chmod(0o755)
res = await probe_one(_cli("faux-agent -p {prompt}"), source="config", path=str(tmp_path))
assert res.status == "ready"
assert res.target == str(exe)
assert res.kind == "cli"
assert res.source == "config"
async def test_a_missing_executable_says_what_installs_it(tmp_path: Path) -> None:
"""The install belongs on the surface a person reads, not in the roster prose.
``which`` can only report the executable it looked for, which is not the
package that ships it: ``qodercli`` does not spell ``@qoder-ai/qodercli``.
Carried by the row's provenance rather than its name, so an owner who renamed
the row keeps the hint.
"""
cfg = ThirdPartyAcpSubagentConfig(name="Renamed By Its Owner", preset="qoder", command="qodercli --acp")
res = await probe_one(cfg, source="config", path=str(tmp_path))
assert res.status == "missing"
assert res.target == "qodercli"
assert res.absent == "qodercli", "what the page offers to install is read from this"
assert "npm i -g @qoder-ai/qodercli" in res.detail
async def test_a_missing_executable_invents_no_install_it_does_not_know(tmp_path: Path) -> None:
"""A hand-written row is not the preset whose name it happens to wear.
Its command is the owner's, so naming a package would send them to install
something they never asked for. The executable name already reported is what
they search with instead.
"""
cfg = ThirdPartyAcpSubagentConfig(name="Qoder", command="my-own-agent --acp")
res = await probe_one(cfg, source="config", path=str(tmp_path))
assert res.status == "missing"
assert res.detail == "my-own-agent is not on the login shell PATH"
assert "npm" not in res.detail
async def test_a_row_launched_through_npx_names_node_js_when_npx_is_absent(tmp_path: Path) -> None:
"""The agent's own installer is the wrong answer when what is absent is npx.
Claude Code and Codex launch through ``npx``, which Node.js brings. With npx
taken off one row's PATH (2026-09-23), the page offered
``npm install -g @anthropic-ai/claude-code``: an npm command, on a machine
whose npm went with its Node.js -- and an agent installed some other way
still launches through npx, so the row stayed absent after it anyway.
"""
cfg = ThirdPartyAcpSubagentConfig(
name="Claude Code", preset="claude_code", command="npx -y @agentclientprotocol/claude-agent-acp@0.79.0"
)
res = await probe_one(cfg, source="preset", path=str(tmp_path))
assert res.status == "missing"
assert res.absent == "npx"
assert (
res.detail
== "npx is not on the login shell PATH; install with Node.js from https://nodejs.org, which brings npx"
)
# A cli row whose command is npx is the same absence, whatever it runs.
cli = await probe_one(_cli("npx some-agent {prompt}"), source="config", path=str(tmp_path))
assert cli.absent == "npx"
assert "https://nodejs.org" in cli.detail
async def test_only_a_lookup_that_found_nothing_names_what_is_absent(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``absent`` is what to install, so it must not ride on a ``missing`` that came from a handshake.
A snapshot recorded ``missing`` is an executable that was found and then did
not start. Naming it absent would put an installer in front of a reader who
already has the thing installed.
"""
npx = _fake_executable(tmp_path, "npx")
snapshot = type("Snapshot", (), {"stale": False, "model_menu_measured": True, "status": "missing", "detail": "x"})
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: snapshot)
cfg = ThirdPartyAcpSubagentConfig(
name="Codex", preset="codex", command="npx -y @agentclientprotocol/codex-acp@1.1.14"
)
res = await probe_one(cfg, source="preset", path=str(tmp_path))
assert res.status == "missing"
assert res.target == str(npx)
assert res.absent is None
async def test_cli_probe_reports_missing_and_still_names_what_it_looked_for(tmp_path: Path) -> None:
res = await probe_one(_cli("definitely-not-installed {prompt}"), source="config", path=str(tmp_path))
assert res.status == "missing"
# A `missing` result still names the executable, or the user cannot tell
# which of several tokens in the command was the one not found.
assert res.target == "definitely-not-installed"
assert "definitely-not-installed" in res.detail
@pytest.mark.parametrize("kind", ["cli", "acp"])
async def test_windows_probes_find_a_legacy_single_quoted_program(tmp_path, monkeypatch, kind):
import shlex
from types import SimpleNamespace
from raven.utils import commands
monkeypatch.setattr(commands, "os", SimpleNamespace(name="nt"))
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda _cfg: None)
program = _fake_executable(tmp_path, "saved agent.exe")
command = shlex.quote(str(program))
cfg = _cli(command) if kind == "cli" else ThirdPartyAcpSubagentConfig(name="saved-command", command=command)
result = await probe_one(cfg, source="config", path=str(tmp_path))
assert result.target == str(program)
assert result.status == ("ready" if kind == "cli" else "attention")
assert result.absent is None
@pytest.mark.skipif(os.name == "nt", reason="POSIX shell line: a lone quote is not a parse error to CreateProcess")
async def test_cli_probe_reports_unparseable_command_as_unknown() -> None:
# This single quote belongs to a non-path token, so it stays literal on
# Windows and the probe reports a missing agent rather than invalid syntax.
res = await probe_one(_cli("claude -p 'unbalanced {prompt}"), source="config", path="/usr/bin")
assert res.status == "unknown"
assert "cannot be parsed" in res.detail
async def test_cli_probe_reports_a_placeholder_executable_as_unknown() -> None:
# `{prompt}` as argv[0] names no executable, so "not installed" would be a
# wrong diagnosis of a malformed command.
res = await probe_one(_cli("{prompt} --go"), source="config", path="/usr/bin")
assert res.status == "unknown"
assert "placeholder" in res.detail
async def test_cli_probe_resolves_on_the_login_shell_path_not_ravens(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
# The whole point: `CliAgentBackend._exec` runs the child under
# `login_shell_env()`, so a probe reading os.environ would report an agent
# missing while real dispatches find it (codex lives on the login PATH only).
exe = tmp_path / "login-only-agent"
exe.write_text("#!/bin/sh\nexit 0\n")
exe.chmod(0o755)
monkeypatch.setenv("PATH", "/nonexistent")
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
res = await probe_one(_cli("login-only-agent {prompt}"), source="config")
assert res.status == "ready"
async def test_cli_probe_honors_a_configured_path_override(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
# CliAgentBackend._exec builds the child's env as {**login_shell_env(), **self.env}
# (cli_agent.py:139-140), so a configured PATH wins over the login shell's for a
# real spawn. The probe must resolve the same way, or a subagent with an env PATH
# override would be reported missing while a real spawn finds it fine.
login_only = tmp_path / "login_only"
login_only.mkdir()
override = tmp_path / "override"
override.mkdir()
exe = override / "custom-agent"
exe.write_text("#!/bin/sh\nexit 0\n")
exe.chmod(0o755)
cfg = ThirdPartyCliSubagentConfig(name="agent", command="custom-agent {prompt}", env={"PATH": str(override)})
res = await probe_one(cfg, source="config", path=str(login_only))
assert res.status == "ready"
assert res.target == str(exe)
# --- openai probe --------------------------------------------------------
async def _serve(handler, route: str = "/v1/models", method: str = "GET") -> tuple[str, web.AppRunner, list[str]]:
"""Start a stub endpoint; returns (base_url, runner, recorded request paths)."""
seen: list[str] = []
async def wrapped(request: web.Request):
seen.append(request.path)
return await handler(request)
port = _free_port()
app = web.Application()
app.router.add_route(method, route, wrapped)
runner = web.AppRunner(app)
await runner.setup()
await web.TCPSite(runner, "127.0.0.1", port).start()
return f"http://127.0.0.1:{port}/v1", runner, seen
async def test_openai_probe_ready_when_the_model_is_listed() -> None:
async def handler(request: web.Request) -> web.Response:
return web.json_response({"object": "list", "data": [{"id": "m1"}, {"id": "m2"}]})
base, runner, _ = await _serve(handler)
try:
res = await probe_one(_openai(base, model="m1"), source="config")
assert res.status == "ready"
assert res.target == base
finally:
await runner.cleanup()
async def test_openai_probe_flags_a_model_absent_from_the_list() -> None:
async def handler(request: web.Request) -> web.Response:
return web.json_response({"data": [{"id": "m1"}, {"id": "m2"}]})
base, runner, _ = await _serve(handler)
try:
res = await probe_one(_openai(base, model="typo-model"), source="config")
assert res.status == "attention"
assert "typo-model" in res.detail
finally:
await runner.cleanup()
async def test_openai_probe_is_ready_but_honest_when_the_body_has_no_model_list() -> None:
async def handler(request: web.Request) -> web.Response:
return web.json_response({"ok": True})
base, runner, _ = await _serve(handler)
try:
res = await probe_one(_openai(base), source="config")
assert res.status == "ready"
assert "unverified" in res.detail
finally:
await runner.cleanup()
async def test_openai_probe_reports_a_rejected_key() -> None:
async def handler(request: web.Request) -> web.Response:
return web.json_response({"error": "invalid api key"}, status=401)
base, runner, _ = await _serve(handler)
try:
res = await probe_one(_openai(base), source="config")
assert res.status == "attention"
assert "401" in res.detail
finally:
await runner.cleanup()
async def test_openai_probe_reports_a_missing_models_endpoint_as_unverified() -> None:
async def handler(request: web.Request) -> web.Response:
return web.Response(status=404)
base, runner, _ = await _serve(handler)
try:
res = await probe_one(_openai(base), source="config")
assert res.status == "attention"
assert "unverified" in res.detail
finally:
await runner.cleanup()
async def test_openai_probe_reports_an_unreachable_endpoint_as_missing() -> None:
res = await probe_one(_openai(f"http://127.0.0.1:{_free_port()}/v1"), source="config")
assert res.status == "missing"
assert "unreachable" in res.detail
async def test_openai_probe_honors_env_proxy(monkeypatch: pytest.MonkeyPatch) -> None:
# Same guarantee `test_openai_backend_honors_env_proxy` pins for the backend,
# and it must hold here too: aiohttp ignores HTTP_PROXY unless trust_env is
# set, so on a host that can only reach the provider through a proxy a probe
# without it would report a working agent unreachable.
async def handler(request: web.Request) -> web.Response:
return web.json_response({"data": [{"id": "m1"}]})
proxy_base, runner, _ = await _serve(handler, route="/{tail:.*}")
dead_port = _free_port()
for var in ("no_proxy", "NO_PROXY", "https_proxy", "HTTPS_PROXY", "HTTP_PROXY"):
monkeypatch.delenv(var, raising=False)
monkeypatch.setenv("http_proxy", proxy_base.removesuffix("/v1"))
try:
res = await probe_one(_openai(f"http://127.0.0.1:{dead_port}/v1", model="m1"), source="config")
assert res.status == "ready"
finally:
await runner.cleanup()
async def test_openai_probe_sends_nothing_for_a_keyless_preset() -> None:
async def handler(request: web.Request) -> web.Response:
return web.json_response({"data": []})
base, runner, seen = await _serve(handler)
try:
res = await probe_one(_openai(base, api_key=""), source="preset")
assert res.status == "attention"
assert seen == [] # a template must not fire a request that is certain to 401
finally:
await runner.cleanup()
async def test_openai_probe_does_send_for_a_keyless_configured_entry() -> None:
# A keyless endpoint is legitimate (a local vLLM), so short-circuiting a
# configured entry would report a working agent as broken.
async def handler(request: web.Request) -> web.Response:
return web.json_response({"data": [{"id": "m1"}]})
base, runner, seen = await _serve(handler)
try:
res = await probe_one(_openai(base, api_key=""), source="config")
assert res.status == "ready"
assert seen == ["/v1/models"]
finally:
await runner.cleanup()
# --- capability limits ---------------------------------------------------
def test_openai_config_coerces_declared_local_file_access_away() -> None:
# `reads_local_files` is rendered into the roster that closes the spawn tool's
# description as a `local-files` tag, so honouring it here would licence the dispatching model
# to hand this agent a path that nothing can open. It is coerced rather than
# rejected because the older web form defaulted the checkbox to true and
# rendered it for both kinds: rejecting on load would raise inside the whole
# top-level Config for anyone with such an entry stored, so raven would stop
# starting and the UI that could fix the field sits behind that config.
cfg = ThirdPartyOpenAISubagentConfig(name="api", base_url="http://x/v1", model="m", reads_local_files=True)
assert cfg.reads_local_files is False
def test_a_stored_openai_entry_with_local_file_access_still_loads() -> None:
# The regression this guards is startup, not the field: one such entry used
# to fail the entire Config, taking every unrelated section with it.
config = Config.model_validate(
{
"subagents": {
"thirdParty": [
{
"name": "legacy",
"kind": "openai",
"baseUrl": "http://x/v1",
"model": "m",
"readsLocalFiles": True,
}
]
}
}
)
assert config.subagents.agents[0].reads_local_files is False
def test_openai_config_accepts_false_and_omitted() -> None:
assert ThirdPartyOpenAISubagentConfig(name="a", base_url="http://x/v1", model="m").reads_local_files is False
explicit = ThirdPartyOpenAISubagentConfig(name="b", base_url="http://x/v1", model="m", reads_local_files=False)
assert explicit.reads_local_files is False
def test_cli_config_still_accepts_local_file_access() -> None:
# A cli agent is a local subprocess, so the declaration is real there.
assert ThirdPartyCliSubagentConfig(name="c", command="echo {prompt}", reads_local_files=True).reads_local_files
# --- batch ---------------------------------------------------------------
async def test_probe_all_preserves_order_and_captures_the_path_once(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
calls: list[int] = []
def counted() -> str:
calls.append(1)
return str(tmp_path)
monkeypatch.setattr(probe_mod, "_login_path", counted)
entries = [(_cli("a {prompt}", name="a"), "config"), (_cli("b {prompt}", name="b"), "preset")]
results = await probe_all(entries)
assert [r.name for r in results] == ["a", "b"]
assert [r.source for r in results] == ["config", "preset"]
# login_shell_env shells out to `bash -lic` and can block for seconds on its
# first call; per-agent capture would serialise the whole batch behind it.
assert len(calls) == 1
async def test_probe_all_survives_a_raising_login_path_capture(monkeypatch: pytest.MonkeyPatch) -> None:
# Neither probe_one nor probe_all may raise, on any path -- including the
# shared PATH capture, which today only degrades (login_shell_env falls
# back to os.environ) but is not guaranteed to.
def boom() -> str:
raise OSError("no bash")
monkeypatch.setattr(probe_mod, "_login_path", boom)
entries = [(_cli("a {prompt}", name="a"), "config"), (_cli("b {prompt}", name="b"), "preset")]
results = await probe_all(entries)
assert [r.name for r in results] == ["a", "b"]
for res in results:
assert res.status in {"ready", "attention", "missing", "unknown"}
async def test_probe_result_wire_shape_is_camel_case(tmp_path: Path) -> None:
res = await probe_one(_cli("nope {prompt}"), source="config", path=str(tmp_path))
assert set(res.to_wire()) == {"name", "source", "kind", "status", "detail", "target", "elapsedMs", "lastTest"}
async def test_probe_all_attaches_a_verdict_to_the_matching_result(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
from raven.agent.subagent.probe_state import LastTest
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
a = _cli("a {prompt}", name="a")
b = _cli("b {prompt}", name="b")
verdicts = {"config:a": LastTest(ok=False, detail="auth failed", tested_at_ms=1700)}
results = await probe_all([(a, "config"), (b, "config")], verdicts=verdicts)
assert results[0].last_test == LastTest(ok=False, detail="auth failed", tested_at_ms=1700)
assert results[1].last_test is None
async def test_probe_wire_shape_carries_last_test(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
from raven.agent.subagent.probe_state import LastTest
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
cfg = _cli("nope {prompt}", name="a")
[bare] = await probe_all([(cfg, "config")])
assert bare.to_wire()["lastTest"] is None
[withv] = await probe_all([(cfg, "config")], verdicts={"config:a": LastTest(True, "ran", 42)})
assert withv.to_wire()["lastTest"] == {"ok": True, "detail": "ran", "testedAtMs": 42}
# --- explicit test -------------------------------------------------------
def _patch_login_env_for_spawn(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""Point both the probe's PATH read and a real spawn's child env at ``tmp_path``.
Patching ``probe_mod._login_path`` alone (as the probe-only tests above do)
only steers ``probe_one``: ``CliAgentBackend._exec`` builds its child's
environment from ``login_shell_env()`` directly, never through anything the
probe passes, so a run that reaches a real spawn needs the shared cache both
layers read from patched instead - the same pattern
``test_subagent_third_party.py``'s ``_clear_login_env_cache`` fixture uses.
"""
monkeypatch.setattr(
env_mod,
"_LOGIN_ENV",
{"PATH": f"{tmp_path}:/usr/bin:/bin", "HOME": os.environ.get("HOME", "/root")},
)
async def test_test_of_a_missing_cli_fails_without_building_a_backend(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
# A failed probe must short-circuit: there is nothing to execute, so no
# backend is built and no process is spawned.
def must_not_build(*args, **kwargs):
raise AssertionError("a failed probe must not reach the backend")
monkeypatch.setattr(probe_mod, "build_third_party_backend", must_not_build)
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
res = await run_test(_cli("not-installed-at-all {prompt}"), source="config")
assert res.ok is False
assert res.kind == "cli"
assert "not on the login shell PATH" in res.detail
async def test_test_of_a_working_cli_returns_the_reply(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
exe = tmp_path / "echo-agent"
exe.write_text("#!/bin/sh\necho PONG\n")
exe.chmod(0o755)
_patch_login_env_for_spawn(monkeypatch, tmp_path)
res = await run_test(_cli("echo-agent {prompt}"), source="config")
assert res.ok is True
assert res.reply == "PONG"
assert res.elapsed_ms >= 0
async def test_test_of_a_cli_that_returns_nothing_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
# Exit 0 with an empty answer is exactly the openclaw symptom of replying to
# its workspace bootstrap instead of the task, so it is not a pass.
exe = tmp_path / "silent-agent"
exe.write_text("#!/bin/sh\nexit 0\n")
exe.chmod(0o755)
_patch_login_env_for_spawn(monkeypatch, tmp_path)
res = await run_test(_cli("silent-agent {prompt}"), source="config")
assert res.ok is False
assert "returned nothing" in res.detail
async def test_test_of_a_failing_cli_reports_its_error_rather_than_raising(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
exe = tmp_path / "broken-agent"
exe.write_text("#!/bin/sh\necho 'ProviderAuthError' >&2\nexit 1\n")
exe.chmod(0o755)
_patch_login_env_for_spawn(monkeypatch, tmp_path)
res = await run_test(_cli("broken-agent {prompt}"), source="config")
assert res.ok is False
assert "ProviderAuthError" in res.detail
@pytest.mark.skipif(os.name == "nt", reason="the stand-in CLI is a POSIX shell script")
async def test_a_cli_test_whose_directory_stays_in_use_reports_the_reply(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""On Windows a process the CLI left running keeps the directory; the test reports what the CLI said."""
exe = tmp_path / "echo-agent"
exe.write_text("#!/bin/sh\necho PONG\n")
exe.chmod(0o755)
_patch_login_env_for_spawn(monkeypatch, tmp_path)
root = scratch_root(tmp_path, monkeypatch)
refused = windows_removal(monkeypatch, lambda path: path.parent == root)
res = await run_test(_cli("echo-agent {prompt}"), source="config")
assert (res.ok, res.reply) == (True, "PONG")
assert refused, "no removal was refused, so this tested nothing"
async def test_stateful_cli_test_leaves_the_real_registry_untouched(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
exe = tmp_path / "stateful-agent"
exe.write_text("#!/bin/sh\necho PONG\n")
exe.chmod(0o755)
_patch_login_env_for_spawn(monkeypatch, tmp_path)
cfg = ThirdPartyCliSubagentConfig(
name="stateful",
command="stateful-agent {prompt} --session-id {agent_id}",
resume_command="stateful-agent {prompt} --resume {agent_id}",
)
res = await run_test(cfg, source="config")
assert res.ok is True
# The create commits a handle binding; it must land in the probe's throwaway
# registry, not in the one the running gateway reads.
assert instances_mod._registry.list_instances() == []
async def test_an_openai_test_asks_the_endpoint_to_answer() -> None:
"""Both halves, in this order, and the order is the whole economy of it.
The free models probe goes first and refuses on its own when the endpoint
does not know the credential, so nothing is spent on one that cannot answer.
Past it, the endpoint is asked the same prompt every other kind is asked,
because knowing the credential is not the same as being able to serve the
model behind it.
"""
async def models(request: web.Request) -> web.Response:
return web.json_response({"data": [{"id": "m1"}]})
seen: list[str] = []
port = _free_port()
app = web.Application()
async def record_models(request: web.Request) -> web.Response:
seen.append(request.path)
return await models(request)
async def record_completion(request: web.Request) -> web.Response:
seen.append(request.path)
return web.json_response({"choices": [{"message": {"content": "billed!"}}]})
app.router.add_get("/v1/models", record_models)
app.router.add_post("/v1/chat/completions", record_completion)
runner = web.AppRunner(app)
await runner.setup()
await web.TCPSite(runner, "127.0.0.1", port).start()
try:
res = await run_test(_openai(f"http://127.0.0.1:{port}/v1", model="m1"), source="config")
assert res.ok is True
assert res.reply is None
assert seen == ["/v1/models", "/v1/chat/completions"]
finally:
await runner.cleanup()
async def test_an_openai_test_asks_an_endpoint_that_serves_no_model_list() -> None:
"""The free probe vetoes only when it has proved there is nothing to send to.
`/models` is optional: the backend only ever POSTs `/chat/completions`, so an
endpoint that serves completions and answers 404 for the list is a working
agent. Letting the probe veto it would fail a Test that Connect accepts --
the very disagreement this gate exists to remove -- so anything reachable is
settled by asking it.
"""
seen: list[str] = []
port = _free_port()
app = web.Application()
async def no_model_list(request: web.Request) -> web.Response:
seen.append(request.path)
return web.json_response({"error": "not found"}, status=404)
async def record_completion(request: web.Request) -> web.Response:
seen.append(request.path)
return web.json_response({"choices": [{"message": {"content": "PONG"}}]})
app.router.add_get("/v1/models", no_model_list)
app.router.add_post("/v1/chat/completions", record_completion)
runner = web.AppRunner(app)
await runner.setup()
await web.TCPSite(runner, "127.0.0.1", port).start()
try:
res = await run_test(_openai(f"http://127.0.0.1:{port}/v1", model="m1"), source="config")
assert res.ok is True, res.detail
assert seen == ["/v1/models", "/v1/chat/completions"]
finally:
await runner.cleanup()
async def test_an_openai_test_spends_nothing_on_an_endpoint_it_cannot_reach() -> None:
"""The one verdict the free probe can reach alone: nothing is listening, so
there is no request to make and no quota to consider."""
port = _free_port()
res = await run_test(_openai(f"http://127.0.0.1:{port}/v1", model="m1"), source="config")
assert res.ok is False
assert "unreachable" in res.detail
async def test_test_result_wire_shape_is_camel_case(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
wire = (await run_test(_cli("nope {prompt}"), source="config")).to_wire()
assert set(wire) == {"name", "source", "kind", "ok", "detail", "reply", "elapsedMs"}
async def test_testing_a_builtin_agent_is_refused_rather_than_attempted() -> None:
"""There is nothing a test could check about an in-process agent.
No command to launch, no endpoint to authenticate against, no transcript
parser to exercise -- the three things a test exists to catch. Falling through
to the cli branch did run, though: the probe now reports a built-in row
"ready", so it reached ``build_third_party_backend``, which raises on the kind,
and the bare-except turned that into a verdict claiming ``kind == "cli"`` with
the detail "unknown third-party subagent kind". The caller then *records*
verdicts, so that row would have shown a failed test permanently.
"""
from raven.agent.subagent.probe import run_test
from raven.config.schema import BuiltinAgentConfig
result = await run_test(BuiltinAgentConfig(name="research-raven"), source="config")
assert result.kind == "builtin"
assert result.ok is False
assert "nothing to test" in result.detail
assert "unknown third-party" not in result.detail
# --- ping ----------------------------------------------------------------
async def test_ping_dispatches_the_probe_prompt_and_passes_on_a_reply(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Layer 2 is a real turn, because layer 1 cannot see a credential.
Measured 2026-09-07: six registry agents answered `initialize` and
`session/new` and then failed the first model call, which is where their
credential is actually checked.
"""
sent: list[str] = []
class _Answers:
async def run(self, prompt, **kwargs):
sent.append(prompt)
return "PONG"
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Answers())
cfg = ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp")
result = await probe_mod.ping_agent(cfg)
assert sent == [probe_mod.PROBE_PROMPT]
assert result.ok is True
assert "replied" in result.detail
async def test_ping_fails_when_the_agent_answers_nothing(monkeypatch: pytest.MonkeyPatch) -> None:
"""An empty reply is a signal, not a pass -- openclaw answered its own
workspace bootstrap instead of the task and returned nothing."""
class _Silent:
async def run(self, prompt, **kwargs):
return " "
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Silent())
cfg = ThirdPartyAcpSubagentConfig(name="silent-acp", kind="acp", command="fake-agent acp")
result = await probe_mod.ping_agent(cfg)
assert result.ok is False
assert "answered nothing" in result.detail
async def test_ping_fails_and_keeps_the_reason_when_the_model_call_is_refused(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The exact shape six measured agents have: layer 1 fine, model call refused.
The reason has to survive into the detail, because it is the whole value of
the failure -- "sign in" and "-32603" send an operator to different places.
"""
class _Refuses:
async def run(self, prompt, **kwargs):
raise RuntimeError("Internal error: You need to sign in to use this model.")
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Refuses())
cfg = ThirdPartyAcpSubagentConfig(name="unauthed-acp", kind="acp", command="fake-agent acp")
result = await probe_mod.ping_agent(cfg)
assert result.ok is False
assert "sign in" in result.detail
async def test_ping_fails_on_a_timeout_rather_than_waiting(monkeypatch: pytest.MonkeyPatch) -> None:
"""gemini and dirac hang rather than refuse, so an unbounded wait would sit
on the settings switch and then say nothing useful."""
class _Hangs:
async def run(self, prompt, **kwargs):
await asyncio.sleep(30)
return "too late"
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Hangs())
monkeypatch.setattr(probe_mod, "_ENABLE_PING_TIMEOUT_SECONDS", 1)
# The wait is the start window plus the answer's (`_ping_bounds`), so the
# row declares a short start window of its own: the hang has to outlast both.
cfg = ThirdPartyAcpSubagentConfig(name="hanging-acp", kind="acp", command="fake-agent acp", ready_timeout_ms=1000)
result = await probe_mod.ping_agent(cfg)
assert result.ok is False
assert "did not answer within 2s" in result.detail
async def test_ping_hands_the_backend_the_same_bound_it_waits_for(monkeypatch: pytest.MonkeyPatch) -> None:
"""One number, or the inner bound is unreachable: a backend allowed the
longer explicit-Test budget can only ever be cancelled by the `wait_for`
around it, never reach its own timeout and describe the failure.
This half asserts what `ping_agent` asks for, and can assert nothing more:
the builder is replaced below, so a builder that accepted both numbers and
used neither would satisfy every line of it. The other half is
`test_the_factory_honours_the_bounds_its_caller_overrides`, which builds for
real and reads them back off the backend."""
seen: dict = {}
class _Answers:
async def run(self, prompt, **kwargs):
return "PONG"
def _capture(cfg, **kw):
seen.update(kw)
return _Answers()
monkeypatch.setattr(probe_mod, "build_third_party_backend", _capture)
await probe_mod.ping_agent(ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp"))
assert seen["timeout"] == probe_mod._ENABLE_PING_TIMEOUT_SECONDS
# A row that asks for less than the cap still gets its own number.
tight = ThirdPartyAcpSubagentConfig(name="ping-tight", kind="acp", command="fake-agent acp", timeout=5)
await probe_mod.ping_agent(tight)
assert seen["timeout"] == 5
# The handshake gets the row's own start window, uncapped: several shipped
# presets declare 120s because `npx` downloads the adapter there on a first
# connect, and npm takes about 70s to report a registry it cannot reach. Cut
# to the answer's cap, both came back "did not answer within 60s".
slow_start = ThirdPartyAcpSubagentConfig(
name="ping-slow-start", kind="acp", command="fake-agent acp", ready_timeout_ms=120000
)
waited: list[float] = []
real_wait_for = asyncio.wait_for
async def _spy(awaitable, timeout):
waited.append(timeout)
return await real_wait_for(awaitable, timeout)
monkeypatch.setattr(probe_mod.asyncio, "wait_for", _spy)
await probe_mod.ping_agent(slow_start)
assert seen["ready_timeout_ms"] == 120000
# ...and the wait around the run leaves room for both inner bounds, so
# neither is cancelled from outside before the backend can say which ran out.
assert waited == [120 + probe_mod._ENABLE_PING_TIMEOUT_SECONDS]
quick_start = ThirdPartyAcpSubagentConfig(
name="ping-quick-start", kind="acp", command="fake-agent acp", ready_timeout_ms=4000
)
await probe_mod.ping_agent(quick_start)
assert seen["ready_timeout_ms"] == 4000
assert waited[-1] == 4 + probe_mod._ENABLE_PING_TIMEOUT_SECONDS
# A kind with no handshake is waited on for its answer alone.
cli = ThirdPartyCliSubagentConfig(name="ping-cli", kind="cli", command="fake-agent {prompt}")
await probe_mod.ping_agent(cli)
assert waited[-1] == probe_mod._ENABLE_PING_TIMEOUT_SECONDS
async def test_ping_runs_on_a_connection_pool_of_its_own(monkeypatch: pytest.MonkeyPatch) -> None:
"""A ping must not reach the pool the running roster dispatches on.
The pool keys a connection on its launch arguments, and ``cwd`` falls back
to the caller's workspace -- which for a ping is a fresh temporary directory
every time. So a ping on the shared pool always computes a launch key no
held connection matches, and ``acquire`` retires every connection of that
agent before launching its own. Two consequences, both measured against a
live adapter on 2026-09-20: a second Connect pressed while the first was
still running killed the first one's connection, and the first came back
"did not answer a test message" for a failure raven had caused; and a ping
fired while that agent was serving a real run would have taken that run's
connection down with it.
This half asserts what ``ping_agent`` asks the factory for. The other half
is ``test_the_factory_honours_the_pool_its_caller_overrides``.
"""
from raven.acp_client.pool import AcpConnectionPool, get_pool
seen: dict = {}
class _Answers:
async def run(self, prompt, **kwargs):
return "PONG"
def _capture(cfg, **kw):
seen.update(kw)
return _Answers()
monkeypatch.setattr(probe_mod, "build_third_party_backend", _capture)
await probe_mod.ping_agent(ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp"))
assert isinstance(seen["pool"], AcpConnectionPool)
assert seen["pool"] is not get_pool()
async def test_ping_closes_the_pool_it_opened(monkeypatch: pytest.MonkeyPatch) -> None:
"""Or every press leaks the agent process it launched.
The private pool above is per-ping, so nothing else will ever close it: a
pool left open holds a live child process for the rest of the gateway's
life, and the roster would accumulate one per Connect pressed.
"""
import raven.acp_client.pool as pool_mod
closed: list[int] = []
class _Counting(pool_mod.AcpConnectionPool):
async def close_all(self) -> None:
closed.append(1)
await super().close_all()
monkeypatch.setattr(pool_mod, "AcpConnectionPool", _Counting)
class _Answers:
async def run(self, prompt, **kwargs):
return "PONG"
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Answers())
await probe_mod.ping_agent(ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp"))
assert closed == [1]
# And on the failing path too, which is the one that runs when an agent is
# the reason the press is happening at all.
class _Refuses:
async def run(self, prompt, **kwargs):
raise RuntimeError("Internal error: You need to sign in to use this model.")
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Refuses())
result = await probe_mod.ping_agent(
ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp")
)
assert result.ok is False
assert closed == [1, 1]
async def test_a_directory_its_agent_will_not_let_go_does_not_fail_a_ping_that_answered(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The verdict is the agent's answer; the directory is logged and left, not reported in its place.
A refused removal raised out of the ping was reported as "did not answer a
test message" about an agent that had answered: on native Windows that is
every Connect of an agent whose processes still worked in the directory.
"""
root = scratch_root(tmp_path, monkeypatch)
class _Answers:
async def run(self, prompt, **kwargs):
return "PONG"
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Answers())
refused = windows_removal(monkeypatch, lambda path: path.parent == root)
result = await probe_mod.ping_agent(
ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp")
)
assert result == probe_mod.PingResult(True, "it ran and replied")
assert refused, "no removal was refused, so this tested nothing"
assert [path.name.startswith("raven_subagent_ping_") for path in root.iterdir()] == [True]
async def test_a_ping_that_runs_out_says_so_while_its_directory_is_still_in_use(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A refusal raised on the way out took the timeout's place, so a silent agent read as a file error."""
root = scratch_root(tmp_path, monkeypatch)
class _Hangs:
async def run(self, prompt, **kwargs):
await asyncio.sleep(30)
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Hangs())
monkeypatch.setattr(probe_mod, "_ENABLE_PING_TIMEOUT_SECONDS", 0.2)
refused = windows_removal(monkeypatch, lambda path: path.parent == root)
cfg = ThirdPartyAcpSubagentConfig(name="hanging-acp", kind="acp", command="fake-agent acp", ready_timeout_ms=100)
result = await probe_mod.ping_agent(cfg)
assert result.ok is False
assert result.detail.startswith("it did not answer within"), result.detail
assert refused, "no removal was refused, so this tested nothing"
async def test_a_ping_with_nowhere_to_make_its_directory_says_so_rather_than_raising(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
import tempfile
monkeypatch.setattr(tempfile, "tempdir", str(tmp_path / "gone"))
def must_not_build(*args, **kwargs):
raise AssertionError("there is no directory to hand the agent")
monkeypatch.setattr(probe_mod, "build_third_party_backend", must_not_build)
result = await probe_mod.ping_agent(
ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp")
)
assert result.ok is False
assert "gone" in result.detail
def test_the_factory_honours_the_pool_its_caller_overrides() -> None:
"""Read off a real build, for the reason the bounds test states: a request
recorded by a replaced builder proves nothing about what the backend then
dispatches on."""
from raven.acp_client.pool import AcpConnectionPool, get_pool
from raven.agent.subagent.backends import build_third_party_backend
cfg = ThirdPartyAcpSubagentConfig(name="factory-pool", kind="acp", command="fake-agent acp")
private = AcpConnectionPool()
assert build_third_party_backend(cfg, pool=private).pool is private
# Omitted, and a dispatch goes where every other dispatch goes.
assert build_third_party_backend(cfg).pool is get_pool()
def test_the_factory_honours_the_bounds_its_caller_overrides() -> None:
"""Both overrides exist for one caller, so both are read off a real build.
Asserting on the arguments a replaced builder recorded proves the request and
not the effect: dropping either honouring line left the whole subagent suite
green. Read them back off the object instead, and the test stays true of the
next bound the factory learns to take.
"""
from raven.agent.subagent.backends import build_third_party_backend
cfg = ThirdPartyAcpSubagentConfig(
name="factory-bounds",
kind="acp",
command="fake-agent acp",
ready_timeout_ms=120000,
timeout=300,
)
backend = build_third_party_backend(cfg, ready_timeout_ms=45000, timeout=60)
assert backend.ready_timeout_ms == 45000
assert backend.timeout == 60
# Omitted, and the config's own numbers stand -- the overrides are for the
# caller that has a shorter budget, not a rewrite of the row.
untouched = build_third_party_backend(cfg)
assert untouched.ready_timeout_ms == 120000
assert untouched.timeout == 300
def _host_with_a_provider_key() -> None:
"""Give the isolated home one usable provider.
The empty-menu re-measure is bounded on this: a child raven advertises no
model at all while the catalogue it inherits is empty, so a host with no
credentials would be relaunching children to be told what it already knows.
Written to the real config the bound reads rather than stubbing the check,
since "which sections count as usable" is half of what is under test.
"""
from raven.home import get_config_path
get_config_path().write_text(json.dumps({"providers": {"openai": {"apiKey": "sk-test"}}}), encoding="utf-8")
class _FakeRow:
def __init__(self, name: str, *, kind: str = "acp", enabled: bool = True, config: object = None) -> None:
self.name = name
self.kind = kind
self.enabled = enabled
self.config = config if config is not None else type("C", (), {"name": name})()
class _FakeManager:
def __init__(self, rows: list[_FakeRow], *, with_refresh: bool = False) -> None:
self.registry = type("R", (), {"rows": lambda self: rows})()
if with_refresh:
self.refresh_calls = 0
def refresh_agents() -> None:
self.refresh_calls += 1
self.refresh_agents = refresh_agents
class TestAutomaticSnapshotVerification:
async def test_only_missing_or_stale_rows_are_verified(self, monkeypatch: pytest.MonkeyPatch) -> None:
from dataclasses import dataclass
from raven.agent.subagent.probe import schedule_snapshot_verification
@dataclass
class Snap:
status: str = "ready"
stale: bool = False
fresh, missing, stale = _FakeRow("fresh"), _FakeRow("missing"), _FakeRow("stale")
recorded: list[str] = []
def fake_snapshot_for(cfg: object) -> Snap | None:
return {id(fresh.config): Snap(), id(missing.config): None, id(stale.config): Snap(stale=True)}[id(cfg)]
async def fake_verify(cfg: object) -> Snap:
return Snap()
monkeypatch.setattr(probe_mod, "acp_snapshot_for", fake_snapshot_for)
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(lambda s: recorded.append(getattr(s, "status")))})(),
)
# This one pins the registry half. The preset half reads the machine's
# own PATH, so leaving it live would make the assertions below depend on
# which agents happen to be installed here.
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
manager = _FakeManager([fresh, missing, stale], with_refresh=True)
task = schedule_snapshot_verification(manager)
assert task is not None
assert task in probe_mod._VERIFY_TASKS, "an unrefed task can be collected mid-run"
await task
assert task not in probe_mod._VERIFY_TASKS, "the done callback must release the reference"
assert len(recorded) == 2 # both missing and stale round-tripped to a fresh snapshot
assert manager.refresh_calls == 1, "the materialized rows must be rebuilt after recording"
async def test_a_credential_refusal_is_recorded_and_not_only_a_pass(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""Recording only ``ready`` meant the one verdict worth remembering was
the one thrown away.
An agent that answers the handshake and then refuses to open a session
without a credential has told us something durable about itself, and the
agents page reads exactly that to say `Unauthorized` instead of offering
a Connect that spends a launch to be refused again. Any other failure is
recorded only over a row whose last record had already failed, replacing
that verdict and keeping its capabilities. Over a row with no record, as
here, or a passing one, a timeout or a crashed adapter stays unrecorded:
it is a fact about this minute, and freezing it into a snapshot would
label a working agent broken until someone pressed Test.
"""
from raven.agent.subagent.probe import schedule_snapshot_verification
recorded: list[str] = []
verdicts = {
"refused": type("S", (), {"status": "attention", "needs_auth": True})(),
"wedged": type("S", (), {"status": "unknown", "needs_auth": False})(),
"fine": type("S", (), {"status": "ready", "needs_auth": False})(),
}
async def fake_verify(cfg: object) -> object:
return verdicts[cfg.name]
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(lambda s: recorded.append(s.status))})(),
)
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
await schedule_snapshot_verification(_FakeManager([_FakeRow("refused"), _FakeRow("wedged"), _FakeRow("fine")]))
assert sorted(recorded) == ["attention", "ready"]
@staticmethod
def _mirasim_boot(
monkeypatch: pytest.MonkeyPatch, snapshot: object, installed: frozenset[str], own_host: str | None = None
) -> tuple[list, list]:
"""A boot whose unconfigured presets are three Mirasim agents and one other; what it verified and asked.
``own_host`` names the one Mirasim row whose ``env`` points it at a host of its own.
"""
from raven.agent.subagent import mirasim_presets
from raven.agent.subagent.presets import third_party_subagent_preset
from raven.config.schema import SubagentsConfig
keys = ("mirasim_claude", "mirasim_zcode", "mirasim_qwen")
rows = [
{**third_party_subagent_preset(key), **({"env": {"RAVEN_MIRASIM_HOST": "4971"}} if key == own_host else {})}
for key in keys
]
configs = SubagentsConfig(agents=rows).agents
verified: list[str] = []
asked: list[str | None] = []
async def fake_verify(cfg: object) -> object:
verified.append(cfg.name)
return type("S", (), {"status": "ready", "needs_auth": False})()
async def fake_installed(*, path: str | None) -> frozenset[str]:
asked.append(path)
return installed
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: snapshot)
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
)
monkeypatch.setattr(
probe_mod,
"_unconfigured_acp_preset_rows",
lambda configured, path=None: [*(probe_mod._PresetRow(cfg) for cfg in configs), _FakeRow("a-preset")],
)
monkeypatch.setattr(mirasim_presets, "installed_agents", fake_installed, raising=False)
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
return verified, asked
async def test_a_mirasim_preset_whose_agent_mirasim_lacks_is_not_handshaken(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Its handshake fails and a preset's failure is not kept, so verifying it would handshake it again at
every start: two adapter launches or more for nothing, per such agent. Mirasim is asked once
instead."""
from raven.agent.subagent.probe import schedule_snapshot_verification
verified, asked = self._mirasim_boot(monkeypatch, None, frozenset({"claude", "qwen"}))
await schedule_snapshot_verification(_FakeManager([]))
assert verified == ["Claude Code-Mirasim", "Qwen Code-Mirasim", "a-preset"]
assert len(asked) == 1
async def test_with_no_mirasim_host_running_only_a_row_that_names_its_own_host_is_handshaken(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Every other handshake would fail on the same missing host, and be tried again at the next start."""
from raven.agent.subagent.probe import schedule_snapshot_verification
verified, asked = self._mirasim_boot(monkeypatch, None, frozenset(), own_host="mirasim_qwen")
await schedule_snapshot_verification(_FakeManager([]))
assert verified == ["Qwen Code-Mirasim", "a-preset"]
assert len(asked) == 1
async def test_mirasim_is_asked_only_when_one_of_its_presets_is_to_be_handshaken(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The question launches the adapter and waits out the host's agent list, which a fully measured boot never needs."""
from raven.agent.subagent.probe import schedule_snapshot_verification
measured = type("S", (), {"status": "ready", "stale": False, "needs_auth": False})()
verified, asked = self._mirasim_boot(monkeypatch, measured, frozenset())
await schedule_snapshot_verification(_FakeManager([]))
assert verified == [] and asked == []
async def test_a_preset_nobody_has_configured_is_verified_too(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""Or the page can never say `Unauthorized` before the first press.
The rows this backfill was written for come off the registry, which
holds configured agents only -- so an agent the reader has not connected
yet is exactly the one it never reached, and exactly the one whose
Connect button is about to lie to them.
"""
from raven.agent.subagent.probe import schedule_snapshot_verification
seen: list[str] = []
async def fake_verify(cfg: object) -> object:
seen.append(cfg.name)
return type("S", (), {"status": "ready", "needs_auth": False})()
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
)
monkeypatch.setattr(
probe_mod,
"_unconfigured_acp_preset_rows",
lambda configured, path=None: [_FakeRow("a-preset")] if configured else [],
)
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
await schedule_snapshot_verification(_FakeManager([_FakeRow("configured")]))
assert seen == ["configured", "a-preset"], "the configured rows first: they are the ones a run can dispatch to"
async def test_a_recorded_credential_refusal_is_measured_again(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""A recorded refusal is a verdict a restart must not take on trust.
A fresh `ready` snapshot is skipped, barring the two model-menu cases
tested below, and rightly: a `ready` agent that has not been relaunched is
still ready, and re-proving it would spend a process per row per boot. A
fresh failure is measured again, and a credential refusal is the failure
that most needs it -- it is a state the user is expected to go and fix,
and nothing about fixing it touches the launch config, so the snapshot
never goes stale and the row would keep saying "Unauthorized" across
every restart after the sign-in that cured it.
"""
from dataclasses import dataclass
from raven.agent.subagent.probe import schedule_snapshot_verification
@dataclass
class Snap:
status: str = "ready"
stale: bool = False
needs_auth: bool = False
fine, refused = _FakeRow("fine"), _FakeRow("refused")
verified: list[str] = []
def fake_snapshot_for(cfg: object) -> Snap:
return {
id(fine.config): Snap(),
id(refused.config): Snap(status="attention", needs_auth=True),
}[id(cfg)]
async def fake_verify(cfg: object) -> Snap:
verified.append(cfg.name)
return Snap()
monkeypatch.setattr(probe_mod, "acp_snapshot_for", fake_snapshot_for)
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
)
# This one pins the registry half. The preset half reads the machine's
# own PATH, so leaving it live would make the assertions below depend on
# which agents happen to be installed here.
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
await schedule_snapshot_verification(_FakeManager([fine, refused]))
assert verified == ["refused"], "a fresh pass is still trusted; a fresh refusal is not"
async def test_failed_verification_does_not_stop_the_rest(self, monkeypatch: pytest.MonkeyPatch) -> None:
from raven.agent.subagent.probe import schedule_snapshot_verification
calls: list[str] = []
async def fake_verify(cfg: object) -> object:
calls.append(getattr(cfg, "tag", "?"))
if getattr(cfg, "tag", "") == "boom":
raise RuntimeError("no")
return type("S", (), {"status": "ready"})()
def make(tag: str) -> object:
return type(
"Row", (), {"name": tag, "kind": "acp", "enabled": True, "config": type("C", (), {"tag": tag})()}
)()
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
)
# This one pins the registry half. The preset half reads the machine's
# own PATH, so leaving it live would make the assertions below depend on
# which agents happen to be installed here.
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
task = schedule_snapshot_verification(_FakeManager([make("boom"), make("after")]))
await task
assert calls == ["boom", "after"]
async def test_a_manager_without_a_registry_defers_to_the_host(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""A mounted stack's subagents substitute may not carry the registry the
backfill reads; scheduling nothing must not burn the once-per-process
ticket, so a real host later in the same process still gets its backfill."""
from raven.agent.subagent.probe import schedule_snapshot_verification
called: list[str] = []
monkeypatch.setattr(
probe_mod,
"acp_snapshot_for",
lambda cfg: None,
)
async def fake_verify(cfg: object) -> object:
called.append("verified")
return type("S", (), {"status": "ready"})()
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
)
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
stub = type("Stub", (), {})()
assert schedule_snapshot_verification(stub) is None
assert probe_mod._SCHEDULED is False, "the ticket must not be spent on a mount that cannot backfill"
assert called == []
async def test_disabled_and_non_acp_rows_are_skipped(self, monkeypatch: pytest.MonkeyPatch) -> None:
from raven.agent.subagent.probe import schedule_snapshot_verification
called: list[str] = []
monkeypatch.setattr(
probe_mod,
"acp_snapshot_for",
lambda cfg: None,
)
async def fake_verify(cfg: object) -> object:
called.append(cfg.name)
return type("S", (), {"status": "ready"})()
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
)
# This one pins the registry half. The preset half reads the machine's
# own PATH, so leaving it live would make the assertions below depend on
# which agents happen to be installed here.
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
rows = [
_FakeRow("off", enabled=False),
_FakeRow("cli-kind", kind="cli"),
_FakeRow("on"),
]
task = schedule_snapshot_verification(_FakeManager(rows))
await task
assert called == ["on"]
async def test_a_store_without_has_model_menu_does_not_force_a_reverify(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A caller's stand-in store (a bare ``record``-only object, as several
tests in this class substitute) must not be treated as "every row is on
the older format" -- the predicate is optional, and its absence must
leave a fresh, non-stale snapshot alone exactly as before this hook."""
from dataclasses import dataclass
from raven.agent.subagent.probe import schedule_snapshot_verification
@dataclass
class Snap:
status: str = "ready"
stale: bool = False
row = _FakeRow("fresh")
recorded: list[str] = []
async def fake_verify(cfg: object) -> Snap:
recorded.append(getattr(cfg, "name", "?"))
return Snap()
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: Snap())
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
)
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
task = schedule_snapshot_verification(_FakeManager([row]))
await task
assert recorded == []
async def test_a_snapshot_missing_the_model_choices_key_is_reverified(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A snapshot recorded before ``modelChoices`` existed is fresh and not
stale by every other measure, but a row stuck on it would never learn
the agent's model menu until the agent is edited or Test is pressed by
hand -- so the older format alone must trigger a re-verify."""
from dataclasses import dataclass
from raven.agent.subagent.probe import schedule_snapshot_verification
@dataclass
class Snap:
status: str = "ready"
stale: bool = False
row = _FakeRow("old-format")
recorded: list[str] = []
async def fake_verify(cfg: object) -> Snap:
recorded.append(getattr(cfg, "name", "?"))
return Snap()
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: Snap())
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type(
"S",
(),
{
"record": staticmethod(lambda s: None),
"has_model_menu": staticmethod(lambda agent: False),
},
)(),
)
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
task = schedule_snapshot_verification(_FakeManager([row]))
await task
assert recorded == ["old-format"]
async def test_one_of_ravens_own_rows_that_measured_no_menu_is_measured_again(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Raven's own agents run on raven's provider catalogue, so an empty menu
there is a handshake from before that catalogue reached them -- and one
nothing invalidates, since the launch config it was measured against has
not moved. A third party that advertised none is taken at its word: it
would otherwise be relaunched at every boot to say so again."""
from dataclasses import dataclass
from raven.agent.subagent.probe import schedule_snapshot_verification
@dataclass
class Snap:
agent_name: str = "raven"
status: str = "ready"
stale: bool = False
needs_auth: bool = False
model_menu_measured: bool = True
model_choices: tuple = ()
_host_with_a_provider_key()
own, third_party = _FakeRow("Raven-PPT"), _FakeRow("Codex")
snapshots = {id(own.config): Snap(), id(third_party.config): Snap(agent_name="codex")}
verified: list[str] = []
async def fake_verify(cfg: object) -> Snap:
verified.append(getattr(cfg, "name", "?"))
return Snap(model_choices=("v/a",))
monkeypatch.setattr(probe_mod, "_MENULESS_OWN_RE_MEASURED", set())
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: snapshots[id(cfg)])
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type(
"S",
(),
{"record": staticmethod(lambda s: None), "has_model_menu": staticmethod(lambda agent: True)},
)(),
)
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
await schedule_snapshot_verification(_FakeManager([own, third_party]))
assert verified == ["Raven-PPT"]
snapshots[id(own.config)] = Snap(model_choices=("v/a",))
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
await schedule_snapshot_verification(_FakeManager([own, third_party]))
assert verified == ["Raven-PPT"], "the menu is measured once; the record it wrote is then trusted"
async def test_a_row_told_again_it_has_no_menu_is_not_re_measured_a_third_time(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The one re-measure reason nothing invalidates has to spend itself.
An agent that answers "no models" twice is answering about itself, not
about a catalogue that had not arrived yet -- and since the record it
writes is the record that arms this reason, an unspent one would put a
child launch on every boot and on every connect after it, forever.
"""
from dataclasses import dataclass
from raven.agent.subagent.probe import capabilities_wanted, schedule_snapshot_verification
@dataclass
class Snap:
agent_name: str = "raven"
status: str = "ready"
stale: bool = False
needs_auth: bool = False
model_menu_measured: bool = True
model_choices: tuple = ()
_host_with_a_provider_key()
own = _FakeRow("Raven-PPT")
verified: list[str] = []
async def fake_verify(cfg: object) -> Snap:
verified.append(getattr(cfg, "name", "?"))
return Snap()
monkeypatch.setattr(probe_mod, "_MENULESS_OWN_RE_MEASURED", set())
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: Snap())
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type(
"S",
(),
{"record": staticmethod(lambda s: None), "has_model_menu": staticmethod(lambda agent: True)},
)(),
)
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
await schedule_snapshot_verification(_FakeManager([own]))
assert verified == ["Raven-PPT"]
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
await schedule_snapshot_verification(_FakeManager([own]))
assert verified == ["Raven-PPT"], "told once that this row really has none, and it is not asked again"
assert capabilities_wanted(own.config) is False, "nor by the connect, which reads the same rule"
async def test_a_host_with_no_provider_of_its_own_re_measures_nothing(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Nothing for the child to inherit, so nothing for a handshake to find:
the fallback's premise is a catalogue this raven can pass on."""
from dataclasses import dataclass
from raven.agent.subagent.probe import capabilities_wanted, schedule_snapshot_verification
@dataclass
class Snap:
agent_name: str = "raven"
status: str = "ready"
stale: bool = False
needs_auth: bool = False
model_menu_measured: bool = True
model_choices: tuple = ()
own = _FakeRow("Raven-PPT")
verified: list[str] = []
async def fake_verify(cfg: object) -> Snap:
verified.append(getattr(cfg, "name", "?"))
return Snap()
monkeypatch.setattr(probe_mod, "_MENULESS_OWN_RE_MEASURED", set())
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: Snap())
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type(
"S",
(),
{"record": staticmethod(lambda s: None), "has_model_menu": staticmethod(lambda agent: True)},
)(),
)
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
await schedule_snapshot_verification(_FakeManager([own]))
assert verified == []
assert capabilities_wanted(own.config) is False
async def test_capabilities_wanted_re_measures_one_of_ravens_own_rows_with_an_empty_menu(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The connect's own reading of the backfill's rule, off real stored rows.
The two must answer alike: a connect that skipped this row would leave it
menuless until the next restart, and one that re-measured every third party
with no menu would spend a handshake per connect to be told the same thing.
"""
from raven.acp_client.capabilities import AcpModelChoice, CapabilitySnapshot, SnapshotStore, snapshot_fingerprint
from raven.agent.subagent.probe import capabilities_wanted
store_path = tmp_path / "caps.json"
monkeypatch.setattr("raven.acp_client.capabilities.default_snapshot_path", lambda: store_path)
def recorded(
name: str, *, agent_name: str, menu: tuple[str, ...] = (), status: str = "ready"
) -> ThirdPartyAcpSubagentConfig:
cfg = ThirdPartyAcpSubagentConfig(name=name, command=f"{name} acp")
SnapshotStore(path=store_path).record(
CapabilitySnapshot(
agent=name,
fingerprint=snapshot_fingerprint(cfg),
status=status,
detail="",
measured_at_ms=1,
agent_name=agent_name,
model_choices=tuple(AcpModelChoice(value=v, name=v, group="V") for v in menu),
)
)
return cfg
assert capabilities_wanted(recorded("own-empty", agent_name="raven")) is False, "no provider of raven's own yet"
_host_with_a_provider_key()
monkeypatch.setattr(probe_mod, "_MENULESS_OWN_RE_MEASURED", set())
assert capabilities_wanted(recorded("own-empty", agent_name="raven")) is True
assert capabilities_wanted(recorded("own-menu", agent_name="raven", menu=("v/a",))) is False
assert capabilities_wanted(recorded("third-party-empty", agent_name="codex")) is False
# Ready, or the empty menu says nothing about the catalogue: an agent that
# did not come up is the missing-snapshot case, not this one.
assert capabilities_wanted(recorded("own-broken", agent_name="raven", status="attention")) is False
def test_a_preset_the_table_cannot_be_read_from_offers_nothing(monkeypatch: pytest.MonkeyPatch) -> None:
"""Three ways the shipped table can fail to answer, and none may reach the boot.
This runs at startup, before anything a user did: a packaging fault here is
not their typo to see, and taking the gateway down over it would trade a
stale row for no gateway. Each branch answers with the rows it could build,
which for these inputs is none.
"""
# A command no shell could split. `shlex` raises rather than returning, and
# the entry is simply not a candidate.
monkeypatch.setattr(
probe_mod,
"third_party_subagent_presets",
lambda: [{"name": "Torn", "preset": "claude_code", "kind": "acp", "command": 'x "unclosed'}],
)
assert probe_mod._unconfigured_acp_preset_rows(set(), path=None) == []
# Nothing on this machine resolves, so there is nothing to hand back and the
# schema is never asked.
monkeypatch.setattr(
probe_mod,
"third_party_subagent_presets",
lambda: [{"name": "Gone", "preset": "codex", "kind": "acp", "command": "absent-agent acp"}],
)
monkeypatch.setattr(probe_mod.shutil, "which", lambda exe, path=None: None)
assert probe_mod._unconfigured_acp_preset_rows(set(), path=None) == []
# And a table the schema refuses: logged, and the boot goes on without it.
monkeypatch.setattr(probe_mod.shutil, "which", lambda exe, path=None: "/bin/x")
monkeypatch.setattr(
probe_mod,
"third_party_subagent_presets",
lambda: [{"name": "Bad", "preset": "codex", "kind": "acp", "command": "x acp", "readyTimeoutMs": "soon"}],
)
assert probe_mod._unconfigured_acp_preset_rows(set(), path=None) == []
def test_only_resolvable_unconfigured_acp_presets_are_offered_for_verification(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Three filters, each for its own reason.
Already configured: the registry half of the backfill has it. Not acp: there
is no handshake to record. Not on PATH: launching it is how you learn it is
not there, and the free probe answered that for nothing.
"""
from raven.config.schema import ThirdPartyAcpSubagentConfig
# Real preset keys: the schema checks `preset` against the shipped table, so
# a made-up one would fail validation rather than the filter under test.
presets = [
{"name": "Here", "preset": "claude_code", "kind": "acp", "command": "present-agent acp"},
{"name": "Gone", "preset": "codex", "kind": "acp", "command": "absent-agent acp"},
{"name": "Mine", "preset": "opencode", "kind": "acp", "command": "present-agent acp"},
{"name": "Http", "preset": "mirothinker", "kind": "openai", "baseUrl": "https://x/v1", "model": "m"},
]
seen_paths: list[str | None] = []
def fake_which(exe: str, path: str | None = None) -> str | None:
seen_paths.append(path)
return "/bin/x" if exe == "present-agent" else None
monkeypatch.setattr(probe_mod, "third_party_subagent_presets", lambda: presets)
monkeypatch.setattr(probe_mod.shutil, "which", fake_which)
rows = probe_mod._unconfigured_acp_preset_rows({"Mine"}, path="/login/shell/bin")
assert [row.name for row in rows] == ["Here"]
assert isinstance(rows[0].config, ThirdPartyAcpSubagentConfig)
# Resolved against the caller's PATH, not this process's. `_probe_acp` answers
# the row on screen from the login shell's, so reading a different one here
# would skip an agent the page is reporting as installed.
assert set(seen_paths) == {"/login/shell/bin"}
def test_a_shim_preset_whose_agent_is_absent_is_not_offered_for_verification(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A shim's command resolves on any machine; the agent it drives may not.
``_probe_acp`` reads such a row as missing, so handshaking it at boot would
pay to learn what the free probe already said. Sharpest for a Mirasim row,
whose command is Raven's own interpreter.
"""
presets = [
{
"name": "Claude Code-Mirasim",
"preset": "mirasim_claude",
"kind": "acp",
"command": "/venv/bin/python -P -m raven.acp.mirasim --agent claude",
},
{"name": "Pi", "preset": "pi", "kind": "acp", "command": "npx -y pi-acp@0.0.33"},
]
present = {"/venv/bin/python", "npx", "pi"}
monkeypatch.setattr(probe_mod, "third_party_subagent_presets", lambda: presets)
monkeypatch.setattr(probe_mod.shutil, "which", lambda exe, path=None: f"/bin/{exe}" if exe in present else None)
assert [row.name for row in probe_mod._unconfigured_acp_preset_rows(set(), path=None)] == ["Pi"]
present.add("mirasim")
rows = probe_mod._unconfigured_acp_preset_rows(set(), path=None)
assert [row.name for row in rows] == ["Claude Code-Mirasim", "Pi"]
async def test_a_hand_written_npx_row_is_not_held_to_a_shims_requirement(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The requirement rides on provenance: a row that merely wears the preset's name runs its own command."""
_fake_executable(tmp_path, "npx")
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
cfg = ThirdPartyAcpSubagentConfig(name="Pi", command="npx -y my-own-acp-shim@1.0.0")
res = await probe_one(cfg, source="config", path=str(tmp_path))
assert res.status == "attention"
async def test_a_shim_that_ships_its_own_agent_asks_after_nothing_else(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``codex-acp`` bundles the agent as its own binary; what it wants is a login, not an install."""
_fake_executable(tmp_path, "npx")
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
cfg = ThirdPartyAcpSubagentConfig(
name="Codex", preset="codex", command="npx -y @agentclientprotocol/codex-acp@1.1.14"
)
res = await probe_one(cfg, source="preset", path=str(tmp_path))
assert res.status == "attention"
async def test_a_shim_row_whose_agent_is_installed_goes_on_to_the_snapshot(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
npx = _fake_executable(tmp_path, "npx")
_fake_executable(tmp_path, "pi")
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
cfg = ThirdPartyAcpSubagentConfig(name="Pi", preset="pi", command="npx -y pi-acp@0.0.33")
res = await probe_one(cfg, source="preset", path=str(tmp_path))
assert res.status == "attention"
assert res.target == str(npx)
async def test_a_shim_row_is_missing_when_the_agent_it_drives_is_absent(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``npx`` resolving says nothing about whether ``pi`` is installed.
The shim is fetched on connect and fails a minute later with "executable not
found" when the agent it drives is absent. That is the sentence the probe can
say up front, with the install beside it, so the row reads absent rather
than connectable -- the same reading a local-executable row gets.
"""
_fake_executable(tmp_path, "npx")
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
cfg = ThirdPartyAcpSubagentConfig(name="Pi", preset="pi", command="npx -y pi-acp@0.0.33")
res = await probe_one(cfg, source="preset", path=str(tmp_path))
assert res.status == "missing"
assert res.target == "pi"
assert res.absent == "pi", "npx is there; the agent it drives is what to install"
assert res.detail == (
"pi is not on the login shell PATH; install with npm install -g @earendil-works/pi-coding-agent"
)
def _fake_executable(tmp_path: Path, name: str) -> Path:
exe = tmp_path / name
exe.write_text("#!/bin/sh\nexit 0\n")
exe.chmod(0o755)
return exe
def test_a_machine_failure_says_where_the_fix_is_made_unless_the_fix_is_the_node_installed_from_here():
from raven.agent.subagent import probe
there = probe._on_machine("'claude' is not on the login shell PATH", "ravenx2")
assert there.endswith("(on machine 'ravenx2': any fix is made there)")
here = probe._on_machine(
"Raven node 0.2.4-abc is not installed on machine 'RavenX 2' (ravenx2); install it with "
"`raven ops connection install-node ravenx2`",
"ravenx2",
)
assert "any fix is made there" not in here and here.endswith("install-node ravenx2`")
@pytest.mark.parametrize(
"status,detail",
[
("unknown", "unsupported session configuration"),
("attention", "session/new timed out"),
("missing", "transport disconnected"),
],
)
async def test_boot_replaces_an_auth_verdict_and_retries_until_ready(tmp_path, monkeypatch, status, detail):
from dataclasses import replace
from unittest.mock import AsyncMock
import raven.acp_client.capabilities as caps
cfg = ThirdPartyAcpSubagentConfig(name="measured", command="measured --acp")
monkeypatch.setattr(caps, "default_snapshot_path", lambda: tmp_path / "caps.json")
monkeypatch.setattr(probe_mod, "_launch_kw", lambda cfg: {})
old = caps.CapabilitySnapshot(
agent=cfg.name,
fingerprint=caps.snapshot_fingerprint(cfg),
status="attention",
detail="sign in",
measured_at_ms=1,
needs_auth=True,
can_load=True,
model_choices=(caps.AcpModelChoice(value="a", name="A"),),
)
store = caps.SnapshotStore()
store.record(old)
failed = caps.CapabilitySnapshot(
agent=cfg.name,
fingerprint=old.fingerprint,
status=status,
detail=detail,
measured_at_ms=2,
needs_auth=False,
)
verified = AsyncMock(side_effect=[failed, replace(old, status="ready", needs_auth=False, measured_at_ms=3)])
monkeypatch.setattr(caps, "verify_agent", verified)
rows = [_FakeRow(cfg.name, config=cfg)]
manager = _FakeManager(rows, with_refresh=True)
await probe_mod._verify_missing_snapshots(manager, rows)
saved = store.load([cfg])[cfg.name]
assert not saved.needs_auth
assert not saved.usable
assert (saved.status, saved.detail) == (status, detail)
assert saved.can_load and saved.model_choices == old.model_choices
await probe_mod._verify_missing_snapshots(manager, rows)
assert store.load([cfg])[cfg.name].usable
await probe_mod._verify_missing_snapshots(manager, rows)
assert verified.await_count == 2
assert manager.refresh_calls == 2
@pytest.mark.parametrize(
"tested_status,late",
[
("ready", {"status": "attention", "detail": "session/new timed out"}),
("ready", {"status": "attention", "detail": "Authentication required", "needs_auth": True}),
("attention", {"status": "ready", "detail": "connected"}),
],
ids=[
"a-late-timeout-loses-to-a-test-pass",
"a-late-refusal-loses-to-a-test-pass",
"a-late-pass-loses-to-a-test-failure",
],
)
async def test_a_record_written_while_the_boot_verify_ran_outranks_its_late_verdict(
tmp_path, monkeypatch, tested_status, late
):
"""The newest evidence wins: a Test recorded while the boot's handshake was in
flight stands, whatever that handshake returns afterwards.
The boot reads the row before it launches anything, and the handshake can
then hold that read for the whole ready timeout. A write built from it took
the Test's verdict and the menu the Test measured both -- a late timeout
included, which the boot records over any row whose last record failed.
"""
from dataclasses import replace
import raven.acp_client.capabilities as caps
cfg = ThirdPartyAcpSubagentConfig(name="measured", command="measured --acp")
monkeypatch.setattr(caps, "default_snapshot_path", lambda: tmp_path / "caps.json")
monkeypatch.setattr(probe_mod, "_launch_kw", lambda cfg: {})
old = caps.CapabilitySnapshot(
agent=cfg.name,
fingerprint=caps.snapshot_fingerprint(cfg),
status="attention",
detail="sign in",
measured_at_ms=1,
needs_auth=True,
can_load=True,
model_choices=(caps.AcpModelChoice(value="a", name="A"),),
)
store = caps.SnapshotStore()
store.record(old)
if tested_status == "ready":
tested = replace(
old,
status="ready",
detail="connected",
needs_auth=False,
measured_at_ms=3,
model_choices=(caps.AcpModelChoice(value="b", name="B"),),
)
else:
tested = replace(old, status="attention", detail="session/new timed out", needs_auth=False, measured_at_ms=3)
late_result = replace(
old, **{"needs_auth": False, "measured_at_ms": 4, "model_choices": (caps.AcpModelChoice(value="c"),), **late}
)
calls: list[str] = []
boot_parked, released = asyncio.Event(), asyncio.Event()
async def fake_verify(cfg, **kwargs):
if not calls:
calls.append("boot")
boot_parked.set()
await released.wait()
return late_result
calls.append("test")
return tested
monkeypatch.setattr(caps, "verify_agent", fake_verify)
rows = [_FakeRow(cfg.name, config=cfg)]
boot = asyncio.ensure_future(probe_mod._verify_missing_snapshots(_FakeManager(rows, with_refresh=True), rows))
await asyncio.wait_for(boot_parked.wait(), 2)
await probe_mod.record_capabilities(cfg)
after_test = store.load([cfg])[cfg.name]
assert not boot.done(), "the Test has to land while the boot's handshake is still open"
assert after_test == tested, "and record its own measurement there"
released.set()
await asyncio.wait_for(boot, 2)
assert calls == ["boot", "test"]
assert store.load([cfg])[cfg.name] == after_test, "the boot's late verdict must not replace the Test's"
@pytest.mark.parametrize("late_status", ["ready", "attention"])
async def test_a_modes_relearn_during_the_boot_verify_does_not_stop_its_write(tmp_path, monkeypatch, late_status):
"""A dispatch that relearns the agent's modes mid-handshake measures nothing.
It keeps the record's measurement stamp, so the boot's verdict still lands:
otherwise a failed row that a dispatch merely touched would stay failed until
the next boot. A late failure is written over the re-read record, so the
relearned modes survive it; a late ``ready`` is a whole measurement, its own
menu included, and is written as measured.
"""
from dataclasses import replace
import raven.acp_client.capabilities as caps
cfg = ThirdPartyAcpSubagentConfig(name="measured", command="measured --acp")
monkeypatch.setattr(caps, "default_snapshot_path", lambda: tmp_path / "caps.json")
monkeypatch.setattr(probe_mod, "_launch_kw", lambda cfg: {})
old = caps.CapabilitySnapshot(
agent=cfg.name,
fingerprint=caps.snapshot_fingerprint(cfg),
status="attention",
detail="session/new timed out",
measured_at_ms=1,
can_load=True,
available_modes=(caps.AcpMode(id="old", name="Old"),),
)
store = caps.SnapshotStore()
store.record(old)
late_result = replace(
old,
status=late_status,
detail="connected" if late_status == "ready" else "session/new timed out again",
measured_at_ms=4,
available_modes=(caps.AcpMode(id="measured", name="Measured"),),
)
boot_parked, released = asyncio.Event(), asyncio.Event()
async def fake_verify(cfg, **kwargs):
boot_parked.set()
await released.wait()
return late_result
monkeypatch.setattr(caps, "verify_agent", fake_verify)
rows = [_FakeRow(cfg.name, config=cfg)]
manager = _FakeManager(rows, with_refresh=True)
boot = asyncio.ensure_future(probe_mod._verify_missing_snapshots(manager, rows))
await asyncio.wait_for(boot_parked.wait(), 2)
relearned = caps.relearn_session_modes(
store.load([cfg])[cfg.name],
{"sessionId": "s1", "modes": {"availableModes": [{"id": "fast", "name": "Fast"}], "currentModeId": "fast"}},
store=store,
)
assert relearned is not None and relearned.measured_at_ms == 1, "a relearn keeps the stamp it found"
assert not boot.done(), "the relearn has to land while the boot's handshake is still open"
released.set()
await asyncio.wait_for(boot, 2)
stored = store.load([cfg])[cfg.name]
assert (stored.status, stored.measured_at_ms) == (late_status, 4), "the boot's verdict must still land"
expected_modes = ("measured",) if late_status == "ready" else ("fast",)
assert tuple(m.id for m in stored.available_modes) == expected_modes
assert manager.refresh_calls == 1
def _menu_records() -> tuple[object, object]:
"""A record with a measured menu, and a later answer that was ready but could not read one."""
from raven.acp_client.capabilities import AcpMode, AcpModelChoice, CapabilitySnapshot
measured = CapabilitySnapshot(
agent="Claude Code-Mirasim",
fingerprint="old",
status="ready",
detail="",
measured_at_ms=1,
model_choices=(AcpModelChoice("own:sonnet", "Sonnet", "Claude Code"),),
available_modes=(AcpMode("high", "High"),),
stale=True,
)
unread = CapabilitySnapshot(
agent="Claude Code-Mirasim",
fingerprint="new",
status="ready",
detail="ready",
measured_at_ms=2,
model_menu_measured=False,
)
return measured, unread
def test_a_ready_answer_whose_menu_went_unread_keeps_the_menu_it_had() -> None:
"""A connect or a Test the Mirasim host leaves unanswered is menuless: a read that failed, not a measured none."""
measured, unread = _menu_records()
saved = probe_mod._test_record(unread, measured)
assert (saved.model_choices, saved.available_modes) == (measured.model_choices, measured.available_modes)
assert saved.model_menu_measured
assert (saved.status, saved.fingerprint, saved.measured_at_ms) == ("ready", "new", 2)
async def test_the_boot_keeps_a_measured_menu_over_an_answer_that_could_not_read_it(
monkeypatch: pytest.MonkeyPatch,
) -> None:
measured, unread = _menu_records()
recorded: list[object] = []
async def fake_verify(cfg: object) -> object:
return unread
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: measured)
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
monkeypatch.setattr(
"raven.acp_client.capabilities.SnapshotStore",
lambda: type("S", (), {"record": staticmethod(recorded.append)})(),
)
row = _FakeRow("Claude Code-Mirasim")
await probe_mod._verify_missing_snapshots(_FakeManager([row]), [row])
assert [snapshot.model_choices for snapshot in recorded] == [measured.model_choices]
@pytest.mark.parametrize("needs_auth,unfetched", [(True, False), (False, True), (False, False)])
@pytest.mark.parametrize("previous_status", ["ready", "attention"])
def test_manual_test_merges_current_failure_flags_with_old_capabilities(needs_auth, unfetched, previous_status):
from dataclasses import replace
from raven.acp_client.capabilities import CapabilitySnapshot
old = CapabilitySnapshot(
agent="agent",
fingerprint="old",
status=previous_status,
detail="old refusal",
measured_at_ms=1,
needs_auth=previous_status == "attention",
can_load=True,
)
measured = replace(
old,
status="attention",
fingerprint="new",
measured_at_ms=2,
detail="current refusal",
can_load=False,
needs_auth=needs_auth,
unfetched=unfetched,
)
saved = probe_mod._test_record(measured, old)
assert saved.can_load
assert saved.fingerprint == "new"
assert (saved.needs_auth, saved.unfetched) == (needs_auth, unfetched)