## Summary Raven's host acts as the user's agent manager: it delegates production work, including small edits, to the agent whose ownership or description fits the task -- or, on a turn bound to a Worker Table, to the worker whose brief covers it -- keeps track of the task and its instance, routes follow-ups back to that instance, and gathers the results for the user. Its own tools support briefing, checking and coordination. - Add the resident `multi-agent-coding` skill for choosing coding agents, keeping one instance per task, passing results by file, isolating concurrent edits in worktrees, and handing over work when an agent exhausts its quota. - Keep the new host role authoritative in both the DAG skill's resident description and its full guide. A single task uses `spawn`, or a one-node graph when only the DAG dispatch path is available. - Rename the generic sub-agent to `Raven-General`, with `Raven` and `raven` retained as compatibility aliases. Instance records, handle locks, live activity, cancellation, preferences and RPC lookups use the same canonical identity. Historical spawn and DAG readers normalize names before liveness checks and serialization, so tasks and transcripts join the current instance roster. Concurrent alias calls preserve both turns instead of overwriting one conversation snapshot. The instance registry reads former-name rows under the current name and, on its next write, saves every loaded row under it; transcripts, direct chats and stored graphs stay as written. - Resolve built-in configuration aliases before merging fields: `Raven-General` wins over `Raven`, then `raven`, regardless of file order. Discard older rows wholesale with a warning, preventing stale model, permission or transport settings from leaking into the selected row. - Resume the most recently written message state and instance log across former names. Saved messages take precedence over an empty chat directory; equal timestamps prefer the current name and then the most recent former name. Existing files stay in place. Legacy spawn records, completed DAG manifests and unfinished graphs remain readable through the current name. - Emit the agent roster once at the end of `spawn`'s description. The registry places `run_subagent_dag` and then `spawn` last, including after late MCP or session tools. The graph tool renders the roster itself when `spawn` is unavailable. With a Worker Table bound, dispatch guidance selects worker labels from the live enum; the roster explains underlying capabilities. The tool descriptions and resident DAG guide follow the same rule. - On a turn bound to a Worker Table, such as a Persona session, `spawn` and the DAG nodes offer worker labels alone, and a generated Persona never includes the generic agent. The host role on such a turn routes production work to the worker whose brief covers it, by label, and puts work no worker covers to the user, who can add a worker or give the conversation another Harness, instead of doing it itself or naming an agent outside the enum. Ordinary turns render byte for byte as before. A Harness that `load_playbook` binds mid-turn takes effect after the identity was assembled, so that call's result carries the same routing as this system's trusted note, after the untrusted fence. The coding skill picks worker labels on such a turn, for its first pick and for a quota takeover. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Measured at this head on Linux/Python 3.12.13. - `UV_NO_SYNC=1 uv run --no-sync pytest -q -n 12`: **33,767 passed, 178 skipped, 2 xfailed; 9 failed and 1 error.** Eight of the failures are bound to this box and fail the same way on current main, measured the same day: five provider tests that read the box's proxy environment, a Node.js path case, a LibreOffice conversion, and a fixture that root can read anyway. The ninth, with its teardown error, is the `remote_runs` race tracked in #990; that test passed 5 of 5 runs alone, and its file 58 of 58. - 13 new regression tests for Worker Table routing: 12 fail against the previous head's source and all pass now; the thirteenth checks that a Harness loaded mid-turn leaves the next turn's role plain, and passes on both by design. The end-to-end case drives a real Persona-bound turn: both enums offer `research-a` alone, dispatching `Raven-General` is refused, and the role must name neither that agent nor the roster fallback. - Nine single-construct mutants, each applied to an isolated copy of the tree, each fail at least one new test: the role ignoring the table, the default path not reading the turn's table, the uncovered-work rule removed, the generic fallback kept in the worker routing, the coding skill reverted, and the mid-turn note never added, added on every call, added without a dispatch path, or phrased apart from the role. - The host identity on ordinary turns is byte-identical to the previous head in all 16 rendered shapes: a launched sub-agent process or not, with or without ownership lines, and every combination of the two dispatch tools. - Instance registry probe, run with this head and with main on one synthetic file: reads leave the file byte-identical; one unrelated `upsert_spawn` saves every former-name row as `Raven-General`, keeping the more recently updated of two colliding spellings and the later one in the file on a tie; main then finds none of those rows under `Raven` or `raven`. - `uv run --no-sync python scripts/coverage_gate.py --coverage-json <coverage.json> diff --base-ref origin/main --threshold 90`, over a coverage run of 225 test files (8,952 passed): **99.40%**, 167/168 executable changed lines covered. - `UV_NO_SYNC=1 make lint-types lint-imports check-large-files check-source-language COMMIT_RANGE=origin/main...HEAD`: passed; all 10 import contracts kept. - `UV_NO_SYNC=1 make check-commits`: both commit-message validators passed; each also refused a dangling control commit with a malformed subject. - `ruff check` and `ruff format --check` over the branch's Python files passed. `node ui-tui/scripts/gen-i18n.mjs --check` passed after the rebase onto #1015, which also changed the i18n catalogue. - `uv run --no-sync pre-commit run --from-ref origin/main --to-ref HEAD`: every hook passed except the whole-TUI Prettier hook, which reports only `newInstancePicker.tsx` and `subagentsHub.tsx`; both are byte-identical to current main. - Live-model replay and native Windows/macOS testing were not repeated for these fixes. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - The host delegates work it previously performed itself, including small scripts. Task-to-instance routing remains prompt-based; a task can still stay with the host after it has already answered a follow-up itself. - In a conversation bound to a Persona or another Harness with workers, work no worker's brief covers now waits for the user to add a worker or change the conversation's Harness, rather than being carried out by the host. - Old agent spellings remain accepted. The instance registry's first write after the upgrade, of any row for any session, saves every former-name row in `subagent_instances.json` as `Raven-General`, and keeps only the more recently updated row where two spellings of one instance collide. Builds from before the rename cannot look those rows up, so reverting the squash commit alone does not restore their instance bindings: that needs the registry file as it stood before that first write. Renaming rows back by hand cannot recover a colliding row the write dropped. Transcripts, direct chats and stored graphs are not rewritten. - When multiple generic-agent configuration spellings coexist, only the newest row contributes overrides. Fields supplied exclusively by an older row are intentionally discarded. - Moving the dispatch tools changes the tool-array cache prefix once after upgrade. - A plugin participant that removes `spawn` through `select_tools` can still leave the graph tool pointing at an absent roster, because that selection occurs after the registry's offer check. No bundled participant does this. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A
2164 lines
92 KiB
Python
2164 lines
92 KiB
Python
"""Free availability probe for third-party subagents (cli PATH + openai /models)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import json
|
|
import os
|
|
import socket
|
|
from contextlib import closing
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
from aiohttp import web
|
|
|
|
import raven.agent.subagent.backends.env as env_mod
|
|
import raven.agent.subagent.probe as probe_mod
|
|
from raven.agent.subagent import instances as instances_mod
|
|
from raven.agent.subagent.probe import probe_all, probe_one, run_test
|
|
from raven.config.schema import (
|
|
Config,
|
|
ThirdPartyAcpSubagentConfig,
|
|
ThirdPartyCliSubagentConfig,
|
|
ThirdPartyOpenAISubagentConfig,
|
|
)
|
|
from tests._windows_dirs import scratch_root, windows_removal
|
|
|
|
|
|
def _free_port() -> int:
|
|
with closing(socket.socket()) as s:
|
|
s.bind(("127.0.0.1", 0))
|
|
return s.getsockname()[1]
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _isolated_instance_registry(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""No test here may touch the real ~/.raven/subagent_instances.json.
|
|
|
|
Same reasoning as the fixture of this name in
|
|
`tests/test_subagent_third_party.py`: a stateful create commits a handle
|
|
binding through the process-wide `get_registry()` singleton, so without
|
|
this every run would accumulate junk rows in the user's real file.
|
|
"""
|
|
monkeypatch.setattr(
|
|
instances_mod, "_registry", instances_mod.InstanceRegistry(path=tmp_path / "_autouse_inst.json")
|
|
)
|
|
|
|
|
|
def _cli(command: str, name: str = "agent") -> ThirdPartyCliSubagentConfig:
|
|
return ThirdPartyCliSubagentConfig(name=name, command=command)
|
|
|
|
|
|
def _openai(base_url: str, model: str = "m1", api_key: str = "k") -> ThirdPartyOpenAISubagentConfig:
|
|
return ThirdPartyOpenAISubagentConfig(name="api", base_url=base_url, model=model, api_key=api_key)
|
|
|
|
|
|
# --- cli probe -----------------------------------------------------------
|
|
|
|
|
|
async def test_cli_probe_reports_the_resolved_absolute_path(tmp_path: Path) -> None:
|
|
exe = tmp_path / "faux-agent"
|
|
exe.write_text("#!/bin/sh\nexit 0\n")
|
|
exe.chmod(0o755)
|
|
res = await probe_one(_cli("faux-agent -p {prompt}"), source="config", path=str(tmp_path))
|
|
assert res.status == "ready"
|
|
assert res.target == str(exe)
|
|
assert res.kind == "cli"
|
|
assert res.source == "config"
|
|
|
|
|
|
async def test_a_missing_executable_says_what_installs_it(tmp_path: Path) -> None:
|
|
"""The install belongs on the surface a person reads, not in the roster prose.
|
|
|
|
``which`` can only report the executable it looked for, which is not the
|
|
package that ships it: ``qodercli`` does not spell ``@qoder-ai/qodercli``.
|
|
Carried by the row's provenance rather than its name, so an owner who renamed
|
|
the row keeps the hint.
|
|
"""
|
|
cfg = ThirdPartyAcpSubagentConfig(name="Renamed By Its Owner", preset="qoder", command="qodercli --acp")
|
|
res = await probe_one(cfg, source="config", path=str(tmp_path))
|
|
assert res.status == "missing"
|
|
assert res.target == "qodercli"
|
|
assert res.absent == "qodercli", "what the page offers to install is read from this"
|
|
assert "npm i -g @qoder-ai/qodercli" in res.detail
|
|
|
|
|
|
async def test_a_missing_executable_invents_no_install_it_does_not_know(tmp_path: Path) -> None:
|
|
"""A hand-written row is not the preset whose name it happens to wear.
|
|
|
|
Its command is the owner's, so naming a package would send them to install
|
|
something they never asked for. The executable name already reported is what
|
|
they search with instead.
|
|
"""
|
|
cfg = ThirdPartyAcpSubagentConfig(name="Qoder", command="my-own-agent --acp")
|
|
res = await probe_one(cfg, source="config", path=str(tmp_path))
|
|
assert res.status == "missing"
|
|
assert res.detail == "my-own-agent is not on the login shell PATH"
|
|
assert "npm" not in res.detail
|
|
|
|
|
|
async def test_a_row_launched_through_npx_names_node_js_when_npx_is_absent(tmp_path: Path) -> None:
|
|
"""The agent's own installer is the wrong answer when what is absent is npx.
|
|
|
|
Claude Code and Codex launch through ``npx``, which Node.js brings. With npx
|
|
taken off one row's PATH (2026-09-23), the page offered
|
|
``npm install -g @anthropic-ai/claude-code``: an npm command, on a machine
|
|
whose npm went with its Node.js -- and an agent installed some other way
|
|
still launches through npx, so the row stayed absent after it anyway.
|
|
"""
|
|
cfg = ThirdPartyAcpSubagentConfig(
|
|
name="Claude Code", preset="claude_code", command="npx -y @agentclientprotocol/claude-agent-acp@0.79.0"
|
|
)
|
|
res = await probe_one(cfg, source="preset", path=str(tmp_path))
|
|
assert res.status == "missing"
|
|
assert res.absent == "npx"
|
|
assert (
|
|
res.detail
|
|
== "npx is not on the login shell PATH; install with Node.js from https://nodejs.org, which brings npx"
|
|
)
|
|
|
|
# A cli row whose command is npx is the same absence, whatever it runs.
|
|
cli = await probe_one(_cli("npx some-agent {prompt}"), source="config", path=str(tmp_path))
|
|
assert cli.absent == "npx"
|
|
assert "https://nodejs.org" in cli.detail
|
|
|
|
|
|
async def test_only_a_lookup_that_found_nothing_names_what_is_absent(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""``absent`` is what to install, so it must not ride on a ``missing`` that came from a handshake.
|
|
|
|
A snapshot recorded ``missing`` is an executable that was found and then did
|
|
not start. Naming it absent would put an installer in front of a reader who
|
|
already has the thing installed.
|
|
"""
|
|
npx = _fake_executable(tmp_path, "npx")
|
|
snapshot = type("Snapshot", (), {"stale": False, "model_menu_measured": True, "status": "missing", "detail": "x"})
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: snapshot)
|
|
cfg = ThirdPartyAcpSubagentConfig(
|
|
name="Codex", preset="codex", command="npx -y @agentclientprotocol/codex-acp@1.1.14"
|
|
)
|
|
res = await probe_one(cfg, source="preset", path=str(tmp_path))
|
|
assert res.status == "missing"
|
|
assert res.target == str(npx)
|
|
assert res.absent is None
|
|
|
|
|
|
async def test_cli_probe_reports_missing_and_still_names_what_it_looked_for(tmp_path: Path) -> None:
|
|
res = await probe_one(_cli("definitely-not-installed {prompt}"), source="config", path=str(tmp_path))
|
|
assert res.status == "missing"
|
|
# A `missing` result still names the executable, or the user cannot tell
|
|
# which of several tokens in the command was the one not found.
|
|
assert res.target == "definitely-not-installed"
|
|
assert "definitely-not-installed" in res.detail
|
|
|
|
|
|
@pytest.mark.parametrize("kind", ["cli", "acp"])
|
|
async def test_windows_probes_find_a_legacy_single_quoted_program(tmp_path, monkeypatch, kind):
|
|
import shlex
|
|
from types import SimpleNamespace
|
|
|
|
from raven.utils import commands
|
|
|
|
monkeypatch.setattr(commands, "os", SimpleNamespace(name="nt"))
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda _cfg: None)
|
|
program = _fake_executable(tmp_path, "saved agent.exe")
|
|
command = shlex.quote(str(program))
|
|
cfg = _cli(command) if kind == "cli" else ThirdPartyAcpSubagentConfig(name="saved-command", command=command)
|
|
|
|
result = await probe_one(cfg, source="config", path=str(tmp_path))
|
|
|
|
assert result.target == str(program)
|
|
assert result.status == ("ready" if kind == "cli" else "attention")
|
|
assert result.absent is None
|
|
|
|
|
|
@pytest.mark.skipif(os.name == "nt", reason="POSIX shell line: a lone quote is not a parse error to CreateProcess")
|
|
async def test_cli_probe_reports_unparseable_command_as_unknown() -> None:
|
|
# This single quote belongs to a non-path token, so it stays literal on
|
|
# Windows and the probe reports a missing agent rather than invalid syntax.
|
|
res = await probe_one(_cli("claude -p 'unbalanced {prompt}"), source="config", path="/usr/bin")
|
|
assert res.status == "unknown"
|
|
assert "cannot be parsed" in res.detail
|
|
|
|
|
|
async def test_cli_probe_reports_a_placeholder_executable_as_unknown() -> None:
|
|
# `{prompt}` as argv[0] names no executable, so "not installed" would be a
|
|
# wrong diagnosis of a malformed command.
|
|
res = await probe_one(_cli("{prompt} --go"), source="config", path="/usr/bin")
|
|
assert res.status == "unknown"
|
|
assert "placeholder" in res.detail
|
|
|
|
|
|
async def test_cli_probe_resolves_on_the_login_shell_path_not_ravens(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
# The whole point: `CliAgentBackend._exec` runs the child under
|
|
# `login_shell_env()`, so a probe reading os.environ would report an agent
|
|
# missing while real dispatches find it (codex lives on the login PATH only).
|
|
exe = tmp_path / "login-only-agent"
|
|
exe.write_text("#!/bin/sh\nexit 0\n")
|
|
exe.chmod(0o755)
|
|
monkeypatch.setenv("PATH", "/nonexistent")
|
|
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
|
|
res = await probe_one(_cli("login-only-agent {prompt}"), source="config")
|
|
assert res.status == "ready"
|
|
|
|
|
|
async def test_cli_probe_honors_a_configured_path_override(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# CliAgentBackend._exec builds the child's env as {**login_shell_env(), **self.env}
|
|
# (cli_agent.py:139-140), so a configured PATH wins over the login shell's for a
|
|
# real spawn. The probe must resolve the same way, or a subagent with an env PATH
|
|
# override would be reported missing while a real spawn finds it fine.
|
|
login_only = tmp_path / "login_only"
|
|
login_only.mkdir()
|
|
override = tmp_path / "override"
|
|
override.mkdir()
|
|
exe = override / "custom-agent"
|
|
exe.write_text("#!/bin/sh\nexit 0\n")
|
|
exe.chmod(0o755)
|
|
cfg = ThirdPartyCliSubagentConfig(name="agent", command="custom-agent {prompt}", env={"PATH": str(override)})
|
|
res = await probe_one(cfg, source="config", path=str(login_only))
|
|
assert res.status == "ready"
|
|
assert res.target == str(exe)
|
|
|
|
|
|
# --- openai probe --------------------------------------------------------
|
|
|
|
|
|
async def _serve(handler, route: str = "/v1/models", method: str = "GET") -> tuple[str, web.AppRunner, list[str]]:
|
|
"""Start a stub endpoint; returns (base_url, runner, recorded request paths)."""
|
|
seen: list[str] = []
|
|
|
|
async def wrapped(request: web.Request):
|
|
seen.append(request.path)
|
|
return await handler(request)
|
|
|
|
port = _free_port()
|
|
app = web.Application()
|
|
app.router.add_route(method, route, wrapped)
|
|
runner = web.AppRunner(app)
|
|
await runner.setup()
|
|
await web.TCPSite(runner, "127.0.0.1", port).start()
|
|
return f"http://127.0.0.1:{port}/v1", runner, seen
|
|
|
|
|
|
async def test_openai_probe_ready_when_the_model_is_listed() -> None:
|
|
async def handler(request: web.Request) -> web.Response:
|
|
return web.json_response({"object": "list", "data": [{"id": "m1"}, {"id": "m2"}]})
|
|
|
|
base, runner, _ = await _serve(handler)
|
|
try:
|
|
res = await probe_one(_openai(base, model="m1"), source="config")
|
|
assert res.status == "ready"
|
|
assert res.target == base
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
async def test_openai_probe_flags_a_model_absent_from_the_list() -> None:
|
|
async def handler(request: web.Request) -> web.Response:
|
|
return web.json_response({"data": [{"id": "m1"}, {"id": "m2"}]})
|
|
|
|
base, runner, _ = await _serve(handler)
|
|
try:
|
|
res = await probe_one(_openai(base, model="typo-model"), source="config")
|
|
assert res.status == "attention"
|
|
assert "typo-model" in res.detail
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
async def test_openai_probe_is_ready_but_honest_when_the_body_has_no_model_list() -> None:
|
|
async def handler(request: web.Request) -> web.Response:
|
|
return web.json_response({"ok": True})
|
|
|
|
base, runner, _ = await _serve(handler)
|
|
try:
|
|
res = await probe_one(_openai(base), source="config")
|
|
assert res.status == "ready"
|
|
assert "unverified" in res.detail
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
async def test_openai_probe_reports_a_rejected_key() -> None:
|
|
async def handler(request: web.Request) -> web.Response:
|
|
return web.json_response({"error": "invalid api key"}, status=401)
|
|
|
|
base, runner, _ = await _serve(handler)
|
|
try:
|
|
res = await probe_one(_openai(base), source="config")
|
|
assert res.status == "attention"
|
|
assert "401" in res.detail
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
async def test_openai_probe_reports_a_missing_models_endpoint_as_unverified() -> None:
|
|
async def handler(request: web.Request) -> web.Response:
|
|
return web.Response(status=404)
|
|
|
|
base, runner, _ = await _serve(handler)
|
|
try:
|
|
res = await probe_one(_openai(base), source="config")
|
|
assert res.status == "attention"
|
|
assert "unverified" in res.detail
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
async def test_openai_probe_reports_an_unreachable_endpoint_as_missing() -> None:
|
|
res = await probe_one(_openai(f"http://127.0.0.1:{_free_port()}/v1"), source="config")
|
|
assert res.status == "missing"
|
|
assert "unreachable" in res.detail
|
|
|
|
|
|
async def test_openai_probe_honors_env_proxy(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# Same guarantee `test_openai_backend_honors_env_proxy` pins for the backend,
|
|
# and it must hold here too: aiohttp ignores HTTP_PROXY unless trust_env is
|
|
# set, so on a host that can only reach the provider through a proxy a probe
|
|
# without it would report a working agent unreachable.
|
|
async def handler(request: web.Request) -> web.Response:
|
|
return web.json_response({"data": [{"id": "m1"}]})
|
|
|
|
proxy_base, runner, _ = await _serve(handler, route="/{tail:.*}")
|
|
dead_port = _free_port()
|
|
for var in ("no_proxy", "NO_PROXY", "https_proxy", "HTTPS_PROXY", "HTTP_PROXY"):
|
|
monkeypatch.delenv(var, raising=False)
|
|
monkeypatch.setenv("http_proxy", proxy_base.removesuffix("/v1"))
|
|
try:
|
|
res = await probe_one(_openai(f"http://127.0.0.1:{dead_port}/v1", model="m1"), source="config")
|
|
assert res.status == "ready"
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
async def test_openai_probe_sends_nothing_for_a_keyless_preset() -> None:
|
|
async def handler(request: web.Request) -> web.Response:
|
|
return web.json_response({"data": []})
|
|
|
|
base, runner, seen = await _serve(handler)
|
|
try:
|
|
res = await probe_one(_openai(base, api_key=""), source="preset")
|
|
assert res.status == "attention"
|
|
assert seen == [] # a template must not fire a request that is certain to 401
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
async def test_openai_probe_does_send_for_a_keyless_configured_entry() -> None:
|
|
# A keyless endpoint is legitimate (a local vLLM), so short-circuiting a
|
|
# configured entry would report a working agent as broken.
|
|
async def handler(request: web.Request) -> web.Response:
|
|
return web.json_response({"data": [{"id": "m1"}]})
|
|
|
|
base, runner, seen = await _serve(handler)
|
|
try:
|
|
res = await probe_one(_openai(base, api_key=""), source="config")
|
|
assert res.status == "ready"
|
|
assert seen == ["/v1/models"]
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
# --- capability limits ---------------------------------------------------
|
|
|
|
|
|
def test_openai_config_coerces_declared_local_file_access_away() -> None:
|
|
# `reads_local_files` is rendered into the roster that closes the spawn tool's
|
|
# description as a `local-files` tag, so honouring it here would licence the dispatching model
|
|
# to hand this agent a path that nothing can open. It is coerced rather than
|
|
# rejected because the older web form defaulted the checkbox to true and
|
|
# rendered it for both kinds: rejecting on load would raise inside the whole
|
|
# top-level Config for anyone with such an entry stored, so raven would stop
|
|
# starting and the UI that could fix the field sits behind that config.
|
|
cfg = ThirdPartyOpenAISubagentConfig(name="api", base_url="http://x/v1", model="m", reads_local_files=True)
|
|
assert cfg.reads_local_files is False
|
|
|
|
|
|
def test_a_stored_openai_entry_with_local_file_access_still_loads() -> None:
|
|
# The regression this guards is startup, not the field: one such entry used
|
|
# to fail the entire Config, taking every unrelated section with it.
|
|
config = Config.model_validate(
|
|
{
|
|
"subagents": {
|
|
"thirdParty": [
|
|
{
|
|
"name": "legacy",
|
|
"kind": "openai",
|
|
"baseUrl": "http://x/v1",
|
|
"model": "m",
|
|
"readsLocalFiles": True,
|
|
}
|
|
]
|
|
}
|
|
}
|
|
)
|
|
assert config.subagents.agents[0].reads_local_files is False
|
|
|
|
|
|
def test_openai_config_accepts_false_and_omitted() -> None:
|
|
assert ThirdPartyOpenAISubagentConfig(name="a", base_url="http://x/v1", model="m").reads_local_files is False
|
|
explicit = ThirdPartyOpenAISubagentConfig(name="b", base_url="http://x/v1", model="m", reads_local_files=False)
|
|
assert explicit.reads_local_files is False
|
|
|
|
|
|
def test_cli_config_still_accepts_local_file_access() -> None:
|
|
# A cli agent is a local subprocess, so the declaration is real there.
|
|
assert ThirdPartyCliSubagentConfig(name="c", command="echo {prompt}", reads_local_files=True).reads_local_files
|
|
|
|
|
|
# --- batch ---------------------------------------------------------------
|
|
|
|
|
|
async def test_probe_all_preserves_order_and_captures_the_path_once(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
calls: list[int] = []
|
|
|
|
def counted() -> str:
|
|
calls.append(1)
|
|
return str(tmp_path)
|
|
|
|
monkeypatch.setattr(probe_mod, "_login_path", counted)
|
|
entries = [(_cli("a {prompt}", name="a"), "config"), (_cli("b {prompt}", name="b"), "preset")]
|
|
results = await probe_all(entries)
|
|
assert [r.name for r in results] == ["a", "b"]
|
|
assert [r.source for r in results] == ["config", "preset"]
|
|
# login_shell_env shells out to `bash -lic` and can block for seconds on its
|
|
# first call; per-agent capture would serialise the whole batch behind it.
|
|
assert len(calls) == 1
|
|
|
|
|
|
async def test_probe_all_survives_a_raising_login_path_capture(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# Neither probe_one nor probe_all may raise, on any path -- including the
|
|
# shared PATH capture, which today only degrades (login_shell_env falls
|
|
# back to os.environ) but is not guaranteed to.
|
|
def boom() -> str:
|
|
raise OSError("no bash")
|
|
|
|
monkeypatch.setattr(probe_mod, "_login_path", boom)
|
|
entries = [(_cli("a {prompt}", name="a"), "config"), (_cli("b {prompt}", name="b"), "preset")]
|
|
results = await probe_all(entries)
|
|
assert [r.name for r in results] == ["a", "b"]
|
|
for res in results:
|
|
assert res.status in {"ready", "attention", "missing", "unknown"}
|
|
|
|
|
|
async def test_probe_result_wire_shape_is_camel_case(tmp_path: Path) -> None:
|
|
res = await probe_one(_cli("nope {prompt}"), source="config", path=str(tmp_path))
|
|
assert set(res.to_wire()) == {"name", "source", "kind", "status", "detail", "target", "elapsedMs", "lastTest"}
|
|
|
|
|
|
async def test_probe_all_attaches_a_verdict_to_the_matching_result(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
from raven.agent.subagent.probe_state import LastTest
|
|
|
|
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
|
|
a = _cli("a {prompt}", name="a")
|
|
b = _cli("b {prompt}", name="b")
|
|
verdicts = {"config:a": LastTest(ok=False, detail="auth failed", tested_at_ms=1700)}
|
|
results = await probe_all([(a, "config"), (b, "config")], verdicts=verdicts)
|
|
assert results[0].last_test == LastTest(ok=False, detail="auth failed", tested_at_ms=1700)
|
|
assert results[1].last_test is None
|
|
|
|
|
|
async def test_probe_wire_shape_carries_last_test(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
from raven.agent.subagent.probe_state import LastTest
|
|
|
|
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
|
|
cfg = _cli("nope {prompt}", name="a")
|
|
[bare] = await probe_all([(cfg, "config")])
|
|
assert bare.to_wire()["lastTest"] is None
|
|
[withv] = await probe_all([(cfg, "config")], verdicts={"config:a": LastTest(True, "ran", 42)})
|
|
assert withv.to_wire()["lastTest"] == {"ok": True, "detail": "ran", "testedAtMs": 42}
|
|
|
|
|
|
# --- explicit test -------------------------------------------------------
|
|
|
|
|
|
def _patch_login_env_for_spawn(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
|
"""Point both the probe's PATH read and a real spawn's child env at ``tmp_path``.
|
|
|
|
Patching ``probe_mod._login_path`` alone (as the probe-only tests above do)
|
|
only steers ``probe_one``: ``CliAgentBackend._exec`` builds its child's
|
|
environment from ``login_shell_env()`` directly, never through anything the
|
|
probe passes, so a run that reaches a real spawn needs the shared cache both
|
|
layers read from patched instead - the same pattern
|
|
``test_subagent_third_party.py``'s ``_clear_login_env_cache`` fixture uses.
|
|
"""
|
|
monkeypatch.setattr(
|
|
env_mod,
|
|
"_LOGIN_ENV",
|
|
{"PATH": f"{tmp_path}:/usr/bin:/bin", "HOME": os.environ.get("HOME", "/root")},
|
|
)
|
|
|
|
|
|
async def test_test_of_a_missing_cli_fails_without_building_a_backend(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
# A failed probe must short-circuit: there is nothing to execute, so no
|
|
# backend is built and no process is spawned.
|
|
def must_not_build(*args, **kwargs):
|
|
raise AssertionError("a failed probe must not reach the backend")
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", must_not_build)
|
|
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
|
|
res = await run_test(_cli("not-installed-at-all {prompt}"), source="config")
|
|
assert res.ok is False
|
|
assert res.kind == "cli"
|
|
assert "not on the login shell PATH" in res.detail
|
|
|
|
|
|
async def test_test_of_a_working_cli_returns_the_reply(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
exe = tmp_path / "echo-agent"
|
|
exe.write_text("#!/bin/sh\necho PONG\n")
|
|
exe.chmod(0o755)
|
|
_patch_login_env_for_spawn(monkeypatch, tmp_path)
|
|
res = await run_test(_cli("echo-agent {prompt}"), source="config")
|
|
assert res.ok is True
|
|
assert res.reply == "PONG"
|
|
assert res.elapsed_ms >= 0
|
|
|
|
|
|
async def test_test_of_a_cli_that_returns_nothing_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# Exit 0 with an empty answer is exactly the openclaw symptom of replying to
|
|
# its workspace bootstrap instead of the task, so it is not a pass.
|
|
exe = tmp_path / "silent-agent"
|
|
exe.write_text("#!/bin/sh\nexit 0\n")
|
|
exe.chmod(0o755)
|
|
_patch_login_env_for_spawn(monkeypatch, tmp_path)
|
|
res = await run_test(_cli("silent-agent {prompt}"), source="config")
|
|
assert res.ok is False
|
|
assert "returned nothing" in res.detail
|
|
|
|
|
|
async def test_test_of_a_failing_cli_reports_its_error_rather_than_raising(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
exe = tmp_path / "broken-agent"
|
|
exe.write_text("#!/bin/sh\necho 'ProviderAuthError' >&2\nexit 1\n")
|
|
exe.chmod(0o755)
|
|
_patch_login_env_for_spawn(monkeypatch, tmp_path)
|
|
res = await run_test(_cli("broken-agent {prompt}"), source="config")
|
|
assert res.ok is False
|
|
assert "ProviderAuthError" in res.detail
|
|
|
|
|
|
@pytest.mark.skipif(os.name == "nt", reason="the stand-in CLI is a POSIX shell script")
|
|
async def test_a_cli_test_whose_directory_stays_in_use_reports_the_reply(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""On Windows a process the CLI left running keeps the directory; the test reports what the CLI said."""
|
|
exe = tmp_path / "echo-agent"
|
|
exe.write_text("#!/bin/sh\necho PONG\n")
|
|
exe.chmod(0o755)
|
|
_patch_login_env_for_spawn(monkeypatch, tmp_path)
|
|
root = scratch_root(tmp_path, monkeypatch)
|
|
refused = windows_removal(monkeypatch, lambda path: path.parent == root)
|
|
|
|
res = await run_test(_cli("echo-agent {prompt}"), source="config")
|
|
|
|
assert (res.ok, res.reply) == (True, "PONG")
|
|
assert refused, "no removal was refused, so this tested nothing"
|
|
|
|
|
|
async def test_stateful_cli_test_leaves_the_real_registry_untouched(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
exe = tmp_path / "stateful-agent"
|
|
exe.write_text("#!/bin/sh\necho PONG\n")
|
|
exe.chmod(0o755)
|
|
_patch_login_env_for_spawn(monkeypatch, tmp_path)
|
|
cfg = ThirdPartyCliSubagentConfig(
|
|
name="stateful",
|
|
command="stateful-agent {prompt} --session-id {agent_id}",
|
|
resume_command="stateful-agent {prompt} --resume {agent_id}",
|
|
)
|
|
res = await run_test(cfg, source="config")
|
|
assert res.ok is True
|
|
# The create commits a handle binding; it must land in the probe's throwaway
|
|
# registry, not in the one the running gateway reads.
|
|
assert instances_mod._registry.list_instances() == []
|
|
|
|
|
|
async def test_an_openai_test_asks_the_endpoint_to_answer() -> None:
|
|
"""Both halves, in this order, and the order is the whole economy of it.
|
|
|
|
The free models probe goes first and refuses on its own when the endpoint
|
|
does not know the credential, so nothing is spent on one that cannot answer.
|
|
Past it, the endpoint is asked the same prompt every other kind is asked,
|
|
because knowing the credential is not the same as being able to serve the
|
|
model behind it.
|
|
"""
|
|
|
|
async def models(request: web.Request) -> web.Response:
|
|
return web.json_response({"data": [{"id": "m1"}]})
|
|
|
|
seen: list[str] = []
|
|
port = _free_port()
|
|
app = web.Application()
|
|
|
|
async def record_models(request: web.Request) -> web.Response:
|
|
seen.append(request.path)
|
|
return await models(request)
|
|
|
|
async def record_completion(request: web.Request) -> web.Response:
|
|
seen.append(request.path)
|
|
return web.json_response({"choices": [{"message": {"content": "billed!"}}]})
|
|
|
|
app.router.add_get("/v1/models", record_models)
|
|
app.router.add_post("/v1/chat/completions", record_completion)
|
|
runner = web.AppRunner(app)
|
|
await runner.setup()
|
|
await web.TCPSite(runner, "127.0.0.1", port).start()
|
|
try:
|
|
res = await run_test(_openai(f"http://127.0.0.1:{port}/v1", model="m1"), source="config")
|
|
assert res.ok is True
|
|
assert res.reply is None
|
|
assert seen == ["/v1/models", "/v1/chat/completions"]
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
async def test_an_openai_test_asks_an_endpoint_that_serves_no_model_list() -> None:
|
|
"""The free probe vetoes only when it has proved there is nothing to send to.
|
|
|
|
`/models` is optional: the backend only ever POSTs `/chat/completions`, so an
|
|
endpoint that serves completions and answers 404 for the list is a working
|
|
agent. Letting the probe veto it would fail a Test that Connect accepts --
|
|
the very disagreement this gate exists to remove -- so anything reachable is
|
|
settled by asking it.
|
|
"""
|
|
seen: list[str] = []
|
|
port = _free_port()
|
|
app = web.Application()
|
|
|
|
async def no_model_list(request: web.Request) -> web.Response:
|
|
seen.append(request.path)
|
|
return web.json_response({"error": "not found"}, status=404)
|
|
|
|
async def record_completion(request: web.Request) -> web.Response:
|
|
seen.append(request.path)
|
|
return web.json_response({"choices": [{"message": {"content": "PONG"}}]})
|
|
|
|
app.router.add_get("/v1/models", no_model_list)
|
|
app.router.add_post("/v1/chat/completions", record_completion)
|
|
runner = web.AppRunner(app)
|
|
await runner.setup()
|
|
await web.TCPSite(runner, "127.0.0.1", port).start()
|
|
try:
|
|
res = await run_test(_openai(f"http://127.0.0.1:{port}/v1", model="m1"), source="config")
|
|
assert res.ok is True, res.detail
|
|
assert seen == ["/v1/models", "/v1/chat/completions"]
|
|
finally:
|
|
await runner.cleanup()
|
|
|
|
|
|
async def test_an_openai_test_spends_nothing_on_an_endpoint_it_cannot_reach() -> None:
|
|
"""The one verdict the free probe can reach alone: nothing is listening, so
|
|
there is no request to make and no quota to consider."""
|
|
port = _free_port()
|
|
|
|
res = await run_test(_openai(f"http://127.0.0.1:{port}/v1", model="m1"), source="config")
|
|
|
|
assert res.ok is False
|
|
assert "unreachable" in res.detail
|
|
|
|
|
|
async def test_test_result_wire_shape_is_camel_case(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setattr(probe_mod, "_login_path", lambda: str(tmp_path))
|
|
wire = (await run_test(_cli("nope {prompt}"), source="config")).to_wire()
|
|
assert set(wire) == {"name", "source", "kind", "ok", "detail", "reply", "elapsedMs"}
|
|
|
|
|
|
async def test_testing_a_builtin_agent_is_refused_rather_than_attempted() -> None:
|
|
"""There is nothing a test could check about an in-process agent.
|
|
|
|
No command to launch, no endpoint to authenticate against, no transcript
|
|
parser to exercise -- the three things a test exists to catch. Falling through
|
|
to the cli branch did run, though: the probe now reports a built-in row
|
|
"ready", so it reached ``build_third_party_backend``, which raises on the kind,
|
|
and the bare-except turned that into a verdict claiming ``kind == "cli"`` with
|
|
the detail "unknown third-party subagent kind". The caller then *records*
|
|
verdicts, so that row would have shown a failed test permanently.
|
|
"""
|
|
from raven.agent.subagent.probe import run_test
|
|
from raven.config.schema import BuiltinAgentConfig
|
|
|
|
result = await run_test(BuiltinAgentConfig(name="research-raven"), source="config")
|
|
|
|
assert result.kind == "builtin"
|
|
assert result.ok is False
|
|
assert "nothing to test" in result.detail
|
|
assert "unknown third-party" not in result.detail
|
|
|
|
|
|
# --- ping ----------------------------------------------------------------
|
|
|
|
|
|
async def test_ping_dispatches_the_probe_prompt_and_passes_on_a_reply(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""Layer 2 is a real turn, because layer 1 cannot see a credential.
|
|
|
|
Measured 2026-09-07: six registry agents answered `initialize` and
|
|
`session/new` and then failed the first model call, which is where their
|
|
credential is actually checked.
|
|
"""
|
|
sent: list[str] = []
|
|
|
|
class _Answers:
|
|
async def run(self, prompt, **kwargs):
|
|
sent.append(prompt)
|
|
return "PONG"
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Answers())
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp")
|
|
result = await probe_mod.ping_agent(cfg)
|
|
|
|
assert sent == [probe_mod.PROBE_PROMPT]
|
|
assert result.ok is True
|
|
assert "replied" in result.detail
|
|
|
|
|
|
async def test_ping_fails_when_the_agent_answers_nothing(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""An empty reply is a signal, not a pass -- openclaw answered its own
|
|
workspace bootstrap instead of the task and returned nothing."""
|
|
|
|
class _Silent:
|
|
async def run(self, prompt, **kwargs):
|
|
return " "
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Silent())
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(name="silent-acp", kind="acp", command="fake-agent acp")
|
|
result = await probe_mod.ping_agent(cfg)
|
|
|
|
assert result.ok is False
|
|
assert "answered nothing" in result.detail
|
|
|
|
|
|
async def test_ping_fails_and_keeps_the_reason_when_the_model_call_is_refused(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""The exact shape six measured agents have: layer 1 fine, model call refused.
|
|
|
|
The reason has to survive into the detail, because it is the whole value of
|
|
the failure -- "sign in" and "-32603" send an operator to different places.
|
|
"""
|
|
|
|
class _Refuses:
|
|
async def run(self, prompt, **kwargs):
|
|
raise RuntimeError("Internal error: You need to sign in to use this model.")
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Refuses())
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(name="unauthed-acp", kind="acp", command="fake-agent acp")
|
|
result = await probe_mod.ping_agent(cfg)
|
|
|
|
assert result.ok is False
|
|
assert "sign in" in result.detail
|
|
|
|
|
|
async def test_ping_fails_on_a_timeout_rather_than_waiting(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""gemini and dirac hang rather than refuse, so an unbounded wait would sit
|
|
on the settings switch and then say nothing useful."""
|
|
|
|
class _Hangs:
|
|
async def run(self, prompt, **kwargs):
|
|
await asyncio.sleep(30)
|
|
return "too late"
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Hangs())
|
|
monkeypatch.setattr(probe_mod, "_ENABLE_PING_TIMEOUT_SECONDS", 1)
|
|
|
|
# The wait is the start window plus the answer's (`_ping_bounds`), so the
|
|
# row declares a short start window of its own: the hang has to outlast both.
|
|
cfg = ThirdPartyAcpSubagentConfig(name="hanging-acp", kind="acp", command="fake-agent acp", ready_timeout_ms=1000)
|
|
result = await probe_mod.ping_agent(cfg)
|
|
|
|
assert result.ok is False
|
|
assert "did not answer within 2s" in result.detail
|
|
|
|
|
|
async def test_ping_hands_the_backend_the_same_bound_it_waits_for(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""One number, or the inner bound is unreachable: a backend allowed the
|
|
longer explicit-Test budget can only ever be cancelled by the `wait_for`
|
|
around it, never reach its own timeout and describe the failure.
|
|
|
|
This half asserts what `ping_agent` asks for, and can assert nothing more:
|
|
the builder is replaced below, so a builder that accepted both numbers and
|
|
used neither would satisfy every line of it. The other half is
|
|
`test_the_factory_honours_the_bounds_its_caller_overrides`, which builds for
|
|
real and reads them back off the backend."""
|
|
seen: dict = {}
|
|
|
|
class _Answers:
|
|
async def run(self, prompt, **kwargs):
|
|
return "PONG"
|
|
|
|
def _capture(cfg, **kw):
|
|
seen.update(kw)
|
|
return _Answers()
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", _capture)
|
|
|
|
await probe_mod.ping_agent(ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp"))
|
|
assert seen["timeout"] == probe_mod._ENABLE_PING_TIMEOUT_SECONDS
|
|
|
|
# A row that asks for less than the cap still gets its own number.
|
|
tight = ThirdPartyAcpSubagentConfig(name="ping-tight", kind="acp", command="fake-agent acp", timeout=5)
|
|
await probe_mod.ping_agent(tight)
|
|
assert seen["timeout"] == 5
|
|
|
|
# The handshake gets the row's own start window, uncapped: several shipped
|
|
# presets declare 120s because `npx` downloads the adapter there on a first
|
|
# connect, and npm takes about 70s to report a registry it cannot reach. Cut
|
|
# to the answer's cap, both came back "did not answer within 60s".
|
|
slow_start = ThirdPartyAcpSubagentConfig(
|
|
name="ping-slow-start", kind="acp", command="fake-agent acp", ready_timeout_ms=120000
|
|
)
|
|
waited: list[float] = []
|
|
real_wait_for = asyncio.wait_for
|
|
|
|
async def _spy(awaitable, timeout):
|
|
waited.append(timeout)
|
|
return await real_wait_for(awaitable, timeout)
|
|
|
|
monkeypatch.setattr(probe_mod.asyncio, "wait_for", _spy)
|
|
await probe_mod.ping_agent(slow_start)
|
|
assert seen["ready_timeout_ms"] == 120000
|
|
# ...and the wait around the run leaves room for both inner bounds, so
|
|
# neither is cancelled from outside before the backend can say which ran out.
|
|
assert waited == [120 + probe_mod._ENABLE_PING_TIMEOUT_SECONDS]
|
|
|
|
quick_start = ThirdPartyAcpSubagentConfig(
|
|
name="ping-quick-start", kind="acp", command="fake-agent acp", ready_timeout_ms=4000
|
|
)
|
|
await probe_mod.ping_agent(quick_start)
|
|
assert seen["ready_timeout_ms"] == 4000
|
|
assert waited[-1] == 4 + probe_mod._ENABLE_PING_TIMEOUT_SECONDS
|
|
|
|
# A kind with no handshake is waited on for its answer alone.
|
|
cli = ThirdPartyCliSubagentConfig(name="ping-cli", kind="cli", command="fake-agent {prompt}")
|
|
await probe_mod.ping_agent(cli)
|
|
assert waited[-1] == probe_mod._ENABLE_PING_TIMEOUT_SECONDS
|
|
|
|
|
|
async def test_ping_runs_on_a_connection_pool_of_its_own(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""A ping must not reach the pool the running roster dispatches on.
|
|
|
|
The pool keys a connection on its launch arguments, and ``cwd`` falls back
|
|
to the caller's workspace -- which for a ping is a fresh temporary directory
|
|
every time. So a ping on the shared pool always computes a launch key no
|
|
held connection matches, and ``acquire`` retires every connection of that
|
|
agent before launching its own. Two consequences, both measured against a
|
|
live adapter on 2026-09-20: a second Connect pressed while the first was
|
|
still running killed the first one's connection, and the first came back
|
|
"did not answer a test message" for a failure raven had caused; and a ping
|
|
fired while that agent was serving a real run would have taken that run's
|
|
connection down with it.
|
|
|
|
This half asserts what ``ping_agent`` asks the factory for. The other half
|
|
is ``test_the_factory_honours_the_pool_its_caller_overrides``.
|
|
"""
|
|
from raven.acp_client.pool import AcpConnectionPool, get_pool
|
|
|
|
seen: dict = {}
|
|
|
|
class _Answers:
|
|
async def run(self, prompt, **kwargs):
|
|
return "PONG"
|
|
|
|
def _capture(cfg, **kw):
|
|
seen.update(kw)
|
|
return _Answers()
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", _capture)
|
|
|
|
await probe_mod.ping_agent(ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp"))
|
|
|
|
assert isinstance(seen["pool"], AcpConnectionPool)
|
|
assert seen["pool"] is not get_pool()
|
|
|
|
|
|
async def test_ping_closes_the_pool_it_opened(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Or every press leaks the agent process it launched.
|
|
|
|
The private pool above is per-ping, so nothing else will ever close it: a
|
|
pool left open holds a live child process for the rest of the gateway's
|
|
life, and the roster would accumulate one per Connect pressed.
|
|
"""
|
|
import raven.acp_client.pool as pool_mod
|
|
|
|
closed: list[int] = []
|
|
|
|
class _Counting(pool_mod.AcpConnectionPool):
|
|
async def close_all(self) -> None:
|
|
closed.append(1)
|
|
await super().close_all()
|
|
|
|
monkeypatch.setattr(pool_mod, "AcpConnectionPool", _Counting)
|
|
|
|
class _Answers:
|
|
async def run(self, prompt, **kwargs):
|
|
return "PONG"
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Answers())
|
|
|
|
await probe_mod.ping_agent(ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp"))
|
|
assert closed == [1]
|
|
|
|
# And on the failing path too, which is the one that runs when an agent is
|
|
# the reason the press is happening at all.
|
|
class _Refuses:
|
|
async def run(self, prompt, **kwargs):
|
|
raise RuntimeError("Internal error: You need to sign in to use this model.")
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Refuses())
|
|
|
|
result = await probe_mod.ping_agent(
|
|
ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp")
|
|
)
|
|
assert result.ok is False
|
|
assert closed == [1, 1]
|
|
|
|
|
|
async def test_a_directory_its_agent_will_not_let_go_does_not_fail_a_ping_that_answered(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""The verdict is the agent's answer; the directory is logged and left, not reported in its place.
|
|
|
|
A refused removal raised out of the ping was reported as "did not answer a
|
|
test message" about an agent that had answered: on native Windows that is
|
|
every Connect of an agent whose processes still worked in the directory.
|
|
"""
|
|
root = scratch_root(tmp_path, monkeypatch)
|
|
|
|
class _Answers:
|
|
async def run(self, prompt, **kwargs):
|
|
return "PONG"
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Answers())
|
|
refused = windows_removal(monkeypatch, lambda path: path.parent == root)
|
|
|
|
result = await probe_mod.ping_agent(
|
|
ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp")
|
|
)
|
|
|
|
assert result == probe_mod.PingResult(True, "it ran and replied")
|
|
assert refused, "no removal was refused, so this tested nothing"
|
|
assert [path.name.startswith("raven_subagent_ping_") for path in root.iterdir()] == [True]
|
|
|
|
|
|
async def test_a_ping_that_runs_out_says_so_while_its_directory_is_still_in_use(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""A refusal raised on the way out took the timeout's place, so a silent agent read as a file error."""
|
|
root = scratch_root(tmp_path, monkeypatch)
|
|
|
|
class _Hangs:
|
|
async def run(self, prompt, **kwargs):
|
|
await asyncio.sleep(30)
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", lambda cfg, **kw: _Hangs())
|
|
monkeypatch.setattr(probe_mod, "_ENABLE_PING_TIMEOUT_SECONDS", 0.2)
|
|
refused = windows_removal(monkeypatch, lambda path: path.parent == root)
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(name="hanging-acp", kind="acp", command="fake-agent acp", ready_timeout_ms=100)
|
|
result = await probe_mod.ping_agent(cfg)
|
|
|
|
assert result.ok is False
|
|
assert result.detail.startswith("it did not answer within"), result.detail
|
|
assert refused, "no removal was refused, so this tested nothing"
|
|
|
|
|
|
async def test_a_ping_with_nowhere_to_make_its_directory_says_so_rather_than_raising(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
import tempfile
|
|
|
|
monkeypatch.setattr(tempfile, "tempdir", str(tmp_path / "gone"))
|
|
|
|
def must_not_build(*args, **kwargs):
|
|
raise AssertionError("there is no directory to hand the agent")
|
|
|
|
monkeypatch.setattr(probe_mod, "build_third_party_backend", must_not_build)
|
|
|
|
result = await probe_mod.ping_agent(
|
|
ThirdPartyAcpSubagentConfig(name="ping-acp", kind="acp", command="fake-agent acp")
|
|
)
|
|
|
|
assert result.ok is False
|
|
assert "gone" in result.detail
|
|
|
|
|
|
def test_the_factory_honours_the_pool_its_caller_overrides() -> None:
|
|
"""Read off a real build, for the reason the bounds test states: a request
|
|
recorded by a replaced builder proves nothing about what the backend then
|
|
dispatches on."""
|
|
from raven.acp_client.pool import AcpConnectionPool, get_pool
|
|
from raven.agent.subagent.backends import build_third_party_backend
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(name="factory-pool", kind="acp", command="fake-agent acp")
|
|
private = AcpConnectionPool()
|
|
|
|
assert build_third_party_backend(cfg, pool=private).pool is private
|
|
# Omitted, and a dispatch goes where every other dispatch goes.
|
|
assert build_third_party_backend(cfg).pool is get_pool()
|
|
|
|
|
|
def test_the_factory_honours_the_bounds_its_caller_overrides() -> None:
|
|
"""Both overrides exist for one caller, so both are read off a real build.
|
|
|
|
Asserting on the arguments a replaced builder recorded proves the request and
|
|
not the effect: dropping either honouring line left the whole subagent suite
|
|
green. Read them back off the object instead, and the test stays true of the
|
|
next bound the factory learns to take.
|
|
"""
|
|
from raven.agent.subagent.backends import build_third_party_backend
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(
|
|
name="factory-bounds",
|
|
kind="acp",
|
|
command="fake-agent acp",
|
|
ready_timeout_ms=120000,
|
|
timeout=300,
|
|
)
|
|
backend = build_third_party_backend(cfg, ready_timeout_ms=45000, timeout=60)
|
|
assert backend.ready_timeout_ms == 45000
|
|
assert backend.timeout == 60
|
|
|
|
# Omitted, and the config's own numbers stand -- the overrides are for the
|
|
# caller that has a shorter budget, not a rewrite of the row.
|
|
untouched = build_third_party_backend(cfg)
|
|
assert untouched.ready_timeout_ms == 120000
|
|
assert untouched.timeout == 300
|
|
|
|
|
|
def _host_with_a_provider_key() -> None:
|
|
"""Give the isolated home one usable provider.
|
|
|
|
The empty-menu re-measure is bounded on this: a child raven advertises no
|
|
model at all while the catalogue it inherits is empty, so a host with no
|
|
credentials would be relaunching children to be told what it already knows.
|
|
Written to the real config the bound reads rather than stubbing the check,
|
|
since "which sections count as usable" is half of what is under test.
|
|
"""
|
|
from raven.home import get_config_path
|
|
|
|
get_config_path().write_text(json.dumps({"providers": {"openai": {"apiKey": "sk-test"}}}), encoding="utf-8")
|
|
|
|
|
|
class _FakeRow:
|
|
def __init__(self, name: str, *, kind: str = "acp", enabled: bool = True, config: object = None) -> None:
|
|
self.name = name
|
|
self.kind = kind
|
|
self.enabled = enabled
|
|
self.config = config if config is not None else type("C", (), {"name": name})()
|
|
|
|
|
|
class _FakeManager:
|
|
def __init__(self, rows: list[_FakeRow], *, with_refresh: bool = False) -> None:
|
|
self.registry = type("R", (), {"rows": lambda self: rows})()
|
|
if with_refresh:
|
|
self.refresh_calls = 0
|
|
|
|
def refresh_agents() -> None:
|
|
self.refresh_calls += 1
|
|
|
|
self.refresh_agents = refresh_agents
|
|
|
|
|
|
class TestAutomaticSnapshotVerification:
|
|
async def test_only_missing_or_stale_rows_are_verified(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
from dataclasses import dataclass
|
|
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
@dataclass
|
|
class Snap:
|
|
status: str = "ready"
|
|
stale: bool = False
|
|
|
|
fresh, missing, stale = _FakeRow("fresh"), _FakeRow("missing"), _FakeRow("stale")
|
|
recorded: list[str] = []
|
|
|
|
def fake_snapshot_for(cfg: object) -> Snap | None:
|
|
return {id(fresh.config): Snap(), id(missing.config): None, id(stale.config): Snap(stale=True)}[id(cfg)]
|
|
|
|
async def fake_verify(cfg: object) -> Snap:
|
|
return Snap()
|
|
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", fake_snapshot_for)
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(lambda s: recorded.append(getattr(s, "status")))})(),
|
|
)
|
|
# This one pins the registry half. The preset half reads the machine's
|
|
# own PATH, so leaving it live would make the assertions below depend on
|
|
# which agents happen to be installed here.
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
manager = _FakeManager([fresh, missing, stale], with_refresh=True)
|
|
task = schedule_snapshot_verification(manager)
|
|
assert task is not None
|
|
assert task in probe_mod._VERIFY_TASKS, "an unrefed task can be collected mid-run"
|
|
await task
|
|
assert task not in probe_mod._VERIFY_TASKS, "the done callback must release the reference"
|
|
|
|
assert len(recorded) == 2 # both missing and stale round-tripped to a fresh snapshot
|
|
assert manager.refresh_calls == 1, "the materialized rows must be rebuilt after recording"
|
|
|
|
async def test_a_credential_refusal_is_recorded_and_not_only_a_pass(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Recording only ``ready`` meant the one verdict worth remembering was
|
|
the one thrown away.
|
|
|
|
An agent that answers the handshake and then refuses to open a session
|
|
without a credential has told us something durable about itself, and the
|
|
agents page reads exactly that to say `Unauthorized` instead of offering
|
|
a Connect that spends a launch to be refused again. Any other failure is
|
|
recorded only over a row whose last record had already failed, replacing
|
|
that verdict and keeping its capabilities. Over a row with no record, as
|
|
here, or a passing one, a timeout or a crashed adapter stays unrecorded:
|
|
it is a fact about this minute, and freezing it into a snapshot would
|
|
label a working agent broken until someone pressed Test.
|
|
"""
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
recorded: list[str] = []
|
|
verdicts = {
|
|
"refused": type("S", (), {"status": "attention", "needs_auth": True})(),
|
|
"wedged": type("S", (), {"status": "unknown", "needs_auth": False})(),
|
|
"fine": type("S", (), {"status": "ready", "needs_auth": False})(),
|
|
}
|
|
|
|
async def fake_verify(cfg: object) -> object:
|
|
return verdicts[cfg.name]
|
|
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(lambda s: recorded.append(s.status))})(),
|
|
)
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
await schedule_snapshot_verification(_FakeManager([_FakeRow("refused"), _FakeRow("wedged"), _FakeRow("fine")]))
|
|
assert sorted(recorded) == ["attention", "ready"]
|
|
|
|
@staticmethod
|
|
def _mirasim_boot(
|
|
monkeypatch: pytest.MonkeyPatch, snapshot: object, installed: frozenset[str], own_host: str | None = None
|
|
) -> tuple[list, list]:
|
|
"""A boot whose unconfigured presets are three Mirasim agents and one other; what it verified and asked.
|
|
|
|
``own_host`` names the one Mirasim row whose ``env`` points it at a host of its own.
|
|
"""
|
|
from raven.agent.subagent import mirasim_presets
|
|
from raven.agent.subagent.presets import third_party_subagent_preset
|
|
from raven.config.schema import SubagentsConfig
|
|
|
|
keys = ("mirasim_claude", "mirasim_zcode", "mirasim_qwen")
|
|
rows = [
|
|
{**third_party_subagent_preset(key), **({"env": {"RAVEN_MIRASIM_HOST": "4971"}} if key == own_host else {})}
|
|
for key in keys
|
|
]
|
|
configs = SubagentsConfig(agents=rows).agents
|
|
verified: list[str] = []
|
|
asked: list[str | None] = []
|
|
|
|
async def fake_verify(cfg: object) -> object:
|
|
verified.append(cfg.name)
|
|
return type("S", (), {"status": "ready", "needs_auth": False})()
|
|
|
|
async def fake_installed(*, path: str | None) -> frozenset[str]:
|
|
asked.append(path)
|
|
return installed
|
|
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: snapshot)
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
|
|
)
|
|
monkeypatch.setattr(
|
|
probe_mod,
|
|
"_unconfigured_acp_preset_rows",
|
|
lambda configured, path=None: [*(probe_mod._PresetRow(cfg) for cfg in configs), _FakeRow("a-preset")],
|
|
)
|
|
monkeypatch.setattr(mirasim_presets, "installed_agents", fake_installed, raising=False)
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
return verified, asked
|
|
|
|
async def test_a_mirasim_preset_whose_agent_mirasim_lacks_is_not_handshaken(
|
|
self, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Its handshake fails and a preset's failure is not kept, so verifying it would handshake it again at
|
|
every start: two adapter launches or more for nothing, per such agent. Mirasim is asked once
|
|
instead."""
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
verified, asked = self._mirasim_boot(monkeypatch, None, frozenset({"claude", "qwen"}))
|
|
await schedule_snapshot_verification(_FakeManager([]))
|
|
assert verified == ["Claude Code-Mirasim", "Qwen Code-Mirasim", "a-preset"]
|
|
assert len(asked) == 1
|
|
|
|
async def test_with_no_mirasim_host_running_only_a_row_that_names_its_own_host_is_handshaken(
|
|
self, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Every other handshake would fail on the same missing host, and be tried again at the next start."""
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
verified, asked = self._mirasim_boot(monkeypatch, None, frozenset(), own_host="mirasim_qwen")
|
|
await schedule_snapshot_verification(_FakeManager([]))
|
|
assert verified == ["Qwen Code-Mirasim", "a-preset"]
|
|
assert len(asked) == 1
|
|
|
|
async def test_mirasim_is_asked_only_when_one_of_its_presets_is_to_be_handshaken(
|
|
self, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""The question launches the adapter and waits out the host's agent list, which a fully measured boot never needs."""
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
measured = type("S", (), {"status": "ready", "stale": False, "needs_auth": False})()
|
|
verified, asked = self._mirasim_boot(monkeypatch, measured, frozenset())
|
|
await schedule_snapshot_verification(_FakeManager([]))
|
|
assert verified == [] and asked == []
|
|
|
|
async def test_a_preset_nobody_has_configured_is_verified_too(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Or the page can never say `Unauthorized` before the first press.
|
|
|
|
The rows this backfill was written for come off the registry, which
|
|
holds configured agents only -- so an agent the reader has not connected
|
|
yet is exactly the one it never reached, and exactly the one whose
|
|
Connect button is about to lie to them.
|
|
"""
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
seen: list[str] = []
|
|
|
|
async def fake_verify(cfg: object) -> object:
|
|
seen.append(cfg.name)
|
|
return type("S", (), {"status": "ready", "needs_auth": False})()
|
|
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
|
|
)
|
|
monkeypatch.setattr(
|
|
probe_mod,
|
|
"_unconfigured_acp_preset_rows",
|
|
lambda configured, path=None: [_FakeRow("a-preset")] if configured else [],
|
|
)
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
await schedule_snapshot_verification(_FakeManager([_FakeRow("configured")]))
|
|
assert seen == ["configured", "a-preset"], "the configured rows first: they are the ones a run can dispatch to"
|
|
|
|
async def test_a_recorded_credential_refusal_is_measured_again(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""A recorded refusal is a verdict a restart must not take on trust.
|
|
|
|
A fresh `ready` snapshot is skipped, barring the two model-menu cases
|
|
tested below, and rightly: a `ready` agent that has not been relaunched is
|
|
still ready, and re-proving it would spend a process per row per boot. A
|
|
fresh failure is measured again, and a credential refusal is the failure
|
|
that most needs it -- it is a state the user is expected to go and fix,
|
|
and nothing about fixing it touches the launch config, so the snapshot
|
|
never goes stale and the row would keep saying "Unauthorized" across
|
|
every restart after the sign-in that cured it.
|
|
"""
|
|
from dataclasses import dataclass
|
|
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
@dataclass
|
|
class Snap:
|
|
status: str = "ready"
|
|
stale: bool = False
|
|
needs_auth: bool = False
|
|
|
|
fine, refused = _FakeRow("fine"), _FakeRow("refused")
|
|
verified: list[str] = []
|
|
|
|
def fake_snapshot_for(cfg: object) -> Snap:
|
|
return {
|
|
id(fine.config): Snap(),
|
|
id(refused.config): Snap(status="attention", needs_auth=True),
|
|
}[id(cfg)]
|
|
|
|
async def fake_verify(cfg: object) -> Snap:
|
|
verified.append(cfg.name)
|
|
return Snap()
|
|
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", fake_snapshot_for)
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
|
|
)
|
|
# This one pins the registry half. The preset half reads the machine's
|
|
# own PATH, so leaving it live would make the assertions below depend on
|
|
# which agents happen to be installed here.
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
await schedule_snapshot_verification(_FakeManager([fine, refused]))
|
|
assert verified == ["refused"], "a fresh pass is still trusted; a fresh refusal is not"
|
|
|
|
async def test_failed_verification_does_not_stop_the_rest(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
calls: list[str] = []
|
|
|
|
async def fake_verify(cfg: object) -> object:
|
|
calls.append(getattr(cfg, "tag", "?"))
|
|
if getattr(cfg, "tag", "") == "boom":
|
|
raise RuntimeError("no")
|
|
return type("S", (), {"status": "ready"})()
|
|
|
|
def make(tag: str) -> object:
|
|
return type(
|
|
"Row", (), {"name": tag, "kind": "acp", "enabled": True, "config": type("C", (), {"tag": tag})()}
|
|
)()
|
|
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
|
|
)
|
|
# This one pins the registry half. The preset half reads the machine's
|
|
# own PATH, so leaving it live would make the assertions below depend on
|
|
# which agents happen to be installed here.
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
task = schedule_snapshot_verification(_FakeManager([make("boom"), make("after")]))
|
|
await task
|
|
|
|
assert calls == ["boom", "after"]
|
|
|
|
async def test_a_manager_without_a_registry_defers_to_the_host(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""A mounted stack's subagents substitute may not carry the registry the
|
|
backfill reads; scheduling nothing must not burn the once-per-process
|
|
ticket, so a real host later in the same process still gets its backfill."""
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
called: list[str] = []
|
|
monkeypatch.setattr(
|
|
probe_mod,
|
|
"acp_snapshot_for",
|
|
lambda cfg: None,
|
|
)
|
|
|
|
async def fake_verify(cfg: object) -> object:
|
|
called.append("verified")
|
|
return type("S", (), {"status": "ready"})()
|
|
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
|
|
)
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
stub = type("Stub", (), {})()
|
|
assert schedule_snapshot_verification(stub) is None
|
|
assert probe_mod._SCHEDULED is False, "the ticket must not be spent on a mount that cannot backfill"
|
|
assert called == []
|
|
|
|
async def test_disabled_and_non_acp_rows_are_skipped(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
called: list[str] = []
|
|
monkeypatch.setattr(
|
|
probe_mod,
|
|
"acp_snapshot_for",
|
|
lambda cfg: None,
|
|
)
|
|
|
|
async def fake_verify(cfg: object) -> object:
|
|
called.append(cfg.name)
|
|
return type("S", (), {"status": "ready"})()
|
|
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
|
|
)
|
|
# This one pins the registry half. The preset half reads the machine's
|
|
# own PATH, so leaving it live would make the assertions below depend on
|
|
# which agents happen to be installed here.
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
rows = [
|
|
_FakeRow("off", enabled=False),
|
|
_FakeRow("cli-kind", kind="cli"),
|
|
_FakeRow("on"),
|
|
]
|
|
task = schedule_snapshot_verification(_FakeManager(rows))
|
|
await task
|
|
|
|
assert called == ["on"]
|
|
|
|
async def test_a_store_without_has_model_menu_does_not_force_a_reverify(
|
|
self, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""A caller's stand-in store (a bare ``record``-only object, as several
|
|
tests in this class substitute) must not be treated as "every row is on
|
|
the older format" -- the predicate is optional, and its absence must
|
|
leave a fresh, non-stale snapshot alone exactly as before this hook."""
|
|
from dataclasses import dataclass
|
|
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
@dataclass
|
|
class Snap:
|
|
status: str = "ready"
|
|
stale: bool = False
|
|
|
|
row = _FakeRow("fresh")
|
|
recorded: list[str] = []
|
|
|
|
async def fake_verify(cfg: object) -> Snap:
|
|
recorded.append(getattr(cfg, "name", "?"))
|
|
return Snap()
|
|
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: Snap())
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(lambda s: None)})(),
|
|
)
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
task = schedule_snapshot_verification(_FakeManager([row]))
|
|
await task
|
|
|
|
assert recorded == []
|
|
|
|
async def test_a_snapshot_missing_the_model_choices_key_is_reverified(
|
|
self, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""A snapshot recorded before ``modelChoices`` existed is fresh and not
|
|
stale by every other measure, but a row stuck on it would never learn
|
|
the agent's model menu until the agent is edited or Test is pressed by
|
|
hand -- so the older format alone must trigger a re-verify."""
|
|
from dataclasses import dataclass
|
|
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
@dataclass
|
|
class Snap:
|
|
status: str = "ready"
|
|
stale: bool = False
|
|
|
|
row = _FakeRow("old-format")
|
|
recorded: list[str] = []
|
|
|
|
async def fake_verify(cfg: object) -> Snap:
|
|
recorded.append(getattr(cfg, "name", "?"))
|
|
return Snap()
|
|
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: Snap())
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type(
|
|
"S",
|
|
(),
|
|
{
|
|
"record": staticmethod(lambda s: None),
|
|
"has_model_menu": staticmethod(lambda agent: False),
|
|
},
|
|
)(),
|
|
)
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
task = schedule_snapshot_verification(_FakeManager([row]))
|
|
await task
|
|
|
|
assert recorded == ["old-format"]
|
|
|
|
async def test_one_of_ravens_own_rows_that_measured_no_menu_is_measured_again(
|
|
self, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Raven's own agents run on raven's provider catalogue, so an empty menu
|
|
there is a handshake from before that catalogue reached them -- and one
|
|
nothing invalidates, since the launch config it was measured against has
|
|
not moved. A third party that advertised none is taken at its word: it
|
|
would otherwise be relaunched at every boot to say so again."""
|
|
from dataclasses import dataclass
|
|
|
|
from raven.agent.subagent.probe import schedule_snapshot_verification
|
|
|
|
@dataclass
|
|
class Snap:
|
|
agent_name: str = "raven"
|
|
status: str = "ready"
|
|
stale: bool = False
|
|
needs_auth: bool = False
|
|
model_menu_measured: bool = True
|
|
model_choices: tuple = ()
|
|
|
|
_host_with_a_provider_key()
|
|
own, third_party = _FakeRow("Raven-PPT"), _FakeRow("Codex")
|
|
snapshots = {id(own.config): Snap(), id(third_party.config): Snap(agent_name="codex")}
|
|
verified: list[str] = []
|
|
|
|
async def fake_verify(cfg: object) -> Snap:
|
|
verified.append(getattr(cfg, "name", "?"))
|
|
return Snap(model_choices=("v/a",))
|
|
|
|
monkeypatch.setattr(probe_mod, "_MENULESS_OWN_RE_MEASURED", set())
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: snapshots[id(cfg)])
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type(
|
|
"S",
|
|
(),
|
|
{"record": staticmethod(lambda s: None), "has_model_menu": staticmethod(lambda agent: True)},
|
|
)(),
|
|
)
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
await schedule_snapshot_verification(_FakeManager([own, third_party]))
|
|
assert verified == ["Raven-PPT"]
|
|
|
|
snapshots[id(own.config)] = Snap(model_choices=("v/a",))
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
await schedule_snapshot_verification(_FakeManager([own, third_party]))
|
|
assert verified == ["Raven-PPT"], "the menu is measured once; the record it wrote is then trusted"
|
|
|
|
async def test_a_row_told_again_it_has_no_menu_is_not_re_measured_a_third_time(
|
|
self, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""The one re-measure reason nothing invalidates has to spend itself.
|
|
|
|
An agent that answers "no models" twice is answering about itself, not
|
|
about a catalogue that had not arrived yet -- and since the record it
|
|
writes is the record that arms this reason, an unspent one would put a
|
|
child launch on every boot and on every connect after it, forever.
|
|
"""
|
|
from dataclasses import dataclass
|
|
|
|
from raven.agent.subagent.probe import capabilities_wanted, schedule_snapshot_verification
|
|
|
|
@dataclass
|
|
class Snap:
|
|
agent_name: str = "raven"
|
|
status: str = "ready"
|
|
stale: bool = False
|
|
needs_auth: bool = False
|
|
model_menu_measured: bool = True
|
|
model_choices: tuple = ()
|
|
|
|
_host_with_a_provider_key()
|
|
own = _FakeRow("Raven-PPT")
|
|
verified: list[str] = []
|
|
|
|
async def fake_verify(cfg: object) -> Snap:
|
|
verified.append(getattr(cfg, "name", "?"))
|
|
return Snap()
|
|
|
|
monkeypatch.setattr(probe_mod, "_MENULESS_OWN_RE_MEASURED", set())
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: Snap())
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type(
|
|
"S",
|
|
(),
|
|
{"record": staticmethod(lambda s: None), "has_model_menu": staticmethod(lambda agent: True)},
|
|
)(),
|
|
)
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
await schedule_snapshot_verification(_FakeManager([own]))
|
|
assert verified == ["Raven-PPT"]
|
|
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
await schedule_snapshot_verification(_FakeManager([own]))
|
|
assert verified == ["Raven-PPT"], "told once that this row really has none, and it is not asked again"
|
|
assert capabilities_wanted(own.config) is False, "nor by the connect, which reads the same rule"
|
|
|
|
async def test_a_host_with_no_provider_of_its_own_re_measures_nothing(
|
|
self, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Nothing for the child to inherit, so nothing for a handshake to find:
|
|
the fallback's premise is a catalogue this raven can pass on."""
|
|
from dataclasses import dataclass
|
|
|
|
from raven.agent.subagent.probe import capabilities_wanted, schedule_snapshot_verification
|
|
|
|
@dataclass
|
|
class Snap:
|
|
agent_name: str = "raven"
|
|
status: str = "ready"
|
|
stale: bool = False
|
|
needs_auth: bool = False
|
|
model_menu_measured: bool = True
|
|
model_choices: tuple = ()
|
|
|
|
own = _FakeRow("Raven-PPT")
|
|
verified: list[str] = []
|
|
|
|
async def fake_verify(cfg: object) -> Snap:
|
|
verified.append(getattr(cfg, "name", "?"))
|
|
return Snap()
|
|
|
|
monkeypatch.setattr(probe_mod, "_MENULESS_OWN_RE_MEASURED", set())
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: Snap())
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type(
|
|
"S",
|
|
(),
|
|
{"record": staticmethod(lambda s: None), "has_model_menu": staticmethod(lambda agent: True)},
|
|
)(),
|
|
)
|
|
monkeypatch.setattr(probe_mod, "_unconfigured_acp_preset_rows", lambda configured, path=None: [])
|
|
monkeypatch.setattr(probe_mod, "_SCHEDULED", False)
|
|
|
|
await schedule_snapshot_verification(_FakeManager([own]))
|
|
assert verified == []
|
|
assert capabilities_wanted(own.config) is False
|
|
|
|
|
|
async def test_capabilities_wanted_re_measures_one_of_ravens_own_rows_with_an_empty_menu(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""The connect's own reading of the backfill's rule, off real stored rows.
|
|
|
|
The two must answer alike: a connect that skipped this row would leave it
|
|
menuless until the next restart, and one that re-measured every third party
|
|
with no menu would spend a handshake per connect to be told the same thing.
|
|
"""
|
|
from raven.acp_client.capabilities import AcpModelChoice, CapabilitySnapshot, SnapshotStore, snapshot_fingerprint
|
|
from raven.agent.subagent.probe import capabilities_wanted
|
|
|
|
store_path = tmp_path / "caps.json"
|
|
monkeypatch.setattr("raven.acp_client.capabilities.default_snapshot_path", lambda: store_path)
|
|
|
|
def recorded(
|
|
name: str, *, agent_name: str, menu: tuple[str, ...] = (), status: str = "ready"
|
|
) -> ThirdPartyAcpSubagentConfig:
|
|
cfg = ThirdPartyAcpSubagentConfig(name=name, command=f"{name} acp")
|
|
SnapshotStore(path=store_path).record(
|
|
CapabilitySnapshot(
|
|
agent=name,
|
|
fingerprint=snapshot_fingerprint(cfg),
|
|
status=status,
|
|
detail="",
|
|
measured_at_ms=1,
|
|
agent_name=agent_name,
|
|
model_choices=tuple(AcpModelChoice(value=v, name=v, group="V") for v in menu),
|
|
)
|
|
)
|
|
return cfg
|
|
|
|
assert capabilities_wanted(recorded("own-empty", agent_name="raven")) is False, "no provider of raven's own yet"
|
|
_host_with_a_provider_key()
|
|
monkeypatch.setattr(probe_mod, "_MENULESS_OWN_RE_MEASURED", set())
|
|
assert capabilities_wanted(recorded("own-empty", agent_name="raven")) is True
|
|
assert capabilities_wanted(recorded("own-menu", agent_name="raven", menu=("v/a",))) is False
|
|
assert capabilities_wanted(recorded("third-party-empty", agent_name="codex")) is False
|
|
# Ready, or the empty menu says nothing about the catalogue: an agent that
|
|
# did not come up is the missing-snapshot case, not this one.
|
|
assert capabilities_wanted(recorded("own-broken", agent_name="raven", status="attention")) is False
|
|
|
|
|
|
def test_a_preset_the_table_cannot_be_read_from_offers_nothing(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Three ways the shipped table can fail to answer, and none may reach the boot.
|
|
|
|
This runs at startup, before anything a user did: a packaging fault here is
|
|
not their typo to see, and taking the gateway down over it would trade a
|
|
stale row for no gateway. Each branch answers with the rows it could build,
|
|
which for these inputs is none.
|
|
"""
|
|
# A command no shell could split. `shlex` raises rather than returning, and
|
|
# the entry is simply not a candidate.
|
|
monkeypatch.setattr(
|
|
probe_mod,
|
|
"third_party_subagent_presets",
|
|
lambda: [{"name": "Torn", "preset": "claude_code", "kind": "acp", "command": 'x "unclosed'}],
|
|
)
|
|
assert probe_mod._unconfigured_acp_preset_rows(set(), path=None) == []
|
|
|
|
# Nothing on this machine resolves, so there is nothing to hand back and the
|
|
# schema is never asked.
|
|
monkeypatch.setattr(
|
|
probe_mod,
|
|
"third_party_subagent_presets",
|
|
lambda: [{"name": "Gone", "preset": "codex", "kind": "acp", "command": "absent-agent acp"}],
|
|
)
|
|
monkeypatch.setattr(probe_mod.shutil, "which", lambda exe, path=None: None)
|
|
assert probe_mod._unconfigured_acp_preset_rows(set(), path=None) == []
|
|
|
|
# And a table the schema refuses: logged, and the boot goes on without it.
|
|
monkeypatch.setattr(probe_mod.shutil, "which", lambda exe, path=None: "/bin/x")
|
|
monkeypatch.setattr(
|
|
probe_mod,
|
|
"third_party_subagent_presets",
|
|
lambda: [{"name": "Bad", "preset": "codex", "kind": "acp", "command": "x acp", "readyTimeoutMs": "soon"}],
|
|
)
|
|
assert probe_mod._unconfigured_acp_preset_rows(set(), path=None) == []
|
|
|
|
|
|
def test_only_resolvable_unconfigured_acp_presets_are_offered_for_verification(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""Three filters, each for its own reason.
|
|
|
|
Already configured: the registry half of the backfill has it. Not acp: there
|
|
is no handshake to record. Not on PATH: launching it is how you learn it is
|
|
not there, and the free probe answered that for nothing.
|
|
"""
|
|
from raven.config.schema import ThirdPartyAcpSubagentConfig
|
|
|
|
# Real preset keys: the schema checks `preset` against the shipped table, so
|
|
# a made-up one would fail validation rather than the filter under test.
|
|
presets = [
|
|
{"name": "Here", "preset": "claude_code", "kind": "acp", "command": "present-agent acp"},
|
|
{"name": "Gone", "preset": "codex", "kind": "acp", "command": "absent-agent acp"},
|
|
{"name": "Mine", "preset": "opencode", "kind": "acp", "command": "present-agent acp"},
|
|
{"name": "Http", "preset": "mirothinker", "kind": "openai", "baseUrl": "https://x/v1", "model": "m"},
|
|
]
|
|
seen_paths: list[str | None] = []
|
|
|
|
def fake_which(exe: str, path: str | None = None) -> str | None:
|
|
seen_paths.append(path)
|
|
return "/bin/x" if exe == "present-agent" else None
|
|
|
|
monkeypatch.setattr(probe_mod, "third_party_subagent_presets", lambda: presets)
|
|
monkeypatch.setattr(probe_mod.shutil, "which", fake_which)
|
|
|
|
rows = probe_mod._unconfigured_acp_preset_rows({"Mine"}, path="/login/shell/bin")
|
|
|
|
assert [row.name for row in rows] == ["Here"]
|
|
assert isinstance(rows[0].config, ThirdPartyAcpSubagentConfig)
|
|
# Resolved against the caller's PATH, not this process's. `_probe_acp` answers
|
|
# the row on screen from the login shell's, so reading a different one here
|
|
# would skip an agent the page is reporting as installed.
|
|
assert set(seen_paths) == {"/login/shell/bin"}
|
|
|
|
|
|
def test_a_shim_preset_whose_agent_is_absent_is_not_offered_for_verification(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""A shim's command resolves on any machine; the agent it drives may not.
|
|
|
|
``_probe_acp`` reads such a row as missing, so handshaking it at boot would
|
|
pay to learn what the free probe already said. Sharpest for a Mirasim row,
|
|
whose command is Raven's own interpreter.
|
|
"""
|
|
presets = [
|
|
{
|
|
"name": "Claude Code-Mirasim",
|
|
"preset": "mirasim_claude",
|
|
"kind": "acp",
|
|
"command": "/venv/bin/python -P -m raven.acp.mirasim --agent claude",
|
|
},
|
|
{"name": "Pi", "preset": "pi", "kind": "acp", "command": "npx -y pi-acp@0.0.33"},
|
|
]
|
|
present = {"/venv/bin/python", "npx", "pi"}
|
|
monkeypatch.setattr(probe_mod, "third_party_subagent_presets", lambda: presets)
|
|
monkeypatch.setattr(probe_mod.shutil, "which", lambda exe, path=None: f"/bin/{exe}" if exe in present else None)
|
|
|
|
assert [row.name for row in probe_mod._unconfigured_acp_preset_rows(set(), path=None)] == ["Pi"]
|
|
present.add("mirasim")
|
|
rows = probe_mod._unconfigured_acp_preset_rows(set(), path=None)
|
|
assert [row.name for row in rows] == ["Claude Code-Mirasim", "Pi"]
|
|
|
|
|
|
async def test_a_hand_written_npx_row_is_not_held_to_a_shims_requirement(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""The requirement rides on provenance: a row that merely wears the preset's name runs its own command."""
|
|
_fake_executable(tmp_path, "npx")
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
|
|
cfg = ThirdPartyAcpSubagentConfig(name="Pi", command="npx -y my-own-acp-shim@1.0.0")
|
|
|
|
res = await probe_one(cfg, source="config", path=str(tmp_path))
|
|
|
|
assert res.status == "attention"
|
|
|
|
|
|
async def test_a_shim_that_ships_its_own_agent_asks_after_nothing_else(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""``codex-acp`` bundles the agent as its own binary; what it wants is a login, not an install."""
|
|
_fake_executable(tmp_path, "npx")
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
|
|
cfg = ThirdPartyAcpSubagentConfig(
|
|
name="Codex", preset="codex", command="npx -y @agentclientprotocol/codex-acp@1.1.14"
|
|
)
|
|
|
|
res = await probe_one(cfg, source="preset", path=str(tmp_path))
|
|
|
|
assert res.status == "attention"
|
|
|
|
|
|
async def test_a_shim_row_whose_agent_is_installed_goes_on_to_the_snapshot(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
npx = _fake_executable(tmp_path, "npx")
|
|
_fake_executable(tmp_path, "pi")
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
|
|
cfg = ThirdPartyAcpSubagentConfig(name="Pi", preset="pi", command="npx -y pi-acp@0.0.33")
|
|
|
|
res = await probe_one(cfg, source="preset", path=str(tmp_path))
|
|
|
|
assert res.status == "attention"
|
|
assert res.target == str(npx)
|
|
|
|
|
|
async def test_a_shim_row_is_missing_when_the_agent_it_drives_is_absent(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""``npx`` resolving says nothing about whether ``pi`` is installed.
|
|
|
|
The shim is fetched on connect and fails a minute later with "executable not
|
|
found" when the agent it drives is absent. That is the sentence the probe can
|
|
say up front, with the install beside it, so the row reads absent rather
|
|
than connectable -- the same reading a local-executable row gets.
|
|
"""
|
|
_fake_executable(tmp_path, "npx")
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: None)
|
|
cfg = ThirdPartyAcpSubagentConfig(name="Pi", preset="pi", command="npx -y pi-acp@0.0.33")
|
|
|
|
res = await probe_one(cfg, source="preset", path=str(tmp_path))
|
|
|
|
assert res.status == "missing"
|
|
assert res.target == "pi"
|
|
assert res.absent == "pi", "npx is there; the agent it drives is what to install"
|
|
assert res.detail == (
|
|
"pi is not on the login shell PATH; install with npm install -g @earendil-works/pi-coding-agent"
|
|
)
|
|
|
|
|
|
def _fake_executable(tmp_path: Path, name: str) -> Path:
|
|
exe = tmp_path / name
|
|
exe.write_text("#!/bin/sh\nexit 0\n")
|
|
exe.chmod(0o755)
|
|
return exe
|
|
|
|
|
|
def test_a_machine_failure_says_where_the_fix_is_made_unless_the_fix_is_the_node_installed_from_here():
|
|
from raven.agent.subagent import probe
|
|
|
|
there = probe._on_machine("'claude' is not on the login shell PATH", "ravenx2")
|
|
assert there.endswith("(on machine 'ravenx2': any fix is made there)")
|
|
|
|
here = probe._on_machine(
|
|
"Raven node 0.2.4-abc is not installed on machine 'RavenX 2' (ravenx2); install it with "
|
|
"`raven ops connection install-node ravenx2`",
|
|
"ravenx2",
|
|
)
|
|
assert "any fix is made there" not in here and here.endswith("install-node ravenx2`")
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"status,detail",
|
|
[
|
|
("unknown", "unsupported session configuration"),
|
|
("attention", "session/new timed out"),
|
|
("missing", "transport disconnected"),
|
|
],
|
|
)
|
|
async def test_boot_replaces_an_auth_verdict_and_retries_until_ready(tmp_path, monkeypatch, status, detail):
|
|
from dataclasses import replace
|
|
from unittest.mock import AsyncMock
|
|
|
|
import raven.acp_client.capabilities as caps
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(name="measured", command="measured --acp")
|
|
monkeypatch.setattr(caps, "default_snapshot_path", lambda: tmp_path / "caps.json")
|
|
monkeypatch.setattr(probe_mod, "_launch_kw", lambda cfg: {})
|
|
old = caps.CapabilitySnapshot(
|
|
agent=cfg.name,
|
|
fingerprint=caps.snapshot_fingerprint(cfg),
|
|
status="attention",
|
|
detail="sign in",
|
|
measured_at_ms=1,
|
|
needs_auth=True,
|
|
can_load=True,
|
|
model_choices=(caps.AcpModelChoice(value="a", name="A"),),
|
|
)
|
|
store = caps.SnapshotStore()
|
|
store.record(old)
|
|
failed = caps.CapabilitySnapshot(
|
|
agent=cfg.name,
|
|
fingerprint=old.fingerprint,
|
|
status=status,
|
|
detail=detail,
|
|
measured_at_ms=2,
|
|
needs_auth=False,
|
|
)
|
|
verified = AsyncMock(side_effect=[failed, replace(old, status="ready", needs_auth=False, measured_at_ms=3)])
|
|
monkeypatch.setattr(caps, "verify_agent", verified)
|
|
rows = [_FakeRow(cfg.name, config=cfg)]
|
|
manager = _FakeManager(rows, with_refresh=True)
|
|
await probe_mod._verify_missing_snapshots(manager, rows)
|
|
saved = store.load([cfg])[cfg.name]
|
|
assert not saved.needs_auth
|
|
assert not saved.usable
|
|
assert (saved.status, saved.detail) == (status, detail)
|
|
assert saved.can_load and saved.model_choices == old.model_choices
|
|
await probe_mod._verify_missing_snapshots(manager, rows)
|
|
assert store.load([cfg])[cfg.name].usable
|
|
await probe_mod._verify_missing_snapshots(manager, rows)
|
|
assert verified.await_count == 2
|
|
assert manager.refresh_calls == 2
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"tested_status,late",
|
|
[
|
|
("ready", {"status": "attention", "detail": "session/new timed out"}),
|
|
("ready", {"status": "attention", "detail": "Authentication required", "needs_auth": True}),
|
|
("attention", {"status": "ready", "detail": "connected"}),
|
|
],
|
|
ids=[
|
|
"a-late-timeout-loses-to-a-test-pass",
|
|
"a-late-refusal-loses-to-a-test-pass",
|
|
"a-late-pass-loses-to-a-test-failure",
|
|
],
|
|
)
|
|
async def test_a_record_written_while_the_boot_verify_ran_outranks_its_late_verdict(
|
|
tmp_path, monkeypatch, tested_status, late
|
|
):
|
|
"""The newest evidence wins: a Test recorded while the boot's handshake was in
|
|
flight stands, whatever that handshake returns afterwards.
|
|
|
|
The boot reads the row before it launches anything, and the handshake can
|
|
then hold that read for the whole ready timeout. A write built from it took
|
|
the Test's verdict and the menu the Test measured both -- a late timeout
|
|
included, which the boot records over any row whose last record failed.
|
|
"""
|
|
from dataclasses import replace
|
|
|
|
import raven.acp_client.capabilities as caps
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(name="measured", command="measured --acp")
|
|
monkeypatch.setattr(caps, "default_snapshot_path", lambda: tmp_path / "caps.json")
|
|
monkeypatch.setattr(probe_mod, "_launch_kw", lambda cfg: {})
|
|
old = caps.CapabilitySnapshot(
|
|
agent=cfg.name,
|
|
fingerprint=caps.snapshot_fingerprint(cfg),
|
|
status="attention",
|
|
detail="sign in",
|
|
measured_at_ms=1,
|
|
needs_auth=True,
|
|
can_load=True,
|
|
model_choices=(caps.AcpModelChoice(value="a", name="A"),),
|
|
)
|
|
store = caps.SnapshotStore()
|
|
store.record(old)
|
|
if tested_status == "ready":
|
|
tested = replace(
|
|
old,
|
|
status="ready",
|
|
detail="connected",
|
|
needs_auth=False,
|
|
measured_at_ms=3,
|
|
model_choices=(caps.AcpModelChoice(value="b", name="B"),),
|
|
)
|
|
else:
|
|
tested = replace(old, status="attention", detail="session/new timed out", needs_auth=False, measured_at_ms=3)
|
|
late_result = replace(
|
|
old, **{"needs_auth": False, "measured_at_ms": 4, "model_choices": (caps.AcpModelChoice(value="c"),), **late}
|
|
)
|
|
calls: list[str] = []
|
|
boot_parked, released = asyncio.Event(), asyncio.Event()
|
|
|
|
async def fake_verify(cfg, **kwargs):
|
|
if not calls:
|
|
calls.append("boot")
|
|
boot_parked.set()
|
|
await released.wait()
|
|
return late_result
|
|
calls.append("test")
|
|
return tested
|
|
|
|
monkeypatch.setattr(caps, "verify_agent", fake_verify)
|
|
rows = [_FakeRow(cfg.name, config=cfg)]
|
|
boot = asyncio.ensure_future(probe_mod._verify_missing_snapshots(_FakeManager(rows, with_refresh=True), rows))
|
|
await asyncio.wait_for(boot_parked.wait(), 2)
|
|
await probe_mod.record_capabilities(cfg)
|
|
after_test = store.load([cfg])[cfg.name]
|
|
assert not boot.done(), "the Test has to land while the boot's handshake is still open"
|
|
assert after_test == tested, "and record its own measurement there"
|
|
released.set()
|
|
await asyncio.wait_for(boot, 2)
|
|
|
|
assert calls == ["boot", "test"]
|
|
assert store.load([cfg])[cfg.name] == after_test, "the boot's late verdict must not replace the Test's"
|
|
|
|
|
|
@pytest.mark.parametrize("late_status", ["ready", "attention"])
|
|
async def test_a_modes_relearn_during_the_boot_verify_does_not_stop_its_write(tmp_path, monkeypatch, late_status):
|
|
"""A dispatch that relearns the agent's modes mid-handshake measures nothing.
|
|
|
|
It keeps the record's measurement stamp, so the boot's verdict still lands:
|
|
otherwise a failed row that a dispatch merely touched would stay failed until
|
|
the next boot. A late failure is written over the re-read record, so the
|
|
relearned modes survive it; a late ``ready`` is a whole measurement, its own
|
|
menu included, and is written as measured.
|
|
"""
|
|
from dataclasses import replace
|
|
|
|
import raven.acp_client.capabilities as caps
|
|
|
|
cfg = ThirdPartyAcpSubagentConfig(name="measured", command="measured --acp")
|
|
monkeypatch.setattr(caps, "default_snapshot_path", lambda: tmp_path / "caps.json")
|
|
monkeypatch.setattr(probe_mod, "_launch_kw", lambda cfg: {})
|
|
old = caps.CapabilitySnapshot(
|
|
agent=cfg.name,
|
|
fingerprint=caps.snapshot_fingerprint(cfg),
|
|
status="attention",
|
|
detail="session/new timed out",
|
|
measured_at_ms=1,
|
|
can_load=True,
|
|
available_modes=(caps.AcpMode(id="old", name="Old"),),
|
|
)
|
|
store = caps.SnapshotStore()
|
|
store.record(old)
|
|
late_result = replace(
|
|
old,
|
|
status=late_status,
|
|
detail="connected" if late_status == "ready" else "session/new timed out again",
|
|
measured_at_ms=4,
|
|
available_modes=(caps.AcpMode(id="measured", name="Measured"),),
|
|
)
|
|
boot_parked, released = asyncio.Event(), asyncio.Event()
|
|
|
|
async def fake_verify(cfg, **kwargs):
|
|
boot_parked.set()
|
|
await released.wait()
|
|
return late_result
|
|
|
|
monkeypatch.setattr(caps, "verify_agent", fake_verify)
|
|
rows = [_FakeRow(cfg.name, config=cfg)]
|
|
manager = _FakeManager(rows, with_refresh=True)
|
|
boot = asyncio.ensure_future(probe_mod._verify_missing_snapshots(manager, rows))
|
|
await asyncio.wait_for(boot_parked.wait(), 2)
|
|
relearned = caps.relearn_session_modes(
|
|
store.load([cfg])[cfg.name],
|
|
{"sessionId": "s1", "modes": {"availableModes": [{"id": "fast", "name": "Fast"}], "currentModeId": "fast"}},
|
|
store=store,
|
|
)
|
|
assert relearned is not None and relearned.measured_at_ms == 1, "a relearn keeps the stamp it found"
|
|
assert not boot.done(), "the relearn has to land while the boot's handshake is still open"
|
|
released.set()
|
|
await asyncio.wait_for(boot, 2)
|
|
|
|
stored = store.load([cfg])[cfg.name]
|
|
assert (stored.status, stored.measured_at_ms) == (late_status, 4), "the boot's verdict must still land"
|
|
expected_modes = ("measured",) if late_status == "ready" else ("fast",)
|
|
assert tuple(m.id for m in stored.available_modes) == expected_modes
|
|
assert manager.refresh_calls == 1
|
|
|
|
|
|
def _menu_records() -> tuple[object, object]:
|
|
"""A record with a measured menu, and a later answer that was ready but could not read one."""
|
|
from raven.acp_client.capabilities import AcpMode, AcpModelChoice, CapabilitySnapshot
|
|
|
|
measured = CapabilitySnapshot(
|
|
agent="Claude Code-Mirasim",
|
|
fingerprint="old",
|
|
status="ready",
|
|
detail="",
|
|
measured_at_ms=1,
|
|
model_choices=(AcpModelChoice("own:sonnet", "Sonnet", "Claude Code"),),
|
|
available_modes=(AcpMode("high", "High"),),
|
|
stale=True,
|
|
)
|
|
unread = CapabilitySnapshot(
|
|
agent="Claude Code-Mirasim",
|
|
fingerprint="new",
|
|
status="ready",
|
|
detail="ready",
|
|
measured_at_ms=2,
|
|
model_menu_measured=False,
|
|
)
|
|
return measured, unread
|
|
|
|
|
|
def test_a_ready_answer_whose_menu_went_unread_keeps_the_menu_it_had() -> None:
|
|
"""A connect or a Test the Mirasim host leaves unanswered is menuless: a read that failed, not a measured none."""
|
|
measured, unread = _menu_records()
|
|
saved = probe_mod._test_record(unread, measured)
|
|
assert (saved.model_choices, saved.available_modes) == (measured.model_choices, measured.available_modes)
|
|
assert saved.model_menu_measured
|
|
assert (saved.status, saved.fingerprint, saved.measured_at_ms) == ("ready", "new", 2)
|
|
|
|
|
|
async def test_the_boot_keeps_a_measured_menu_over_an_answer_that_could_not_read_it(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
measured, unread = _menu_records()
|
|
recorded: list[object] = []
|
|
|
|
async def fake_verify(cfg: object) -> object:
|
|
return unread
|
|
|
|
monkeypatch.setattr(probe_mod, "acp_snapshot_for", lambda cfg: measured)
|
|
monkeypatch.setattr("raven.acp_client.capabilities.verify_agent", fake_verify)
|
|
monkeypatch.setattr(
|
|
"raven.acp_client.capabilities.SnapshotStore",
|
|
lambda: type("S", (), {"record": staticmethod(recorded.append)})(),
|
|
)
|
|
row = _FakeRow("Claude Code-Mirasim")
|
|
await probe_mod._verify_missing_snapshots(_FakeManager([row]), [row])
|
|
assert [snapshot.model_choices for snapshot in recorded] == [measured.model_choices]
|
|
|
|
|
|
@pytest.mark.parametrize("needs_auth,unfetched", [(True, False), (False, True), (False, False)])
|
|
@pytest.mark.parametrize("previous_status", ["ready", "attention"])
|
|
def test_manual_test_merges_current_failure_flags_with_old_capabilities(needs_auth, unfetched, previous_status):
|
|
from dataclasses import replace
|
|
|
|
from raven.acp_client.capabilities import CapabilitySnapshot
|
|
|
|
old = CapabilitySnapshot(
|
|
agent="agent",
|
|
fingerprint="old",
|
|
status=previous_status,
|
|
detail="old refusal",
|
|
measured_at_ms=1,
|
|
needs_auth=previous_status == "attention",
|
|
can_load=True,
|
|
)
|
|
measured = replace(
|
|
old,
|
|
status="attention",
|
|
fingerprint="new",
|
|
measured_at_ms=2,
|
|
detail="current refusal",
|
|
can_load=False,
|
|
needs_auth=needs_auth,
|
|
unfetched=unfetched,
|
|
)
|
|
saved = probe_mod._test_record(measured, old)
|
|
assert saved.can_load
|
|
assert saved.fingerprint == "new"
|
|
assert (saved.needs_auth, saved.unfetched) == (needs_auth, unfetched)
|