## Summary Bundled ACP agents could fail to start on Windows when rendering configuration, reading stdin, or replacing the launcher process. Closing an ACP connection could also fail before releasing its process and transport because Windows has no os.killpg. - Keep all five bundled Windows launchers alive until their ACP child exits, forward the exit status, and apply the same behavior to new scaffolded launchers. POSIX retains exec. - Preserve live rendered configurations with the cross-platform PID probe, retain POSIX file permissions, and preserve seed-file line endings. - Read Windows stdin pipes through a daemon feeder with PeekNamedPipe and reads bounded by the available bytes. - End the live launcher's Windows process tree with bounded taskkill before releasing the transport. Leave an already-exited launcher's PID alone. - Preserve a completed session/new reply when EOF shutdown cancels its optional model-menu lookup. The first LiteLLM import can exceed the five-second grace even after #909 removed the metadata-network delay. Active-connection cancellation and cancellation during required session setup still propagate. - Run real stdio handshakes, all five launchers, command registration, and native teardown cases in Windows CI. The handshake now also runs with a fresh bytecode cache. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Native Windows build 10.0.26300, Python 3.13.12, PYTHONUTF8=1. The final tree is based on the fetched main revision; all eight patches replayed unchanged. The last upstream update adds bug-report upload/receiver code and has no file overlap with this PR. The 19-case selection below was rerun on the final tree; the broader selection and controls predate that last unrelated update. Tests used isolated user/Raven homes, frozen dependencies, and verified imports from the tested checkout. Run-owned --basetemp and --junitxml output paths are omitted from the selections below. - `uv run --frozen --no-sync pytest tests/test_acp_server.py::TestFrameLoop::test_session_creation_answers_when_eof_cancels_the_catalogue tests/test_acp_methods.py::TestConfigOptions::test_session_creation_preserves_cancellation_outside_shutdown_metadata tests/integration/test_acp_stdio_smoke.py tests/test_cli_agents_commands.py::test_register_pins_the_resolved_row_into_the_host_roster tests/test_subagent_acp.py::test_close_without_killpg_ends_the_tree_through_taskkill tests/test_subagent_acp.py::test_close_without_killpg_kills_nothing_by_the_pid_of_an_exited_launcher -m 'integration or not integration' -n 0 --tb=short`: **19 passed, no skips** after the final rebase. This includes both EOF timings, cancellation controls, both bytecode-cache states, all five real launchers, and real taskkill. One non-failing idle-ceiling warning was reported. - `uv run --frozen --no-sync pytest tests/test_acp_server.py tests/test_acp_methods.py tests/test_acp_updates.py tests/test_acp_config_options.py tests/test_cli_acp_commands.py tests/test_config_product_render.py tests/test_utils_pid.py tests/test_agents_code_launcher.py tests/test_agents_design_launcher.py tests/test_agents_oncall_launcher.py tests/test_agents_ppt_launcher.py tests/test_agents_research_launcher.py tests/test_cli_agents_commands.py tests/test_subagent_acp.py tests/test_rpc_bootstrap.py tests/test_subagent_host_env.py tests/test_subagent_probe.py -n 0 --tb=short`: **1284 passed, 58 failed, 10 skipped, 4 errors**. The same selection on the rebased PR immediately before the EOF fix returned **1280 passed, the identical 58 failing and 4 error test IDs, and 10 skipped**. No new failure/error IDs from the EOF fix. - All 62 failure/error IDs were also selected on main: 56 failed, 4 errored, and 2 passed. Those two passed in isolation on this PR too, and failed in the complete selection both before and after the EOF fix. The remaining failures include Windows path, shell/command, and symlink-privilege cases. The attempted complete main selection stalled in the pre-existing Windows stdin tests and was terminated; no complete main-suite result is claimed. - Before the EOF fix, `uv run --frozen --no-sync pytest tests/integration/test_acp_stdio_smoke.py::test_a_whole_handshake_runs_in_one_process[cold_imports] -m integration -n 0 --tb=short` failed with response IDs [1, 3]. A separate stack-capture probe measured 6.76 seconds in the first LiteLLM import; the handler awaited model.options while its worker imported LiteLLM and its dependencies. - `uv run --frozen --no-sync pytest tests/test_acp_server.py::TestFrameLoop::test_session_creation_answers_when_eof_cancels_the_catalogue tests/test_acp_methods.py::TestConfigOptions::test_session_creation_preserves_cancellation_outside_shutdown_metadata -n 0 --tb=short`: before the fix, 2 failed and 2 passed. Both EOF cases failed with the missing session reply. They pass with the fix and are included in the broader post-rebase selection above. Warm/cold real handshakes plus these four cases also passed as a six-case selection. - A mutant that treats every connection as closing makes the ordinary-cancellation control fail. Cancellation during required session setup still propagates. - `uv run --frozen --no-sync ruff check raven evolver agents plugins-dist tests scripts docs-site` and the corresponding `ruff format --check`: passed; 2266 files already formatted. - `uv run --frozen --no-sync lint-imports`: 10 contracts kept. `uv run --frozen --no-sync ty check raven/acp/methods.py raven/acp/updates.py raven/acp_client/client.py raven/cli/acp_commands.py raven/config/product_render.py`: passed. - `uv run --frozen --no-sync pre-commit run --from-ref origin/main --to-ref HEAD`: every applicable hook passed. The two final docstring edits also passed their file-scoped pre-commit run. - `node node_modules/@commitlint/cli/cli.js --from origin/main --to HEAD --config commitlint.config.cjs`, `uv run --frozen --no-sync python scripts/check_commit_messages.py origin/main..HEAD`, `uv run --frozen --no-sync python scripts/check_large_files.py origin/main...HEAD`, `uv run --frozen --no-sync python scripts/check_source_language.py origin/main...HEAD`, and `git diff --check origin/main...HEAD`: passed. - The PR body passed the repository body checker and an explicit ASCII scan. The broader Windows selection remains red, so the relevant-tests checkbox is unchecked. This revision has no new Linux or full repository run, live LLM workload, rendering workload, or ACL audit. POSIX-only skips are not counted as Windows passes. - [ ] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes At EOF, a completed session/new can omit optional configOptions when shutdown cancels a slow lookup; the sessionId and session modes are retained. Active connections keep their existing model-menu behavior. The handler grace remains five seconds. Windows keeps the launcher present while its server runs, and connection close waits up to five seconds for taskkill. Idle Windows pipe input polls every 10 ms. Redirected files and text streams retain their existing reader path. POSIX retains exec and 0600 permissions; Windows rendered files inherit directory ACLs. There is no configuration-format change. Previously generated agent folders retain their existing launchers; the scaffold change applies to new folders. Revert this PR to roll back the launcher, stdin, process-cleanup and EOF-response changes. ## Related Issues #909
1199 lines
49 KiB
Python
1199 lines
49 KiB
Python
"""Tests for ``raven agents new`` -- the scaffold command and its refusals.
|
|
|
|
The command instantiates ``raven/templates/agents_scaffold/`` into a fresh
|
|
agent folder. Pinned here: the dry-run tree, the three conflict refusals
|
|
(existing directory, roster row, reserved name), the four name derivations,
|
|
the instantiation itself (schema-valid manifest, compilable python, zero
|
|
token residue), and the two landing spots.
|
|
|
|
Every test isolates the raven home through ``RAVEN_HOME``: ``agents_root()``
|
|
takes the first existing tree, so a real home would shadow everything a test
|
|
builds (the S1 spike measured exactly that).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import py_compile
|
|
import re
|
|
import sys
|
|
import tomllib
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
from typer.testing import CliRunner
|
|
|
|
from raven.cli.commands import app
|
|
|
|
runner = CliRunner()
|
|
|
|
EXPECTED_TREE = {
|
|
".env.example",
|
|
"README.md",
|
|
"config.json",
|
|
"install.py",
|
|
"plugins/demo-agent-flow/demo_agent_flow/__init__.py",
|
|
"plugins/demo-agent-flow/demo_agent_flow/plugin.py",
|
|
"plugins/demo-agent-flow/demo_agent_flow/tools/__init__.py",
|
|
"plugins/demo-agent-flow/demo_agent_flow/tools/hello.py",
|
|
"plugins/demo-agent-flow/raven-plugin.toml",
|
|
"run.py",
|
|
"subagent.json",
|
|
}
|
|
|
|
TOKENS = ("my-agent", "My-Agent", "my_agent", "MY_AGENT")
|
|
|
|
|
|
@pytest.fixture
|
|
def raven_home(tmp_path: Path, monkeypatch) -> Path:
|
|
home = tmp_path / "home"
|
|
home.mkdir()
|
|
monkeypatch.setenv("RAVEN_HOME", str(home))
|
|
return home
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# dry run
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_dry_run_prints_the_instantiated_tree_and_writes_nothing(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--dry-run"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
for rel in EXPECTED_TREE:
|
|
assert rel in r.output
|
|
assert "dry run: nothing written" in r.output
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# the three conflict refusals
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_an_existing_target_directory_is_refused(raven_home: Path) -> None:
|
|
(raven_home / "agents" / "demo-agent").mkdir(parents=True)
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "already exists" in r.output
|
|
assert not (raven_home / "agents" / "demo-agent" / "subagent.json").exists()
|
|
|
|
|
|
def test_a_roster_row_with_the_same_name_is_refused(raven_home: Path) -> None:
|
|
(raven_home / "config.json").write_text(
|
|
json.dumps({"subagents": {"agents": [{"name": "Demo-Agent", "kind": "acp", "command": "demo --acp"}]}}),
|
|
encoding="utf-8",
|
|
)
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "roster" in r.output
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
@pytest.mark.parametrize("name", ["raven-code", "raven-research-ng", "raven"])
|
|
def test_a_reserved_name_is_refused(raven_home: Path, name: str) -> None:
|
|
r = runner.invoke(app, ["agents", "new", name])
|
|
|
|
assert r.exit_code != 0
|
|
assert "choose another name" in r.output
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
def test_a_reserved_display_name_is_refused(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "fine-name", "--display", "Raven-Code"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "choose another name" in r.output
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# name derivations
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_api_key_var_normalizes_hyphens() -> None:
|
|
"""Pins the existing behavior the scaffold leans on (the blueprint's fix-2)."""
|
|
from raven.agent.subagent.vendored_agents import api_key_var
|
|
|
|
assert api_key_var("my-agent") == "MY_AGENT_API_KEY"
|
|
|
|
|
|
def test_the_four_identities_derive_from_one_kebab_name() -> None:
|
|
from raven.cli.agents_commands import _display_name, _env_prefix
|
|
|
|
assert _display_name("my-agent") == "My-Agent"
|
|
assert _display_name("a-b2-c") == "A-B2-C"
|
|
assert _env_prefix("my-agent") == "MY_AGENT"
|
|
|
|
|
|
@pytest.mark.parametrize("name", ["My-Agent", "my_agent", "-lead", "trail-", "double--dash", "0day", "one.dot"])
|
|
def test_a_name_that_is_not_kebab_case_is_refused(raven_home: Path, name: str) -> None:
|
|
r = runner.invoke(app, ["agents", "new", name])
|
|
|
|
assert r.exit_code != 0
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# instantiation
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_the_generated_folder_is_valid_compilable_and_token_free(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
target = raven_home / "agents" / "demo-agent"
|
|
generated = {p.relative_to(target).as_posix() for p in target.rglob("*") if p.is_file()}
|
|
assert generated == EXPECTED_TREE
|
|
|
|
from raven.config.schema import ThirdPartyAcpSubagentConfig
|
|
|
|
row = json.loads((target / "subagent.json").read_text(encoding="utf-8"))
|
|
validated = ThirdPartyAcpSubagentConfig.model_validate(row)
|
|
assert validated.name == "Demo-Agent"
|
|
assert validated.memory and validated.memory.model_dump(exclude_none=True)["userId"] == "demo-agent"
|
|
assert "{PYTHON}" in row["command"] and "{SUBAGENT_DIR}" in row["command"]
|
|
|
|
for rel in generated:
|
|
text = (target / rel).read_text(encoding="utf-8")
|
|
for token in TOKENS:
|
|
assert token not in rel, (rel, token)
|
|
assert token not in text, (rel, token)
|
|
|
|
for rel in generated:
|
|
if rel.endswith(".py"):
|
|
py_compile.compile(str(target / rel), doraise=True)
|
|
|
|
config = json.loads((target / "config.json").read_text(encoding="utf-8"))
|
|
assert config == {"plugins": {"config": {"demo-agent-flow": {"enabled": True}}}}
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# landing spots
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_the_default_landing_is_the_raven_home_agents_tree(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert (raven_home / "agents" / "demo-agent" / "run.py").is_file()
|
|
launcher = (raven_home / "agents" / "demo-agent" / "run.py").read_text(encoding="utf-8")
|
|
assert 'sys.platform == "win32"' in launcher
|
|
assert "subprocess.call(argv)" in launcher
|
|
|
|
|
|
def test_here_lands_in_the_working_directory_agents_tree(raven_home: Path, tmp_path: Path, monkeypatch) -> None:
|
|
checkout = tmp_path / "checkout"
|
|
checkout.mkdir()
|
|
monkeypatch.chdir(checkout)
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--here", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert (checkout / "agents" / "demo-agent" / "run.py").is_file()
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# options
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_kind_cli_is_reserved_for_v2(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--kind", "cli"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "v2" in r.output
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
def test_an_unknown_kind_is_refused(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--kind", "carrier-pigeon"])
|
|
|
|
assert r.exit_code != 0
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
def test_display_overrides_the_derived_display_name(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--display", "Demo Deluxe", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
row = json.loads((raven_home / "agents" / "demo-agent" / "subagent.json").read_text(encoding="utf-8"))
|
|
assert row["name"] == "Demo Deluxe"
|
|
assert row["memory"]["userId"] == "demo-agent"
|
|
|
|
|
|
def test_register_pins_the_resolved_row_into_the_host_roster(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--register", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
from raven.config.update_subagents import get_agents
|
|
from raven.utils.commands import command_argv
|
|
|
|
rows = {row["name"]: row for row in get_agents(config_path=raven_home / "config.json")}
|
|
row = rows["Demo-Agent"]
|
|
target = raven_home / "agents" / "demo-agent"
|
|
assert "{PYTHON}" not in row["command"] and "{SUBAGENT_DIR}" not in row["command"]
|
|
argv = command_argv(row["command"])
|
|
assert argv[0] == sys.executable
|
|
assert Path(argv[1]) == target / "run.py"
|
|
assert argv[2:] == ["--acp"]
|
|
assert row["cwd"] == str(target)
|
|
|
|
|
|
def test_no_smoke_prints_the_skip_line_and_spawns_nothing(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert "smoke: skipped (--no-smoke)" in r.output
|
|
assert "handshake" not in r.output
|
|
|
|
|
|
@pytest.mark.parametrize("display", ['Demo "Deluxe"', "Demo\\Agent", "Demo\nAgent", " Demo", "-Demo"])
|
|
def test_a_display_name_that_would_corrupt_the_manifest_is_refused(raven_home: Path, display: str) -> None:
|
|
"""The display is substituted verbatim into JSON, so the charset is fenced at the door."""
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--display", display])
|
|
|
|
assert r.exit_code != 0
|
|
assert "letters, digits, spaces and hyphens" in r.output
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# degraded worlds
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_a_malformed_host_config_reads_as_no_roster(raven_home: Path) -> None:
|
|
"""The scaffold writes no config, so a broken one must not block the folder."""
|
|
(raven_home / "config.json").write_text("{not json", encoding="utf-8")
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert (raven_home / "agents" / "demo-agent" / "subagent.json").is_file()
|
|
|
|
|
|
def test_an_empty_template_tree_is_refused(raven_home: Path, tmp_path: Path, monkeypatch) -> None:
|
|
empty = tmp_path / "no-templates"
|
|
empty.mkdir()
|
|
monkeypatch.setattr("raven.cli.agents_commands._templates_root", lambda: empty)
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "broken" in r.output
|
|
|
|
|
|
def test_cache_droppings_in_the_template_tree_are_not_scaffolded(raven_home: Path, tmp_path: Path, monkeypatch) -> None:
|
|
"""Only the stub lands (no .pyc, no .DS_Store); the doctor then rightly reds the stub tree."""
|
|
root = tmp_path / "templates"
|
|
(root / "__pycache__").mkdir(parents=True)
|
|
(root / "__pycache__" / "run.cpython-312.pyc").write_text("x", encoding="utf-8")
|
|
(root / ".DS_Store").write_text("x", encoding="utf-8")
|
|
(root / "run.py").write_text('"""Launcher stub for my-agent."""\n', encoding="utf-8")
|
|
monkeypatch.setattr("raven.cli.agents_commands._templates_root", lambda: root)
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
|
|
target = raven_home / "agents" / "demo-agent"
|
|
assert {p.relative_to(target).as_posix() for p in target.rglob("*") if p.is_file()} == {"run.py"}
|
|
assert r.exit_code != 0
|
|
assert "subagent.json" in r.output
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# the gold tests: new -> doctor -> discovery ready -> handshake
|
|
# ---------------------------------------------------------------------------
|
|
|
|
# The escape hatch for environments whose sandbox cannot spawn the launcher
|
|
# (the S1 spike proved an in-test spawn works on the dev boxes; a restricted
|
|
# runner sets the variable instead of deleting the test).
|
|
spawns_the_launcher = pytest.mark.skipif(
|
|
os.environ.get("RAVEN_TESTS_NO_SPAWN") == "1",
|
|
reason="RAVEN_TESTS_NO_SPAWN=1: this environment cannot spawn the generated launcher",
|
|
)
|
|
|
|
|
|
@spawns_the_launcher
|
|
def test_gold_new_is_discovered_ready_and_handshakes_green(raven_home: Path, monkeypatch) -> None:
|
|
"""The whole closed loop, no human in it: write, doctor, discovery, ACP initialize.
|
|
|
|
A dummy own-key is enough for the handshake (S1: initialize spends no
|
|
tokens and never touches the provider), so the test needs no credentials.
|
|
"""
|
|
monkeypatch.setenv("DEMO_AGENT_API_KEY", "dummy-key-for-the-smoke")
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert "subagent.json: valid (Demo-Agent, kind=acp)" in r.output
|
|
assert "raven-plugin.toml: parsed (demo-agent-flow: 1 tool(s), 1 hook(s))" in r.output
|
|
assert "py_compile: 6 file(s) OK" in r.output
|
|
assert "readiness=ready" in r.output
|
|
assert "handshake GREEN" in r.output and "protocolVersion=1" in r.output
|
|
|
|
from raven.agent.subagent.vendored_agents import discover_product_rows, product_state
|
|
|
|
root = raven_home / "agents"
|
|
assert any(row.name == "Demo-Agent" and row.enabled for row in discover_product_rows(root))
|
|
assert product_state(root)["Demo-Agent"].ready
|
|
|
|
|
|
@spawns_the_launcher
|
|
def test_gold_a_keyless_world_reports_the_fail_closed_refusal(raven_home: Path, monkeypatch) -> None:
|
|
"""No key anywhere: the launcher's loud exit is passed through as a readable
|
|
outcome (the chain is wired; the refusal is its fail-closed design), not a
|
|
doctor failure."""
|
|
monkeypatch.delenv("DEMO_AGENT_API_KEY", raising=False)
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert "readiness=ready" in r.output
|
|
assert "fail-closed" in r.output
|
|
assert "DEMO_AGENT_API_KEY is not set" in r.output
|
|
assert re.search(r"cannot be inherited \([^)]+\); put the key in", r.output), r.output
|
|
assert "handshake GREEN" not in r.output
|
|
|
|
|
|
def test_a_scaffold_still_refuses_in_words_on_a_raven_without_inherit_refusal(raven_home: Path, monkeypatch) -> None:
|
|
"""A generated folder is not refreshed when raven changes version, so it can
|
|
run on a raven that predates ``inherit_refusal``: that one refuses without
|
|
the reason instead of raising AttributeError."""
|
|
import importlib.util
|
|
|
|
from raven.config import product_render
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
assert r.exit_code == 0, r.output
|
|
target = raven_home / "agents" / "demo-agent"
|
|
monkeypatch.delenv("DEMO_AGENT_API_KEY", raising=False)
|
|
spec = importlib.util.spec_from_file_location("older_raven_generated_run", target / "run.py")
|
|
run = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(run)
|
|
|
|
with pytest.raises(SystemExit, match=r"cannot be inherited \([^)]+\); put the key in"):
|
|
run.render_config(target / "config.json")
|
|
monkeypatch.delattr(product_render, "inherit_refusal")
|
|
with pytest.raises(
|
|
SystemExit, match=r"DEMO_AGENT_API_KEY is not set and the host's model cannot be inherited; put"
|
|
):
|
|
run.render_config(target / "config.json")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# doctor and smoke failure shapes
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_doctor_names_every_broken_artifact_and_exits_nonzero(raven_home: Path, tmp_path: Path, monkeypatch) -> None:
|
|
"""A template tree whose files are wrong lands, then the doctor reds each check by name."""
|
|
root = tmp_path / "templates"
|
|
root.mkdir()
|
|
(root / "subagent.json").write_text("{not json", encoding="utf-8")
|
|
plugin = root / "plugins" / "my-agent-flow"
|
|
plugin.mkdir(parents=True)
|
|
(plugin / "raven-plugin.toml").write_text("[plugin\nid=", encoding="utf-8")
|
|
(root / "run.py").write_text('"""Stub for my-agent."""\ndef broken(:\n', encoding="utf-8")
|
|
monkeypatch.setattr("raven.cli.agents_commands._templates_root", lambda: root)
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "subagent.json: INVALID" in r.output
|
|
assert "raven-plugin.toml: INVALID" in r.output
|
|
assert "BROKEN" in r.output
|
|
assert "left at" in r.output
|
|
|
|
|
|
def test_doctor_reports_a_row_that_is_listed_but_not_ready(raven_home: Path, tmp_path: Path, monkeypatch) -> None:
|
|
"""A valid manifest whose launcher file is missing: discovery lists the row disabled."""
|
|
root = tmp_path / "templates"
|
|
root.mkdir()
|
|
(root / "subagent.json").write_text(
|
|
json.dumps(
|
|
{
|
|
"name": "My-Agent",
|
|
"kind": "acp",
|
|
"command": "{PYTHON} {SUBAGENT_DIR}/run.py --acp",
|
|
"cwd": "{SUBAGENT_DIR}",
|
|
}
|
|
),
|
|
encoding="utf-8",
|
|
)
|
|
plugin = root / "plugins" / "my-agent-flow"
|
|
plugin.mkdir(parents=True)
|
|
(plugin / "raven-plugin.toml").write_text('[plugin]\nid = "my-agent-flow"\nversion = "0.1.0"\n', encoding="utf-8")
|
|
monkeypatch.setattr("raven.cli.agents_commands._templates_root", lambda: root)
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "NOT ready" in r.output
|
|
assert "launcher" in r.output
|
|
|
|
|
|
def test_a_failed_smoke_exits_nonzero_and_keeps_the_folder(raven_home: Path, monkeypatch) -> None:
|
|
monkeypatch.setattr("raven.cli.agents_commands._smoke_handshake", lambda row: ("failed", "no reply"))
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "smoke: FAILED" in r.output
|
|
assert (raven_home / "agents" / "demo-agent" / "subagent.json").is_file()
|
|
|
|
|
|
@spawns_the_launcher
|
|
@pytest.mark.parametrize(
|
|
("script", "expected"),
|
|
[
|
|
("print('plain text', flush=True)\nimport time\ntime.sleep(3)\n", "not JSON-RPC"),
|
|
(
|
|
"import json\nprint(json.dumps({'jsonrpc': '2.0', 'id': 1, 'result': {}}), flush=True)\n",
|
|
"not a legal InitializeResponse",
|
|
),
|
|
("pass\n", "no reply"),
|
|
],
|
|
)
|
|
def test_smoke_judges_the_three_bad_reply_shapes(tmp_path: Path, script: str, expected: str) -> None:
|
|
"""The smoke's own verdicts, against tiny stand-in launchers: garbage, bad shape, silence.
|
|
|
|
The stand-in is a file, like the launcher a scaffolded row names.
|
|
"""
|
|
from types import SimpleNamespace
|
|
|
|
from raven.cli.agents_commands import _smoke_handshake
|
|
|
|
stub = tmp_path / "stub_launcher.py"
|
|
stub.write_text(script, encoding="utf-8")
|
|
row = SimpleNamespace(command=f"{sys.executable} {stub}", cwd="", ready_timeout_ms=15000)
|
|
|
|
status, detail = _smoke_handshake(row)
|
|
|
|
assert status == "failed"
|
|
assert expected in detail
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# the engine-wheel variant
|
|
# ---------------------------------------------------------------------------
|
|
|
|
ENGINE_AGENT_TREE = {
|
|
".env.example",
|
|
"README.md",
|
|
"config.json",
|
|
"install.py",
|
|
"run.py",
|
|
"subagent.json",
|
|
}
|
|
|
|
ENGINE_TREE = {
|
|
"pyproject.toml",
|
|
"demo_agent_engine/__init__.py",
|
|
"demo_agent_engine/raven-plugin.toml",
|
|
"demo_agent_engine/plugin.py",
|
|
"demo_agent_engine/tools/__init__.py",
|
|
"demo_agent_engine/tools/hello.py",
|
|
}
|
|
|
|
|
|
@pytest.fixture
|
|
def workdir(tmp_path: Path, monkeypatch) -> Path:
|
|
work = tmp_path / "work"
|
|
work.mkdir()
|
|
monkeypatch.chdir(work)
|
|
return work
|
|
|
|
|
|
def test_engine_wheel_scaffolds_both_trees_in_the_user_landing(raven_home: Path, workdir: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
target = raven_home / "agents" / "demo-agent"
|
|
engine = workdir / "demo-agent-engine"
|
|
assert {p.relative_to(target).as_posix() for p in target.rglob("*") if p.is_file()} == ENGINE_AGENT_TREE
|
|
assert {p.relative_to(engine).as_posix() for p in engine.rglob("*") if p.is_file()} == ENGINE_TREE
|
|
|
|
for root in (target, engine):
|
|
# Materialized before compiling: py_compile writes bytecode, and a live
|
|
# rglob would walk into the fresh __pycache__ and try to read it as text.
|
|
files = [p for p in root.rglob("*") if p.is_file()]
|
|
for index, path in enumerate(files):
|
|
rel = path.relative_to(root).as_posix()
|
|
text = path.read_text(encoding="utf-8")
|
|
for token in TOKENS:
|
|
assert token not in rel, (rel, token)
|
|
assert token not in text, (rel, token)
|
|
if rel.endswith(".py"):
|
|
py_compile.compile(str(path), cfile=str(root.parent / f"scratch-{index}.pyc"), doraise=True)
|
|
|
|
config = json.loads((target / "config.json").read_text(encoding="utf-8"))
|
|
assert config == {"plugins": {"config": {"demo-agent-engine": {"enabled": True}}}}
|
|
run_py = (target / "run.py").read_text(encoding="utf-8")
|
|
assert 'ENGINE_PACKAGE = "demo_agent_engine"' in run_py
|
|
assert 'FLOW_PLUGIN_ID = "demo-agent-engine"' in run_py
|
|
row = json.loads((target / "subagent.json").read_text(encoding="utf-8"))
|
|
assert row["engine"] == {"package": "demo_agent_engine", "wheel": "demo-agent-engine"}
|
|
|
|
assert "disabled as designed" in r.output
|
|
assert "demo-agent-engine" in r.output
|
|
assert "pip install -e" in r.output
|
|
|
|
|
|
def test_engine_wheel_here_lands_on_the_plugins_dist_shelf(raven_home: Path, workdir: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--here", "--engine-wheel", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert (workdir / "agents" / "demo-agent" / "run.py").is_file()
|
|
engine = workdir / "plugins-dist" / "demo-agent-engine"
|
|
assert {p.relative_to(engine).as_posix() for p in engine.rglob("*") if p.is_file()} == ENGINE_TREE
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
def test_engine_pyproject_and_manifest_parse(raven_home: Path, workdir: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
engine = workdir / "demo-agent-engine"
|
|
with (engine / "pyproject.toml").open("rb") as f:
|
|
pyproject = tomllib.load(f)
|
|
assert pyproject["project"]["name"] == "demo-agent-engine"
|
|
assert pyproject["project"]["entry-points"]["raven.plugins"] == {"demo-agent-engine": "demo_agent_engine"}
|
|
assert pyproject["tool"]["hatch"]["build"]["targets"]["wheel"]["packages"] == ["demo_agent_engine"]
|
|
|
|
from raven.plugins.manifest import PluginManifest
|
|
|
|
manifest = PluginManifest.from_toml_path(engine / "demo_agent_engine" / "raven-plugin.toml")
|
|
assert manifest.id == "demo-agent-engine"
|
|
assert [t.name for t in manifest.contributes.tools] == ["demo_agent_hello"]
|
|
assert [h.name for h in manifest.contributes.hooks] == ["demo_agent_engine"]
|
|
|
|
|
|
def test_the_engine_row_passes_the_exported_subagent_schema(raven_home: Path, workdir: Path) -> None:
|
|
import jsonschema
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
row = json.loads((raven_home / "agents" / "demo-agent" / "subagent.json").read_text(encoding="utf-8"))
|
|
schema_path = Path(__file__).resolve().parent.parent / "schemas" / "subagent.schema.json"
|
|
schema = json.loads(schema_path.read_text(encoding="utf-8"))
|
|
|
|
jsonschema.Draft202012Validator(schema).validate(row)
|
|
|
|
|
|
def test_engine_readiness_disabled_until_the_wheel_is_importable(raven_home: Path, workdir: Path, monkeypatch) -> None:
|
|
"""Both halves against the real probe: sys.path injection over a find_spec
|
|
monkeypatch, so the fact under test (importability where raven runs) is
|
|
exercised, not faked. invalidate_caches because the injected directory was
|
|
created after this interpreter started -- a fresh process (the real flow:
|
|
pip install, restart the gateway) needs no such step, measured."""
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
from raven.agent.subagent.vendored_agents import discover_product_rows, product_state
|
|
|
|
root = raven_home / "agents"
|
|
row = next(row for row in discover_product_rows(root) if row.name == "Demo-Agent")
|
|
state = product_state(root)["Demo-Agent"]
|
|
assert not row.enabled
|
|
assert not state.ready and state.kind == "engine"
|
|
assert "demo-agent-engine" in state.detail
|
|
|
|
import importlib
|
|
|
|
monkeypatch.syspath_prepend(str(workdir / "demo-agent-engine"))
|
|
importlib.invalidate_caches()
|
|
|
|
row = next(row for row in discover_product_rows(root) if row.name == "Demo-Agent")
|
|
assert row.enabled
|
|
assert product_state(root)["Demo-Agent"].ready
|
|
|
|
|
|
def test_engine_wheel_dry_run_previews_both_trees(raven_home: Path, workdir: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel", "--dry-run"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert r.output.count("would create") == 2
|
|
assert "demo_agent_engine/plugin.py" in r.output
|
|
assert "pyproject.toml" in r.output
|
|
assert "plugins/demo-agent-flow" not in r.output
|
|
assert not (raven_home / "agents").exists()
|
|
assert not (workdir / "demo-agent-engine").exists()
|
|
|
|
|
|
def test_an_existing_engine_directory_is_refused(raven_home: Path, workdir: Path) -> None:
|
|
(workdir / "demo-agent-engine").mkdir()
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel", "--no-smoke"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "already exists" in r.output
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
@spawns_the_launcher
|
|
def test_gold_engine_variant_reports_the_designed_disabled_state(raven_home: Path, workdir: Path) -> None:
|
|
"""The engine gold test: generate, doctor reports disabled-as-designed, and
|
|
the smoke passes through the launcher's own engine refusal (which fires
|
|
before any key handling, so no credentials are needed)."""
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert "disabled as designed" in r.output
|
|
assert "fail-closed" in r.output
|
|
assert "engine wheel is not installed" in r.output
|
|
assert "pip install -e" in r.output
|
|
assert "handshake GREEN" not in r.output
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# the c7 hardening wave: findings from the pre-push adversarial review
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_register_is_refused_while_the_engine_row_is_not_ready(raven_home: Path, workdir: Path) -> None:
|
|
"""H1: a pinned row overrides the discovered one wholesale and is never
|
|
readiness-checked, so pinning a cannot-start row would put an enabled name
|
|
on the dispatch roster."""
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel", "--register", "--no-smoke"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "--register refused" in r.output
|
|
from raven.config.update_subagents import get_agents
|
|
|
|
assert get_agents(config_path=raven_home / "config.json") == []
|
|
assert (raven_home / "agents" / "demo-agent" / "subagent.json").is_file()
|
|
|
|
|
|
def test_a_display_collision_with_a_discovered_folder_is_refused(raven_home: Path) -> None:
|
|
"""H2: the tree is a roster without registration, so the name fence must
|
|
read it -- not only config."""
|
|
r1 = runner.invoke(app, ["agents", "new", "demo", "--no-smoke"])
|
|
assert r1.exit_code == 0, r1.output
|
|
|
|
r2 = runner.invoke(app, ["agents", "new", "demo-x", "--display", "Demo", "--no-smoke"])
|
|
|
|
assert r2.exit_code != 0
|
|
assert "one namespace across casings" in r2.output
|
|
assert not (raven_home / "agents" / "demo-x").exists()
|
|
|
|
|
|
def test_a_folder_id_collision_in_another_casing_is_refused(raven_home: Path) -> None:
|
|
r1 = runner.invoke(app, ["agents", "new", "demo", "--no-smoke"])
|
|
assert r1.exit_code == 0, r1.output
|
|
|
|
r2 = runner.invoke(app, ["agents", "new", "other", "--display", "demo", "--no-smoke"])
|
|
|
|
assert r2.exit_code != 0
|
|
assert not (raven_home / "agents" / "other").exists()
|
|
|
|
|
|
def test_the_doctor_reads_its_own_folder_not_a_same_named_one(raven_home: Path) -> None:
|
|
"""H2: pre-existing duplicates must not let the doctor misattribute --
|
|
a broken same-named neighbor is not this scaffold's failure."""
|
|
r1 = runner.invoke(app, ["agents", "new", "demo-x", "--display", "Alpha", "--no-smoke"])
|
|
assert r1.exit_code == 0, r1.output
|
|
(raven_home / "agents" / "demo-x" / "run.py").unlink()
|
|
|
|
r2 = runner.invoke(app, ["agents", "new", "alpha", "--no-smoke"])
|
|
|
|
assert r2.exit_code != 0
|
|
assert "one namespace across casings" in r2.output
|
|
|
|
|
|
@pytest.mark.parametrize("name", ["nl-demo\n", "nl-demo\nx"])
|
|
def test_a_name_with_an_embedded_newline_is_refused(raven_home: Path, name: str) -> None:
|
|
"""M1: re.match with $ let a trailing newline through and created a
|
|
newline-named directory in the real home."""
|
|
r = runner.invoke(app, ["agents", "new", name])
|
|
|
|
assert r.exit_code != 0
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
def test_a_display_with_a_trailing_newline_is_refused(raven_home: Path) -> None:
|
|
r = runner.invoke(app, ["agents", "new", "nl-two", "--display", "Nl Two\n"])
|
|
|
|
assert r.exit_code != 0
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
def test_stale_staging_residue_is_swept_and_never_discovered(raven_home: Path) -> None:
|
|
"""M2: what a SIGKILL between the last file write and the rename leaves."""
|
|
residue = raven_home / "agents" / ".demo-two.partial-k3jx9q"
|
|
residue.mkdir(parents=True)
|
|
(residue / "subagent.json").write_text(
|
|
json.dumps({"name": "Demo-Two", "kind": "acp", "command": "{PYTHON} {SUBAGENT_DIR}/run.py --acp"}),
|
|
encoding="utf-8",
|
|
)
|
|
|
|
from raven.agent.subagent.vendored_agents import discover_product_rows
|
|
|
|
assert discover_product_rows(raven_home / "agents") == []
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-two", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert not residue.exists()
|
|
rows = discover_product_rows(raven_home / "agents")
|
|
assert [row.name for row in rows] == ["Demo-Two"]
|
|
|
|
|
|
def test_a_builtin_name_in_another_casing_is_refused(raven_home: Path) -> None:
|
|
"""M3: 'RAVEN' beside the built-in 'Raven' would be an enabled impostor."""
|
|
r = runner.invoke(app, ["agents", "new", "impostor", "--display", "RAVEN"])
|
|
|
|
assert r.exit_code != 0
|
|
assert "built-in" in r.output
|
|
assert not (raven_home / "agents").exists()
|
|
|
|
|
|
def test_the_engine_declaration_round_trips_through_the_row_schema(raven_home: Path, workdir: Path) -> None:
|
|
"""L1: a pinned copy of the manifest must not silently drop the engine key."""
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel", "--no-smoke"])
|
|
assert r.exit_code == 0, r.output
|
|
|
|
from raven.config.schema import ThirdPartyAcpSubagentConfig
|
|
|
|
row = json.loads((raven_home / "agents" / "demo-agent" / "subagent.json").read_text(encoding="utf-8"))
|
|
validated = ThirdPartyAcpSubagentConfig.model_validate(row)
|
|
assert validated.engine is not None
|
|
assert validated.engine.package == "demo_agent_engine"
|
|
dumped = validated.model_dump(by_alias=True)
|
|
assert dumped["engine"] == {"package": "demo_agent_engine", "wheel": "demo-agent-engine"}
|
|
|
|
|
|
@pytest.mark.skipif(
|
|
os.name == "nt" or os.geteuid() == 0,
|
|
reason="Windows and root cannot make a directory unwritable with POSIX permission bits",
|
|
)
|
|
def test_an_unwritable_target_refuses_cleanly(raven_home: Path, tmp_path: Path, monkeypatch) -> None:
|
|
"""L4: a traceback is not a refusal."""
|
|
checkout = tmp_path / "frozen"
|
|
checkout.mkdir()
|
|
(checkout / "agents").mkdir()
|
|
(checkout / "agents").chmod(0o555)
|
|
monkeypatch.chdir(checkout)
|
|
try:
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--here", "--no-smoke"])
|
|
finally:
|
|
(checkout / "agents").chmod(0o755)
|
|
|
|
assert r.exit_code != 0
|
|
assert "Error:" in r.output and "cannot write" in r.output
|
|
assert "Traceback" not in r.output
|
|
|
|
|
|
def test_the_landed_folder_is_not_mkdtemps_0700(raven_home: Path) -> None:
|
|
"""L5: the folder should match hand-made siblings, not the staging dir."""
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
mode = (raven_home / "agents" / "demo-agent").stat().st_mode & 0o777
|
|
assert mode == 0o755, oct(mode)
|
|
|
|
|
|
def test_engine_factories_decline_without_their_config_slice(raven_home: Path, workdir: Path) -> None:
|
|
"""H3b: an installed engine wheel activates in every raven process in the
|
|
environment, so the factories themselves scope it to the one agent whose
|
|
rendered config enables the slice."""
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--engine-wheel", "--no-smoke"])
|
|
assert r.exit_code == 0, r.output
|
|
|
|
import importlib
|
|
import sys
|
|
|
|
sys.path.insert(0, str(workdir / "demo-agent-engine"))
|
|
importlib.invalidate_caches()
|
|
try:
|
|
plugin = importlib.import_module("demo_agent_engine.plugin")
|
|
hello = importlib.import_module("demo_agent_engine.tools.hello")
|
|
|
|
class _Ctx:
|
|
def __init__(self, config):
|
|
self.config = config
|
|
self.services = None
|
|
self.logger = None
|
|
|
|
assert plugin.make_hook(_Ctx({})) is None
|
|
assert hello.make_hello(_Ctx({})) is None
|
|
assert plugin.make_hook(_Ctx({"enabled": True})) is not None
|
|
tool = hello.make_hello(_Ctx({"enabled": True}))
|
|
assert tool is not None and tool.name == "demo_agent_hello"
|
|
finally:
|
|
sys.path.remove(str(workdir / "demo-agent-engine"))
|
|
for mod in list(sys.modules):
|
|
if mod.startswith("demo_agent_engine"):
|
|
del sys.modules[mod]
|
|
|
|
|
|
@spawns_the_launcher
|
|
def test_smoke_fails_a_green_handshake_whose_plugin_activation_failed(tmp_path: Path) -> None:
|
|
"""H3c: a legal reply while a plugin failed to load is a broken agent behind
|
|
a green light -- the agent runs without what that plugin contributes, its own
|
|
tools, hooks and memory backend when it is the agent's engine, so the smoke
|
|
reds it instead of blessing it. The stub prints the notice the host itself
|
|
writes, so the notice and the smoke's reader cannot drift apart."""
|
|
from types import SimpleNamespace
|
|
|
|
from raven.cli.agents_commands import _smoke_handshake
|
|
from raven.core.plugin_stack import plugin_failure_note
|
|
from raven.plugins import PluginActivationFailure, PluginManifest
|
|
|
|
note = plugin_failure_note(
|
|
PluginActivationFailure(
|
|
plugin_id="demo-flow",
|
|
reason="tool 'x' contributed by both 'other' and 'demo-flow'",
|
|
manifest=PluginManifest(id="demo-flow", version="0.1.0"),
|
|
)
|
|
)
|
|
stub = tmp_path / "stub_launcher.py"
|
|
stub.write_text(
|
|
"import json\n"
|
|
"import sys\n"
|
|
f"print({note!r}, file=sys.stderr, flush=True)\n"
|
|
"print(json.dumps({'jsonrpc': '2.0', 'id': 1, 'result': {'protocolVersion': 1}}), flush=True)\n"
|
|
"import time\n"
|
|
"time.sleep(3)\n",
|
|
encoding="utf-8",
|
|
)
|
|
row = SimpleNamespace(command=f"{sys.executable} {stub}", cwd="", ready_timeout_ms=15000)
|
|
|
|
status, detail = _smoke_handshake(row)
|
|
|
|
assert status == "failed"
|
|
assert "WITHOUT what it contributes" in detail
|
|
assert "Plugin 'demo-flow' did not load" in detail
|
|
|
|
|
|
def test_an_installed_engine_pollutes_nothing_and_two_scaffolds_no_longer_collide(
|
|
raven_home: Path, workdir: Path, tmp_path: Path, monkeypatch
|
|
) -> None:
|
|
"""H3 end to end (the review's probe-w/probe-h scenario, in process): an
|
|
engine wheel visible to entry-point discovery activates everywhere, but
|
|
the derived tool names cannot collide with a sibling's, and the declined
|
|
factories keep the host's and the sibling's turns empty of engine tools."""
|
|
r1 = runner.invoke(app, ["agents", "new", "probe-w", "--engine-wheel", "--no-smoke"])
|
|
assert r1.exit_code == 0, r1.output
|
|
r2 = runner.invoke(app, ["agents", "new", "probe-h", "--no-smoke"])
|
|
assert r2.exit_code == 0, r2.output
|
|
|
|
# Make the engine what pip install -e makes it: importable AND registered
|
|
# on the raven.plugins entry-point group (a dist-info beside the package).
|
|
engine_root = workdir / "probe-w-engine"
|
|
dist_info = engine_root / "probe_w_engine-0.1.0.dist-info"
|
|
dist_info.mkdir()
|
|
(dist_info / "METADATA").write_text(
|
|
"Metadata-Version: 2.1\nName: probe-w-engine\nVersion: 0.1.0\n", encoding="utf-8"
|
|
)
|
|
(dist_info / "entry_points.txt").write_text("[raven.plugins]\nprobe-w-engine = probe_w_engine\n", encoding="utf-8")
|
|
|
|
import importlib
|
|
|
|
monkeypatch.syspath_prepend(str(engine_root))
|
|
importlib.invalidate_caches()
|
|
|
|
from raven.config.raven import load_raven_config
|
|
from raven.core.plugin_stack import build_plugin_registry, build_plugin_tools
|
|
|
|
probe_h_plugins = raven_home / "agents" / "probe-h" / "plugins"
|
|
sibling_config = tmp_path / "sibling-config.json"
|
|
sibling_config.write_text(
|
|
json.dumps({"plugins": {"dirs": [str(probe_h_plugins)], "config": {"probe-h-flow": {"enabled": True}}}}),
|
|
encoding="utf-8",
|
|
)
|
|
cfg = load_raven_config(sibling_config)
|
|
registry = build_plugin_registry(cfg)
|
|
activated = set(registry.activated_ids())
|
|
assert {"probe-h-flow", "probe-w-engine"} <= activated, activated
|
|
|
|
workspace = tmp_path / "ws"
|
|
workspace.mkdir()
|
|
tool_names = {tool.name for tool in build_plugin_tools(workspace, cfg, registry=registry)}
|
|
assert "probe_h_hello" in tool_names
|
|
assert "probe_w_hello" not in tool_names
|
|
|
|
host_config = tmp_path / "host-config.json"
|
|
host_config.write_text("{}", encoding="utf-8")
|
|
host_cfg = load_raven_config(host_config)
|
|
host_registry = build_plugin_registry(host_cfg)
|
|
assert "probe-w-engine" in set(host_registry.activated_ids())
|
|
host_tools = {tool.name for tool in build_plugin_tools(workspace, host_cfg, registry=host_registry)}
|
|
assert "probe_w_hello" not in host_tools
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_dry_run_leaves_stale_staging_residue_untouched(raven_home: Path) -> None:
|
|
"""A dry run promises zero disk mutation; sweeping residue is a mutation,
|
|
so it belongs to the write path alone."""
|
|
residue = raven_home / "agents" / ".victim.partial-abc123"
|
|
residue.mkdir(parents=True)
|
|
(residue / "half-written.txt").write_text("x", encoding="utf-8")
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--dry-run"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert "dry run: nothing written" in r.output
|
|
assert residue.is_dir() and (residue / "half-written.txt").is_file()
|
|
|
|
r2 = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
|
|
assert r2.exit_code == 0, r2.output
|
|
assert not residue.exists()
|
|
|
|
|
|
def test_a_home_path_with_whitespace_registers_with_a_quoted_command(tmp_path: Path, monkeypatch) -> None:
|
|
"""A spacey landing is quoted into the roster command, not refused: the
|
|
folder scaffolds and the registered command still spawns the launcher as
|
|
one token."""
|
|
home = tmp_path / "space y" / "home"
|
|
home.mkdir(parents=True)
|
|
monkeypatch.setenv("RAVEN_HOME", str(home))
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--register", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert (home / "agents" / "demo-agent").is_dir()
|
|
from raven.config.update_subagents import get_agents
|
|
from raven.utils.commands import command_argv
|
|
|
|
(row,) = get_agents(config_path=home / "config.json")
|
|
argv = command_argv(row["command"])
|
|
assert str(home / "agents" / "demo-agent" / "run.py") in argv
|
|
|
|
|
|
def test_a_spacey_working_directory_registers_in_here_mode(raven_home: Path, tmp_path: Path, monkeypatch) -> None:
|
|
checkout = tmp_path / "space y checkout"
|
|
checkout.mkdir()
|
|
monkeypatch.chdir(checkout)
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--here", "--register", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
assert (checkout / "agents" / "demo-agent").is_dir()
|
|
from raven.config.update_subagents import get_agents
|
|
from raven.utils.commands import command_argv
|
|
|
|
(row,) = get_agents(config_path=raven_home / "config.json")
|
|
assert str(checkout / "agents" / "demo-agent" / "run.py") in command_argv(row["command"])
|
|
|
|
|
|
def test_an_explicit_schema_default_workspace_renders_like_an_omitted_one(raven_home: Path, monkeypatch) -> None:
|
|
"""The sentinel is the paper's WORKSPACE_DEFAULT_SENTINEL, not the literal
|
|
'workspace': a config that carries the default explicitly must land the
|
|
engine's Agent home exactly where an omitted value does -- outside the
|
|
host Agent home, never as a literal path under the state root."""
|
|
import importlib.util
|
|
|
|
from raven.contracts.path_policy import WORKSPACE_DEFAULT_SENTINEL
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
assert r.exit_code == 0, r.output
|
|
target = raven_home / "agents" / "demo-agent"
|
|
|
|
monkeypatch.setenv("DEMO_AGENT_API_KEY", "dummy-key-for-render")
|
|
spec = importlib.util.spec_from_file_location("c10_generated_run", target / "run.py")
|
|
run = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(run)
|
|
|
|
explicit = target / "explicit-default.json"
|
|
config = json.loads((target / "config.json").read_text(encoding="utf-8"))
|
|
config["agents"] = {"defaults": {"workspace": WORKSPACE_DEFAULT_SENTINEL}}
|
|
explicit.write_text(json.dumps(config), encoding="utf-8")
|
|
|
|
explicit_ws = json.loads(run.render_config(explicit).read_text(encoding="utf-8"))["agents"]["defaults"]["workspace"]
|
|
omitted_ws = json.loads(run.render_config(target / "config.json").read_text(encoding="utf-8"))["agents"][
|
|
"defaults"
|
|
]["workspace"]
|
|
|
|
assert explicit_ws == omitted_ws
|
|
host_agent_home = raven_home / "workspace"
|
|
assert not Path(explicit_ws).is_relative_to(host_agent_home)
|
|
assert "~" not in explicit_ws
|
|
explicit.unlink()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# the c12 wave: one {PYTHON} resolution for discovery, register and installer
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.fixture
|
|
def clean_alt_python(tmp_path: Path) -> Path:
|
|
"""An existing, space-free interpreter path that is not sys.executable."""
|
|
alt = tmp_path / "pyalt"
|
|
alt.symlink_to(sys.executable)
|
|
return alt
|
|
|
|
|
|
def test_register_writes_subagent_python_over_sys_executable(
|
|
raven_home: Path, clean_alt_python: Path, monkeypatch
|
|
) -> None:
|
|
"""``--register`` resolves ``{PYTHON}`` the way discovery does: a
|
|
``SUBAGENT_PYTHON`` set over a spacey ``sys.executable`` is the
|
|
interpreter the pinned row carries."""
|
|
monkeypatch.setenv("SUBAGENT_PYTHON", str(clean_alt_python))
|
|
monkeypatch.setattr(sys, "executable", "/spa cey/python")
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--register", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
from raven.config.update_subagents import get_agents
|
|
from raven.utils.commands import command_argv
|
|
|
|
(row,) = get_agents(config_path=raven_home / "config.json")
|
|
assert command_argv(row["command"])[0] == str(clean_alt_python)
|
|
assert "spa cey" not in row["command"]
|
|
|
|
|
|
def test_a_spacey_subagent_python_registers_with_a_quoted_command(
|
|
raven_home: Path, tmp_path: Path, monkeypatch
|
|
) -> None:
|
|
"""A real, spaced interpreter is quoted into the roster command, not refused."""
|
|
spacey = tmp_path / "spa cey" / "python"
|
|
spacey.parent.mkdir(parents=True)
|
|
spacey.symlink_to(sys.executable)
|
|
monkeypatch.setenv("SUBAGENT_PYTHON", str(spacey))
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--register", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
from raven.config.update_subagents import get_agents
|
|
from raven.utils.commands import command_argv
|
|
|
|
(row,) = get_agents(config_path=raven_home / "config.json")
|
|
assert command_argv(row["command"])[0] == str(spacey)
|
|
|
|
|
|
def test_the_registered_interpreter_equals_the_discovered_one(
|
|
raven_home: Path, clean_alt_python: Path, monkeypatch
|
|
) -> None:
|
|
"""The consistency pin: one folder, one interpreter, whichever side reads it."""
|
|
monkeypatch.setenv("SUBAGENT_PYTHON", str(clean_alt_python))
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--register", "--no-smoke"])
|
|
|
|
assert r.exit_code == 0, r.output
|
|
from raven.agent.subagent.vendored_agents import discover_product_rows
|
|
from raven.config.update_subagents import get_agents
|
|
from raven.utils.commands import command_argv
|
|
|
|
(discovered,) = discover_product_rows(raven_home / "agents")
|
|
(registered,) = get_agents(config_path=raven_home / "config.json")
|
|
assert registered["command"] == discovered.command
|
|
assert command_argv(registered["command"])[0] == str(clean_alt_python)
|
|
|
|
|
|
def test_the_generated_installer_honors_subagent_python_and_quotes_whitespace(
|
|
raven_home: Path, clean_alt_python: Path, tmp_path: Path, monkeypatch
|
|
) -> None:
|
|
"""The template installer resolves {PYTHON} the way discovery does, and a
|
|
spacey resolved interpreter is quoted into the row, not refused."""
|
|
import importlib.util
|
|
|
|
from raven.utils.commands import command_argv
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
assert r.exit_code == 0, r.output
|
|
target = raven_home / "agents" / "demo-agent"
|
|
|
|
spec = importlib.util.spec_from_file_location("c12_generated_install", target / "install.py")
|
|
install = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(install)
|
|
|
|
monkeypatch.setenv("SUBAGENT_PYTHON", str(clean_alt_python))
|
|
assert install.main() == 0
|
|
from raven.config.update_subagents import get_agents
|
|
|
|
(row,) = get_agents(config_path=raven_home / "config.json")
|
|
assert command_argv(row["command"])[0] == str(clean_alt_python)
|
|
assert row["cwd"] == str(target)
|
|
|
|
spacey = tmp_path / "spa cey" / "python"
|
|
spacey.parent.mkdir(parents=True)
|
|
spacey.symlink_to(sys.executable)
|
|
monkeypatch.setenv("SUBAGENT_PYTHON", str(spacey))
|
|
assert install.main() == 0
|
|
(spacey_row,) = get_agents(config_path=raven_home / "config.json")
|
|
assert command_argv(spacey_row["command"])[0] == str(spacey)
|
|
|
|
|
|
def test_the_generated_installer_under_a_raven_without_the_quoting_rule_writes_the_unquoted_row(
|
|
raven_home: Path, clean_alt_python: Path, monkeypatch
|
|
) -> None:
|
|
"""A generated folder is never refreshed, so its installer can run under a raven
|
|
older than ``raven.utils.commands``; there it writes the unquoted row rather than
|
|
failing on the import."""
|
|
import importlib.util
|
|
|
|
r = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
assert r.exit_code == 0, r.output
|
|
target = raven_home / "agents" / "demo-agent"
|
|
|
|
spec = importlib.util.spec_from_file_location("c12_generated_install_older_raven", target / "install.py")
|
|
install = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(install)
|
|
written: list[dict] = []
|
|
monkeypatch.setattr("raven.config.update_subagents.add_third_party_subagent", written.append)
|
|
monkeypatch.setenv("SUBAGENT_PYTHON", str(clean_alt_python))
|
|
monkeypatch.setitem(sys.modules, "raven.utils.commands", None)
|
|
|
|
assert install.main() == 0
|
|
|
|
(row,) = written
|
|
here = target.resolve()
|
|
assert row["command"] == f"{clean_alt_python} {here}/run.py --acp"
|
|
assert row["cwd"] == str(here)
|
|
|
|
|
|
@pytest.mark.parametrize("spaced", ["none", "folder", "python", "cwd"])
|
|
def test_the_generated_installer_on_older_raven_checks_paths_before_registering(
|
|
raven_home: Path, tmp_path: Path, monkeypatch, spaced: str
|
|
) -> None:
|
|
import importlib.util
|
|
|
|
from raven.config.update_subagents import get_agents
|
|
|
|
if spaced in {"folder", "cwd"}:
|
|
raven_home = tmp_path / "my home"
|
|
monkeypatch.setenv("RAVEN_HOME", str(raven_home))
|
|
result = runner.invoke(app, ["agents", "new", "demo-agent", "--no-smoke"])
|
|
assert result.exit_code == 0, result.output
|
|
target = raven_home / "agents" / "demo-agent"
|
|
spec = importlib.util.spec_from_file_location("generated_install_older_raven_whitespace", target / "install.py")
|
|
install = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(install)
|
|
python = tmp_path / ("my python" if spaced == "python" else "python")
|
|
if spaced == "cwd":
|
|
manifest_path = target / "subagent.json"
|
|
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
|
manifest["command"] = "python run.py --acp"
|
|
manifest_path.write_text(json.dumps(manifest), encoding="utf-8")
|
|
monkeypatch.setenv("SUBAGENT_PYTHON", str(python))
|
|
monkeypatch.setitem(sys.modules, "raven.utils.commands", None)
|
|
|
|
if spaced in {"folder", "python"}:
|
|
with pytest.raises(SystemExit, match="Upgrade Raven"):
|
|
install.main()
|
|
|
|
assert get_agents(config_path=raven_home / "config.json") == []
|
|
else:
|
|
assert install.main() == 0
|
|
(row,) = get_agents(config_path=raven_home / "config.json")
|
|
expected = "python run.py --acp" if spaced == "cwd" else f"{python} {target}/run.py --acp"
|
|
assert row["command"] == expected
|
|
assert row["cwd"] == str(target)
|