Raven-1/tests/test_core_plugin_stack.py
0xKT 44309dde3a feat(cli): raven agents new scaffolds a complete agent end to end
## Summary

raven agents new scaffolds a complete, working agent on the installed
raven: eleven fire-tested template files land in the raven home's agents/
tree (discovered with zero registration), a four-segment health card
examines the result (atomic write, doctor, a real ACP handshake smoke,
next steps), and --engine-wheel grows the same scaffold into the
wheel-shaped harness the shipped engines use. The developer guide ships
with it, in-repo and editor-checkable.

Key decisions:

- **Templates are real code, fire-tested before templating.** The eleven
  files under raven/templates/agents_scaffold/ pass ruff and py_compile
  like any source, ride the wheel, and their content was proven live
  before landing: discovery, handshake, and plugin tool/hook injection
  all green on a hand-assembled probe in a disposable home. A dedicated
  spike also proved the placement is invisible to the import-linter
  contracts and needs exactly one coverage-gate exclusion line.
- **Home-first landing, registration as an explicit act.** The default
  target is the raven home's agents/ tree, discovery's first root: a
  scaffolded folder is on the roster without any registration.
  --register mirrors the generated install.py through the same pinned
  write door; --here lands in the checkout for in-repo agent work.
- **The scaffold examines its own work.** Doctor re-validates the
  manifest, re-parses the plugin manifest, compiles every generated
  file, and really scans discovery; smoke spawns the launcher for one
  initialize frame. Three smoke outcomes are told apart: GREEN, the
  launcher's loud no-key refusal (fail-closed design, passed through
  verbatim), and FAILED with the scene preserved. Display names are
  fenced so no input can corrupt the generated manifest.
- **Schemas derive from the models.** scripts/export_agent_schemas.py
  renders schemas/subagent.schema.json (a discriminated union on kind)
  and schemas/raven-plugin.schema.json from the pydantic models; a
  contract family holds the on-disk files byte-equal to a fresh render
  and validates every real manifest on the tree against them. Editors
  get instant validation with two settings lines, documented in the
  guide.
- **The lexicon follows the terminology ruling.** agents/BUILDING.md is
  the from-zero developer guide (gold standard: a developer who never
  read the source reaches a green handshake in ten minutes); CONTEXT.md
  renames the category entry to Discovered agent -- agent the thing,
  harness the mechanism, subagent the protocol seat -- with product
  absorbed into Avoid; the charter gains the scaffold entry and the
  engine-ownership line.
- **--engine-wheel lands both shapes.** User mode drops the engine
  project in the working directory (pip install -e territory); --here
  lands it on the plugins-dist shelf. The launcher adopts the official
  ENGINE_PACKAGE convention and refuses loudly pre-exec when the engine
  is missing; the doctor reports that row as disabled by design with
  the install cure printed. No plugins/ directory ships beside an
  engine-shaped agent, matching the shipped engines.
- **A pre-push adversarial review hardened the whole surface.** Two
  hostile lenses (filesystem edges of a command that writes into real
  raven homes; template quality as the file set every future agent
  copies) found 3 HIGH / 6 MED / 5 LOW -- among them: --register could
  pin an enabled row that cannot start, the name fence ignored the
  discovered tree, one installed engine wheel rode the host's and every
  sibling's turns through a fixed tool name, and crash residue was
  discovered as a hidden live agent. All fourteen close in a
  dedicated hardening commit, each with a regression test built from the review's own
  reproduction.
- **Near-zero kernel touch, and the two touches are the review's.**
  Nothing under raven/contracts/ or the agent loop changed. The branch
  carries exactly two narrow runtime fixes exposed by the review:
  discovery now skips dot-directories (crash residue is never an
  agent), and the config schema gains an engine round-trip field
  (declared as a field-level neutral seam; merge and readiness
  semantics untouched).

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- Every commit landed through a full-suite rehearsal on a clean bench
  before touching the branch; tails over the original base's 19953
  passed / 137 skipped: 19980 (templates + command), 19993 (doctor +
  smoke + gold tests), 20010 (schemas), 20010 (docs, count-identical as
  expected), 20018 (engine wheel) -- exit 0 and 137 skipped throughout,
  every step a pure addition, +65 tests net. The branch diff is
  fingerprint-stable across the rebase (git patch-id --stable verified
  pairwise across two clean rebases); the branch closes at 20094 passed
  / 137 skipped / exit 0 on the current main base, +80 tests net over
  whatever base it sits on -- re-measured in full after every rebase.
- A real-machine pass on the live raven home closed the loop end to
  end: one command scaffolds into ~/.raven/agents, the four-segment
  card runs all green, and a genuine LLM turn has the agent's own model
  call its own scaffolded hello tool through the permission gate and
  relay the exact output; the probe was then removed without touching
  the roster.
- Gold tests spawn for real: scaffold into an isolated home, assert the
  four doctor lines, readiness ready, and a live ACP initialize
  handshake GREEN (~3s, zero tokens); a keyless twin pins the refusal
  passthrough and an engine twin pins disabled-as-designed.
- Schema contract: mutation audits both directions (tampered model and
  tampered file each turn the family red); all six subagent manifests
  and all eight raven-plugin.toml files on the tree validate.
- make lint-python, uv run lint-imports (9 contracts kept, 0 broken),
  make check-large-files, make check-source-language: green at every
  step; wheel build carries all seventeen template files and none of
  schemas/.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

New command surface only: no existing agent, roster row, or runtime path
changes behavior. The generated .env stays untracked and the wheel
builder still refuses secrets. The one gate change is a coverage-gate
exclusion declaring templates non-production. Rollback: revert the
squash commit; scaffolded agents already created keep working, since
they run on the installed raven like any discovered folder.

## Related Issues

N/A
2026-09-10 07:07:15 +00:00

423 lines
15 KiB
Python

"""CLI plugin-stack helper.
Exercises :func:`build_plugin_registry` and
:func:`maybe_build_memory_backend` against the ``raven_everos`` plugin,
which the dev environment installs from ``plugins-dist/everos-memory`` and
which discovery finds through the ``raven.plugins`` entry-point group.
"""
from __future__ import annotations
import ast
from pathlib import Path
import pytest
import raven
from raven.config.raven import (
MemoryConfig,
PluginsConfig,
RavenConfig,
)
from raven.config.schema import Config, MediaGenConfig, ToolsConfig, WebToolsConfig
from raven.contracts.memory import MemoryBackend
from raven.core.plugin_stack import (
build_plugin_hooks,
build_plugin_registry,
build_plugin_tools,
everos_plugin_installed,
everos_plugin_missing_note,
maybe_build_memory_backend,
named_plugin_roots,
)
from raven.plugins import PluginRegistry
from tests._everos_presence import everos_plugin_absent, everos_plugin_broken
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _config(
*,
memory_backend: str | None = "everos",
disabled: list[str] | None = None,
plugin_config: dict | None = None,
dirs: list[str] | None = None,
tools: ToolsConfig | None = None,
) -> RavenConfig:
return RavenConfig(
memory=MemoryConfig(backend=memory_backend),
plugins=PluginsConfig(
disabled=list(disabled or []),
config=dict(plugin_config or {}),
dirs=list(dirs or []),
),
base=Config(tools=tools) if tools is not None else Config(),
)
# ---------------------------------------------------------------------------
# build_plugin_registry
# ---------------------------------------------------------------------------
class TestBuildRegistry:
def test_returns_registry_with_everos_activated(self) -> None:
reg = build_plugin_registry(_config())
assert isinstance(reg, PluginRegistry)
assert "everos-memory" in reg.activated_ids()
assert "everos" in reg.memory_backend_names()
def test_disabled_plugin_id_skipped(self) -> None:
reg = build_plugin_registry(
_config(disabled=["everos-memory"]),
)
assert "everos-memory" not in reg.activated_ids()
assert "everos" not in reg.memory_backend_names()
# ---------------------------------------------------------------------------
# maybe_build_memory_backend
# ---------------------------------------------------------------------------
class TestMaybeBuildBackend:
def test_default_config_builds_everos(self, tmp_path: Path) -> None:
backend = maybe_build_memory_backend(tmp_path, _config())
assert backend is not None
assert isinstance(backend, MemoryBackend)
def test_memory_backend_none_returns_none(
self,
tmp_path: Path,
) -> None:
backend = maybe_build_memory_backend(
tmp_path,
_config(memory_backend=None),
)
assert backend is None
def test_unknown_backend_returns_none_no_raise(
self,
tmp_path: Path,
) -> None:
"""A user-config typo / missing plugin must NOT crash boot —
the helper logs + degrades, AgentLoop falls back to legacy."""
backend = maybe_build_memory_backend(
tmp_path,
_config(memory_backend="nonexistent"),
)
assert backend is None
def test_disabled_backend_returns_none(self, tmp_path: Path) -> None:
backend = maybe_build_memory_backend(
tmp_path,
_config(
memory_backend="everos",
disabled=["everos-memory"],
),
)
assert backend is None
# ---------------------------------------------------------------------------
# Per-plugin config slice resolution
# ---------------------------------------------------------------------------
class TestConfigSliceResolution:
def test_config_by_plugin_id(self, tmp_path: Path) -> None:
backend = maybe_build_memory_backend(
tmp_path,
_config(
plugin_config={
"everos-memory": {"mode": "embedded", "base_url": "http://x"},
}
),
)
# Backend received the config slice keyed by plugin id.
assert backend._config["mode"] == "embedded"
assert backend._config["base_url"] == "http://x"
def test_config_by_backend_name_fallback(
self,
tmp_path: Path,
) -> None:
"""When the user uses the shorter key (backend name), the
helper still finds it. Useful for handwritten configs."""
backend = maybe_build_memory_backend(
tmp_path,
_config(
plugin_config={
"everos": {"mode": "http", "base_url": "http://y"},
}
),
)
assert backend._config["mode"] == "http"
assert backend._config["base_url"] == "http://y"
def test_plugin_id_takes_precedence_over_backend_name(
self,
tmp_path: Path,
) -> None:
"""If both keys are present, the canonical (plugin id) wins."""
backend = maybe_build_memory_backend(
tmp_path,
_config(
plugin_config={
"everos-memory": {"marker": "canonical"},
"everos": {"marker": "fallback"},
}
),
)
assert backend._config["marker"] == "canonical"
def test_no_config_slice_yields_empty_dict(
self,
tmp_path: Path,
) -> None:
backend = maybe_build_memory_backend(tmp_path, _config())
assert backend._config == {}
# ---------------------------------------------------------------------------
# Registry injection — caller can pass a pre-built registry
# ---------------------------------------------------------------------------
class TestRegistryInjection:
def test_caller_supplied_registry_used(self, tmp_path: Path) -> None:
reg = build_plugin_registry(_config())
backend = maybe_build_memory_backend(
tmp_path,
_config(),
registry=reg,
)
# Backend constructed via the explicit registry.
assert backend is not None
def test_default_construction_creates_internal_registry(
self,
tmp_path: Path,
) -> None:
# Sanity: with no registry passed, the helper still works.
backend = maybe_build_memory_backend(tmp_path, _config())
assert backend is not None
# ---------------------------------------------------------------------------
# Workspace plumbing through ServiceLocator
# ---------------------------------------------------------------------------
class TestServiceLocatorPlumbing:
def test_workspace_reaches_backend(self, tmp_path: Path) -> None:
backend = maybe_build_memory_backend(tmp_path, _config())
# EverosBackend stores ctx.services on construction.
assert backend._services.workspace == tmp_path
def test_the_two_identities_do_not_arrive_swapped(self, tmp_path: Path) -> None:
"""The whole point of moving identity into ServiceLocator is that a
mismatch makes every written memory unrecallable with no warning. Until
this assertion existed, swapping the two arguments at the only
production wiring point left the entire suite green."""
config = _config()
config.memory.user_id = "u-distinct"
config.memory.agent_id = "a-distinct"
backend = maybe_build_memory_backend(tmp_path, config)
assert backend._services.user_id == "u-distinct"
assert backend._services.agent_id == "a-distinct"
# ---------------------------------------------------------------------------
# plugins.dirs
# ---------------------------------------------------------------------------
class TestConfiguredDirs:
def test_named_roots_resolve_from_config(self, tmp_path: Path) -> None:
assert named_plugin_roots(_config(dirs=[str(tmp_path)])) == (tmp_path,)
assert named_plugin_roots(_config()) == ()
assert named_plugin_roots(None) == ()
def test_a_plugin_under_a_named_root_activates(self, tmp_path: Path) -> None:
root = tmp_path / "plugins"
(root / "shelf").mkdir(parents=True)
(root / "shelf" / "raven-plugin.toml").write_text(
'[plugin]\nid = "shelf"\nversion = "0.1.0"\nenabled_by_default = true\n',
encoding="utf-8",
)
assert "shelf" in build_plugin_registry(_config(dirs=[str(root)])).activated_ids()
assert "shelf" not in build_plugin_registry(_config()).activated_ids()
# ---------------------------------------------------------------------------
# The provider grant
# ---------------------------------------------------------------------------
class _GrantWatch:
"""A registry stand-in that records what locator each factory was handed."""
def __init__(self, seen: list) -> None:
self._seen = seen
def hook_names(self):
return ["h"]
def tool_names(self):
return ["t"]
def hook_plugin_id(self, name):
return "p"
def tool_plugin_id(self, name):
return "p"
def build_hook(self, name, *, config, services):
self._seen.append(("hook", services))
return object()
def build_tool(self, name, *, config, services):
self._seen.append(("tool", services))
return object()
class TestProviderGrant:
def test_the_connections_provider_reaches_hooks_and_tools(self, tmp_path: Path) -> None:
seen: list = []
lent = object()
build_plugin_hooks(tmp_path, _config(), registry=_GrantWatch(seen), provider=lent)
build_plugin_tools(tmp_path, _config(), registry=_GrantWatch(seen), provider=lent)
assert [(kind, services.provider) for kind, services in seen] == [("hook", lent), ("tool", lent)]
def test_no_provider_lends_none(self, tmp_path: Path) -> None:
seen: list = []
build_plugin_hooks(tmp_path, _config(), registry=_GrantWatch(seen))
assert [(kind, services.provider) for kind, services in seen] == [("hook", None)]
class TestHostToolsGrants:
"""The host side of ``media_proxy``, ``web_config`` and ``media_config``.
The plugin side is tested with a hand-built locator; this is the link nothing
else pins, and the one that was broken: every entrance hands the stack its
``RavenConfig``, whose ``tools`` lives under ``base``, so reading ``tools`` off
the extension object lent None to every deployment.
"""
def test_the_hosts_media_proxy_and_web_section_reach_tools(self, tmp_path: Path) -> None:
seen: list = []
tools = ToolsConfig(
media=MediaGenConfig(proxy="http://127.0.0.1:7890"),
web=WebToolsConfig(proxy="socks5://127.0.0.1:1080"),
)
config = _config(tools=tools)
build_plugin_tools(tmp_path, config, registry=_GrantWatch(seen))
[(kind, services)] = seen
assert kind == "tool"
assert services.media_proxy == "http://127.0.0.1:7890"
assert services.web_config is not None
assert services.web_config() is config.base.tools.web
assert services.web_config().proxy == "socks5://127.0.0.1:1080"
assert callable(services.media_config)
def test_a_host_without_a_proxy_lends_none_and_its_web_section(self, tmp_path: Path) -> None:
seen: list = []
config = _config()
build_plugin_tools(tmp_path, config, registry=_GrantWatch(seen))
[(_, services)] = seen
assert services.media_proxy is None
assert services.web_config is not None
assert services.web_config() is config.base.tools.web
def test_a_base_config_is_read_as_itself(self, tmp_path: Path) -> None:
"""A caller holding the base ``Config`` rather than the extension is read
directly; the two lanes name the same section."""
from raven.core.plugin_stack import _host_config
base = Config(tools=ToolsConfig(media=MediaGenConfig(proxy="http://10.0.0.1:3128")))
assert _host_config(base) is base
assert _host_config(_config(tools=base.tools)).tools.media.proxy == "http://10.0.0.1:3128"
# ---------------------------------------------------------------------------
# everos plugin presence
# ---------------------------------------------------------------------------
class TestEverosPresence:
def test_the_dev_environment_carries_the_plugin(self) -> None:
assert everos_plugin_installed() is True
def test_an_uninstalled_distribution_reads_as_absent(self) -> None:
with everos_plugin_absent():
assert everos_plugin_installed() is False
def test_a_plugin_that_is_broken_inside_still_reads_as_installed(self) -> None:
"""The two failures a host must not confuse: gone, and here but broken.
A package whose own imports fail is present -- the guard says so, and the
import the caller then makes raises the plugin's own error instead of
being reported as an absence nobody can act on.
"""
with everos_plugin_broken():
assert everos_plugin_installed() is True
with pytest.raises(ImportError):
from raven_everos.health import capability_available # noqa: F401
def test_no_host_module_imports_the_plugin_at_module_level(self) -> None:
"""Every one of these imports has to stay inside a function.
A module-level one fails at ``import raven.cli``, before any guard can
run, and no amount of degrading downstream can recover from that.
"""
root = Path(raven.__file__).parent
offenders = [
f"{path.relative_to(root)}:{node.lineno}"
for path in sorted(root.rglob("*.py"))
for node in ast.parse(path.read_text(encoding="utf-8")).body
if isinstance(node, (ast.Import, ast.ImportFrom))
for name in ([node.module or ""] if isinstance(node, ast.ImportFrom) else [a.name for a in node.names])
if name.startswith("raven_everos")
]
assert offenders == []
def test_the_note_names_the_distribution_and_the_default(self) -> None:
note = everos_plugin_missing_note()
assert "everos-memory" in note
assert "memory.backend" in note
def test_build_plugin_tools_stamps_the_contributing_plugin(tmp_path):
"""[seam-1] The wake grant's namespace comes from this stamp; a plugin
that could name its own namespace could steal another's partition."""
from types import SimpleNamespace
from raven.core.plugin_stack import build_plugin_tools
class _Tool:
name = "t1"
class _Reg:
def tool_names(self):
return ["t1"]
def tool_plugin_id(self, name):
return "plug-a"
def build_tool(self, name, config, services):
return _Tool()
cfg = SimpleNamespace(
memory=SimpleNamespace(user_id="u", agent_id="a"),
plugins=SimpleNamespace(config={}),
)
tools = build_plugin_tools(tmp_path, cfg, registry=_Reg(), provider=None)
assert len(tools) == 1
assert tools[0].contributed_by == "plug-a"