Raven-1/tests/test_plugin_registry.py
Zuyi Zhou 34ed138cc7
fix(plugins): a plugin that fails to load no longer takes every other plugin with it (#703)
## Summary

One plugin that failed to activate dropped every plugin.
`PluginRegistry.activate` raised on the first plugin whose factory would
not import (or whose contribution name collided), and
`build_plugin_registry` (`raven/core/plugin_stack.py`) answered by
returning an empty registry. A single broken user plugin therefore
silently removed research-flow, ppt-engine, design-engine, the memory
backend and the bundled playbook tools, with one stderr warning as the
only trace. Reproduced with two mock plugins under a scratch
`RAVEN_HOME`: `aa-broken` (factory raises `ImportError`) beside
`zz-good`; before this change the registry came back with `activated:
[]`, after it with `design-engine, everos-memory, playbook, ppt-engine,
zz-good` activated and `aa-broken` named as failed.

Changes:

- **Per-plugin activation** (`raven/plugins/registry.py`). Each admitted
plugin activates on its own. One that fails is rolled back whole (no
partial contribution in any table, and the `sys.path` entry
`_ensure_importable` appended for it is withdrawn), recorded as a
`PluginActivationFailure` (`activation_failures()`), and the plugins
after it still activate. In a name conflict the plugin activated first
(discovery order, by id) keeps the name and the other is recorded as
failed. A module `__getattr__` that raises something other than
`AttributeError` is now a recorded failure rather than an uncaught
crash.
- **Said to the user** (`plugin_stack.build_plugin_registry`,
`runtime.build_runtime`). Each failure is one notice naming the plugin,
its cause and the `plugins.disabled` escape, through the host's notifier
(`HostWiring.notify`), else stderr - the same channel the
missing-memory-backend notice already uses. When the configured memory
backend belongs to a plugin that failed, the memory notice names that
plugin and its cause instead of "no installed plugin provides it".
- **`raven plugins`** shows a `failed` status and the cause under the
table (the command called `activate` directly and previously crashed on
such a plugin).
- **Agents smoke check** (`raven/cli/agents_commands.py`) matched the
old log line "plugin activation failed", which no longer exists; it now
reads the shared `PLUGIN_FAILURE_MARKER`, and its test prints the notice
the real producer writes, so the two cannot drift.
- **Sub-agents inherit the host's opt-outs**
(`product_render.write_rendered`, `inherit_plugin_opt_outs`). A product
engine inherits `RAVEN_HOME`, so it scans the host's `<home>/plugins`
and the shared interpreter's entry points, but it read
`plugins.disabled` only from its own rendered config; a plugin disabled
on the host came back in every product. Every render now merges the
host's list after the product's own. The product's own engine plugin
never travels (`own_plugins=`, passed by all five launchers and the
scaffold): the PPT agent is `ppt-engine`, and a host turning it off for
its own agent is not a request to run the PPT agent without it. A
contract test reads every launcher's `write_rendered` call and requires
it to name its `*_PLUGIN_ID`.
- `CONTEXT.md` (Plugin Registry) and
`docs-site/docs/building-plugin{,.zh}.md` describe the new behaviour.

Not in this PR: the Web UI plugin list (`ext.list`) is discovery-only
and does not show a failed plugin; adding it is a wire change
(`ExtListResult`, openrpc, generated TS, the page) and is left for a
follow-up. The TUI, gateway and `raven agent` hosts show the notice
through their notifiers; the rpc stack (serve/acp) prints it to stderr,
which is also what the agents smoke check reads.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

**Rebased**: head 7cd000dbc on origin/main 8ee5ae0d8, without conflicts.
`git range-diff` shows every commit replayed unchanged. The branch's own
test files rerun at this head: 395 passed. Numbers elsewhere in this
description are from earlier heads.

Head b4d51aee8, rebased onto origin/main 6551f8259 after #698 and #708
merged. #698 conflicted in `raven/config/product_render.py` and
`tests/test_config_product_render.py`: both PRs added a function at the
same spot (`inherit_host_denials` there, `inherit_plugin_opt_outs` here)
and a call in `write_rendered`. The resolution keeps both functions and
all their tests; `write_rendered` now reads `host_config()` once and
passes it to both. #708 touches none of these files. At this head:

```
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q -p no:cacheprovider
  25948 passed, 109 skipped
uv run --frozen --python 3.12 --all-extras pytest -q tests/test_config_product_render.py
  44 passed
ruff check / ruff format --check / make lint-imports lint-deps lint-types / git diff --check origin/main...HEAD
check_source_language.py origin/main...HEAD / make check-commits / check-large-files
  all clean
cd ui-tui && npm run lint:rpc && npm run type-check && npm test
  144 files, 2106 passed
```

The earlier `TUI checks` failure (run 35834019105, head 5fc374035) was
`newInstancePicker.test.tsx > quick-selects by the number it printed`.
This PR changes nothing under `ui-tui/`; the test passed 5 of 5 runs
locally at this head and in the full ui-tui run above.

The numbers below are from the earlier heads and are kept for the
record.

```
uv run --frozen --python 3.12 --extra dev ruff check raven evolver agents plugins-dist tests scripts
  All checks passed
uv run --frozen --python 3.12 --extra dev ruff format --check raven evolver agents plugins-dist tests scripts
  2083 files already formatted
make lint-imports && make lint-deps && make lint-types
  Contracts: 10 kept, 0 broken / No dependency issues found / All checks passed
git diff --check origin/main...HEAD
  clean
COLORTERM=truecolor make coverage        (full suite, head before rebasing onto 7fdf296f4)
  25847 passed, 109 skipped
COVERAGE_BASE_REF=origin/main make coverage-diff
  Diff coverage: 100.00% (71/71 executable changed lines)
PYTHONPATH=. uv run --frozen --python 3.12 --extra dev python scripts/check_source_language.py origin/main...HEAD
  exit 0
make check-commits
  pass
git merge-tree --write-tree HEAD origin/main
  clean
```

After rebasing onto 7fdf296f4 (its two new commits touch none of these
files), the touched suites at this head:

```
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q tests/test_plugin_*.py tests/test_core_plugin_stack.py tests/test_cli_plugin_commands.py tests/test_cli_agents_commands.py tests/test_cli_tui_commands.py tests/test_config_product_render.py tests/test_agents_*launcher*.py tests/test_everos_plugin_discovery.py tests/test_release_plugin_list.py tests/test_cli_gateway_commands.py
  727 passed
uv run --frozen --python 3.12 --all-extras pytest -q tests/test_subagent_vendored_agents.py tests/test_rpc_subagents.py tests/test_subagent_acp.py
  452 passed
```

Revert-to-red, one production file restored to origin/main at a time
with the tests kept:

```
raven/plugins/registry.py        -> 2 errors (plugin_stack imports PluginActivationFailure)
raven/core/plugin_stack.py       -> 6 failed
raven/core/runtime.py            -> 1 failed (build_runtime hands the host's notifier to the registry)
raven/config/product_render.py   -> 42 failed
raven/cli/agents_commands.py     -> 1 failed (smoke reds a handshake whose plugin did not load)
raven/cli/plugin_commands.py     -> 1 failed (failed row and escaped cause)
agents/raven-ppt/run.py          -> 2 failed (engine exemption, launcher contract)
```

Ten existing tests pinned "activation raises" (`test_plugin_registry`,
`test_plugin_bootstrap`, and the conflict tests for tools, hooks, tool
gates and session observers). They now assert the new contract instead:
the later plugin is recorded with the same reason text, the earlier one
keeps the name, and nothing of the loser is registered. Two stubs in
`test_cli_tui_commands.py` took `build_plugin_registry(cfg)` and now
accept the `notify` keyword.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- A plugin conflict used to disable all plugins; it now keeps the plugin
that activates first and skips the other with a notice. Two plugins
contributing the same name therefore run with one of them instead of
none.
- A product sub-agent now starts with the host's `plugins.disabled`
merged into its config (its own engine excepted). A plugin that was
disabled on the host but relied on inside a product stops loading there;
the remedy is to enable it on the host.
- `PluginRegistry.activate` no longer raises `PluginConflictError` /
`PluginFactoryImportError`; the only in-tree callers were
`assemble_plugin_registry` and `raven plugins`. The exception classes
stay exported.
- Security: the change only narrows what loads. A failed plugin's
directory leaves sys.path, and the inherited opt-outs can only remove
plugins from a product, never add one.
- Rollback: revert the squash commit.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-09-23 22:41:21 +08:00

468 lines
16 KiB
Python

"""PG-2 — PluginRegistry activation + factory resolution + conflict detection."""
from __future__ import annotations
import sys
import types
from pathlib import Path
import pytest
from raven.plugins import (
Contributes,
DiscoveredPlugin,
ManifestOrigin,
MemoryBackendContribution,
OnboardContribution,
PluginContext,
PluginManifest,
PluginNotFoundError,
PluginRegistry,
ServiceLocator,
)
# ---------------------------------------------------------------------------
# In-memory test plugin modules
# ---------------------------------------------------------------------------
def _install_test_module(name: str, attrs: dict[str, object]) -> None:
"""Inject a fake module into ``sys.modules`` for factory resolution
tests. The module is removed in the per-test cleanup fixture."""
mod = types.ModuleType(name)
for k, v in attrs.items():
setattr(mod, k, v)
sys.modules[name] = mod
@pytest.fixture(autouse=True)
def _cleanup_modules():
"""Remove every test module we inject so cross-test pollution is
impossible."""
snapshot = set(sys.modules)
yield
extras = set(sys.modules) - snapshot
for k in extras:
sys.modules.pop(k, None)
def _make_discovered(
plugin_id: str,
*,
backends: list[tuple[str, str]] | None = None,
onboard: list[tuple[str, str]] | None = None,
bundled: bool = False,
) -> DiscoveredPlugin:
mf = PluginManifest(
id=plugin_id,
version="0.1.0",
bundled=bundled,
contributes=Contributes(
memory_backends=[MemoryBackendContribution(name=n, factory=f) for n, f in (backends or [])],
onboard=[OnboardContribution(name=n, factory=f) for n, f in (onboard or [])],
),
)
return DiscoveredPlugin(
manifest=mf,
source=ManifestOrigin.BUNDLED if bundled else ManifestOrigin.USER,
location=None,
)
# ---------------------------------------------------------------------------
# Successful activation
# ---------------------------------------------------------------------------
class TestActivation:
def test_activates_single_plugin_with_one_backend(self) -> None:
def fake_factory(ctx):
return "backend-instance"
_install_test_module("_test_plugin_a", {"make_backend": fake_factory})
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"alpha",
backends=[
("everos", "_test_plugin_a:make_backend"),
],
),
]
)
assert reg.activated_ids() == ["alpha"]
assert reg.memory_backend_names() == ["everos"]
factory = reg.get_memory_backend_factory("everos")
assert factory is fake_factory
def test_factory_invoked_with_context(self, tmp_path: Path) -> None:
captured = {}
def fake_factory(ctx: PluginContext):
captured["ctx"] = ctx
return ("backend", ctx.config)
_install_test_module("_test_plugin_b", {"make_backend": fake_factory})
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"plug",
backends=[
("everos", "_test_plugin_b:make_backend"),
],
),
]
)
ctx = PluginContext(
config={"mode": "embedded"},
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
)
result = reg.get_memory_backend_factory("everos")(ctx)
assert result == ("backend", {"mode": "embedded"})
assert captured["ctx"] is ctx
# ---------------------------------------------------------------------------
# Opt-out / opt-in gating
# ---------------------------------------------------------------------------
class TestEnablement:
def test_disabled_plugin_is_skipped(self) -> None:
def fake_factory(ctx):
return "x"
_install_test_module("_test_plugin_c", {"make_backend": fake_factory})
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"plug",
backends=[
("everos", "_test_plugin_c:make_backend"),
],
),
],
disabled=frozenset({"plug"}),
)
assert reg.activated_ids() == []
assert reg.memory_backend_names() == []
def test_every_discovered_plugin_not_disabled_is_activated(self) -> None:
def fake_factory(ctx):
return "x"
_install_test_module("_test_plugin_d", {"make_backend": fake_factory})
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"plug",
backends=[
("everos", "_test_plugin_d:make_backend"),
],
),
]
)
assert reg.activated_ids() == ["plug"]
# ---------------------------------------------------------------------------
# Conflicts
# ---------------------------------------------------------------------------
class TestConflicts:
def test_two_plugins_contribute_same_backend_name(self) -> None:
def fake_a(ctx):
return "a"
def fake_b(ctx):
return "b"
_install_test_module("_test_plugin_e", {"make_backend": fake_a})
_install_test_module("_test_plugin_f", {"make_backend": fake_b})
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"alpha",
backends=[
("everos", "_test_plugin_e:make_backend"),
],
),
_make_discovered(
"beta",
backends=[
("everos", "_test_plugin_f:make_backend"),
],
),
]
)
assert reg.activated_ids() == ["alpha"]
assert reg.get_memory_backend_factory("everos") is fake_a
[failure] = reg.activation_failures()
assert failure.plugin_id == "beta"
assert "memory_backend 'everos' contributed by both 'alpha' and 'beta'" in failure.reason
# ---------------------------------------------------------------------------
# One plugin's failure is its own
# ---------------------------------------------------------------------------
class TestFailureIsolation:
def test_a_broken_plugin_leaves_the_others_activated(self) -> None:
_install_test_module("_test_iso_ok", {"make_backend": lambda ctx: "ok"})
reg = PluginRegistry()
reg.activate(
[
_make_discovered("alpha", backends=[("a", "_test_iso_ok:make_backend")]),
_make_discovered("broken", backends=[("b", "_nonexistent_iso_zzz:make_backend")]),
_make_discovered("gamma", backends=[("c", "_test_iso_ok:make_backend")]),
]
)
assert reg.activated_ids() == ["alpha", "gamma"]
assert reg.memory_backend_names() == ["a", "c"]
assert [f.plugin_id for f in reg.activation_failures()] == ["broken"]
def test_a_plugin_that_fails_halfway_registers_nothing(self) -> None:
_install_test_module("_test_iso_half", {"make_backend": lambda ctx: "half"})
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"half",
backends=[("half", "_test_iso_half:make_backend")],
onboard=[("half", "_test_iso_half:missing_step")],
),
]
)
assert reg.activated_ids() == []
assert reg.manifest_for("half") is None
assert reg.memory_backend_names() == []
assert reg.onboard_names() == []
[failure] = reg.activation_failures()
assert failure.manifest.id == "half"
def test_an_attribute_lookup_that_raises_is_a_failure_not_a_crash(self) -> None:
mod = types.ModuleType("_test_iso_getattr")
def boom(name: str):
raise RuntimeError(f"lazy import of {name} failed")
mod.__getattr__ = boom # type: ignore[attr-defined]
sys.modules["_test_iso_getattr"] = mod
_install_test_module("_test_iso_ok2", {"make_backend": lambda ctx: "ok"})
reg = PluginRegistry()
reg.activate(
[
_make_discovered("lazy", backends=[("lazy", "_test_iso_getattr:make_backend")]),
_make_discovered("steady", backends=[("steady", "_test_iso_ok2:make_backend")]),
]
)
assert reg.activated_ids() == ["steady"]
[failure] = reg.activation_failures()
assert "lazy import of make_backend failed" in failure.reason
def test_a_failed_file_plugin_takes_its_directory_back_off_sys_path(self, tmp_path: Path) -> None:
broken_dir = tmp_path / "broken"
(broken_dir / "_iso_broken_pkg").mkdir(parents=True)
(broken_dir / "_iso_broken_pkg" / "__init__.py").write_text("raise ImportError('boom')\n")
good_dir = tmp_path / "good"
(good_dir / "_iso_good_pkg").mkdir(parents=True)
(good_dir / "_iso_good_pkg" / "__init__.py").write_text("def make(ctx):\n return 'good'\n")
def file_plugin(plugin_id: str, root: Path, ref: str) -> DiscoveredPlugin:
d = _make_discovered(plugin_id, backends=[(plugin_id, ref)])
return DiscoveredPlugin(
manifest=d.manifest, source=ManifestOrigin.USER, location=root / "raven-plugin.toml"
)
before = list(sys.path)
try:
reg = PluginRegistry()
reg.activate(
[
file_plugin("broken", broken_dir, "_iso_broken_pkg:make"),
file_plugin("good", good_dir, "_iso_good_pkg:make"),
]
)
assert reg.activated_ids() == ["good"]
assert str(broken_dir) not in sys.path
assert str(good_dir) in sys.path
finally:
sys.path[:] = before
# ---------------------------------------------------------------------------
# Factory resolution errors
# ---------------------------------------------------------------------------
class TestFactoryResolutionErrors:
def test_missing_module(self) -> None:
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"plug",
backends=[
("everos", "_nonexistent_module_zzz:make_backend"),
],
),
]
)
assert reg.activated_ids() == []
assert reg.memory_backend_names() == []
[failure] = reg.activation_failures()
assert failure.plugin_id == "plug"
assert "importing '_nonexistent_module_zzz' failed" in failure.reason
def test_module_lacks_attribute(self) -> None:
_install_test_module("_test_plugin_g", {"other_thing": object()})
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"plug",
backends=[
("everos", "_test_plugin_g:make_backend"),
],
),
]
)
assert reg.activated_ids() == []
assert reg.memory_backend_names() == []
[failure] = reg.activation_failures()
assert failure.plugin_id == "plug"
assert "'_test_plugin_g' has no attribute 'make_backend'" in failure.reason
def test_attribute_not_callable(self) -> None:
_install_test_module("_test_plugin_h", {"make_backend": 42})
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"plug",
backends=[
("everos", "_test_plugin_h:make_backend"),
],
),
]
)
assert reg.activated_ids() == []
assert reg.memory_backend_names() == []
[failure] = reg.activation_failures()
assert failure.plugin_id == "plug"
assert "resolved to a non-callable int" in failure.reason
# ---------------------------------------------------------------------------
# Lookups
# ---------------------------------------------------------------------------
class TestLookup:
def test_unknown_name_raises_not_found(self) -> None:
reg = PluginRegistry()
with pytest.raises(PluginNotFoundError, match="no memory_backend named 'x'"):
reg.get_memory_backend_factory("x")
def test_manifest_for_returns_none_when_missing(self) -> None:
reg = PluginRegistry()
assert reg.manifest_for("nope") is None
def test_manifest_for_after_activation(self) -> None:
def fake(ctx):
return "x"
_install_test_module("_test_plugin_i", {"make_backend": fake})
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"plug",
backends=[
("everos", "_test_plugin_i:make_backend"),
],
),
]
)
mf = reg.manifest_for("plug")
assert mf is not None
assert mf.id == "plug"
def test_build_onboard_step_calls_the_factory(tmp_path: Path) -> None:
_install_test_module(
"_test_onboard",
{
"make_onboard_step": lambda ctx: ("step", ctx.config),
"make_backend": lambda ctx: "backend",
},
)
reg = PluginRegistry()
reg.activate(
[
_make_discovered(
"plug",
backends=[("plug", "_test_onboard:make_backend")],
onboard=[("plug", "_test_onboard:make_onboard_step")],
)
]
)
assert reg.onboard_names() == ["plug"]
assert reg.onboard_plugin_id("plug") == "plug"
step = reg.build_onboard_step(
"plug",
config={"k": 1},
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
)
assert step == ("step", {"k": 1})
def test_build_onboard_step_rejects_an_unknown_name(tmp_path: Path) -> None:
reg = PluginRegistry()
with pytest.raises(PluginNotFoundError):
reg.build_onboard_step(
"nobody",
config={},
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
)
def test_two_plugins_contribute_same_onboard_name() -> None:
"""mrbot's cross-owner scenario: plugin A owns backend 'shared'; plugin
B, to satisfy the manifest rule that its onboard name matches its own
backend name, must also contribute a backend named 'shared'. That
backend collides with A's before either onboard step registers, so
activation refuses -- B can never receive A's config slice because B
never activates at all."""
_install_test_module("_test_onboard_a", {"make_backend": lambda ctx: "a"})
_install_test_module(
"_test_onboard_b",
{"make_backend": lambda ctx: "b", "make_onboard_step": lambda ctx: "b"},
)
reg = PluginRegistry()
reg.activate(
[
_make_discovered("alpha", backends=[("shared", "_test_onboard_a:make_backend")]),
_make_discovered(
"beta",
backends=[("shared", "_test_onboard_b:make_backend")],
onboard=[("shared", "_test_onboard_b:make_onboard_step")],
),
]
)
assert reg.activated_ids() == ["alpha"]
assert reg.onboard_names() == []
[failure] = reg.activation_failures()
assert failure.plugin_id == "beta"
assert "memory_backend 'shared'" in failure.reason