Raven-1/tests/test_plugin_bootstrap.py
Zuyi Zhou 34ed138cc7
fix(plugins): a plugin that fails to load no longer takes every other plugin with it (#703)
## Summary

One plugin that failed to activate dropped every plugin.
`PluginRegistry.activate` raised on the first plugin whose factory would
not import (or whose contribution name collided), and
`build_plugin_registry` (`raven/core/plugin_stack.py`) answered by
returning an empty registry. A single broken user plugin therefore
silently removed research-flow, ppt-engine, design-engine, the memory
backend and the bundled playbook tools, with one stderr warning as the
only trace. Reproduced with two mock plugins under a scratch
`RAVEN_HOME`: `aa-broken` (factory raises `ImportError`) beside
`zz-good`; before this change the registry came back with `activated:
[]`, after it with `design-engine, everos-memory, playbook, ppt-engine,
zz-good` activated and `aa-broken` named as failed.

Changes:

- **Per-plugin activation** (`raven/plugins/registry.py`). Each admitted
plugin activates on its own. One that fails is rolled back whole (no
partial contribution in any table, and the `sys.path` entry
`_ensure_importable` appended for it is withdrawn), recorded as a
`PluginActivationFailure` (`activation_failures()`), and the plugins
after it still activate. In a name conflict the plugin activated first
(discovery order, by id) keeps the name and the other is recorded as
failed. A module `__getattr__` that raises something other than
`AttributeError` is now a recorded failure rather than an uncaught
crash.
- **Said to the user** (`plugin_stack.build_plugin_registry`,
`runtime.build_runtime`). Each failure is one notice naming the plugin,
its cause and the `plugins.disabled` escape, through the host's notifier
(`HostWiring.notify`), else stderr - the same channel the
missing-memory-backend notice already uses. When the configured memory
backend belongs to a plugin that failed, the memory notice names that
plugin and its cause instead of "no installed plugin provides it".
- **`raven plugins`** shows a `failed` status and the cause under the
table (the command called `activate` directly and previously crashed on
such a plugin).
- **Agents smoke check** (`raven/cli/agents_commands.py`) matched the
old log line "plugin activation failed", which no longer exists; it now
reads the shared `PLUGIN_FAILURE_MARKER`, and its test prints the notice
the real producer writes, so the two cannot drift.
- **Sub-agents inherit the host's opt-outs**
(`product_render.write_rendered`, `inherit_plugin_opt_outs`). A product
engine inherits `RAVEN_HOME`, so it scans the host's `<home>/plugins`
and the shared interpreter's entry points, but it read
`plugins.disabled` only from its own rendered config; a plugin disabled
on the host came back in every product. Every render now merges the
host's list after the product's own. The product's own engine plugin
never travels (`own_plugins=`, passed by all five launchers and the
scaffold): the PPT agent is `ppt-engine`, and a host turning it off for
its own agent is not a request to run the PPT agent without it. A
contract test reads every launcher's `write_rendered` call and requires
it to name its `*_PLUGIN_ID`.
- `CONTEXT.md` (Plugin Registry) and
`docs-site/docs/building-plugin{,.zh}.md` describe the new behaviour.

Not in this PR: the Web UI plugin list (`ext.list`) is discovery-only
and does not show a failed plugin; adding it is a wire change
(`ExtListResult`, openrpc, generated TS, the page) and is left for a
follow-up. The TUI, gateway and `raven agent` hosts show the notice
through their notifiers; the rpc stack (serve/acp) prints it to stderr,
which is also what the agents smoke check reads.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

**Rebased**: head 7cd000dbc on origin/main 8ee5ae0d8, without conflicts.
`git range-diff` shows every commit replayed unchanged. The branch's own
test files rerun at this head: 395 passed. Numbers elsewhere in this
description are from earlier heads.

Head b4d51aee8, rebased onto origin/main 6551f8259 after #698 and #708
merged. #698 conflicted in `raven/config/product_render.py` and
`tests/test_config_product_render.py`: both PRs added a function at the
same spot (`inherit_host_denials` there, `inherit_plugin_opt_outs` here)
and a call in `write_rendered`. The resolution keeps both functions and
all their tests; `write_rendered` now reads `host_config()` once and
passes it to both. #708 touches none of these files. At this head:

```
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q -p no:cacheprovider
  25948 passed, 109 skipped
uv run --frozen --python 3.12 --all-extras pytest -q tests/test_config_product_render.py
  44 passed
ruff check / ruff format --check / make lint-imports lint-deps lint-types / git diff --check origin/main...HEAD
check_source_language.py origin/main...HEAD / make check-commits / check-large-files
  all clean
cd ui-tui && npm run lint:rpc && npm run type-check && npm test
  144 files, 2106 passed
```

The earlier `TUI checks` failure (run 35834019105, head 5fc374035) was
`newInstancePicker.test.tsx > quick-selects by the number it printed`.
This PR changes nothing under `ui-tui/`; the test passed 5 of 5 runs
locally at this head and in the full ui-tui run above.

The numbers below are from the earlier heads and are kept for the
record.

```
uv run --frozen --python 3.12 --extra dev ruff check raven evolver agents plugins-dist tests scripts
  All checks passed
uv run --frozen --python 3.12 --extra dev ruff format --check raven evolver agents plugins-dist tests scripts
  2083 files already formatted
make lint-imports && make lint-deps && make lint-types
  Contracts: 10 kept, 0 broken / No dependency issues found / All checks passed
git diff --check origin/main...HEAD
  clean
COLORTERM=truecolor make coverage        (full suite, head before rebasing onto 7fdf296f4)
  25847 passed, 109 skipped
COVERAGE_BASE_REF=origin/main make coverage-diff
  Diff coverage: 100.00% (71/71 executable changed lines)
PYTHONPATH=. uv run --frozen --python 3.12 --extra dev python scripts/check_source_language.py origin/main...HEAD
  exit 0
make check-commits
  pass
git merge-tree --write-tree HEAD origin/main
  clean
```

After rebasing onto 7fdf296f4 (its two new commits touch none of these
files), the touched suites at this head:

```
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q tests/test_plugin_*.py tests/test_core_plugin_stack.py tests/test_cli_plugin_commands.py tests/test_cli_agents_commands.py tests/test_cli_tui_commands.py tests/test_config_product_render.py tests/test_agents_*launcher*.py tests/test_everos_plugin_discovery.py tests/test_release_plugin_list.py tests/test_cli_gateway_commands.py
  727 passed
uv run --frozen --python 3.12 --all-extras pytest -q tests/test_subagent_vendored_agents.py tests/test_rpc_subagents.py tests/test_subagent_acp.py
  452 passed
```

Revert-to-red, one production file restored to origin/main at a time
with the tests kept:

```
raven/plugins/registry.py        -> 2 errors (plugin_stack imports PluginActivationFailure)
raven/core/plugin_stack.py       -> 6 failed
raven/core/runtime.py            -> 1 failed (build_runtime hands the host's notifier to the registry)
raven/config/product_render.py   -> 42 failed
raven/cli/agents_commands.py     -> 1 failed (smoke reds a handshake whose plugin did not load)
raven/cli/plugin_commands.py     -> 1 failed (failed row and escaped cause)
agents/raven-ppt/run.py          -> 2 failed (engine exemption, launcher contract)
```

Ten existing tests pinned "activation raises" (`test_plugin_registry`,
`test_plugin_bootstrap`, and the conflict tests for tools, hooks, tool
gates and session observers). They now assert the new contract instead:
the later plugin is recorded with the same reason text, the earlier one
keeps the name, and nothing of the loser is registered. Two stubs in
`test_cli_tui_commands.py` took `build_plugin_registry(cfg)` and now
accept the `notify` keyword.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- A plugin conflict used to disable all plugins; it now keeps the plugin
that activates first and skips the other with a notice. Two plugins
contributing the same name therefore run with one of them instead of
none.
- A product sub-agent now starts with the host's `plugins.disabled`
merged into its config (its own engine excepted). A plugin that was
disabled on the host but relied on inside a product stops loading there;
the remedy is to enable it on the host.
- `PluginRegistry.activate` no longer raises `PluginConflictError` /
`PluginFactoryImportError`; the only in-tree callers were
`assemble_plugin_registry` and `raven plugins`. The exception classes
stay exported.
- Security: the change only narrows what loads. A failed plugin's
directory leaves sys.path, and the inherited opt-outs can only remove
plugins from a product, never add one.
- Rollback: revert the squash commit.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-09-23 22:41:21 +08:00

332 lines
10 KiB
Python

"""PG-3 — end-to-end bootstrap + build_memory_backend integration."""
from __future__ import annotations
import sys
import textwrap
import types
from pathlib import Path
import pytest
from raven.plugins import (
PluginNotFoundError,
PluginRegistry,
ServiceLocator,
assemble_plugin_registry,
)
def _write_manifest(
root: Path,
plugin_id: str,
*,
factory_ref: str,
backend_name: str = "everos",
bundled: bool = True,
) -> None:
sub = root / plugin_id
sub.mkdir(parents=True, exist_ok=True)
flags = f"bundled = {str(bundled).lower()}\n"
(sub / "raven-plugin.toml").write_text(
textwrap.dedent(f"""
[plugin]
id = "{plugin_id}"
version = "0.1.0"
{flags}
[[plugin.contributes.memory_backends]]
name = "{backend_name}"
factory = "{factory_ref}"
"""),
encoding="utf-8",
)
def _install_test_module(name: str, attrs: dict[str, object]) -> None:
mod = types.ModuleType(name)
for k, v in attrs.items():
setattr(mod, k, v)
sys.modules[name] = mod
@pytest.fixture(autouse=True)
def _cleanup_modules():
snapshot = set(sys.modules)
yield
extras = set(sys.modules) - snapshot
for k in extras:
sys.modules.pop(k, None)
@pytest.fixture(autouse=True)
def _restore_syspath():
snapshot = list(sys.path)
yield
sys.path[:] = snapshot
def _write_real_plugin(
root: Path,
plugin_id: str,
*,
pkg: str,
backend_name: str | None = None,
tool_name: str | None = None,
) -> None:
"""Write a genuine on-disk plugin: manifest + an importable package
whose factory module is *not* pre-seeded into ``sys.modules``.
Loading it therefore only succeeds if activation put ``<root>/<id>/``
on ``sys.path`` — the exact behaviour under test. Contrast with
``_install_test_module``, which sidesteps import resolution entirely.
"""
sub = root / plugin_id
(sub / pkg).mkdir(parents=True, exist_ok=True)
(sub / pkg / "__init__.py").write_text("", encoding="utf-8")
(sub / pkg / "factories.py").write_text(
textwrap.dedent("""
def make_backend(ctx):
return {"kind": "backend", **ctx.config}
def make_tool(ctx):
return {"kind": "tool"}
"""),
encoding="utf-8",
)
blocks = ""
if backend_name is not None:
blocks += textwrap.dedent(f"""
[[plugin.contributes.memory_backends]]
name = "{backend_name}"
factory = "{pkg}.factories:make_backend"
""")
if tool_name is not None:
blocks += textwrap.dedent(f"""
[[plugin.contributes.tools]]
name = "{tool_name}"
factory = "{pkg}.factories:make_tool"
""")
(sub / "raven-plugin.toml").write_text(
textwrap.dedent(f"""
[plugin]
id = "{plugin_id}"
version = "0.1.0"
""")
+ blocks,
encoding="utf-8",
)
# ---------------------------------------------------------------------------
# End-to-end: discover → activate → build
# ---------------------------------------------------------------------------
class TestEndToEnd:
def test_discover_activate_build(self, tmp_path: Path) -> None:
"""A single bundled plugin reaches the constructed backend."""
def fake_factory(ctx):
return {"workspace": str(ctx.services.workspace), **ctx.config}
_install_test_module("_test_eos", {"make_backend": fake_factory})
bundled = tmp_path / "bundled"
_write_manifest(bundled, "everos-memory", factory_ref="_test_eos:make_backend")
registry = assemble_plugin_registry(
bundled_dir=bundled,
entry_points_group=None,
)
assert registry.activated_ids() == ["everos-memory"]
ws = tmp_path / "chanwork"
ws.mkdir()
backend = registry.build_memory_backend(
"everos",
config={"mode": "embedded"},
services=ServiceLocator(workspace=ws, user_id="default", agent_id="default"),
)
assert backend == {"workspace": str(ws), "mode": "embedded"}
def test_unknown_backend_name(self, tmp_path: Path) -> None:
registry = assemble_plugin_registry(
bundled_dir=tmp_path,
entry_points_group=None,
)
with pytest.raises(PluginNotFoundError):
registry.build_memory_backend(
"everos",
config={},
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
)
def test_disabled_plugin_does_not_register(self, tmp_path: Path) -> None:
def fake_factory(ctx):
return "should-not-be-built"
_install_test_module("_test_disabled", {"mk": fake_factory})
bundled = tmp_path / "bundled"
_write_manifest(bundled, "myplug", factory_ref="_test_disabled:mk")
registry = assemble_plugin_registry(
bundled_dir=bundled,
entry_points_group=None,
disabled=frozenset({"myplug"}),
)
assert registry.activated_ids() == []
assert registry.memory_backend_names() == []
# ---------------------------------------------------------------------------
# Cross-source: bundled wins over user-level for same id
# ---------------------------------------------------------------------------
class TestCrossSource:
def test_bundled_shadows_user(self, tmp_path: Path) -> None:
def bundled_factory(ctx):
return "BUNDLED"
def user_factory(ctx):
return "USER"
_install_test_module("_bp", {"mk": bundled_factory})
_install_test_module("_up", {"mk": user_factory})
bundled = tmp_path / "bundled"
user = tmp_path / "user"
_write_manifest(bundled, "everos-memory", factory_ref="_bp:mk")
_write_manifest(user, "everos-memory", factory_ref="_up:mk")
registry = assemble_plugin_registry(
bundled_dir=bundled,
user_dir=user,
entry_points_group=None,
)
# Bundled wins — the user copy is shadowed and never imported.
result = registry.build_memory_backend(
"everos",
config={},
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
)
assert result == "BUNDLED"
# ---------------------------------------------------------------------------
# Conflict: two activated plugins same backend name
# ---------------------------------------------------------------------------
class TestConflict:
def test_two_plugins_same_backend_name_keeps_the_first(self, tmp_path: Path) -> None:
def fa(ctx):
return "a"
def fb(ctx):
return "b"
_install_test_module("_pa", {"mk": fa})
_install_test_module("_pb", {"mk": fb})
bundled = tmp_path / "bundled"
_write_manifest(
bundled,
"plug-a",
factory_ref="_pa:mk",
backend_name="everos",
)
_write_manifest(
bundled,
"plug-b",
factory_ref="_pb:mk",
backend_name="everos",
)
registry = assemble_plugin_registry(
bundled_dir=bundled,
entry_points_group=None,
)
assert registry.activated_ids() == ["plug-a"]
assert registry.get_memory_backend_factory("everos") is fa
[failure] = registry.activation_failures()
assert failure.plugin_id == "plug-b"
assert "everos" in failure.reason
# ---------------------------------------------------------------------------
# Empty bootstrap
# ---------------------------------------------------------------------------
class TestEmpty:
def test_no_sources_returns_empty_registry(self) -> None:
registry = assemble_plugin_registry(entry_points_group=None)
assert isinstance(registry, PluginRegistry)
assert registry.activated_ids() == []
# ---------------------------------------------------------------------------
# User/project-dir plugins ship their factory package in the plugin dir;
# activation must put that dir on sys.path so the factory imports.
# ---------------------------------------------------------------------------
class TestUserDirImport:
def test_user_dir_backend_loads_from_ondisk_package(self, tmp_path: Path) -> None:
user = tmp_path / "user"
_write_real_plugin(
user,
"ud-backend",
pkg="udpkg_backend",
backend_name="ud_mem",
)
registry = assemble_plugin_registry(user_dir=user, entry_points_group=None)
assert registry.activated_ids() == ["ud-backend"]
backend = registry.build_memory_backend(
"ud_mem",
config={"mode": "embedded"},
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
)
assert backend == {"kind": "backend", "mode": "embedded"}
def test_project_dir_tool_loads_from_ondisk_package(self, tmp_path: Path) -> None:
project = tmp_path / "project"
_write_real_plugin(
project,
"pd-tool",
pkg="pdpkg_tool",
tool_name="pd_tool",
)
registry = assemble_plugin_registry(project_dir=project, entry_points_group=None)
assert registry.activated_ids() == ["pd-tool"]
tool = registry.build_tool(
"pd_tool",
config={},
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
)
assert tool == {"kind": "tool"}
def test_user_dir_mixed_backend_and_tool_load(self, tmp_path: Path) -> None:
user = tmp_path / "user"
_write_real_plugin(
user,
"ud-mixed",
pkg="udpkg_mixed",
backend_name="mixed_mem",
tool_name="mixed_tool",
)
registry = assemble_plugin_registry(user_dir=user, entry_points_group=None)
assert registry.activated_ids() == ["ud-mixed"]
backend = registry.build_memory_backend(
"mixed_mem",
config={},
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
)
tool = registry.build_tool(
"mixed_tool",
config={},
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
)
assert backend == {"kind": "backend"}
assert tool == {"kind": "tool"}