## Summary
One plugin that failed to activate dropped every plugin.
`PluginRegistry.activate` raised on the first plugin whose factory would
not import (or whose contribution name collided), and
`build_plugin_registry` (`raven/core/plugin_stack.py`) answered by
returning an empty registry. A single broken user plugin therefore
silently removed research-flow, ppt-engine, design-engine, the memory
backend and the bundled playbook tools, with one stderr warning as the
only trace. Reproduced with two mock plugins under a scratch
`RAVEN_HOME`: `aa-broken` (factory raises `ImportError`) beside
`zz-good`; before this change the registry came back with `activated:
[]`, after it with `design-engine, everos-memory, playbook, ppt-engine,
zz-good` activated and `aa-broken` named as failed.
Changes:
- **Per-plugin activation** (`raven/plugins/registry.py`). Each admitted
plugin activates on its own. One that fails is rolled back whole (no
partial contribution in any table, and the `sys.path` entry
`_ensure_importable` appended for it is withdrawn), recorded as a
`PluginActivationFailure` (`activation_failures()`), and the plugins
after it still activate. In a name conflict the plugin activated first
(discovery order, by id) keeps the name and the other is recorded as
failed. A module `__getattr__` that raises something other than
`AttributeError` is now a recorded failure rather than an uncaught
crash.
- **Said to the user** (`plugin_stack.build_plugin_registry`,
`runtime.build_runtime`). Each failure is one notice naming the plugin,
its cause and the `plugins.disabled` escape, through the host's notifier
(`HostWiring.notify`), else stderr - the same channel the
missing-memory-backend notice already uses. When the configured memory
backend belongs to a plugin that failed, the memory notice names that
plugin and its cause instead of "no installed plugin provides it".
- **`raven plugins`** shows a `failed` status and the cause under the
table (the command called `activate` directly and previously crashed on
such a plugin).
- **Agents smoke check** (`raven/cli/agents_commands.py`) matched the
old log line "plugin activation failed", which no longer exists; it now
reads the shared `PLUGIN_FAILURE_MARKER`, and its test prints the notice
the real producer writes, so the two cannot drift.
- **Sub-agents inherit the host's opt-outs**
(`product_render.write_rendered`, `inherit_plugin_opt_outs`). A product
engine inherits `RAVEN_HOME`, so it scans the host's `<home>/plugins`
and the shared interpreter's entry points, but it read
`plugins.disabled` only from its own rendered config; a plugin disabled
on the host came back in every product. Every render now merges the
host's list after the product's own. The product's own engine plugin
never travels (`own_plugins=`, passed by all five launchers and the
scaffold): the PPT agent is `ppt-engine`, and a host turning it off for
its own agent is not a request to run the PPT agent without it. A
contract test reads every launcher's `write_rendered` call and requires
it to name its `*_PLUGIN_ID`.
- `CONTEXT.md` (Plugin Registry) and
`docs-site/docs/building-plugin{,.zh}.md` describe the new behaviour.
Not in this PR: the Web UI plugin list (`ext.list`) is discovery-only
and does not show a failed plugin; adding it is a wire change
(`ExtListResult`, openrpc, generated TS, the page) and is left for a
follow-up. The TUI, gateway and `raven agent` hosts show the notice
through their notifiers; the rpc stack (serve/acp) prints it to stderr,
which is also what the agents smoke check reads.
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
**Rebased**: head 7cd000dbc on origin/main 8ee5ae0d8, without conflicts.
`git range-diff` shows every commit replayed unchanged. The branch's own
test files rerun at this head: 395 passed. Numbers elsewhere in this
description are from earlier heads.
Head b4d51aee8, rebased onto origin/main 6551f8259 after #698 and #708
merged. #698 conflicted in `raven/config/product_render.py` and
`tests/test_config_product_render.py`: both PRs added a function at the
same spot (`inherit_host_denials` there, `inherit_plugin_opt_outs` here)
and a call in `write_rendered`. The resolution keeps both functions and
all their tests; `write_rendered` now reads `host_config()` once and
passes it to both. #708 touches none of these files. At this head:
```
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q -p no:cacheprovider
25948 passed, 109 skipped
uv run --frozen --python 3.12 --all-extras pytest -q tests/test_config_product_render.py
44 passed
ruff check / ruff format --check / make lint-imports lint-deps lint-types / git diff --check origin/main...HEAD
check_source_language.py origin/main...HEAD / make check-commits / check-large-files
all clean
cd ui-tui && npm run lint:rpc && npm run type-check && npm test
144 files, 2106 passed
```
The earlier `TUI checks` failure (run 35834019105, head 5fc374035) was
`newInstancePicker.test.tsx > quick-selects by the number it printed`.
This PR changes nothing under `ui-tui/`; the test passed 5 of 5 runs
locally at this head and in the full ui-tui run above.
The numbers below are from the earlier heads and are kept for the
record.
```
uv run --frozen --python 3.12 --extra dev ruff check raven evolver agents plugins-dist tests scripts
All checks passed
uv run --frozen --python 3.12 --extra dev ruff format --check raven evolver agents plugins-dist tests scripts
2083 files already formatted
make lint-imports && make lint-deps && make lint-types
Contracts: 10 kept, 0 broken / No dependency issues found / All checks passed
git diff --check origin/main...HEAD
clean
COLORTERM=truecolor make coverage (full suite, head before rebasing onto 7fdf296f4)
25847 passed, 109 skipped
COVERAGE_BASE_REF=origin/main make coverage-diff
Diff coverage: 100.00% (71/71 executable changed lines)
PYTHONPATH=. uv run --frozen --python 3.12 --extra dev python scripts/check_source_language.py origin/main...HEAD
exit 0
make check-commits
pass
git merge-tree --write-tree HEAD origin/main
clean
```
After rebasing onto 7fdf296f4 (its two new commits touch none of these
files), the touched suites at this head:
```
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q tests/test_plugin_*.py tests/test_core_plugin_stack.py tests/test_cli_plugin_commands.py tests/test_cli_agents_commands.py tests/test_cli_tui_commands.py tests/test_config_product_render.py tests/test_agents_*launcher*.py tests/test_everos_plugin_discovery.py tests/test_release_plugin_list.py tests/test_cli_gateway_commands.py
727 passed
uv run --frozen --python 3.12 --all-extras pytest -q tests/test_subagent_vendored_agents.py tests/test_rpc_subagents.py tests/test_subagent_acp.py
452 passed
```
Revert-to-red, one production file restored to origin/main at a time
with the tests kept:
```
raven/plugins/registry.py -> 2 errors (plugin_stack imports PluginActivationFailure)
raven/core/plugin_stack.py -> 6 failed
raven/core/runtime.py -> 1 failed (build_runtime hands the host's notifier to the registry)
raven/config/product_render.py -> 42 failed
raven/cli/agents_commands.py -> 1 failed (smoke reds a handshake whose plugin did not load)
raven/cli/plugin_commands.py -> 1 failed (failed row and escaped cause)
agents/raven-ppt/run.py -> 2 failed (engine exemption, launcher contract)
```
Ten existing tests pinned "activation raises" (`test_plugin_registry`,
`test_plugin_bootstrap`, and the conflict tests for tools, hooks, tool
gates and session observers). They now assert the new contract instead:
the later plugin is recorded with the same reason text, the earlier one
keeps the name, and nothing of the loser is registered. Two stubs in
`test_cli_tui_commands.py` took `build_plugin_registry(cfg)` and now
accept the `notify` keyword.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
- A plugin conflict used to disable all plugins; it now keeps the plugin
that activates first and skips the other with a notice. Two plugins
contributing the same name therefore run with one of them instead of
none.
- A product sub-agent now starts with the host's `plugins.disabled`
merged into its config (its own engine excepted). A plugin that was
disabled on the host but relied on inside a product stops loading there;
the remedy is to enable it on the host.
- `PluginRegistry.activate` no longer raises `PluginConflictError` /
`PluginFactoryImportError`; the only in-tree callers were
`assemble_plugin_registry` and `raven plugins`. The exception classes
stay exported.
- Security: the change only narrows what loads. A failed plugin's
directory leaves sys.path, and the inherited opt-outs can only remove
plugins from a product, never add one.
- Rollback: revert the squash commit.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
633 lines
24 KiB
Python
633 lines
24 KiB
Python
"""Plugin ``tools`` contribution point — manifest, registry, CLI stack,
|
|
render surfacing, and the EverOS ``understand_media`` tool itself.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
import textwrap
|
|
import types
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
from pydantic import ValidationError
|
|
|
|
from raven.plugins import (
|
|
Contributes,
|
|
DiscoveredPlugin,
|
|
ManifestOrigin,
|
|
PluginContext,
|
|
PluginManifest,
|
|
PluginNotFoundError,
|
|
PluginRegistry,
|
|
ServiceLocator,
|
|
ToolContribution,
|
|
)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Test-module injection (mirrors test_plugin_registry.py)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
_INJECTED_MODULES: set[str] = set()
|
|
|
|
|
|
def _install_test_module(name: str, attrs: dict[str, object]) -> None:
|
|
mod = types.ModuleType(name)
|
|
for k, v in attrs.items():
|
|
setattr(mod, k, v)
|
|
sys.modules[name] = mod
|
|
_INJECTED_MODULES.add(name)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _cleanup_modules():
|
|
# Only the fakes injected above -- evicting every module a test imported
|
|
# also drops real ones, and a Rust extension that installs a global logger
|
|
# on import (lancedb, reached via everos) panics on the second import.
|
|
yield
|
|
for name in _INJECTED_MODULES:
|
|
sys.modules.pop(name, None)
|
|
_INJECTED_MODULES.clear()
|
|
|
|
|
|
def _discovered_with_tools(
|
|
plugin_id: str,
|
|
tools: list[tuple[str, str]],
|
|
) -> DiscoveredPlugin:
|
|
mf = PluginManifest(
|
|
id=plugin_id,
|
|
version="0.1.0",
|
|
contributes=Contributes(
|
|
tools=[ToolContribution(name=n, factory=f) for n, f in tools],
|
|
),
|
|
)
|
|
return DiscoveredPlugin(manifest=mf, source=ManifestOrigin.USER, location=None)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Manifest
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestManifestTools:
|
|
def test_parses_tools_contribution(self) -> None:
|
|
toml = textwrap.dedent("""
|
|
[plugin]
|
|
id = "everos-memory"
|
|
version = "0.1.0"
|
|
[[plugin.contributes.tools]]
|
|
name = "understand_media"
|
|
factory = "raven_everos.tools:make_understand_media_tool"
|
|
""")
|
|
mf = PluginManifest.from_toml_str(toml)
|
|
assert [t.name for t in mf.contributes.tools] == ["understand_media"]
|
|
assert mf.contributes.tools[0].factory.endswith(":make_understand_media_tool")
|
|
|
|
def test_default_tools_empty(self) -> None:
|
|
mf = PluginManifest(id="p", version="0.1.0")
|
|
assert mf.contributes.tools == []
|
|
|
|
def test_bad_factory_ref_rejected(self) -> None:
|
|
with pytest.raises(ValidationError):
|
|
ToolContribution(name="t", factory="not-a-ref")
|
|
|
|
def test_duplicate_tool_names_rejected(self) -> None:
|
|
with pytest.raises(ValidationError, match="duplicate tool name"):
|
|
PluginManifest(
|
|
id="p",
|
|
version="0.1.0",
|
|
contributes=Contributes(
|
|
tools=[
|
|
ToolContribution(name="dup", factory="m:a"),
|
|
ToolContribution(name="dup", factory="m:b"),
|
|
],
|
|
),
|
|
)
|
|
|
|
def test_backend_and_tool_may_share_name(self) -> None:
|
|
# Uniqueness is per-kind; a backend and a tool named the same is fine.
|
|
from raven.plugins import MemoryBackendContribution
|
|
|
|
mf = PluginManifest(
|
|
id="p",
|
|
version="0.1.0",
|
|
contributes=Contributes(
|
|
memory_backends=[MemoryBackendContribution(name="x", factory="m:b")],
|
|
tools=[ToolContribution(name="x", factory="m:t")],
|
|
),
|
|
)
|
|
assert mf.contributes.tools[0].name == "x"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Registry
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestRegistryTools:
|
|
def test_activates_and_resolves_tool(self) -> None:
|
|
def fake_factory(ctx):
|
|
return "tool-instance"
|
|
|
|
_install_test_module("_tp_tools_a", {"make_tool": fake_factory})
|
|
reg = PluginRegistry()
|
|
reg.activate([_discovered_with_tools("alpha", [("understand_media", "_tp_tools_a:make_tool")])])
|
|
|
|
assert reg.tool_names() == ["understand_media"]
|
|
assert reg.tool_plugin_id("understand_media") == "alpha"
|
|
assert reg.get_tool_factory("understand_media") is fake_factory
|
|
|
|
def test_build_tool_invokes_factory_with_context(self, tmp_path: Path) -> None:
|
|
captured = {}
|
|
|
|
def fake_factory(ctx: PluginContext):
|
|
captured["config"] = ctx.config
|
|
return "built"
|
|
|
|
_install_test_module("_tp_tools_b", {"make_tool": fake_factory})
|
|
reg = PluginRegistry()
|
|
reg.activate([_discovered_with_tools("p", [("t", "_tp_tools_b:make_tool")])])
|
|
|
|
out = reg.build_tool(
|
|
"t",
|
|
config={"k": 1},
|
|
services=ServiceLocator(workspace=tmp_path, user_id="default", agent_id="default"),
|
|
)
|
|
assert out == "built"
|
|
assert captured["config"] == {"k": 1}
|
|
|
|
def test_tool_name_conflict_across_plugins(self) -> None:
|
|
def f(ctx):
|
|
return None
|
|
|
|
_install_test_module("_tp_tools_c", {"make_tool": f})
|
|
reg = PluginRegistry()
|
|
reg.activate(
|
|
[
|
|
_discovered_with_tools("one", [("dup", "_tp_tools_c:make_tool")]),
|
|
_discovered_with_tools("two", [("dup", "_tp_tools_c:make_tool")]),
|
|
]
|
|
)
|
|
assert reg.activated_ids() == ["one"]
|
|
assert reg.tool_plugin_id("dup") == "one"
|
|
[failure] = reg.activation_failures()
|
|
assert failure.plugin_id == "two"
|
|
assert "tool 'dup' contributed by both 'one' and 'two'" in failure.reason
|
|
|
|
def test_unknown_tool_raises(self) -> None:
|
|
reg = PluginRegistry()
|
|
with pytest.raises(PluginNotFoundError):
|
|
reg.get_tool_factory("nope")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# CLI stack — build_plugin_tools
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestBuildPluginTools:
|
|
def _config(self, plugin_config: dict | None = None):
|
|
from raven.config.raven import PluginsConfig, RavenConfig
|
|
|
|
return RavenConfig(plugins=PluginsConfig(config=dict(plugin_config or {})))
|
|
|
|
def test_builds_tools_from_registry(self, tmp_path: Path) -> None:
|
|
from raven.core.plugin_stack import build_plugin_tools
|
|
|
|
seen = {}
|
|
|
|
def fake_factory(ctx):
|
|
seen["config"] = ctx.config
|
|
return f"tool::{ctx.config.get('flag')}"
|
|
|
|
_install_test_module("_tp_tools_d", {"make_tool": fake_factory})
|
|
reg = PluginRegistry()
|
|
reg.activate([_discovered_with_tools("myplugin", [("t1", "_tp_tools_d:make_tool")])])
|
|
|
|
cfg = self._config({"myplugin": {"flag": "on"}})
|
|
tools = build_plugin_tools(tmp_path, cfg, registry=reg)
|
|
assert tools == ["tool::on"]
|
|
assert seen["config"] == {"flag": "on"}
|
|
|
|
def test_empty_when_no_tools(self, tmp_path: Path) -> None:
|
|
from raven.core.plugin_stack import build_plugin_tools
|
|
|
|
assert build_plugin_tools(tmp_path, self._config(), registry=PluginRegistry()) == []
|
|
|
|
def test_failing_factory_is_skipped(self, tmp_path: Path) -> None:
|
|
from raven.core.plugin_stack import build_plugin_tools
|
|
|
|
def boom(ctx):
|
|
raise RuntimeError("nope")
|
|
|
|
_install_test_module("_tp_tools_e", {"make_tool": boom})
|
|
reg = PluginRegistry()
|
|
reg.activate([_discovered_with_tools("p", [("t", "_tp_tools_e:make_tool")])])
|
|
# One bad tool doesn't crash the build — it's logged + skipped.
|
|
assert build_plugin_tools(tmp_path, self._config(), registry=reg) == []
|
|
|
|
def test_none_factory_is_skipped(self, tmp_path: Path) -> None:
|
|
from raven.core.plugin_stack import build_plugin_tools
|
|
|
|
# A factory may return None to decline contribution (e.g. an
|
|
# optional dependency is absent) — skipped without error.
|
|
def opt_out(ctx):
|
|
return None
|
|
|
|
_install_test_module("_tp_tools_f", {"make_tool": opt_out})
|
|
reg = PluginRegistry()
|
|
reg.activate([_discovered_with_tools("p", [("t", "_tp_tools_f:make_tool")])])
|
|
assert build_plugin_tools(tmp_path, self._config(), registry=reg) == []
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# render.build_user_content — attachment surfacing
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestRenderAttachments:
|
|
def test_non_image_surfaced_as_note(self, tmp_path: Path) -> None:
|
|
from raven.context_engine.segments import render
|
|
|
|
pdf = tmp_path / "report.pdf"
|
|
pdf.write_bytes(b"%PDF-1.4 data")
|
|
# Named explicitly: the hint is only emitted when a description tool
|
|
# is actually registered, and it is absent on a default install.
|
|
out = render.build_user_content("summarize this", [str(pdf)], describe_tool="understand_media")
|
|
assert isinstance(out, str)
|
|
assert "report.pdf" in out
|
|
assert "understand_media" in out
|
|
|
|
bare = render.build_user_content("summarize this", [str(pdf)])
|
|
assert "report.pdf" in bare and f"(path: {pdf})" in bare
|
|
assert "understand_media" not in bare
|
|
assert "summarize this" in out
|
|
|
|
def test_image_inlined_as_block(self, tmp_path: Path) -> None:
|
|
import base64 as _b64
|
|
|
|
from raven.context_engine.segments import render
|
|
|
|
png = tmp_path / "a.png"
|
|
png.write_bytes(
|
|
_b64.b64decode(
|
|
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
|
|
)
|
|
)
|
|
out = render.build_user_content("look", [str(png)])
|
|
assert isinstance(out, list)
|
|
assert out[0]["type"] == "image_url"
|
|
assert out[0]["image_url"]["url"].startswith("data:image/png;base64,")
|
|
# The user's own text comes first, then the path. The base64 survives
|
|
# only this turn (the session keeps a placeholder), so the path is what
|
|
# lets a later turn re-read the picture.
|
|
text_block = out[-1]["text"]
|
|
assert text_block.startswith("look\n\n")
|
|
assert f"(path: {png})" in text_block
|
|
assert "read_file" in text_block
|
|
|
|
def test_mixed_image_and_doc(self, tmp_path: Path) -> None:
|
|
import base64 as _b64
|
|
|
|
from raven.context_engine.segments import render
|
|
|
|
png = tmp_path / "a.png"
|
|
png.write_bytes(
|
|
_b64.b64decode(
|
|
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
|
|
)
|
|
)
|
|
pdf = tmp_path / "d.pdf"
|
|
pdf.write_bytes(b"%PDF-1.4")
|
|
out = render.build_user_content("q", [str(png), str(pdf)], describe_tool="understand_media")
|
|
assert isinstance(out, list)
|
|
assert out[0]["type"] == "image_url"
|
|
text_block = out[-1]["text"]
|
|
assert "d.pdf" in text_block and "understand_media" in text_block
|
|
|
|
def test_no_media_returns_text(self) -> None:
|
|
from raven.context_engine.segments import render
|
|
|
|
assert render.build_user_content("hi", None) == "hi"
|
|
|
|
def test_image_becomes_a_note_when_the_model_cannot_see(self, tmp_path: Path) -> None:
|
|
"""A text-only model must be told the picture exists and how to read it.
|
|
|
|
Inlining it instead is the one outcome with no recovery: the endpoint
|
|
either rejects the request or drops the image and answers from the text
|
|
alone, which reads as a correct reply built on nothing.
|
|
"""
|
|
import base64 as _b64
|
|
|
|
from raven.context_engine.segments import render
|
|
|
|
png = tmp_path / "a.png"
|
|
png.write_bytes(
|
|
_b64.b64decode(
|
|
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
|
|
)
|
|
)
|
|
out = render.build_user_content("look", [str(png)], can_see_images=False, describe_tool="understand_media")
|
|
|
|
assert isinstance(out, str)
|
|
assert "base64" not in out
|
|
assert f"(path: {png})" in out
|
|
assert "understand_media" in out
|
|
assert out.startswith("look\n\n")
|
|
|
|
def test_legacy_builder_names_the_image_path_too(self, tmp_path: Path) -> None:
|
|
"""There are two content builders (context engine and legacy
|
|
ContextBuilder); a path dropped in either one is a path the model cannot
|
|
use to re-read the picture."""
|
|
import base64 as _b64
|
|
|
|
from raven.agent.context.builder import ContextBuilder
|
|
|
|
png = tmp_path / "a.png"
|
|
png.write_bytes(
|
|
_b64.b64decode(
|
|
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
|
|
)
|
|
)
|
|
out = ContextBuilder._build_user_content(object.__new__(ContextBuilder), "look", [str(png)])
|
|
|
|
assert isinstance(out, list)
|
|
assert out[0]["type"] == "image_url"
|
|
text_block = out[-1]["text"]
|
|
assert text_block.startswith("look\n\n")
|
|
assert f"(path: {png})" in text_block
|
|
assert "read_file" in text_block
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# EverOS understand_media tool (needs raven_everos importable)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
from raven_everos.tools import UnderstandMediaTool, make_understand_media_tool
|
|
|
|
|
|
class TestUnderstandMediaTool:
|
|
def test_factory_returns_tool_when_extra_available(self, monkeypatch) -> None:
|
|
import raven_everos.tools as tools_mod
|
|
from raven.contracts.tool import Tool
|
|
|
|
# Gate on the parser extra; force it "available" so the assertion
|
|
# holds regardless of whether the heavy extra is installed here.
|
|
monkeypatch.setattr(tools_mod, "_multimodal_available", lambda: True)
|
|
t = make_understand_media_tool(None)
|
|
assert isinstance(t, Tool)
|
|
assert t.name == "understand_media"
|
|
assert "paths" in t.parameters["properties"]
|
|
|
|
def test_factory_returns_none_when_extra_missing(self, monkeypatch) -> None:
|
|
# No everos[multimodal] extra → factory declines to contribute the
|
|
# tool (returns None) so the host never registers it.
|
|
import raven_everos.tools as tools_mod
|
|
|
|
monkeypatch.setattr(tools_mod, "_multimodal_available", lambda: False)
|
|
assert make_understand_media_tool(None) is None
|
|
|
|
def test_description_defers_images_to_read_file(self) -> None:
|
|
# Both tools accept an image path, so without an explicit steer the
|
|
# model picks either and may get a transcription instead of the picture.
|
|
assert "read_file" in UnderstandMediaTool().description
|
|
|
|
async def test_missing_paths_errors(self) -> None:
|
|
t = UnderstandMediaTool()
|
|
assert (await t.execute(paths=None)).startswith("Error")
|
|
assert (await t.execute(paths=[])).startswith("Error")
|
|
|
|
async def test_degrades_when_unavailable(self, monkeypatch) -> None:
|
|
# When the multimodal runtime is unavailable, understand_files
|
|
# raises MultimodalUnavailableError and the tool surfaces one clear
|
|
# error message rather than propagating the exception.
|
|
import raven_everos.tools as tools_mod
|
|
from raven_everos.multimodal import MultimodalUnavailableError
|
|
|
|
async def fake_unavailable(paths):
|
|
raise MultimodalUnavailableError("multimodal extra not configured")
|
|
|
|
monkeypatch.setattr(tools_mod, "understand_files", fake_unavailable)
|
|
out = await UnderstandMediaTool().execute(paths=["/tmp/x.pdf"])
|
|
assert "unavailable" in out.lower()
|
|
|
|
async def test_formats_results(self, monkeypatch) -> None:
|
|
import raven_everos.tools as tools_mod
|
|
|
|
async def fake_understand(paths):
|
|
return [
|
|
{"path": paths[0], "name": "a.pdf", "text": "hello world"},
|
|
{"path": paths[1], "name": "b.mp4", "error": "video not supported"},
|
|
]
|
|
|
|
monkeypatch.setattr(tools_mod, "understand_files", fake_understand)
|
|
out = await UnderstandMediaTool().execute(paths=["/x/a.pdf", "/x/b.mp4"])
|
|
assert "## a.pdf" in out and "hello world" in out
|
|
assert "## b.mp4" in out and "video not supported" in out
|
|
|
|
|
|
class TestContentItemRouting:
|
|
"""Input routing in multimodal._content_item_for — file path vs URL.
|
|
|
|
Unit-level so it needs no LLM / network: just asserts the ContentItem
|
|
shape everalgo will receive.
|
|
"""
|
|
|
|
def test_http_url_detection(self) -> None:
|
|
from raven_everos.multimodal import _is_http_url
|
|
|
|
assert _is_http_url("https://example.com/page")
|
|
assert _is_http_url("http://example.com")
|
|
assert not _is_http_url("/tmp/a.png")
|
|
assert not _is_http_url("file:///tmp/a.png")
|
|
assert not _is_http_url("relative/path.pdf")
|
|
|
|
def test_url_becomes_uri_item(self) -> None:
|
|
from raven_everos.multimodal import _content_item_for
|
|
|
|
item = _content_item_for("https://example.com/doc")
|
|
# uri-backed → everalgo fetches + dispatches by Content-Type;
|
|
# no base64 payload, no local read.
|
|
assert item == {
|
|
"type": "url",
|
|
"uri": "https://example.com/doc",
|
|
"name": "https://example.com/doc",
|
|
}
|
|
|
|
def test_local_file_becomes_base64_item(self, tmp_path: Path) -> None:
|
|
from raven_everos.multimodal import _content_item_for
|
|
|
|
f = tmp_path / "hello.txt"
|
|
f.write_text("hi")
|
|
item = _content_item_for(str(f))
|
|
assert item["name"] == "hello.txt"
|
|
assert item["ext"] == ".txt"
|
|
assert "base64" in item and "uri" not in item
|
|
|
|
def test_missing_local_file_raises(self) -> None:
|
|
from raven_everos.multimodal import _content_item_for
|
|
|
|
with pytest.raises(FileNotFoundError):
|
|
_content_item_for("/no/such/file.pdf")
|
|
|
|
|
|
class TestEverosParserContract:
|
|
"""``understand_files`` driving the *real* everos parser call chain.
|
|
|
|
Every other test here replaces ``understand_files`` wholesale, so a
|
|
signature change in ``everos.memory.extract.parser.enrich_content_items``
|
|
reaches users as a per-file "could not understand" note instead of a test
|
|
failure. These stub only the two external boundaries -- the optional
|
|
parser extra's availability probe and the vision LLM call -- leaving the
|
|
call into everos, and therefore its signature, genuinely exercised.
|
|
"""
|
|
|
|
@staticmethod
|
|
def _stub_boundaries(monkeypatch: pytest.MonkeyPatch, parse) -> None:
|
|
import everos.component.llm.client as llm_client
|
|
import everos.component.parser as component_parser
|
|
|
|
monkeypatch.setattr(component_parser, "parser_available", lambda: True)
|
|
monkeypatch.setattr(component_parser, "aparse_file", parse)
|
|
monkeypatch.setattr(llm_client, "get_multimodal_llm_client", lambda: object())
|
|
|
|
@staticmethod
|
|
def _png(tmp_path: Path, name: str = "shot.png") -> Path:
|
|
f = tmp_path / name
|
|
f.write_bytes(b"payload")
|
|
return f
|
|
|
|
async def test_local_file_parses_through_real_enrich(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
from raven_everos.multimodal import understand_files
|
|
|
|
async def parse(raw_file):
|
|
assert raw_file.content == b"payload"
|
|
assert raw_file.extension == "png"
|
|
return types.SimpleNamespace(text="a diagram")
|
|
|
|
self._stub_boundaries(monkeypatch, parse)
|
|
f = self._png(tmp_path)
|
|
assert await understand_files([str(f)]) == [{"path": str(f), "name": "shot.png", "text": "a diagram"}]
|
|
|
|
async def test_interface_error_aborts_the_batch(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# A TypeError out of the parser means our call no longer matches
|
|
# everos's API -- true of every file, so it must not be reported as
|
|
# one file the parser could not read.
|
|
from raven_everos.multimodal import (
|
|
MultimodalUnavailableError,
|
|
understand_files,
|
|
)
|
|
|
|
async def parse(raw_file):
|
|
raise TypeError("unexpected keyword argument")
|
|
|
|
self._stub_boundaries(monkeypatch, parse)
|
|
f = self._png(tmp_path)
|
|
with pytest.raises(MultimodalUnavailableError):
|
|
await understand_files([str(f)])
|
|
|
|
async def test_one_failed_file_does_not_abort_the_batch(
|
|
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
from raven_everos.multimodal import understand_files
|
|
|
|
async def parse(raw_file):
|
|
if raw_file.content == b"payload":
|
|
raise ValueError("cannot decode")
|
|
return types.SimpleNamespace(text="second one")
|
|
|
|
self._stub_boundaries(monkeypatch, parse)
|
|
bad = self._png(tmp_path, "bad.png")
|
|
good = tmp_path / "good.png"
|
|
good.write_bytes(b"other")
|
|
out = await understand_files([str(bad), str(good)])
|
|
assert "cannot decode" in out[0]["error"]
|
|
assert out[1]["text"] == "second one"
|
|
|
|
async def test_unsupported_modality_degrades_per_item(
|
|
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
from everos.core.errors import UnsupportedModalityError
|
|
|
|
from raven_everos.multimodal import understand_files
|
|
|
|
async def parse(raw_file):
|
|
if raw_file.content == b"payload":
|
|
raise UnsupportedModalityError("video is not supported")
|
|
return types.SimpleNamespace(text="fine")
|
|
|
|
self._stub_boundaries(monkeypatch, parse)
|
|
video = self._png(tmp_path, "clip.png")
|
|
good = tmp_path / "good.png"
|
|
good.write_bytes(b"other")
|
|
out = await understand_files([str(video), str(good)])
|
|
assert "not supported" in out[0]["error"]
|
|
assert out[1]["text"] == "fine"
|
|
|
|
async def test_unconfigured_llm_fails_the_call_once(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# The reason understand_files probes the client up front: without it
|
|
# the parser resolves its own client per file and the same failure
|
|
# would come back once per attachment.
|
|
import everos.component.llm.client as llm_client
|
|
|
|
from raven_everos.multimodal import (
|
|
MultimodalUnavailableError,
|
|
understand_files,
|
|
)
|
|
|
|
parsed: list[object] = []
|
|
|
|
async def parse(raw_file):
|
|
parsed.append(raw_file)
|
|
return types.SimpleNamespace(text="never")
|
|
|
|
self._stub_boundaries(monkeypatch, parse)
|
|
|
|
def unconfigured():
|
|
raise llm_client.LLMNotConfiguredError("multimodal LLM not configured")
|
|
|
|
monkeypatch.setattr(llm_client, "get_multimodal_llm_client", unconfigured)
|
|
with pytest.raises(MultimodalUnavailableError, match="not configured"):
|
|
await understand_files([str(self._png(tmp_path))])
|
|
assert parsed == []
|
|
|
|
async def test_missing_parser_extra_fails_the_call(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
import everos.component.parser as component_parser
|
|
|
|
from raven_everos.multimodal import (
|
|
MultimodalUnavailableError,
|
|
understand_files,
|
|
)
|
|
|
|
monkeypatch.setattr(component_parser, "parser_available", lambda: False)
|
|
with pytest.raises(MultimodalUnavailableError, match="not installed"):
|
|
await understand_files([str(self._png(tmp_path))])
|
|
|
|
async def test_llm_failure_reads_back_parse_error(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# everos degrades an LLMError in place (parse_error set, no
|
|
# parsed_content); understand_files must surface that as the error.
|
|
from everalgo.llm import LLMError
|
|
|
|
from raven_everos.multimodal import understand_files
|
|
|
|
async def parse(raw_file):
|
|
raise LLMError("upstream 500")
|
|
|
|
self._stub_boundaries(monkeypatch, parse)
|
|
f = self._png(tmp_path)
|
|
assert await understand_files([str(f)]) == [{"path": str(f), "name": "shot.png", "error": "LLMError"}]
|
|
|
|
async def test_missing_file_is_reported_not_fatal(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
from raven_everos.multimodal import understand_files
|
|
|
|
async def parse(raw_file):
|
|
return types.SimpleNamespace(text="fine")
|
|
|
|
self._stub_boundaries(monkeypatch, parse)
|
|
good = self._png(tmp_path)
|
|
missing = tmp_path / "gone.png"
|
|
out = await understand_files([str(missing), str(good)])
|
|
assert out[0] == {"path": str(missing), "name": "gone.png", "error": "file not found"}
|
|
assert out[1]["text"] == "fine"
|