Raven-1/tests/test_cli_plugin_commands.py
zhanghui 16784d3a1e
build(*): upgrade everos and the memory plugin to 1.4.2 (#950)
## Summary

EverOS 1.4.2 shipped today. It fixes what a memory server that stays up
for
days runs into: a LanceDB index reader leaked file descriptors until
searches
took seconds, an interactive search queued behind the cascade's
background
embedding (vector search p50 46 s under three concurrent writers, 0.60 s
now),
and a cascade pass sent one embedding request per entry instead of one
per
file. The memory plugin's exact pin moves from
`everos[multimodal]==1.4.1` to
`==1.4.2`, and the plugin's own version from `1.4.0` to `1.4.2` -- the
same
number as the EverOS it ships. Both EverOS releases declare the same 36
requirements, so nothing else in the lock follows everos down.

**No adapter change.** Every internal everos symbol the adapter reaches
is
present in 1.4.2 with the same signature, the package layout churn
underneath
included: `infra.persistence.lancedb.tables` became
`infra.persistence.records` and `infra.persistence.backends` became
`lancedb.LanceIndexBackend`, but the adapter imports `episode_repo` and
`agent_skill_repo` from the `lancedb` package, which re-exports both the
way
it did before.

**The derived index repairs itself, once.** 1.4.2 makes a table's
primary key
`(app_id, project_id, id)` rather than `id` alone, and the first full
cascade
pass after the upgrade re-reconciles every indexed markdown file against
the
index, re-embedding only the rows that are missing and recording that it
ran
in `.index/.index_reconcile_version`. A healthy root pays one read pass
and no
embedding; markdown is the source either way and was never affected by
the bug
this fixes. The two roots EverOS says need `everos cascade rebuild` by
hand are
a Milvus derived index and a root where reflection already ran across
projects;
raven writes one project into a LanceDB root and reaches neither, though
a user
who pointed raven at a root of their own might.

**The plugin version now equals the EverOS it pins.** The only precedent
was
plugin `1.2.0` pinning EverOS `1.2.3` and plugin `1.4.0` pinning EverOS
`1.4.1`: the plugin tracked EverOS major.minor and kept its patch for
itself,
and nothing said so anywhere. The result reads as a version one release
behind
the thing it ships, so step 5 of the upgrade SOP now says to hold the
two
equal, and this upgrade skips `1.4.1` to close the gap. That step also
named a
test that no longer exists for the literal assertion
(`test_bundled_shadows_lower_priority_source`); it is
`TestPackageSurface::test_imports_clean`, and the step now lists all
three
files that carry the version rather than the manifest alone.

Recorded in `docs/memory-plugin-architecture.md` section 7.5, which the
upgrade
SOP in section 7.2 asks for.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [x] Other (external dependency upgrade)

## Verification

```
uv run --frozen --python 3.12 --all-extras pytest -q \
  tests/test_everos_plugin_discovery.py tests/test_everos_backend.py \
  tests/test_everos_backend_contract.py tests/test_everos_http_adapter.py \
  tests/test_everos_config.py tests/test_everos_server.py \
  tests/test_everos_discover.py tests/test_memory_backend_protocol.py \
  tests/test_memory_backend_contract.py tests/test_cli_plugin_commands.py \
  tests/test_config_update_everos.py tests/test_dependency_constraints.py \
  tests/test_release_plugin_list.py tests/test_skill_router_everos_source.py \
  tests/test_agent_loop_everos_pipeline.py
-> 567 passed, 1 skipped (the skip is Linux-only inotify)

uv run --frozen --python 3.12 --all-extras pytest -q \
  tests/test_everos_plugin_discovery.py tests/test_cli_plugin_commands.py \
  tests/test_release_plugin_list.py tests/test_plugin_manifest.py \
  tests/test_plugin_discover.py tests/test_everos_discover.py
-> 91 passed (re-run after the plugin version moved to 1.4.2)

make lint-types                           -> all checks passed

make lint-python lint-imports lint-deps   -> clean, 10 import contracts kept
make check-source-language check-large-files COMMIT_RANGE=github/main..HEAD -> clean
```

Against a copy of a real 55 MB store (568 episodes, seven LanceDB
tables)
written under 1.2.x and last served by 1.4.1, booted on a scratch root:

- 1.4.2 opens it with no schema complaint, `/health` answers
`version 1.4.2` with `cascade.healthy true`, and the reconcile pass
writes
  `.index/.index_reconcile_version` with no error in 441 log lines.
- The same keyword search (`POST /api/v2/memory/search`, the body the
adapter
sends) returns the same five episode ids in the same order under 1.4.1
and
under 1.4.2; BM25 scores drift in the third decimal, as they did on the
  1.2.3 to 1.4.1 move.
- Raven's own `installed_everos_version()` reads 1.4.2, and every
internal
  symbol the adapter imports resolves with the signature it had.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

User-visible change: memory searches stop queueing behind background
writes,
and a long-lived server stops leaking file descriptors. An existing
index pays
one read pass on the first start after the upgrade.

Rollback is `git revert`. The reconcile pass only re-embeds rows the old
primary key had lost, and `lancedb` does not move, so 1.4.1 reads an
index
1.4.2 has written. The new `.index/.index_reconcile_version` file is
ignored by
1.4.1; a re-upgrade later would skip the pass it records, which is the
intent.

Not exercised here: a Milvus derived index (raven configures LanceDB), a
multi-project root, and a real embedding or LLM round trip (the scratch
root
ran without provider keys, so the checks above are the keyword leg and
the
cascade's own reconcile).

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: gloryfromca <23442919+gloryfromca@users.noreply.github.com>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-10-15 11:01:45 +08:00

336 lines
12 KiB
Python

"""Tests for ``raven plugins`` and ``raven plugin auth``.
``plugins`` is read-only: it renders the activated plugins -- here the
registered entry-points one, raven_everos -- and resolves
``config.memory.backend`` against the live registry without invoking any
plugin runtime (no ``MemoryBackend.start`` is awaited). These tests pin the
table's contents and the three branches of the backend-selection block:
present-active, present-unknown, and explicitly-disabled.
``plugin auth`` is a chain of refusals in front of one network call: an
unknown server, a disabled one, and one not configured for OAuth are each
rejected before anything connects, and a connect that comes back in any state
other than ``connected`` exits non-zero. Only the last of those needs a
manager, and it gets a fake one -- the real flow opens a browser.
"""
from __future__ import annotations
import json
import os
from pathlib import Path
from typing import Any
import pytest
from typer.testing import CliRunner
from raven.cli.commands import app
from raven.config.loader import set_config_path
runner = CliRunner()
# Through the root app rather than `plugin_app` directly: a Typer group with a
# single command collapses into that command, so invoking the sub-app with
# ["auth", name] parses "auth" as the server argument.
# ---------------------------------------------------------------------------
# ``raven plugin auth`` -- refusals in front of the one network call
# ---------------------------------------------------------------------------
@pytest.fixture
def config_with(tmp_path: Path):
"""Write a config carrying the given mcp_servers block, and point the
loader at it. The command reads the loader's own path, not an argument."""
def _write(servers: dict) -> Path:
path = tmp_path / "config.json"
path.write_text(json.dumps({"tools": {"mcpServers": servers}}))
set_config_path(path)
return path
yield _write
set_config_path(None) # type: ignore[arg-type]
def test_an_unknown_server_is_refused_before_anything_connects(config_with) -> None:
config_with({})
r = runner.invoke(app, ["plugin", "auth", "ghost"])
assert r.exit_code == 1
assert "no MCP server named" in r.stdout
def test_a_disabled_server_is_refused(config_with) -> None:
config_with({"svc": {"url": "https://svc.example/mcp", "auth": "oauth", "enabled": False}})
r = runner.invoke(app, ["plugin", "auth", "svc"])
assert r.exit_code == 1
assert "disabled" in r.stdout
def test_a_server_not_configured_for_oauth_is_refused(config_with) -> None:
"""The message has to name the current mode, or the fix is a guess."""
config_with({"svc": {"url": "https://svc.example/mcp", "auth": "apikey", "enabled": True}})
r = runner.invoke(app, ["plugin", "auth", "svc"])
assert r.exit_code == 1
assert "apikey" in r.stdout
class _FakeManager:
"""Stands in for MCPConnectionManager: the real `connect` opens a browser."""
def __init__(self, snapshot: dict):
self._snapshot = snapshot
self.closed = False
async def connect(self, name: str, cfg) -> dict:
return self._snapshot
async def aclose(self) -> None:
self.closed = True
def _patch_manager(monkeypatch: pytest.MonkeyPatch, snapshot: dict) -> list:
made: list = []
import raven.mcp.manager as mgr_mod
def _factory(registry):
m = _FakeManager(snapshot)
made.append(m)
return m
monkeypatch.setattr(mgr_mod, "MCPConnectionManager", _factory)
return made
def test_a_connected_snapshot_reports_the_tool_count(config_with, monkeypatch: pytest.MonkeyPatch) -> None:
config_with({"svc": {"url": "https://svc.example/mcp", "auth": "oauth", "enabled": True}})
made = _patch_manager(monkeypatch, {"state": "connected", "tool_count": 3, "error": None})
r = runner.invoke(app, ["plugin", "auth", "svc"])
assert r.exit_code == 0, r.stdout
assert "authorized" in r.stdout
assert "3 tools" in r.stdout
assert made[0].closed, "the manager must be closed even on the happy path"
def test_a_failed_authorization_exits_non_zero_and_names_the_reason(
config_with, monkeypatch: pytest.MonkeyPatch
) -> None:
"""`auth_required` back from connect means the flow did not complete; the
command must not report success just because the call returned."""
config_with({"svc": {"url": "https://svc.example/mcp", "auth": "oauth", "enabled": True}})
made = _patch_manager(monkeypatch, {"state": "auth_required", "tool_count": 0, "error": "denied by user"})
r = runner.invoke(app, ["plugin", "auth", "svc"])
assert r.exit_code == 1
assert "authorization failed" in r.stdout
assert "denied by user" in r.stdout
assert made[0].closed, "the manager must be closed on the failure path too"
# ---------------------------------------------------------------------------
# ``raven plugins`` -- the listing table and the backend-selection block
# ---------------------------------------------------------------------------
def _make_runner_args(tmp_path: Path, config: dict[str, Any]) -> list[str]:
"""Write a config file + return the typer args to point at it."""
config_path = tmp_path / "config.json"
config_path.write_text(json.dumps(config), encoding="utf-8")
# ``raven plugins -c <path>``
return ["plugins", "-c", str(config_path)]
@pytest.fixture(autouse=True)
def _wide_console(monkeypatch: pytest.MonkeyPatch) -> None:
"""Give every command in this file a console wide enough to print a
factory reference.
``COLUMNS`` in the invocation's environment is not enough on its own. The
command renders through ``plugin_commands.console``, a module-level
``Console()`` whose width is fixed when the module is first imported, so
the variable only lands if this file is what imported it -- which held
until the suite started running in parallel, where another worker's test
gets there first and the width is Rich's default 80. The factory column
then truncates to ``...`` and the assertions read as a missing string
rather than a narrow terminal.
Reproduce by importing the CLI at 80 columns before the test runs: the
failure is `assert 'raven_everos.backend:make_backend' in`
a string that ends `...___/`.
"""
from rich.console import Console
from raven.cli import plugin_commands
monkeypatch.setattr(plugin_commands, "console", Console(width=200))
def _invoke(args: list[str], tmp_path: Path):
"""Run the CLI with a sandboxed HOME so user-level plugin discovery
doesn't surface unrelated plugins on the developer's machine.
``COLUMNS=200`` is kept for the child's own sake -- anything the command
renders through a console it builds itself reads it -- but the table this
file asserts on comes from the module-level console that ``_wide_console``
replaces.
"""
fake_home = tmp_path / "home"
fake_home.mkdir(parents=True, exist_ok=True)
env = {**os.environ, "HOME": str(fake_home), "COLUMNS": "200"}
return runner.invoke(app, args, env=env)
# ---------------------------------------------------------------------------
# Default config -- backend selected and available
# ---------------------------------------------------------------------------
class TestActiveBackend:
def test_lists_everos_memory(self, tmp_path: Path) -> None:
result = _invoke(
_make_runner_args(tmp_path, {}),
tmp_path,
)
assert result.exit_code == 0, result.stdout
assert "everos-memory" in result.stdout
assert "1.4.2" in result.stdout
assert "entry_points" in result.stdout
def test_shows_active_backend_with_track_ids(
self,
tmp_path: Path,
) -> None:
result = _invoke(
_make_runner_args(
tmp_path,
{
"memory": {
"backend": "everos",
"userId": "alice",
"agentId": "robo",
},
},
),
tmp_path,
)
assert result.exit_code == 0
assert "Active memory backend" in result.stdout
assert "everos" in result.stdout
assert "from plugin: everos-memory" in result.stdout
assert "User id:" in result.stdout
assert "alice" in result.stdout
assert "Agent id:" in result.stdout
assert "robo" in result.stdout
# ---------------------------------------------------------------------------
# Backend explicitly disabled
# ---------------------------------------------------------------------------
class TestBackendDisabled:
def test_null_backend_reports_none(self, tmp_path: Path) -> None:
result = _invoke(
_make_runner_args(tmp_path, {"memory": {"backend": None}}),
tmp_path,
)
assert result.exit_code == 0
assert "Active memory backend" in result.stdout
assert "none" in result.stdout
# The legacy-fallback hint surfaces.
assert "legacy" in result.stdout
# ---------------------------------------------------------------------------
# Backend name set but no contribution matches
# ---------------------------------------------------------------------------
class TestBackendUnavailable:
def test_unknown_backend_flagged(self, tmp_path: Path) -> None:
result = _invoke(
_make_runner_args(
tmp_path,
{
"memory": {"backend": "nonexistent"},
},
),
tmp_path,
)
assert result.exit_code == 0
assert "nonexistent" in result.stdout
assert "not available" in result.stdout
# ---------------------------------------------------------------------------
# Plugin disabled list shows in table
# ---------------------------------------------------------------------------
class TestDisabledList:
def test_disabled_plugin_status(self, tmp_path: Path) -> None:
result = _invoke(
_make_runner_args(
tmp_path,
{
"plugins": {"disabled": ["everos-memory"]},
},
),
tmp_path,
)
assert result.exit_code == 0
assert "everos-memory" in result.stdout
# The status column shows "disabled" for the row.
assert "disabled" in result.stdout
# ---------------------------------------------------------------------------
# A plugin that fails to load gets its own row, not the whole table
# ---------------------------------------------------------------------------
class TestFailedPlugin:
def test_a_broken_plugin_reads_failed_with_its_reason(self, tmp_path: Path) -> None:
package = "_cli_wreck_pkg"
plug = tmp_path / "shelf" / "wreck"
(plug / package).mkdir(parents=True)
(plug / package / "__init__.py").write_text("raise ImportError('boom [red]')\n", encoding="utf-8")
(plug / "raven-plugin.toml").write_text(
'[plugin]\nid = "wreck"\nversion = "0.1.0"\n'
f'[[plugin.contributes.tools]]\nname = "wreck_tool"\nfactory = "{package}:make"\n',
encoding="utf-8",
)
result = _invoke(_make_runner_args(tmp_path, {"plugins": {"dirs": [str(tmp_path / "shelf")]}}), tmp_path)
assert result.exit_code == 0, result.stdout
rows = {line.split()[1]: line for line in result.stdout.splitlines() if line.startswith("│")}
assert "failed" in rows["wreck"]
assert "activated" in rows["everos-memory"]
assert "wreck failed to load: plugin 'wreck': importing '_cli_wreck_pkg' failed: boom [red]" in result.stdout
# ---------------------------------------------------------------------------
# Verbose flag shows factory references
# ---------------------------------------------------------------------------
class TestVerboseFlag:
def test_verbose_shows_factory(self, tmp_path: Path) -> None:
args = _make_runner_args(tmp_path, {})
args.append("--verbose")
result = _invoke(args, tmp_path)
assert result.exit_code == 0
# The factory reference is the canonical ``module:callable`` form.
assert "raven_everos.backend:make_backend" in result.stdout