Raven-1/tests/test_plugin_contribution_ledger.py
zhanghui e70a868e9c
refactor(*): seal the memory backend seam so a memory plugin needs no host edits (#414)
## Summary

Make `MemoryBackend` the only seam between the host and the memory
subsystem, so a second memory plugin can be installed without editing
host code.

The abstraction is the goal; no second backend is being added.

What the branch does:

- `health()` joins the Protocol. `raven doctor` and `raven import` read
a `BackendHealth(ready, checks)` the backend fills in with its own
vocabulary, instead of reaching into EverOS internals. Doctor prints the
checks verbatim and never interprets a label.
- A new `[[plugin.contributes.onboard]]` kind, with
`raven/contracts/onboard.py` as its paper. The host lends the wizard
shell as one `OnboardUI` and records the result; the screen returns a
`StepOutcome`. The plugin never writes `memory.backend` -- that key
stays the host's. EverOS's step-4 screen and its `everos.toml`
management move into the plugin.
- The skill source and the after-turn feedback prefix are named from
`memory.backend` rather than the literal `everos`.
- `enabled_by_default` is gone from the manifest: an installed plugin
the user has not disabled is active.
- `tests/test_plugin_boundary.py` pins the dependency direction in both
directions, with an equality assertion so the remaining exceptions
cannot grow.

That guard's allowlist started at five host files still speaking
EverOS's wire protocol. The follow-ups below close three of them.

Two things this branch itself introduced were found in manual acceptance
on an install without the plugin and fixed here: a turn printed which
backends were installed -- empty on exactly the install that needs the
sentence -- and `settings.everos` / `settings.everosSet` let
`ModuleNotFoundError` reach the dispatcher as an internal error carrying
a traceback. Both now carry the note doctor and the wizard already use,
naming the distribution to install.

Review on this PR found four blockers, all fixed and covered. A fifth
defect came out of manual acceptance rather than review, and four more
out of the rounds that followed it -- every one of those in the
embedding endpoint's own plumbing, each found by the round after the fix
that introduced it. They are listed under Risk; the blockers are:

1. `onboard` is an optional contribution, but the wizard equated "no
matching screen" with "not configured", so one `raven onboard
--skip-memory` wrote `memory.backend` back to null for a third-party
backend configured by hand. A backend with no screen now answers on the
strength of having been selected; only one whose own screen says it is
unconfigured is cleared, which is the case that rule exists for.
2. The record a sub-agent call leaves behind ran through a backend
nobody had started. The shipped adapter begins in an unknown state, so
the first flushing write only scheduled a readiness probe and answered
false -- the record said "unavailable" while the service was up the
whole time, and each per-record HTTP client was left open. One context
manager now owns start and stop around a record, in both the manager and
the DAG runner.
3. That record's write and the recall after it addressed different
owners: a real config spells its keys in camelCase and the block reaches
the backend as written, while the adapter read only the contract's
snake_case, so a sub-agent's memories were filed under the host
identity. The adapter reads both spellings.
4. Legacy embedding discovery had narrowed to the recorded plugin root,
so an install predating root recording lost its knowledge-base endpoint
with a valid file still on disk.

**The five follow-ups the seam left behind are in this PR too**, rather
than in three stacked ones, because they are one goal and the branch
protection dismisses an approval on every push:

- The embedding endpoint is raven's. A knowledge base indexes inside the
gateway process and never speaks to the memory service, yet it read its
endpoint out of everos.toml, so selecting another backend left it with
no vectors and no message naming the cause. Raven holds the block now
and hands it to the backend at start -- as a shared default, not an
override: a plugin that was given an endpoint of its own keeps it, and
only one that has none takes raven's. Discovery of an existing
everos.toml still checks all three places the plugin itself looks, read
with plain tomllib so it works where the plugin is not installed. No
data migrates: a base is keyed on its model and measured width, not on
the base url.
- `delete()` joins the Protocol. The browser used to delete a row out of
EverOS's index, in the host process, through the plugin's repository
layer -- while the markdown EverOS treats as the source of truth kept
the text, so the memory came back on the next rebuild, after the user
was told it was gone. The adapter now uses only operations EverOS
already performs, for the kinds it performs them for.
- Sub-agent memory goes through the contract. The read half is a new
optional `recall_session`; the write half needed no new surface, only
two metadata conventions written down. `agents.<name>.config.everos`
becomes `config.memory`, opaque to the host except the two keys the host
reads; the old spelling still loads.
- The memory and settings pages say why they are empty -- no plugin, or
a backend this page does not read -- instead of four zeros and a retry
button that cannot help.
- The `everos` package leaves the host's dependencies for the plugin's
own, at the same exact pin; the host reaches the tree through the
plugin. This is the package layer only: the installers already deliver
`everos-memory` as a sibling wheel from the same release, so an ordinary
install still has long-term memory.

The boundary guard's allowlist falls from five entries to two, both GUI
surfaces still shaped around EverOS's own store. Generalising their wire
is a frontend change and is deliberately not in scope.

This branch was reviewed on GitLab as merge request 639 over three
rounds, including a post-rebase range-diff check. It is being moved here
because GitHub is now the source of truth.

## Type

- [x] Refactor

## Verification

- `uv run pytest` on this branch, rebased onto `main`: 22146 passed, 4
failed, 45 skipped. Three of the four (`test_agents_oncall_launcher`,
`test_agent_loop_token_budget`, `test_agents_code_launcher`) were re-run
on a detached checkout of that same `main` commit in this environment
and fail there identically. The fourth (`test_ppt_engine_image_search`)
could not run in that checkout, whose environment lacks the deck
engine's extras; this branch's only change anywhere under
`plugins-dist/ppt-engine` is the removal of the `enabled_by_default`
manifest key, which that test does not reach. So this branch adds no
regression.
- `npx vitest run` in ui-web: 1680 passed across 103 files. `npm run
type-check` and `npm run gen:check` clean, so the generated client
matches the contract.
- Every fix in the follow-up commits was mutated away and its test
confirmed red, then restored and confirmed green. For the record path
that is three separate mutations: dropping `start()`, dropping `stop()`,
and narrowing the owner lookup back to one spelling.
- `uv run ruff check .` and `uv run ruff format --check` on the changed
files: clean.
- `uv lock --check`: consistent.
- `scripts/check_commit_messages.py`, `scripts/check_large_files.py`,
`scripts/check_source_language.py` over the branch range: clean.
- Manual acceptance against a real isolated install (temporary
RAVEN_HOME, isolated EverOS root and port), 34 cases: `raven plugins`,
`raven doctor` across four backend states, `raven onboard` interactive
and non-interactive including all three step-4 lanes, `raven import`
refusing and importing, an end-to-end write and recall, the web UI's
memory and settings pages driven through a real browser, a sub-agent
trace record, and an install with no memory plugin at all. 31 passed; 1
failed and was fixed; 1 partial; 1 carried over from the earlier round.
- The failure is the one worth reading. On a fresh managed install the
wizard reported embedding configured and the service it started the next
moment ran keyword-only: the spawned process carried no
`EVEROS_EMBEDDING__` values, and EverOS logged that the embedding key
was not configured. The rule deciding whether everos.toml already held
an operator's own endpoint asked whether its model name was a `"<...>"`
placeholder; the shipped template seeds every section with a real model
name and an empty key, so every fresh install read as a deliberate
choice and the host's endpoint was never bound. It now asks the
criterion the module already had -- model and key. The unit tests missed
it because each built its "operator's own" section with a key in it; the
regression copies the shipped template itself.
- The partial: the memory page's search returned relevant results
without error, but every memory in the test corpus was on one subject,
so narrowing could not be observed. The carried-over case is the
skill-source prefix, whose path this branch does not touch.
- Findings recorded and not fixed here, none introduced by this branch:
a progress spinner is still drawn after `import` reports it imported
nothing; an unsupported delete surfaces as a generic internal error; the
TUI log and `doctor` word "no memory plugin" differently; a turn that
did reach disk still prints that it was not written.
- The partial case: an induced agent skill is discovered through
`find_skill` with the backend's prefix, but `read_skill` / `use_skill`
on that id fail. Nothing writes the on-disk layer those tools look in,
on this branch or on `main`, and no test covers it. Pre-existing; filed
as a follow-up.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

User-visible changes:

1. User-track recall is now strictly bounded by `top_k`. A profile row
used to ride outside that bound.
2. `raven doctor` builds the plugin factory, so a managed EverOS setup
writes its env var and config templates during a doctor run. It used to
be read-only. The `--json` memory section is reshaped around
`health.checks`.
3. A third-party drop-in plugin without `enabled_by_default = true` used
to be skipped and now loads.
4. The loader no longer back-fills the EverOS config slice's `root` /
`owned`, nor strips a legacy `mode` key; a leftover `mode` now draws an
unknown-key warning.
5. The wizard's recap shows the contribution name `everos` rather than
`EverOS`.
6. `use_skill` with an unknown prefix reports "not found" after a disk
lookup instead of "unknown source".

From the follow-ups:

7. Config gains an optional top-level `embedding` block. Absent, the
knowledge base reads the EverOS file as before and says so once. An
operator who moves the values across changes nothing else; one who names
a *different* model there gets the existing `StaleBaseError` telling
them to rebuild that base. The block is a default for the memory plugin
rather than an override, so an install that had configured the plugin
separately keeps what it had. Note that a config carrying this block
does not load on an older Raven, which rejects unknown top-level keys.
8. `agents.<name>.everos` is written back as `agents.<name>.memory`. The
old spelling still loads; a per-agent `baseUrl` is ignored with a
warning, having never had a user.
9. Deleting a memory now changes what EverOS treats as the source of
truth, so it stays deleted. Profiles and cases report that they cannot
be deleted rather than appearing to succeed.
10. `pip install raven` no longer pulls the memory engine in.
`install.sh` and `install.ps1` are unaffected: both already resolve
`everos_memory-*.whl` from the release they install and pass it as
`--with everos-memory`, and the source path installs it editable, so
those installs keep long-term memory. An install that reaches neither --
a release predating the sibling wheel, or a bare `pip install` -- now
boots without memory and says which distribution to install rather than
failing on an import.

Found after the acceptance run, in review, all fixed here. They share
one cause worth stating once: the endpoint acquired a second home, and
each fix taught one more reader about it while leaving the next one
behind.

11. The endpoint the wizard saved never reached the EverOS process it
then started. The rule for "does everos.toml already hold an operator's
own endpoint" asked whether its model name was a `"<...>"` placeholder;
the shipped template seeds every section with a real model name and an
empty key, so every fresh managed install read as a deliberate choice
and the host's endpoint was never bound. The service came out of the
wizard running keyword-only while the wizard said embedding was
configured. The rule now asks the criterion the module already had:
model and key.
12. The exported `EVEROS_EMBEDDING__*` variables are a third home,
documented by EverOS and ranked above its file. They were being
overwritten. A complete triple is now an operator's choice; an
incomplete one is not an endpoint and is replaced whole rather than
topped up from the host.
13. Raven writes those same variables itself at startup, so a reader
that asked only whether all three were set called raven's own binding an
operator's export -- a blank settings card and a save refused by naming
variables nobody had set. The module now records which of them it set.
14. That record lived in a module-level set, and `raven gateway
--restart` re-launches through `os.execv`: the values survive, module
state does not. The record now travels in a variable beside them, and is
stripped from the environment handed to EverOS.
15. A settings save the environment would outrank is refused with the
variable names, rather than written and silently ignored.

Four of the five arrived one per review round, each in the fix for the
one before it. The tests that missed them all shared a shape: each built
its "an operator configured this" fixture with a key in it, or cleared
the environment before the case, so the states the product actually
produces -- a template with no key, a live process that has just bound
its own endpoint -- were never the input. The regressions now copy the
shipped template, and run the startup sequence and a second interpreter
rather than a prepared state.

Rollback is a revert of the branch: the seam is additive, and
`memory.backend` keeps its shipped default.

- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: gloryfromca <23442919+gloryfromca@users.noreply.github.com>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-16 22:19:50 +08:00

181 lines
7.4 KiB
Python

"""The contribution surface's ledger: the plugin-facing vocabulary, pinned.
``ServiceLocator`` and ``RuntimeHandles`` are papers now
(``raven.contracts.plugin_surface``, under the contract tier's version and
ledger); ``PluginManifest`` stays plugin-side because paper-izing a pydantic
model would pull pydantic into the kernel closure. This file keeps what the
tier ledger does not pin: the exact field rosters, the frozenness, the
manifest vocabulary, and the re-export address plugin authors import from.
"""
from __future__ import annotations
import dataclasses
def test_the_service_locator_grants_are_the_ledgered_nine() -> None:
from raven.plugins.context import ServiceLocator
assert sorted(f.name for f in dataclasses.fields(ServiceLocator)) == [
"agent_id",
# The host's embedding endpoint (model, base url, key, optional width).
# A credential grant, and a deliberate single source: raven.knowledge
# reads the same block, so one endpoint serves the installation rather
# than each feature carrying a copy to rotate. A backend that wants its
# own still has one -- this is what the host hands over, not a ceiling.
"embedding",
# The host's tools.media.<kind> section, read live and resolved as the
# host's own image_generate resolves it (paper: contracts/plugin_surface.py).
# A credential grant: whoever holds it can spend the deployment's image
# budget, so a plugin that generates pictures rides the host's tool and
# its usage ledger instead of carrying a key of its own.
"media_config",
# The proxy the host's media calls go through; generation state, not a
# preference, so it is a value rather than a reader.
"media_proxy",
"notify",
"provider",
"user_id",
# The host's tools.web section (proxy, search vendor keys), so a plugin
# with a web-facing tool defaults to the host's routing and credentials
# and its own slice keys become overrides rather than the only source.
"web_config",
"workspace",
], "a new field here is a new grant to every plugin factory: ledger it and say why"
assert ServiceLocator.__dataclass_params__.frozen, "grants are handed over, never handed back"
def test_the_runtime_handles_grants_are_ledgered() -> None:
from raven.plugins.context import RuntimeHandles
assert sorted(f.name for f in dataclasses.fields(RuntimeHandles)) == [
# The loop's own user-question face, lent to a bound holder. A BIG
# grant: whoever holds it can interrupt the user.
"direct_ask",
"playbook_runtime",
# Repoints one session's working root, durable truth first. A BIG
# grant: whoever holds it moves where every subsequent write lands.
"rebind_workdir",
"session_dir",
"subagent_registry",
"subagents_paused",
# Where a tool that bills per call reports its spend: the loop's own
# image-usage recorder. Hands over the power to write into the host's
# usage ledger, nothing more; None means the host keeps none.
"usage_recorder",
# Keyed one-shot wakes on the host scheduler, namespaced to the
# contributing plugin (paper: contracts/scheduling.py).
"wake_scheduler",
], "a new field here is a new late-bound grant: ledger it and name the power it hands over"
assert RuntimeHandles.__dataclass_params__.frozen, "grants are handed over, never handed back"
def test_the_manifest_kinds_are_ledgered() -> None:
from raven.plugins.manifest import Contributes, PluginManifest
assert sorted(Contributes.model_fields) == [
"hooks",
"memory_backends",
"onboard",
"services",
"session_observers",
"tool_gates",
"tools",
], (
"a new contribution kind changes what every raven-plugin.toml can say: "
"ledger it here in the change that teaches the registry to consume it"
)
assert sorted(PluginManifest.model_fields) == [
"bundled",
"config_schema",
"contributes",
"display_name",
"id",
"raven",
"version",
], "the manifest header is the plugin file format: a new key is a format change, ledger it"
def test_the_decline_vocabulary_is_two_shapes() -> None:
"""A factory declines by returning None (pinned by the plugin-tools
tests); a binder declines by raising the one sanctioned exception. Both
are configuration facts, not failures -- the ledger pins that the
exception exists on the contribution surface and is an Exception a loop
can catch narrowly.
"""
from raven.plugins.context import BindDeclinedError
assert issubclass(BindDeclinedError, Exception)
assert not issubclass(BindDeclinedError, (ValueError, TypeError, RuntimeError)), (
"the decline must stay its own class: a loop catches it narrowly, and riding a "
"builtin would catch real bugs as declines"
)
def test_the_documented_import_address_serves_the_papers_objects() -> None:
"""Plugin authors import from ``raven.plugins.context``; the definitions
live in the papers. Both spellings must hand out the same objects, or two
half-surfaces drift apart under one name."""
from raven.contracts import plugin_surface
from raven.plugins import context
assert context.ServiceLocator is plugin_surface.ServiceLocator
assert context.RuntimeHandles is plugin_surface.RuntimeHandles
assert context.BindDeclinedError is plugin_surface.BindDeclinedError
def test_a_service_contribution_parses_registers_and_builds(tmp_path) -> None:
"""[seam-2] The fourth kind travels the same road as the other three:
manifest row -> activation (duplicate names refused across plugins) ->
factory call with a fresh PluginContext -> an inert object the host may
start. The paper's face is what the built object satisfies."""
import textwrap
from types import SimpleNamespace
from raven.contracts.services import PluginService
from raven.plugins.manifest import PluginManifest
toml = textwrap.dedent("""
[plugin]
id = "watcher-plug"
version = "0.1.0"
[[plugin.contributes.services]]
name = "event_watcher"
factory = "raven.plugins.manifest:PluginManifest"
""")
mf = PluginManifest.from_toml_str(toml)
assert [s.name for s in mf.contributes.services] == ["event_watcher"]
assert mf.contributes.services[0].factory.endswith(":PluginManifest")
class _Svc:
def __init__(self) -> None:
self.started = []
async def start(self, handles) -> None:
self.started.append(handles)
async def stop(self) -> None:
pass
assert isinstance(_Svc(), PluginService), "the paper face is start/stop"
class _Reg:
def service_names(self):
return ["event_watcher"]
def service_plugin_id(self, name):
return "watcher-plug"
def build_service(self, name, config, services):
return _Svc()
from raven.core.plugin_stack import build_plugin_services
cfg = SimpleNamespace(
memory=SimpleNamespace(user_id="u", agent_id="a"),
plugins=SimpleNamespace(config={}),
)
built = build_plugin_services(tmp_path, cfg, registry=_Reg(), provider=None)
assert len(built) == 1
assert built[0].contributed_by == "watcher-plug"