## Summary The image role in Settings offered OpenRouter alone, although `image_generate` already speaks the OpenAI Images API (`/images/generations`, `/images/edits`) on any other base. A user with an OpenAI or OpenAI-compatible provider could reach it only by hand-editing `tools.media.image.apiKey` and `apiBase`. `tools.media.image` gains `provider`. A section naming one runs on that provider's address, key and headers as one group, taken from the endpoint the key comes from and resolved at call time the way the embedding pin is (`provider_connection`, factored out of `resolve_provider_credentials`); the section's own `apiKey`, `apiBase` and `extraHeaders` stand aside. Every media request sends the section's `extraHeaders` (a new field, secret like a provider's own) after the bearer key, the order the chat route gives a provider's `extra_headers`. The registry declares which providers the image tool can run on (`ProviderSpec.image_api`: OpenRouter, OpenAI, Custom), `model.options` carries the flag per row, the roles card offers exactly those for the image slot, and `settings.set` refuses any other. An OpenRouter pick writes an empty provider, so existing sections keep their current meaning. Speech and video keep to OpenRouter: their sections have no `provider` field, so one written there is ignored on every reader, and `settings.set` and `set_media` refuse it. Two fixes on the same path: - The OpenRouter key, `providers.openrouter.apiKey` or `OPENROUTER_API_KEY`, is borrowed only by a section that calls OpenRouter's address. A keyless section pointed at another endpoint used to send that endpoint the OpenRouter key as its bearer token. - Off OpenRouter, every image model goes to the Images API. The chat route with output modalities is OpenRouter's own; an OpenAI-compatible relay answered it with its own 404 (`model is not found`), which the OpenRouter-only fallback did not recognise, so a model outside the Images API name list could not generate there. The live config reader resolves the named provider's section exactly as boot does: every spelling, former name and field alias of it goes through `ProvidersConfig`, which folds them into one section, and no other provider's section is validated. A section header replaces the key's header of the same name in any case. raven-ppt and raven-design inherit the host's image section through the same resolver. When raven-ppt pins an image setting, the section becomes a snapshot read as written: it drops the provider name and takes the Wire Model that provider uses. A pinned key or base leaves the inherited headers behind, since they complete the host's key at the host's address. A pinned base also clears whatever key the host resolved, so the deck's key slot decides afresh: `PPT_IMAGE_API_KEY`, the host section's own key, or, toward OpenRouter only, the key that pays for the words. That closes a leak every 0.2 release has, the OpenRouter key the host borrowed for OpenRouter's address going to a pinned deck base. On a named provider's own address the deck sends that provider's key or none. The launcher can outlive its raven, since a stamped home tree outranks a checkout's own `agents/`, so it calls only what every 0.2 release has: the reader's two-argument form, and `media_provider` looked up where it exists. The settings dialog's design and acceptance specs (C14, A22, A24 and their decisions) are amended to the new rule. Overlap: open #432 and #433 (MiniMax images and video) edit `_generate_one`, the video tool's `execute`, a `_resolve_key` override on each tool, and the lines beside the removed `borrows_openrouter_key`. Their MiniMax routes return before the routing this change touches and their overrides delegate to the base resolver, so the conflicts are textual. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - `uv run --frozen --python 3.12 --all-extras pytest -q`: 27601 passed, 7 failed, 115 skipped. The 7 fail identically on `origin/main` 3632e6040 on this machine (five proxy tests in `test_config_update_providers.py` under the shell's proxy variables, `test_install_script.py::test_resolve_node_dir_answers_each_case_it_exists_for`, and `test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing` run as root); 0 introduced. - ui-web, `node node_modules/vitest/vitest.mjs run --no-file-parallelism` (both test roots, 214 files): 3188 passed, 0 failed, 0 unhandled errors. Measured before the review commits, which touch no TypeScript, generated file or RPC contract. - ui-tui, the same command (144 files): 2089 passed, 9 failed, 0 unhandled errors. The 9 are rendering tests in four files that fail with identical ids on `origin/main`'s ui-tui on this machine; 0 introduced. - `tsc --noEmit` in ui-web and ui-tui, `gen-rpc-client.mjs --check`, `gen-rpc-types.mjs --check`, eslint over the touched ui-web directories, and `ruff format --check` / `ruff check` over the changed Python files: all clean. - `scripts/check_commit_messages.py origin/main..HEAD`, commitlint over the same range, `scripts/check_source_language.py origin/main...HEAD`, `scripts/check_large_files.py origin/main...HEAD`: all exit 0. - The new tests were written first and watched fail for the missing behaviour, the review rounds' tests on the head before each fix. A mutation pass then removed each construct this change adds, one at a time: 40 Python and 3 TypeScript mutants, all killed but one, controls green. The survivor matches a provider on its name instead of its `route_names`, which is equivalent today because every former name in the registry is also a field alias. - Each commit added in review passes the touched test files on its own. - Live, against an OpenAI-compatible relay serving `sensenova-u1.5-lite`: before, `image_generate` with the section's own `apiKey`/`apiBase` posted to `/chat/completions` and got 404. After, on an isolated `raven web` in Chromium, the image row offered Custom, OpenAI and OpenRouter (speech and video OpenRouter only), the typed id was added to Custom as an image model and saved as `{model, quality, provider: "custom"}`, and one call through the live config reader returned a 2048x1536 PNG from `/images/generations` with the Custom provider's key. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed No user-facing documentation covers the media tools; the settings dialog's specs carry the rule. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes - Security: credentials travel to fewer places. A section borrows only from the provider it names, address and key together, and a provider with no known address supplies no key. `tools.media.image.provider` joins the `settings.set` whitelist with the reach the subsystem pins already have: it moves the tool to a provider the operator configured, and names no address and carries no key. - Provider headers can carry credentials: they travel with the key to the provider's own address only, a video poll or content URL on another origin gets neither, and a media section's `extraHeaders` is masked like a provider's. The OpenRouter default lane still borrows only the OpenRouter key, not that provider's headers. - A keyless image, speech or video section whose `apiBase` is not OpenRouter's now reports a missing key instead of borrowing the OpenRouter key. A relay in front of OpenRouter at another address needs `apiKey` on the section. - On such a base, a model outside the Images API name list (a Gemini image model, say) now goes to `/images/generations` instead of `/chat/completions`. - Choosing a provider other than OpenRouter in Settings sets aside a hand-written `apiKey`/`apiBase` on the image section; they stay in the file and apply again once an OpenRouter model is picked. - A `provider` hand-written into `tools.media.speech` or `tools.media.video` is now ignored: that section runs on OpenRouter, with the OpenRouter key. - raven-ppt: a deck pinning `PPT_IMAGE_API_KEY` no longer sends the host's `extraHeaders`, so an endpoint that also routes by header cannot take the deck's own key; before this change no header travelled at all. A deck pinning `PPT_IMAGE_API_BASE` without a key of its own no longer receives the OpenRouter key the host borrowed; it needs `PPT_IMAGE_API_KEY`, the host section's own key, or a base on OpenRouter's address. - The raven-ppt launcher still renders under every 0.2 release, which lacks `media_provider`; a test runs it against a reader with the 0.2 signature. - Rollback: revert the squash commit. The previous config models ignore a saved `provider`, so the image tool returns to OpenRouter with the earlier key rules. ## Related Issues Fixes #828 --------- Co-authored-by: Claude (claude-opus-5-5[1m]) <noreply@anthropic.com>
634 lines
27 KiB
Python
634 lines
27 KiB
Python
"""The capability table must describe what AgentLoop actually does.
|
|
|
|
A second description of an existing rule is only worth having while it stays
|
|
true. These drive a real ``AgentLoop`` and compare what it offers the model
|
|
against what the table predicts, for every combination that changes an answer --
|
|
so the table cannot quietly become a third opinion about which tools are
|
|
available. Registration stopped being the gate: every gated tool is registered
|
|
now and *withheld* while unconfigured, so the predicate a deployer feels is
|
|
``offers_by_name``, never ``has``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import tempfile
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from raven.agent.loop import AgentLoop
|
|
from raven.agent.tools.capabilities import (
|
|
CAPABILITIES,
|
|
Need,
|
|
borrowable_credential,
|
|
configured_from,
|
|
disabled_by,
|
|
has_credential,
|
|
is_configured,
|
|
is_disabled,
|
|
is_offered,
|
|
)
|
|
from raven.config.loader import load_config
|
|
from raven.providers.base import LLMProvider, LLMResponse
|
|
from tests._wiring import wire
|
|
|
|
|
|
class _StubProvider(LLMProvider):
|
|
def __init__(self) -> None:
|
|
super().__init__(api_key="test")
|
|
|
|
async def chat(
|
|
self,
|
|
messages,
|
|
tools=None,
|
|
model=None,
|
|
max_tokens=4096,
|
|
temperature=0.7,
|
|
reasoning_effort=None,
|
|
tool_choice=None,
|
|
):
|
|
return LLMResponse(content="stub", finish_reason="stop")
|
|
|
|
def get_default_model(self) -> str:
|
|
return "stub"
|
|
|
|
|
|
@pytest.fixture
|
|
def workspace():
|
|
with tempfile.TemporaryDirectory() as td:
|
|
yield Path(td)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _no_ambient_keys(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Every credential here also resolves from the environment, so a developer
|
|
who exported one would otherwise see these pass for the wrong reason.
|
|
|
|
``OPENROUTER_API_KEY`` matters as much as the search key: it is a third
|
|
source for the media family, behind the section and the borrow, and a case
|
|
asserting that nothing was available to borrow silently stops testing that
|
|
on any machine that exports it.
|
|
"""
|
|
for var in ("SERPER_API_KEY", "OPENROUTER_API_KEY"):
|
|
monkeypatch.delenv(var, raising=False)
|
|
|
|
|
|
def _config(tmp_path: Path):
|
|
"""A real Config with genuine defaults, independent of whoever runs this."""
|
|
return load_config(tmp_path / "absent.json")
|
|
|
|
|
|
def _resolved(config, attr):
|
|
return getattr(config.effective_media_config(), attr)
|
|
|
|
|
|
def _media_attrs() -> list[str]:
|
|
"""Every media tool the schema declares, asked of the schema.
|
|
|
|
Listing them here instead would make the coverage test below blind to the
|
|
one case it exists to catch: a media tool added to the schema and to
|
|
registration but not to the table. A hardcoded list never configures the new
|
|
tool, so it never appears among the gated ones, so the assertion holds while
|
|
the table is already wrong.
|
|
"""
|
|
from raven.config.schema import MediaGenConfig, MediaToolConfig
|
|
|
|
return [
|
|
n
|
|
for n, f in MediaGenConfig.model_fields.items()
|
|
if isinstance(f.annotation, type) and issubclass(f.annotation, MediaToolConfig)
|
|
]
|
|
|
|
|
|
def _loop(workspace: Path, config, **kw) -> AgentLoop:
|
|
return AgentLoop(
|
|
provider=_StubProvider(),
|
|
workspace=workspace,
|
|
model="stub",
|
|
**wire(media_config=config.effective_media_config(), **kw),
|
|
)
|
|
|
|
|
|
def test_the_table_names_exactly_the_credential_gated_tools(workspace, tmp_path: Path) -> None:
|
|
"""Derived rather than listed: the tools that appear only once credentials
|
|
are supplied *are* the credential-gated ones, so this catches both halves --
|
|
a table entry for a tool that no longer exists, and a newly gated tool whose
|
|
author forgot the table. The second is the one that puts a deployer back to
|
|
guessing what is missing."""
|
|
bare = _config(tmp_path)
|
|
loop_bare = _loop(workspace, bare)
|
|
|
|
full = _config(tmp_path)
|
|
full.tools.web.search.api_key = "sk-serper"
|
|
# image_search is gated on a key like web_search, but registered only where
|
|
# `tools.web.search.images` asks for it, so "everything supplied" has to
|
|
# turn the switch on too or the tool never appears among the gated ones.
|
|
full.tools.web.search.images = True
|
|
for attr in _media_attrs():
|
|
getattr(full.tools.media, attr).model = "some/model"
|
|
full.providers.openrouter.api_key = "sk-or-test"
|
|
loop_full = _loop(workspace, full, search_api_key="sk-serper", image_search=True)
|
|
|
|
def offered(loop) -> set[str]:
|
|
return {n for n in loop.tools.tool_names if loop.tools.offers_by_name(n)}
|
|
|
|
gated = offered(loop_full) - offered(loop_bare)
|
|
declared = {c.tool for c in CAPABILITIES if c.need is not Need.NOTHING}
|
|
|
|
assert gated == declared, (
|
|
f"gated but undeclared: {sorted(gated - declared)}; declared but not gated: {sorted(declared - gated)}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("cap", CAPABILITIES, ids=lambda c: c.tool)
|
|
def test_the_table_agrees_with_the_loop_when_unconfigured(cap, workspace, tmp_path: Path) -> None:
|
|
config = _config(tmp_path)
|
|
loop = _loop(workspace, config)
|
|
|
|
assert is_configured(cap, config) is loop.tools.offers_by_name(cap.tool), (
|
|
f"{cap.tool}: table says configured={is_configured(cap, config)}, "
|
|
f"loop offers={loop.tools.offers_by_name(cap.tool)}"
|
|
)
|
|
|
|
|
|
def test_a_configured_search_key_agrees_on_both_sides(workspace, tmp_path: Path) -> None:
|
|
config = _config(tmp_path)
|
|
config.tools.web.search.api_key = "sk-serper"
|
|
loop = _loop(workspace, config, search_api_key="sk-serper")
|
|
|
|
cap = next(c for c in CAPABILITIES if c.tool == "web_search")
|
|
assert is_configured(cap, config) and loop.tools.offers_by_name("web_search")
|
|
assert configured_from(cap, config) == "tools.web.search.apiKey"
|
|
|
|
|
|
def test_the_env_var_alone_agrees_on_both_sides(workspace, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The half of the deploys a config-only reading would answer wrong."""
|
|
monkeypatch.setenv("SERPER_API_KEY", "sk-from-env")
|
|
config = _config(tmp_path)
|
|
loop = _loop(workspace, config)
|
|
|
|
cap = next(c for c in CAPABILITIES if c.tool == "web_search")
|
|
assert is_configured(cap, config) and loop.tools.offers_by_name("web_search")
|
|
assert configured_from(cap, config) == "SERPER_API_KEY"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("attr", "tool"),
|
|
[("image", "image_generate"), ("speech", "text_to_speech"), ("video", "video_generate")],
|
|
)
|
|
def test_a_media_model_alone_agrees_on_both_sides(attr, tool, workspace, tmp_path: Path) -> None:
|
|
"""A model with no key is the switch-on case: the key is borrowed."""
|
|
config = _config(tmp_path)
|
|
getattr(config.tools.media, attr).model = "some/model"
|
|
config.providers.openrouter.api_key = "sk-or-test"
|
|
loop = _loop(workspace, config)
|
|
|
|
cap = next(c for c in CAPABILITIES if c.tool == tool)
|
|
assert is_configured(cap, config) and loop.tools.offers_by_name(tool)
|
|
assert configured_from(cap, config) == "borrowed: providers.openrouter.apiKey"
|
|
|
|
|
|
def test_an_openrouter_key_alone_switches_nothing_on(workspace, tmp_path: Path) -> None:
|
|
"""The property that makes the media rule "model *or* key" rather than
|
|
"resolved key": a chat credential must not silently enable three paid tools."""
|
|
config = _config(tmp_path)
|
|
config.providers.openrouter.api_key = "sk-or-test"
|
|
loop = _loop(workspace, config)
|
|
|
|
for cap in (c for c in CAPABILITIES if c.media_attr):
|
|
assert not is_configured(cap, config), cap.tool
|
|
assert not loop.tools.offers_by_name(cap.tool), cap.tool
|
|
|
|
|
|
def test_the_free_capability_needs_no_credential(workspace, tmp_path: Path) -> None:
|
|
config = _config(tmp_path)
|
|
loop = _loop(workspace, config)
|
|
|
|
cap = next(c for c in CAPABILITIES if c.need is Need.NOTHING)
|
|
assert is_configured(cap, config) and loop.tools.offers_by_name(cap.tool)
|
|
assert configured_from(cap, config) == "", "nothing was required, so nothing supplied it"
|
|
|
|
|
|
def test_every_entry_carries_what_a_deployer_has_to_act_on() -> None:
|
|
"""The table is read by someone deciding what to go and do, so the fields
|
|
that tell them are not optional for the needs that require action."""
|
|
for cap in CAPABILITIES:
|
|
assert cap.summary and cap.tool
|
|
if cap.need is Need.NEW_ACCOUNT:
|
|
assert cap.obtain_from, f"{cap.tool}: no url for an account the deployer must create"
|
|
assert cap.config_path, f"{cap.tool}: no config path to put the key in"
|
|
if cap.need is Need.OWN_CREDENTIAL:
|
|
assert cap.cost_note, f"{cap.tool}: switched on without saying it bills per call"
|
|
# The row that says "you already have this key" has to name where,
|
|
# and the doctor prints both unguarded -- a blank one renders as an
|
|
# instruction with the answer missing.
|
|
assert cap.env_var, f"{cap.tool}: reuses a credential without naming the variable"
|
|
assert cap.key_path != cap.config_path, f"{cap.tool}: model path doubling as the key path"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("attr", "tool"),
|
|
[("image", "image_generate"), ("speech", "text_to_speech"), ("video", "video_generate")],
|
|
)
|
|
def test_a_media_model_with_no_key_to_borrow_still_counts(attr, tool, workspace, tmp_path: Path) -> None:
|
|
"""The half of the media rule the borrow hides.
|
|
|
|
With an OpenRouter key present, `effective_media_config` fills `api_key` in,
|
|
so a rule reading only the key still answers correctly and a test that sets
|
|
both proves nothing about the `or model` half. With no key to borrow, the
|
|
model is the only thing making this configured -- and the loop registers it,
|
|
so the table must agree.
|
|
"""
|
|
config = _config(tmp_path)
|
|
getattr(config.tools.media, attr).model = "some/model"
|
|
assert not config.providers.openrouter.api_key, "this case needs nothing to borrow"
|
|
loop = _loop(workspace, config)
|
|
|
|
cap = next(c for c in CAPABILITIES if c.tool == tool)
|
|
assert not _resolved(config, attr).api_key, "nothing should have been borrowed"
|
|
assert is_configured(cap, config) and loop.tools.offers_by_name(tool)
|
|
# Offered, and unusable: no key resolves from the section, the borrow, or
|
|
# the environment, so every call returns the tool's missing-key error. There
|
|
# is no source to name, and naming the model path -- the only path this
|
|
# capability has -- would tell the deployer a key sits somewhere it does not.
|
|
assert configured_from(cap, config) == ""
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("attr", "tool"),
|
|
[("image", "image_generate"), ("speech", "text_to_speech"), ("video", "video_generate")],
|
|
)
|
|
def test_a_media_key_is_reported_at_its_own_path_not_the_model_path(attr, tool, workspace, tmp_path: Path) -> None:
|
|
"""``config_path`` names the model for this family, so reusing it as the
|
|
credential source sends the deployer to edit a line holding no key."""
|
|
config = _config(tmp_path)
|
|
getattr(config.tools.media, attr).api_key = "sk-tool-own"
|
|
loop = _loop(workspace, config)
|
|
|
|
cap = next(c for c in CAPABILITIES if c.tool == tool)
|
|
assert is_configured(cap, config) and loop.tools.offers_by_name(tool)
|
|
assert configured_from(cap, config) == f"tools.media.{attr}.apiKey"
|
|
assert configured_from(cap, config) != cap.config_path
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("attr", "tool"),
|
|
[("image", "image_generate"), ("speech", "text_to_speech"), ("video", "video_generate")],
|
|
)
|
|
def test_a_media_key_from_the_environment_is_a_source_like_any_other(
|
|
attr, tool, monkeypatch: pytest.MonkeyPatch, workspace, tmp_path: Path
|
|
) -> None:
|
|
"""The tool resolves ``OPENROUTER_API_KEY`` at call time, so a report that
|
|
consults only config answers "no credential" for a working install."""
|
|
monkeypatch.setenv("OPENROUTER_API_KEY", "sk-from-env")
|
|
config = _config(tmp_path)
|
|
getattr(config.tools.media, attr).model = "some/model"
|
|
loop = _loop(workspace, config)
|
|
|
|
cap = next(c for c in CAPABILITIES if c.tool == tool)
|
|
assert is_configured(cap, config) and loop.tools.offers_by_name(tool)
|
|
assert configured_from(cap, config) == "OPENROUTER_API_KEY"
|
|
# The half that decides whether the row carries a warning. This install
|
|
# works, so calling it keyless would send a deployer to fix what is not
|
|
# broken -- and only the tool's chain reaches the variable to know that.
|
|
assert has_credential(cap, config)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("attr", "tool"),
|
|
[("image", "image_generate"), ("speech", "text_to_speech"), ("video", "video_generate")],
|
|
)
|
|
def test_no_borrow_is_claimed_when_there_is_nothing_to_borrow(attr, tool, tmp_path: Path) -> None:
|
|
"""The instruction "reuse the key you already have" is wrong when no key
|
|
exists, and wrong in the expensive direction: it is the reason a deployer
|
|
sets a model, gets a registered tool, and sees every call fail."""
|
|
config = _config(tmp_path)
|
|
cap = next(c for c in CAPABILITIES if c.tool == tool)
|
|
assert not config.providers.openrouter.api_key, "this case needs nothing to borrow"
|
|
|
|
assert borrowable_credential(cap, config) == ""
|
|
|
|
config.providers.openrouter.api_key = "sk-or-test"
|
|
assert borrowable_credential(cap, config) == "providers.openrouter.apiKey"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("attr", "tool"),
|
|
[("image", "image_generate"), ("speech", "text_to_speech"), ("video", "video_generate")],
|
|
)
|
|
def test_an_exported_key_is_reusable_too(attr, tool, monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
|
"""A deployment that exports the key and configures nothing needs the model
|
|
and nothing else, so telling it to go and set a key is the same misreport in
|
|
the other direction."""
|
|
monkeypatch.setenv("OPENROUTER_API_KEY", "sk-from-env")
|
|
config = _config(tmp_path)
|
|
cap = next(c for c in CAPABILITIES if c.tool == tool)
|
|
assert not config.providers.openrouter.api_key, "the environment must be the only source"
|
|
|
|
assert borrowable_credential(cap, config) == "OPENROUTER_API_KEY"
|
|
|
|
|
|
def test_a_media_section_on_a_named_provider_is_paid_by_that_provider(workspace, tmp_path: Path) -> None:
|
|
"""The provider the section names supplies its address and its key together; the
|
|
OpenRouter key on the same install is not consulted."""
|
|
config = _config(tmp_path)
|
|
config.tools.media.image.model = "openai/gpt-image-2"
|
|
config.tools.media.image.provider = "openai"
|
|
config.providers.openai.api_key = "sk-openai"
|
|
config.providers.openrouter.api_key = "sk-or-test"
|
|
loop = _loop(workspace, config)
|
|
|
|
cap = next(c for c in CAPABILITIES if c.tool == "image_generate")
|
|
resolved = _resolved(config, "image")
|
|
assert (resolved.api_base, resolved.api_key) == ("https://api.openai.com/v1", "sk-openai")
|
|
assert is_configured(cap, config) and loop.tools.offers_by_name("image_generate")
|
|
assert configured_from(cap, config) == "borrowed: providers.openai.apiKey"
|
|
assert has_credential(cap, config)
|
|
|
|
|
|
def test_a_named_provider_whose_section_does_not_validate_lends_nothing_at_boot() -> None:
|
|
"""An undeclared provider's section loads as raw data and is validated only when
|
|
read, so the boot resolution reads it as absent rather than raising on start."""
|
|
from raven.config.schema import Config
|
|
|
|
config = Config.model_validate(
|
|
{
|
|
"providers": {"weirdvendor": {"apiKey": 123}},
|
|
"tools": {"media": {"image": {"model": "m", "provider": "weirdvendor"}}},
|
|
}
|
|
)
|
|
resolved = _resolved(config, "image")
|
|
assert (resolved.api_base, resolved.api_key) == ("", "")
|
|
|
|
|
|
def test_an_unconfigured_section_on_a_named_provider_would_borrow_that_providers_key(tmp_path: Path) -> None:
|
|
config = _config(tmp_path)
|
|
config.tools.media.image.provider = "openai"
|
|
config.providers.openrouter.api_key = "sk-or-test"
|
|
cap = next(c for c in CAPABILITIES if c.tool == "image_generate")
|
|
assert borrowable_credential(cap, config) == ""
|
|
|
|
config.providers.openai.api_key = "sk-openai"
|
|
assert borrowable_credential(cap, config) == "providers.openai.apiKey"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("attr", "tool"),
|
|
[("image", "image_generate"), ("speech", "text_to_speech"), ("video", "video_generate")],
|
|
)
|
|
def test_a_media_section_pointed_elsewhere_claims_no_openrouter_key(
|
|
attr, tool, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
"""Neither OpenRouter credential reaches a section whose address is somewhere else,
|
|
so neither may be reported as its source -- that row would read as working."""
|
|
monkeypatch.setenv("OPENROUTER_API_KEY", "sk-from-env")
|
|
config = _config(tmp_path)
|
|
config.providers.openrouter.api_key = "sk-or-test"
|
|
section = getattr(config.tools.media, attr)
|
|
section.api_base = "https://relay.test/v1"
|
|
cap = next(c for c in CAPABILITIES if c.tool == tool)
|
|
assert borrowable_credential(cap, config) == ""
|
|
|
|
section.model = "some/model"
|
|
assert not _resolved(config, attr).api_key
|
|
assert configured_from(cap, config) == ""
|
|
assert not has_credential(cap, config)
|
|
|
|
|
|
def test_only_the_media_family_borrows(tmp_path: Path) -> None:
|
|
"""web_search cannot reuse anything -- its row already names what to get --
|
|
and web_fetch needs nothing at all."""
|
|
config = _config(tmp_path)
|
|
config.providers.openrouter.api_key = "sk-or-test"
|
|
|
|
for cap in (c for c in CAPABILITIES if not c.media_attr):
|
|
assert borrowable_credential(cap, config) == "", cap.tool
|
|
|
|
|
|
def _on_offer(loop, tool: str) -> bool:
|
|
"""What the agent is actually handed: registered, and not withheld now.
|
|
|
|
Registry membership alone is not the answer here -- a disabled tool stays
|
|
registered and is dropped when the tool array is built.
|
|
"""
|
|
return loop.tools.offers_by_name(tool) and tool not in loop._withheld_tool_names()
|
|
|
|
|
|
def test_a_switched_off_tool_is_configured_and_still_not_offered(workspace, tmp_path: Path) -> None:
|
|
"""The combination the report used to get wrong.
|
|
|
|
A key is set, so the credential gate is satisfied and saying "unconfigured"
|
|
would send the deployer to set it again. What decides whether the agent is
|
|
offered the tool is `tools.disabledTools` -- so the predicate that has to
|
|
agree with the loop is `is_offered`, not `is_configured`.
|
|
|
|
Upstream compared `is_offered` against registry membership, because there a
|
|
disabled tool is unregistered at startup. This loop keeps it registered and
|
|
withholds it per request (`_withheld_tool_names`), so that a switch flipped
|
|
now takes effect on the next turn instead of destroying its own subject.
|
|
What the agent is offered is therefore "registered and not withheld".
|
|
"""
|
|
config = _config(tmp_path)
|
|
config.tools.web.search.api_key = "sk-serper"
|
|
config.tools.disabled_tools = ["web_search"]
|
|
# Passed in, the way the CLI entry points do: the loop takes the list as an
|
|
# argument rather than reading the config.
|
|
loop = _loop(workspace, config, search_api_key="sk-serper", disabled_tools=config.tools.disabled_tools)
|
|
cap = next(c for c in CAPABILITIES if c.tool == "web_search")
|
|
|
|
assert _on_offer(loop, "web_search") is False
|
|
assert is_configured(cap, config) is True
|
|
assert is_disabled(cap, config) is True
|
|
assert is_offered(cap, config) is _on_offer(loop, "web_search")
|
|
|
|
|
|
@pytest.mark.parametrize("cap", CAPABILITIES, ids=lambda c: c.tool)
|
|
def test_being_offered_matches_the_registry_for_every_capability(cap, workspace, tmp_path: Path) -> None:
|
|
"""Every entry, switched on and then off by name, against the real loop.
|
|
|
|
Parametrised rather than written for web_search alone: the disabled list
|
|
covers any tool, and a family that stops agreeing is exactly the drift the
|
|
rest of this file exists to catch.
|
|
"""
|
|
config = _config(tmp_path)
|
|
config.tools.web.search.api_key = "sk-serper"
|
|
config.tools.web.search.images = True
|
|
for attr in _media_attrs():
|
|
getattr(config.tools.media, attr).model = "some/model"
|
|
config.providers.openrouter.api_key = "sk-or-test"
|
|
|
|
on = _loop(workspace, config, search_api_key="sk-serper", image_search=True)
|
|
assert is_offered(cap, config) is _on_offer(on, cap.tool)
|
|
|
|
config.tools.disabled_tools = [cap.tool]
|
|
off = _loop(
|
|
workspace,
|
|
config,
|
|
search_api_key="sk-serper",
|
|
image_search=True,
|
|
disabled_tools=config.tools.disabled_tools,
|
|
)
|
|
assert is_offered(cap, config) is _on_offer(off, cap.tool)
|
|
assert _on_offer(off, cap.tool) is False
|
|
|
|
|
|
def test_the_picture_switch_reads_as_off_rather_than_unconfigured(workspace, tmp_path: Path) -> None:
|
|
"""A Serper key with `tools.web.search.images` left off.
|
|
|
|
The credential gate is satisfied and the loop still holds no image_search,
|
|
so a table that answered on the key alone would tick a capability the agent
|
|
does not have -- and would send the deployer to set a key already there.
|
|
"""
|
|
config = _config(tmp_path)
|
|
config.tools.web.search.api_key = "sk-serper"
|
|
loop = _loop(workspace, config, search_api_key="sk-serper")
|
|
cap = next(c for c in CAPABILITIES if c.tool == "image_search")
|
|
|
|
assert _on_offer(loop, "image_search") is False
|
|
assert is_configured(cap, config) is True
|
|
assert is_disabled(cap, config) is True
|
|
assert is_offered(cap, config) is _on_offer(loop, "image_search")
|
|
# The setting a report must name: `disabledTools` need not contain the
|
|
# tool, and editing it cannot turn this one on.
|
|
assert disabled_by(cap, config) == "tools.web.search.images"
|
|
|
|
|
|
def test_the_off_switch_each_capability_names_is_the_one_that_undoes_it(workspace, tmp_path: Path) -> None:
|
|
"""Both branches of the field, and the order where both are shut.
|
|
|
|
The registration gate is named ahead of the deny list because taking
|
|
image_search out of `disabledTools` with pictures off restarts into the
|
|
same missing tool.
|
|
"""
|
|
config = _config(tmp_path)
|
|
config.tools.web.search.api_key = "sk-serper"
|
|
config.tools.web.search.images = True
|
|
pictures = next(c for c in CAPABILITIES if c.tool == "image_search")
|
|
search = next(c for c in CAPABILITIES if c.tool == "web_search")
|
|
|
|
assert disabled_by(pictures, config) == ""
|
|
assert disabled_by(search, config) == ""
|
|
|
|
config.tools.disabled_tools = ["web_search", "image_search"]
|
|
assert disabled_by(search, config) == "tools.disabledTools"
|
|
assert disabled_by(pictures, config) == "tools.disabledTools"
|
|
|
|
config.tools.web.search.images = False
|
|
assert disabled_by(pictures, config) == "tools.web.search.images"
|
|
|
|
|
|
def test_a_media_key_added_after_start_surfaces_the_tool(workspace, tmp_path: Path, monkeypatch) -> None:
|
|
"""The whole point of moving the gate to the withheld axis: the user edits
|
|
the config file, nothing re-registers, and the next assembly offers the tool
|
|
with the very key the call will use."""
|
|
cfg_file = tmp_path / "config.json"
|
|
cfg_file.write_text(json.dumps({}), encoding="utf-8")
|
|
monkeypatch.setattr("raven.home._current_config_path", cfg_file)
|
|
loop = _loop(workspace, _config(tmp_path))
|
|
assert not loop.tools.offers_by_name("image_generate")
|
|
|
|
cfg_file.write_text(json.dumps({"tools": {"media": {"image": {"apiKey": "sk-added"}}}}), encoding="utf-8")
|
|
|
|
assert loop.tools.offers_by_name("image_generate")
|
|
assert loop.tools.get("image_generate").api_key == "sk-added"
|
|
|
|
|
|
def test_a_media_key_removed_after_start_withdraws_the_tool(workspace, tmp_path: Path, monkeypatch) -> None:
|
|
cfg_file = tmp_path / "config.json"
|
|
cfg_file.write_text(json.dumps({"tools": {"media": {"image": {"apiKey": "sk-was-here"}}}}), encoding="utf-8")
|
|
monkeypatch.setattr("raven.home._current_config_path", cfg_file)
|
|
config = load_config(cfg_file)
|
|
loop = _loop(workspace, config)
|
|
assert loop.tools.offers_by_name("image_generate")
|
|
|
|
cfg_file.write_text(json.dumps({"tools": {"media": {"image": {}}}}), encoding="utf-8")
|
|
|
|
assert not loop.tools.offers_by_name("image_generate")
|
|
|
|
|
|
def test_a_plugin_shadowing_a_media_tool_is_not_gated_by_the_builtin_config(workspace, tmp_path: Path) -> None:
|
|
from raven.contracts.tool import Tool
|
|
|
|
class _PluginImage(Tool):
|
|
name = "image_generate"
|
|
description = "plugin image tool with its own credential story"
|
|
parameters = {"type": "object", "properties": {}}
|
|
|
|
async def execute(self, **kwargs):
|
|
return "plugin ran"
|
|
|
|
plugin = _PluginImage()
|
|
loop = _loop(workspace, _config(tmp_path), plugin_tools=[plugin])
|
|
|
|
assert loop.tools.get("image_generate") is plugin
|
|
assert loop.tools.offers_by_name("image_generate"), "the built-in's empty section must not gate the plugin"
|
|
|
|
|
|
def test_a_plugin_tool_that_declares_configured_rides_the_live_withheld_lane(workspace, tmp_path: Path) -> None:
|
|
"""The paper in plugin_surface.py: a contributed tool that declares `configured()`
|
|
is asked per assembly and withheld while it answers False -- offered the moment
|
|
the deployment configures it, withdrawn the moment it does not, no restart."""
|
|
from raven.contracts.tool import Tool
|
|
|
|
state = {"configured": False}
|
|
|
|
class _DeckPicture(Tool):
|
|
name = "deck_picture"
|
|
description = "a plugin tool that bills against a credential the host may not have"
|
|
parameters = {"type": "object", "properties": {}}
|
|
|
|
def configured(self) -> bool:
|
|
return state["configured"]
|
|
|
|
async def execute(self, **kwargs):
|
|
return "drawn"
|
|
|
|
loop = _loop(workspace, _config(tmp_path), plugin_tools=[_DeckPicture()])
|
|
assert loop.tools.has("deck_picture"), "withheld, not unregistered"
|
|
assert not loop.tools.offers_by_name("deck_picture")
|
|
state["configured"] = True
|
|
assert loop.tools.offers_by_name("deck_picture")
|
|
state["configured"] = False
|
|
assert not loop.tools.offers_by_name("deck_picture")
|
|
|
|
|
|
def test_a_plugin_subclass_of_a_media_tool_is_not_gated_either(workspace, tmp_path: Path) -> None:
|
|
from raven.agent.tools.media_gen import ImageGenerateTool
|
|
from raven.config.schema import MediaToolConfig
|
|
|
|
class _PluginSubclassImage(ImageGenerateTool):
|
|
def __init__(self) -> None:
|
|
super().__init__(MediaToolConfig(api_key="sk-plugin-own", model="plugin/model"))
|
|
|
|
plugin = _PluginSubclassImage()
|
|
loop = _loop(workspace, _config(tmp_path), plugin_tools=[plugin])
|
|
|
|
assert loop.tools.get("image_generate") is plugin
|
|
assert loop.tools.offers_by_name("image_generate"), "identity, not type: a subclass is not the built-in"
|
|
|
|
|
|
def test_the_web_rows_resolve_to_the_selected_vendor(tmp_path: Path) -> None:
|
|
"""The table is static; the deployer's instructions are not. A Tavily
|
|
deployment is sent to Tavily's slot, env var and sign-up page, and the
|
|
source named for a satisfied row is the slot that actually holds the key."""
|
|
from raven.agent.tools.capabilities import configured_from, resolve
|
|
|
|
config = _config(tmp_path)
|
|
config.tools.web.search.provider = "tavily"
|
|
config.tools.web.fetch.provider = "firecrawl"
|
|
rows = {c.tool: resolve(c, config) for c in CAPABILITIES}
|
|
assert rows["web_search"].config_path == "tools.web.providers.tavily.apiKey"
|
|
assert rows["web_search"].env_var == "TAVILY_API_KEY"
|
|
assert rows["web_search"].obtain_from == "https://tavily.com"
|
|
assert rows["web_fetch"].config_path == "tools.web.providers.firecrawl.apiKey"
|
|
assert rows["web_fetch"].need is Need.NOTHING, "web_fetch is always offered; a keyless reader falls back to Jina"
|
|
|
|
search = next(c for c in CAPABILITIES if c.tool == "web_search")
|
|
config.tools.web.providers.tavily.api_key = "tv"
|
|
assert configured_from(search, config) == "tools.web.providers.tavily.apiKey"
|
|
|
|
legacy = _config(tmp_path)
|
|
legacy.tools.web.search.api_key = "sk-serper"
|
|
assert configured_from(search, legacy) == "tools.web.search.apiKey", "the pre-vendor leaf is named as itself"
|