Takes the web UI's agents page (rail entry "Agent Hub", page "Agents") to the prototype's agent hub, and gives the wire what the prototype needed and did not have: a default model on an agent row. Base: `refactor/ui_web_architecture`, the long-lived architecture branch (#475), not `main`. The diff shown here is the diff that merges into that branch. What the page does now: - Three sections, never folded: connected, available to connect, not installed on this machine. One control per row (connect / disconnect / retry / install), a status dot only where there is something to say, and a one-line summary from a small catalogue of what each agent is and who makes it. - A 520x500 sheet in the shared detail host: what the agent is good at (editable, saved on blur), the API key where one is needed, the install command and site for an absent agent with a re-check, a Test button, and the model pill. - A write in flight is held on the row as "connecting" for its length; a refusal -- a model pick included -- stays on the row as red text with Retry rather than as a toast. A running test keeps a Stop control. The model pill, by the row's own rule (`model_source`, a new wire field): - The built-in row picks from raven's connected providers. The pick is stored naming its provider (`stored_model_id`, the way `config.set model` stores the host's), validated against a provider raven knows that holds a usable credential, and paired with that credential on every dispatch through `ProviderPool.bind_pin` -- the backend resolves the pin per run, so a cached backend never runs one conversation's key against another's model. The write and the dispatch read the provider off the stored id through one function, `stored_provider_name` (the inverse of `stored_model_id`), so a section raven has no spec for resolves to that section on both sides. - An acp row, raven's own shipped products included, picks from the choices its handshake advertised (`model_choices`) and the value is pushed over `session/set_config_option`, through one resolver (`session_model_for`) for a spawn, a direct chat and a DAG node (`model_for` threaded through the runner). - An openai or cli row has no menu; the pill is disabled. Unset reads by ownership: one of raven's own "follows the main Raven", a third party runs on "its own default". Ownership is the wire's `own` (a new field: built-in, vendored, or an acp row whose handshake named Raven), with the catalogue as the fallback for a server that predates it. The picker offers listed ids only; there is no typed entry, since an acp row's write takes exact menu values and a built-in row's adds to no provider. Deviations from the prototype, deliberately: - No cancel while connecting; the row shows "connecting" until the server answers. - Third parties do not pick from raven's providers: there is no path that would deliver raven's credential to them, so they pick from their own advertised menu. - Raven-PPT is a hidden row and is not listed; "build an agent" is not in this PR. - The old page's rename control, transport chips and test-cost notes are gone; the prototype has none of them. - The built-in row's description stays read-only in the sheet (the server now takes the write; the field follows later). Install data (commands and sites) comes from each vendor's own documentation: Claude Code, Codex, OpenCode, Hermes, OpenClaw, MiroThinker, GitHub Copilot, Qwen Code, CodeBuddy, Qoder, Grok, Kimi Code, Pi. Wire changes: `SubagentRow.own / model / model_choices / model_source`; `subagents.update` takes `model`, `provider`, `clear_model` (clear wins) and `description: null` (back to the factory text; a built-in row goes back to following its seed); `subagents.test` takes `source: "vendored"`; `ThirdPartyAcpSubagentConfig.model` (and `schemas/subagent.schema.json` re-exported for it); `SubagentManager(provider_pool=...)` is injected by the loop and the playbook CLI. Reviewed in two passes (three read-only reviewers plus a refuter per finding): ten findings confirmed, all fixed in the backend commit -- among them a dispatch that withheld the model from a cached built-in backend, a `model_source` that promised a write the server refused, and a built-in override edit that would have shadowed a legacy-spelled row. Review round (gloryfromca, LivXue): all five of the first review's blockers and the second review's two blockers plus its two "fix or file" items where a fix was in reach are addressed in four commits -- the one-resolver read of a stored built-in model (`stored_provider_name`) with the spec-less section case tested on both sides and an unreadable config raised rather than blamed on the pick, the unusable-pin warning keyed per pin, a refused acp push said once per value instead of once per turn, the wire's `own` flag read by the page, the stored-prefix spelling matched to the host slug, no typed picks, a Stop for a running test, the model op held on the row, and the stale narrative in the manifest, the CONTRIBUTING ledger rows and two comments. The first reviewer's two later nonblocking follow-ups are in as well: the Claude Code preset no longer requires a `claude` on PATH (the adapter's SDK pin carries the CLI), and a stored id whose prefix names no provider section is read as part of the id, so a hand-written gateway-served id keeps its pin. The second review's finding 4 is fixed on the record side: a failed manual Test writes its verdict over the previous record's capabilities rather than erasing them, and a row recorded before the menu existed reads "attention, model menu not measured" until a re-verify lands instead of a stored "ready" (`CapabilitySnapshot.model_menu_measured`). Deferred by the owner as a feature the prototype does not draw: a row field saying whether the stored model is in effect (pin resolved / push accepted), which would be the durable answer to the second review's "silently" in findings 1 and 2. Not in this PR and not tracked as an issue for now. Four pre-existing defects the live regression surfaced are fixed here as well, each in its own commit: - `raven acp` answered `session/set_config_option` (and `session/new`) with the process default as the model option's `currentValue`, because it asked `model.options` without a session id. A client that renders `currentValue` as the selected item snapped back to the old model after every switch. Both call sites now ask about the session whose answer it is. - The acp probe resolved `argv[0]` alone, and a shim-launched preset's command is an `npx` one that resolves on any machine with node, so Pi read as connectable wherever `pi` was not installed and the connect failed a minute later inside the adapter. `SHIM_REQUIRED_EXECUTABLES` (in `presets.py`) names the local agent a shim reaches for with its install command; the probe asks after it too and reports the row missing with the install beside it. Only Pi is listed: `codex-acp` bundles its agent, and `claude-agent-acp` runs the CLI its `@anthropic-ai/claude-agent-sdk` pin carries as a per-platform optional dependency (verified in the 0.66.0 package: the SDK's platform package holds the `claude` binary, and the live run's "Claude Code 2.1.220" is that bundled build, not the machine's), so neither wants an install. - The Claude Code preset could not connect for an account whose Claude settings name a recent model: `claude-agent-acp@0.66.0` bundles CLI 2.1.220 through its SDK pin, and the CLI refuses such a model ("version 2.1.251 or newer is required"). The pin is now 0.79.0 (CLI 2.1.274), measured: handshake with resume, fork and load, six model choices, five modes; connect from the row in 9 s where 0.66.0 failed after 155 s. - A row connected from the page recorded no capability snapshot, so it read "capabilities not recorded -- run a test", stateless and menuless, until a Test or a restart. The connect now records the handshake the Test would (`record_capabilities`, shared by both), only when the record is missing, stale or from before the menu. - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other Backend (worktree venv with the dev extra): ``` uv run pytest tests/test_rpc_subagents.py tests/test_rpc_subagents_model.py tests/test_rpc_schema_match.py \ tests/test_subagent_manager.py tests/test_subagent_probe.py tests/test_subagent_dag_runner.py \ tests/test_subagent_acp.py tests/test_subagent_builtin_agents.py tests/test_rpc_registration.py \ tests/test_agent_loop_session_model.py tests/test_agent_loop_subagent_role.py \ tests/test_subagent_direct_chat.py tests/test_subagent_workdir.py tests/test_agent_loop_web_tools.py \ tests/test_cli_playbook_commands.py tests/test_config_live.py -q -p no:randomly 1471 passed uv run pytest tests/test_acp_methods.py tests/test_acp_config_options.py tests/test_subagent_probe.py \ tests/test_subagent_third_party.py tests/test_rpc_subagents.py -q # the two fixes: 581 passed uv run pytest tests/test_rpc_subagents_model.py tests/test_subagent_manager.py tests/test_provider_pool.py \ tests/test_provider_wire_model.py tests/test_subagent_acp.py tests/test_rpc_subagents.py tests/test_acp_methods.py \ tests/test_acp_config_options.py tests/test_subagent_probe.py tests/test_subagent_third_party.py -q 1345 passed # after the review round; 948 passed on the five files the rebase's auto-merge touched make test-python # 24016 passed, 109 skipped (the full unit suite) make lint-python # clean ``` Web and TUI: ``` npm run gen:check --prefix ui-web # generated.ts matches the contract (189 methods) npm run type-check --prefix ui-web # clean npm run lint --prefix ui-web # 0 errors (5 pre-existing warnings in other domains) npm test --prefix ui-web # 190 files, 2550 tests passed (after the rebase onto the current base) node ui-web/scripts/check-class-namespace.mjs # OK, extAgents pinned at 0 / 0 npm run lint:rpc --prefix ui-tui # in sync npm run lint:i18n --prefix ui-tui # up to date ``` Served page (a test gateway from this branch, an isolated RAVEN_HOME copied from a real one): roster of 5 connected / 4 available / 9 not installed with display names; Raven-Research's pill lists the 55 choices its handshake advertised, a pick lands on the row and reads back from `subagents.list`; the built-in row's pick is stored as `openrouter/anthropic/claude-opus-5` and clears back to null; the three products whose snapshots predated the menu key were re-verified at startup and got their menus. Live regression on the same gateway: a product row switched off and back on through the row control (the readiness ping relaunches it); MiroThinker connected with a key typed in the sheet; an absent row's install block, copy and re-check ("still not found"); a pick and a clear on Raven-Code's own menu. Then, with the session on DeepSeek and Raven-Research's row set to `deepseek/deepseek/deepseek-v4-flash` from its menu, one direct-chat turn to a Raven-Research instance: the host sent `session/set_config_option {configId: model}` with that value (acp frame journal), the child logged `Session model: deepseek/deepseek-v4-flash` and ran the turn on it (`Iteration 1/60 model=deepseek/deepseek-v4-flash`), and answered in 15 s. Connecting Claude Code, Codex and Pi from the row each ended in the server's refusal shown red on the row with Retry -- the machine's Claude Code CLI is too old for the adapter's model, Codex is not logged in, Pi's executable is not installed -- so their Test buttons were not reached. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed - Behaviour change: the agents page is redrawn; a built-in row's `model` in config (previously masked by the turn binding on every dispatch) now takes effect, paired with its own credential, and falls back to the conversation's model when no credential can serve it (one warning in the log). An acp row's `model` is new and off by default. - The Pi row now reads "not installed" on a machine without the `pi` executable, where it read "available" before; a hand-written row is not held to this, only a row carrying the preset's provenance. - An acp row whose capability snapshot predates the model menu reads "attention" with a "run a test" detail until the boot-time re-verify or a manual Test records the menu; before, it read the old "ready" with a disabled model pill. A failed manual Test no longer drops a row's recorded menu and statefulness. - A built-in row's bare stored id (written by hand before this branch) is now paired by keyword on read and re-stored prefixed on its next write; a pick under a spec-less section is accepted only for an id that section lists. A hand-written id whose prefix names no provider section (`deepseek-ai/DeepSeek-V3`) keeps its provider derived by the pool -- the configured gateway, as such an id always ran. - Rollback: revert the PR; the new config fields (`model` on an acp row, a built-in override row) are ignored by the previous code. - [ ] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes N/A --------- Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
555 lines
19 KiB
Python
555 lines
19 KiB
Python
"""Where a model id becomes a model id plus the credential that serves it.
|
|
|
|
A pin like ``context.curator_model`` is a model id and nothing else, so
|
|
resolving it has to answer a second question: which configured provider
|
|
serves that vendor. Answering it in one place is what stops a subsystem
|
|
from sending one vendor's key to another's endpoint -- either the pool
|
|
returns a complete pair, or it returns nothing and the caller follows the
|
|
conversation instead.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
|
|
import pytest
|
|
|
|
from raven.config.schema import Config
|
|
from raven.providers.binding import ModelBinding, active_binding, resolve, use_binding
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _restore_env():
|
|
"""Building a keyed provider writes the vendor's env var; keep that here."""
|
|
before = dict(os.environ)
|
|
yield
|
|
os.environ.clear()
|
|
os.environ.update(before)
|
|
|
|
|
|
def _config(model: str = "claude-opus-4-5", **keys: str) -> Config:
|
|
cfg = Config()
|
|
cfg.agents.defaults.model = model
|
|
cfg.agents.defaults.provider = "auto"
|
|
for name, key in keys.items():
|
|
section = cfg.providers.get(name)
|
|
assert section is not None, name
|
|
section.api_key = key
|
|
return cfg
|
|
|
|
|
|
def _pool(model: str = "claude-opus-4-5", **kwargs):
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
return ProviderPool(_config(model, **kwargs))
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Binding a model
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_a_model_gets_its_own_vendors_key() -> None:
|
|
"""The point of the pool: the credential follows the model id, not whatever
|
|
provider the caller happened to be holding.
|
|
"""
|
|
pool = _pool(anthropic="sk-ant", gemini="AIza")
|
|
|
|
assert pool.bind("gemini-2.5-flash").provider.api_key == "AIza"
|
|
assert pool.bind("claude-sonnet-4-5").provider.api_key == "sk-ant"
|
|
|
|
|
|
def test_the_same_pair_is_built_once() -> None:
|
|
"""Building a provider imports LiteLLM and writes vendor env vars, so a
|
|
session flipping between two models must not pay for it per turn.
|
|
"""
|
|
pool = _pool(anthropic="sk-ant")
|
|
first = pool.bind("claude-opus-4-5")
|
|
|
|
assert pool.bind("claude-opus-4-5") is first
|
|
|
|
|
|
def test_two_models_of_one_vendor_are_two_bindings() -> None:
|
|
pool = _pool(anthropic="sk-ant")
|
|
|
|
assert pool.bind("claude-opus-4-5") is not pool.bind("claude-sonnet-4-5")
|
|
|
|
|
|
def test_an_explicit_provider_name_wins_over_derivation() -> None:
|
|
"""The picker sends the provider alongside the model; a gateway serving
|
|
another vendor's id would be mis-derived without it.
|
|
"""
|
|
pool = _pool("anthropic/claude-opus-4-5", openrouter="sk-or")
|
|
binding = pool.bind("anthropic/claude-opus-4-5", "openrouter")
|
|
|
|
assert binding.provider.api_key == "sk-or"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Pins
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_a_pin_with_credentials_becomes_a_pair() -> None:
|
|
pool = _pool(anthropic="sk-ant", gemini="AIza")
|
|
pin = pool.bind_pin("gemini-2.5-flash")
|
|
|
|
assert pin is not None
|
|
assert pin.model == "gemini-2.5-flash"
|
|
assert pin.provider.api_key == "AIza"
|
|
|
|
|
|
def test_a_pin_without_credentials_is_not_a_pair() -> None:
|
|
"""The case that used to 401 every call: a model id whose vendor has no key.
|
|
None tells the caller to follow the conversation instead of sending this id
|
|
on the conversation's credential.
|
|
"""
|
|
pool = _pool(anthropic="sk-ant")
|
|
|
|
assert pool.bind_pin("gemini-2.5-flash") is None
|
|
assert pool.bind_pin("openai/gpt-5-mini") is None
|
|
|
|
|
|
def test_an_unset_pin_is_not_a_pair() -> None:
|
|
pool = _pool(anthropic="sk-ant")
|
|
|
|
assert pool.bind_pin(None) is None
|
|
assert pool.bind_pin("") is None
|
|
|
|
|
|
def test_a_pin_whose_vendor_cannot_be_derived_is_not_a_pair() -> None:
|
|
"""With no vendor there is no section to check, so there is no way to tell a
|
|
working pin from a mis-paired one. Following the conversation is at least a
|
|
pair.
|
|
"""
|
|
pool = _pool(anthropic="sk-ant")
|
|
|
|
assert pool.bind_pin("some-unknown-local-model") is None
|
|
|
|
|
|
def test_a_declared_but_empty_section_is_not_credentials() -> None:
|
|
"""Every vendor exists as an empty section whether or not it was configured,
|
|
so presence proves nothing.
|
|
"""
|
|
pool = _pool(anthropic="sk-ant")
|
|
assert pool.config.providers.get("gemini") is not None
|
|
|
|
assert pool.bind_pin("gemini-2.5-flash") is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The binding context
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class _Stub:
|
|
def get_default_model(self) -> str:
|
|
return "stub/default"
|
|
|
|
|
|
def test_a_binding_needs_a_model() -> None:
|
|
with pytest.raises(ValueError):
|
|
ModelBinding(_Stub(), "")
|
|
|
|
|
|
def test_nothing_is_bound_outside_a_turn() -> None:
|
|
assert active_binding() is None
|
|
|
|
|
|
def test_a_binding_is_visible_for_its_block_and_no_longer() -> None:
|
|
binding = ModelBinding(_Stub(), "vendor/model")
|
|
with use_binding(binding):
|
|
assert active_binding() is binding
|
|
assert active_binding() is None
|
|
|
|
|
|
def test_a_binding_is_restored_when_the_block_raises() -> None:
|
|
with pytest.raises(RuntimeError):
|
|
with use_binding(ModelBinding(_Stub(), "vendor/model")):
|
|
raise RuntimeError("boom")
|
|
assert active_binding() is None
|
|
|
|
|
|
def test_bindings_nest() -> None:
|
|
outer = ModelBinding(_Stub(), "outer/model")
|
|
inner = ModelBinding(_Stub(), "inner/model")
|
|
with use_binding(outer):
|
|
with use_binding(inner):
|
|
assert active_binding() is inner
|
|
assert active_binding() is outer
|
|
|
|
|
|
def test_resolve_prefers_a_pin_then_the_turn_then_the_fallback() -> None:
|
|
pin = ModelBinding(_Stub(), "pin/model")
|
|
turn = ModelBinding(_Stub(), "turn/model")
|
|
fallback = ModelBinding(_Stub(), "fallback/model")
|
|
|
|
assert resolve(None, fallback) is fallback
|
|
with use_binding(turn):
|
|
assert resolve(None, fallback) is turn
|
|
assert resolve(pin, fallback) is pin, "a configured subsystem does not follow the turn"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_detached_task_keeps_the_binding_it_was_created_under() -> None:
|
|
"""This is what makes a spawned subagent finish on the model it started on:
|
|
``create_task`` copies the context, so a later switch cannot reach it.
|
|
"""
|
|
import asyncio
|
|
|
|
started = ModelBinding(_Stub(), "started/model")
|
|
seen: list[str | None] = []
|
|
release = asyncio.Event()
|
|
|
|
async def _detached() -> None:
|
|
await release.wait()
|
|
binding = active_binding()
|
|
seen.append(binding.model if binding else None)
|
|
|
|
with use_binding(started):
|
|
task = asyncio.create_task(_detached())
|
|
|
|
# Outside the block now, and a different binding is current.
|
|
with use_binding(ModelBinding(_Stub(), "switched/model")):
|
|
release.set()
|
|
await task
|
|
|
|
assert seen == ["started/model"]
|
|
|
|
|
|
def test_a_config_supplier_is_re_read_and_drops_stale_bindings() -> None:
|
|
"""A credential fixed after start (an OAuth re-login, an edited file) has
|
|
to be visible without a restart, which a snapshot would prevent.
|
|
"""
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
broken = _config(anthropic="")
|
|
fixed = _config(anthropic="sk-ant")
|
|
current = [broken]
|
|
pool = ProviderPool(lambda: current[0])
|
|
|
|
assert pool.bind_pin("claude-opus-4-5") is None, "no key yet"
|
|
|
|
current[0] = fixed
|
|
pin = pool.bind_pin("claude-opus-4-5")
|
|
assert pin is not None
|
|
assert pin.provider.api_key == "sk-ant"
|
|
|
|
|
|
def test_a_gateway_serves_a_pin_its_upstream_vendor_has_no_key_for() -> None:
|
|
"""A gateway bills its own key for whatever id it is handed, so asking
|
|
whether the upstream vendor has a key of its own would drop a pin that
|
|
works -- which is how a working setup got broken once already.
|
|
"""
|
|
cfg = _config("anthropic/claude-opus-4-5", openrouter="sk-or")
|
|
cfg.agents.defaults.provider = "openrouter"
|
|
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
pin = ProviderPool(cfg).bind_pin("gemini-2.5-flash")
|
|
|
|
assert pin is not None
|
|
assert pin.model == "gemini-2.5-flash"
|
|
assert pin.provider.api_key == "sk-or"
|
|
|
|
|
|
def test_an_unbuildable_pin_is_reported_not_raised(monkeypatch) -> None:
|
|
"""``bind_pin`` is called from the factory at construction; a vendor whose
|
|
provider cannot be built must leave the subsystem following the
|
|
conversation, not stop the agent from starting.
|
|
"""
|
|
import raven.providers.factory as helpers
|
|
|
|
pool = _pool(anthropic="sk-ant", gemini="AIza")
|
|
|
|
def _boom(_cfg):
|
|
raise RuntimeError("cannot build")
|
|
|
|
monkeypatch.setattr(helpers, "make_provider", _boom)
|
|
assert pool.bind_pin("gemini-2.5-flash") is None
|
|
|
|
|
|
def test_a_credential_added_after_boot_is_visible_without_a_restart() -> None:
|
|
"""The production wiring hands the pool a loader, not the boot config: a
|
|
user who connects a provider in the picker and then picks its model must
|
|
not be told the model is unavailable.
|
|
"""
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
cfg = _config()
|
|
pool = ProviderPool(lambda: cfg)
|
|
|
|
assert pool.bind_pin("gemini-2.5-flash") is None, "no key yet"
|
|
|
|
cfg.providers.gemini.api_key = "AIza-added-after-boot"
|
|
pin = pool.bind_pin("gemini-2.5-flash")
|
|
|
|
assert pin is not None
|
|
assert pin.provider.api_key == "AIza-added-after-boot"
|
|
|
|
|
|
def test_a_re_reading_supplier_still_reuses_bindings() -> None:
|
|
"""Invalidating on config identity would clear the cache on every call for
|
|
a supplier that re-reads, which defeats the pool. Only a credential change
|
|
may drop it.
|
|
"""
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
def _fresh():
|
|
# A new object every call, same contents -- what a file loader does.
|
|
return _config(anthropic="sk-ant")
|
|
|
|
pool = ProviderPool(_fresh)
|
|
first = pool.bind("claude-opus-4-5")
|
|
|
|
assert pool.bind("claude-opus-4-5") is first
|
|
|
|
|
|
def test_an_edited_window_drops_the_binding_that_carried_the_old_one() -> None:
|
|
"""The cache key has to cover the whole cached value, not just the credential.
|
|
|
|
``bind`` copies ``contextWindowTokens`` onto the binding, so a key made only
|
|
of credentials leaves a stale binding answering with the old number until a
|
|
restart -- the "I changed it and nothing happened" shape the window ladder
|
|
exists to prevent, arriving through the cache instead.
|
|
"""
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
state = {"window": None}
|
|
|
|
def _fresh():
|
|
cfg = _config(anthropic="sk-ant")
|
|
cfg.agents.defaults.context_window_tokens = state["window"]
|
|
return cfg
|
|
|
|
pool = ProviderPool(_fresh)
|
|
first = pool.bind("claude-opus-4-5")
|
|
assert first.configured_window is None
|
|
|
|
state["window"] = 32768
|
|
second = pool.bind("claude-opus-4-5")
|
|
|
|
assert second is not first, "the stale binding was reused"
|
|
assert second.configured_window == 32768
|
|
assert second.context_window == 32768
|
|
|
|
|
|
def test_a_gateway_pin_that_cannot_be_built_is_reported_not_raised(monkeypatch) -> None:
|
|
"""The gateway branch is reached from the factory at construction too, so
|
|
it needs the same guard as the direct-vendor branch -- otherwise a missing
|
|
gateway key stops the agent from starting.
|
|
"""
|
|
import raven.providers.factory as helpers
|
|
|
|
cfg = _config("anthropic/claude-opus-4-5", openrouter="sk-or")
|
|
cfg.agents.defaults.provider = "openrouter"
|
|
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
pool = ProviderPool(cfg)
|
|
|
|
def _boom(_cfg):
|
|
raise RuntimeError("no gateway key")
|
|
|
|
monkeypatch.setattr(helpers, "make_provider", _boom)
|
|
assert pool.bind_pin("gemini-2.5-flash") is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# A pin configured as a pair
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_a_configured_provider_beats_the_gateway_guess() -> None:
|
|
"""The id alone cannot say which credential was meant.
|
|
|
|
On a gateway, ``anthropic/claude-...`` served by the gateway and
|
|
``claude-...`` served by Anthropic direct are both valid, name different
|
|
credentials and different bills. Guessing picks one; the configured pair
|
|
says which.
|
|
"""
|
|
cfg = _config("anthropic/claude-opus-4-5", openrouter="sk-or", anthropic="sk-ant")
|
|
cfg.agents.defaults.provider = "openrouter"
|
|
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
pin = ProviderPool(cfg).bind_pin("claude-haiku-4-5", "anthropic")
|
|
|
|
assert pin is not None
|
|
assert pin.model == "claude-haiku-4-5"
|
|
assert pin.provider.api_key == "sk-ant", "the configured provider serves it, not the gateway"
|
|
|
|
|
|
def test_a_configured_provider_beats_the_vendor_the_id_names() -> None:
|
|
"""The other direction: a gateway reselling a vendor's model. Deriving from
|
|
the id would send it to Anthropic direct on a key the user may not hold.
|
|
"""
|
|
cfg = _config("claude-opus-4-5", openrouter="sk-or", anthropic="sk-ant")
|
|
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
pin = ProviderPool(cfg).bind_pin("anthropic/claude-haiku-4-5", "openrouter")
|
|
|
|
assert pin is not None
|
|
assert pin.provider.api_key == "sk-or", "the gateway serves it, not the vendor in the id"
|
|
|
|
|
|
def test_a_configured_provider_without_credentials_is_not_a_pair() -> None:
|
|
"""Explicitly configured and still unusable is a config error, and the
|
|
subsystem follows the conversation rather than borrowing another key.
|
|
"""
|
|
cfg = _config("claude-opus-4-5", anthropic="sk-ant")
|
|
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
assert ProviderPool(cfg).bind_pin("gemini-2.5-flash", "gemini") is None
|
|
|
|
|
|
def test_an_unpaired_pin_still_derives_its_vendor() -> None:
|
|
"""Configs written before the provider field existed keep working."""
|
|
cfg = _config("claude-opus-4-5", anthropic="sk-ant", gemini="AIza")
|
|
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
pin = ProviderPool(cfg).bind_pin("gemini-2.5-flash")
|
|
|
|
assert pin is not None
|
|
assert pin.provider.api_key == "AIza"
|
|
|
|
|
|
def test_a_pin_that_cannot_be_built_at_all_does_not_stop_the_agent(monkeypatch) -> None:
|
|
"""``bind_pin`` runs in the context-engine factory at construction, and
|
|
building a provider imports a vendor module -- so the failures are not only
|
|
the credential ones the narrower guard covered.
|
|
"""
|
|
import raven.providers.factory as helpers
|
|
|
|
cfg = _config("claude-opus-4-5", anthropic="sk-ant")
|
|
|
|
from raven.providers.pool import ProviderPool
|
|
|
|
def _boom(_cfg):
|
|
raise ModuleNotFoundError("no module named 'litellm'")
|
|
|
|
monkeypatch.setattr(helpers, "make_provider", _boom)
|
|
assert ProviderPool(cfg).bind_pin("claude-haiku-4-5", "anthropic") is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# live_pin_resolver: the pin as a question asked per call
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class _AskedPool:
|
|
"""Records what it was asked for; pairs everything but ``"unpaired"``."""
|
|
|
|
def __init__(self) -> None:
|
|
self.asked: list[tuple[str | None, str | None]] = []
|
|
|
|
def bind_pin(self, model, provider_name=None):
|
|
self.asked.append((model, provider_name))
|
|
return None if model == "unpaired" else ModelBinding(object(), model)
|
|
|
|
|
|
def test_a_pin_is_re_read_on_every_call() -> None:
|
|
"""The whole point: a subsystem repointed on a settings surface runs on the
|
|
new pair from its next call, not from the next restart."""
|
|
from raven.providers.pool import live_pin_resolver
|
|
|
|
pool = _AskedPool()
|
|
configured = {"pin": ("first", "deepseek")}
|
|
resolve_pin = live_pin_resolver(
|
|
pool, lambda: configured["pin"], key="context.curator_model", follower="the curator"
|
|
)
|
|
|
|
assert resolve_pin().model == "first"
|
|
configured["pin"] = ("second", "gemini")
|
|
assert resolve_pin().model == "second"
|
|
assert pool.asked == [("first", "deepseek"), ("second", "gemini")]
|
|
|
|
|
|
def test_an_unset_pin_asks_the_pool_nothing() -> None:
|
|
"""Unset is the common case and the configured intent -- follow the
|
|
conversation -- so it must not cost a resolution per call."""
|
|
from raven.providers.pool import live_pin_resolver
|
|
|
|
pool = _AskedPool()
|
|
resolve_pin = live_pin_resolver(pool, lambda: (None, None), key="k", follower="the curator")
|
|
|
|
assert resolve_pin() is None
|
|
assert pool.asked == []
|
|
|
|
|
|
def test_an_unusable_pin_is_reported_once() -> None:
|
|
"""Configured and unusable is a config error the operator can fix, and
|
|
silence is how a pinned subsystem stayed inert. Once, because the
|
|
alternative is a line per call for as long as the config stays wrong."""
|
|
from loguru import logger
|
|
|
|
from raven.providers.pool import live_pin_resolver
|
|
|
|
said: list[str] = []
|
|
sink_id = logger.add(lambda message: said.append(message.record["message"]), level="WARNING")
|
|
try:
|
|
resolve_pin = live_pin_resolver(
|
|
_AskedPool(), lambda: ("unpaired", None), key="context.curator_model", follower="the curator"
|
|
)
|
|
assert resolve_pin() is None
|
|
assert resolve_pin() is None
|
|
finally:
|
|
logger.remove(sink_id)
|
|
|
|
mine = [m for m in said if "context.curator_model" in m]
|
|
assert len(mine) == 1
|
|
assert "the curator follows the conversation's model instead" in mine[0]
|
|
|
|
|
|
def test_a_later_different_unusable_pin_is_reported_on_its_own() -> None:
|
|
"""The resolver outlives the config that produced it, so once-per-resolver
|
|
would leave a second wrong pin silent for the life of the process. Once
|
|
per pin: each distinct problem gets one line, and the same problem still
|
|
gets only one."""
|
|
from loguru import logger
|
|
|
|
from raven.providers.pool import live_pin_resolver
|
|
|
|
said: list[str] = []
|
|
sink_id = logger.add(lambda message: said.append(message.record["message"]), level="WARNING")
|
|
try:
|
|
configured = {"pin": ("unpaired", None)}
|
|
resolve_pin = live_pin_resolver(
|
|
_AskedPool(), lambda: configured["pin"], key="context.curator_model", follower="the curator"
|
|
)
|
|
assert resolve_pin() is None
|
|
configured["pin"] = ("unpaired", "deepseek")
|
|
assert resolve_pin() is None
|
|
assert resolve_pin() is None
|
|
finally:
|
|
logger.remove(sink_id)
|
|
|
|
mine = [m for m in said if "context.curator_model" in m]
|
|
assert len(mine) == 2
|
|
|
|
|
|
def test_a_pin_is_resolved_once_for_the_turn_that_holds_it() -> None:
|
|
"""The curator asks for its pin once per step of a tool-calling
|
|
conversation that carries the previous steps with it. Answering two steps
|
|
differently sends the continuation to another provider, and the new model
|
|
picks up the old one's partial plan."""
|
|
from raven.config.live import hold_for_this_turn
|
|
from raven.providers.pool import live_pin_resolver
|
|
|
|
pool = _AskedPool()
|
|
configured = {"pin": ("first", None)}
|
|
resolve_pin = live_pin_resolver(
|
|
pool, lambda: configured["pin"], key="context.curator_model", follower="the curator"
|
|
)
|
|
|
|
with hold_for_this_turn():
|
|
assert resolve_pin().model == "first"
|
|
configured["pin"] = ("second", None)
|
|
assert resolve_pin().model == "first"
|
|
|
|
assert resolve_pin().model == "second"
|
|
assert pool.asked == [("first", None), ("second", None)]
|