Raven-1/tests/test_provider_pool.py
KT 68002ec25d feat(*): take the agents page to the agent hub prototype (#559)
Takes the web UI's agents page (rail entry "Agent Hub", page "Agents")
to the prototype's agent hub, and gives the wire what the prototype
needed and did not have: a default model on an agent row.

Base: `refactor/ui_web_architecture`, the long-lived architecture branch
(#475), not `main`. The diff shown here is the diff that merges into
that branch.

What the page does now:

- Three sections, never folded: connected, available to connect, not
installed on this machine. One control per row (connect / disconnect /
retry / install), a status dot only where there is something to say, and
a one-line summary from a small catalogue of what each agent is and who
makes it.
- A 520x500 sheet in the shared detail host: what the agent is good at
(editable, saved on blur), the API key where one is needed, the install
command and site for an absent agent with a re-check, a Test button, and
the model pill.
- A write in flight is held on the row as "connecting" for its length; a
refusal -- a model pick included -- stays on the row as red text with
Retry rather than as a toast. A running test keeps a Stop control.

The model pill, by the row's own rule (`model_source`, a new wire
field):

- The built-in row picks from raven's connected providers. The pick is
stored naming its provider (`stored_model_id`, the way `config.set
model` stores the host's), validated against a provider raven knows that
holds a usable credential, and paired with that credential on every
dispatch through `ProviderPool.bind_pin` -- the backend resolves the pin
per run, so a cached backend never runs one conversation's key against
another's model. The write and the dispatch read the provider off the
stored id through one function, `stored_provider_name` (the inverse of
`stored_model_id`), so a section raven has no spec for resolves to that
section on both sides.
- An acp row, raven's own shipped products included, picks from the
choices its handshake advertised (`model_choices`) and the value is
pushed over `session/set_config_option`, through one resolver
(`session_model_for`) for a spawn, a direct chat and a DAG node
(`model_for` threaded through the runner).
- An openai or cli row has no menu; the pill is disabled. Unset reads by
ownership: one of raven's own "follows the main Raven", a third party
runs on "its own default". Ownership is the wire's `own` (a new field:
built-in, vendored, or an acp row whose handshake named Raven), with the
catalogue as the fallback for a server that predates it. The picker
offers listed ids only; there is no typed entry, since an acp row's
write takes exact menu values and a built-in row's adds to no provider.

Deviations from the prototype, deliberately:

- No cancel while connecting; the row shows "connecting" until the
server answers.
- Third parties do not pick from raven's providers: there is no path
that would deliver raven's credential to them, so they pick from their
own advertised menu.
- Raven-PPT is a hidden row and is not listed; "build an agent" is not
in this PR.
- The old page's rename control, transport chips and test-cost notes are
gone; the prototype has none of them.
- The built-in row's description stays read-only in the sheet (the
server now takes the write; the field follows later).

Install data (commands and sites) comes from each vendor's own
documentation: Claude Code, Codex, OpenCode, Hermes, OpenClaw,
MiroThinker, GitHub Copilot, Qwen Code, CodeBuddy, Qoder, Grok, Kimi
Code, Pi.

Wire changes: `SubagentRow.own / model / model_choices / model_source`;
`subagents.update` takes `model`, `provider`, `clear_model` (clear wins)
and `description: null` (back to the factory text; a built-in row goes
back to following its seed); `subagents.test` takes `source:
"vendored"`; `ThirdPartyAcpSubagentConfig.model` (and
`schemas/subagent.schema.json` re-exported for it);
`SubagentManager(provider_pool=...)` is injected by the loop and the
playbook CLI.

Reviewed in two passes (three read-only reviewers plus a refuter per
finding): ten findings confirmed, all fixed in the backend commit --
among them a dispatch that withheld the model from a cached built-in
backend, a `model_source` that promised a write the server refused, and
a built-in override edit that would have shadowed a legacy-spelled row.

Review round (gloryfromca, LivXue): all five of the first review's
blockers and the second review's two blockers plus its two "fix or file"
items where a fix was in reach are addressed in four commits -- the
one-resolver read of a stored built-in model (`stored_provider_name`)
with the spec-less section case tested on both sides and an unreadable
config raised rather than blamed on the pick, the unusable-pin warning
keyed per pin, a refused acp push said once per value instead of once
per turn, the wire's `own` flag read by the page, the stored-prefix
spelling matched to the host slug, no typed picks, a Stop for a running
test, the model op held on the row, and the stale narrative in the
manifest, the CONTRIBUTING ledger rows and two comments. The first
reviewer's two later nonblocking follow-ups are in as well: the Claude
Code preset no longer requires a `claude` on PATH (the adapter's SDK pin
carries the CLI), and a stored id whose prefix names no provider section
is read as part of the id, so a hand-written gateway-served id keeps its
pin. The second review's finding 4 is fixed on the record side: a failed
manual Test writes its verdict over the previous record's capabilities
rather than erasing them, and a row recorded before the menu existed
reads "attention, model menu not measured" until a re-verify lands
instead of a stored "ready" (`CapabilitySnapshot.model_menu_measured`).
Deferred by the owner as a feature the prototype does not draw: a row
field saying whether the stored model is in effect (pin resolved / push
accepted), which would be the durable answer to the second review's
"silently" in findings 1 and 2. Not in this PR and not tracked as an
issue for now.

Four pre-existing defects the live regression surfaced are fixed here as
well, each in its own commit:

- `raven acp` answered `session/set_config_option` (and `session/new`)
with the process default as the model option's `currentValue`, because
it asked `model.options` without a session id. A client that renders
`currentValue` as the selected item snapped back to the old model after
every switch. Both call sites now ask about the session whose answer it
is.
- The acp probe resolved `argv[0]` alone, and a shim-launched preset's
command is an `npx` one that resolves on any machine with node, so Pi
read as connectable wherever `pi` was not installed and the connect
failed a minute later inside the adapter. `SHIM_REQUIRED_EXECUTABLES`
(in `presets.py`) names the local agent a shim reaches for with its
install command; the probe asks after it too and reports the row missing
with the install beside it. Only Pi is listed: `codex-acp` bundles its
agent, and `claude-agent-acp` runs the CLI its
`@anthropic-ai/claude-agent-sdk` pin carries as a per-platform optional
dependency (verified in the 0.66.0 package: the SDK's platform package
holds the `claude` binary, and the live run's "Claude Code 2.1.220" is
that bundled build, not the machine's), so neither wants an install.
- The Claude Code preset could not connect for an account whose Claude
settings name a recent model: `claude-agent-acp@0.66.0` bundles CLI
2.1.220 through its SDK pin, and the CLI refuses such a model ("version
2.1.251 or newer is required"). The pin is now 0.79.0 (CLI 2.1.274),
measured: handshake with resume, fork and load, six model choices, five
modes; connect from the row in 9 s where 0.66.0 failed after 155 s.
- A row connected from the page recorded no capability snapshot, so it
read "capabilities not recorded -- run a test", stateless and menuless,
until a Test or a restart. The connect now records the handshake the
Test would (`record_capabilities`, shared by both), only when the record
is missing, stale or from before the menu.

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

Backend (worktree venv with the dev extra):

```
uv run pytest tests/test_rpc_subagents.py tests/test_rpc_subagents_model.py tests/test_rpc_schema_match.py \
  tests/test_subagent_manager.py tests/test_subagent_probe.py tests/test_subagent_dag_runner.py \
  tests/test_subagent_acp.py tests/test_subagent_builtin_agents.py tests/test_rpc_registration.py \
  tests/test_agent_loop_session_model.py tests/test_agent_loop_subagent_role.py \
  tests/test_subagent_direct_chat.py tests/test_subagent_workdir.py tests/test_agent_loop_web_tools.py \
  tests/test_cli_playbook_commands.py tests/test_config_live.py -q -p no:randomly
1471 passed
uv run pytest tests/test_acp_methods.py tests/test_acp_config_options.py tests/test_subagent_probe.py \
  tests/test_subagent_third_party.py tests/test_rpc_subagents.py -q      # the two fixes: 581 passed
uv run pytest tests/test_rpc_subagents_model.py tests/test_subagent_manager.py tests/test_provider_pool.py \
  tests/test_provider_wire_model.py tests/test_subagent_acp.py tests/test_rpc_subagents.py tests/test_acp_methods.py \
  tests/test_acp_config_options.py tests/test_subagent_probe.py tests/test_subagent_third_party.py -q
1345 passed                              # after the review round; 948 passed on the five files the rebase's auto-merge touched
make test-python                         # 24016 passed, 109 skipped (the full unit suite)
make lint-python                         # clean
```

Web and TUI:

```
npm run gen:check --prefix ui-web        # generated.ts matches the contract (189 methods)
npm run type-check --prefix ui-web       # clean
npm run lint --prefix ui-web             # 0 errors (5 pre-existing warnings in other domains)
npm test --prefix ui-web                 # 190 files, 2550 tests passed (after the rebase onto the current base)
node ui-web/scripts/check-class-namespace.mjs   # OK, extAgents pinned at 0 / 0
npm run lint:rpc --prefix ui-tui         # in sync
npm run lint:i18n --prefix ui-tui        # up to date
```

Served page (a test gateway from this branch, an isolated RAVEN_HOME
copied from a real one): roster of 5 connected / 4 available / 9 not
installed with display names; Raven-Research's pill lists the 55 choices
its handshake advertised, a pick lands on the row and reads back from
`subagents.list`; the built-in row's pick is stored as
`openrouter/anthropic/claude-opus-5` and clears back to null; the three
products whose snapshots predated the menu key were re-verified at
startup and got their menus.

Live regression on the same gateway: a product row switched off and back
on through the row control (the readiness ping relaunches it);
MiroThinker connected with a key typed in the sheet; an absent row's
install block, copy and re-check ("still not found"); a pick and a clear
on Raven-Code's own menu. Then, with the session on DeepSeek and
Raven-Research's row set to `deepseek/deepseek/deepseek-v4-flash` from
its menu, one direct-chat turn to a Raven-Research instance: the host
sent `session/set_config_option {configId: model}` with that value (acp
frame journal), the child logged `Session model:
deepseek/deepseek-v4-flash` and ran the turn on it (`Iteration 1/60
model=deepseek/deepseek-v4-flash`), and answered in 15 s. Connecting
Claude Code, Codex and Pi from the row each ended in the server's
refusal shown red on the row with Retry -- the machine's Claude Code CLI
is too old for the adapter's model, Codex is not logged in, Pi's
executable is not installed -- so their Test buttons were not reached.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

- Behaviour change: the agents page is redrawn; a built-in row's `model`
in config (previously masked by the turn binding on every dispatch) now
takes effect, paired with its own credential, and falls back to the
conversation's model when no credential can serve it (one warning in the
log). An acp row's `model` is new and off by default.
- The Pi row now reads "not installed" on a machine without the `pi`
executable, where it read "available" before; a hand-written row is not
held to this, only a row carrying the preset's provenance.
- An acp row whose capability snapshot predates the model menu reads
"attention" with a "run a test" detail until the boot-time re-verify or
a manual Test records the menu; before, it read the old "ready" with a
disabled model pill. A failed manual Test no longer drops a row's
recorded menu and statefulness.
- A built-in row's bare stored id (written by hand before this branch)
is now paired by keyword on read and re-stored prefixed on its next
write; a pick under a spec-less section is accepted only for an id that
section lists. A hand-written id whose prefix names no provider section
(`deepseek-ai/DeepSeek-V3`) keeps its provider derived by the pool --
the configured gateway, as such an id always ran.
- Rollback: revert the PR; the new config fields (`model` on an acp row,
a built-in override row) are ignored by the previous code.

- [ ] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

N/A

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-22 01:05:18 +08:00

555 lines
19 KiB
Python

"""Where a model id becomes a model id plus the credential that serves it.
A pin like ``context.curator_model`` is a model id and nothing else, so
resolving it has to answer a second question: which configured provider
serves that vendor. Answering it in one place is what stops a subsystem
from sending one vendor's key to another's endpoint -- either the pool
returns a complete pair, or it returns nothing and the caller follows the
conversation instead.
"""
from __future__ import annotations
import os
import pytest
from raven.config.schema import Config
from raven.providers.binding import ModelBinding, active_binding, resolve, use_binding
@pytest.fixture(autouse=True)
def _restore_env():
"""Building a keyed provider writes the vendor's env var; keep that here."""
before = dict(os.environ)
yield
os.environ.clear()
os.environ.update(before)
def _config(model: str = "claude-opus-4-5", **keys: str) -> Config:
cfg = Config()
cfg.agents.defaults.model = model
cfg.agents.defaults.provider = "auto"
for name, key in keys.items():
section = cfg.providers.get(name)
assert section is not None, name
section.api_key = key
return cfg
def _pool(model: str = "claude-opus-4-5", **kwargs):
from raven.providers.pool import ProviderPool
return ProviderPool(_config(model, **kwargs))
# ---------------------------------------------------------------------------
# Binding a model
# ---------------------------------------------------------------------------
def test_a_model_gets_its_own_vendors_key() -> None:
"""The point of the pool: the credential follows the model id, not whatever
provider the caller happened to be holding.
"""
pool = _pool(anthropic="sk-ant", gemini="AIza")
assert pool.bind("gemini-2.5-flash").provider.api_key == "AIza"
assert pool.bind("claude-sonnet-4-5").provider.api_key == "sk-ant"
def test_the_same_pair_is_built_once() -> None:
"""Building a provider imports LiteLLM and writes vendor env vars, so a
session flipping between two models must not pay for it per turn.
"""
pool = _pool(anthropic="sk-ant")
first = pool.bind("claude-opus-4-5")
assert pool.bind("claude-opus-4-5") is first
def test_two_models_of_one_vendor_are_two_bindings() -> None:
pool = _pool(anthropic="sk-ant")
assert pool.bind("claude-opus-4-5") is not pool.bind("claude-sonnet-4-5")
def test_an_explicit_provider_name_wins_over_derivation() -> None:
"""The picker sends the provider alongside the model; a gateway serving
another vendor's id would be mis-derived without it.
"""
pool = _pool("anthropic/claude-opus-4-5", openrouter="sk-or")
binding = pool.bind("anthropic/claude-opus-4-5", "openrouter")
assert binding.provider.api_key == "sk-or"
# ---------------------------------------------------------------------------
# Pins
# ---------------------------------------------------------------------------
def test_a_pin_with_credentials_becomes_a_pair() -> None:
pool = _pool(anthropic="sk-ant", gemini="AIza")
pin = pool.bind_pin("gemini-2.5-flash")
assert pin is not None
assert pin.model == "gemini-2.5-flash"
assert pin.provider.api_key == "AIza"
def test_a_pin_without_credentials_is_not_a_pair() -> None:
"""The case that used to 401 every call: a model id whose vendor has no key.
None tells the caller to follow the conversation instead of sending this id
on the conversation's credential.
"""
pool = _pool(anthropic="sk-ant")
assert pool.bind_pin("gemini-2.5-flash") is None
assert pool.bind_pin("openai/gpt-5-mini") is None
def test_an_unset_pin_is_not_a_pair() -> None:
pool = _pool(anthropic="sk-ant")
assert pool.bind_pin(None) is None
assert pool.bind_pin("") is None
def test_a_pin_whose_vendor_cannot_be_derived_is_not_a_pair() -> None:
"""With no vendor there is no section to check, so there is no way to tell a
working pin from a mis-paired one. Following the conversation is at least a
pair.
"""
pool = _pool(anthropic="sk-ant")
assert pool.bind_pin("some-unknown-local-model") is None
def test_a_declared_but_empty_section_is_not_credentials() -> None:
"""Every vendor exists as an empty section whether or not it was configured,
so presence proves nothing.
"""
pool = _pool(anthropic="sk-ant")
assert pool.config.providers.get("gemini") is not None
assert pool.bind_pin("gemini-2.5-flash") is None
# ---------------------------------------------------------------------------
# The binding context
# ---------------------------------------------------------------------------
class _Stub:
def get_default_model(self) -> str:
return "stub/default"
def test_a_binding_needs_a_model() -> None:
with pytest.raises(ValueError):
ModelBinding(_Stub(), "")
def test_nothing_is_bound_outside_a_turn() -> None:
assert active_binding() is None
def test_a_binding_is_visible_for_its_block_and_no_longer() -> None:
binding = ModelBinding(_Stub(), "vendor/model")
with use_binding(binding):
assert active_binding() is binding
assert active_binding() is None
def test_a_binding_is_restored_when_the_block_raises() -> None:
with pytest.raises(RuntimeError):
with use_binding(ModelBinding(_Stub(), "vendor/model")):
raise RuntimeError("boom")
assert active_binding() is None
def test_bindings_nest() -> None:
outer = ModelBinding(_Stub(), "outer/model")
inner = ModelBinding(_Stub(), "inner/model")
with use_binding(outer):
with use_binding(inner):
assert active_binding() is inner
assert active_binding() is outer
def test_resolve_prefers_a_pin_then_the_turn_then_the_fallback() -> None:
pin = ModelBinding(_Stub(), "pin/model")
turn = ModelBinding(_Stub(), "turn/model")
fallback = ModelBinding(_Stub(), "fallback/model")
assert resolve(None, fallback) is fallback
with use_binding(turn):
assert resolve(None, fallback) is turn
assert resolve(pin, fallback) is pin, "a configured subsystem does not follow the turn"
@pytest.mark.asyncio
async def test_a_detached_task_keeps_the_binding_it_was_created_under() -> None:
"""This is what makes a spawned subagent finish on the model it started on:
``create_task`` copies the context, so a later switch cannot reach it.
"""
import asyncio
started = ModelBinding(_Stub(), "started/model")
seen: list[str | None] = []
release = asyncio.Event()
async def _detached() -> None:
await release.wait()
binding = active_binding()
seen.append(binding.model if binding else None)
with use_binding(started):
task = asyncio.create_task(_detached())
# Outside the block now, and a different binding is current.
with use_binding(ModelBinding(_Stub(), "switched/model")):
release.set()
await task
assert seen == ["started/model"]
def test_a_config_supplier_is_re_read_and_drops_stale_bindings() -> None:
"""A credential fixed after start (an OAuth re-login, an edited file) has
to be visible without a restart, which a snapshot would prevent.
"""
from raven.providers.pool import ProviderPool
broken = _config(anthropic="")
fixed = _config(anthropic="sk-ant")
current = [broken]
pool = ProviderPool(lambda: current[0])
assert pool.bind_pin("claude-opus-4-5") is None, "no key yet"
current[0] = fixed
pin = pool.bind_pin("claude-opus-4-5")
assert pin is not None
assert pin.provider.api_key == "sk-ant"
def test_a_gateway_serves_a_pin_its_upstream_vendor_has_no_key_for() -> None:
"""A gateway bills its own key for whatever id it is handed, so asking
whether the upstream vendor has a key of its own would drop a pin that
works -- which is how a working setup got broken once already.
"""
cfg = _config("anthropic/claude-opus-4-5", openrouter="sk-or")
cfg.agents.defaults.provider = "openrouter"
from raven.providers.pool import ProviderPool
pin = ProviderPool(cfg).bind_pin("gemini-2.5-flash")
assert pin is not None
assert pin.model == "gemini-2.5-flash"
assert pin.provider.api_key == "sk-or"
def test_an_unbuildable_pin_is_reported_not_raised(monkeypatch) -> None:
"""``bind_pin`` is called from the factory at construction; a vendor whose
provider cannot be built must leave the subsystem following the
conversation, not stop the agent from starting.
"""
import raven.providers.factory as helpers
pool = _pool(anthropic="sk-ant", gemini="AIza")
def _boom(_cfg):
raise RuntimeError("cannot build")
monkeypatch.setattr(helpers, "make_provider", _boom)
assert pool.bind_pin("gemini-2.5-flash") is None
def test_a_credential_added_after_boot_is_visible_without_a_restart() -> None:
"""The production wiring hands the pool a loader, not the boot config: a
user who connects a provider in the picker and then picks its model must
not be told the model is unavailable.
"""
from raven.providers.pool import ProviderPool
cfg = _config()
pool = ProviderPool(lambda: cfg)
assert pool.bind_pin("gemini-2.5-flash") is None, "no key yet"
cfg.providers.gemini.api_key = "AIza-added-after-boot"
pin = pool.bind_pin("gemini-2.5-flash")
assert pin is not None
assert pin.provider.api_key == "AIza-added-after-boot"
def test_a_re_reading_supplier_still_reuses_bindings() -> None:
"""Invalidating on config identity would clear the cache on every call for
a supplier that re-reads, which defeats the pool. Only a credential change
may drop it.
"""
from raven.providers.pool import ProviderPool
def _fresh():
# A new object every call, same contents -- what a file loader does.
return _config(anthropic="sk-ant")
pool = ProviderPool(_fresh)
first = pool.bind("claude-opus-4-5")
assert pool.bind("claude-opus-4-5") is first
def test_an_edited_window_drops_the_binding_that_carried_the_old_one() -> None:
"""The cache key has to cover the whole cached value, not just the credential.
``bind`` copies ``contextWindowTokens`` onto the binding, so a key made only
of credentials leaves a stale binding answering with the old number until a
restart -- the "I changed it and nothing happened" shape the window ladder
exists to prevent, arriving through the cache instead.
"""
from raven.providers.pool import ProviderPool
state = {"window": None}
def _fresh():
cfg = _config(anthropic="sk-ant")
cfg.agents.defaults.context_window_tokens = state["window"]
return cfg
pool = ProviderPool(_fresh)
first = pool.bind("claude-opus-4-5")
assert first.configured_window is None
state["window"] = 32768
second = pool.bind("claude-opus-4-5")
assert second is not first, "the stale binding was reused"
assert second.configured_window == 32768
assert second.context_window == 32768
def test_a_gateway_pin_that_cannot_be_built_is_reported_not_raised(monkeypatch) -> None:
"""The gateway branch is reached from the factory at construction too, so
it needs the same guard as the direct-vendor branch -- otherwise a missing
gateway key stops the agent from starting.
"""
import raven.providers.factory as helpers
cfg = _config("anthropic/claude-opus-4-5", openrouter="sk-or")
cfg.agents.defaults.provider = "openrouter"
from raven.providers.pool import ProviderPool
pool = ProviderPool(cfg)
def _boom(_cfg):
raise RuntimeError("no gateway key")
monkeypatch.setattr(helpers, "make_provider", _boom)
assert pool.bind_pin("gemini-2.5-flash") is None
# ---------------------------------------------------------------------------
# A pin configured as a pair
# ---------------------------------------------------------------------------
def test_a_configured_provider_beats_the_gateway_guess() -> None:
"""The id alone cannot say which credential was meant.
On a gateway, ``anthropic/claude-...`` served by the gateway and
``claude-...`` served by Anthropic direct are both valid, name different
credentials and different bills. Guessing picks one; the configured pair
says which.
"""
cfg = _config("anthropic/claude-opus-4-5", openrouter="sk-or", anthropic="sk-ant")
cfg.agents.defaults.provider = "openrouter"
from raven.providers.pool import ProviderPool
pin = ProviderPool(cfg).bind_pin("claude-haiku-4-5", "anthropic")
assert pin is not None
assert pin.model == "claude-haiku-4-5"
assert pin.provider.api_key == "sk-ant", "the configured provider serves it, not the gateway"
def test_a_configured_provider_beats_the_vendor_the_id_names() -> None:
"""The other direction: a gateway reselling a vendor's model. Deriving from
the id would send it to Anthropic direct on a key the user may not hold.
"""
cfg = _config("claude-opus-4-5", openrouter="sk-or", anthropic="sk-ant")
from raven.providers.pool import ProviderPool
pin = ProviderPool(cfg).bind_pin("anthropic/claude-haiku-4-5", "openrouter")
assert pin is not None
assert pin.provider.api_key == "sk-or", "the gateway serves it, not the vendor in the id"
def test_a_configured_provider_without_credentials_is_not_a_pair() -> None:
"""Explicitly configured and still unusable is a config error, and the
subsystem follows the conversation rather than borrowing another key.
"""
cfg = _config("claude-opus-4-5", anthropic="sk-ant")
from raven.providers.pool import ProviderPool
assert ProviderPool(cfg).bind_pin("gemini-2.5-flash", "gemini") is None
def test_an_unpaired_pin_still_derives_its_vendor() -> None:
"""Configs written before the provider field existed keep working."""
cfg = _config("claude-opus-4-5", anthropic="sk-ant", gemini="AIza")
from raven.providers.pool import ProviderPool
pin = ProviderPool(cfg).bind_pin("gemini-2.5-flash")
assert pin is not None
assert pin.provider.api_key == "AIza"
def test_a_pin_that_cannot_be_built_at_all_does_not_stop_the_agent(monkeypatch) -> None:
"""``bind_pin`` runs in the context-engine factory at construction, and
building a provider imports a vendor module -- so the failures are not only
the credential ones the narrower guard covered.
"""
import raven.providers.factory as helpers
cfg = _config("claude-opus-4-5", anthropic="sk-ant")
from raven.providers.pool import ProviderPool
def _boom(_cfg):
raise ModuleNotFoundError("no module named 'litellm'")
monkeypatch.setattr(helpers, "make_provider", _boom)
assert ProviderPool(cfg).bind_pin("claude-haiku-4-5", "anthropic") is None
# ---------------------------------------------------------------------------
# live_pin_resolver: the pin as a question asked per call
# ---------------------------------------------------------------------------
class _AskedPool:
"""Records what it was asked for; pairs everything but ``"unpaired"``."""
def __init__(self) -> None:
self.asked: list[tuple[str | None, str | None]] = []
def bind_pin(self, model, provider_name=None):
self.asked.append((model, provider_name))
return None if model == "unpaired" else ModelBinding(object(), model)
def test_a_pin_is_re_read_on_every_call() -> None:
"""The whole point: a subsystem repointed on a settings surface runs on the
new pair from its next call, not from the next restart."""
from raven.providers.pool import live_pin_resolver
pool = _AskedPool()
configured = {"pin": ("first", "deepseek")}
resolve_pin = live_pin_resolver(
pool, lambda: configured["pin"], key="context.curator_model", follower="the curator"
)
assert resolve_pin().model == "first"
configured["pin"] = ("second", "gemini")
assert resolve_pin().model == "second"
assert pool.asked == [("first", "deepseek"), ("second", "gemini")]
def test_an_unset_pin_asks_the_pool_nothing() -> None:
"""Unset is the common case and the configured intent -- follow the
conversation -- so it must not cost a resolution per call."""
from raven.providers.pool import live_pin_resolver
pool = _AskedPool()
resolve_pin = live_pin_resolver(pool, lambda: (None, None), key="k", follower="the curator")
assert resolve_pin() is None
assert pool.asked == []
def test_an_unusable_pin_is_reported_once() -> None:
"""Configured and unusable is a config error the operator can fix, and
silence is how a pinned subsystem stayed inert. Once, because the
alternative is a line per call for as long as the config stays wrong."""
from loguru import logger
from raven.providers.pool import live_pin_resolver
said: list[str] = []
sink_id = logger.add(lambda message: said.append(message.record["message"]), level="WARNING")
try:
resolve_pin = live_pin_resolver(
_AskedPool(), lambda: ("unpaired", None), key="context.curator_model", follower="the curator"
)
assert resolve_pin() is None
assert resolve_pin() is None
finally:
logger.remove(sink_id)
mine = [m for m in said if "context.curator_model" in m]
assert len(mine) == 1
assert "the curator follows the conversation's model instead" in mine[0]
def test_a_later_different_unusable_pin_is_reported_on_its_own() -> None:
"""The resolver outlives the config that produced it, so once-per-resolver
would leave a second wrong pin silent for the life of the process. Once
per pin: each distinct problem gets one line, and the same problem still
gets only one."""
from loguru import logger
from raven.providers.pool import live_pin_resolver
said: list[str] = []
sink_id = logger.add(lambda message: said.append(message.record["message"]), level="WARNING")
try:
configured = {"pin": ("unpaired", None)}
resolve_pin = live_pin_resolver(
_AskedPool(), lambda: configured["pin"], key="context.curator_model", follower="the curator"
)
assert resolve_pin() is None
configured["pin"] = ("unpaired", "deepseek")
assert resolve_pin() is None
assert resolve_pin() is None
finally:
logger.remove(sink_id)
mine = [m for m in said if "context.curator_model" in m]
assert len(mine) == 2
def test_a_pin_is_resolved_once_for_the_turn_that_holds_it() -> None:
"""The curator asks for its pin once per step of a tool-calling
conversation that carries the previous steps with it. Answering two steps
differently sends the continuation to another provider, and the new model
picks up the old one's partial plan."""
from raven.config.live import hold_for_this_turn
from raven.providers.pool import live_pin_resolver
pool = _AskedPool()
configured = {"pin": ("first", None)}
resolve_pin = live_pin_resolver(
pool, lambda: configured["pin"], key="context.curator_model", follower="the curator"
)
with hold_for_this_turn():
assert resolve_pin().model == "first"
configured["pin"] = ("second", None)
assert resolve_pin().model == "first"
assert resolve_pin().model == "second"
assert pool.asked == [("first", None), ("second", None)]