Raven-1/tests/test_agent_loop_web_tools.py
Zuyi Zhou 6083657412
feat(agent): pause a web tool whose vendor refused the key (#572)
## Summary

A 401 or 402 from a reader, or a 401, 402 or 403 from a search vendor,
is about the key or the account, not the page or the query, so every
call after it meets the same answer. The web tools kept asking: on
2026-09-20 Jina answered HTTP 402 on every `web_fetch` of a session, the
tool returned one identical error envelope per call, and the model went
on calling it.

`web_fetch`, `web_search` and `image_search` now remember a refusal per
tool instance (`_VendorRefusal` in `raven/agent/tools/web.py`) and
answer later calls without a request:

- the `error` string is the same for the refusing call and every paused
call (`<Vendor> refused the key (HTTP 402)`), so `failure_class` reads
them as one cause and the loop's stop-repeating nudge fires;
`is_hard_tool_failure` reads the envelope as deterministic (none of the
transient markers appear in the text);
- the `detail` says what the status means (401 rejected key, 402 out of
credit, 403 not allowed), that this call was not sent, and what the user
has to do: the vendor's `tools.web.providers.<vendor>.apiKey` slot, its
env var, its sign-up page, or another vendor under
`tools.web.<search|fetch>.provider`. The `Tell the user:` framing is
deliberate: a model cannot fix a key or a bill, and the incident's model
retried instead of saying so;
- the pause is per tool rather than per session because the key is
shared by every session of the process, so one session's refusal is
every session's; per-session marking would send each session into the
same 402 once;
- all three tools read their key live, in both loops: `WebFetchTool`
takes a callable key source the way `WebSearchTool` and
`ImageSearchTool` do (main gave it the same source in #548 while this PR
was open; the rebase keeps main's version), the main loop hands each of
them one through `_live_vendor_key`, and the sub-agent lane through one
`live_key` helper over the same resolver on its module `LiveConfig` (it
used to hand `web_search` and `image_search` the key it was constructed
with, while their envelope told the user the slot is read on the next
call). The vendor is live too since #543: the main loop hands the tools
a callable `provider`, and the lane runs on the vendor main reads at
spawn. That resolver, `live_vendor_key` in `raven/config/live.py`, is
one function the main loop's two live readers now delegate to, and it
follows `WebToolsConfig.vendor_key` in full: the slot, then the
pre-vendor leaf that is Serper's (`tools.web.search.apiKey`) or Jina's
(`tools.web.jinaApiKey`, through a new `web_jina_key` reader with the
same last-good memory as Serper's), then a revocation only where the
file names the vendor with an empty key, then the boot value where the
file says nothing about the vendor. Review measured the earlier shape
losing a paid Jina key the moment any other vendor's key was set (the
schema-backed slot reader answers an empty key for a vendor the subtree
does not name, and that empty was read as a revocation before the leaf
or the boot value), and a harness-supplied key the same way; both rows
now answer what the canonical resolver answers. A key set at
`tools.web.providers.<vendor>.apiKey` therefore reaches the next call
and lifts the pause without a restart, which is what the envelope tells
the user to do; otherwise one real request goes through after
`VENDOR_REFUSAL_PAUSE_S` (10 minutes) and re-arms the pause if refused
again. The envelope's advice names what each route needs: the config
slot and the vendor selection are read from the config file without a
restart (per call in the main loop, at spawn for a sub-agent), the env
var needs a restart;
- each call resolves its (vendor, key) pair once, at its start, through
a `_resolve()` on each tool -- and for `web_fetch` that one read also
decides the Jina substitution, so a keyed reader whose key is cleared in
the file (the edit the refusal text points at) reads through Jina,
keyless, from the next call instead of sending an empty credential that
could arm no pause
(`test_a_keyed_reader_whose_key_is_cleared_falls_back_to_jina_per_call`,
and the same through the main loop's wiring) -- and that pair is what
the request carries, what the envelope names and what a refusal is
recorded against; the request builders and normalisers take the vendor
and the key as parameters and read the live properties no more. Read
again on the way, a key the user replaced while a request was in flight
was recorded as refused without ever having been sent, and the next call
was answered from the pause (in the main loop the tool serves every
session of the process, so all of them lost it for ten minutes); and
with the vendor live, an envelope could name a backend the request never
reached, since `WebFetchTool.provider` decides the Jina substitution per
access. The pause is keyed on the pair, so another vendor given the same
key string is a new request;
- which statuses count differs by tool, because what a status can mean
differs by endpoint. A search vendor's API is the endpoint, so 401, 402
and 403 all speak of the account: Serper answers a rejected key with 403
(measured against the live endpoint). A reader's 403 speaks of the URL:
Jina answers an anonymous request for a domain it has blocked with 403
for every URL under it (measured:
`r.jina.ai/https://httpbin.org/status/200` answers 403 with Jina's own
`AbuseAlleviationError` envelope, "Anonymous access to domain
httpbin.org blocked ... due to previous abuse"), and Firecrawl answers
403 for a site its policy does not scrape ("This website is no longer
supported"). So `SEARCH_REFUSAL_STATUSES` is `{401, 402, 403}` and
`FETCH_REFUSAL_STATUSES` is `{401, 402}`; a reader's 403 is reported per
URL exactly as before this PR;
- a request that carried no key is never paused, whatever the status,
because there was no key to refuse. That is the default configuration,
Jina without a key (`needs_key=False`), where review found that one
blocked domain paused `web_fetch` for every URL for ten minutes and told
the user to replace a key that does not exist;
- statuses outside a tool's set (500, 429, a reader's 403, ...) are
reported exactly as before and pause nothing.

Not changed: which vendor is configured. The incident's Jina account is
out of credit; whether to top it up or select Tavily/Exa under
`tools.web.fetch.provider` is a configuration decision this PR only
makes visible. Tavily reports an exhausted plan with its own 432 and 433
rather than 402; those stay outside the set and are reported per call.
The research sub-agent's web tools are the fork's own classes
(`agents/raven-research/.../research_flow/tools/web.py`, which imports
only `raven.contracts.tool` and `raven.security.network`): they already
classify these statuses as `quota_err`, carry no `_VendorRefusal`, and
are untouched, so the fork has no pause to lift and gets none of this
PR's behaviour. An earlier revision of this description said its pause
lifts on the plugin's rebuild; there is no such pause.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

At head 40893f7f, eight commits on origin/main at a40e911d (rebased once
more for the changelog anchor; main's commits since d50a13c1 touch no
file this PR touches). Main merged #543 and #548 while this PR was open
(a callable `provider` on the three tools, the Jina substitution decided
per call, a callable key source on `WebFetchTool`, live vendors and keys
handed by the gateway loop); `git range-diff` against 38222344: the four
earlier commits replayed with main's version taken where the two had
written the same thing (the `WebFetchTool` key source, the gateway's
`web_fetch` registration) and no other hunk changed, the fifth commit
adapts the pause to the live vendor, the sixth makes the live key reader
follow the canonical order (the Jina leaf, and the boot value for a
vendor the file says nothing about), the seventh pins a leaf rotated in
the file, and the eighth is `ruff format` on one test file:

```
uv run --frozen --python 3.12 --all-extras pytest -q -n 0 -p no:randomly tests/test_agent_tools_web_providers.py tests/test_agent_loop_web_tools.py tests/test_config_live.py tests/test_security_web_ssrf.py
  235 passed (17 pre-existing PytestWarnings: asyncio marks on sync tests in that file; the four files, test_config_live.py included since main's live-vendor tests live there)
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q      (full suite, in its own worktree at this head)
  24717 passed, 110 skipped in 260.48s (0:04:20)
make lint-python lint-imports lint-deps lint-types                            # all pass
PYTHONPATH=. uv run --frozen --python 3.12 --extra dev python scripts/check_source_language.py origin/main...HEAD   # exit 0
make check-commits; PR_TITLE=... make check-pr-title; COMMIT_RANGE=origin/main...HEAD make check-large-files       # exit 0
git diff --check origin/main...HEAD; git merge-tree --write-tree HEAD origin/main                                   # clean
```

The reviewer's probe, run through the real `WebFetchTool` (Jina, no key;
a transport answering 403 for one host and a page for the others), at
the previous head and at this one:

```
previous head 067f6232 (web.py swapped in):
  PAUSED https://blocked.example/a   Jina Reader refused the key (HTTP 403)
  PAUSED https://example.org/b       Jina Reader refused the key (HTTP 403)
  PAUSED https://example.net/c       Jina Reader refused the key (HTTP 403)
  requests sent: 1
this head:
  error  https://blocked.example/a   Jina Reader answered HTTP 403
  ok     https://example.org/b       len=4
  ok     https://example.net/c       len=4
  requests sent: 3
```

Revert-to-red: `raven/agent/tools/web.py` reverted to `origin/main`, or
to the previous head, with the tests kept fails at collection
(`AttributeError: module 'raven.agent.tools.web' has no attribute
'FETCH_REFUSAL_STATUSES'`, exit 2).

Mutation, third round, each mutant applied alone with the file restored
between runs (the two web test files, 152 tests at this head, `-n 0`; a
mutant that changes a status set also changes how many cases the
parametrised tests collect, so the rows give the failures):

```
note() records a refusal from a request that carried no key    2 failed  (keyless reader, 401 and 402; its 403 case is held by the fetch set alone)
FETCH_REFUSAL_STATUSES keeps 403                                5 failed  (a reader's 403, all five readers)
FETCH_REFUSAL_STATUSES drops 402                                3 failed  (the 402 pause tests, including the loop-level key rotation)
SEARCH_REFUSAL_STATUSES drops 403                               1 failed  (image_search on Serper's 403)
web_fetch built on the search set                               5 failed
control (nothing mutated)                                     152 passed
```

Fourth round, one key read per call and the sub-agent lane's two tools
(the two web test files, 155 tests, `-n 0`):

```
baseline                                                   155 passed
MUTANT  web_fetch's note() re-reads the key                1 failed   (the in-flight case for web_fetch)
MUTANT  web_search's note() re-reads the key               1 failed   (for web_search)
MUTANT  image_search's note() re-reads the key             1 failed   (for image_search)
MUTANT  web_fetch's request re-reads the key               1 failed   (the request carries a key the pause was not recorded against)
MUTANT  sub-agent lane: web_search on a boot key           1 failed   (the lane test)
MUTANT  sub-agent lane: image_search on a boot key         1 failed   (the lane test)
MUTANT  vendor switch loses its restart qualifier          34 failed  (every pause test, both tools)
REVERT  web.py + raven_loop.py to reviewed head 428cbe6b   38 failed  (the three in-flight cases, the lane test, the 34 wording cases)
CONTROL restored                                           155 passed
```

Fifth round, the live vendor (the two web test files plus
`test_config_live.py`, 219 tests, `-n 0`):

```
baseline                                                                   219 passed
MUTANT  the pause ignores the vendor                                       1 failed
MUTANT  web_fetch resolves the pair twice (key read again for the request) 1 failed
MUTANT  web_fetch's envelope names the vendor read again                   1 failed
MUTANT  the vendor switch loses its live qualifier                         34 failed
REVERT  web.py to the rebased head 9c788e0b (before the adaptation)        36 failed
CONTROL restored                                                           219 passed
```

Sixth round, the live key reader (the two web test files plus
`test_config_live.py`, 232 tests, `-n 0`):

```
baseline                                                                     232 passed
MUTANT  the jina leaf is not consulted (serper's alone, as reviewed)         2 failed
MUTANT  an empty slot is a revocation whether or not the file names the vendor 1 failed
MUTANT  the leaf reader forgets its last admitted key on a rejected edit     1 failed
MUTANT  web_fetch runs on the selected vendor even when its key is gone      3 failed
REVERT  live.py + schema.py to the reviewed head dbc8ffe0                    5 failed
CONTROL restored                                                             232 passed
```

The reviewer's measurement of the key reader, `boot` from
`WebToolsConfig.vendor_key` and `live` from `live_vendor_key` through
the real readers, at the reviewed head and at this one:

```
### reviewed head dbc8ffe0
  tools.web.jinaApiKey only                    boot='sk-jina-paid'       live='sk-jina-paid'
  same key + providers.serper                  boot='sk-jina-paid'       live=''
  serper's legacy leaf + providers.firecrawl   boot='sk-serper-legacy'   live='sk-serper-legacy'
  a cleared slot beside the serper leaf        boot='sk-serper-legacy'   live='sk-serper-legacy'
  harness key, providers names other vendors   boot='tv-harness'         live=''
### this head 40893f7fb
  tools.web.jinaApiKey only                    boot='sk-jina-paid'       live='sk-jina-paid'
  same key + providers.serper                  boot='sk-jina-paid'       live='sk-jina-paid'
  serper's legacy leaf + providers.firecrawl   boot='sk-serper-legacy'   live='sk-serper-legacy'
  a cleared slot beside the serper leaf        boot='sk-serper-legacy'   live='sk-serper-legacy'
  harness key, providers names other vendors   boot='tv-harness'         live='tv-harness'
```

The reviewer's in-flight probe (the real `WebFetchTool` on Jina, a
transport answering 402 and rotating the key source while the request is
in flight), at the fourth round's previous head and at its head
(unchanged by the fifth round: the pair is still read once and the old
key is still the one sent and paused):

```
### reviewed head 428cbe6b
  key-source reads during one execute(): 4  ['KEY-OLD', 'KEY-OLD', 'KEY-OLD', 'KEY-NEW']
  Authorization actually sent          : ['KEY-OLD']
  next call                            : answered from the pause, 0 request sent, carrying []  (api_key resolves to KEY-NEW)
### this head 38222344
  key-source reads during one execute(): 1  ['KEY-OLD']
  Authorization actually sent          : ['KEY-OLD']
  next call                            : served, 1 request sent, carrying ['KEY-NEW']  (api_key resolves to KEY-NEW)
```

Earlier rounds, kept for the record; each was measured at the head of
its round, and the tests they name are unchanged since except for the
parametrisations noted below. First round (131 tests in the two files,
at f87ec320):

```
refusals never remembered                     25 failed
a new key does not lift the pause              1 failed
the pause never ends                           1 failed
a 500 pauses too                              19 failed
paused calls change the error string          23 failed
web_search never short-circuits                8 failed
image_search never short-circuits              1 failed
web_fetch never short-circuits                16 failed
control (nothing mutated)                     131 passed
```

Second round, the live key (133 tests, at 145344f0):

```
the four production files reverted to the first-round head   3 failed
WebFetchTool ignores its key source                          4 failed
main loop hands the reader a one-time key                    1 failed
sub-agent lane hands the reader a one-time key               1 failed
live_vendor_key never reads the file                         6 failed
control                                                     133 passed
```

Tests changed across the rounds:
`test_a_new_key_lifts_the_pause_at_once` rotates the key through the
callable source the loops hand the tool instead of a private field; the
"vendor and status only, never the key" rule for search (was 401) and
fetch (was 402) now uses a 500, since a refusal status renders
differently on purpose; `test_a_reader_refusing_the_key_pauses_the_tool`
is parametrised over the reader set (401, 402) and
`test_a_search_vendor_refusing_the_key_pauses_the_tool` over the search
set (401, 402, 403). New in the third round:
`test_a_keyless_reader_is_never_paused_by_a_status[401|402|403]` (the
probe above) and `test_a_readers_403_is_about_the_page_not_the_key` for
every reader. Fourth round:
`test_the_subagent_lanes_reader_reads_its_key_live_too` became
`test_the_subagent_lanes_web_tools_read_their_keys_live_too` and covers
all three tools; new:
`test_a_key_rotated_while_a_request_is_in_flight_is_tried_before_it_is_paused[web_search|image_search|web_fetch]`;
the two pause tests assert the remedy wording. Fifth round: new
`test_a_refusal_by_one_vendor_pauses_no_other`; the two pause tests
assert the remedy's new wording (the config routes are read without a
restart, the env var is not); main's own `test_config_live.py` tests for
the callable provider pass unchanged. Sixth round:
`TestLiveVendorKeyFollowsTheCanonicalOrder` (the reviewer's three rows
and three more against `WebToolsConfig.vendor_key` as the oracle),
`test_a_vendor_the_file_says_nothing_about_keeps_its_boot_key`,
`test_a_revoked_leaf_is_an_answer_not_a_miss`,
`test_a_leaf_the_schema_rejects_dispenses_no_new_answer`,
`test_a_keyed_reader_whose_key_is_cleared_falls_back_to_jina_per_call`
and `test_a_reader_whose_key_is_cleared_in_the_file_reads_through_jina`.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed (CHANGELOG
`Unreleased / Fixed`)

## Risk

User-visible behaviour changes:

- After a 401 or 402 from a reader, or a 401, 402 or 403 from a search
vendor, the tool answers for up to ten minutes (or until the key
changes) without contacting the vendor, with an envelope/text that names
the cause and the fix. A key set at the vendor's config slot is read on
the next call and lifts the pause; a user who instead tops up the same
account gets one retry at the window's end.
- A request that carried no key never pauses, and a reader's 403 never
pauses: on the default configuration (Jina without a key) `web_fetch`
behaves exactly as before this PR.
- All three tools in both loops read their key from the live config on
every call (the main loop's `web_search` and `image_search` already did;
`web_fetch` and the sub-agent lane's two now do), and resolve their
vendor and key once per call, so the pair a request carried is what its
refusal is recorded against and a refusal by one vendor pauses no other;
a key removed from the file (an empty slot) is a revocation for them
too.
- The refusal text says the config slot and the vendor selection under
`tools.web.<kind>.provider` are read from the config file without a
restart, and the env var on restart.
- `WebFetchTool.provider` and `api_key` now both come from one
`_resolve()`; their values are what main's properties returned, read
once per access rather than the key source twice.
- A Jina key held at the pre-vendor leaf `tools.web.jinaApiKey` keeps
being sent when a `providers` subtree exists beside it, as it did before
this PR and as `WebToolsConfig.vendor_key` resolves it; a key the
process booted with is kept for a vendor the file's `providers` subtree
does not name, and revoked where the subtree names it with an empty key.
- The refusing call's own error text changes from `<Vendor> answered
HTTP 402` to `<Vendor> refused the key (HTTP 402)` plus a detail, for
the statuses in the tool's set. Anything matching the old text for those
statuses would need updating; nothing in the repo does.

Rollback: revert the squash commit. The pause is process-local state; no
config or stored data changes shape.

- [x] Security impact considered (the key is never echoed; the envelope
names the config path and env var, not the value)
- [x] Backward compatibility considered (tool schemas unchanged;
non-refusal statuses render exactly as before)
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-21 22:58:23 -07:00

720 lines
30 KiB
Python

"""Registration rules for the web tools inside ``_register_default_tools``.
``web_search`` needs a Serper key. Registering it without one let the model
reach for a search it could not run, and the tool's error -- naming a config
file and an env var -- was relayed to whoever was on the other end of the
channel. It is withheld instead, on the same terms as the media tools right
below it.
``web_fetch`` is the contrast and is asserted alongside: it works with no key,
so it is registered unconditionally and must stay that way.
"""
from __future__ import annotations
import json
import tempfile
from pathlib import Path
import pytest
from raven.agent.loop import AgentLoop
from raven.agent.subagent.backends.raven_loop import RavenLoopBackend
from raven.agent.tools.registry import ToolRegistry
from raven.agent.tools.web import ImageSearchTool, WebSearchTool
from raven.contracts.tool import Tool
from raven.providers.base import LLMProvider, LLMResponse
from tests._wiring import wire
class _StubProvider(LLMProvider):
def __init__(self) -> None:
super().__init__(api_key="test")
async def chat(
self,
messages,
tools=None,
model=None,
max_tokens=4096,
temperature=0.7,
reasoning_effort=None,
tool_choice=None,
):
return LLMResponse(content="stub", finish_reason="stop")
def get_default_model(self) -> str:
return "stub"
@pytest.fixture
def workspace():
with tempfile.TemporaryDirectory() as td:
yield Path(td)
@pytest.fixture(autouse=True)
def _no_ambient_serper_key(monkeypatch: pytest.MonkeyPatch) -> None:
"""The tool falls back to ``SERPER_API_KEY``, so a developer who exports one
would otherwise see these tests pass for the wrong reason."""
monkeypatch.delenv("SERPER_API_KEY", raising=False)
def _loop(workspace: Path, **kw) -> AgentLoop:
return AgentLoop(provider=_StubProvider(), workspace=workspace, model="stub", **wire(**kw))
def test_web_search_is_withheld_without_a_key(workspace) -> None:
# Registered but withheld: registration stopped being the gate so a key
# added while the process runs can surface the tool without a restart.
loop = _loop(workspace)
assert not loop.tools.offers_by_name("web_search"), (
"offering a search that cannot run makes the model relay the tool's setup error to the user"
)
assert loop.tools.has("web_search"), "withheld, not unregistered -- the switch must stay reversible"
assert loop.tools.offers_by_name("web_fetch"), "web_fetch needs no key and must stay unconditional"
def test_a_configured_key_offers_web_search(workspace) -> None:
loop = _loop(workspace, search_api_key="sk-serper")
assert loop.tools.offers_by_name("web_search")
def test_the_env_var_alone_offers_web_search(workspace, monkeypatch: pytest.MonkeyPatch) -> None:
# The tool resolves its key at call time from the config value *or*
# SERPER_API_KEY, so a gate that reads only the config would withdraw the
# tool from a deploy that exports the variable and configures nothing.
monkeypatch.setenv("SERPER_API_KEY", "sk-from-env")
loop = _loop(workspace)
assert loop.tools.offers_by_name("web_search")
#: Every layout a key can be added in, and the vendor selected while it is.
#: The canonical slot is what the settings page and the wizard write; the
#: pre-vendor leaf is what an unmigrated config still holds. A reader wired for
#: the leaf alone left the canonical case -- the common one -- withheld until
#: the next process, which no caller could tell from the tool being unkeyed.
_LIVE_KEY_LAYOUTS = [
("pre-vendor leaf", "serper", {"tools": {"web": {"search": {"apiKey": "sk-added-later"}}}}),
(
"canonical serper slot",
"serper",
{"tools": {"web": {"providers": {"serper": {"apiKey": "sk-added-later"}}}}},
),
(
"canonical slot of another vendor",
"tavily",
{
"tools": {
"web": {
"search": {"provider": "tavily"},
"providers": {"tavily": {"apiKey": "sk-added-later"}},
}
}
},
),
]
@pytest.mark.parametrize(("layout", "vendor", "written"), _LIVE_KEY_LAYOUTS, ids=[c[0] for c in _LIVE_KEY_LAYOUTS])
def test_a_key_added_after_start_surfaces_web_search(
workspace, tmp_path: Path, monkeypatch, layout: str, vendor: str, written: dict
) -> None:
# The reversibility the registration gate could not give: the user edits the
# config file, nothing re-registers, and the next assembly reads a different
# answer -- both the veil and the credential the call then uses.
monkeypatch.delenv("TAVILY_API_KEY", raising=False)
cfg = tmp_path / "config.json"
cfg.write_text(json.dumps({}), encoding="utf-8")
monkeypatch.setattr("raven.home._current_config_path", cfg)
loop = _loop(workspace, web_search_provider=vendor)
assert not loop.tools.offers_by_name("web_search")
cfg.write_text(json.dumps(written), encoding="utf-8")
assert loop.tools.offers_by_name("web_search"), f"{layout}: the added key never reached the tool"
assert loop.tools.get("web_search").api_key == "sk-added-later"
def test_a_key_cleared_after_start_withdraws_web_search(workspace, tmp_path: Path, monkeypatch) -> None:
"""The other direction, and the reason an empty slot is an answer rather
than a miss: a revoked key must not fall through to the boot value the
process started with."""
cfg = tmp_path / "config.json"
cfg.write_text(json.dumps({"tools": {"web": {"providers": {"serper": {"apiKey": "sk-boot"}}}}}), encoding="utf-8")
monkeypatch.setattr("raven.home._current_config_path", cfg)
loop = _loop(workspace, web_provider_keys={"serper": "sk-boot"})
assert loop.tools.offers_by_name("web_search")
cfg.write_text(json.dumps({"tools": {"web": {"providers": {"serper": {"apiKey": ""}}}}}), encoding="utf-8")
assert not loop.tools.offers_by_name("web_search")
assert loop.tools.has("web_search"), "withheld, not unregistered"
def test_the_subagent_loop_applies_the_same_rule(workspace, monkeypatch: pytest.MonkeyPatch) -> None:
# A sub-agent reaching for a search it cannot run reports the failure to its
# caller, and that text lands in the parent turn.
#
# The backend builds its registry inside `run` and keeps no reference, so the
# names are observed as they are registered. The collector opens before the
# backend is constructed and drops nothing on the floor: were a registration
# ever to happen outside a window, it would land in the previous run's list
# and be caught, rather than vanishing and leaving an assertion that passes
# over an empty list.
registered: list[list[str]] = []
real = ToolRegistry.register
def _spy(self, tool): # noqa: ANN001, ANN202
real(self, tool)
assert registered, f"{tool.name} was registered outside a collection window"
registered[-1].append(tool.name)
monkeypatch.setattr(ToolRegistry, "register", _spy)
async def _names(**kw) -> list[str]:
registered.append([])
backend = RavenLoopBackend(provider=_StubProvider(), model="stub", agent_home=workspace / "home", **kw)
await backend.run("task", task_id="t1", workspace=workspace, executor=None)
return registered[-1]
import asyncio
without = asyncio.run(_names())
with_key = asyncio.run(_names(search_api_key="sk-serper"))
# Baselines first: an empty list would satisfy the "not in" assertion below
# without proving anything about the gate.
assert "read_file" in without and "web_fetch" in without, without
assert "read_file" in with_key and "web_fetch" in with_key, with_key
assert "web_search" not in without
assert "web_search" in with_key
def test_the_selected_vendors_key_is_what_offers_web_search(workspace, monkeypatch: pytest.MonkeyPatch) -> None:
"""The gate asks the tool built for the selected vendor. A Serper key does
not offer a Tavily search, and Tavily's own env var does.
Offered rather than registered: the tool is always registered and withheld
from the advertised array while no key resolves, so the assertions here are
about the offer. A gate that named ``SERPER_API_KEY`` itself instead of
asking the tool passed every case but the first.
"""
monkeypatch.delenv("TAVILY_API_KEY", raising=False)
serper_keyed = _loop(workspace, web_search_provider="tavily", search_api_key="sk-serper")
assert not serper_keyed.tools.offers_by_name("web_search")
assert serper_keyed.tools.has("web_search"), "withheld, not unregistered"
with_key = _loop(workspace, web_search_provider="tavily", web_provider_keys={"tavily": "tv"})
assert with_key.tools.offers_by_name("web_search")
assert with_key.tools.get("web_search").provider == "tavily"
monkeypatch.setenv("TAVILY_API_KEY", "tv-env")
assert _loop(workspace, web_search_provider="tavily").tools.offers_by_name("web_search")
def test_a_keyed_reader_without_a_key_registers_jina_instead(workspace, monkeypatch: pytest.MonkeyPatch) -> None:
"""web_fetch stays unconditional: the reader that cannot run is replaced by
the one that needs no key, out loud, rather than offered to fail."""
monkeypatch.delenv("FIRECRAWL_API_KEY", raising=False)
loop = _loop(workspace, web_fetch_provider="firecrawl")
assert loop.tools.get("web_fetch").provider == "jina"
keyed = _loop(workspace, web_fetch_provider="firecrawl", web_provider_keys={"firecrawl": "fc"})
assert keyed.tools.get("web_fetch").provider == "firecrawl"
assert keyed.tools.get("web_fetch").api_key == "fc"
class _RefusingThenServing:
"""Stands in for ``httpx.AsyncClient``: refuses the boot key, serves the new one."""
def __init__(self) -> None:
self.keys_seen: list[str] = []
def __call__(self, *args, **kwargs):
return self
async def __aenter__(self):
return self
async def __aexit__(self, *exc):
return None
async def post(self, url: str, **kwargs):
import httpx
key = kwargs["headers"]["Authorization"].removeprefix("Bearer ")
self.keys_seen.append(key)
request = httpx.Request("POST", url)
if key == "sk-boot":
return httpx.Response(402, json={}, request=request)
return httpx.Response(200, json={"results": [{"url": url, "raw_content": "PAGE"}]}, request=request)
@pytest.mark.asyncio
async def test_a_reader_key_set_after_a_refusal_reaches_the_next_call(workspace, tmp_path: Path, monkeypatch) -> None:
"""The refusal tells the user to set a working key at the vendor's slot. The
tool used to hold the key it was built with, so following that instruction
did nothing until a restart; the slot is now what the next call reads."""
monkeypatch.delenv("TAVILY_API_KEY", raising=False)
cfg = tmp_path / "config.json"
cfg.write_text(json.dumps({"tools": {"web": {"providers": {"tavily": {"apiKey": "sk-boot"}}}}}), encoding="utf-8")
monkeypatch.setattr("raven.home._current_config_path", cfg)
monkeypatch.setattr("raven.agent.tools.web.validate_url_target", lambda url: (True, ""))
client = _RefusingThenServing()
monkeypatch.setattr("raven.agent.tools.web.httpx.AsyncClient", client)
loop = _loop(workspace, web_fetch_provider="tavily", web_provider_keys={"tavily": "sk-boot"})
tool = loop.tools.get("web_fetch")
assert tool.provider == "tavily"
refused = json.loads(await tool.execute("https://a.example"))
paused = json.loads(await tool.execute("https://b.example"))
assert refused["error"] == "Tavily refused the key (HTTP 402)" and paused["paused"] is True
assert client.keys_seen == ["sk-boot"], "the paused call was not sent"
cfg.write_text(
json.dumps({"tools": {"web": {"providers": {"tavily": {"apiKey": "sk-rotated"}}}}}), encoding="utf-8"
)
served = json.loads(await tool.execute("https://c.example"))
assert served["text"] == "PAGE"
assert client.keys_seen == ["sk-boot", "sk-rotated"]
assert tool.api_key == "sk-rotated"
@pytest.mark.asyncio
async def test_a_reader_whose_key_is_cleared_in_the_file_reads_through_jina(
workspace, tmp_path: Path, monkeypatch
) -> None:
"""Clearing ``tools.web.providers.<vendor>.apiKey`` is the edit the refusal
text points the user at. The reader registered on that vendor then runs on
Jina, keyless, from the next call, rather than sending an empty credential."""
monkeypatch.delenv("TAVILY_API_KEY", raising=False)
monkeypatch.delenv("JINA_API_KEY", raising=False)
cfg = tmp_path / "config.json"
cfg.write_text(json.dumps({"tools": {"web": {"providers": {"tavily": {"apiKey": "sk-boot"}}}}}), encoding="utf-8")
monkeypatch.setattr("raven.home._current_config_path", cfg)
monkeypatch.setattr("raven.agent.tools.web.validate_url_target", lambda url: (True, ""))
sent: list[tuple[str, dict]] = []
class _Reader:
def __call__(self, *a, **k):
return self
async def __aenter__(self):
return self
async def __aexit__(self, *exc):
return None
async def post(self, url, **kwargs):
import httpx
sent.append((url, kwargs.get("headers") or {}))
payload = {"results": [{"url": url, "raw_content": "TAVILY"}]}
return httpx.Response(200, json=payload, request=httpx.Request("POST", url))
async def get(self, url, **kwargs):
import httpx
sent.append((url, kwargs.get("headers") or {}))
return httpx.Response(200, text="JINA", request=httpx.Request("GET", url))
monkeypatch.setattr("raven.agent.tools.web.httpx.AsyncClient", _Reader())
loop = _loop(workspace, web_fetch_provider="tavily", web_provider_keys={"tavily": "sk-boot"})
tool = loop.tools.get("web_fetch")
first = json.loads(await tool.execute("https://a.example"))
cfg.write_text(json.dumps({"tools": {"web": {"providers": {"tavily": {"apiKey": ""}}}}}), encoding="utf-8")
second = json.loads(await tool.execute("https://b.example"))
assert first["extractor"] == "tavily-extract" and sent[0][1]["Authorization"] == "Bearer sk-boot"
assert second["extractor"] == "jina-reader" and second["text"] == "JINA"
assert sent[1][0].startswith("https://r.jina.ai/") and "Authorization" not in sent[1][1]
assert tool.provider == "jina"
@pytest.mark.asyncio
async def test_the_subagent_lanes_web_tools_read_their_keys_live_too(tmp_path: Path, monkeypatch) -> None:
"""All three tools carry the advice to set a key at the config slot, so all
three read it there; a boot-snapshot key on two of them made that advice
false for one sub-agent run."""
from raven.agent.subagent.backends.raven_loop import RavenLoopBackend
from raven.agent.tools.registry import ToolRegistry
monkeypatch.delenv("TAVILY_API_KEY", raising=False)
cfg = tmp_path / "config.json"
cfg.write_text(json.dumps({}), encoding="utf-8")
monkeypatch.setattr("raven.home._current_config_path", cfg)
registered: list = []
real = ToolRegistry.register
def _spy(self, tool):
real(self, tool)
registered.append(tool)
monkeypatch.setattr(ToolRegistry, "register", _spy)
backend = RavenLoopBackend(
provider=_StubProvider(),
model="stub",
agent_home=tmp_path,
web_search_provider="tavily",
web_fetch_provider="tavily",
web_provider_keys={"tavily": "sk-boot"},
image_search=True,
)
await backend.run("task", task_id="t1", workspace=tmp_path, executor=None)
tools = {t.name: t for t in registered if t.name in ("web_search", "image_search", "web_fetch")}
assert sorted(tools) == ["image_search", "web_fetch", "web_search"]
assert {t.provider for t in tools.values()} == {"tavily"}
assert {t.api_key for t in tools.values()} == {"sk-boot"}
cfg.write_text(
json.dumps({"tools": {"web": {"providers": {"tavily": {"apiKey": "sk-rotated"}}}}}), encoding="utf-8"
)
assert {name: t.api_key for name, t in tools.items()} == {name: "sk-rotated" for name in tools}
@pytest.mark.asyncio
async def test_the_unconfigured_error_names_the_config_actually_in_force(tmp_path: Path) -> None:
"""Reachable only if the key disappears after registration, but the message
used to hard-code ``~/.raven/config.json`` and send anyone running with
``--config`` to edit a file the process never reads."""
from raven.config.loader import get_config_path, set_config_path
before = get_config_path()
chosen = tmp_path / "elsewhere.json"
try:
set_config_path(chosen)
out = await WebSearchTool().execute(query="anything")
finally:
set_config_path(before)
assert str(chosen) in out
assert "~/.raven/config.json" not in out
class _PluginSearch(Tool):
"""A plugin's own web_search, registered last so it shadows the built-in."""
name = "web_search"
description = "plugin search with its own credential story"
parameters = {"type": "object", "properties": {}}
async def execute(self, **kwargs) -> str:
return "plugin ran"
def test_a_plugin_shadowing_web_search_is_not_gated_by_the_builtin_config(workspace) -> None:
# Plugin tools register last precisely so one can replace a built-in by
# name; the unconfigured gate reads the BUILT-IN's config and must judge
# only the built-in instance, or it hides the plugin behind a section that
# says nothing about it.
plugin = _PluginSearch()
loop = _loop(workspace, plugin_tools=[plugin])
assert loop.tools.get("web_search") is plugin
assert loop.tools.offers_by_name("web_search"), "the built-in's empty section must not gate the plugin"
def test_a_plugin_subclass_of_web_search_is_not_gated_either(workspace) -> None:
# The ownership check is identity, not type: an isinstance gate read a
# plugin SUBCLASS as the built-in itself and withheld it on the built-in's
# empty section, even though the subclass carries its own credential story.
class _PluginSubclassSearch(WebSearchTool):
def __init__(self) -> None:
super().__init__(api_key="sk-plugin-own")
plugin = _PluginSubclassSearch()
loop = _loop(workspace, plugin_tools=[plugin])
assert loop.tools.get("web_search") is plugin
assert loop.tools.offers_by_name("web_search")
def test_a_subagent_exec_tool_reads_the_live_deny_source(workspace, tmp_path: Path, monkeypatch) -> None:
"""A tightened permission gates a delegated shell the same call it gates a
direct one: the sub-agent's gate reads the same live deny source the main
loop's does, not a construction-time snapshot."""
import asyncio
from raven.agent.tools.shell import ExecTool
from raven.permissions.turn import start_permission_turn
cfg = tmp_path / "config.json"
cfg.write_text(json.dumps({"tools": {"exec": {"extraDenyPatterns": []}}}), encoding="utf-8")
monkeypatch.setattr("raven.home._current_config_path", cfg)
captured: list[ToolRegistry] = []
real = ToolRegistry.register
def _spy(self, tool, **kw): # noqa: ANN001, ANN202
real(self, tool, **kw)
if isinstance(tool, ExecTool):
captured.append(self)
monkeypatch.setattr(ToolRegistry, "register", _spy)
backend = RavenLoopBackend(provider=_StubProvider(), model="stub", agent_home=workspace / "home")
asyncio.run(backend.run("task", task_id="t1", workspace=workspace, executor=None))
assert captured, "the backend registered no ExecTool"
registry = captured[-1]
cfg.write_text(json.dumps({"tools": {"exec": {"extraDenyPatterns": ["\\bosascript\\b"]}}}), encoding="utf-8")
start_permission_turn(None, conversation_id="sub", turn_id="t1")
result = asyncio.run(registry.execute("exec", {"command": "osascript -e beep"}))
assert "blocked" in str(result)
def test_a_subagent_gate_asks_through_the_parent_turns_responder(workspace, tmp_path: Path, monkeypatch) -> None:
"""A sub-agent's gate reads the turn's mode instead of pinning full, and
reaches the responder the parent turn bound (a ContextVar its task
inherits): a delegated exec in ask mode asks the same human a direct one
would, rather than running in silence or being refused outright."""
import asyncio
from raven.agent.tools.shell import ExecTool
from raven.contracts.permissions import ApprovalChoice, ApprovalOutcome
from raven.permissions.turn import start_permission_turn
cfg = tmp_path / "config.json"
cfg.write_text(json.dumps({"permissions": {"mode": "ask"}}), encoding="utf-8")
monkeypatch.setattr("raven.home._current_config_path", cfg)
captured: list[ToolRegistry] = []
real = ToolRegistry.register
def _spy(self, tool, **kw): # noqa: ANN001, ANN202
real(self, tool, **kw)
if isinstance(tool, ExecTool):
captured.append(self)
monkeypatch.setattr(ToolRegistry, "register", _spy)
backend = RavenLoopBackend(provider=_StubProvider(), model="stub", agent_home=workspace / "home")
asyncio.run(backend.run("task", task_id="t1", workspace=workspace, executor=None))
assert captured, "the backend registered no ExecTool"
registry = captured[-1]
class _Click:
def __init__(self) -> None:
self.seen: list[str] = []
async def await_approval(self, **request): # noqa: ANN003, ANN202
self.seen.append(request["command"])
return ApprovalOutcome(choice=ApprovalChoice.DENY)
click = _Click()
start_permission_turn(click, conversation_id="sub", turn_id="t1")
# A write, not a read: `echo` would run without asking under the read-only
# default, and this test is about the ask reaching the parent's responder.
result = asyncio.run(registry.execute("exec", {"command": "touch delegated"}))
assert click.seen == ["touch delegated"]
assert "denied" in str(result).lower()
@pytest.mark.parametrize("initially_enabled", [False, True])
def test_design_image_offer_and_execution_follow_host_selection(tmp_path, initially_enabled):
from raven.config.live import LiveConfig
from raven.config.schema import MediaGenConfig
host = tmp_path / "host.json"
selection = {"apiKey": "host-key", "model": "openai/gpt-image-2"}
def write(enabled):
host.write_text(json.dumps({"tools": {"media": {"image": selection if enabled else {}}}}))
write(initially_enabled)
startup = {**(selection if initially_enabled else {}), "selectionConfig": str(host)}
loop = _loop(tmp_path, media_config=MediaGenConfig.model_validate({"image": startup}))
loop._live_config = LiveConfig(tmp_path / "absent-render.json")
tool = loop.tools.get("image_generate")
for enabled in [initially_enabled, not initially_enabled, initially_enabled]:
write(enabled)
assert ("image_generate" not in loop._unconfigured_tool_names()) is enabled
assert tool._config.api_key == ("host-key" if enabled else "")
assert tool._config.model == ("openai/gpt-image-2" if enabled else "")
host.write_text(json.dumps({"tools": {}}))
assert "image_generate" in loop._unconfigured_tool_names()
assert not tool._config.api_key
def test_image_search_is_withheld_without_a_serper_key_and_offered_with_one(workspace) -> None:
"""The Design lane was told to call Serper's image endpoint from `exec` with a key in
a file nothing named; a live run found no key and generated every picture. A tool,
gated like web_search, is what the lane -- and the host -- reach for instead."""
unasked = _loop(workspace, search_api_key="sk-serper")
assert not unasked.tools.has("image_search"), (
"a loop that did not ask for pictures has no picture tool, keyed or not"
)
bare = _loop(workspace, image_search=True)
assert bare.tools.has("image_search") and not bare.tools.offers_by_name("image_search")
keyed = _loop(workspace, search_api_key="sk-serper", image_search=True)
assert keyed.tools.offers_by_name("image_search")
def test_image_search_follows_a_selected_vendor_that_has_an_image_surface(workspace) -> None:
loop = _loop(workspace, web_search_provider="tavily", web_provider_keys={"tavily": "tv-key"}, image_search=True)
assert loop.tools.offers_by_name("web_search") and loop.tools.offers_by_name("image_search")
assert loop.tools.get("image_search").provider == "tavily"
def test_image_search_falls_back_to_serper_when_the_selected_vendor_searches_pages_only(workspace) -> None:
on_exa = _loop(workspace, web_search_provider="exa", web_provider_keys={"exa": "exa-key"}, image_search=True)
assert on_exa.tools.offers_by_name("web_search"), "Exa searches pages"
assert not on_exa.tools.offers_by_name("image_search"), "Exa has no image surface and Serper holds no key"
assert on_exa.tools.get("image_search").provider == "serper"
with_serper = _loop(
workspace, web_search_provider="exa", web_provider_keys={"exa": "exa-key", "serper": "sk"}, image_search=True
)
assert with_serper.tools.offers_by_name("image_search"), "Serper's key opens pictures beside Exa's pages"
unkeyed_tavily = _loop(workspace, web_search_provider="tavily", search_api_key="sk-serper", image_search=True)
assert unkeyed_tavily.tools.get("image_search").provider == "serper", "a selected vendor without a key gives way"
assert unkeyed_tavily.tools.offers_by_name("image_search")
class _ImageResponse:
def __init__(self, hits):
self._hits = hits
def raise_for_status(self) -> None:
return None
def json(self):
return {"images": self._hits}
class _ImageClient:
asked: list[str] = []
def __init__(self, *args, **kwargs) -> None:
pass
async def __aenter__(self):
return self
async def __aexit__(self, *exc) -> None:
return None
async def post(self, url, json=None, headers=None, timeout=None):
type(self).asked.append(json["q"])
return _ImageResponse(
[
{
"title": f"{json['q']} wide",
"imageUrl": f"https://pics.example/{json['q']}.png",
"imageWidth": 1600,
"imageHeight": 900,
"domain": "example.com",
"link": f"https://example.com/{json['q']}",
},
{
"title": "thumbnail",
"imageUrl": "https://pics.example/small.png",
"imageWidth": 200,
"imageHeight": 150,
"domain": "example.com",
},
]
)
@pytest.mark.asyncio
async def test_image_search_returns_url_size_and_source_and_drops_what_a_screen_cannot_use(monkeypatch) -> None:
monkeypatch.setattr("raven.agent.tools.web.httpx.AsyncClient", _ImageClient)
_ImageClient.asked = []
tool = ImageSearchTool(api_key="k")
said = await tool.execute(queries=["gugong", "tiantan"])
assert _ImageClient.asked == ["gugong", "tiantan"]
assert "Image results for: gugong" in said and "Image results for: tiantan" in said
assert "https://pics.example/gugong.png" in said and "1600x900px - example.com" in said
assert "from: https://example.com/gugong" in said
assert "small.png" not in said, "200px wide is a thumbnail, not a picture"
assert "Error" in await tool.execute(), "neither form is a search"
assert "API key not configured" in await ImageSearchTool(api_key=None).execute(query="x")
@pytest.mark.parametrize(
("vendor", "payload"),
[
(
"serpapi",
{
"images_results": [
{
"title": "gate",
"original": "https://p.example/g.jpg",
"original_width": 1600,
"original_height": 900,
"source": "example.com",
"link": "https://example.com/gate",
}
]
},
),
(
"brave",
{
"results": [
{
"title": "gate",
"url": "https://example.com/gate",
"source": "example.com",
"properties": {"url": "https://p.example/g.jpg", "width": 1600, "height": 900},
}
]
},
),
(
"firecrawl",
{
"data": {
"images": [
{
"title": "gate",
"imageUrl": "https://p.example/g.jpg",
"imageWidth": 1600,
"imageHeight": 900,
"url": "https://example.com/gate",
"position": 1,
}
]
}
},
),
],
)
def test_every_vendor_with_an_image_surface_is_read_into_the_same_hit(vendor, payload) -> None:
hit = ImageSearchTool(api_key="k", provider=vendor).normalise_hits(payload)[0]
assert (hit.title, hit.image_url, hit.width, hit.height) == ("gate", "https://p.example/g.jpg", 1600, 900)
assert hit.source == "example.com" and hit.page == "https://example.com/gate"
def test_tavily_hits_come_without_a_size_and_are_offered_as_such() -> None:
tool = ImageSearchTool(api_key="k", provider="tavily")
hits = tool.normalise_hits(
{
"images": [
{"url": "https://cdn.example/a.jpg", "description": "the gate at dusk"},
"https://cdn.example/b.jpg",
]
}
)
assert [h.image_url for h in hits] == ["https://cdn.example/a.jpg", "https://cdn.example/b.jpg"]
assert hits[0].title == "the gate at dusk" and hits[0].width is None and hits[0].source == "cdn.example"
def test_a_vendor_without_an_image_surface_is_refused_at_construction() -> None:
with pytest.raises(ValueError):
ImageSearchTool(api_key="k", provider="exa")