Raven-1/tests/test_agent_loop_tool_search.py
zhanghui db9adde64c
feat(*): fold a large tool catalog by default, with the meta-pair reserved (#616)
## Summary

Make progressive tool disclosure safe to ship on, then ship it on.

`tools.toolSearch.enabled` now defaults to true. Below the compaction
threshold
that costs a deploy nothing (the strategy drops `tool_search` from every
request
while the catalog fits); above it, most schemas are withheld and reached
through
`tool_search` / `tool_call`, so context stops scaling with tool count
and an MCP
connect stops moving the prompt-cache prefix.

Two things had to land first, and both are here:

1. **The meta-pair is reserved from `tools.disabledTools`.** Their
absence from an
array is how the fold reads "this request has no search route", and it
answers
by shipping every schema. So an off switch there did not slim a folded
request,
it unfolded it -- mid-turn, since the switch is read once per assembled
array.
A measured turn went from 12 tools to 61 between two model calls,
discarding
the whole cached prefix. They join the MCP resource and prompt
meta-tools as
names the loop owns for its lifetime; `tools.toolSearch.enabled` speaks
for
both. The settings page needs no change: `ext.list` already reports them
as
   builtin and the page draws a fixed control for those.
2. **A folded request names the connected MCP servers whose schemas it
withheld.**
The runtime context named a server only while it awaited authorization
or was
still connecting, so a connected one read as nothing at all and a model
that
did not think to search reported a capability it held as missing. One
line per
server, counting the tools this turn's array carries and the fold
removes --
not the manager's registered count, which includes a switched-off tool
that
   search filters out and an always-visible one whose schema is present.

Knock-on changes: the four packaged agents drop their now-inert disable
rows, so
`tool_call` joins their faces and the ledger tests record it; each pins
`toolSearch.enabled` to false, which keeps a curated face off the fold.
The
strategy also stops assuming the array's shape -- replay feeds recorded
arrays
back through the same chain, and a corrupt recording used to end the
turn with a
TypeError instead of reaching replay's own backstop.

This replaces #609, which was cut from the pre-rebuild main. The commits
are the
same four plus one test fixture adjustment; the only dropped hunk is an
edit to
`ui-web/src/live/090-extensions.js`, which the frontend rebuild deleted
and whose
behaviour the rebuilt settings page already has.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run on the branch tip, against github/main at 52eb09b25:

- `uv run --frozen --python 3.12 --extra dev pytest -q` with
SERPER_API_KEY unset:
  24325 passed, 121 skipped, 0 failed.
- `uv run ruff check raven/ tests/` and `ruff format --check` on every
touched
  Python file: clean.
- Both new guards were reverse-verified: with the reservation removed
`test_disabled_tools_cannot_withhold_the_meta_tools` fails; with the
notice
  filter removed

`test_the_count_is_what_the_fold_withholds_not_what_the_server_registered`
  fails.
- Note for reviewers: the packaged-agent face tests read SERPER_API_KEY
from the
environment despite being hermetic, so `web_search` joins the face and
they fail
on a machine that has one. That predates this branch and is untouched
here.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

User-visible: a deploy whose catalog passes
`tools.toolSearch.compactionThreshold`
(50 by default) now sends most tool schemas only on demand. A config
file that
spells `enabled: false` keeps today's behaviour with no action. Rollback
is that
one setting, or `compactionThreshold` raised past the catalog size.

Also user-visible: `tools.disabledTools` entries naming `tool_search` or
`tool_call` stop taking effect and are reported once at startup, and the
four
packaged agents now offer `tool_call`.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

Replaces #609. N/A otherwise.

---------

Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>
2026-09-22 14:05:27 +08:00

201 lines
8.7 KiB
Python

"""Wiring tests for tool-search registration inside AgentLoop.__init__.
Covers the ``_register_default_tools`` block: the meta-tools land in the
registry and the strategy is inserted *first* (so it filters before
CacheOptimizer marks the final tool), and the whole thing is a no-op when the
feature is disabled.
"""
from __future__ import annotations
import tempfile
from pathlib import Path
import pytest
from raven.agent.loop import AgentLoop
from raven.agent.loop.bundles import EngineWiring, ToolWiring, TurnPolicy
from raven.config.schema import ToolSearchConfig
from raven.contracts.token_strategy import TokenStrategy
from raven.providers.base import LLMProvider, LLMResponse
from raven.token_wise.registry import StrategyRegistry
class _StubProvider(LLMProvider):
def __init__(self) -> None:
super().__init__(api_key="test")
async def chat(
self,
messages,
tools=None,
model=None,
max_tokens=4096,
temperature=0.7,
reasoning_effort=None,
tool_choice=None,
):
return LLMResponse(content="stub", finish_reason="stop")
def get_default_model(self) -> str:
return "stub"
class _MarkerStrategy(TokenStrategy):
@property
def name(self) -> str:
return "marker"
@pytest.fixture
def workspace():
with tempfile.TemporaryDirectory() as td:
yield Path(td)
def _make_loop(workspace: Path, cfg, strategies=None, disabled_tools=None) -> AgentLoop:
return AgentLoop(
provider=_StubProvider(),
workspace=workspace,
model="stub",
# web_search is the cataloged domain tool these tests fold away, and the
# loop only registers it when a search key resolves. Supplying one keeps
# the subject of the test present for the right reason.
policy=TurnPolicy(max_iterations=2),
tools=ToolWiring(
restrict_to_workspace=True,
tool_search_config=cfg,
search_api_key="test-serper-key",
disabled_tools=disabled_tools or [],
),
engine=EngineWiring(strategies=strategies),
)
def test_enabled_registers_meta_tools(workspace) -> None:
loop = _make_loop(workspace, ToolSearchConfig(enabled=True))
for name in ("tool_search", "tool_call"):
assert loop.tools.has(name), f"{name} should be registered"
assert loop.strategies.get("tool_search") is not None
def test_strategy_registered_first(workspace) -> None:
# A pre-existing strategy must end up *after* tool_search (which is inserted
# first so it filters tools before any cache-marking strategy runs).
registry = StrategyRegistry([_MarkerStrategy()])
loop = _make_loop(workspace, ToolSearchConfig(enabled=True), strategies=registry)
names = [s.name for s in loop.strategies.strategies]
assert names[0] == "tool_search", f"tool_search must run first, got {names}"
assert "marker" in names
def test_disabled_registers_no_search_but_keeps_tool_call(workspace) -> None:
# tool_call is not part of the feature switch: it is the only route by which
# a model can name a schema-hidden tool (the DAG controls), and those exist
# whether or not this deploy folds its catalog.
loop = _make_loop(workspace, ToolSearchConfig(enabled=False))
assert not loop.tools.has("tool_search")
assert loop.strategies.get("tool_search") is None
assert loop.tools.has("tool_call")
def test_none_config_takes_the_factory_default(workspace) -> None:
# A host that wires no tool-search config gets the schema's, and the schema
# ships the fold on -- so "no config" is the on case, not a third state. It
# costs a small catalog nothing: the strategy drops tool_search from every
# request until the catalog passes the threshold.
loop = _make_loop(workspace, None)
assert loop.tools.has("tool_search")
assert loop.strategies.get("tool_search") is not None
assert loop.tools.has("tool_call")
def test_dag_controls_are_reachable_without_progressive_disclosure(workspace) -> None:
# The bug this pair of assertions pins: cancel_dag / dag_status /
# resolve_dag_node are hidden from the schema and advertised in
# run_subagent_dag's own result text, so an unreachable tool_call left the
# advertisement pointing at nothing and a suspended node timing out.
loop = _make_loop(workspace, ToolSearchConfig(enabled=False))
hidden = loop.tools.schema_hidden_names()
assert {"cancel_dag", "dag_status", "resolve_dag_node"} <= hidden
assert loop.dag_control_reachable()
@pytest.mark.asyncio
async def test_enabled_loop_keeps_interaction_primitives_visible(workspace) -> None:
# Above the threshold the strategy compacts the real loop's tool list: the
# file/interaction/orchestration primitives (read_file / message / ask_user /
# spawn) and the meta-tools keep their schema, while a cataloged domain tool
# (web_search) is withheld and reachable only via tool_search.
loop = _make_loop(workspace, ToolSearchConfig(enabled=True, compaction_threshold=5))
assert loop.tools.has("ask_user") and loop.tools.has("spawn") and loop.tools.has("web_search")
tools = loop.tools.get_definitions()
_, out, _ = await loop.strategies.before_llm_call([], tools, "stub")
names = {t["function"]["name"] for t in out}
assert {"read_file", "message", "ask_user", "spawn"} <= names, "primitives must stay visible"
assert {"tool_search", "tool_call"} <= names, "meta-tools must stay visible"
assert "web_search" not in names, "cataloged domain tools are withheld above threshold"
@pytest.mark.asyncio
async def test_tool_call_does_not_name_tool_search_when_it_is_absent(workspace) -> None:
# The refusal text used to point at tool_search unconditionally, which is the
# same broken promise -- a route named in a result that this deploy does not
# ship -- that leaving tool_call behind the feature switch created.
loop = _make_loop(workspace, ToolSearchConfig(enabled=False))
out = await loop.tools.execute("tool_call", {"name": "no_such_tool"})
assert "tool_search" not in out
# Enabled is not enough: below the threshold the strategy drops tool_search
# from the request too, so the pointer is only honest above the fold.
unfolded = _make_loop(workspace, ToolSearchConfig(enabled=True, compaction_threshold=500))
out = await unfolded.tools.execute("tool_call", {"name": "no_such_tool"})
assert "tool_search" not in out
folded = _make_loop(workspace, ToolSearchConfig(enabled=True, compaction_threshold=5))
out = await folded.tools.execute("tool_call", {"name": "no_such_tool"})
assert "tool_search lists what is currently loaded" in out
def test_the_reserved_warning_names_the_remedy_each_tool_actually_has(workspace) -> None:
# The two are not owned the same way: turning the fold off does stop
# tool_search registering, and leaves tool_call exactly where it was. A
# shared remedy sent an operator who wrote tool_call to a setting that
# changes nothing for it.
from loguru import logger
records: list[str] = []
sink_id = logger.add(lambda message: records.append(message.record["message"]), level="WARNING")
try:
_make_loop(workspace, ToolSearchConfig(enabled=True), disabled_tools=["tool_search", "tool_call"])
finally:
logger.remove(sink_id)
lines = { # one line per entry, keyed by the name it names
name: next((text for text in records if f"names '{name}'" in text), "") for name in ("tool_search", "tool_call")
}
assert "tools.tool_search.enabled" in lines["tool_search"], lines["tool_search"]
assert "tools.tool_search.enabled" not in lines["tool_call"], lines["tool_call"]
assert "no switch" in lines["tool_call"], lines["tool_call"]
@pytest.mark.asyncio
async def test_disabled_tools_cannot_withhold_the_meta_tools(workspace) -> None:
# The off switch reaches every other name, but not these two. The strategy
# reads their absence as "this request has no search route" and ships every
# schema instead, so an entry here would unfold the array rather than slim
# it -- and mid-turn at that, since the switch is read once per assembly.
loop = _make_loop(
workspace,
ToolSearchConfig(enabled=True, compaction_threshold=5),
disabled_tools=["tool_search", "tool_call", "web_search"],
)
assert loop.tools.offers_by_name("tool_search")
assert loop.tools.offers_by_name("tool_call")
assert not loop.tools.offers_by_name("web_search"), "every other name still switches off"
tools = loop.tools.get_definitions()
_, out, _ = await loop.strategies.before_llm_call([], tools, "stub")
names = {t["function"]["name"] for t in out}
assert {"tool_search", "tool_call"} <= names
assert len(names) < len(tools), "the fold must still fold, not fall through to passthrough"