## Summary Make progressive tool disclosure safe to ship on, then ship it on. `tools.toolSearch.enabled` now defaults to true. Below the compaction threshold that costs a deploy nothing (the strategy drops `tool_search` from every request while the catalog fits); above it, most schemas are withheld and reached through `tool_search` / `tool_call`, so context stops scaling with tool count and an MCP connect stops moving the prompt-cache prefix. Two things had to land first, and both are here: 1. **The meta-pair is reserved from `tools.disabledTools`.** Their absence from an array is how the fold reads "this request has no search route", and it answers by shipping every schema. So an off switch there did not slim a folded request, it unfolded it -- mid-turn, since the switch is read once per assembled array. A measured turn went from 12 tools to 61 between two model calls, discarding the whole cached prefix. They join the MCP resource and prompt meta-tools as names the loop owns for its lifetime; `tools.toolSearch.enabled` speaks for both. The settings page needs no change: `ext.list` already reports them as builtin and the page draws a fixed control for those. 2. **A folded request names the connected MCP servers whose schemas it withheld.** The runtime context named a server only while it awaited authorization or was still connecting, so a connected one read as nothing at all and a model that did not think to search reported a capability it held as missing. One line per server, counting the tools this turn's array carries and the fold removes -- not the manager's registered count, which includes a switched-off tool that search filters out and an always-visible one whose schema is present. Knock-on changes: the four packaged agents drop their now-inert disable rows, so `tool_call` joins their faces and the ledger tests record it; each pins `toolSearch.enabled` to false, which keeps a curated face off the fold. The strategy also stops assuming the array's shape -- replay feeds recorded arrays back through the same chain, and a corrupt recording used to end the turn with a TypeError instead of reaching replay's own backstop. This replaces #609, which was cut from the pre-rebuild main. The commits are the same four plus one test fixture adjustment; the only dropped hunk is an edit to `ui-web/src/live/090-extensions.js`, which the frontend rebuild deleted and whose behaviour the rebuilt settings page already has. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run on the branch tip, against github/main at 52eb09b25: - `uv run --frozen --python 3.12 --extra dev pytest -q` with SERPER_API_KEY unset: 24325 passed, 121 skipped, 0 failed. - `uv run ruff check raven/ tests/` and `ruff format --check` on every touched Python file: clean. - Both new guards were reverse-verified: with the reservation removed `test_disabled_tools_cannot_withhold_the_meta_tools` fails; with the notice filter removed `test_the_count_is_what_the_fold_withholds_not_what_the_server_registered` fails. - Note for reviewers: the packaged-agent face tests read SERPER_API_KEY from the environment despite being hermetic, so `web_search` joins the face and they fail on a machine that has one. That predates this branch and is untouched here. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk User-visible: a deploy whose catalog passes `tools.toolSearch.compactionThreshold` (50 by default) now sends most tool schemas only on demand. A config file that spells `enabled: false` keeps today's behaviour with no action. Rollback is that one setting, or `compactionThreshold` raised past the catalog size. Also user-visible: `tools.disabledTools` entries naming `tool_search` or `tool_call` stop taking effect and are reported once at startup, and the four packaged agents now offer `tool_call`. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues Replaces #609. N/A otherwise. --------- Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>
834 lines
33 KiB
Python
834 lines
33 KiB
Python
"""Tests for progressive tool disclosure (tool_index + tool_search)."""
|
|
|
|
import json
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from raven.agent.tools.registry import ToolRegistry
|
|
from raven.agent.tools.tool_index import ToolIndex, _schema_text
|
|
from raven.agent.tools.tool_search import (
|
|
DEFAULT_ALWAYS_VISIBLE,
|
|
META_TOOL_NAMES,
|
|
TOOL_CALL_NAME,
|
|
TOOL_SEARCH_NAME,
|
|
ToolCallTool,
|
|
ToolSearchController,
|
|
ToolSearchStrategy,
|
|
ToolSearchTool,
|
|
)
|
|
from raven.config.schema import ToolSearchConfig
|
|
from raven.contracts.tool import Tool
|
|
|
|
|
|
class _FakeTool(Tool):
|
|
def __init__(
|
|
self,
|
|
name: str,
|
|
description: str,
|
|
parameters: dict[str, Any] | None = None,
|
|
) -> None:
|
|
self._name = name
|
|
self._description = description
|
|
self._parameters = parameters or {"type": "object", "properties": {}}
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return self._name
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return self._description
|
|
|
|
@property
|
|
def parameters(self) -> dict[str, Any]:
|
|
return self._parameters
|
|
|
|
async def execute(self, **kwargs: Any) -> str:
|
|
return f"ran {self._name}"
|
|
|
|
|
|
# ---- ToolIndex ----
|
|
|
|
|
|
def test_index_ranks_name_match_first() -> None:
|
|
idx = ToolIndex()
|
|
tools = [
|
|
_FakeTool("create_issue", "open a github issue"),
|
|
_FakeTool("send_message", "post to a slack channel"),
|
|
]
|
|
idx.ensure(tools)
|
|
assert idx.search("issue", limit=5)[0] == "create_issue"
|
|
|
|
|
|
def test_index_chinese_query_hits() -> None:
|
|
idx = ToolIndex()
|
|
tools = [
|
|
_FakeTool("image_generate", "生成图片 from a text prompt"),
|
|
_FakeTool("read_file", "读取文件 contents"),
|
|
]
|
|
idx.ensure(tools)
|
|
assert idx.search("生成图片", limit=5)[0] == "image_generate"
|
|
|
|
|
|
def test_index_search_before_ensure_returns_empty() -> None:
|
|
assert ToolIndex().search("anything", limit=5) == []
|
|
|
|
|
|
def test_index_rebuilds_on_name_or_description_change() -> None:
|
|
idx = ToolIndex()
|
|
idx.ensure([_FakeTool("a", "alpha")])
|
|
first = idx._bm25
|
|
idx.ensure([_FakeTool("a", "alpha")]) # identical catalog
|
|
assert idx._bm25 is first, "should not rebuild when (name, description) is unchanged"
|
|
idx.ensure([_FakeTool("a", "alpha changed")]) # description changed
|
|
assert idx._bm25 is not first, "should rebuild when a description changes"
|
|
idx.ensure([_FakeTool("a", "alpha changed"), _FakeTool("b", "beta")]) # name set grew
|
|
assert idx._bm25 is not first, "should rebuild when the name set changes"
|
|
|
|
|
|
def test_index_matches_parameter_schema_keywords() -> None:
|
|
# A discriminating keyword living only in the parameter schema (not the
|
|
# one-line description) should still make the tool findable.
|
|
idx = ToolIndex()
|
|
tools = [
|
|
_FakeTool(
|
|
"create_issue",
|
|
"open a ticket",
|
|
parameters={
|
|
"type": "object",
|
|
"properties": {
|
|
"repository": {"type": "string", "description": "target github repository"},
|
|
},
|
|
},
|
|
),
|
|
_FakeTool("send_message", "post to a channel"),
|
|
]
|
|
idx.ensure(tools)
|
|
assert idx.search("github repository", limit=5)[0] == "create_issue"
|
|
|
|
|
|
def test_schema_text_extracts_names_descriptions_enums_and_nesting() -> None:
|
|
schema = {
|
|
"type": "object",
|
|
"properties": {
|
|
"channel": {"type": "string", "description": "slack channel id"},
|
|
"mode": {"type": "string", "enum": ["fast", "thorough"]},
|
|
"opts": {
|
|
"type": "object",
|
|
"properties": {"retries": {"type": "integer", "description": "retry count"}},
|
|
},
|
|
"tags": {"type": "array", "items": {"type": "string", "description": "a label"}},
|
|
},
|
|
}
|
|
text = _schema_text(schema)
|
|
for token in (
|
|
"channel",
|
|
"slack channel id",
|
|
"mode",
|
|
"fast",
|
|
"thorough",
|
|
"opts",
|
|
"retries",
|
|
"retry count",
|
|
"tags",
|
|
"a label",
|
|
):
|
|
assert token in text, f"{token!r} missing from schema text"
|
|
|
|
|
|
def test_schema_text_handles_non_dict_and_empty() -> None:
|
|
assert _schema_text(None) == ""
|
|
assert _schema_text([1, 2]) == "" # type: ignore[arg-type]
|
|
assert _schema_text({"type": "object"}) == "" # no properties/desc/enum
|
|
|
|
|
|
def test_schema_text_respects_depth_cap() -> None:
|
|
# Build nesting deeper than the cap; the deepest description must be dropped.
|
|
node: dict[str, Any] = {"type": "string", "description": "TOODEEP"}
|
|
for _ in range(10):
|
|
node = {"type": "object", "properties": {"n": node}}
|
|
assert "TOODEEP" not in _schema_text(node)
|
|
|
|
|
|
def test_index_rebuilds_on_parameters_change() -> None:
|
|
base = {"type": "object", "properties": {"a": {"type": "string", "description": "alpha"}}}
|
|
idx = ToolIndex()
|
|
idx.ensure([_FakeTool("t", "desc", parameters=base)])
|
|
first = idx._bm25
|
|
idx.ensure([_FakeTool("t", "desc", parameters=dict(base))]) # same schema content
|
|
assert idx._bm25 is first, "should not rebuild when parameters are unchanged"
|
|
changed = {"type": "object", "properties": {"a": {"type": "string", "description": "beta"}}}
|
|
idx.ensure([_FakeTool("t", "desc", parameters=changed)]) # param description changed
|
|
assert idx._bm25 is not first, "should rebuild when a parameter description changes"
|
|
|
|
|
|
def test_index_reused_across_instances_for_same_catalog() -> None:
|
|
tools = [_FakeTool("x", "xray"), _FakeTool("y", "yankee")]
|
|
first = ToolIndex()
|
|
first.ensure(tools)
|
|
second = ToolIndex()
|
|
second.ensure([_FakeTool("x", "xray"), _FakeTool("y", "yankee")])
|
|
assert second._bm25 is first._bm25, "same catalog should reuse the cached BM25"
|
|
|
|
|
|
# ---- ToolSearchController ----
|
|
|
|
|
|
def _controller(registry: ToolRegistry) -> ToolSearchController:
|
|
return ToolSearchController(
|
|
registry,
|
|
always_visible={"read_file"},
|
|
search_result_limit=10,
|
|
)
|
|
|
|
|
|
def test_controller_search_includes_parameters() -> None:
|
|
schema = {
|
|
"type": "object",
|
|
"properties": {"repo": {"type": "string", "description": "target repository"}},
|
|
"required": ["repo"],
|
|
}
|
|
reg = ToolRegistry()
|
|
reg.register(_FakeTool("create_issue", "open a github issue", parameters=schema))
|
|
ctrl = _controller(reg)
|
|
ctrl.refresh()
|
|
hits = ctrl.search("github issue")
|
|
assert hits[0]["name"] == "create_issue"
|
|
assert hits[0]["parameters"] == schema
|
|
|
|
|
|
def test_meta_includes_tool_call() -> None:
|
|
assert TOOL_CALL_NAME in META_TOOL_NAMES
|
|
|
|
|
|
def test_meta_no_longer_includes_describe() -> None:
|
|
assert "tool_describe" not in META_TOOL_NAMES
|
|
assert META_TOOL_NAMES == {"tool_search", TOOL_CALL_NAME}
|
|
|
|
|
|
def test_default_search_result_limit_is_ten() -> None:
|
|
assert ToolSearchConfig().search_result_limit == 10
|
|
ctrl = ToolSearchController(ToolRegistry(), always_visible=set())
|
|
assert ctrl.search_result_limit == 10
|
|
|
|
|
|
def test_default_always_visible_covers_core_and_interaction_primitives() -> None:
|
|
# File/search/exec primitives plus the interaction/orchestration primitives
|
|
# (message / ask_user / spawn) the agent must reach on any turn. Guards
|
|
# against silently dropping one (which would strand it behind tool_search).
|
|
assert set(DEFAULT_ALWAYS_VISIBLE) >= {
|
|
"read_file",
|
|
"write_file",
|
|
"edit_file",
|
|
"list_dir",
|
|
"grep",
|
|
"find",
|
|
"exec",
|
|
"message",
|
|
"ask_user",
|
|
"spawn",
|
|
}
|
|
|
|
|
|
def test_optional_orchestration_and_skill_fetch_stay_searchable() -> None:
|
|
"""A minute-scale sub-agent fan-out and a skill-body fetch are deliberate
|
|
acts, not per-turn primitives: they are reached through tool_search so the
|
|
resident tool list does not grow with every optional capability. A deploy
|
|
that wants either resident adds it via tools.tool_search.always_visible."""
|
|
always = set(DEFAULT_ALWAYS_VISIBLE)
|
|
assert "run_subagent_dag" not in always
|
|
assert "read_skill" not in always
|
|
assert "use_skill" not in always
|
|
|
|
|
|
def test_visible_names_are_stable_and_include_meta() -> None:
|
|
ctrl = ToolSearchController(ToolRegistry(), always_visible={"read_file"})
|
|
assert META_TOOL_NAMES <= ctrl.visible_names()
|
|
assert ctrl.visible_names() == ctrl.visible_names()
|
|
|
|
|
|
def test_meta_tools_not_self_searchable() -> None:
|
|
reg = ToolRegistry()
|
|
ctrl = _controller(reg)
|
|
reg.register(ToolSearchTool(ctrl))
|
|
reg.register(ToolCallTool(ctrl))
|
|
reg.register(_FakeTool("create_issue", "open a github issue"))
|
|
ctrl.refresh()
|
|
names = [h["name"] for h in ctrl.search("tool search describe call")]
|
|
assert not (META_TOOL_NAMES & set(names))
|
|
|
|
|
|
# ---- meta-tools execute ----
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_search_tool_returns_json_hits_with_parameters() -> None:
|
|
schema = {"type": "object", "properties": {"repo": {"type": "string"}}}
|
|
reg = ToolRegistry()
|
|
reg.register(_FakeTool("create_issue", "open a github issue", parameters=schema))
|
|
ctrl = _controller(reg)
|
|
ctrl.refresh()
|
|
out = await ToolSearchTool(ctrl).execute(query="github issue")
|
|
hit = json.loads(out)[0]
|
|
assert hit["name"] == "create_issue"
|
|
assert hit["parameters"] == schema
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_search_tool_no_match_message() -> None:
|
|
reg = ToolRegistry()
|
|
reg.register(_FakeTool("create_issue", "open a github issue"))
|
|
ctrl = _controller(reg)
|
|
ctrl.refresh()
|
|
out = await ToolSearchTool(ctrl).execute(query="zzzznomatch")
|
|
assert "No tools matched" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_call_forwards_to_registry() -> None:
|
|
reg = ToolRegistry()
|
|
reg.register(_FakeTool("create_issue", "open a github issue"))
|
|
ctrl = _controller(reg)
|
|
out = await ToolCallTool(ctrl).execute(name="create_issue", arguments={})
|
|
assert out == "ran create_issue"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_call_rejects_meta_and_missing() -> None:
|
|
ctrl = _controller(ToolRegistry())
|
|
assert "cannot be invoked" in await ctrl.call("tool_search", {})
|
|
assert "not available" in await ctrl.call("nope", {})
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_call_parses_stringified_arguments() -> None:
|
|
reg = ToolRegistry()
|
|
reg.register(_FakeTool("create_issue", "open a github issue"))
|
|
ctrl = _controller(reg)
|
|
# Model emitted the nested arguments as a JSON string instead of an object.
|
|
out = await ctrl.call("create_issue", '{"x": 1}')
|
|
assert out == "ran create_issue"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_call_rejects_unparseable_arguments() -> None:
|
|
ctrl = _controller(ToolRegistry())
|
|
assert "must be a JSON object" in await ctrl.call("anything", "not json {")
|
|
|
|
|
|
# ---- ToolSearchStrategy ----
|
|
|
|
|
|
def _registry_with_n(n: int) -> tuple[ToolRegistry, ToolSearchController]:
|
|
reg = ToolRegistry()
|
|
for i in range(n):
|
|
reg.register(_FakeTool(f"extra_{i}", f"extra tool number {i}"))
|
|
ctrl = ToolSearchController(
|
|
reg,
|
|
always_visible=set(DEFAULT_ALWAYS_VISIBLE),
|
|
search_result_limit=10,
|
|
)
|
|
reg.register(ToolSearchTool(ctrl))
|
|
reg.register(ToolCallTool(ctrl))
|
|
return reg, ctrl
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_strategy_small_catalog_passthrough_drops_only_tool_search() -> None:
|
|
reg, ctrl = _registry_with_n(3)
|
|
strat = ToolSearchStrategy(ctrl, compaction_threshold=25)
|
|
tools = reg.get_definitions()
|
|
_, out, _ = await strat.before_llm_call([], tools, "m")
|
|
out_names = {t["function"]["name"] for t in out}
|
|
assert TOOL_SEARCH_NAME not in out_names, "nothing to search below threshold"
|
|
assert TOOL_CALL_NAME in out_names, "schema-hidden tools need a name route at any catalog size"
|
|
assert "extra_0" in out_names, "all real tools exposed below threshold"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_strategy_large_catalog_compacts_to_visible() -> None:
|
|
reg, ctrl = _registry_with_n(40)
|
|
strat = ToolSearchStrategy(ctrl, compaction_threshold=25)
|
|
tools = reg.get_definitions()
|
|
_, out, _ = await strat.before_llm_call([], tools, "m")
|
|
out_names = {t["function"]["name"] for t in out}
|
|
assert META_TOOL_NAMES <= out_names, "meta-tools stay visible above threshold"
|
|
assert "extra_0" not in out_names, "cataloged tools are withheld above threshold"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_strategy_keeps_interaction_primitives_visible_above_threshold() -> None:
|
|
# ask_user / spawn are in DEFAULT_ALWAYS_VISIBLE, so above the threshold they
|
|
# keep their schema while ordinary cataloged tools are withheld.
|
|
reg, ctrl = _registry_with_n(40)
|
|
reg.register(_FakeTool("ask_user", "ask the user a clarifying question"))
|
|
reg.register(_FakeTool("spawn", "spawn a subagent to handle a subtask"))
|
|
strat = ToolSearchStrategy(ctrl, compaction_threshold=25)
|
|
_, out, _ = await strat.before_llm_call([], reg.get_definitions(), "m")
|
|
out_names = {t["function"]["name"] for t in out}
|
|
assert {"ask_user", "spawn"} <= out_names, "interaction primitives must stay visible"
|
|
assert "extra_0" not in out_names, "ordinary cataloged tools are still withheld"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_strategy_tool_list_stable_across_turns() -> None:
|
|
# Core guarantee: the compacted tool list never changes turn-to-turn, so the
|
|
# prompt cache stays valid (tools sit ahead of system+messages in the prefix).
|
|
reg, ctrl = _registry_with_n(40)
|
|
strat = ToolSearchStrategy(ctrl, compaction_threshold=25)
|
|
first = {t["function"]["name"] for t in (await strat.before_llm_call([], reg.get_definitions(), "m"))[1]}
|
|
second = {t["function"]["name"] for t in (await strat.before_llm_call([], reg.get_definitions(), "m"))[1]}
|
|
assert first == second
|
|
assert META_TOOL_NAMES <= first and "extra_0" not in first
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_strategy_none_tools_passthrough() -> None:
|
|
_, ctrl = _registry_with_n(40)
|
|
strat = ToolSearchStrategy(ctrl, compaction_threshold=25)
|
|
msgs, out, model = await strat.before_llm_call([{"role": "user"}], None, "m")
|
|
assert out is None and model == "m"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_strategy_passthrough_when_meta_tools_absent() -> None:
|
|
# Above threshold but meta-tools missing (a dispatch charter that narrows the
|
|
# turn, or a host that never registered them -- tools.disabled_tools cannot
|
|
# reach them): expose everything rather than strand cataloged tools.
|
|
reg, ctrl = _registry_with_n(40)
|
|
reg.unregister("tool_search")
|
|
reg.unregister(TOOL_CALL_NAME)
|
|
strat = ToolSearchStrategy(ctrl, compaction_threshold=25)
|
|
tools = reg.get_definitions()
|
|
_, out, _ = await strat.before_llm_call([], tools, "m")
|
|
assert {t["function"]["name"] for t in out} == {t["function"]["name"] for t in tools}
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unreadable_entry_passes_through_instead_of_raising() -> None:
|
|
# Replay feeds a recorded array back through this chain, so a corrupt
|
|
# recording arrives as an entry whose "function" is a string. That has to
|
|
# end at replay's own backstop as an unreplayable recording; a TypeError
|
|
# here ends the turn instead, and ends it somewhere that cannot say why.
|
|
reg, ctrl = _registry_with_n(40)
|
|
strat = ToolSearchStrategy(ctrl, compaction_threshold=25)
|
|
tools = [*reg.get_definitions(), {"function": "bad"}, "not-a-dict"]
|
|
_, out, _ = await strat.before_llm_call([], tools, "m")
|
|
assert {"function": "bad"} in out and "not-a-dict" in out, "what the fold cannot read, it keeps"
|
|
assert any(t.get("function", {}).get("name") == TOOL_CALL_NAME for t in out if isinstance(t, dict))
|
|
|
|
|
|
def test_registry_register_first_runs_before_others() -> None:
|
|
from raven.contracts.token_strategy import TokenStrategy
|
|
from raven.token_wise.registry import StrategyRegistry
|
|
|
|
class _Noop(TokenStrategy):
|
|
def __init__(self, tag: str) -> None:
|
|
self._tag = tag
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return self._tag
|
|
|
|
reg = StrategyRegistry([_Noop("a"), _Noop("b")])
|
|
reg.register(_Noop("front"), first=True)
|
|
reg.register(_Noop("back"))
|
|
assert [s.name for s in reg.strategies] == ["front", "a", "b", "back"]
|
|
|
|
|
|
# ---- tool_call must not flatten its target's blocking verdict ----
|
|
|
|
|
|
def test_tool_call_reports_its_target_as_blocking() -> None:
|
|
"""``tool_call`` is a passthrough, so its blocking verdict is the target's.
|
|
|
|
Reading the meta-tool's own flag instead double-wraps a blocking target: the
|
|
registry applies its default ceiling to the outer ``tool_call`` and kills a
|
|
sub-agent run that is deliberately deadline-free, and the turn stream is told
|
|
the call is clockable when it is not.
|
|
"""
|
|
reg = ToolRegistry()
|
|
reg.register(_FakeTool("grep", "search files"))
|
|
blocking = _FakeTool("run_subagent_dag", "orchestrate sub-agents")
|
|
blocking.blocking_interaction = True
|
|
reg.register(blocking)
|
|
ctrl = _controller(reg)
|
|
reg.register(ToolCallTool(ctrl))
|
|
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": "run_subagent_dag"}) is True
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": "grep"}) is False
|
|
|
|
|
|
def test_tool_call_with_an_unknown_or_meta_target_is_not_blocking() -> None:
|
|
"""No target, an unknown one, or a meta-tool: fall back to non-blocking so the
|
|
registry keeps its backstop on a call that is going to error out anyway."""
|
|
reg = ToolRegistry()
|
|
ctrl = _controller(reg)
|
|
reg.register(ToolCallTool(ctrl))
|
|
reg.register(ToolSearchTool(ctrl))
|
|
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {}) is False
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": "nope"}) is False
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": TOOL_CALL_NAME}) is False
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": "tool_search"}) is False
|
|
|
|
|
|
class _ArgSensitive(_FakeTool):
|
|
"""Blocking only when asked about run r1 -- the shape ResolveDagNodeTool has."""
|
|
|
|
def blocking_for(self, params: dict[str, Any]) -> bool:
|
|
return params.get("run_id") == "r1"
|
|
|
|
|
|
def test_tool_call_forwards_its_arguments_to_the_target_verdict() -> None:
|
|
"""A target whose blocking verdict depends on its arguments must see them.
|
|
|
|
`resolve_dag_node` blocks only for a bound foreground run, which it can tell
|
|
only from `run_id`; forwarded without arguments it always said "not
|
|
blocking", and the registry then put its default ceiling on a wait that has
|
|
none -- cancelling it, which the tool reads as the user stopping the agent.
|
|
"""
|
|
reg = ToolRegistry()
|
|
reg.register(_ArgSensitive("resolve_dag_node", "answer a suspended node"))
|
|
ctrl = _controller(reg)
|
|
reg.register(ToolCallTool(ctrl))
|
|
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": "resolve_dag_node", "arguments": {"run_id": "r1"}}) is True
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": "resolve_dag_node", "arguments": {"run_id": "r2"}}) is False
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": "resolve_dag_node", "arguments": '{"run_id": "r1"}'}) is True, (
|
|
"models sometimes emit the nested arguments as a JSON string; call() tolerates it, so must the verdict"
|
|
)
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": "resolve_dag_node"}) is False
|
|
assert reg.is_blocking(TOOL_CALL_NAME, {"name": "resolve_dag_node", "arguments": "not json"}) is False
|
|
|
|
|
|
class _MetadataTool(_FakeTool):
|
|
def take_metadata(self) -> dict[str, Any] | None:
|
|
return {"raven_delivery": {"files": ["report.pdf"]}}
|
|
|
|
|
|
def test_tool_call_hands_back_its_target_metadata() -> None:
|
|
"""``tool_call`` produces no metadata of its own, so a forwarded tool's
|
|
payload has to be collected from the target. Reading the forwarder instead
|
|
strands the payload: the call succeeds, the model is told so, and the UI is
|
|
handed nothing at all."""
|
|
reg = ToolRegistry()
|
|
reg.register(_MetadataTool("deliver_files", "hand files to the user"))
|
|
ctrl = _controller(reg)
|
|
reg.register(ToolCallTool(ctrl))
|
|
|
|
payload = reg.take_metadata(TOOL_CALL_NAME, {"name": "deliver_files", "arguments": {}})
|
|
|
|
assert payload == {"raven_delivery": {"files": ["report.pdf"]}}
|
|
|
|
|
|
def test_tool_call_with_an_unknown_or_meta_target_has_no_metadata() -> None:
|
|
"""No target, an unknown one, or a meta-tool: fall back to the forwarder's
|
|
own (absent) metadata rather than raising on a call that already errors."""
|
|
reg = ToolRegistry()
|
|
ctrl = _controller(reg)
|
|
reg.register(ToolCallTool(ctrl))
|
|
reg.register(ToolSearchTool(ctrl))
|
|
|
|
assert reg.take_metadata(TOOL_CALL_NAME, {}) is None
|
|
assert reg.take_metadata(TOOL_CALL_NAME, {"name": "nope"}) is None
|
|
assert reg.take_metadata(TOOL_CALL_NAME, {"name": TOOL_CALL_NAME}) is None
|
|
assert reg.take_metadata("nope", {}) is None
|
|
|
|
|
|
def test_direct_call_still_takes_its_own_metadata() -> None:
|
|
reg = ToolRegistry()
|
|
reg.register(_MetadataTool("deliver_files", "hand files to the user"))
|
|
|
|
assert reg.take_metadata("deliver_files", {}) == {"raven_delivery": {"files": ["report.pdf"]}}
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_call_does_not_timer_kill_a_blocking_target() -> None:
|
|
"""End-to-end: a slow blocking target reached via tool_call outlives the
|
|
registry ceiling, instead of being cut off by the outer wrapper."""
|
|
import asyncio
|
|
|
|
class _SlowBlocking(_FakeTool):
|
|
blocking_interaction = True
|
|
|
|
async def execute(self, **kwargs: Any) -> str:
|
|
await asyncio.sleep(0.2)
|
|
return "ran run_subagent_dag"
|
|
|
|
reg = ToolRegistry()
|
|
reg.DEFAULT_TOOL_TIMEOUT_S = 0.05
|
|
reg.register(_SlowBlocking("run_subagent_dag", "orchestrate sub-agents"))
|
|
ctrl = _controller(reg)
|
|
reg.register(ToolCallTool(ctrl))
|
|
|
|
out = await reg.execute(TOOL_CALL_NAME, {"name": "run_subagent_dag", "arguments": {}})
|
|
|
|
assert out == "ran run_subagent_dag"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# channel-bound tools: hidden from search must also mean unreachable by name
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class _WebOnlyTool(_FakeTool):
|
|
channels = frozenset({"web"})
|
|
|
|
|
|
def _web_only_registry() -> tuple[ToolRegistry, ToolSearchController]:
|
|
reg = ToolRegistry()
|
|
reg.register(_WebOnlyTool("deliver_files", "deliver finished output files to the user"))
|
|
ctrl = _controller(reg)
|
|
reg.register(ToolCallTool(ctrl))
|
|
ctrl.refresh()
|
|
return reg, ctrl
|
|
|
|
|
|
def test_search_hides_a_tool_this_channel_cannot_use() -> None:
|
|
reg, ctrl = _web_only_registry()
|
|
reg.set_channel("whatsapp")
|
|
assert ctrl.search("deliver files") == []
|
|
|
|
|
|
def test_search_surfaces_it_on_its_own_channel() -> None:
|
|
reg, ctrl = _web_only_registry()
|
|
reg.set_channel("web")
|
|
assert [h["name"] for h in ctrl.search("deliver files")] == ["deliver_files"]
|
|
|
|
|
|
def test_the_index_itself_stays_channel_independent() -> None:
|
|
"""The BM25 index is keyed on the catalog and shared by concurrent turns, so
|
|
it must not vary per channel -- filtering happens on the read instead."""
|
|
reg, ctrl = _web_only_registry()
|
|
reg.set_channel("whatsapp")
|
|
ctrl.refresh()
|
|
assert "deliver_files" in [t.name for t in ctrl._catalog_tools()]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_call_refuses_a_tool_this_channel_cannot_use() -> None:
|
|
"""Naming it directly is the way around a search-only filter, so tool_call
|
|
has to apply the same predicate rather than fall through to the registry."""
|
|
reg, _ = _web_only_registry()
|
|
reg.set_channel("whatsapp")
|
|
|
|
out = await reg.execute(TOOL_CALL_NAME, {"name": "deliver_files", "arguments": {}})
|
|
|
|
assert "not available" in out
|
|
# Worded exactly as an absent tool is: "it exists but not for you" is the
|
|
# way around the filter that keeping it out of the schema was meant to close.
|
|
assert out.replace("deliver_files", "ghost") == await reg.execute(
|
|
TOOL_CALL_NAME, {"name": "ghost", "arguments": {}}
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_withheld_tool_does_not_consume_a_result_slot() -> None:
|
|
"""The filter must run before truncation. With it after, a withheld rank-1
|
|
hit ate the only slot and the model was told nothing matched while a usable
|
|
rank-2 tool existed -- worse than not having the filter at all."""
|
|
reg = ToolRegistry()
|
|
reg.register(_WebOnlyTool("deliver_files", "deliver files to the user"))
|
|
reg.register(_FakeTool("message", "send a message with files to the user"))
|
|
ctrl = _controller(reg)
|
|
ctrl.refresh()
|
|
|
|
# Rank 1 is the web-only tool, so a telegram turn has to fall through to it.
|
|
assert ctrl._index.search("deliver files to the user", 10)[0] == "deliver_files"
|
|
|
|
reg.set_channel("telegram")
|
|
assert [h["name"] for h in ctrl.search("deliver files to the user", limit=1)] == ["message"]
|
|
|
|
reg.set_channel("web")
|
|
assert [h["name"] for h in ctrl.search("deliver files to the user", limit=1)] == ["deliver_files"]
|
|
|
|
|
|
class TestAlwaysVisibleResolvesConfiguredNames:
|
|
"""A configured name has to keep matching after its tool is renamed.
|
|
|
|
``always_visible`` is what a deploy uses to hold a tool resident while the
|
|
rest of the catalog is folded behind ``tool_search``. It matched by exact
|
|
set membership, so a namespaced tool whose registered name differs from the
|
|
configured spelling -- sanitising and the length cap both do that -- dropped
|
|
out of the resident set and back into the catalog. No error: the model just
|
|
has to search for it first.
|
|
"""
|
|
|
|
def _mcp(self, registry, server: str, tool: str):
|
|
from raven.mcp.naming import MCPToolRef, tool_name
|
|
|
|
name = tool_name(server, tool, taken=registry)
|
|
registry.register(
|
|
_FakeTool(name, f"{tool} on {server}"), origin=MCPToolRef(name=name, server=server, tool=tool)
|
|
)
|
|
return name
|
|
|
|
def test_a_configured_name_that_was_sanitised_still_resolves(self):
|
|
reg = ToolRegistry()
|
|
# The registered name loses the slash; the config entry still has it.
|
|
registered = self._mcp(reg, "gh", "actions/download-logs")
|
|
assert registered == "mcp_gh_actions_download-logs"
|
|
|
|
search = ToolSearchController(reg, always_visible={"mcp_gh_actions/download-logs"})
|
|
assert registered in search.visible_names()
|
|
|
|
def test_an_exactly_named_tool_is_visible(self):
|
|
reg = ToolRegistry()
|
|
registered = self._mcp(reg, "gh", "list_repos")
|
|
search = ToolSearchController(reg, always_visible={registered})
|
|
assert registered in search.visible_names()
|
|
|
|
def test_a_builtin_still_matches_by_its_own_name(self):
|
|
reg = ToolRegistry()
|
|
reg.register(_FakeTool("grep", "search files"))
|
|
search = ToolSearchController(reg, always_visible={"grep"})
|
|
assert "grep" in search.visible_names()
|
|
|
|
def test_a_name_matching_nothing_is_kept_as_written(self):
|
|
# Usually a tool this deploy does not have, which is ordinary. Dropping
|
|
# it would hide a typo rather than surface one.
|
|
search = ToolSearchController(ToolRegistry(), always_visible={"not_installed_here"})
|
|
assert "not_installed_here" in search.visible_names()
|
|
|
|
def test_the_meta_tools_are_always_in(self):
|
|
search = ToolSearchController(ToolRegistry(), always_visible=set())
|
|
assert META_TOOL_NAMES <= search.visible_names()
|
|
|
|
def test_a_tool_registered_after_construction_resolves(self):
|
|
# MCP servers attach while raven runs, so resolution cannot happen once
|
|
# at construction: the tool is not there yet.
|
|
reg = ToolRegistry()
|
|
search = ToolSearchController(reg, always_visible={"mcp_gh_actions/download-logs"})
|
|
assert "mcp_gh_actions_download-logs" not in search.visible_names()
|
|
|
|
registered = self._mcp(reg, "gh", "actions/download-logs")
|
|
assert registered in search.visible_names()
|
|
|
|
|
|
class TestAnAbsentNameOnTheFoldedPath:
|
|
"""``tool_call`` answers a name it cannot resolve by pointing at tool_search.
|
|
|
|
This surface has a catalog to search, so that is the useful advice --
|
|
unlike ``ToolRegistry.execute``, which also serves the unfolded surface
|
|
where there is nothing to search and therefore names both readings instead.
|
|
A tool whose MCP server was unloaded mid-turn lands here too and the search
|
|
finds nothing: one wasted hop, and the accepted price of not tracking what
|
|
used to exist.
|
|
|
|
Registering ``ToolSearchTool`` is what makes these the folded path: the
|
|
pointer is withheld where the fold is off, because ``tool_call`` ships there
|
|
too and naming a tool the deploy does not have is the broken promise this
|
|
text is supposed to avoid.
|
|
"""
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unknown_name_is_told_where_the_catalog_is(self):
|
|
reg = ToolRegistry()
|
|
reg.register(_FakeTool("real_one", "a cataloged tool"))
|
|
ctrl = ToolSearchController(reg, always_visible=set(), compaction_threshold=0)
|
|
reg.register(ToolSearchTool(ctrl))
|
|
out = await ctrl.call("mcp_ghost_thing", {})
|
|
assert "not available" in out
|
|
assert "tool_search" in out
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unknown_name_is_not_pointed_at_an_absent_tool_search(self):
|
|
reg = ToolRegistry()
|
|
ctrl = ToolSearchController(reg, always_visible=set())
|
|
out = await ctrl.call("mcp_ghost_thing", {})
|
|
assert "not available" in out
|
|
assert "tool_search" not in out
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_it_reads_the_same_way_as_the_direct_path(self):
|
|
# Both surfaces answer a name that did not resolve, and neither can tell
|
|
# a hallucinated one from a tool unloaded mid-turn. Saying it two
|
|
# different ways -- one of them "you got the name wrong" -- was the
|
|
# disagreement this pins shut. Only the catalog pointer differs, because
|
|
# only this surface has a catalog the model cannot already see.
|
|
reg = ToolRegistry()
|
|
ctrl = ToolSearchController(reg, always_visible=set())
|
|
folded = await ctrl.call("ghost", {})
|
|
direct = await reg.execute("ghost", {})
|
|
for phrase in ("is not available", "may have been unloaded", "the name may be wrong"):
|
|
assert phrase in folded, phrase
|
|
assert phrase in direct, phrase
|
|
assert "tool_search" not in direct
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unregistered_tool_is_answered_the_same_way(self):
|
|
from raven.mcp.naming import MCPToolRef
|
|
|
|
reg = ToolRegistry()
|
|
name = "mcp_openseo_search"
|
|
reg.register(_FakeTool(name, "search openseo"), origin=MCPToolRef(name=name, server="openseo", tool="search"))
|
|
reg.unregister(name)
|
|
reg.register(_FakeTool("real_one", "a cataloged tool"))
|
|
ctrl = ToolSearchController(reg, always_visible=set(), compaction_threshold=0)
|
|
reg.register(ToolSearchTool(ctrl))
|
|
out = await ctrl.call(name, {})
|
|
assert "not available" in out
|
|
assert "tool_search" in out
|
|
# Same answer as a name that never existed: the registry keeps no record
|
|
# of what it used to hold, so there is nothing to say differently.
|
|
assert out.replace(name, "GHOST") == await ctrl.call("GHOST", {})
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_channel_withheld_tool_is_refused_as_absent(self):
|
|
# The channel filter must not become reachable by naming a tool
|
|
# directly: "it exists but not for you" is the way around the filter
|
|
# that keeping it out of the schema was meant to close.
|
|
class _Bound(_FakeTool):
|
|
channels = frozenset({"web"})
|
|
|
|
reg = ToolRegistry()
|
|
reg.register(_Bound("only_on_web", "web-only tool"))
|
|
reg.set_channel("telegram")
|
|
ctrl = ToolSearchController(reg, always_visible=set())
|
|
|
|
out = await ctrl.call("only_on_web", {})
|
|
assert "not available" in out
|
|
# And it must not read as "it exists, just not for you" -- pointing at
|
|
# the catalog is safe because ``search`` filters by channel too.
|
|
assert [h["name"] for h in ctrl.search("web-only tool")] == []
|
|
|
|
def test_the_search_index_never_offers_an_unregistered_tool(self):
|
|
from raven.mcp.naming import MCPToolRef
|
|
|
|
reg = ToolRegistry()
|
|
name = "mcp_openseo_search"
|
|
reg.register(_FakeTool(name, "search openseo"), origin=MCPToolRef(name=name, server="openseo", tool="search"))
|
|
reg.unregister(name)
|
|
ctrl = ToolSearchController(reg, always_visible=set())
|
|
ctrl.refresh()
|
|
assert [h["name"] for h in ctrl.search("openseo search")] == []
|
|
|
|
|
|
def test_tool_call_describes_the_provenance_a_report_gives_a_name() -> None:
|
|
"""A schema-hidden tool can be named by something that is not a tool result.
|
|
|
|
The DAG exception report reaches the model as an injected turn, not as the
|
|
result of a call it made, and it tells the model to invoke `resolve_dag_node`
|
|
through this tool. A description enumerating only `tool_search` and "another
|
|
tool's result" contradicts that instruction at the moment it has to be obeyed,
|
|
which is the same broken promise the hidden-tool route exists to close.
|
|
"""
|
|
tool = ToolCallTool(ToolSearchController(ToolRegistry(), always_visible=set()))
|
|
# Both authored strings ship to the provider inside one tool definition, and the
|
|
# second is what the model is reading at the moment it fills that argument in, so
|
|
# a provenance named in one and not the other is a contradiction on the wire.
|
|
texts = {
|
|
"description": tool.description,
|
|
"name parameter": tool.parameters["properties"]["name"]["description"],
|
|
}
|
|
|
|
for where, text in texts.items():
|
|
assert TOOL_SEARCH_NAME in text, f"{where}: the search-first provenance stays"
|
|
assert "result" in text, f"{where}: and the tool-result provenance stays"
|
|
assert any(word in text for word in ("report", "notice")), (
|
|
f"{where}: a name handed over in a report or notice is a third provenance and has "
|
|
f"to be named, or the DAG report asks for something this text denies: {text!r}"
|
|
)
|