Raven-1/tests/test_tool_registry_timeout.py
KT a7f102c292
refactor(*): retire the deep_research tool in favour of the research agents (#717)
## Summary

Retires the built-in `deep_research` tool: the MiroThinker HTTP tool,
its
unconfigured stand-in of the same name, the async channel manager, the
`raven deep-research` command group, `tools.deepResearch` /
`DeepResearchToolConfig`, the loop's registration / mid-session
promotion /
broker and submit wiring, the `inline_tool_stream` flag (its only
consumer
was the tool's stream callback) and the `delivered_sink` plumbing from
#705
(its only writer was that callback). The WebUI settings page loses the
row
and its MiroThinker key panel, the ACP command menu loses
`deep-research`,
and the three transcript i18n keys go (readers fall back to the raw
name).

Research keeps two surfaces: the Raven-Research agent and the
MiroThinker
sub-agent preset; neither shared code with the tool.
`TurnRequest.deliver_text`
stays as a spine contract with its own tests, although nothing in-tree
produces it any more; the comments that used deep research as its
example
now name the sub-agent relay.

Config: a load-time migration (floor 10) drops `tools.deepResearch` /
`tools.deep_research`, one notice per spelling it finds, silent and
idempotent when nothing changes (it
re-runs on every load for configs parked by the research rename, and
`load_raven_config` calls it a third time without writing the stamp). A
`deep_research` entry in `tools.disabledTools` is left as written: that
list
is the general name denylist and a plugin tool may carry the name
(plugin
tools register last so one can shadow a built-in), so dropping the entry
would put such a tool back on offer. The first revision stripped it; the
review caught that and the strip is gone.

Kept on purpose (see the grep gate below): the MiroThinker sub-agent
preset
and its fixtures, `agents/raven-research/**`, the four trajectory
cassettes
that recorded the tool's schema (the replay serves the recording's own
tool
list), `benchmarks/skill_evals/queries.jsonl`, and the frozen
vendored-fork
fixtures, and the `deep_research` deny rows in the five
`agents/*/config.json`
files (the same denylist argument as the migration; the launcher parity
tests
are unchanged).

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [x] Refactor
- [ ] Other

## Verification

- `uv run ruff check raven/ tests/` and `uv run ruff format --check
raven/ tests/`: clean.
- `uv run pytest -q` (whole suite, with coverage, at the rebased head):
25840 passed, 109 skipped; `python scripts/coverage_gate.py ratchet
--tolerance 0.05`: line 90.10% vs baseline 87.30%, branch 83.73% vs
79.43%, passed.
- `uv run pytest tests/test_config_loader.py
tests/test_config_update_tools.py tests/test_agent_loop_run_emit.py
tests/test_cli_one_shot_spine.py tests/test_rpc_bootstrap.py
tests/test_rpc_spine.py tests/test_rpc_turn_send.py
tests/test_turn_wire_shape_conformance.py tests/test_core_cron_stack.py
tests/test_cli_gateway_commands.py tests/test_cli_smoke.py
tests/test_acp_methods.py tests/test_rpc_console.py
tests/test_tool_registry_timeout.py tests/test_question_broker.py
tests/test_provider_auth_method.py tests/test_agents_ppt_launcher.py
tests/test_agents_design_launcher.py
tests/test_agents_research_launcher.py tests/test_tool_capabilities.py
tests/test_l4_entrances.py tests/test_settings_tool_groups.py
tests/test_trajectory_regressions.py tests/test_living_docs.py
tests/test_cli_onboard_commands.py tests/test_cli_theme.py -q`: 1500
passed.
- Mutation check on the migration: with the `_migrate_config` dispatch
neutered, three of the four new `tests/test_config_loader.py` cases
fail; with the `_persist_migrations` dispatch neutered,
`test_the_deep_research_section_also_leaves_the_file_on_disk` fails;
with the pre-review stripping behaviour restored, all four fail.
Restored: 74 passed.
- `npm test --prefix ui-web` (208 files, 2884 tests), `npm run lint
--prefix ui-web`, `npm run type-check --prefix ui-web`, `npm run
gen:check --prefix ui-web`: pass.
- `npm run lint:i18n --prefix ui-tui`, `npm run type-check --prefix
ui-tui`, `npm test --prefix ui-tui` (144 files, 2106 tests): pass;
`messages.generated.ts` regenerated with `npm run gen:i18n`.
- Grep gate: `grep -rniE 'deep[ _-]?research' .` (excluding
node_modules, .venv, dist) hits only the kept surfaces listed above plus
CHANGELOG history.
- Real machine, isolated `RAVEN_HOME` whose config carried
`tools.deepResearch` and `disabledTools: ["deep_research", "exec"]` with
a version-9 stamp: `raven agent -m "..."` printed the `Config updated:`
notice for `tools.deepResearch` once, answered, and left the file
without that section, `disabledTools` unchanged (`["deep_research",
"exec"]`), stamp 10; `raven deep-research` answers `No such command`;
the served WebUI (`raven serve`) Settings > Tools page lists 36 rows
with no `deep_research` and `exec` still off, and `ext.list` carries no
`deep_research`.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- User-visible: the `deep_research` tool, the settings row and `raven
deep-research` are gone; a config that still carries the section loses
it once at load with a notice; a `deep_research` entry in
`tools.disabledTools` stays as written. Reports the tool wrote under
`<workspace>/deep_research/` stay on disk. Archived transcripts that
recorded a `deep_research` call replay with the generic icon and the raw
tool name instead of the `researched` verb.
- Coverage: the two deleted modules were well covered, but the live
totals sit several points above the committed baseline, so the ratchet
passes with room to spare (measured locally, see Verification).
- Rollback: revert the PR; the migration is additive on the stamp, so a
reverted build reads a migrated config unchanged (the section it would
have read is simply absent).

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

#156 (its deep-research half no longer applies; tracing half unaffected)

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-23 18:19:42 +08:00

264 lines
9.6 KiB
Python

"""Per-tool timeout wrapping in ``ToolRegistry.execute``.
The registry wraps every non-blocking tool in ``asyncio.wait_for`` so a tool
without its own timeout can't wedge the agent loop. Covers:
- a hanging tool is killed at the ceiling and returns an error (no hang)
- a fast tool returns its result normally
- ``timeout_seconds`` overrides the registry default
- ``blocking_interaction`` tools are NOT wrapped (run past the ceiling)
- ``is_blocking`` reports that flag (the fact source the web channel mirrors)
- a CancelledError (e.g. /stop) is not swallowed as a tool error
"""
from __future__ import annotations
import asyncio
import time
import pytest
from raven.agent.tools.registry import ToolRegistry
from raven.contracts.tool import Tool
class _SleepTool(Tool):
"""Sleeps for ``delay`` then returns 'done'. Configurable timeout/blocking."""
def __init__(self, delay: float, *, timeout_seconds=None, blocking=False):
self._delay = delay
self.timeout_seconds = timeout_seconds
self.blocking_interaction = blocking
@property
def name(self) -> str:
return "sleeper"
@property
def description(self) -> str:
return "sleeps"
@property
def parameters(self) -> dict:
return {"type": "object", "properties": {}}
async def execute(self, **kwargs) -> str:
await asyncio.sleep(self._delay)
return "done"
def _registry(tool: Tool, default_timeout: float | None = None) -> ToolRegistry:
reg = ToolRegistry()
if default_timeout is not None:
reg.DEFAULT_TOOL_TIMEOUT_S = default_timeout
reg.register(tool)
return reg
@pytest.mark.asyncio
async def test_fast_tool_returns_result():
reg = _registry(_SleepTool(0.0), default_timeout=1.0)
assert await reg.execute("sleeper", {}) == "done"
@pytest.mark.asyncio
async def test_hanging_tool_times_out_at_default_ceiling():
reg = _registry(_SleepTool(5.0), default_timeout=0.05)
result = await reg.execute("sleeper", {})
assert "timed out after" in result
assert "try a different approach" in result
@pytest.mark.asyncio
async def test_per_tool_timeout_seconds_overrides_default():
# default would allow it, but the tool's own tighter ceiling fires first
reg = _registry(_SleepTool(5.0, timeout_seconds=0.05), default_timeout=100.0)
result = await reg.execute("sleeper", {})
assert "timed out after 0s" in result
@pytest.mark.asyncio
async def test_blocking_interaction_tool_is_not_wrapped():
# Sleeps well past the ceiling, but blocking tools are never timer-killed.
reg = _registry(_SleepTool(0.2, blocking=True), default_timeout=0.05)
assert await reg.execute("sleeper", {}) == "done"
def test_is_blocking_mirrors_the_tool_flag():
assert _registry(_SleepTool(0.0, blocking=True)).is_blocking("sleeper") is True
assert _registry(_SleepTool(0.0)).is_blocking("sleeper") is False
def test_is_blocking_unknown_tool_is_false():
assert ToolRegistry().is_blocking("no-such-tool") is False
def test_every_subagent_invoking_tool_is_blocking():
"""A tool that runs a sub-agent must declare ``blocking_interaction``.
The flag is the single fact source the web channel mirrors onto
``tool.start`` to suspend its turn-stream idle clock. A sub-agent run stays
silent for far longer than that clock, so a tool missing the flag ends the
turn in the web UI while the run is still going.
"""
from raven.agent.subagent.dag_tool import SubAgentDagTool
from raven.agent.subagent.spawn_tool import SpawnTool
for cls in (SpawnTool, SubAgentDagTool):
assert cls.blocking_interaction is True, cls.__name__
@pytest.mark.asyncio
async def test_cancelled_error_propagates_not_swallowed():
class _CancelTool(_SleepTool):
async def execute(self, **kwargs) -> str:
raise asyncio.CancelledError()
reg = _registry(_CancelTool(0.0), default_timeout=1.0)
with pytest.raises(asyncio.CancelledError):
await reg.execute("sleeper", {})
@pytest.mark.asyncio
async def test_long_running_tools_keep_generous_ceilings():
# Guard against regressing the overrides on the genuinely-slow tools.
from raven.agent.subagent.spawn_tool import SpawnTool
from raven.agent.tools.media_gen import VideoGenerateTool
from raven.agent.tools.shell import ExecTool
assert ExecTool.timeout_seconds >= 600
assert VideoGenerateTool.timeout_seconds >= 600
assert SpawnTool.timeout_seconds >= 600
# Default-class tools inherit None -> registry default applies.
assert Tool.timeout_seconds is None
assert Tool.blocking_interaction is False
class _HeartbeatLog:
"""Capture the registry's own INFO lines for the duration of a call."""
def __init__(self) -> None:
self.lines: list[str] = []
self.gaps: list[float] = []
self._sink: int | None = None
self._last = 0.0
def __enter__(self) -> "_HeartbeatLog":
from loguru import logger
self._last = time.monotonic()
self._sink = logger.add(self._record, level="INFO")
return self
def _record(self, message) -> None:
now = time.monotonic()
self.lines.append(message.record["message"])
if "still running" in message.record["message"]:
self.gaps.append(now - self._last)
self._last = now
def __exit__(self, *exc: object) -> None:
from loguru import logger
if self._sink is not None:
logger.remove(self._sink)
@property
def beats(self) -> list[str]:
return [line for line in self.lines if "still running" in line]
@pytest.mark.asyncio
async def test_a_tool_that_outlives_the_first_beat_says_so(monkeypatch):
"""Between its start line and its result line a tool said nothing, so a
running call and a stopped process left the same record. Measured on a real
run: ``ppt_prepare`` held the turn 600s twice and the whole window holds
three lines -- the call, the timeout, and the next iteration."""
from raven.agent.tools import registry as registry_module
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 0.02)
monkeypatch.setattr(registry_module, "HEARTBEAT_EVERY_S", 0.02)
reg = _registry(_SleepTool(0.12), default_timeout=5.0)
with _HeartbeatLog() as log:
assert await reg.execute("sleeper", {}) == "done"
assert log.beats, "a tool running six beats' worth of time logged nothing"
assert "sleeper" in log.beats[0]
# The ceiling is in the line because "still running" alone does not say
# whether the call is near being killed or nowhere near it.
assert "5s ceiling" in log.beats[0]
@pytest.mark.asyncio
async def test_an_ordinary_call_is_silent(monkeypatch):
"""The first beat is late on purpose. A line per tool call would bury the
slow one it exists to point at."""
from raven.agent.tools import registry as registry_module
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 5.0)
reg = _registry(_SleepTool(0.0), default_timeout=5.0)
with _HeartbeatLog() as log:
assert await reg.execute("sleeper", {}) == "done"
assert log.beats == []
@pytest.mark.asyncio
async def test_the_beat_stops_when_the_call_does(monkeypatch):
"""Otherwise the registry would go on reporting a tool that had returned --
a line saying "still running" about a finished call is worse than none."""
from raven.agent.tools import registry as registry_module
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 0.02)
monkeypatch.setattr(registry_module, "HEARTBEAT_EVERY_S", 0.02)
reg = _registry(_SleepTool(0.06), default_timeout=5.0)
with _HeartbeatLog() as log:
await reg.execute("sleeper", {})
settled = len(log.beats)
await asyncio.sleep(0.15)
assert len(log.beats) == settled
@pytest.mark.asyncio
async def test_a_timed_out_call_still_reports_the_timeout(monkeypatch):
"""The heartbeat is a bystander: it must not hold the result or move the
deadline. This is the case that would notice either."""
from raven.agent.tools import registry as registry_module
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 0.01)
monkeypatch.setattr(registry_module, "HEARTBEAT_EVERY_S", 0.01)
reg = _registry(_SleepTool(5.0), default_timeout=0.05)
with _HeartbeatLog() as log:
result = await reg.execute("sleeper", {})
assert "timed out after" in result
assert log.beats
@pytest.mark.asyncio
async def test_the_first_beat_is_late_and_the_rest_are_not(monkeypatch):
"""Two different intervals, and the difference is the whole design: the
first is late so ordinary calls stay silent, and the ones after it are not,
so a reader tailing a slow call sees it advancing rather than waiting the
same long gap again. A single interval can only have one of those."""
from raven.agent.tools import registry as registry_module
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 0.30)
monkeypatch.setattr(registry_module, "HEARTBEAT_EVERY_S", 0.02)
reg = _registry(_SleepTool(0.60), default_timeout=5.0)
with _HeartbeatLog() as log:
await reg.execute("sleeper", {})
# The two gaps, not the number of beats: a loaded box delivers however many
# it can get round to -- four workers on a small runner once turned the
# fifteen this should produce into three -- but it cannot make the wait
# before the first beat short or the wait after it long.
assert len(log.gaps) >= 2, log.beats
assert log.gaps[0] >= 0.25, f"the first beat did not wait out HEARTBEAT_FIRST_S: {log.gaps}"
assert log.gaps[1] < log.gaps[0] / 2, f"the beats after the first are on the same interval: {log.gaps}"