## Summary Retires the built-in `deep_research` tool: the MiroThinker HTTP tool, its unconfigured stand-in of the same name, the async channel manager, the `raven deep-research` command group, `tools.deepResearch` / `DeepResearchToolConfig`, the loop's registration / mid-session promotion / broker and submit wiring, the `inline_tool_stream` flag (its only consumer was the tool's stream callback) and the `delivered_sink` plumbing from #705 (its only writer was that callback). The WebUI settings page loses the row and its MiroThinker key panel, the ACP command menu loses `deep-research`, and the three transcript i18n keys go (readers fall back to the raw name). Research keeps two surfaces: the Raven-Research agent and the MiroThinker sub-agent preset; neither shared code with the tool. `TurnRequest.deliver_text` stays as a spine contract with its own tests, although nothing in-tree produces it any more; the comments that used deep research as its example now name the sub-agent relay. Config: a load-time migration (floor 10) drops `tools.deepResearch` / `tools.deep_research`, one notice per spelling it finds, silent and idempotent when nothing changes (it re-runs on every load for configs parked by the research rename, and `load_raven_config` calls it a third time without writing the stamp). A `deep_research` entry in `tools.disabledTools` is left as written: that list is the general name denylist and a plugin tool may carry the name (plugin tools register last so one can shadow a built-in), so dropping the entry would put such a tool back on offer. The first revision stripped it; the review caught that and the strip is gone. Kept on purpose (see the grep gate below): the MiroThinker sub-agent preset and its fixtures, `agents/raven-research/**`, the four trajectory cassettes that recorded the tool's schema (the replay serves the recording's own tool list), `benchmarks/skill_evals/queries.jsonl`, and the frozen vendored-fork fixtures, and the `deep_research` deny rows in the five `agents/*/config.json` files (the same denylist argument as the migration; the launcher parity tests are unchanged). ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [x] Refactor - [ ] Other ## Verification - `uv run ruff check raven/ tests/` and `uv run ruff format --check raven/ tests/`: clean. - `uv run pytest -q` (whole suite, with coverage, at the rebased head): 25840 passed, 109 skipped; `python scripts/coverage_gate.py ratchet --tolerance 0.05`: line 90.10% vs baseline 87.30%, branch 83.73% vs 79.43%, passed. - `uv run pytest tests/test_config_loader.py tests/test_config_update_tools.py tests/test_agent_loop_run_emit.py tests/test_cli_one_shot_spine.py tests/test_rpc_bootstrap.py tests/test_rpc_spine.py tests/test_rpc_turn_send.py tests/test_turn_wire_shape_conformance.py tests/test_core_cron_stack.py tests/test_cli_gateway_commands.py tests/test_cli_smoke.py tests/test_acp_methods.py tests/test_rpc_console.py tests/test_tool_registry_timeout.py tests/test_question_broker.py tests/test_provider_auth_method.py tests/test_agents_ppt_launcher.py tests/test_agents_design_launcher.py tests/test_agents_research_launcher.py tests/test_tool_capabilities.py tests/test_l4_entrances.py tests/test_settings_tool_groups.py tests/test_trajectory_regressions.py tests/test_living_docs.py tests/test_cli_onboard_commands.py tests/test_cli_theme.py -q`: 1500 passed. - Mutation check on the migration: with the `_migrate_config` dispatch neutered, three of the four new `tests/test_config_loader.py` cases fail; with the `_persist_migrations` dispatch neutered, `test_the_deep_research_section_also_leaves_the_file_on_disk` fails; with the pre-review stripping behaviour restored, all four fail. Restored: 74 passed. - `npm test --prefix ui-web` (208 files, 2884 tests), `npm run lint --prefix ui-web`, `npm run type-check --prefix ui-web`, `npm run gen:check --prefix ui-web`: pass. - `npm run lint:i18n --prefix ui-tui`, `npm run type-check --prefix ui-tui`, `npm test --prefix ui-tui` (144 files, 2106 tests): pass; `messages.generated.ts` regenerated with `npm run gen:i18n`. - Grep gate: `grep -rniE 'deep[ _-]?research' .` (excluding node_modules, .venv, dist) hits only the kept surfaces listed above plus CHANGELOG history. - Real machine, isolated `RAVEN_HOME` whose config carried `tools.deepResearch` and `disabledTools: ["deep_research", "exec"]` with a version-9 stamp: `raven agent -m "..."` printed the `Config updated:` notice for `tools.deepResearch` once, answered, and left the file without that section, `disabledTools` unchanged (`["deep_research", "exec"]`), stamp 10; `raven deep-research` answers `No such command`; the served WebUI (`raven serve`) Settings > Tools page lists 36 rows with no `deep_research` and `exec` still off, and `ext.list` carries no `deep_research`. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - User-visible: the `deep_research` tool, the settings row and `raven deep-research` are gone; a config that still carries the section loses it once at load with a notice; a `deep_research` entry in `tools.disabledTools` stays as written. Reports the tool wrote under `<workspace>/deep_research/` stay on disk. Archived transcripts that recorded a `deep_research` call replay with the generic icon and the raw tool name instead of the `researched` verb. - Coverage: the two deleted modules were well covered, but the live totals sit several points above the committed baseline, so the ratchet passes with room to spare (measured locally, see Verification). - Rollback: revert the PR; the migration is additive on the stamp, so a reverted build reads a migrated config unchanged (the section it would have read is simply absent). - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues #156 (its deep-research half no longer applies; tracing half unaffected) --------- Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
264 lines
9.6 KiB
Python
264 lines
9.6 KiB
Python
"""Per-tool timeout wrapping in ``ToolRegistry.execute``.
|
|
|
|
The registry wraps every non-blocking tool in ``asyncio.wait_for`` so a tool
|
|
without its own timeout can't wedge the agent loop. Covers:
|
|
- a hanging tool is killed at the ceiling and returns an error (no hang)
|
|
- a fast tool returns its result normally
|
|
- ``timeout_seconds`` overrides the registry default
|
|
- ``blocking_interaction`` tools are NOT wrapped (run past the ceiling)
|
|
- ``is_blocking`` reports that flag (the fact source the web channel mirrors)
|
|
- a CancelledError (e.g. /stop) is not swallowed as a tool error
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import time
|
|
|
|
import pytest
|
|
|
|
from raven.agent.tools.registry import ToolRegistry
|
|
from raven.contracts.tool import Tool
|
|
|
|
|
|
class _SleepTool(Tool):
|
|
"""Sleeps for ``delay`` then returns 'done'. Configurable timeout/blocking."""
|
|
|
|
def __init__(self, delay: float, *, timeout_seconds=None, blocking=False):
|
|
self._delay = delay
|
|
self.timeout_seconds = timeout_seconds
|
|
self.blocking_interaction = blocking
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "sleeper"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "sleeps"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}}
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
await asyncio.sleep(self._delay)
|
|
return "done"
|
|
|
|
|
|
def _registry(tool: Tool, default_timeout: float | None = None) -> ToolRegistry:
|
|
reg = ToolRegistry()
|
|
if default_timeout is not None:
|
|
reg.DEFAULT_TOOL_TIMEOUT_S = default_timeout
|
|
reg.register(tool)
|
|
return reg
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_fast_tool_returns_result():
|
|
reg = _registry(_SleepTool(0.0), default_timeout=1.0)
|
|
assert await reg.execute("sleeper", {}) == "done"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_hanging_tool_times_out_at_default_ceiling():
|
|
reg = _registry(_SleepTool(5.0), default_timeout=0.05)
|
|
result = await reg.execute("sleeper", {})
|
|
assert "timed out after" in result
|
|
assert "try a different approach" in result
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_per_tool_timeout_seconds_overrides_default():
|
|
# default would allow it, but the tool's own tighter ceiling fires first
|
|
reg = _registry(_SleepTool(5.0, timeout_seconds=0.05), default_timeout=100.0)
|
|
result = await reg.execute("sleeper", {})
|
|
assert "timed out after 0s" in result
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_blocking_interaction_tool_is_not_wrapped():
|
|
# Sleeps well past the ceiling, but blocking tools are never timer-killed.
|
|
reg = _registry(_SleepTool(0.2, blocking=True), default_timeout=0.05)
|
|
assert await reg.execute("sleeper", {}) == "done"
|
|
|
|
|
|
def test_is_blocking_mirrors_the_tool_flag():
|
|
assert _registry(_SleepTool(0.0, blocking=True)).is_blocking("sleeper") is True
|
|
assert _registry(_SleepTool(0.0)).is_blocking("sleeper") is False
|
|
|
|
|
|
def test_is_blocking_unknown_tool_is_false():
|
|
assert ToolRegistry().is_blocking("no-such-tool") is False
|
|
|
|
|
|
def test_every_subagent_invoking_tool_is_blocking():
|
|
"""A tool that runs a sub-agent must declare ``blocking_interaction``.
|
|
|
|
The flag is the single fact source the web channel mirrors onto
|
|
``tool.start`` to suspend its turn-stream idle clock. A sub-agent run stays
|
|
silent for far longer than that clock, so a tool missing the flag ends the
|
|
turn in the web UI while the run is still going.
|
|
"""
|
|
from raven.agent.subagent.dag_tool import SubAgentDagTool
|
|
from raven.agent.subagent.spawn_tool import SpawnTool
|
|
|
|
for cls in (SpawnTool, SubAgentDagTool):
|
|
assert cls.blocking_interaction is True, cls.__name__
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_cancelled_error_propagates_not_swallowed():
|
|
class _CancelTool(_SleepTool):
|
|
async def execute(self, **kwargs) -> str:
|
|
raise asyncio.CancelledError()
|
|
|
|
reg = _registry(_CancelTool(0.0), default_timeout=1.0)
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await reg.execute("sleeper", {})
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_long_running_tools_keep_generous_ceilings():
|
|
# Guard against regressing the overrides on the genuinely-slow tools.
|
|
from raven.agent.subagent.spawn_tool import SpawnTool
|
|
from raven.agent.tools.media_gen import VideoGenerateTool
|
|
from raven.agent.tools.shell import ExecTool
|
|
|
|
assert ExecTool.timeout_seconds >= 600
|
|
assert VideoGenerateTool.timeout_seconds >= 600
|
|
assert SpawnTool.timeout_seconds >= 600
|
|
# Default-class tools inherit None -> registry default applies.
|
|
assert Tool.timeout_seconds is None
|
|
assert Tool.blocking_interaction is False
|
|
|
|
|
|
class _HeartbeatLog:
|
|
"""Capture the registry's own INFO lines for the duration of a call."""
|
|
|
|
def __init__(self) -> None:
|
|
self.lines: list[str] = []
|
|
self.gaps: list[float] = []
|
|
self._sink: int | None = None
|
|
self._last = 0.0
|
|
|
|
def __enter__(self) -> "_HeartbeatLog":
|
|
from loguru import logger
|
|
|
|
self._last = time.monotonic()
|
|
self._sink = logger.add(self._record, level="INFO")
|
|
return self
|
|
|
|
def _record(self, message) -> None:
|
|
now = time.monotonic()
|
|
self.lines.append(message.record["message"])
|
|
if "still running" in message.record["message"]:
|
|
self.gaps.append(now - self._last)
|
|
self._last = now
|
|
|
|
def __exit__(self, *exc: object) -> None:
|
|
from loguru import logger
|
|
|
|
if self._sink is not None:
|
|
logger.remove(self._sink)
|
|
|
|
@property
|
|
def beats(self) -> list[str]:
|
|
return [line for line in self.lines if "still running" in line]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_tool_that_outlives_the_first_beat_says_so(monkeypatch):
|
|
"""Between its start line and its result line a tool said nothing, so a
|
|
running call and a stopped process left the same record. Measured on a real
|
|
run: ``ppt_prepare`` held the turn 600s twice and the whole window holds
|
|
three lines -- the call, the timeout, and the next iteration."""
|
|
from raven.agent.tools import registry as registry_module
|
|
|
|
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 0.02)
|
|
monkeypatch.setattr(registry_module, "HEARTBEAT_EVERY_S", 0.02)
|
|
reg = _registry(_SleepTool(0.12), default_timeout=5.0)
|
|
|
|
with _HeartbeatLog() as log:
|
|
assert await reg.execute("sleeper", {}) == "done"
|
|
|
|
assert log.beats, "a tool running six beats' worth of time logged nothing"
|
|
assert "sleeper" in log.beats[0]
|
|
# The ceiling is in the line because "still running" alone does not say
|
|
# whether the call is near being killed or nowhere near it.
|
|
assert "5s ceiling" in log.beats[0]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_ordinary_call_is_silent(monkeypatch):
|
|
"""The first beat is late on purpose. A line per tool call would bury the
|
|
slow one it exists to point at."""
|
|
from raven.agent.tools import registry as registry_module
|
|
|
|
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 5.0)
|
|
reg = _registry(_SleepTool(0.0), default_timeout=5.0)
|
|
|
|
with _HeartbeatLog() as log:
|
|
assert await reg.execute("sleeper", {}) == "done"
|
|
|
|
assert log.beats == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_beat_stops_when_the_call_does(monkeypatch):
|
|
"""Otherwise the registry would go on reporting a tool that had returned --
|
|
a line saying "still running" about a finished call is worse than none."""
|
|
from raven.agent.tools import registry as registry_module
|
|
|
|
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 0.02)
|
|
monkeypatch.setattr(registry_module, "HEARTBEAT_EVERY_S", 0.02)
|
|
reg = _registry(_SleepTool(0.06), default_timeout=5.0)
|
|
|
|
with _HeartbeatLog() as log:
|
|
await reg.execute("sleeper", {})
|
|
settled = len(log.beats)
|
|
await asyncio.sleep(0.15)
|
|
|
|
assert len(log.beats) == settled
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_timed_out_call_still_reports_the_timeout(monkeypatch):
|
|
"""The heartbeat is a bystander: it must not hold the result or move the
|
|
deadline. This is the case that would notice either."""
|
|
from raven.agent.tools import registry as registry_module
|
|
|
|
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 0.01)
|
|
monkeypatch.setattr(registry_module, "HEARTBEAT_EVERY_S", 0.01)
|
|
reg = _registry(_SleepTool(5.0), default_timeout=0.05)
|
|
|
|
with _HeartbeatLog() as log:
|
|
result = await reg.execute("sleeper", {})
|
|
|
|
assert "timed out after" in result
|
|
assert log.beats
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_first_beat_is_late_and_the_rest_are_not(monkeypatch):
|
|
"""Two different intervals, and the difference is the whole design: the
|
|
first is late so ordinary calls stay silent, and the ones after it are not,
|
|
so a reader tailing a slow call sees it advancing rather than waiting the
|
|
same long gap again. A single interval can only have one of those."""
|
|
from raven.agent.tools import registry as registry_module
|
|
|
|
monkeypatch.setattr(registry_module, "HEARTBEAT_FIRST_S", 0.30)
|
|
monkeypatch.setattr(registry_module, "HEARTBEAT_EVERY_S", 0.02)
|
|
reg = _registry(_SleepTool(0.60), default_timeout=5.0)
|
|
|
|
with _HeartbeatLog() as log:
|
|
await reg.execute("sleeper", {})
|
|
|
|
# The two gaps, not the number of beats: a loaded box delivers however many
|
|
# it can get round to -- four workers on a small runner once turned the
|
|
# fifteen this should produce into three -- but it cannot make the wait
|
|
# before the first beat short or the wait after it long.
|
|
assert len(log.gaps) >= 2, log.beats
|
|
assert log.gaps[0] >= 0.25, f"the first beat did not wait out HEARTBEAT_FIRST_S: {log.gaps}"
|
|
assert log.gaps[1] < log.gaps[0] / 2, f"the beats after the first are on the same interval: {log.gaps}"
|