## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
412 lines
14 KiB
Python
412 lines
14 KiB
Python
"""Tests for the legacy-callback adapter hooks.
|
|
|
|
Each adapter wraps a pre-existing AgentLoop constructor parameter so
|
|
the legacy invocation semantics are preserved while internal call
|
|
sites go through a single ``CompositeHook`` chain.
|
|
|
|
Two layers of coverage here:
|
|
|
|
1. **Adapter-level** unit tests — exercise each adapter in isolation
|
|
(no AgentLoop) so we pin the translation contract.
|
|
2. **AgentLoop wire-up** tests — verify the constructor actually builds
|
|
``self.hooks`` with the expected composition order from the three
|
|
legacy params, and that an explicit ``hooks`` argument is inserted
|
|
between the on-user-inbound and response-modifier slots.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import tempfile
|
|
from dataclasses import dataclass, field
|
|
from pathlib import Path
|
|
from typing import Any
|
|
from unittest.mock import AsyncMock
|
|
|
|
import pytest
|
|
|
|
from raven.agent.hook import (
|
|
AgentHook,
|
|
AgentHookContext,
|
|
CompositeHook,
|
|
DecisionConsumerAdapter,
|
|
OnUserInboundAdapter,
|
|
ResponseModifierAdapter,
|
|
)
|
|
from raven.spine.message import ChatType, Source
|
|
from raven.spine.turn import Origin, TurnRequest
|
|
from tests._wiring import wire
|
|
|
|
|
|
@dataclass
|
|
class _Reply:
|
|
channel: str
|
|
chat_id: str
|
|
content: str
|
|
media: list[str] = field(default_factory=list)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Shared fixtures
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _msg(text: str = "hello", **extras: Any) -> TurnRequest:
|
|
return TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(
|
|
channel="cli",
|
|
chat_id="c1",
|
|
sender_id="user",
|
|
chat_type=ChatType.DM,
|
|
extras=extras or {},
|
|
),
|
|
text=text,
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def ctx():
|
|
return AgentHookContext(session_key="cli:test", turn_request=_msg())
|
|
|
|
|
|
# ===========================================================================
|
|
# OnUserInboundAdapter
|
|
# ===========================================================================
|
|
|
|
|
|
class TestOnUserInboundAdapter:
|
|
async def test_sync_callback_invoked(self, ctx):
|
|
recorded = []
|
|
|
|
adapter = OnUserInboundAdapter(lambda m: recorded.append(m.text))
|
|
decision = await adapter.before_user_inbound(ctx)
|
|
|
|
assert recorded == ["hello"]
|
|
# Observer — never short-circuits.
|
|
assert decision.short_circuit_result is None
|
|
assert decision.modified_content is None
|
|
|
|
async def test_async_callback_awaited(self, ctx):
|
|
recorded = []
|
|
|
|
async def cb(m):
|
|
recorded.append(m.text)
|
|
|
|
adapter = OnUserInboundAdapter(cb)
|
|
await adapter.before_user_inbound(ctx)
|
|
assert recorded == ["hello"]
|
|
|
|
async def test_sync_exception_swallowed(self, ctx):
|
|
def bad(m):
|
|
raise RuntimeError("boom")
|
|
|
|
adapter = OnUserInboundAdapter(bad)
|
|
# Should NOT raise — observer must not crash chain.
|
|
decision = await adapter.before_user_inbound(ctx)
|
|
assert decision.short_circuit_result is None
|
|
|
|
async def test_async_exception_swallowed(self, ctx):
|
|
async def bad(m):
|
|
raise RuntimeError("boom")
|
|
|
|
adapter = OnUserInboundAdapter(bad)
|
|
decision = await adapter.before_user_inbound(ctx)
|
|
assert decision.short_circuit_result is None
|
|
|
|
async def test_no_inbound_message_is_noop(self):
|
|
called = []
|
|
ctx_no_msg = AgentHookContext(session_key="cli:test") # turn_request=None
|
|
adapter = OnUserInboundAdapter(lambda m: called.append(m))
|
|
await adapter.before_user_inbound(ctx_no_msg)
|
|
assert called == []
|
|
|
|
async def test_other_phases_are_passthrough(self, ctx):
|
|
adapter = OnUserInboundAdapter(lambda m: None)
|
|
for phase in (
|
|
"before_iteration",
|
|
"before_execute_tools",
|
|
"after_iteration",
|
|
"after_send",
|
|
):
|
|
decision = await getattr(adapter, phase)(ctx)
|
|
assert decision.short_circuit_result is None
|
|
assert decision.modified_content is None
|
|
|
|
def test_name(self):
|
|
adapter = OnUserInboundAdapter(lambda m: None)
|
|
assert adapter.name == "callback(on_user_inbound)"
|
|
|
|
|
|
# ===========================================================================
|
|
# DecisionConsumerAdapter
|
|
# ===========================================================================
|
|
|
|
|
|
class TestDecisionConsumerAdapter:
|
|
async def test_none_return_means_pass_through(self, ctx):
|
|
cb = AsyncMock(return_value=None)
|
|
adapter = DecisionConsumerAdapter(cb)
|
|
|
|
decision = await adapter.before_user_inbound(ctx)
|
|
|
|
assert decision.short_circuit_result is None
|
|
cb.assert_awaited_once()
|
|
|
|
async def test_non_none_return_short_circuits(self, ctx):
|
|
handled = _Reply(channel="cli", chat_id="c1", content="menu picked")
|
|
cb = AsyncMock(return_value=handled)
|
|
adapter = DecisionConsumerAdapter(cb)
|
|
|
|
decision = await adapter.before_user_inbound(ctx)
|
|
|
|
# The reply is normalized to a (content, media) tuple.
|
|
assert decision.short_circuit_result == ("menu picked", [])
|
|
|
|
async def test_exception_treated_as_pass_through(self, ctx):
|
|
async def boom(_):
|
|
raise RuntimeError("boom")
|
|
|
|
adapter = DecisionConsumerAdapter(boom)
|
|
decision = await adapter.before_user_inbound(ctx)
|
|
assert decision.short_circuit_result is None
|
|
|
|
async def test_no_inbound_message_is_noop(self):
|
|
cb = AsyncMock(return_value="should not see")
|
|
ctx_no_msg = AgentHookContext(session_key="cli:test")
|
|
adapter = DecisionConsumerAdapter(cb)
|
|
decision = await adapter.before_user_inbound(ctx_no_msg)
|
|
assert decision.short_circuit_result is None
|
|
cb.assert_not_awaited()
|
|
|
|
def test_name(self):
|
|
adapter = DecisionConsumerAdapter(AsyncMock())
|
|
assert adapter.name == "callback(decision_consumer)"
|
|
|
|
|
|
# ===========================================================================
|
|
# ResponseModifierAdapter
|
|
# ===========================================================================
|
|
|
|
|
|
class TestResponseModifierAdapter:
|
|
async def test_modified_content_propagated(self):
|
|
ctx = AgentHookContext(session_key="cli:test", outbound_content="hi")
|
|
adapter = ResponseModifierAdapter(lambda key, content: f"{content} | session={key}")
|
|
decision = await adapter.after_send(ctx)
|
|
assert decision.modified_content == "hi | session=cli:test"
|
|
|
|
async def test_unchanged_content_returns_passthrough(self):
|
|
# If the modifier returns the same string, we must not signal a
|
|
# change (so downstream content-chaining ignores the call).
|
|
ctx = AgentHookContext(session_key="cli:test", outbound_content="hi")
|
|
adapter = ResponseModifierAdapter(lambda k, c: c)
|
|
decision = await adapter.after_send(ctx)
|
|
assert decision.modified_content is None
|
|
|
|
async def test_empty_content_still_invokes_callback(self):
|
|
called = []
|
|
|
|
def cb(key, content):
|
|
called.append((key, content))
|
|
return content + " appended"
|
|
|
|
ctx = AgentHookContext(session_key="cli:test", outbound_content=None)
|
|
adapter = ResponseModifierAdapter(cb)
|
|
decision = await adapter.after_send(ctx)
|
|
|
|
# Called with an empty string (legacy behavior — modifiers can
|
|
# inject into an otherwise empty reply).
|
|
assert called == [("cli:test", "")]
|
|
assert decision.modified_content == " appended"
|
|
|
|
async def test_exception_swallowed(self):
|
|
def boom(k, c):
|
|
raise RuntimeError("boom")
|
|
|
|
ctx = AgentHookContext(session_key="cli:test", outbound_content="hi")
|
|
adapter = ResponseModifierAdapter(boom)
|
|
decision = await adapter.after_send(ctx)
|
|
# No-op on exception — content unchanged.
|
|
assert decision.modified_content is None
|
|
|
|
async def test_non_string_return_treated_as_no_change(self):
|
|
# Defensive: if the legacy modifier returns a non-string (None,
|
|
# a future Sentinel might return an envelope), we should not
|
|
# corrupt outbound_content.
|
|
adapter = ResponseModifierAdapter(lambda k, c: None) # type: ignore[arg-type]
|
|
ctx = AgentHookContext(session_key="cli:test", outbound_content="hi")
|
|
decision = await adapter.after_send(ctx)
|
|
assert decision.modified_content is None
|
|
|
|
async def test_other_phases_are_passthrough(self):
|
|
adapter = ResponseModifierAdapter(lambda k, c: c + "!")
|
|
ctx = AgentHookContext(session_key="cli:test")
|
|
for phase in (
|
|
"before_user_inbound",
|
|
"before_iteration",
|
|
"before_execute_tools",
|
|
"after_iteration",
|
|
):
|
|
decision = await getattr(adapter, phase)(ctx)
|
|
assert decision.short_circuit_result is None
|
|
assert decision.modified_content is None
|
|
|
|
def test_name(self):
|
|
adapter = ResponseModifierAdapter(lambda k, c: c)
|
|
assert adapter.name == "callback(response_modifier)"
|
|
|
|
|
|
# ===========================================================================
|
|
# AgentLoop wire-up
|
|
# ===========================================================================
|
|
|
|
|
|
@pytest.fixture
|
|
def workspace():
|
|
with tempfile.TemporaryDirectory() as td:
|
|
yield Path(td)
|
|
|
|
|
|
class _StubProvider:
|
|
api_key = "test"
|
|
|
|
def get_default_model(self) -> str:
|
|
return "stub"
|
|
|
|
async def chat(self, *args: Any, **kwargs: Any) -> Any:
|
|
raise NotImplementedError
|
|
|
|
async def chat_with_retry(self, *args: Any, **kwargs: Any) -> Any:
|
|
raise NotImplementedError
|
|
|
|
|
|
def _make_agent(workspace, **kwargs):
|
|
from raven.agent.loop import AgentLoop
|
|
|
|
return AgentLoop(
|
|
provider=_StubProvider(),
|
|
workspace=workspace,
|
|
model="stub",
|
|
**wire(max_iterations=2, restrict_to_workspace=True, **kwargs),
|
|
)
|
|
|
|
|
|
class TestAgentLoopHookWireUp:
|
|
"""Verify the constructor takes finished hooks only: the chain is the
|
|
host's list, in the host's order, and the legacy callback params are
|
|
gone."""
|
|
|
|
def test_empty_composite_when_no_hooks(self, workspace):
|
|
agent = _make_agent(workspace)
|
|
assert isinstance(agent.hooks, CompositeHook)
|
|
assert len(agent.hooks) == 0
|
|
|
|
def test_hooks_extend_in_list_order(self, workspace):
|
|
class Explicit(AgentHook):
|
|
@property
|
|
def name(self) -> str:
|
|
return "Explicit"
|
|
|
|
agent = _make_agent(
|
|
workspace,
|
|
hooks=CompositeHook(
|
|
[
|
|
OnUserInboundAdapter(lambda m: None),
|
|
DecisionConsumerAdapter(AsyncMock()),
|
|
Explicit(),
|
|
ResponseModifierAdapter(lambda k, c: c),
|
|
]
|
|
),
|
|
)
|
|
|
|
hooks = list(agent.hooks)
|
|
assert [type(h) for h in hooks] == [
|
|
OnUserInboundAdapter,
|
|
DecisionConsumerAdapter,
|
|
Explicit,
|
|
ResponseModifierAdapter,
|
|
]
|
|
|
|
def test_the_loop_grows_no_legacy_callback_params(self, workspace):
|
|
# The retirement's bite: a caller still passing the old callables
|
|
# gets the typo error, not a silent drop.
|
|
import pytest
|
|
|
|
for legacy in ("on_user_inbound", "decision_consumer", "response_modifier"):
|
|
with pytest.raises(TypeError):
|
|
_make_agent(workspace, **{legacy: lambda *a: None})
|
|
|
|
async def test_user_inbound_observer_fires_when_no_short_circuit(self, workspace):
|
|
received = []
|
|
agent = _make_agent(
|
|
workspace,
|
|
hooks=CompositeHook(
|
|
[
|
|
OnUserInboundAdapter(lambda m: received.append(m.text)),
|
|
DecisionConsumerAdapter(AsyncMock(return_value=None)),
|
|
]
|
|
),
|
|
)
|
|
|
|
# Drive _process_message indirectly — we only care that the
|
|
# observer adapter was invoked through the hook chain.
|
|
ctx = AgentHookContext(
|
|
session_key="cli:c1",
|
|
turn_request=_msg("hello"),
|
|
)
|
|
await agent.hooks.before_user_inbound(ctx)
|
|
assert received == ["hello"]
|
|
|
|
async def test_decision_consumer_short_circuit_halts_observer(self, workspace):
|
|
"""With both adapters wired, the canonical order has observer
|
|
BEFORE short-circuiter, so engagement observation is preserved
|
|
even when /pick replies short-circuit."""
|
|
received = []
|
|
decision_handled = _Reply(channel="cli", chat_id="c1", content="OUT")
|
|
|
|
agent = _make_agent(
|
|
workspace,
|
|
hooks=CompositeHook(
|
|
[
|
|
OnUserInboundAdapter(lambda m: received.append(m.text)),
|
|
DecisionConsumerAdapter(AsyncMock(return_value=decision_handled)),
|
|
]
|
|
),
|
|
)
|
|
|
|
ctx = AgentHookContext(
|
|
session_key="cli:c1",
|
|
turn_request=_msg("/pick 1"),
|
|
)
|
|
decision = await agent.hooks.before_user_inbound(ctx)
|
|
|
|
# Observer ran before short-circuit halted the chain.
|
|
assert received == ["/pick 1"]
|
|
assert decision.short_circuit_result == ("OUT", [])
|
|
|
|
async def test_response_modifier_invoked_in_after_send(self, workspace):
|
|
agent = _make_agent(
|
|
workspace,
|
|
hooks=CompositeHook([ResponseModifierAdapter(lambda k, c: c + " [appended]")]),
|
|
)
|
|
ctx = AgentHookContext(session_key="cli:c1", outbound_content="hi")
|
|
decision = await agent.hooks.after_send(ctx)
|
|
assert decision.modified_content == "hi [appended]"
|
|
|
|
def test_callbacks_live_only_in_the_hook_chain(self, workspace):
|
|
"""The adapters are the callbacks' one home: the loop keeps no
|
|
parallel copy for a caller to read and find stale."""
|
|
agent = _make_agent(
|
|
workspace,
|
|
hooks=CompositeHook(
|
|
[
|
|
OnUserInboundAdapter(lambda m: None),
|
|
ResponseModifierAdapter(lambda k, c: c),
|
|
]
|
|
),
|
|
)
|
|
assert [type(h) for h in agent.hooks] == [OnUserInboundAdapter, ResponseModifierAdapter]
|
|
assert not hasattr(agent, "on_user_inbound")
|
|
assert not hasattr(agent, "response_modifier")
|
|
assert not hasattr(agent, "decision_consumer")
|