Raven-1/tests/test_acp_protocol.py
Handsome-wzw 7b45527b5b fix(tools): an array sent as its own JSON is still an array
## Summary

Reported as "deliver_files is broken". It is not: **the tool never ran.**

Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:

```
Error: Invalid parameters for tool 'deliver_files': files should be array
```

What the model sent was the array, spelled as a string:

```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```

`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.

## Type

- [x] Fix
- [ ] Feature

## The edge of the forgiving is the design

When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.

**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.

**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.

## Not just deliver_files

Six built-in tools declare an array or object parameter and were all exposed:

| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |

The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.

## Verification

```
make lint-python    ruff check + ruff format --check, 1850 files
make lint-imports   9 contracts kept, 0 broken
pytest tests/       19,699 passed, 7 failed (all pre-existing, see below)
```

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string

**End to end, through the real tool**, with the exact payload that failed five times in production:

```
files sent as:            str -> [{"path": ".../poster.png", "title": "..."}]
returned:                 Delivered 1 file(s): poster.png (72 B).
still "should be array":  False
registered in the store:  True
```

Twelve cases, five mutations, each killing at least one:

| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |

The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.

**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.

## Risk

- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert

## Related Issues

N/A

## Swept

`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.

- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.

Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).

## Deliberately not fixed

The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.

## Moved from !594

Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.

Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00

255 lines
12 KiB
Python

"""The agent direction's wire helpers, and what the handshake declares.
Every frame this file builds is also run through the vendored official schema,
so the assertions cover both "raven does what we intended" and "what we intended
is what the spec says". The second half is the one that catches a hand-written
mapper drifting, and it is why these tests import ``tests/acp_schema.py``
instead of comparing against dicts typed out here.
"""
from __future__ import annotations
import pytest
from raven.acp import protocol
from raven.acp.capabilities import ClientCapabilities, agent_capabilities, initialize_result
from tests.acp_schema import agent_method_names, is_valid_def, validate_def, validate_outbound
class TestIds:
def test_a_string_id_survives_the_round_trip(self):
"""The client mints request ids and JSON-RPC allows a string. Answering a
string id with a number is a correlation failure even though a reply
arrived."""
frame = protocol.request("req-1", "session/prompt", {"sessionId": "s"})
assert frame["id"] == "req-1"
validate_def("ClientRequest", frame)
def test_a_request_carries_params_only_when_there_are_some(self):
"""Absent rather than null, for the same reason ``data`` is: a null reads
as "there are parameters and they are empty"."""
assert "params" not in protocol.request(1, "session/cancel")
assert protocol.request(1, "session/cancel", {})["params"] == {}
def test_an_error_carries_data_only_when_there_is_some(self):
without = protocol.error_response(1, protocol.INVALID_PARAMS, "bad")
with_data = protocol.error_response(1, protocol.INVALID_PARAMS, "bad", {"field": "cwd"})
assert "data" not in without["error"], "a null data reads as empty detail rather than as no detail"
assert with_data["error"]["data"] == {"field": "cwd"}
validate_outbound(without)
validate_outbound(with_data)
def test_the_acp_specific_codes_are_the_ones_the_spec_assigns(self):
# Typed out rather than derived, because these are the load-bearing
# numbers: -32002 is what tells a client its session is gone rather than
# empty, and -32800 is what a cancelled request must answer.
assert (protocol.AUTH_REQUIRED, protocol.RESOURCE_NOT_FOUND, protocol.REQUEST_CANCELLED) == (
-32000,
-32002,
-32800,
)
class TestVersionNegotiation:
@pytest.mark.parametrize(
("raw", "expected"),
[
(1, 1),
(2, 2),
(0, 0),
(1.0, 1),
("1", 1),
(" 7 ", 7),
# A non-ASCII decimal numeral, which ``int()`` parses. Twelve rather
# than one so the assertion can tell reading it apart from falling
# back: the fallback is 1, so a single digit here would pass either
# way and prove nothing.
("\u0661\u0662", 12),
],
)
def test_a_readable_version_is_read(self, raw, expected):
assert protocol.normalize_protocol_version(raw) == expected
@pytest.mark.parametrize("raw", [None, "one", "1.5", 1.5, [], {}, True, False, "\u00b2", "\u00bd"])
def test_an_unreadable_version_falls_back_rather_than_failing(self, raw):
"""Failing the handshake over a malformed version denies the client the
one thing it needs to decide what to do: which version we serve.
``True`` is listed because ``bool`` is an ``int`` subclass -- normalising
it to 1 would be a coincidence, not a reading of intent. Superscript two
is listed because it is the case the ``isdecimal`` check exists for: it
satisfies ``isdigit`` and makes ``int()`` raise, so the looser check would
have guarded nothing.
"""
assert protocol.normalize_protocol_version(raw) == protocol.PROTOCOL_VERSION
def test_an_unsupported_version_is_answered_with_ours(self):
assert protocol.negotiated_version(99) == protocol.PROTOCOL_VERSION
assert protocol.negotiated_version(1) == 1
def test_stop_reasons_are_exactly_the_schema_enum(self):
for reason in protocol.STOP_REASONS:
validate_def("StopReason", reason)
assert not is_valid_def("StopReason", "done")
assert len(protocol.STOP_REASONS) == 5
class TestDeclaredCapabilities:
def test_the_initialize_result_matches_the_schema(self):
validate_def("InitializeResponse", initialize_result({"protocolVersion": 1}))
def test_it_survives_a_client_that_sends_nothing_at_all(self):
"""A handshake is the one exchange with no surface for reporting its own
failure, so it must not have one."""
validate_def("InitializeResponse", initialize_result(None))
validate_def("InitializeResponse", initialize_result({}))
def test_no_authentication_is_declared_as_a_fact_not_an_omission(self):
assert initialize_result({})["authMethods"] == []
def test_nothing_unbuilt_is_declared(self):
"""The worst failure shape in the protocol is a declared capability with
nothing behind it: the client routes work to a method that errors, and
the turn stalls on a promise nobody keeps."""
caps = agent_capabilities()
assert caps["promptCapabilities"]["audio"] is False, "there is no audio path on the prompt side"
assert caps["mcpCapabilities"] == {"http": False, "sse": False}, (
"a session may bring stdio servers, but no http/sse transport is offered"
)
assert caps["auth"] == {}, "declaring auth.logout would put a method on the wire with nothing to end"
# ``list``, ``resume``, ``close`` and ``delete`` are declared and each
# has a method behind it; additionalDirectories is stable in the schema
# and unbuilt here, and each one declared is a method that must then work.
assert set(caps["sessionCapabilities"]) == {"list", "resume", "close", "delete"}
def test_what_is_declared_is_declared_because_it_works(self):
caps = agent_capabilities()
assert caps["promptCapabilities"]["image"] is True
assert caps["promptCapabilities"]["embeddedContext"] is True
def test_load_is_declared_only_because_the_replay_exists(self):
"""The flag is a promise. A client that reopens a session on a false one
shows a person an empty history for a conversation that had one."""
from raven.acp.methods import UNIMPLEMENTED_METHODS
from raven.acp.replay import replay
assert agent_capabilities()["loadSession"] is True
assert "session/load" not in UNIMPLEMENTED_METHODS
assert replay([{"role": "user", "text": "hi"}], session_id="acp:s"), "the replay must produce something"
def test_list_is_declared_only_because_the_method_answers(self):
from raven.acp.methods import UNIMPLEMENTED_METHODS
assert "list" in agent_capabilities()["sessionCapabilities"]
assert "session/list" not in UNIMPLEMENTED_METHODS
def test_resume_is_declared_only_because_the_method_answers(self):
"""Declared because this build answers ``session/resume``, so the
declaration and the method cannot drift apart. Nothing downstream props
it up: a raven acting as this agent's client reads ``loadSession`` to
report the row resumable, not this capability."""
from raven.acp.methods import UNIMPLEMENTED_METHODS
assert "resume" in agent_capabilities()["sessionCapabilities"]
assert "session/resume" not in UNIMPLEMENTED_METHODS
def test_close_and_delete_are_declared_only_because_the_methods_answer(self):
from raven.acp.methods import UNIMPLEMENTED_METHODS
for name, method in (("close", "session/close"), ("delete", "session/delete")):
assert name in agent_capabilities()["sessionCapabilities"], name
assert method not in UNIMPLEMENTED_METHODS, method
def test_per_session_mcp_is_declared_only_because_both_halves_are_built(self):
"""The promise is two claims, and neither has a spec field to make it in.
stdio servers are the ACP baseline, so accepting them cannot be
advertised, and ``mcpCapabilities`` says nothing about isolation. So the
claim goes in ``_meta`` -- and it is only honest while ``session/new`` /
``load`` / ``resume`` really connect what they are handed and the tools
stay scoped to that session's turns.
"""
from raven.acp.methods import AcpMethods
from raven.agent.tools.registry import ToolRegistry
assert protocol.SESSION_MCP_CAPABILITY in agent_capabilities()["_meta"]
# connected, not refused
assert hasattr(AcpMethods, "_adopt_per_session_mcp")
# and released again, so nothing outlives the session that brought it
assert hasattr(AcpMethods, "_release_per_session_mcp")
# visible to that session's turns alone
assert hasattr(ToolRegistry, "session_scope_for")
def test_the_meta_carrier_does_not_break_the_declared_shape(self):
"""``_meta`` is the schema's extension slot, so the whole response must
still validate with raven's claim inside it."""
validate_def("InitializeResponse", initialize_result({"protocolVersion": 1}))
assert agent_capabilities()["_meta"][protocol.SESSION_MCP_CAPABILITY] == {}, (
"the schema spells supported as an empty object, and a reader tests presence not truthiness"
)
def test_the_agent_names_itself_with_a_real_version(self):
info = initialize_result({})["agentInfo"]
assert info["name"] == "raven"
assert info["version"] and info["version"] != "unknown"
validate_def("Implementation", info)
class TestRecordedClientCapabilities:
def test_a_client_that_declares_nothing_can_do_nothing(self):
for params in (None, {}, {"clientCapabilities": None}, {"clientCapabilities": "yes"}):
caps = ClientCapabilities.from_params(params)
assert caps.elicitation is False
assert caps.reads_files is False
assert caps.writes_files is False
assert caps.has_terminal is False
def test_elicitation_is_read_by_presence_because_it_is_an_object(self):
"""The capability is an object in the schema, not a boolean. Reading it
as truthy would make ``{"elicitation": {}}`` -- a client declaring
support with no options -- read as no support."""
assert ClientCapabilities.from_params({"clientCapabilities": {"elicitation": {}}}).elicitation is True
def test_the_fs_flags_are_read_from_inside_their_section(self):
caps = ClientCapabilities.from_params(
{"clientCapabilities": {"fs": {"readTextFile": True, "writeTextFile": False}}}
)
assert caps.reads_files is True
assert caps.writes_files is False
def test_a_malformed_section_does_not_raise(self):
caps = ClientCapabilities.from_params({"clientCapabilities": {"fs": "both"}})
assert caps.reads_files is False
class TestManifestAgreement:
def test_every_method_raven_answers_is_in_the_stable_manifest(self):
"""Guards against serving an unstable method by name. The manifest omits
all 18 of them, so membership is the check."""
from raven.acp.methods import UNIMPLEMENTED_METHODS
served = {"initialize", "authenticate", "session/new", "session/prompt", "session/cancel"}
for method in served | UNIMPLEMENTED_METHODS:
assert method in agent_method_names(), f"{method} is not in the stable manifest"
class TestTheTwoSidesAgree:
def test_the_retraction_raven_sends_is_the_one_raven_listens_for(self):
"""The two ACP packages are deliberately separate -- agent-side and
client-side -- and each spells this method itself. A raven sub-agent
retracting a question only reaches a raven host if the two strings are
the same one, and nothing else in either package would notice if they
drifted apart."""
from raven.acp.protocol import CANCEL_REQUEST_METHOD as SENT
from raven.acp_client.protocol import CANCEL_REQUEST_METHOD as HEARD
assert SENT == HEARD == "$/cancel_request"