## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
294 lines
12 KiB
Python
294 lines
12 KiB
Python
"""A tool name an upstream sent is a key only after it is normalised.
|
|
|
|
An upstream returned tool names carrying a leading space. Raven used them as
|
|
registry keys unchanged, so the lookup missed and the call was refused -- and
|
|
because the name is also written back into the history, the bad one was
|
|
replayed to the model on every turn afterwards, which is why the model could
|
|
not correct it by reasoning about it.
|
|
|
|
Every exit that builds a `ToolCallRequest` out of upstream data is pinned, and
|
|
a guard test enumerates them from the source rather than trusting this list --
|
|
the first version of this change covered two of the four and said it covered
|
|
all of them. Replay is deliberately not one of them: it reconstructs a recorded
|
|
run, and normalising there would make a replay diverge from the run it replays.
|
|
|
|
The boundary is pinned too. Stripping surrounding whitespace is a fact (a
|
|
registry key is written in code and has none); folding case or interior spaces
|
|
would be a guess, and "that tool does not exist" is only worth something while
|
|
it is certain.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import ast
|
|
import json
|
|
from dataclasses import dataclass, field
|
|
from pathlib import Path
|
|
from types import SimpleNamespace
|
|
from typing import Any
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
from raven.providers.azure_openai_provider import AzureOpenAIProvider
|
|
from raven.providers.litellm_provider import LiteLLMProvider
|
|
from raven.providers.streaming import _finalize_tool_calls
|
|
from raven.providers.tool_names import normalized_tool_name, sanitary_form
|
|
|
|
|
|
@dataclass
|
|
class _Msg:
|
|
content: str | None = None
|
|
tool_calls: list[Any] | None = None
|
|
reasoning_content: str | None = None
|
|
|
|
|
|
@dataclass
|
|
class _Choice:
|
|
message: _Msg
|
|
finish_reason: str = "tool_calls"
|
|
|
|
|
|
@dataclass
|
|
class _Usage:
|
|
prompt_tokens: int = 2900
|
|
completion_tokens: int = 12
|
|
total_tokens: int = 2912
|
|
|
|
|
|
@dataclass
|
|
class _Response:
|
|
choices: list[_Choice]
|
|
usage: _Usage = field(default_factory=_Usage)
|
|
|
|
|
|
def _raw_call(name: str) -> SimpleNamespace:
|
|
"""The shape `_parse_response` reads a tool call out of."""
|
|
return SimpleNamespace(
|
|
function=SimpleNamespace(name=name, arguments='{"path": "a.txt"}', provider_specific_fields=None),
|
|
provider_specific_fields=None,
|
|
)
|
|
|
|
|
|
def _slot(name: Any) -> dict[str, Any]:
|
|
"""The shape the streaming accumulator hands to `_finalize_tool_calls`."""
|
|
return {"id": "call_1", "function": {"name": name, "arguments_buf": ['{"path": "a.txt"}']}}
|
|
|
|
|
|
class _SseStream:
|
|
"""SSE response stand-in. Ends rather than stalling: `_consume_sse` returns
|
|
when the stream does, so a trailing stall would trip the idle cap first."""
|
|
|
|
def __init__(self, lines: list[str]) -> None:
|
|
self._lines = lines
|
|
|
|
async def aiter_lines(self):
|
|
for line in self._lines:
|
|
yield line
|
|
|
|
|
|
def _provider() -> LiteLLMProvider:
|
|
with (
|
|
patch("raven.providers.litellm_provider.litellm"),
|
|
patch("raven.providers.litellm_provider.LiteLLMProvider._setup_env"),
|
|
):
|
|
return LiteLLMProvider(api_key="sk-test", provider_name="openrouter")
|
|
|
|
|
|
def _parsed(name: str) -> Any:
|
|
response = _Response(choices=[_Choice(message=_Msg(tool_calls=[_raw_call(name)]))])
|
|
return _provider()._parse_response(response, sent_chars=12000)
|
|
|
|
|
|
# ---------- both parse exits -------------------------------------------------
|
|
|
|
|
|
def test_the_non_streaming_exit_hands_on_a_usable_key() -> None:
|
|
"""The observed shape: a leading space, on the path the incident took."""
|
|
assert [tc.name for tc in _parsed(" read_file").tool_calls] == ["read_file"]
|
|
|
|
|
|
def test_the_streaming_exit_hands_on_a_usable_key() -> None:
|
|
"""The same question on the other path. A verdict reached on one and not
|
|
the other makes the same upstream read differently inside the TUI than
|
|
outside it."""
|
|
assert [tc.name for tc in _finalize_tool_calls([_slot(" read_file")])] == ["read_file"]
|
|
|
|
|
|
def test_the_name_written_back_into_the_history_is_the_clean_one() -> None:
|
|
"""Why this is fixed at the exit and not at the registry lookup.
|
|
|
|
A lookup that stripped would resolve the call and still leave the bad name
|
|
on the request -- which `openai_tool_call` writes into the assistant
|
|
message, so every later turn replays it to the model. The model then reads
|
|
its own bad name as precedent, which is the loop the incident showed.
|
|
"""
|
|
call = _parsed(" read_file\n").tool_calls[0]
|
|
|
|
from raven.providers.tool_calls import openai_tool_call
|
|
|
|
assert openai_tool_call(call)["function"]["name"] == "read_file"
|
|
|
|
|
|
def test_the_azure_exit_hands_on_a_usable_key() -> None:
|
|
"""The third exit. `AgentLoop` reaches `chat_with_retry` -- and so this
|
|
parser -- whenever the turn caller wired no delta callback, which is every
|
|
headless entry point."""
|
|
provider = AzureOpenAIProvider(api_key="sk-test", api_base="https://x.openai.azure.com")
|
|
response = {
|
|
"choices": [
|
|
{
|
|
"message": {
|
|
"content": None,
|
|
"tool_calls": [
|
|
{"id": "call_1", "function": {"name": " read_file", "arguments": '{"path": "a.txt"}'}}
|
|
],
|
|
},
|
|
"finish_reason": "tool_calls",
|
|
}
|
|
]
|
|
}
|
|
|
|
assert [tc.name for tc in provider._parse_response(response).tool_calls] == ["read_file"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_codex_exit_hands_on_a_usable_key() -> None:
|
|
"""The fourth exit. Its name arrives on one SSE event and the call is built
|
|
on a later one, so the raw string outlives the event that carried it."""
|
|
from raven.providers.openai_codex_provider import _consume_sse
|
|
|
|
added = json.dumps(
|
|
{"type": "response.output_item.added", "item": {"type": "function_call", "call_id": "c1", "name": " read_file"}}
|
|
)
|
|
done = json.dumps(
|
|
{
|
|
"type": "response.output_item.done",
|
|
"item": {"type": "function_call", "call_id": "c1", "arguments": '{"path": "a.txt"}'},
|
|
}
|
|
)
|
|
completed = json.dumps({"type": "response.completed"})
|
|
|
|
_, tool_calls, _ = await _consume_sse(
|
|
_SseStream([f"data: {added}", "", f"data: {done}", "", f"data: {completed}", ""]), timeout=1.0
|
|
)
|
|
|
|
assert [tc.name for tc in tool_calls] == ["read_file"]
|
|
|
|
|
|
def test_every_exit_that_builds_a_call_from_upstream_data_normalises_it() -> None:
|
|
"""The guard the first version of this change needed and did not have.
|
|
|
|
Two of the four exits were covered while the docstring and this module both
|
|
claimed all of them were, and nothing went red. Reading the tree rather than
|
|
a hand-kept list is what makes a fifth exit arrive as a failure here instead
|
|
of as a refused tool call in production.
|
|
"""
|
|
exempt = {
|
|
# Replays a recorded run. Repairing a name here would make the replay
|
|
# disagree with the run it is reproducing, which is the one thing it
|
|
# exists not to do.
|
|
"raven/trajectory/replay.py",
|
|
}
|
|
|
|
def _calls(node: ast.AST, name: str) -> list[ast.Call]:
|
|
"""Every call to ``name`` under ``node``, spelled either way.
|
|
|
|
Both spellings, because the bare name is the only one used today and the
|
|
case this guard exists for is the exit written differently later: an
|
|
`ast.Attribute` covers `base.ToolCallRequest(...)` after
|
|
`from raven.providers import base`, and an alias resolves to one or the
|
|
other.
|
|
"""
|
|
found = []
|
|
for n in ast.walk(node):
|
|
if not isinstance(n, ast.Call):
|
|
continue
|
|
called = getattr(n.func, "id", None) or getattr(n.func, "attr", None)
|
|
if called == name:
|
|
found.append(n)
|
|
return found
|
|
|
|
# Anchored to this file, not to the working directory. `Path("raven")` reads
|
|
# as the repo only while pytest happens to run from the root: from anywhere
|
|
# else it enumerates nothing, `offenders` stays empty, and the assert passes
|
|
# having checked nothing -- the one failure mode this guard must not have,
|
|
# since it is what stands between a fifth exit and production.
|
|
root = Path(__file__).resolve().parents[1]
|
|
inspected: list[str] = []
|
|
offenders: list[str] = []
|
|
for path in sorted((root / "raven").rglob("*.py")):
|
|
rel = path.relative_to(root).as_posix()
|
|
if rel in exempt:
|
|
inspected.append(rel)
|
|
continue
|
|
tree = ast.parse(path.read_text(encoding="utf-8"))
|
|
for func in ast.walk(tree):
|
|
if not isinstance(func, ast.FunctionDef | ast.AsyncFunctionDef):
|
|
continue
|
|
builds = _calls(func, "ToolCallRequest")
|
|
if not builds:
|
|
continue
|
|
inspected.append(f"{rel}:{func.name}")
|
|
# Asked of the whole function rather than of the `name=` expression:
|
|
# the streaming exit normalises into a local and passes that, which
|
|
# an inline-only check reads as a miss.
|
|
if not _calls(func, "normalized_tool_name"):
|
|
offenders.extend(f"{rel}:{n.lineno}" for n in builds)
|
|
|
|
# What separates "checked and found nothing" from "checked nothing". The
|
|
# exits are known and counted, so a walk that suddenly sees fewer of them is
|
|
# reporting on a tree it did not read.
|
|
assert len(inspected) >= 5, f"the walk found {len(inspected)} construction sites, expected at least 5: {inspected}"
|
|
|
|
assert not offenders, (
|
|
f"these build a ToolCallRequest from an upstream name without normalising it: {offenders}. "
|
|
"Wrap the name in normalized_tool_name, or add the file to `exempt` here with the reason."
|
|
)
|
|
|
|
|
|
# ---------- the boundary, which is the point ---------------------------------
|
|
|
|
|
|
def test_an_interior_space_is_still_a_name_that_does_not_exist() -> None:
|
|
"""Not repaired, because repairing it would be a guess. Nothing declares
|
|
that `read file` was meant to be `read_file`."""
|
|
assert normalized_tool_name(" read file ") == "read file"
|
|
|
|
|
|
def test_case_is_not_folded() -> None:
|
|
"""`read_file` and `Read_File` can both be registered at once -- an MCP
|
|
server is free to export either -- so folding them would answer a lookup
|
|
with a name the caller never asked for."""
|
|
assert normalized_tool_name(" Read_File") == "Read_File"
|
|
|
|
|
|
def test_a_name_that_is_only_whitespace_is_no_name_at_all() -> None:
|
|
"""It comes out empty, and the streaming exit drops an empty name. Passing
|
|
it on would put a call with no name into the history."""
|
|
assert _finalize_tool_calls([_slot(" ")]) == []
|
|
|
|
|
|
def test_the_outbound_sanitiser_is_not_usable_on_a_received_name() -> None:
|
|
"""Why the two directions share a spec and not a function.
|
|
|
|
`sanitary_form` is what a name goes through on the way *out*, where
|
|
substituting an illegal character is safe because the result is the name
|
|
that then gets registered. Used on a name coming *in* it invents one: a
|
|
leading space becomes an underscore, and an interior space silently becomes
|
|
the very name this repo refuses to guess at.
|
|
"""
|
|
assert sanitary_form(" read_file") == "_read_file"
|
|
assert sanitary_form("read file") == "read_file"
|
|
|
|
assert normalized_tool_name(" read_file") == "read_file"
|
|
assert normalized_tool_name("read file") == "read file"
|
|
|
|
|
|
def test_a_name_that_is_not_a_string_is_handed_back_untouched() -> None:
|
|
"""A different fault with a different owner. Raising here would take down
|
|
the parse of a response that may be otherwise fine, and the name still
|
|
misses the registry exactly as it did before."""
|
|
assert normalized_tool_name(None) is None
|
|
assert normalized_tool_name(7) == 7
|