Raven-1/tests/test_cli_agent_loop_parity.py
Handsome-wzw 7b45527b5b fix(tools): an array sent as its own JSON is still an array
## Summary

Reported as "deliver_files is broken". It is not: **the tool never ran.**

Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:

```
Error: Invalid parameters for tool 'deliver_files': files should be array
```

What the model sent was the array, spelled as a string:

```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```

`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.

## Type

- [x] Fix
- [ ] Feature

## The edge of the forgiving is the design

When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.

**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.

**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.

## Not just deliver_files

Six built-in tools declare an array or object parameter and were all exposed:

| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |

The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.

## Verification

```
make lint-python    ruff check + ruff format --check, 1850 files
make lint-imports   9 contracts kept, 0 broken
pytest tests/       19,699 passed, 7 failed (all pre-existing, see below)
```

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string

**End to end, through the real tool**, with the exact payload that failed five times in production:

```
files sent as:            str -> [{"path": ".../poster.png", "title": "..."}]
returned:                 Delivered 1 file(s): poster.png (72 B).
still "should be array":  False
registered in the store:  True
```

Twelve cases, five mutations, each killing at least one:

| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |

The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.

**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.

## Risk

- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert

## Related Issues

N/A

## Swept

`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.

- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.

Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).

## Deliberately not fixed

The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.

## Moved from !594

Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.

Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00

225 lines
9.6 KiB
Python

"""Cross-entrypoint parity for the assembly-door call.
Three entrypoints used to hand-write their own ``AgentLoop(...)`` call, and a
kwarg added to one and forgotten in the others failed *silently* -- the omitted
feature simply did not exist in that surface (``third_party_subagents`` missing
from two surfaces and ``plugin_tools`` missing from the gateway both shipped
that way). ``build_runtime`` dissolved the three copies: the cargo mapping now
exists once, in ``raven/core/runtime.py``, and parity of everything derived
from config holds by construction.
What can still drift is what the entrances still own -- the transport-side
``TurnPolicy`` / ``HostWiring`` fields and the identity handles they pass to
the door. This file keeps that remainder honest, and keeps the door singular:
an ``AgentLoop(...)`` call reappearing in an entrance is the regression this
guard exists to catch now.
So every difference has to be *declared* here, with the reason. An undeclared
one turns this red. Shrinking :data:`LEDGER` (i.e. fixing a gap) is also a
deliberate edit, because a stale entry turns it red too.
The check is static (``ast`` over the call site) rather than a runtime
kwarg-capture: ``gateway()`` cannot be driven under unit test at all -- it
builds ChannelManager / Cron / Heartbeat stacks whose shutdown paths assume a
running event loop (see the note in ``test_cli_gateway_commands.py``). Reading
the source is the only way to cover all three the same way.
"""
from __future__ import annotations
import ast
from pathlib import Path
from typing import NamedTuple
import pytest
from raven.agent.loop import AgentLoop
_REPO_ROOT = Path(__file__).resolve().parent.parent
ENTRYPOINTS: dict[str, str] = {
"agent": "raven/cli/agent_commands.py",
"gateway": "raven/cli/gateway_commands.py",
"tui": "raven/core/engine_stack.py",
}
class Difference(NamedTuple):
"""One declared asymmetry: who omits the kwarg, and why that is acceptable."""
absent_from: frozenset[str]
reason: str
# Declared asymmetries. Every entry is a deliberate design decision, not a known
# omission: an entrypoint that simply forgot a kwarg is a bug to fix, and the
# tests below make either shape visible -- an undeclared difference fails, and so
# does an entry that no longer matches the code.
LEDGER: dict[str, Difference] = {
"cron_service": Difference(
absent_from=frozenset({"agent"}),
reason=(
"Upstream removed the REPL from `agent`, so that process is never a "
"cron runner and wiring a CronService would create jobs nothing "
"fires (the agent entrance passes no cron_service to "
"build_runtime). Scripted reminder creation is "
"`raven cron add` with an explicit --channel."
),
),
"now_fn": Difference(
absent_from=frozenset({"tui"}),
reason=(
"Fed from a --fake-now CLI option that only agent and gateway "
"expose; the TUI has no such flag, so there is nothing to pass."
),
),
}
def _call_kwargs(relative_path: str, callee: str) -> set[str]:
"""Flattened keyword names passed to the ``callee(...)`` call in one file.
The wiring bundles are transparent: what a site really passes is the
flattened field set, so descend into each bundle constructor and collect
its keyword names.
"""
source = (_REPO_ROOT / relative_path).read_text(encoding="utf-8")
for node in ast.walk(ast.parse(source)):
if not isinstance(node, ast.Call):
continue
func = node.func
name = func.id if isinstance(func, ast.Name) else getattr(func, "attr", None)
if name == callee:
# ``**kwargs`` forwarding would make the set unreadable statically;
# no site does that today, and this asserts it stays that way.
assert all(kw.arg for kw in node.keywords), f"{relative_path}: {callee} called with **kwargs"
flat: set[str] = set()
for kw in node.keywords:
inner = kw.value
if isinstance(inner, ast.Call) and getattr(inner.func, "id", "").endswith(("Wiring", "Policy")):
assert all(k.arg for k in inner.keywords), f"{relative_path}: bundle with **kwargs"
flat |= {k.arg for k in inner.keywords if k.arg}
else:
flat.add(kw.arg)
return flat
raise AssertionError(f"no {callee}(...) call found in {relative_path}")
def _agent_loop_kwargs(relative_path: str) -> set[str]:
"""What one entrance passes through the assembly door."""
source = (_REPO_ROOT / relative_path).read_text(encoding="utf-8")
for node in ast.walk(ast.parse(source)):
if isinstance(node, ast.Call):
name = node.func.id if isinstance(node.func, ast.Name) else getattr(node.func, "attr", None)
assert name != "AgentLoop", (
f"{relative_path} constructs AgentLoop directly again; entrances "
"must assemble through raven.core.runtime.build_runtime."
)
return _call_kwargs(relative_path, "build_runtime")
@pytest.fixture(scope="module")
def kwargs_by_entrypoint() -> dict[str, set[str]]:
return {name: _agent_loop_kwargs(path) for name, path in ENTRYPOINTS.items()}
def _all_kwargs(kwargs_by_entrypoint: dict[str, set[str]]) -> set[str]:
return set().union(*kwargs_by_entrypoint.values())
@pytest.mark.parametrize("entrypoint", sorted(ENTRYPOINTS))
def test_every_entrypoint_has_a_readable_call_site(entrypoint: str) -> None:
"""The parity check is only as good as its ability to find the call."""
assert len(_agent_loop_kwargs(ENTRYPOINTS[entrypoint])) >= 10
def test_the_door_itself_stays_rich() -> None:
"""The single ``AgentLoop(...)`` site in the door still wires the full
cargo surface; the old per-entrance floor moves here."""
assert len(_call_kwargs("raven/core/runtime.py", "AgentLoop")) >= 25
def test_no_undeclared_asymmetry(kwargs_by_entrypoint: dict[str, set[str]]) -> None:
"""Every kwarg is either passed by all three, or declared in LEDGER.
This is the check that catches "added a kwarg to one entrypoint and forgot
the other two".
"""
undeclared: dict[str, set[str]] = {}
for kwarg in sorted(_all_kwargs(kwargs_by_entrypoint)):
absent = {name for name, passed in kwargs_by_entrypoint.items() if kwarg not in passed}
if absent and kwarg not in LEDGER:
undeclared[kwarg] = absent
assert not undeclared, (
"AgentLoop kwargs differ between entrypoints without a LEDGER entry: "
+ "; ".join(f"{k} missing from {sorted(v)}" for k, v in undeclared.items())
+ ". Either pass it everywhere, or declare the asymmetry with its reason."
)
@pytest.mark.parametrize("kwarg", sorted(LEDGER))
def test_declared_asymmetry_still_matches_reality(kwarg: str, kwargs_by_entrypoint: dict[str, set[str]]) -> None:
"""A LEDGER entry must describe the code exactly.
Fails both ways on purpose: if a gap is fixed the entry has to go (so the
file cannot rot into a list of lies), and if the asymmetry spreads to another
entrypoint that is a new omission, not a covered one.
"""
absent = frozenset(name for name, passed in kwargs_by_entrypoint.items() if kwarg not in passed)
assert absent, (
f"{kwarg!r} is now passed by every entrypoint -- delete its LEDGER entry. "
f"Recorded reason was: {LEDGER[kwarg].reason}"
)
assert absent == LEDGER[kwarg].absent_from, (
f"{kwarg!r} is absent from {sorted(absent)} but LEDGER declares {sorted(LEDGER[kwarg].absent_from)}."
)
def test_ledger_entries_name_real_entrypoints() -> None:
for kwarg, difference in LEDGER.items():
unknown = difference.absent_from - set(ENTRYPOINTS)
assert not unknown, f"LEDGER[{kwarg!r}] names unknown entrypoints: {sorted(unknown)}"
assert difference.absent_from, f"LEDGER[{kwarg!r}] declares no absence"
assert difference.reason.strip(), f"LEDGER[{kwarg!r}] has no reason"
def test_the_shared_core_is_not_eroding(kwargs_by_entrypoint: dict[str, set[str]]) -> None:
"""A floor on how much the three entrypoints agree about.
Guards the other direction from :func:`test_no_undeclared_asymmetry`: that one
is satisfiable by declaring ever more asymmetry, this one notices when the
shared core is what shrank.
"""
shared = set.intersection(*kwargs_by_entrypoint.values())
# 9 = the identity five (provider, session_manager, router,
# workdir_resolver, channels_config/on_user_inbound via host) plus the
# three policy fields; provider_pool and deliverables fold into the door.
assert len(shared) >= 9, f"only {len(shared)} kwargs are passed by all three entrypoints: {sorted(shared)}"
class _StubProvider:
"""Construction-only stand-in; no turn is ever run against it here."""
def get_default_model(self) -> str:
return "fake/default"
async def chat_with_retry(self, **kwargs): # pragma: no cover - never invoked
raise NotImplementedError
def _build_loop(tmp_path: Path) -> AgentLoop:
"""A minimal ``AgentLoop``, built only to inspect its post-construction attributes."""
return AgentLoop(provider=_StubProvider(), workspace=tmp_path, model="fake/default")
def test_loop_holds_subagent_questions_and_memory_config(tmp_path: Path) -> None:
loop = _build_loop(tmp_path)
assert loop.subagent_questions_config.autofill_enabled is True
# memory_config was a pass-through parameter with no attribute behind it;
# the resolver needs user_id and memory_top_k from it.
assert loop.memory_config.user_id == "default"
assert loop.memory_config.memory_top_k == 5