## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
143 lines
6.2 KiB
Python
143 lines
6.2 KiB
Python
"""The routes, and the promises that hold across all of them.
|
|
|
|
Two of these tests exist because the predecessor broke exactly here. One route
|
|
became unreachable under every configuration -- a predicate contradicted itself
|
|
-- and the suite had frozen the unreachability as expected behaviour, so 1500
|
|
lines of tooling sat dead for months. The other is the hard invariant about
|
|
physical quantities, which lived in prose and therefore could not fail.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from raven_ppt.contracts import Profile
|
|
from raven_ppt.contracts.findings import Severity
|
|
from raven_ppt.profiles import registry
|
|
from raven_ppt.services.gates.registry import DISPATCH
|
|
|
|
|
|
def test_every_registered_route_is_reachable_by_name() -> None:
|
|
assert registry.names(), "no routes registered"
|
|
for name in registry.names():
|
|
assert isinstance(registry.get(name), Profile)
|
|
assert registry.DEFAULT in registry.names()
|
|
|
|
|
|
def test_an_unknown_route_names_the_ones_that_exist() -> None:
|
|
with pytest.raises(ValueError, match="Registered:"):
|
|
registry.get("no_such_route")
|
|
|
|
|
|
def test_no_route_lets_the_model_write_a_physical_quantity() -> None:
|
|
for name in registry.names():
|
|
caps = registry.get(name).capabilities
|
|
assert caps.physical_geometry is False, f"{name} opened physical geometry"
|
|
assert caps.font_size is False, f"{name} opened font size"
|
|
|
|
|
|
def test_no_route_refuses_a_deck_over_something_the_gate_only_reports() -> None:
|
|
"""A route's fatal list may not contradict the severity the check itself gives.
|
|
|
|
Publication refuses on `severity is BLOCKING or kind in blocking_kinds`, so the
|
|
two say the same thing in different places and nothing made them agree. The band
|
|
gate was downgraded to a warning and stayed fatal on two routes for exactly as
|
|
long as it took someone to try publishing: the finding came back marked
|
|
`warning`, the deck was refused, and the reply told the model to remove the bars.
|
|
|
|
A kind the checks do not declare at all is a route's own -- `unmapped_page` and
|
|
`overlap` build their findings inside the route that cares -- so this only holds
|
|
the kinds both places name.
|
|
"""
|
|
for name in registry.names():
|
|
contradicted = {
|
|
kind
|
|
for kind in registry.get(name).blocking_kinds
|
|
if kind in DISPATCH and DISPATCH[kind] is not Severity.BLOCKING
|
|
}
|
|
assert not contradicted, f"{name} refuses on {sorted(contradicted)}, which the gate only reports"
|
|
|
|
|
|
# The stages that settle what was agreed. `prepare` reads the request and records
|
|
# the part of the brief it already states, asking the user for the rest; `brief` is
|
|
# the same settlement without the reading. A route may open with either, and with
|
|
# nothing else.
|
|
_SETTLES_THE_BRIEF = {"brief", "prepare"}
|
|
|
|
|
|
def test_every_route_agrees_the_brief_first_and_publishes_last() -> None:
|
|
"""The brief is settled first on every route because everything after it is
|
|
checked against it: the page count, the language of the copy, and the room the
|
|
design is judged for. A route that ingested first could read the materials and
|
|
then be told the deck is for a different audience in a different language."""
|
|
for name in registry.names():
|
|
stages = [s.name for s in registry.get(name).stages]
|
|
assert stages[0] in _SETTLES_THE_BRIEF, f"{name} starts with {stages[0]}"
|
|
assert stages[-1] == "publish", f"{name} ends with {stages[-1]}"
|
|
# Ingest comes before anything is planned or drawn, whatever optional
|
|
# gathering sits between it and the brief.
|
|
assert "ingest" in stages, name
|
|
planning = [s for s in ("plan", "build", "background") if s in stages]
|
|
assert all(stages.index("ingest") < stages.index(step) for step in planning), name
|
|
|
|
|
|
def test_what_was_agreed_refuses_a_deck_on_every_route() -> None:
|
|
"""Asking a user to decide something and then ignoring it is worse than not
|
|
asking. The length and the language are checked against the finished file."""
|
|
for name in registry.names():
|
|
assert {"page_budget", "language"} <= registry.get(name).blocking_kinds, name
|
|
|
|
|
|
def test_provenance_refuses_publication_on_every_route() -> None:
|
|
"""A page crediting the wrong figure is fatal whichever way the deck was made."""
|
|
for name in registry.names():
|
|
blocking = registry.get(name).blocking_kinds
|
|
assert {"citation"} <= blocking, f"{name} would publish a miscredited figure"
|
|
|
|
|
|
def test_the_routes_differ_in_what_they_let_the_model_emit() -> None:
|
|
"""Otherwise there is one route wearing three names."""
|
|
emissions = {
|
|
name: (
|
|
registry.get(name).capabilities.raw_script,
|
|
registry.get(name).capabilities.normalized_regions,
|
|
registry.get(name).capabilities.background_prompt,
|
|
)
|
|
for name in registry.names()
|
|
}
|
|
assert len(set(emissions.values())) == len(emissions), emissions
|
|
|
|
|
|
def test_the_routes_do_not_share_a_backend() -> None:
|
|
backends = [registry.get(n).backend for n in registry.names()]
|
|
assert len(set(backends)) == len(backends), backends
|
|
|
|
|
|
def test_availability_names_the_missing_tools_instead_of_guessing() -> None:
|
|
profile = registry.get("script_author")
|
|
ok, missing = registry.available(profile.name, set(profile.tools))
|
|
assert ok and missing == ()
|
|
ok, missing = registry.available(profile.name, {"ppt_ingest"})
|
|
assert not ok
|
|
assert "ppt_build" in missing
|
|
|
|
|
|
def test_the_image_route_keeps_the_words_out_of_the_picture() -> None:
|
|
"""Type baked into a generated image is neither editable nor checkable.
|
|
|
|
It cannot be retranslated or corrected, and it is a raster no check can read
|
|
-- so a deck could state anything at all and pass every check.
|
|
Hence two stages rather than one, and a fail-closed kind for the case.
|
|
"""
|
|
profile = registry.get("image_text")
|
|
stages = [s.name for s in profile.stages]
|
|
assert stages.index("background") < stages.index("place_text")
|
|
assert "text_in_background" in profile.blocking_kinds
|
|
|
|
|
|
def test_the_script_route_refuses_a_deck_that_dropped_a_planned_figure() -> None:
|
|
"""Three fetched pictures, two planned onto pages, none in the deck."""
|
|
from raven_ppt.profiles import registry
|
|
|
|
assert "unplaced_figure" in registry.get("script_author").blocking_kinds
|