## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
4.4 KiB
Raven Test Suite
The test suite covers the CLI, provider routing, context assembly, memory, channels, sandbox behavior, TUI RPC contracts, and proactive engine flows.
Running Tests
Run the default suite from the repository root:
uv run pytest
For focused work, target the relevant file or marker:
uv run pytest tests/test_cli_doctor_commands.py
uv run pytest -m "not real_llm"
The suite runs on four workers by default (-n 4 in addopts), which takes it
from about ten minutes to two. Two things that costs you, both undone by -n0
on the command line:
uv run pytest -n0 --pdb tests/test_cli_doctor_commands.py
--pdbrefuses to start under the workers:--pdb is incompatible with distributing tests. There is no debugger to attach to, since the test runs in a child process.- Output is no longer in file order. Reading a run top to bottom, or
comparing two runs line by line, wants
-n0as well.
Tests should avoid live network calls by default. When a test needs external services or a real model, guard it behind an explicit marker or environment variable so CI and local contributors get deterministic results.
Coverage Governance
Raven uses three related checks:
- Coverage measures line and branch execution across the default Python
suite. It is the broad health signal for the production
raven/package. - Diff coverage measures executable lines added or modified relative to the PR target branch. Comments, deleted lines, and non-executable lines are not part of its denominator. The initial threshold is 90%.
- The ratchet compares total line and branch coverage with the audited
baseline in
.github/coverage-baseline.json. Neither metric may decline by more than 0.05 percentage points. This protects the established level without pretending the current project is already near 100%.
Run the same coverage command used by CI:
make coverage
PYTHON_VERSION defaults to the CI version (3.12) and is shared by all
Python Make targets. It can be overridden for exploratory compatibility runs,
for example make coverage PYTHON_VERSION=3.13; only the configured CI matrix
version may produce an authoritative baseline.
The command prints missing lines and creates coverage.xml, coverage.json,
and htmlcov/. Open htmlcov/index.html in a browser for line-by-line detail.
Generated reports are ignored by git.
After running coverage, use the local gates:
make coverage-summary
make coverage-diff COVERAGE_BASE_REF=origin/main
make coverage-ratchet
make coverage-baseline-check COVERAGE_BASE_REF=origin/main
The summary prints separate total line and branch percentages plus the ten
lowest-covered production files. A diff failure lists every uncovered changed
line as path:line. A ratchet failure prints the current value, baseline,
change in percentage points, and the files with the largest declines.
The CI unit job runs the default suite once and reuses its data for all gates. It uploads XML, JSON, and HTML reports even when tests or a gate fail. Diff coverage runs only for pull requests because a push has no PR target branch; push builds still enforce the overall ratchet.
Baseline updates
make coverage-baseline-candidate creates
coverage-baseline-candidate.json from the current report. The tracked
bootstrap baseline is accepted only after the introducing PR reproduces it in
Python 3.12 Ubuntu CI within the ratchet tolerance. After bootstrap, the
baseline must only be replaced with a candidate downloaded from a successful
main CI run. Review the reference commit and totals in the candidate before
replacement. Never edit the percentages by hand or lower the baseline to make
a change pass. PR CI compares a proposed baseline with the target branch and
rejects any decrease in either line or branch coverage.
Scope and exclusions
The default coverage suite excludes tests marked integration or e2e, plus
tests/integration/, because they require real services, credentials,
subprocess stacks, VMs, or network access and are not deterministic unit-CI
inputs. Provider cases that require live credentials remain skipped.
All production Python under raven/ is measured except:
raven/__main__.pyandevolver/__main__.py, which only forward their module entry points.
Type-checking-only blocks and if __name__ == "__main__" launcher blocks are
excluded as non-runtime paths. No low-coverage feature module is omitted.