## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
233 lines
9.7 KiB
Python
233 lines
9.7 KiB
Python
"""Where the system message stops being the same on the next turn.
|
|
|
|
A prompt cache keys on every block up to its breakpoint, and the system message
|
|
is rebuilt each turn from segments -- the first of which (identity, the bootstrap
|
|
files) read the same every time, while the rest (memory recall, the skill
|
|
router's hits, the Curator's working state) are derived from what the user just
|
|
said. One breakpoint at the end of that message therefore changes key on every
|
|
new turn and re-bills the stable head in front of it.
|
|
|
|
These pin the boundary the assembler measures for a second breakpoint: that it
|
|
falls between segments, that one volatile segment ends the run for every stable
|
|
segment behind it, and that declaring it changes no text the model reads.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from raven.context_engine.assembler import _SEG_SEP, _stable_prefix_chars
|
|
from raven.context_engine.segments import (
|
|
ActiveSkillsSegmentBuilder,
|
|
BootstrapSegmentBuilder,
|
|
IdentitySegmentBuilder,
|
|
MemorySegmentBuilder,
|
|
SkillsSegmentBuilder,
|
|
)
|
|
from raven.context_engine.segments.curator import CuratorSegmentBuilder
|
|
from raven.providers.prompt_cache import STABLE_PREFIX_KEY
|
|
|
|
|
|
def _parts(*spec: tuple[str, bool]) -> list[tuple[int, str, bool]]:
|
|
return [(i, text, stable) for i, (text, stable) in enumerate(spec, start=1)]
|
|
|
|
|
|
class TestTheRunOfStableSegments:
|
|
def test_the_boundary_falls_between_the_run_and_what_follows(self):
|
|
parts = _parts(("IDENT", True), ("BOOT", True), ("RECALL", False))
|
|
|
|
chars = _stable_prefix_chars(parts)
|
|
|
|
whole = _SEG_SEP.join(text for _, text, _ in parts)
|
|
assert whole[:chars] == "IDENT" + _SEG_SEP + "BOOT" + _SEG_SEP
|
|
assert whole[chars:] == "RECALL"
|
|
|
|
def test_one_volatile_segment_ends_the_run_for_everything_behind_it(self):
|
|
"""The reason `active_skills` gains nothing where it sits today.
|
|
|
|
It does not depend on the message, but `memory` is in front of it, so a
|
|
breakpoint any later than `memory` keys on text that changes anyway.
|
|
"""
|
|
parts = _parts(("IDENT", True), ("RECALL", False), ("ALWAYS-ON SKILLS", True))
|
|
|
|
chars = _stable_prefix_chars(parts)
|
|
|
|
whole = _SEG_SEP.join(text for _, text, _ in parts)
|
|
assert whole[:chars] == "IDENT" + _SEG_SEP
|
|
assert "ALWAYS-ON SKILLS" in whole[chars:]
|
|
|
|
@pytest.mark.parametrize(
|
|
("parts", "why"),
|
|
[
|
|
(_parts(("RECALL", False), ("MORE", False)), "no run at all"),
|
|
(_parts(("IDENT", True), ("BOOT", True)), "the run is the whole message and nothing follows"),
|
|
([], "nothing was contributed"),
|
|
],
|
|
)
|
|
def test_a_boundary_at_either_end_is_not_declared(self, parts, why):
|
|
"""Zero, because such a breakpoint would sit where the end-of-message
|
|
one already is and cache nothing it does not."""
|
|
assert _stable_prefix_chars(parts) == 0, why
|
|
|
|
|
|
class TestWhenPhaseBIsTheOnlyThingThatFollows:
|
|
"""An all-stable phase A is not the end of the message if phase B appends.
|
|
|
|
The Curator runs in phase B on every turn it has anything to say, and the
|
|
segments in front of it can all be stable at once: identity and bootstrap
|
|
present, memory and both skill segments contributing nothing. Read over
|
|
phase A alone that run looks terminal, and a boundary at the end of the
|
|
message is worth nothing -- so it used to be dropped, and the assembled
|
|
message went out as `IDENTITY<sep>CURATOR` carrying no key. That is the
|
|
exact miss this measurement exists to remove.
|
|
"""
|
|
|
|
def test_the_whole_stable_prefix_becomes_the_head_when_a_tail_follows(self):
|
|
parts = _parts(("IDENT", True), ("BOOT", True))
|
|
|
|
chars = _stable_prefix_chars(parts, tail_follows=True)
|
|
|
|
prefix = _SEG_SEP.join(text for _, text, _ in parts)
|
|
assert chars == len(prefix) + len(_SEG_SEP)
|
|
finished = prefix + _SEG_SEP + "CURATOR"
|
|
assert finished[:chars] == prefix + _SEG_SEP
|
|
assert finished[chars:] == "CURATOR"
|
|
|
|
def test_nothing_following_still_declares_nothing(self):
|
|
parts = _parts(("IDENT", True), ("BOOT", True))
|
|
|
|
assert _stable_prefix_chars(parts, tail_follows=False) == 0
|
|
|
|
def test_a_run_that_already_ends_inside_phase_a_is_unaffected(self):
|
|
"""`tail_follows` decides only the terminal case."""
|
|
parts = _parts(("IDENT", True), ("RECALL", False))
|
|
|
|
assert _stable_prefix_chars(parts, tail_follows=True) == _stable_prefix_chars(parts, tail_follows=False)
|
|
|
|
async def test_the_assembler_declares_it_for_an_all_stable_phase_a(self):
|
|
from raven.context_engine import ContextAssembler, TurnContext
|
|
from raven.contracts.assembled import TokenBudget
|
|
from raven.contracts.context import AssemblyContext, Segment
|
|
|
|
class _PhaseA:
|
|
def __init__(self, name, order, text):
|
|
self.name, self.order, self.text = name, order, text
|
|
self.needs_prefix, self.stable = False, True
|
|
|
|
async def build(self, ctx: AssemblyContext) -> Segment:
|
|
return Segment(text=self.text)
|
|
|
|
class _Curator:
|
|
name, order, needs_prefix, stable = "curator", 6, True, False
|
|
|
|
async def build(self, ctx: AssemblyContext) -> Segment:
|
|
return Segment(text="CURATOR")
|
|
|
|
eng = ContextAssembler(
|
|
[_PhaseA("identity", 1, "IDENT"), _PhaseA("boot", 2, "BOOT"), _Curator()],
|
|
lambda: [],
|
|
)
|
|
|
|
ac = await eng.assemble(
|
|
"s", [], TokenBudget(100_000, 4_000, 2_000, 1_000, 93_000), turn=TurnContext(current_message="hi")
|
|
)
|
|
|
|
system = ac.messages[0]
|
|
assert STABLE_PREFIX_KEY in system, "an all-stable phase A with a Curator tail declares a head"
|
|
chars = system[STABLE_PREFIX_KEY]
|
|
assert system["content"][:chars] == "IDENT" + _SEG_SEP + "BOOT" + _SEG_SEP
|
|
assert system["content"][chars:] == "CURATOR"
|
|
|
|
async def test_an_all_stable_message_with_no_tail_declares_nothing(self):
|
|
from raven.context_engine import ContextAssembler, TurnContext
|
|
from raven.contracts.assembled import TokenBudget
|
|
from raven.contracts.context import AssemblyContext, Segment
|
|
|
|
class _PhaseA:
|
|
def __init__(self, name, order, text):
|
|
self.name, self.order, self.text = name, order, text
|
|
self.needs_prefix, self.stable = False, True
|
|
|
|
async def build(self, ctx: AssemblyContext) -> Segment:
|
|
return Segment(text=self.text)
|
|
|
|
eng = ContextAssembler([_PhaseA("identity", 1, "IDENT"), _PhaseA("boot", 2, "BOOT")], lambda: [])
|
|
|
|
ac = await eng.assemble(
|
|
"s", [], TokenBudget(100_000, 4_000, 2_000, 1_000, 93_000), turn=TurnContext(current_message="hi")
|
|
)
|
|
|
|
assert STABLE_PREFIX_KEY not in ac.messages[0]
|
|
|
|
|
|
class TestTheShippedSegmentsDeclareThemselves:
|
|
"""Whether each builder's `stable` flag matches what it reads.
|
|
|
|
Asserted here rather than left to the assembler's arithmetic, because the
|
|
flag is a claim about a segment's inputs and the cost of getting one wrong
|
|
is silent: a segment wrongly called stable pays for a write whose key never
|
|
matches again.
|
|
"""
|
|
|
|
@pytest.mark.parametrize(
|
|
("builder", "stable", "why"),
|
|
[
|
|
(IdentitySegmentBuilder, True, "SOUL.md and the identity header"),
|
|
(BootstrapSegmentBuilder, True, "the bootstrap files as they are on disk"),
|
|
(MemorySegmentBuilder, False, "host memory is picked per message; recall queries it"),
|
|
(ActiveSkillsSegmentBuilder, True, "the always-on set, chosen without the message"),
|
|
(SkillsSegmentBuilder, False, "the router's hits for this message"),
|
|
(CuratorSegmentBuilder, False, "working state recomputed for this turn"),
|
|
],
|
|
)
|
|
def test_each_builder_says_whether_it_reads_the_message(self, builder, stable, why):
|
|
assert builder.stable is stable, why
|
|
|
|
|
|
class TestTheAssemblerDeclaresIt:
|
|
async def test_the_offset_points_at_the_message_it_was_measured_over(self):
|
|
from raven.context_engine import ContextAssembler, TurnContext
|
|
from raven.contracts.assembled import TokenBudget
|
|
from raven.contracts.context import AssemblyContext, Segment
|
|
|
|
class _Seg:
|
|
def __init__(self, name, order, text, stable):
|
|
self.name, self.order, self.text, self.stable = name, order, text, stable
|
|
self.needs_prefix = False
|
|
|
|
async def build(self, ctx: AssemblyContext) -> Segment:
|
|
return Segment(text=self.text)
|
|
|
|
eng = ContextAssembler(
|
|
[_Seg("identity", 1, "IDENT", True), _Seg("boot", 2, "BOOT", True), _Seg("recall", 3, "RECALL", False)],
|
|
lambda: [],
|
|
)
|
|
|
|
ac = await eng.assemble(
|
|
"s", [], TokenBudget(100_000, 4_000, 2_000, 1_000, 93_000), turn=TurnContext(current_message="hi")
|
|
)
|
|
|
|
system = ac.messages[0]
|
|
chars = system[STABLE_PREFIX_KEY]
|
|
assert system["content"][:chars] == "IDENT" + _SEG_SEP + "BOOT" + _SEG_SEP
|
|
assert system["content"][chars:] == "RECALL"
|
|
|
|
async def test_an_all_volatile_prefix_declares_nothing(self):
|
|
from raven.context_engine import ContextAssembler, TurnContext
|
|
from raven.contracts.assembled import TokenBudget
|
|
from raven.contracts.context import AssemblyContext, Segment
|
|
|
|
class _Seg:
|
|
name, order, needs_prefix, stable = "recall", 1, False, False
|
|
|
|
async def build(self, ctx: AssemblyContext) -> Segment:
|
|
return Segment(text="RECALL")
|
|
|
|
eng = ContextAssembler([_Seg()], lambda: [])
|
|
|
|
ac = await eng.assemble(
|
|
"s", [], TokenBudget(100_000, 4_000, 2_000, 1_000, 93_000), turn=TurnContext(current_message="hi")
|
|
)
|
|
|
|
assert STABLE_PREFIX_KEY not in ac.messages[0]
|