## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
776 lines
31 KiB
Python
776 lines
31 KiB
Python
"""The main agent recognises run-and-watch work and steers it to the on-call agent.
|
|
|
|
The failure these pin, measured twice: a request naming a solver on this very
|
|
computer, a budget, and a shared machine was read correctly and iteration 1
|
|
still went straight to a local shell -- with the roster entry saying, in as many
|
|
words, that local work counts (2026-08-27). The one channel measured to change
|
|
the next move is a line arriving in a tool result, so that is the only place
|
|
these tests look for it.
|
|
|
|
The judgement's own failure modes have to stay silent: it sits in front of every
|
|
path-touching tool call, and a parse error there would break looking at files.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from raven.agent.subagent import watch_work
|
|
|
|
# --- read_verdict: anything unreadable is "not watched", never an exception ---
|
|
|
|
|
|
def test_a_clean_yes_carries_its_paths():
|
|
v = watch_work.read_verdict('{"watched": true, "paths": ["/tmp/arena"]}')
|
|
assert v.watched and v.subjects == ["/tmp/arena"]
|
|
|
|
|
|
def test_prose_around_the_json_is_tolerated():
|
|
v = watch_work.read_verdict('Sure!\n```{"watched": true, "paths": []}```')
|
|
assert v.watched
|
|
|
|
|
|
@pytest.mark.parametrize("reply", ["", None, "not json", '["list"]', '{"watched": "unsure"}'])
|
|
def test_everything_unreadable_or_unsure_is_a_quiet_no(reply):
|
|
assert watch_work.read_verdict(reply).watched is False
|
|
|
|
|
|
def test_claims_covers_the_path_and_what_sits_under_it():
|
|
v = watch_work.Verdict(watched=True, subjects=["/tmp/arena"])
|
|
assert v.claims("/tmp/arena")
|
|
assert v.claims("/tmp/arena/runs/t1/config.json")
|
|
assert not v.claims("/tmp/elsewhere")
|
|
assert not v.claims("")
|
|
|
|
|
|
def test_a_no_claims_nothing_even_with_paths():
|
|
assert watch_work.Verdict(watched=False, subjects=["/tmp/arena"]).claims("/tmp/arena") is False
|
|
|
|
|
|
def test_a_legacy_paths_reply_still_reads():
|
|
v = watch_work.read_verdict('{"watched": true, "paths": ["/tmp/arena"]}')
|
|
assert v.watched and v.subjects == ["/tmp/arena"]
|
|
|
|
|
|
def test_a_url_subject_claims_the_cli_call_that_reaches_it():
|
|
"""The measured shape (2026-08-28): the owner names a pipeline by web URL,
|
|
and every look arrives as a glab call carrying the project slug
|
|
percent-encoded. Neither side contains the other verbatim."""
|
|
v = watch_work.Verdict(watched=True, subjects=["https://gitlab.com/npc-work/aic/ai/raven/-/pipelines/2798916676"])
|
|
assert v.claims("glab api projects/npc-work%2Faic%2Fai%2Fraven/pipelines/2798916676 2>&1")
|
|
assert v.claims("https://gitlab.com/npc-work/aic/ai/raven/-/pipelines/2798916676")
|
|
assert not v.claims("glab api projects/other%2Fproject/pipelines/999")
|
|
assert not v.claims("ls /tmp")
|
|
|
|
|
|
def test_a_relative_path_keeps_the_containment_it_always_had():
|
|
"""The review's counterexample: a leading-slash dispatch lost this claim,
|
|
while the prior unconditional Path containment granted it."""
|
|
v = watch_work.Verdict(watched=True, subjects=["run"])
|
|
assert v.claims("run/job.sh")
|
|
assert v.claims("run")
|
|
assert not v.claims("elsewhere/job.sh")
|
|
assert not v.claims("run-old/job.sh"), "a sibling with a longer name is not under it"
|
|
|
|
|
|
def test_a_short_handle_is_still_claimable():
|
|
v = watch_work.Verdict(watched=True, subjects=["@bob"])
|
|
assert v.claims("glab api /users/@bob/events")
|
|
assert not v.claims("glab api /users/@alice/events")
|
|
|
|
|
|
def test_a_url_subject_never_claims_a_mere_path_and_vice_versa():
|
|
url = watch_work.Verdict(watched=True, subjects=["https://gitlab.com/g/p/-/pipelines/12345678"])
|
|
assert not url.claims("/tmp/arena/runs/t1")
|
|
path = watch_work.Verdict(watched=True, subjects=["/tmp/arena"])
|
|
assert not path.claims("glab api projects/g%2Fp/pipelines/12345678")
|
|
|
|
|
|
# --- asked_for: the judgement reads the owner's words, not the runtime glue ---
|
|
|
|
|
|
def test_asked_for_takes_the_last_user_message_and_strips_the_metadata_block():
|
|
messages = [
|
|
{"role": "system", "content": "be raven"},
|
|
{"role": "user", "content": "old request"},
|
|
{"role": "assistant", "content": "done"},
|
|
{"role": "user", "content": "[Runtime Context]\nnow=12:00\n\nrun a case, 25 min budget"},
|
|
]
|
|
assert watch_work.asked_for(messages) == "run a case, 25 min budget"
|
|
|
|
|
|
def test_asked_for_handles_block_content_and_empty_history():
|
|
messages = [{"role": "user", "content": [{"type": "text", "text": "watch this"}]}]
|
|
assert watch_work.asked_for(messages) == "watch this"
|
|
assert watch_work.asked_for([]) == ""
|
|
|
|
|
|
# --- the declaration: settled at admission, carried on the table ---
|
|
|
|
|
|
def _admitted(tmp_path, entry: dict):
|
|
"""One folder manifest through the real admission path, to a table row.
|
|
|
|
``discover_product_rows`` for the half that reads the folder, then
|
|
``_row_for`` for the half the registry dispenses -- neither reimplemented
|
|
here, because a stand-in for either would keep passing on the day the real
|
|
one stops carrying the declaration, which is the failure being pinned.
|
|
"""
|
|
from raven.agent.subagent.registry import _row_for
|
|
from raven.agent.subagent.vendored_agents import discover_product_rows
|
|
|
|
folder = tmp_path / "raven-oncall"
|
|
folder.mkdir()
|
|
(folder / "subagent.json").write_text(json.dumps(entry), encoding="utf-8")
|
|
(folder / "install.py").write_text("", encoding="utf-8")
|
|
(cfg,) = discover_product_rows(root=tmp_path)
|
|
return _row_for(cfg)
|
|
|
|
|
|
@pytest.mark.parametrize("kind", ["cli", "acp"])
|
|
@pytest.mark.parametrize("spelling", ["ownsWatchedWork", "owns_watched_work", "runsOnMachines", "runs_on_machines"])
|
|
def test_every_spelling_the_schema_ever_accepted_reaches_the_table(kind, spelling, tmp_path):
|
|
"""Both legacy spellings are what a row or a folder written before the
|
|
rename carries, and ``Base`` sets ``populate_by_name``, so both were valid.
|
|
A declaration admitted here has to arrive on the dispensed row: the reader
|
|
downstream has no other source, so a spelling dropped at admission is a
|
|
steering that goes quiet on upgrade with nothing said."""
|
|
row = _admitted(tmp_path, {"name": "Raven-Oncall", "kind": kind, "command": "run", spelling: True})
|
|
|
|
assert row.owns_watched_work is True
|
|
assert row.meta().owns_watched_work is True
|
|
|
|
|
|
@pytest.mark.parametrize("kind", ["cli", "acp"])
|
|
def test_a_manifest_without_the_declaration_reaches_the_table_without_it(kind, tmp_path):
|
|
row = _admitted(tmp_path, {"name": "Raven-Code", "kind": kind, "command": "run"})
|
|
|
|
assert row.owns_watched_work is False
|
|
assert row.meta().owns_watched_work is False
|
|
|
|
|
|
# --- oncall_agent: which row on this roster the request belongs with ---
|
|
|
|
|
|
def _meta(name: str, *, owns_watched_work: bool):
|
|
from raven.agent.subagent.backends import AgentMeta
|
|
|
|
return AgentMeta(name, "", False, False, owns_watched_work=owns_watched_work)
|
|
|
|
|
|
def test_the_row_that_declares_watched_work_is_the_one_named():
|
|
roster = [_meta("Raven-Code", owns_watched_work=False), _meta("Raven-Oncall", owns_watched_work=True)]
|
|
|
|
assert watch_work.oncall_agent(roster) == "Raven-Oncall"
|
|
|
|
|
|
def test_a_roster_nobody_flags_names_nobody():
|
|
assert watch_work.oncall_agent([_meta("Raven-Code", owns_watched_work=False)]) is None
|
|
|
|
|
|
def test_a_row_that_cannot_answer_the_question_names_nobody():
|
|
"""Duck-typed rows reach this from the DAG runner and from test doubles; one
|
|
that never heard of the declaration is not the specialist."""
|
|
from types import SimpleNamespace
|
|
|
|
assert watch_work.oncall_agent([SimpleNamespace(name="Raven-Code")]) is None
|
|
|
|
|
|
# --- the nudge itself ---
|
|
|
|
|
|
def test_the_nudge_names_the_spawn_and_both_shapes():
|
|
line = watch_work.nudge("Raven-Oncall")
|
|
assert "spawn `Raven-Oncall`" in line
|
|
assert "RUN" in line and "WATCH" in line, (
|
|
"measured 2026-08-21: steered with run-only words, a watch task built its "
|
|
"own monitor out of write_file and cron"
|
|
)
|
|
assert "before running anything by hand" in line
|
|
|
|
|
|
# --- the loop hook, driven through a minimal AgentLoop stand-in ---
|
|
|
|
|
|
class _Response:
|
|
def __init__(self, content):
|
|
self.content = content
|
|
|
|
|
|
class _SpawnStub:
|
|
"""Real ``AgentMeta`` rows, because the hook reads the declaration off them."""
|
|
|
|
def __init__(self, names):
|
|
self._names = names
|
|
|
|
def _agents(self):
|
|
return [_meta(n, owns_watched_work=n == "Raven-Oncall") for n in self._names]
|
|
|
|
|
|
class _Tools:
|
|
def __init__(self, names):
|
|
self._spawn = _SpawnStub(names) if names is not None else None
|
|
|
|
def get(self, name):
|
|
return self._spawn if name == "spawn" else None
|
|
|
|
|
|
def _loop(names=("Raven-Oncall",), verdict='{"watched": true, "paths": ["/tmp/arena"]}'):
|
|
"""An AgentLoop with only what _note_watch_work reads, plus a turn's state.
|
|
|
|
Returns the loop and one turn's TurnWatch: the state lives with the turn,
|
|
not the loop, which is the whole point pinned by the interleaving test.
|
|
"""
|
|
from types import SimpleNamespace
|
|
|
|
from raven.agent.loop.main import AgentLoop
|
|
|
|
loop = AgentLoop.__new__(AgentLoop)
|
|
loop.tools = _Tools(list(names) if names is not None else None)
|
|
# `model` is a property over the active binding; give the fallback leg one.
|
|
loop._default_binding = SimpleNamespace(model="test-model")
|
|
loop._llm_calls = 0
|
|
loop._llm_kwargs = []
|
|
|
|
async def fake_llm(messages, tools, model, **kwargs):
|
|
loop._llm_calls += 1
|
|
loop._llm_kwargs.append(dict(kwargs))
|
|
return _Response(verdict)
|
|
|
|
loop._llm_call_stream = fake_llm
|
|
return loop, watch_work.TurnWatch()
|
|
|
|
|
|
ACCEPTED = "Subagent [x] started (id: abc123). I'll notify you when it completes."
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_look_at_the_named_path_gains_the_line():
|
|
loop, state = _loop()
|
|
out = await loop._note_watch_work(
|
|
state, "read_file", {"path": "/tmp/arena/run.sh"}, "file contents", "run /tmp/arena, 25 min budget"
|
|
)
|
|
assert "spawn `Raven-Oncall`" in out
|
|
assert "file contents" not in out, (
|
|
"the note travels alone: the result is fenced as untrusted data by "
|
|
"add_tool_result, and a note appended to it would ride inside the fence"
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_exec_commands_are_searched_for_the_path():
|
|
loop, state = _loop()
|
|
out = await loop._note_watch_work(
|
|
state, "exec", {"command": "cd /tmp/arena/runs && bash run.sh"}, "ok", "run /tmp/arena"
|
|
)
|
|
assert "spawn `Raven-Oncall`" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_judgement_is_paid_once_per_turn():
|
|
loop, state = _loop()
|
|
for _ in range(3):
|
|
await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert loop._llm_calls == 1
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_judgement_that_stalls_is_cut_and_paid_once(monkeypatch):
|
|
"""2026-09-08: a relay turn's judgement ran 1229s (222k chars of reasoning,
|
|
no answer) and held the conversation. The call is bounded; a turn that hits
|
|
the bound goes on unjudged and does not pay again on its next look."""
|
|
import asyncio
|
|
|
|
from raven.agent.loop.main import AgentLoop
|
|
|
|
loop, state = _loop()
|
|
|
|
async def stalled(messages, tools, model, **kwargs):
|
|
loop._llm_calls += 1
|
|
await asyncio.sleep(5)
|
|
return _Response('{"watched": true, "paths": ["/tmp/arena"]}')
|
|
|
|
loop._llm_call_stream = stalled
|
|
monkeypatch.setattr(AgentLoop, "_WATCH_JUDGEMENT_TIMEOUT_S", 0.05)
|
|
first = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
second = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/y"}, "r", "run /tmp/arena")
|
|
assert first == "" and second == ""
|
|
assert loop._llm_calls == 1, "the cut verdict settles the turn: no second judgement"
|
|
assert state.verdict is not None and not state.verdict.watched
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_judgement_that_raises_leaves_the_turn_unjudged_not_failed():
|
|
"""The callers already swallow a raise, so the turn never failed on one --
|
|
but the verdict stayed unset and every later look paid the failed call
|
|
again. A raise is read like a hang: a settled no verdict, paid once."""
|
|
loop, state = _loop()
|
|
|
|
async def broken(messages, tools, model, **kwargs):
|
|
loop._llm_calls += 1
|
|
raise RuntimeError("provider down")
|
|
|
|
loop._llm_call_stream = broken
|
|
first = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
second = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/y"}, "r", "run /tmp/arena")
|
|
assert first == "" and second == ""
|
|
assert loop._llm_calls == 1
|
|
assert state.verdict is not None and not state.verdict.watched
|
|
|
|
|
|
def test_a_reinjected_turn_reads_as_no_request():
|
|
"""A sub-agent's result relay and a sentinel notice are the runtime speaking,
|
|
not the owner: the judgement has nothing to read, so nothing is paid."""
|
|
from raven.spine import Origin
|
|
|
|
messages = [{"role": "user", "content": "[Subagent 'x' returned]\n\nrun /tmp/arena, 25 min budget"}]
|
|
assert watch_work.asked_for(messages, origin=Origin.SUBAGENT) == ""
|
|
assert watch_work.asked_for(messages, origin=Origin.SENTINEL) == ""
|
|
assert watch_work.asked_for(messages, origin=Origin.USER).startswith("[Subagent")
|
|
assert watch_work.asked_for(messages, origin=Origin.CRON).startswith("[Subagent")
|
|
assert watch_work.asked_for(messages) == watch_work.asked_for(messages, origin=None)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_not_watched_verdict_leaves_every_result_alone():
|
|
loop, state = _loop(verdict='{"watched": false, "paths": []}')
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "read this file for me")
|
|
assert out == ""
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_look_elsewhere_stays_clean_even_on_a_watched_turn():
|
|
loop, state = _loop()
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/etc/hosts"}, "r", "run /tmp/arena")
|
|
assert out == ""
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_after_a_real_oncall_dispatch_the_turn_goes_quiet():
|
|
"""The judgement is paid at most once per turn. A straight-to-spawn turn
|
|
pays it AT the hand-off, to judge the dispatch's shape (2026-09-01: a bare
|
|
watcher spawn on a code-driven request dropped the coding half and every
|
|
nudge then went silent) -- one flash call against hours of misdirected
|
|
budget. Looks after the hand-off still pay nothing."""
|
|
loop, state = _loop()
|
|
await loop._note_watch_work(state, "spawn", {"subagent": "Raven-Oncall"}, ACCEPTED, "run /tmp/arena")
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert out == ""
|
|
assert loop._llm_calls == 1, "the judgement is paid once, at the hand-off, and never again"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unrelated_spawn_silences_nothing():
|
|
"""Reproduced in review: a spawn of another agent read as a hand-off and
|
|
suppressed the next matching path nudge."""
|
|
loop, state = _loop(names=("Raven-Code", "Raven-Oncall"))
|
|
await loop._note_watch_work(
|
|
state, "spawn", {"subagent": "Raven-Code"}, ACCEPTED.replace("x", "code job"), "run /tmp/arena"
|
|
)
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert "spawn `Raven-Oncall`" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"refusal",
|
|
[
|
|
"Spawn refused: delegation is paused",
|
|
"Error: prompt_template names a file this sub-agent cannot be given",
|
|
],
|
|
)
|
|
async def test_a_refused_dispatch_of_any_shape_silences_nothing(refusal):
|
|
loop, state = _loop()
|
|
await loop._note_watch_work(state, "spawn", {"subagent": "Raven-Oncall"}, refusal, "run /tmp/arena")
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert "spawn `Raven-Oncall`" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_two_interleaved_turns_never_share_watch_state():
|
|
"""run_turn lets turns from other sessions run concurrently on the one
|
|
loop object: B's dispatch must not silence A, and A's verdict must not
|
|
answer for B."""
|
|
loop, state_a = _loop()
|
|
state_b = watch_work.TurnWatch()
|
|
|
|
await loop._note_watch_work(state_b, "spawn", {"subagent": "Raven-Oncall"}, ACCEPTED, "run /tmp/arena")
|
|
out_a = await loop._note_watch_work(state_a, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
|
|
assert "spawn `Raven-Oncall`" in out_a, "B's hand-off silenced A"
|
|
assert state_b.dispatched and not state_a.dispatched
|
|
assert state_a.verdict is not None and state_b.verdict is not None
|
|
assert state_a.verdict is not state_b.verdict, "each turn holds its own verdict"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_roster_without_the_specialist_asks_no_judgement():
|
|
loop, state = _loop(names=("Raven-Code",))
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert out == ""
|
|
assert loop._llm_calls == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_judgement_that_blows_up_never_reaches_the_result(monkeypatch):
|
|
loop, state = _loop()
|
|
|
|
def broken(agents):
|
|
raise RuntimeError("probe exploded")
|
|
|
|
monkeypatch.setattr(watch_work, "oncall_agent", broken)
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert out == ""
|
|
|
|
|
|
# --- the spawn itself: nothing stands between the call and the manager ---
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_spawn_reaches_the_manager(tmp_path, monkeypatch):
|
|
"""Including the specialist, with no machine set up anywhere. The host used
|
|
to read the owner's registry here and refuse a spawn it found empty; where
|
|
the work runs is now the sub-agent's own question, asked after it starts."""
|
|
monkeypatch.setenv("RAVEN_CONNECTIONS", str(tmp_path / "never-written.json"))
|
|
from raven.agent.subagent.spawn_tool import SpawnTool
|
|
|
|
class _Manager:
|
|
def __init__(self, tmp):
|
|
self.spawned = False
|
|
self.workspace = str(tmp)
|
|
self._tmp = tmp
|
|
|
|
async def spawn(self, **kwargs):
|
|
self.spawned = True
|
|
return "spawned ok"
|
|
|
|
def reference_roots(self, session_key):
|
|
return []
|
|
|
|
def session_dir_for(self, session_key):
|
|
return self._tmp
|
|
|
|
mgr = _Manager(tmp_path)
|
|
tool = SpawnTool(manager=mgr)
|
|
out = await tool.execute("watch a case", "run the case", subagent="Raven-Oncall", node_id="watch_case")
|
|
|
|
assert mgr.spawned and out == "spawned ok"
|
|
|
|
|
|
# --- a refused dispatch answers nothing: the pressure must survive it ---
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_successful_dispatch_still_goes_quiet():
|
|
loop, state = _loop()
|
|
await loop._note_watch_work(state, "spawn", {"subagent": "Raven-Oncall"}, ACCEPTED, "run /tmp/arena")
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert out == ""
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_declined_graph_silences_nothing():
|
|
"""Reproduced in review: the declined-graph message carries no error prefix,
|
|
and the old DAG leg searched the serialized call, so naming the on-call
|
|
agent in the graph was enough to read a non-dispatch as a hand-off."""
|
|
loop, state = _loop()
|
|
await loop._note_watch_work(
|
|
state,
|
|
"run_subagent_dag",
|
|
{"nodes": [{"id": "watch", "subagent": "Raven-Oncall"}]},
|
|
"The user did not approve this graph, so nothing was run. Do not re-submit it; ask what to change.",
|
|
"run /tmp/arena",
|
|
)
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert "spawn `Raven-Oncall`" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_graph_that_merely_mentions_the_agent_silences_nothing():
|
|
loop, state = _loop(names=("Raven-Code", "Raven-Oncall"))
|
|
await loop._note_watch_work(
|
|
state,
|
|
"run_subagent_dag",
|
|
{"nodes": [{"id": "build", "subagent": "Raven-Code", "prompt_template": "hand off to Raven-Oncall later"}]},
|
|
"DAG run r1 started in the background (1 nodes).",
|
|
"run /tmp/arena",
|
|
)
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert "spawn `Raven-Oncall`" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_accepted_graph_with_an_oncall_node_goes_quiet():
|
|
loop, state = _loop(names=("Raven-Code", "Raven-Oncall"))
|
|
await loop._note_watch_work(
|
|
state,
|
|
"run_subagent_dag",
|
|
{"nodes": [{"id": "build", "subagent": "Raven-Code"}, {"id": "watch", "subagent": "Raven-Oncall"}]},
|
|
"DAG run r2 started in the background (2 nodes).",
|
|
"run /tmp/arena",
|
|
)
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/x"}, "r", "run /tmp/arena")
|
|
assert out == ""
|
|
|
|
|
|
# --- the hard form: the second claimed look is refused, not nudged again ---
|
|
|
|
|
|
def test_the_nudge_names_both_dispatch_doors():
|
|
line = watch_work.nudge("Raven-Oncall")
|
|
assert "spawn `Raven-Oncall`" in line and "run_subagent_dag" in line
|
|
|
|
|
|
def test_the_second_nudge_is_the_hard_form():
|
|
first = watch_work.nudge("Raven-Oncall")
|
|
second = watch_work.nudge("Raven-Oncall", repeat=True)
|
|
assert not first.startswith("\n\nSTOP")
|
|
assert second.startswith("\n\nSTOP")
|
|
assert "do not ask the owner" in second
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_loop_escalates_on_the_second_claimed_look():
|
|
loop, state = _loop()
|
|
first = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/a"}, "r", "run /tmp/arena")
|
|
second = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/b"}, "r", "run /tmp/arena")
|
|
assert "spawn `Raven-Oncall`" in first and not first.startswith("\n\nSTOP")
|
|
assert second.startswith("\n\nSTOP")
|
|
|
|
|
|
# --- the owner-ask gate: a pre-dispatch question never reaches the owner ---
|
|
|
|
|
|
def _watched_state(subjects=("/srv/arena",)):
|
|
state = watch_work.TurnWatch()
|
|
state.verdict = watch_work.Verdict(watched=True, subjects=list(subjects))
|
|
state.agent = "Raven-Oncall"
|
|
return state
|
|
|
|
|
|
def test_a_pre_dispatch_ask_is_preempted_with_the_dispatch_door():
|
|
state = _watched_state()
|
|
out = watch_work.preempt_owner_ask(
|
|
state, {"questions": [{"question": "Path /srv/arena does not exist on this machine, where is it?"}]}
|
|
)
|
|
assert "not sent to the owner" in out and "spawn `Raven-Oncall`" in out
|
|
assert state.nudges == 1
|
|
|
|
|
|
def test_a_paraphrased_or_bundled_ask_is_still_preempted():
|
|
"""Measured 2026-08-28 run 6: the model wrote "this path" for the path and
|
|
bundled two case-answerable questions alongside; a verbatim-subject filter
|
|
let it through to the owner. The gate keys on the turn's judged state."""
|
|
state = _watched_state()
|
|
out = watch_work.preempt_owner_ask(
|
|
state,
|
|
{
|
|
"questions": [
|
|
{"question": "This path is on another machine. How do I reach that machine? SSH address?"},
|
|
{"question": "What are the cantilever beam's geometry and material parameters?"},
|
|
]
|
|
},
|
|
)
|
|
assert "not sent to the owner" in out and "spawn `Raven-Oncall`" in out
|
|
|
|
|
|
def test_no_verdict_or_handed_over_turn_never_intercepts():
|
|
bare = watch_work.TurnWatch()
|
|
assert watch_work.preempt_owner_ask(bare, {"questions": [{"question": "x /srv/arena"}]}) == ""
|
|
done = _watched_state()
|
|
done.dispatched = True
|
|
assert watch_work.preempt_owner_ask(done, {"questions": [{"question": "x /srv/arena"}]}) == ""
|
|
|
|
|
|
# -- hoarded_code_note: bulk source in the main context earns one line --
|
|
|
|
|
|
def _source_blob(n: int = 200) -> str:
|
|
return "\n".join(f"import os\ndef fn_{i}(x):\n return x + {i}" for i in range(n))
|
|
|
|
|
|
def test_a_bulk_source_result_earns_the_hoarding_note():
|
|
"""The 2026-09-01 shape: whole files fetched into the main context."""
|
|
note = watch_work.hoarded_code_note(_source_blob())
|
|
assert "workspace of the node" in note
|
|
assert "PATH" in note
|
|
|
|
|
|
def test_bulk_prose_is_not_code_and_earns_nothing():
|
|
prose = (
|
|
"The experiment concluded with a validation score that held steady "
|
|
"across seeds, and the report describes each round in detail. "
|
|
) * 200
|
|
assert watch_work.hoarded_code_note(prose) == ""
|
|
|
|
|
|
def test_a_small_snippet_earns_nothing():
|
|
"""Quoting a function while discussing it is normal conversation."""
|
|
assert watch_work.hoarded_code_note("def f(x):\n return x\n" * 20) == ""
|
|
|
|
|
|
# -- solo dispatch: a bare watcher spawn on a code-driven request --
|
|
|
|
|
|
CODE_VERDICT = '{"watched": true, "subjects": ["/tmp/arena"], "code_work": true}'
|
|
RUN_VERDICT = '{"watched": true, "subjects": ["/tmp/arena"], "code_work": false}'
|
|
|
|
|
|
def test_read_verdict_parses_code_work():
|
|
assert watch_work.read_verdict(CODE_VERDICT).code_work is True
|
|
assert watch_work.read_verdict(RUN_VERDICT).code_work is False
|
|
assert watch_work.read_verdict('{"watched": true, "subjects": []}').code_work is False
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_bare_watcher_spawn_on_code_work_earns_the_solo_note():
|
|
"""The 2026-09-01 shape: the whole search handed to the watcher as one
|
|
spawn, the coding half owned by nobody, every nudge then silent."""
|
|
loop, state = _loop(verdict=CODE_VERDICT)
|
|
out = await loop._note_watch_work(
|
|
state,
|
|
"spawn",
|
|
{"subagent": "Raven-Oncall", "task": "search"},
|
|
ACCEPTED,
|
|
"edit train.py for the search, run /tmp/arena",
|
|
)
|
|
assert "coding half" in out
|
|
assert "run_subagent_dag" in out
|
|
assert ACCEPTED not in out, (
|
|
"the confirmation travels once, as the fenced tool result; repeating it in the trusted note "
|
|
"was the duplication measured on every dispatch before this branch honoured the caller contract"
|
|
)
|
|
assert state.dispatched and state.solo
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_watcher_spawn_on_run_only_work_is_a_complete_handoff():
|
|
loop, state = _loop(verdict=RUN_VERDICT)
|
|
out = await loop._note_watch_work(
|
|
state,
|
|
"spawn",
|
|
{"subagent": "Raven-Oncall", "task": "run the case"},
|
|
ACCEPTED,
|
|
"run /tmp/arena and report the result",
|
|
)
|
|
assert "coding half" not in out
|
|
assert state.dispatched and not state.solo
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_later_claimed_look_repeats_the_solo_correction():
|
|
loop, state = _loop(verdict=CODE_VERDICT)
|
|
await loop._note_watch_work(
|
|
state, "spawn", {"subagent": "Raven-Oncall", "task": "t"}, ACCEPTED, "edit train.py, run /tmp/arena"
|
|
)
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/train.py"}, "code", "msg")
|
|
assert out.startswith("\n\nSTOP")
|
|
assert "coding half" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_dag_dispatch_clears_the_solo_state():
|
|
loop, state = _loop(verdict=CODE_VERDICT)
|
|
await loop._note_watch_work(
|
|
state, "spawn", {"subagent": "Raven-Oncall", "task": "t"}, ACCEPTED, "edit train.py, run /tmp/arena"
|
|
)
|
|
nodes = [{"subagent": "Raven-Code"}, {"subagent": "Raven-Oncall"}]
|
|
await loop._note_watch_work(state, "run_subagent_dag", {"nodes": nodes}, "run r1 started in the background", "msg")
|
|
assert not state.solo
|
|
out = await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/train.py"}, "code", "msg")
|
|
assert "coding half" not in out
|
|
|
|
|
|
# --- the note channel never carries tool output past the fence ---
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
("tool", "args", "result"),
|
|
[
|
|
("spawn", {"subagent": "Raven-Oncall"}, ACCEPTED),
|
|
("run_subagent_dag", {"task_summary": "run and watch"}, "DAG run r1 accepted: node watch on Raven-Oncall"),
|
|
("spawn", {"subagent": "Raven-Oncall"}, "'Raven-Oncall' has no machine. Nothing was dispatched."),
|
|
],
|
|
)
|
|
async def test_a_dispatch_result_reaches_the_model_once_and_inside_the_fence(tmp_path, tool, args, result):
|
|
"""The whole road: ``_note_watch_work`` -> ``add_tool_result``.
|
|
|
|
``add_tool_result`` places the returned note AFTER the untrusted fence, as
|
|
this system's own voice. Reviewed 2026-09-04: the spawn and DAG exits still
|
|
returned ``result`` under the new contract, so a sub-agent's own text rode
|
|
out of the fence as a trusted note -- a trust-boundary hole, not a display
|
|
duplicate. Only system-composed text may travel that channel.
|
|
"""
|
|
from raven.agent.context.builder import ContextBuilder
|
|
|
|
loop, state = _loop()
|
|
note = await loop._note_watch_work(state, tool, args, result, "run /tmp/arena")
|
|
assert result not in note, "tool output must never be returned as the trusted note"
|
|
|
|
messages = ContextBuilder(workspace=tmp_path).add_tool_result([], "c1", tool, result, trusted_note=note)
|
|
content = messages[-1]["content"]
|
|
assert content.count(result) == 1, "the result travels once, as the fenced tool result"
|
|
fence_end = content.index("[END UNTRUSTED")
|
|
assert result not in content[fence_end:], "and nothing of it appears after the fence closes"
|
|
assert content.index(result) < fence_end
|
|
|
|
|
|
def test_the_resident_dag_skill_digest_routes_code_work_through_the_roster():
|
|
"""The orchestration skill is ``always: true`` with ``inject: description``:
|
|
only its frontmatter description reaches an ordinary turn, and the body
|
|
only after the model calls ``read_skill``. So the door the watch-work notes
|
|
point at (a coding node feeding an on-call node) has to be stated in the
|
|
description itself, or the resident digest keeps telling the model that
|
|
code work is never a DAG while the notes tell it to dispatch one."""
|
|
import raven
|
|
|
|
skill = Path(raven.__file__).parent / "memory_engine" / "skills" / "subagent-dag-orchestration" / "SKILL.md"
|
|
head = skill.read_text(encoding="utf-8").split("---")[1]
|
|
description = next(line for line in head.splitlines() if line.startswith("description:"))
|
|
assert "never a DAG" not in description
|
|
assert "specialist" in description and "run_subagent_dag" in description
|
|
assert "one graph per round" in description
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# the judgement is a model call of the turn, so it runs at the turn's effort #
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_judgement_asks_for_the_turns_effort():
|
|
"""A medium/max session pins an effort for every model call of the turn;
|
|
the classifier this helper pays for on a look is one of them."""
|
|
loop, state = _loop()
|
|
await loop._note_watch_work(
|
|
state, "read_file", {"path": "/tmp/arena/run.sh"}, "r", "run /tmp/arena", reasoning_effort="max"
|
|
)
|
|
assert loop._llm_kwargs == [{"reasoning_effort": "max"}]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_hand_off_judgement_asks_for_the_turns_effort_too():
|
|
"""The other call site: a straight-to-spawn turn judges the hand-off's
|
|
shape with a classifier call of its own."""
|
|
loop, state = _loop(verdict='{"watched": true, "code_work": true, "paths": ["/tmp/arena"]}')
|
|
await loop._note_watch_work(
|
|
state,
|
|
"spawn",
|
|
{"subagent": "Raven-Oncall"},
|
|
ACCEPTED,
|
|
"run /tmp/arena and fix the tests",
|
|
reasoning_effort="medium",
|
|
)
|
|
assert loop._llm_kwargs == [{"reasoning_effort": "medium"}]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_no_pinned_effort_passes_nothing_so_the_provider_default_stands():
|
|
loop, state = _loop()
|
|
await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/run.sh"}, "r", "run /tmp/arena")
|
|
assert loop._llm_kwargs == [{}], "an explicit None would override the provider's sentinel"
|