## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
749 lines
27 KiB
Python
749 lines
27 KiB
Python
"""Contract tests for AgentHook + CompositeHook.
|
|
|
|
Pins the contract that the legacy-callback adapters rely on when porting
|
|
AgentLoop's existing scattered callback fields to hooks, and that the
|
|
eval_engine relies on when implementing its three iteration-phase hooks.
|
|
|
|
The hooks live as a standalone abstraction in ``raven/agent/hook/``.
|
|
This test file's job is to lock in the eight contract dimensions:
|
|
|
|
1. Default no-op for every phase.
|
|
2. ``HookDecision`` default state.
|
|
3. ``AgentHookContext`` field defaults.
|
|
4. ``CompositeHook`` empty.
|
|
5. ``CompositeHook`` single-member dispatch.
|
|
6. ``CompositeHook`` registration-order preservation.
|
|
7. ``CompositeHook`` short-circuit halts the chain.
|
|
8. ``CompositeHook`` ``after_send`` content chains through.
|
|
|
|
Plus exception-isolation (a hook that raises is treated as no-op,
|
|
chain continues) and late-registration via ``append``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from raven.agent.hook import (
|
|
AgentHook,
|
|
AgentHookContext,
|
|
CompositeHook,
|
|
HookDecision,
|
|
)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Fixtures
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.fixture
|
|
def ctx():
|
|
"""Minimal context — only the required ``session_key`` is set."""
|
|
return AgentHookContext(session_key="cli:test")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 1. Default no-op for every phase
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
PHASES = [
|
|
"before_user_inbound",
|
|
"before_iteration",
|
|
"before_execute_tools",
|
|
"after_iteration",
|
|
"terminal_answerless",
|
|
"after_send",
|
|
]
|
|
|
|
|
|
class TestAgentHookDefaults:
|
|
"""All phases on the bare ``AgentHook`` are pass-through no-ops."""
|
|
|
|
@pytest.mark.parametrize("phase", PHASES)
|
|
async def test_default_returns_passthrough(self, ctx, phase):
|
|
class Bare(AgentHook):
|
|
pass
|
|
|
|
hook = Bare()
|
|
decision = await getattr(hook, phase)(ctx)
|
|
assert isinstance(decision, HookDecision)
|
|
assert decision.pass_through is True
|
|
assert decision.short_circuit_result is None
|
|
assert decision.modified_content is None
|
|
|
|
def test_name_defaults_to_class_name(self):
|
|
class Foo(AgentHook):
|
|
pass
|
|
|
|
assert Foo().name == "Foo"
|
|
|
|
def test_name_can_be_overridden(self):
|
|
class Bar(AgentHook):
|
|
@property
|
|
def name(self) -> str:
|
|
return "Custom"
|
|
|
|
assert Bar().name == "Custom"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. HookDecision dataclass
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestHookDecision:
|
|
def test_default_state(self):
|
|
d = HookDecision()
|
|
assert d.pass_through is True
|
|
assert d.short_circuit_result is None
|
|
assert d.modified_content is None
|
|
assert d.rollback is False
|
|
assert d.rollback_overrides is None
|
|
assert d.rollback_inject is None
|
|
assert d.modified_tools is None
|
|
assert d.notes == []
|
|
|
|
def test_short_circuit_carries_arbitrary_value(self):
|
|
d = HookDecision(short_circuit_result="STOP")
|
|
assert d.short_circuit_result == "STOP"
|
|
|
|
d2 = HookDecision(short_circuit_result={"answer": 42})
|
|
assert d2.short_circuit_result == {"answer": 42}
|
|
|
|
def test_modified_content_separate_from_short_circuit(self):
|
|
d = HookDecision(modified_content="hello world")
|
|
assert d.modified_content == "hello world"
|
|
assert d.short_circuit_result is None
|
|
|
|
def test_notes_accumulator(self):
|
|
d = HookDecision(notes=["one", "two"])
|
|
assert d.notes == ["one", "two"]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. AgentHookContext
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestAgentHookContext:
|
|
def test_session_key_is_required(self):
|
|
with pytest.raises(TypeError):
|
|
AgentHookContext() # type: ignore[call-arg]
|
|
|
|
def test_default_fields(self):
|
|
c = AgentHookContext(session_key="cli:default")
|
|
assert c.session_key == "cli:default"
|
|
assert c.turn_request is None
|
|
assert c.iteration is None
|
|
assert c.messages is None
|
|
assert c.tools is None
|
|
assert c.response is None
|
|
assert c.outbound_content is None
|
|
assert c.metadata == {}
|
|
assert c.turn_question == ""
|
|
assert c.turn_base == 0
|
|
assert c.max_iterations is None
|
|
assert c.context_window_tokens is None
|
|
|
|
def test_fields_can_be_set(self):
|
|
c = AgentHookContext(
|
|
session_key="cli:x",
|
|
iteration=3,
|
|
outbound_content="hi",
|
|
metadata={"source": "test"},
|
|
)
|
|
assert c.iteration == 3
|
|
assert c.outbound_content == "hi"
|
|
assert c.metadata == {"source": "test"}
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 4. CompositeHook empty
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCompositeHookEmpty:
|
|
def test_zero_length(self):
|
|
composite = CompositeHook()
|
|
assert len(composite) == 0
|
|
|
|
def test_name_shows_empty(self):
|
|
assert CompositeHook().name == "CompositeHook(empty)"
|
|
|
|
@pytest.mark.parametrize("phase", PHASES)
|
|
async def test_phase_passthrough_when_empty(self, ctx, phase):
|
|
composite = CompositeHook()
|
|
decision = await getattr(composite, phase)(ctx)
|
|
assert decision.short_circuit_result is None
|
|
assert decision.modified_content is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 5. CompositeHook single-member dispatch
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCompositeHookSingleMember:
|
|
async def test_single_hook_called(self, ctx):
|
|
called: list[tuple[str, str]] = []
|
|
|
|
class Recorder(AgentHook):
|
|
async def before_iteration(self, ctx):
|
|
called.append(("before_iteration", ctx.session_key))
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook([Recorder()])
|
|
assert len(composite) == 1
|
|
await composite.before_iteration(ctx)
|
|
assert called == [("before_iteration", "cli:test")]
|
|
|
|
async def test_unrelated_phase_does_not_call_hook(self, ctx):
|
|
called: list[str] = []
|
|
|
|
class OnlyBefore(AgentHook):
|
|
async def before_iteration(self, ctx):
|
|
called.append("before_iteration")
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook([OnlyBefore()])
|
|
# Calling a phase the hook didn't override
|
|
await composite.after_iteration(ctx)
|
|
assert called == [] # before_iteration recorder never ran
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 6. Registration order preserved
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCompositeHookRegistrationOrder:
|
|
async def test_hooks_invoked_in_order(self, ctx):
|
|
order: list[str] = []
|
|
|
|
class Recorder(AgentHook):
|
|
def __init__(self, tag: str) -> None:
|
|
self.tag = tag
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return f"R({self.tag})"
|
|
|
|
async def before_iteration(self, ctx):
|
|
order.append(self.tag)
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook([Recorder("first"), Recorder("second"), Recorder("third")])
|
|
await composite.before_iteration(ctx)
|
|
assert order == ["first", "second", "third"]
|
|
|
|
async def test_name_reflects_order(self):
|
|
class Tagged(AgentHook):
|
|
def __init__(self, tag: str) -> None:
|
|
self.tag = tag
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return self.tag
|
|
|
|
composite = CompositeHook([Tagged("A"), Tagged("B")])
|
|
assert composite.name == "CompositeHook(A, B)"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 7. Short-circuit halts the chain
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCompositeHookShortCircuit:
|
|
async def test_first_short_circuit_wins(self, ctx):
|
|
called: list[str] = []
|
|
|
|
class Halter(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
called.append("halter")
|
|
return HookDecision(short_circuit_result="STOP")
|
|
|
|
class Trailing(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
called.append("trailing")
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook([Halter(), Trailing()])
|
|
decision = await composite.before_user_inbound(ctx)
|
|
|
|
assert decision.short_circuit_result == "STOP"
|
|
# Trailing must NOT be invoked once the chain short-circuits.
|
|
assert called == ["halter"]
|
|
|
|
async def test_short_circuit_only_in_named_phase(self, ctx):
|
|
"""A hook that short-circuits one phase does not affect other
|
|
phases — the chain runs independently per phase."""
|
|
called: list[str] = []
|
|
|
|
class Halter(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
called.append("halter_in_user_inbound")
|
|
return HookDecision(short_circuit_result="STOP")
|
|
|
|
async def before_iteration(self, ctx):
|
|
called.append("halter_in_iteration")
|
|
return HookDecision()
|
|
|
|
class Trailing(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
called.append("trailing_in_user_inbound")
|
|
return HookDecision()
|
|
|
|
async def before_iteration(self, ctx):
|
|
called.append("trailing_in_iteration")
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook([Halter(), Trailing()])
|
|
|
|
await composite.before_user_inbound(ctx)
|
|
# before_user_inbound: halter short-circuits, trailing skipped.
|
|
# before_iteration: both run.
|
|
await composite.before_iteration(ctx)
|
|
|
|
assert called == [
|
|
"halter_in_user_inbound",
|
|
"halter_in_iteration",
|
|
"trailing_in_iteration",
|
|
]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 8. after_send content chains through
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCompositeHookContentChain:
|
|
async def test_after_send_chains_modifications(self, ctx):
|
|
ctx.outbound_content = "original"
|
|
|
|
class Appender(AgentHook):
|
|
def __init__(self, suffix: str) -> None:
|
|
self.suffix = suffix
|
|
|
|
async def after_send(self, ctx):
|
|
base = ctx.outbound_content or ""
|
|
return HookDecision(modified_content=base + self.suffix)
|
|
|
|
composite = CompositeHook([Appender(" A"), Appender(" B")])
|
|
decision = await composite.after_send(ctx)
|
|
|
|
assert decision.modified_content == "original A B"
|
|
# ctx is left updated so any subsequent inspection sees the
|
|
# fully-chained content.
|
|
assert ctx.outbound_content == "original A B"
|
|
|
|
async def test_after_send_no_modification_means_no_change(self, ctx):
|
|
ctx.outbound_content = "untouched"
|
|
|
|
class Observer(AgentHook):
|
|
async def after_send(self, ctx):
|
|
# Observer-only, no modification
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook([Observer(), Observer()])
|
|
decision = await composite.after_send(ctx)
|
|
|
|
assert decision.modified_content is None
|
|
assert ctx.outbound_content == "untouched"
|
|
|
|
async def test_after_send_short_circuit_skips_chain(self, ctx):
|
|
ctx.outbound_content = "original"
|
|
called: list[str] = []
|
|
|
|
class Replacer(AgentHook):
|
|
async def after_send(self, ctx):
|
|
called.append("replacer")
|
|
return HookDecision(short_circuit_result="replaced")
|
|
|
|
class Trailing(AgentHook):
|
|
async def after_send(self, ctx):
|
|
called.append("trailing")
|
|
return HookDecision(modified_content="should not see")
|
|
|
|
composite = CompositeHook([Replacer(), Trailing()])
|
|
decision = await composite.after_send(ctx)
|
|
|
|
assert decision.short_circuit_result == "replaced"
|
|
assert called == ["replacer"]
|
|
|
|
async def test_after_iteration_does_not_chain_content(self, ctx):
|
|
"""Only ``after_send`` chains ``modified_content``. Other phases
|
|
ignore it."""
|
|
ctx.outbound_content = "starting"
|
|
|
|
class WouldModify(AgentHook):
|
|
async def after_iteration(self, ctx):
|
|
return HookDecision(modified_content="ignored")
|
|
|
|
composite = CompositeHook([WouldModify()])
|
|
decision = await composite.after_iteration(ctx)
|
|
|
|
# The dispatcher for non-after_send phases does not propagate
|
|
# modified_content into ctx, and the final decision exposes the
|
|
# raw last-modified (which is None because nothing was chained).
|
|
assert decision.modified_content is None
|
|
assert ctx.outbound_content == "starting" # unchanged
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Exception isolation
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCompositeHookExceptionIsolation:
|
|
async def test_one_hook_raising_does_not_break_chain(self, ctx):
|
|
called: list[str] = []
|
|
|
|
class Bad(AgentHook):
|
|
async def before_iteration(self, ctx):
|
|
called.append("bad")
|
|
raise RuntimeError("boom")
|
|
|
|
class Good(AgentHook):
|
|
async def before_iteration(self, ctx):
|
|
called.append("good")
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook([Bad(), Good()])
|
|
decision = await composite.before_iteration(ctx)
|
|
|
|
# Both ran; Bad's exception was swallowed.
|
|
assert called == ["bad", "good"]
|
|
assert decision.short_circuit_result is None
|
|
|
|
async def test_exception_in_short_circuit_position_does_not_halt(self, ctx):
|
|
"""A hook that would have short-circuited but raises instead
|
|
should NOT short-circuit — subsequent hooks must still run."""
|
|
called: list[str] = []
|
|
|
|
class Bad(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
called.append("bad")
|
|
raise RuntimeError("boom")
|
|
|
|
class Trailing(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
called.append("trailing")
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook([Bad(), Trailing()])
|
|
decision = await composite.before_user_inbound(ctx)
|
|
|
|
assert called == ["bad", "trailing"]
|
|
assert decision.short_circuit_result is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Late registration via append/extend
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCompositeHookRegistration:
|
|
async def test_append_adds_to_tail(self, ctx):
|
|
called: list[str] = []
|
|
|
|
class A(AgentHook):
|
|
async def before_iteration(self, ctx):
|
|
called.append("A")
|
|
return HookDecision()
|
|
|
|
class B(AgentHook):
|
|
async def before_iteration(self, ctx):
|
|
called.append("B")
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook([A()])
|
|
composite.append(B())
|
|
|
|
assert len(composite) == 2
|
|
await composite.before_iteration(ctx)
|
|
assert called == ["A", "B"]
|
|
|
|
async def test_extend_preserves_order(self, ctx):
|
|
called: list[str] = []
|
|
|
|
class Tagged(AgentHook):
|
|
def __init__(self, tag: str) -> None:
|
|
self.tag = tag
|
|
|
|
async def before_iteration(self, ctx):
|
|
called.append(self.tag)
|
|
return HookDecision()
|
|
|
|
composite = CompositeHook()
|
|
composite.extend([Tagged("x"), Tagged("y"), Tagged("z")])
|
|
|
|
await composite.before_iteration(ctx)
|
|
assert called == ["x", "y", "z"]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Smoke test: end-to-end mixed chain (mimics the adapter wire-up shape)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCompositeHookEndToEndScenario:
|
|
async def test_mixed_chain_simulates_sentinel_and_personalizer(self, ctx):
|
|
"""A realistic shape: one short-circuit candidate (Sentinel
|
|
decision_consumer style) sitting in front of an observer
|
|
(FeedbackTracker style) — when the short-circuit fires, the
|
|
observer must not run.
|
|
|
|
Then a separate phase (after_send) shows an injector-style hook
|
|
modifying the outbound content."""
|
|
log: list[str] = []
|
|
|
|
from raven.spine.message import ChatType, Source
|
|
from raven.spine.turn import Origin, TurnRequest
|
|
|
|
def _req(text: str) -> TurnRequest:
|
|
return TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="cli", chat_id="c", sender_id="u", chat_type=ChatType.DM),
|
|
text=text,
|
|
)
|
|
|
|
class FakeDecisionConsumer(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
log.append("decision_consumer.check")
|
|
if ctx.turn_request and ctx.turn_request.text.startswith("/pick"):
|
|
log.append("decision_consumer.short_circuit")
|
|
return HookDecision(short_circuit_result="picked option")
|
|
return HookDecision()
|
|
|
|
class FakeFeedbackTracker(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
# Observer — must not run when decision_consumer fires.
|
|
log.append("feedback_tracker.observe")
|
|
return HookDecision()
|
|
|
|
class FakeNudgeInjector(AgentHook):
|
|
async def after_send(self, ctx):
|
|
log.append("nudge_injector.after_send")
|
|
base = ctx.outbound_content or ""
|
|
return HookDecision(modified_content=base + " [nudge]")
|
|
|
|
composite = CompositeHook([FakeDecisionConsumer(), FakeFeedbackTracker(), FakeNudgeInjector()])
|
|
|
|
# Path 1: /pick reply → short-circuit, observer skipped.
|
|
ctx.turn_request = _req("/pick 2")
|
|
d1 = await composite.before_user_inbound(ctx)
|
|
assert d1.short_circuit_result == "picked option"
|
|
assert log == ["decision_consumer.check", "decision_consumer.short_circuit"]
|
|
|
|
# Path 2: normal reply → consumer doesn't short-circuit, observer runs.
|
|
log.clear()
|
|
ctx.turn_request = _req("hello")
|
|
d2 = await composite.before_user_inbound(ctx)
|
|
assert d2.short_circuit_result is None
|
|
assert log == ["decision_consumer.check", "feedback_tracker.observe"]
|
|
|
|
# after_send: nudge injector appends.
|
|
log.clear()
|
|
ctx.outbound_content = "hi user"
|
|
d3 = await composite.after_send(ctx)
|
|
assert d3.modified_content == "hi user [nudge]"
|
|
assert log == ["nudge_injector.after_send"]
|
|
|
|
|
|
class TestCompositeHaltingAndGrants:
|
|
async def test_rollback_halts_the_chain_like_a_short_circuit(self, ctx):
|
|
seen: list[str] = []
|
|
|
|
class Bouncer(AgentHook):
|
|
async def after_iteration(self, ctx):
|
|
seen.append("bouncer")
|
|
return HookDecision(rollback=True, rollback_inject=[{"role": "user", "content": "again"}])
|
|
|
|
class Later(AgentHook):
|
|
async def after_iteration(self, ctx):
|
|
seen.append("later")
|
|
return HookDecision()
|
|
|
|
decision = await CompositeHook([Bouncer(), Later()]).after_iteration(ctx)
|
|
assert decision.rollback is True
|
|
assert decision.rollback_inject == [{"role": "user", "content": "again"}]
|
|
assert seen == ["bouncer"]
|
|
|
|
async def test_modified_tools_chain_through_before_iteration(self, ctx):
|
|
class DropB(AgentHook):
|
|
async def before_iteration(self, ctx):
|
|
return HookDecision(modified_tools=[t for t in (ctx.tools or []) if t["name"] != "b"])
|
|
|
|
class DropC(AgentHook):
|
|
async def before_iteration(self, ctx):
|
|
return HookDecision(modified_tools=[t for t in (ctx.tools or []) if t["name"] != "c"])
|
|
|
|
ctx.tools = [{"name": "a"}, {"name": "b"}, {"name": "c"}]
|
|
decision = await CompositeHook([DropB(), DropC()]).before_iteration(ctx)
|
|
assert decision.modified_tools == [{"name": "a"}]
|
|
|
|
async def test_terminal_answerless_dispatches(self, ctx):
|
|
class Salvage(AgentHook):
|
|
async def terminal_answerless(self, ctx):
|
|
return HookDecision(short_circuit_result="best-supported answer")
|
|
|
|
decision = await CompositeHook([Salvage()]).terminal_answerless(ctx)
|
|
assert decision.short_circuit_result == "best-supported answer"
|
|
|
|
|
|
class TestNoteAndInboundGrants:
|
|
async def test_notes_chain_in_order_through_the_iteration_phases(self, ctx):
|
|
class Budget(AgentHook):
|
|
async def after_iteration(self, ctx):
|
|
return HookDecision(append_note="[budget: half spent]")
|
|
|
|
class Gate(AgentHook):
|
|
async def after_iteration(self, ctx):
|
|
return HookDecision(append_note="[gate: open a page before searching again]")
|
|
|
|
decision = await CompositeHook([Budget(), Gate()]).after_iteration(ctx)
|
|
assert decision.append_note == "[budget: half spent]\n\n[gate: open a page before searching again]"
|
|
assert (await CompositeHook([Budget()]).after_send(ctx)).append_note is None, "not an iteration phase"
|
|
|
|
async def test_decision_notes_chain_through_the_composite_and_survive_a_halt(self, ctx):
|
|
class Quiet(AgentHook):
|
|
async def after_iteration(self, ctx):
|
|
return HookDecision(notes=["quiet: looked, passed"])
|
|
|
|
class Halter(AgentHook):
|
|
async def after_iteration(self, ctx):
|
|
return HookDecision(short_circuit_result="done", notes=["halter: replaced the draft"])
|
|
|
|
class Never(AgentHook):
|
|
async def after_iteration(self, ctx):
|
|
return HookDecision(notes=["never runs"])
|
|
|
|
halted = await CompositeHook([Quiet(), Halter(), Never()]).after_iteration(ctx)
|
|
assert halted.short_circuit_result == "done"
|
|
assert halted.notes == ["quiet: looked, passed", "halter: replaced the draft"]
|
|
|
|
chained = await CompositeHook([Quiet(), Quiet()]).after_iteration(ctx)
|
|
assert chained.notes == ["quiet: looked, passed", "quiet: looked, passed"]
|
|
|
|
async def test_inbound_text_chains_through_inbound_content(self, ctx):
|
|
class Memo(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
return HookDecision(modified_content=f"[memo]\n\n{ctx.inbound_content}")
|
|
|
|
class Reminder(AgentHook):
|
|
async def before_user_inbound(self, ctx):
|
|
return HookDecision(modified_content=f"{ctx.inbound_content}\n\n[reminder]")
|
|
|
|
ctx.inbound_content = "what changed?"
|
|
decision = await CompositeHook([Memo(), Reminder()]).before_user_inbound(ctx)
|
|
assert decision.modified_content == "[memo]\n\nwhat changed?\n\n[reminder]"
|
|
assert ctx.outbound_content is None
|
|
|
|
|
|
def test_the_factory_loop_surface_is_versioned() -> None:
|
|
"""The tier's word is "versioned with the factory loop" -- but the hook
|
|
vocabulary grew five context fields and a phase with no number moving, so
|
|
versioned was prose rather than a mechanism. The roster below pins the
|
|
surface to the declared version: growing one without the other is red,
|
|
and the bump is what a reviewer (and a product hook author) sees.
|
|
"""
|
|
import dataclasses
|
|
|
|
import raven.contracts.loop_hooks as lh
|
|
|
|
surface = {
|
|
"context": sorted(f.name for f in dataclasses.fields(lh.AgentHookContext)),
|
|
"decision": sorted(f.name for f in dataclasses.fields(lh.HookDecision)),
|
|
"phases": sorted(
|
|
name for name, member in vars(lh.AgentHook).items() if callable(member) and not name.startswith("_")
|
|
),
|
|
}
|
|
pinned = {
|
|
# 1 was the pre-metadata surface (no shared turn dict, no session
|
|
# history, no inbound rewrite); it shipped unnumbered and is not
|
|
# reconstructed here.
|
|
2: {
|
|
"context": [
|
|
"inbound_content",
|
|
"iteration",
|
|
"messages",
|
|
"metadata",
|
|
"outbound_content",
|
|
"response",
|
|
"session_history",
|
|
"session_key",
|
|
"tools",
|
|
"turn_base",
|
|
"turn_question",
|
|
"turn_request",
|
|
],
|
|
"decision": [
|
|
"append_note",
|
|
"modified_content",
|
|
"modified_tools",
|
|
"notes",
|
|
"pass_through",
|
|
"rollback",
|
|
"rollback_inject",
|
|
"rollback_overrides",
|
|
"short_circuit_result",
|
|
],
|
|
"phases": [
|
|
"after_iteration",
|
|
"after_send",
|
|
"before_execute_tools",
|
|
"before_iteration",
|
|
"before_user_inbound",
|
|
"terminal_answerless",
|
|
],
|
|
},
|
|
3: {
|
|
"context": [
|
|
"context_window_tokens",
|
|
"inbound_content",
|
|
"iteration",
|
|
"max_iterations",
|
|
"messages",
|
|
"metadata",
|
|
"outbound_content",
|
|
"response",
|
|
"session_history",
|
|
"session_key",
|
|
"tools",
|
|
"turn_base",
|
|
"turn_question",
|
|
"turn_request",
|
|
],
|
|
"decision": [
|
|
"append_note",
|
|
"modified_content",
|
|
"modified_tools",
|
|
"notes",
|
|
"pass_through",
|
|
"rollback",
|
|
"rollback_inject",
|
|
"rollback_overrides",
|
|
"short_circuit_result",
|
|
],
|
|
"phases": [
|
|
"after_iteration",
|
|
"after_send",
|
|
"before_execute_tools",
|
|
"before_iteration",
|
|
"before_user_inbound",
|
|
"terminal_answerless",
|
|
],
|
|
},
|
|
}
|
|
|
|
assert lh.FACTORY_LOOP_SURFACE_VERSION in pinned, (
|
|
"a new surface version needs its roster pinned here in the same change"
|
|
)
|
|
assert surface == pinned[lh.FACTORY_LOOP_SURFACE_VERSION], (
|
|
"the hook surface changed: bump FACTORY_LOOP_SURFACE_VERSION in "
|
|
"raven/contracts/loop_hooks.py and pin the new roster here -- growing "
|
|
"one without the other is the drift this test exists to stop"
|
|
)
|