Raven-1/tests/test_eval_engine.py
Handsome-wzw 7b45527b5b fix(tools): an array sent as its own JSON is still an array
## Summary

Reported as "deliver_files is broken". It is not: **the tool never ran.**

Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:

```
Error: Invalid parameters for tool 'deliver_files': files should be array
```

What the model sent was the array, spelled as a string:

```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```

`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.

## Type

- [x] Fix
- [ ] Feature

## The edge of the forgiving is the design

When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.

**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.

**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.

## Not just deliver_files

Six built-in tools declare an array or object parameter and were all exposed:

| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |

The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.

## Verification

```
make lint-python    ruff check + ruff format --check, 1850 files
make lint-imports   9 contracts kept, 0 broken
pytest tests/       19,699 passed, 7 failed (all pre-existing, see below)
```

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string

**End to end, through the real tool**, with the exact payload that failed five times in production:

```
files sent as:            str -> [{"path": ".../poster.png", "title": "..."}]
returned:                 Delivered 1 file(s): poster.png (72 B).
still "should be array":  False
registered in the store:  True
```

Twelve cases, five mutations, each killing at least one:

| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |

The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.

**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.

## Risk

- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert

## Related Issues

N/A

## Swept

`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.

- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.

Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).

## Deliberately not fixed

The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.

## Moved from !594

Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.

Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00

394 lines
16 KiB
Python

"""Contract tests for the Eval Engine.
Pins the three-hook scaffold:
- ``BeforeIterationHook``: token-budget gate — short-circuits when the
per-iteration estimate exceeds ``config.max_iteration_tokens``.
- ``ToolAuditHook``: deterministic deny-list — short-circuits when a
tool call in ``ctx.response.tool_calls`` is in the denylist.
- ``AfterIterationHook``: LLM-judge → MemoryEngine writeback.
Default config (``enabled=False``) is a hard requirement — all three
hooks must be no-ops in the default state, so simply instantiating
``EvalEngine()`` and dropping its ``hooks()`` into a ``CompositeHook``
must not change AgentLoop behavior.
"""
from __future__ import annotations
from datetime import datetime
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
from raven.contracts.loop_hooks import AgentHookContext
from raven.eval_engine import (
AfterIterationHook,
BeforeIterationHook,
EvalEngine,
EvalEngineConfig,
JudgeVerdict,
ToolAuditHook,
)
from raven.eval_engine.adapter.adapter import EvalAdapter
from raven.eval_engine.judge.judge import EvalJudge
# ---------------------------------------------------------------------------
# Shared fixtures
# ---------------------------------------------------------------------------
@pytest.fixture
def ctx_with_messages():
return AgentHookContext(
session_key="cli:test",
iteration=1,
messages=[
{"role": "user", "content": "summarize this"},
{"role": "assistant", "content": "ok, summary: foo"},
],
)
# ===========================================================================
# Default config — everything must be no-op
# ===========================================================================
class TestDefaultDisabledBehavior:
def test_config_default_enabled_false(self):
cfg = EvalEngineConfig()
assert cfg.enabled is False
def test_config_subflags_have_safe_defaults(self):
cfg = EvalEngineConfig()
# on_task_completion defaults True so that flipping just
# `enabled=True` activates the most useful hook.
assert cfg.on_task_completion is True
# Expensive hooks default off.
assert cfg.on_tool_audit is False
assert cfg.on_iteration_gate is False
async def test_before_iteration_is_noop_when_disabled(self, ctx_with_messages):
hook = BeforeIterationHook(EvalEngineConfig())
decision = await hook.before_iteration(ctx_with_messages)
assert decision.short_circuit_result is None
async def test_tool_audit_is_noop_when_disabled(self, ctx_with_messages):
hook = ToolAuditHook(EvalEngineConfig(tool_denylist=["dangerous"]))
# response has a denylisted tool call but enabled=False
ctx_with_messages.response = {"tool_calls": [{"name": "dangerous", "arguments": {}}]}
decision = await hook.before_execute_tools(ctx_with_messages)
assert decision.short_circuit_result is None
# ===========================================================================
# BeforeIterationHook
# ===========================================================================
class TestBeforeIterationHook:
async def test_within_budget_passes_through(self, ctx_with_messages):
cfg = EvalEngineConfig(enabled=True, on_iteration_gate=True, max_iteration_tokens=1_000_000)
hook = BeforeIterationHook(cfg)
decision = await hook.before_iteration(ctx_with_messages)
assert decision.short_circuit_result is None
async def test_over_budget_short_circuits(self, ctx_with_messages):
cfg = EvalEngineConfig(enabled=True, on_iteration_gate=True, max_iteration_tokens=1)
hook = BeforeIterationHook(cfg)
decision = await hook.before_iteration(ctx_with_messages)
assert decision.short_circuit_result is not None
assert "budget" in str(decision.short_circuit_result).lower()
assert any("token_budget_exceeded" in n for n in decision.notes)
async def test_empty_messages_is_passthrough(self):
cfg = EvalEngineConfig(enabled=True, on_iteration_gate=True)
hook = BeforeIterationHook(cfg)
ctx = AgentHookContext(session_key="cli:test", messages=[])
decision = await hook.before_iteration(ctx)
assert decision.short_circuit_result is None
async def test_gate_off_keeps_passthrough_even_with_huge_messages(self, ctx_with_messages):
# enabled=True but gate-flag off
cfg = EvalEngineConfig(enabled=True, on_iteration_gate=False, max_iteration_tokens=1)
hook = BeforeIterationHook(cfg)
decision = await hook.before_iteration(ctx_with_messages)
assert decision.short_circuit_result is None
def test_estimator_handles_unserializable_payload(self):
# Object that defies json.dumps — fall back to 0 estimate
# rather than crashing.
cfg = EvalEngineConfig(enabled=True, on_iteration_gate=True)
hook = BeforeIterationHook(cfg)
weird = object()
# default=str in the estimator catches this; should return some int
result = hook._estimate_tokens([{"weird": weird}])
assert isinstance(result, int)
assert result >= 0
# ===========================================================================
# ToolAuditHook
# ===========================================================================
class TestToolAuditHook:
async def test_no_denylist_means_passthrough(self, ctx_with_messages):
cfg = EvalEngineConfig(enabled=True, on_tool_audit=True, tool_denylist=[])
hook = ToolAuditHook(cfg)
ctx_with_messages.response = {"tool_calls": [{"name": "anything", "arguments": {}}]}
decision = await hook.before_execute_tools(ctx_with_messages)
assert decision.short_circuit_result is None
async def test_dict_style_denied_tool_short_circuits(self, ctx_with_messages):
cfg = EvalEngineConfig(enabled=True, on_tool_audit=True, tool_denylist=["shell", "exec"])
hook = ToolAuditHook(cfg)
ctx_with_messages.response = {"tool_calls": [{"name": "shell", "arguments": {"cmd": "rm -rf /"}}]}
decision = await hook.before_execute_tools(ctx_with_messages)
assert decision.short_circuit_result is not None
assert "shell" in decision.short_circuit_result
async def test_object_style_tool_call_supported(self, ctx_with_messages):
cfg = EvalEngineConfig(enabled=True, on_tool_audit=True, tool_denylist=["dangerous"])
hook = ToolAuditHook(cfg)
class FakeCall:
name = "dangerous"
ctx_with_messages.response = MagicMock(tool_calls=[FakeCall()])
decision = await hook.before_execute_tools(ctx_with_messages)
assert decision.short_circuit_result is not None
async def test_function_nested_tool_name_supported(self, ctx_with_messages):
# OpenAI-style nested name (function.name)
cfg = EvalEngineConfig(enabled=True, on_tool_audit=True, tool_denylist=["banned"])
hook = ToolAuditHook(cfg)
ctx_with_messages.response = {"tool_calls": [{"function": {"name": "banned"}}]}
decision = await hook.before_execute_tools(ctx_with_messages)
assert decision.short_circuit_result is not None
async def test_no_response_is_passthrough(self):
cfg = EvalEngineConfig(enabled=True, on_tool_audit=True, tool_denylist=["banned"])
hook = ToolAuditHook(cfg)
ctx = AgentHookContext(session_key="cli:test") # response=None
decision = await hook.before_execute_tools(ctx)
assert decision.short_circuit_result is None
async def test_audit_off_with_denylisted_call_is_passthrough(self, ctx_with_messages):
# enabled=True but on_tool_audit=False
cfg = EvalEngineConfig(enabled=True, on_tool_audit=False, tool_denylist=["banned"])
hook = ToolAuditHook(cfg)
ctx_with_messages.response = {"tool_calls": [{"name": "banned"}]}
decision = await hook.before_execute_tools(ctx_with_messages)
assert decision.short_circuit_result is None
# ===========================================================================
# EvalJudge
# ===========================================================================
class TestEvalJudge:
async def test_completed_verdict(self):
provider = MagicMock()
provider.chat_with_retry = AsyncMock(return_value=MagicMock(content="completed"))
judge = EvalJudge(provider, model="haiku-test")
verdict = await judge.judge("did you do X?", "yes, here is X")
assert verdict is JudgeVerdict.completed
async def test_failed_verdict(self):
provider = MagicMock()
provider.chat_with_retry = AsyncMock(return_value=MagicMock(content="Verdict: failed."))
judge = EvalJudge(provider)
assert await judge.judge("X?", "I couldn't do X") is JudgeVerdict.failed
async def test_unknown_when_response_ambiguous(self):
provider = MagicMock()
provider.chat_with_retry = AsyncMock(return_value=MagicMock(content="hmmm, it's complicated"))
judge = EvalJudge(provider)
assert await judge.judge("X?", "...") is JudgeVerdict.unknown
async def test_provider_exception_returns_unknown(self):
provider = MagicMock()
provider.chat_with_retry = AsyncMock(side_effect=RuntimeError("boom"))
judge = EvalJudge(provider)
assert await judge.judge("X?", "Y") is JudgeVerdict.unknown
async def test_timeout_returns_unknown(self):
async def slow(*args: Any, **kwargs: Any):
import asyncio
await asyncio.sleep(10)
return MagicMock(content="completed")
provider = MagicMock()
provider.chat_with_retry = slow
judge = EvalJudge(provider, timeout_seconds=0.05)
assert await judge.judge("X?", "Y") is JudgeVerdict.unknown
# ===========================================================================
# EvalAdapter
# ===========================================================================
class TestEvalAdapter:
def test_completed_writes_history_entry(self):
memory = MagicMock()
adapter = EvalAdapter(
memory,
now_fn=lambda: datetime(2026, 5, 13, 10, 0),
)
adapter.record_task_completion(
JudgeVerdict.completed,
user_goal="summarize report.pdf",
session_key="cli:c1",
)
memory.append_history.assert_called_once()
entry = memory.append_history.call_args[0][0]
assert "eval verdict=completed" in entry
assert "session=cli:c1" in entry
assert "summarize report.pdf" in entry
def test_unknown_is_noop(self):
memory = MagicMock()
adapter = EvalAdapter(memory)
adapter.record_task_completion(JudgeVerdict.unknown, user_goal="x", session_key="cli:c")
memory.append_history.assert_not_called()
def test_memory_exception_is_swallowed(self):
memory = MagicMock()
memory.append_history.side_effect = RuntimeError("io error")
adapter = EvalAdapter(memory)
# Should NOT raise.
adapter.record_task_completion(JudgeVerdict.failed, user_goal="x", session_key="cli:c")
def test_truncates_long_goal(self):
memory = MagicMock()
adapter = EvalAdapter(memory)
long_goal = "x" * 500
adapter.record_task_completion(JudgeVerdict.completed, user_goal=long_goal, session_key="cli:c")
entry = memory.append_history.call_args[0][0]
# Goal got truncated to <=160 chars within the quoted segment.
quoted = entry.split('goal="', 1)[1].rsplit('"', 1)[0]
assert len(quoted) <= 160
# ===========================================================================
# AfterIterationHook
# ===========================================================================
class TestAfterIterationHook:
@pytest.fixture
def cfg_enabled(self):
return EvalEngineConfig(enabled=True, on_task_completion=True)
@pytest.fixture
def fake_judge(self):
j = MagicMock()
j.judge = AsyncMock(return_value=JudgeVerdict.completed)
return j
@pytest.fixture
def adapter(self):
return MagicMock()
async def test_completed_writes_through_adapter(self, cfg_enabled, fake_judge, adapter, ctx_with_messages):
hook = AfterIterationHook(cfg_enabled, fake_judge, adapter)
decision = await hook.after_iteration(ctx_with_messages)
adapter.record_task_completion.assert_called_once()
kwargs = adapter.record_task_completion.call_args.kwargs
assert kwargs["verdict"] is JudgeVerdict.completed
assert "summarize" in kwargs["user_goal"]
# Hook never short-circuits — evaluator must not interrupt the reply.
assert decision.short_circuit_result is None
async def test_unknown_verdict_skips_adapter(self, cfg_enabled, adapter, ctx_with_messages):
judge = MagicMock()
judge.judge = AsyncMock(return_value=JudgeVerdict.unknown)
hook = AfterIterationHook(cfg_enabled, judge, adapter)
await hook.after_iteration(ctx_with_messages)
adapter.record_task_completion.assert_not_called()
async def test_disabled_is_full_noop(self, fake_judge, adapter, ctx_with_messages):
hook = AfterIterationHook(
EvalEngineConfig(), # disabled
fake_judge,
adapter,
)
await hook.after_iteration(ctx_with_messages)
fake_judge.judge.assert_not_awaited()
adapter.record_task_completion.assert_not_called()
async def test_missing_user_or_assistant_is_noop(self, cfg_enabled, fake_judge, adapter):
hook = AfterIterationHook(cfg_enabled, fake_judge, adapter)
# No assistant message
ctx = AgentHookContext(
session_key="cli:test",
messages=[{"role": "user", "content": "hi"}],
)
await hook.after_iteration(ctx)
fake_judge.judge.assert_not_awaited()
async def test_judge_exception_recovers_as_unknown(self, cfg_enabled, adapter, ctx_with_messages):
judge = MagicMock()
judge.judge = AsyncMock(side_effect=RuntimeError("boom"))
hook = AfterIterationHook(cfg_enabled, judge, adapter)
decision = await hook.after_iteration(ctx_with_messages)
# No write through, no crash.
adapter.record_task_completion.assert_not_called()
assert decision.short_circuit_result is None
# ===========================================================================
# EvalEngine orchestrator
# ===========================================================================
class TestEvalEngineOrchestrator:
def test_default_engine_yields_three_hooks(self):
engine = EvalEngine()
hooks = engine.hooks()
assert len(hooks) == 3
# Order is canonical: before_iteration → tool_audit → after_iteration
assert isinstance(hooks[0], BeforeIterationHook)
assert isinstance(hooks[1], ToolAuditHook)
# When no MemoryEngine is wired, the after-iteration slot is a noop.
# We just assert that *some* AgentHook subclass occupies it.
from raven.contracts.loop_hooks import AgentHook
assert isinstance(hooks[2], AgentHook)
def test_engine_with_memory_engine_uses_real_after_hook(self, tmp_path):
memory = MagicMock()
engine = EvalEngine(memory=memory)
hooks = engine.hooks()
assert isinstance(hooks[2], AfterIterationHook)
def test_engine_exposes_config(self):
cfg = EvalEngineConfig(enabled=True, judge_model="custom-model")
engine = EvalEngine(cfg)
assert engine.config is cfg
assert engine.config.judge_model == "custom-model"
async def test_default_engine_hooks_are_all_noops(self, ctx_with_messages):
"""The smoke contract — mount EvalEngine.hooks() into a chain
with default config and AgentLoop behavior must be unchanged."""
from raven.agent.hook import CompositeHook
engine = EvalEngine() # default config — disabled
composite = CompositeHook(engine.hooks())
for phase in (
"before_iteration",
"before_execute_tools",
"after_iteration",
):
decision = await getattr(composite, phase)(ctx_with_messages)
assert decision.short_circuit_result is None
assert decision.modified_content is None