Raven-1/tests/test_tool_registry_execute.py
Dizhan Xue a8455bae5e
fix(agent): say what arrived when a tool call's arguments are rejected (#985)
## Summary

Rejected tool calls now report the argument path, the expected shape and
what arrived, so the model can correct a misplaced field or a JSON
string instead of repeating the same call. Generic string casting
preserves objects and arrays for validation, and the kernel's
model-facing `ask_user` cast preserves option objects it cannot render
under its accepted field policy.

The validation and compatibility boundaries are:

- Generic validation does not execute schema-supplied `pattern`
expressions. Tool-specific validation, including the DAG parser's
patterns, remains responsible for those checks.
- Suggestions and misplaced-key searches share one budget across an
entire validation, including nested objects: 16 suggestion attempts, 256
candidate comparisons and 256 queued search nodes. Candidates are
limited to 32 names of at most 64 characters per object. Diagnostic
exhaustion suppresses optional hints, while validation continues to
report invalid arguments.
- Displayed keys, paths, individual errors and expected shapes are
bounded at 64, 160, 512 and 300 characters respectively. The rendered
diagnostic body is capped at 3,500 characters. Actual parameter lookup
uses the original keys. The registry keeps the error prefix used by
failure classification.
- Objects with nonempty `patternProperties` leave additional-property
casting and validation to the tool until those rules can be checked
together safely. Explicit properties and required fields are still
checked. This is partial schema support, not an implementation of
`patternProperties`. Array bounds apply only when they are nonnegative
integers other than booleans; malformed bounds remain unchecked.
- `image_search` retains its single-query fallback when `queries` is
empty and `query` is valid. A nonempty batch still wins. The DAG tool
normalizes aliases declared by its parser, including required-field
aliases, and rejects simultaneous alias and canonical names instead of
overwriting either. Node `skills` and `mcps` remain playbook-owned
fields, accepted by the graph parser and rejected at the model-facing
registry boundary.
- Research round-trip questions use the shared public `option_text`
renderer. A choice such as `{"label": "EU only", "id": 1}` reaches the
broker as `EU only`, and a description is retained. The kernel's
model-facing entry continues to reject unknown option fields. These are
separate acceptance policies.
- A closed MCP schema still rejects unknown fields; an open schema
forwards them. This compatibility change is deliberate. A server's
willingness to ignore fields does not change its declared schema.

`const`, `uniqueItems`, valid array bounds and additional-property
constraints without `patternProperties` remain enforced. `pattern`,
`patternProperties`, `anyOf`, `oneOf` and `format` are not fully
implemented by the generic validator. Complete bounded pattern support
and the research renderer's pre-existing fallback for objects with no
label or description remain follow-up work. The research renderer can
still discard unknown fields; this PR does not claim every product cast
is lossless.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

After resolving the test-file conflict with main, on Linux and Python
3.12.13:

```bash
uv run --frozen --no-sync pytest -q -n 0 \
  tests/test_subagent_dag_core.py tests/test_subagent_dag_guard.py \
  tests/test_subagent_dag_runner.py tests/test_subagent_dag_adjudication.py \
  tests/test_subagent_dag_control_tools.py tests/test_subagent_dag_verdict.py \
  tests/test_tool_params_cast.py tests/test_tool_registry_execute.py \
  tests/test_ask_user_tool.py tests/test_mcp_client.py tests/test_agents_research_tools.py \
  tests/test_agents_research_flow_ask_user_surface.py \
  tests/test_agents_research_flow_ask_user.py tests/test_agent_tools_web_providers.py \
  tests/test_security_untrusted_context.py tests/test_tool_admission.py \
  tests/test_tool_refusal_continuation.py tests/test_l3_open_world.py --tb=short \
  --cov=raven.agent.tools.params --cov=raven.agent.tools.ask_user \
  --cov=raven.agent.tools.registry --cov=raven.agent.tools.web \
  --cov=raven.agent.subagent.dag_tool --cov=research_flow.tools.ask_user \
  --cov-branch --cov-report=json:coverage.json --cov-report=term
uv run --frozen --no-sync python scripts/coverage_gate.py \
  --coverage-json coverage.json diff --base-ref origin/main --head HEAD --threshold 90
```

- 1629 passed, with 17 existing asyncio-marker warnings. HTTP(S) and ALL
proxy variables were unset. Coverage filenames above replace the
absolute local report paths used in the run; the diff gate used the
pinned main revision.
- Diff coverage: 98.08% (306/312 executable changed lines), above the
90% gate. This is coverage of the PR's changed production lines, not a
full-suite measurement.
- Conflict resolution preserves every top-level declaration from main
and all five additional PR regression functions. AST comparison reports
no duplicate declarations. All nine commits replayed; the only
range-diff change is the insertion context of the appended DAG tests.
- Through the real registry and graph parser, a synthetic branching
graph with `nodeSummary`, `promptTemplate` and `dependsOn` retains
`routes`, `select` and `when`, both as a node list and as JSON text.
Alias/canonical collisions and playbook-only `skills`/`mcps` remain
rejected. The probe uses a synthetic home and starts no workers.
- `uv run --frozen --no-sync ruff check raven evolver agents
plugins-dist tests scripts docs-site` and the matching `ruff format
--check`: pass.
- `uv run --frozen --no-sync ty check raven evolver agents plugins-dist
scripts`, `uv run --frozen --no-sync lint-imports` and `uv run --frozen
--no-sync deptry raven`: pass. Import contracts: 10 kept, 0 broken.
- Pre-commit on all changed files, `UV_NO_SYNC=1 make check-large-files
check-source-language COMMIT_RANGE=origin/main...HEAD`, and `uv run
--frozen --no-sync python scripts/check_commit_messages.py
origin/main..HEAD`: pass.

The full repository suite and native Windows checks run in CI for the
new head. The earlier head's full-suite counts, recorded-session replays
and native Windows review are not reused as measurements of this rebase.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed (no new
documentation in this conflict-resolution round)

## Risk

Closed schemas, including MCP schemas, and kernel `ask_user` option
objects with unknown fields have a stricter acceptance boundary. The
documented aliases and image-search fallback are preserved at their own
tool boundaries. Unsupported pattern rules remain delegated to tools;
this validator is not a complete JSON Schema implementation.

The budgets limit optional diagnostic work and rendered text, not the
total time required to validate arbitrarily large input. A missing-key
hint can be omitted when its search budget is exhausted. Bounded
previews can still contain a fragment of submitted text; length limits
are not a secret-redaction guarantee.

Four nonblocking re-review items remain outside this conflict-only
update: declared node aliases are not yet normalized at the replan
entry; some documentation still describes playbook-owned node fields as
model-facing; malformed MCP `properties: null` remains unsupported; and
the empty-batch fallback does not yet cover JSON-text empty batches or
`ppt_image_search`.

No new dependency, persisted state or migration. Rollback is a revert.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-10-19 17:37:25 +08:00

807 lines
29 KiB
Python

"""Tests for the ToolRegistry execute boundary (raven/agent/tools/registry.py).
``Tool.execute`` may return ``str`` or :class:`ToolResult`, but every consumer
of the registry other than the agent loop -- the sentinel action executor, the
subagent manager, the context curator, tracing -- treats the return value as
model text and puts it straight into a message, a reply or an artifact. So the
registry unwraps here and hands back a ``str`` with the display string attached.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from raven.agent.tools.filesystem import WriteFileTool
from raven.agent.tools.registry import ToolRegistry
from raven.contracts.llm_provider import RunMeta, TruncationInfo
from raven.contracts.tool import Continuation, Tool, ToolOutput, ToolResult
class _Split(Tool):
def __init__(
self,
model_text: str,
display_text: str | None,
*,
retryable: bool = True,
blocks_call: bool = False,
continuation: Continuation = Continuation.CONTINUE,
) -> None:
self._model_text = model_text
self._display_text = display_text
self._retryable = retryable
self._blocks_call = blocks_call
self._continuation = continuation
@property
def name(self) -> str:
return "split"
@property
def description(self) -> str:
return "returns ToolResult"
@property
def parameters(self) -> dict:
return {"type": "object", "properties": {}, "required": []}
async def execute(self, **kwargs) -> ToolResult:
return ToolResult(
model_text=self._model_text,
display_text=self._display_text,
retryable=self._retryable,
blocks_call=self._blocks_call,
continuation=self._continuation,
)
class _Plain(Tool):
@property
def name(self) -> str:
return "plain"
@property
def description(self) -> str:
return "returns a bare str"
@property
def parameters(self) -> dict:
return {"type": "object", "properties": {}, "required": []}
async def execute(self, **kwargs) -> str:
return "plain output"
def _registry(*tools: Tool) -> ToolRegistry:
reg = ToolRegistry()
for tool in tools:
reg.register(tool)
return reg
@pytest.mark.asyncio
async def test_tool_result_is_unwrapped_to_model_text_at_the_boundary():
reg = _registry(_Split("model sentence", "question -> answer"))
result = await reg.execute("split", {})
# A str, equal to the model text -- not a dataclass, and no repr anywhere.
assert isinstance(result, str)
assert result == "model sentence"
assert "ToolResult(" not in f"{result}"
assert json.loads(json.dumps({"result": result}))["result"] == "model sentence"
# The display string rides along for the agent loop.
assert result.display_text == "question -> answer" # type: ignore[attr-defined]
@pytest.mark.asyncio
async def test_plain_string_tools_carry_no_display_text():
reg = _registry(_Plain())
result = await reg.execute("plain", {})
assert result == "plain output"
assert getattr(result, "display_text", None) is None
@pytest.mark.asyncio
async def test_error_prefixed_tool_result_keeps_hint_and_display():
reg = _registry(_Split("Error: broker unavailable", "asked -> nothing"))
result = await reg.execute("split", {})
assert result.startswith("Error: broker unavailable")
assert "try a different approach" in result
assert result.display_text == "asked -> nothing" # type: ignore[attr-defined]
@pytest.mark.asyncio
async def test_non_retryable_error_omits_hint_and_preserves_abort_signal():
reg = _registry(
_Split(
"Error: denied by safety policy",
None,
retryable=False,
blocks_call=True,
continuation=Continuation.ABORT_TURN,
)
)
result = await reg.execute("split", {})
assert "try a different approach" not in result
assert result.retryable is False # type: ignore[attr-defined]
assert result.blocks_call is True # type: ignore[attr-defined]
assert result.continuation is Continuation.ABORT_TURN # type: ignore[attr-defined]
@pytest.mark.asyncio
async def test_missing_tool_still_returns_a_plain_string():
reg = _registry(_Plain())
result = await reg.execute("nope", {})
assert isinstance(result, str)
assert "not available" in result
@pytest.mark.asyncio
async def test_a_miss_names_both_readings_and_lists_nothing():
"""The registry cannot tell a hallucinated name from one unloaded mid-turn.
Both happen: a turn's prompt is assembled while an MCP server's tools
exist, and the server can go away before the model calls one. Saying only
"not found" accused the model of guessing, which is wrong half the time and
was the whole reason an earlier revision tracked tombstones to say
otherwise. Listing the catalog was worse -- 1470 tokens for a 210-tool
deploy, repeating schemas the request already carries, kept in history for
the rest of the run.
"""
reg = _registry(_Plain())
result = await reg.execute("nope", {})
assert "may have been unloaded" in result
assert "the name may be wrong" in result
assert _Plain().name not in result
assert "try a different approach" not in result
def test_tool_output_is_a_str_subclass():
out = ToolOutput("text", "display")
assert isinstance(out, str)
assert out == "text"
assert out.display_text == "display"
# Attribute is always present, so callers can getattr without a default dance.
assert ToolOutput("text").display_text is None
# ---------------------------------------------------------------------------
# Truncated arguments are reported as truncation, not as a missing field
# ---------------------------------------------------------------------------
class _NeedsPath(Tool):
"""A tool with one required parameter, plus a record of what it received."""
def __init__(self) -> None:
self.received: dict | None = None
@property
def name(self) -> str:
return "write_file"
@property
def description(self) -> str:
return "writes a file"
@property
def parameters(self) -> dict:
return {
"type": "object",
"properties": {"path": {"type": "string"}, "content": {"type": "string"}},
"required": ["path", "content"],
}
async def execute(self, **kwargs) -> str:
self.received = kwargs
return "written"
class TestUnparsableArguments:
"""A call whose arguments were not JSON must be told so.
The loop parks the raw text under ``_raw_arguments`` and flags the call with
``run_meta.arguments_repaired`` when ``json.loads`` fails; the refusal reads
the flag and quotes the parked text back.
Falling through to schema validation instead would report it as "missing
required path" -- and a caller told it forgot a field it did send re-sends
the same malformed JSON forever. One observed session burned eighteen tool
calls in this loop and the model ended up reasoning about
``_raw_arguments``, an internal key it only ever saw because we invented it.
"""
@pytest.mark.asyncio
async def test_the_error_names_the_parse_failure_not_a_missing_field(self):
reg = _registry(_NeedsPath())
result = await reg.execute(
"write_file",
{"_raw_arguments": '{"path": "a.py", "content": "x'},
run_meta=RunMeta(arguments_repaired=True),
)
assert "not valid JSON" in result
assert "missing required" not in result
# The raw text comes back so the caller can see what it actually sent.
assert '{"path": "a.py"' in result
# And the internal key never appears in what the caller is asked to fix.
assert "_raw_arguments" not in result
@pytest.mark.asyncio
async def test_a_genuinely_missing_field_still_reports_as_missing(self):
reg = _registry(_NeedsPath())
result = await reg.execute("write_file", {"path": "a.py"})
assert "missing required content" in result
@pytest.mark.asyncio
async def test_the_internal_key_is_never_named_even_without_run_meta(self):
"""A caller that forwards a repaired call without its ``run_meta`` still
reaches validation, and the key the provider invented must not be listed
as something the model sent."""
reg = _registry(_NeedsPath())
result = await reg.execute("write_file", {"_raw_arguments": '{"path": "a.py", "content": "x'})
assert "missing required path" in result
assert "_raw_arguments" not in result
@pytest.mark.asyncio
async def test_a_long_argument_blob_is_capped(self):
reg = _registry(_NeedsPath())
result = await reg.execute(
"write_file",
{"_raw_arguments": "x" * 5000},
run_meta=RunMeta(arguments_repaired=True),
)
assert result.endswith("...")
assert len(result) < 1000
@pytest.mark.asyncio
async def test_truncated_arguments_reported_as_truncation() -> None:
"""A cut-off call names the real cause instead of the missing field.
The schema is right that something is missing; it is wrong about why.
Telling a model that wrote a path it forgot the path sends it looking for a
mistake it did not make, and its most reasonable next move is to resend the
same oversized payload.
"""
reg = ToolRegistry()
reg.register(_NeedsPath())
out = await reg.execute(
"write_file",
{"_raw_arguments": '{"content": "def foo('},
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=4096)),
)
assert "[truncated]" in out
assert "4096-token output limit" in out
assert "missing required" not in out
# flag_truncation reaches its verdict from arguments_repaired and adds
# truncation to that same run_meta, so this pairing is the ordinary streamed
# cut -- the parked text has to come back on this path as well.
assert '{"content": "def foo(' in out
# The generic hint would still point at "try a different approach", which is
# the advice that produced the retry loop.
assert "different approach" not in out
@pytest.mark.asyncio
async def test_truncated_but_parseable_arguments_also_reported_as_truncation() -> None:
"""Covers the shape where the transport closed the JSON for us.
The blob parses cleanly and simply lacks whatever the model had not reached
yet, so validation fails on a genuinely absent field -- indistinguishable
from a model error unless the truncation flag is honoured.
"""
reg = ToolRegistry()
reg.register(_NeedsPath())
out = await reg.execute(
"write_file",
{"content": "def foo(): ..."},
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=8192)),
)
assert "[truncated]" in out
assert "missing required" not in out
@pytest.mark.asyncio
async def test_untruncated_invalid_arguments_keep_the_schema_message() -> None:
"""A genuine model mistake must still read as one."""
reg = ToolRegistry()
reg.register(_NeedsPath())
out = await reg.execute("write_file", {"content": "hello"})
assert "Invalid parameters" in out
assert "[truncated]" not in out
assert "different approach" in out
@pytest.mark.asyncio
async def test_a_truncated_call_is_never_dispatched() -> None:
"""Not even when every required field happens to have arrived.
A truncated call is not a smaller version of itself; it can be a different
call. `mode` is optional on the real write_file, so a cut landing before it
falls back to "overwrite" and silently replaces everything an earlier
append had written -- then reports success. Validation cannot see that: the
call is well-formed. Only the fact that it was cut says otherwise.
The cost of being wrong is one retry, since a turn can end with a complete
call and be cut in the prose after it. That is cheaper than the overwrite.
"""
tool = _NeedsPath()
reg = ToolRegistry()
reg.register(tool)
out = await reg.execute(
"write_file",
{"path": "a.py", "content": "x"},
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=4096)),
)
assert tool.received is None, "a truncated call must not reach the tool"
assert "[truncated]" in out
@pytest.mark.asyncio
async def test_an_untruncated_call_still_reaches_the_tool() -> None:
"""The refusal is keyed on truncation alone, not on anything else."""
tool = _NeedsPath()
reg = ToolRegistry()
reg.register(tool)
await reg.execute("write_file", {"path": "a.py", "content": "x"})
assert tool.received == {"path": "a.py", "content": "x"}
# ---------------------------------------------------------------------------
# The advice on a truncated call comes from the tool, not from one template
# ---------------------------------------------------------------------------
class _Appendable(_NeedsPath):
"""A tool with a smaller form to fall back to."""
@property
def truncation_hint(self) -> str:
return "Send a smaller first chunk with mode=overwrite, then append the rest with mode=append."
@pytest.mark.asyncio
async def test_a_tools_own_recovery_advice_reaches_the_model() -> None:
"""Observed live: the generic "send it in smaller pieces" produced thirty-two
iterations of the same call with the content field left out. The model had
worked out that it should split the file, and still did not use the append
mode -- it is declared in the tool schema, which a model in a retry loop
never re-reads. The error message is the one text it does read every turn.
"""
reg = ToolRegistry()
reg.register(_Appendable())
out = await reg.execute(
"write_file",
{"path": "snake.py"},
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=1200)),
)
assert "[truncated]" in out
assert "mode=append" in out
@pytest.mark.asyncio
async def test_a_tool_without_advice_still_gets_the_neutral_message() -> None:
"""And nothing more: a generic "split it up" was wrong for tools like exec,
whose argument is one command and has no smaller form.
"""
reg = ToolRegistry()
reg.register(_NeedsPath())
out = await reg.execute(
"write_file",
{"path": "snake.py"},
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=1200)),
)
assert "[truncated]" in out
assert "may have been cut short" in out
# No instruction at all: the facts stand, and nothing tells this tool to do
# something it may have no way of doing.
assert "smaller pieces" not in out
assert out.rstrip().endswith("It was not run. Send it again.")
def test_shipped_tools_that_can_be_split_say_how() -> None:
"""write_file and exec are the two the incident actually cycled through."""
from raven.agent.tools.filesystem import WriteFileTool
from raven.agent.tools.shell import ExecTool
write_hint = WriteFileTool(".").truncation_hint or ""
assert "mode=append" in write_hint
exec_hint = ExecTool().truncation_hint or ""
assert "horten the command" in exec_hint
# `truncation_hint` speaks only where the upstream confirmed the cut, so it
# states rather than hedges. The hedging moved to `incomplete_hint`, which
# is read under an "If it was the output limit:" heading -- lower-case and
# continuing a sentence rather than opening one.
for hint in (write_hint, exec_hint):
assert not hint.lstrip().startswith("If "), "no condition where the upstream was explicit"
for tool in (WriteFileTool("."), ExecTool()):
incomplete = tool.incomplete_hint or ""
assert incomplete and incomplete[0].islower(), "reads as the consequent of a condition"
@pytest.mark.asyncio
async def test_the_message_does_not_tell_the_model_to_withhold_the_content() -> None:
"""It used to say "do not resend the same content", and that was wrong.
Nothing past the cut is saved: the upstream drops the unfinished key, so
the call that reaches the tool carries only the fields that completed, and
the conversation history stores that same stub. The content genuinely has
to come again. Observed live: the model followed the old sentence exactly
-- forty turns of `write_file({"path": ...})` with the content withheld.
"""
reg = ToolRegistry()
reg.register(_NeedsPath())
out = await reg.execute(
"write_file",
{"path": "snake.py"},
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=1200)),
)
assert "may have been cut short" in out
assert "Send it again" in out
assert "not resend" not in out.lower()
@pytest.mark.asyncio
async def test_an_unparsed_earlier_call_is_told_it_is_bad_json() -> None:
"""Calls arrived after it, so a cut cannot explain it -- and the advice for
a cut would be wrong."""
registry = ToolRegistry()
registry.register(WriteFileTool(workspace=Path(".")))
text = await registry.execute(
"write_file", {"path": "a.py", "content": "x"}, run_meta=RunMeta(arguments_repaired=True)
)
assert "invalid arguments" in text
assert "well-formed arguments" in text
assert "mode=append" not in text, "splitting it up is not the fix here"
@pytest.mark.asyncio
async def test_an_ambiguous_refusal_never_asserts_that_a_cut_happened() -> None:
"""`truncation_hint` is advice for a turn that ran out of room, so it is
only ever attached when the upstream said so. Here nothing did: the
arguments failed to parse and no call followed, which a cut explains and
plain malformed JSON explains equally well.
Acting on truncation advice after writing bad JSON sends a model hunting
for a size problem it does not have, so both readings are offered, each
under its own condition, and the message says outright that nothing here
tells them apart. Longer is acceptable; asserting is not.
"""
registry = ToolRegistry()
registry.register(WriteFileTool(workspace=Path(".")))
text = await registry.execute(
"write_file",
{"path": "a.py", "content": "x"},
run_meta=RunMeta(arguments_repaired=True, last_of_turn=True),
)
assert "[incomplete arguments]" in text
assert "[truncated]" not in text, "that marker is the upstream's to earn"
assert "output token limit" in text, "the reading is named"
assert "tells the two apart" in text, "and disclaimed"
for branch in ("If it was the output limit", "If the arguments were malformed"):
assert branch in text, f"missing the {branch!r} branch"
@pytest.mark.asyncio
async def test_each_case_carries_its_own_tool_advice() -> None:
"""A certain cut and an ambiguous one are different situations, so a tool
speaks to each in its own words rather than one string doing double duty."""
registry = ToolRegistry()
registry.register(WriteFileTool(workspace=Path(".")))
certain = await registry.execute(
"write_file", {"path": "a.py", "content": "x"}, run_meta=RunMeta(truncation=TruncationInfo(at_tokens=8192))
)
ambiguous = await registry.execute(
"write_file", {"path": "a.py", "content": "x"}, run_meta=RunMeta(arguments_repaired=True, last_of_turn=True)
)
assert "[truncated]" in certain and "8192-token" in certain
assert "mode=append" in certain and "mode=append" in ambiguous
assert "If it was the output limit" not in certain, "no hedging where the upstream was explicit"
def test_a_tool_that_speaks_to_one_case_speaks_to_both() -> None:
"""The two hints say nearly the same thing in different frames, which is
exactly how they drift apart. A tool that answers one and not the other
leaves the ambiguous refusal with no way forward -- the case where the
upstream lied, which is the one this machinery exists for."""
from raven.agent.tools.filesystem import EditFileTool, ListDirTool, ReadFileTool
from raven.agent.tools.shell import ExecTool
tools = [
WriteFileTool(workspace=Path(".")),
ReadFileTool(workspace=Path(".")),
EditFileTool(workspace=Path(".")),
ListDirTool(workspace=Path(".")),
ExecTool(working_dir="."),
]
mismatched = [type(t).__name__ for t in tools if bool(t.truncation_hint) != bool(t.incomplete_hint)]
assert not mismatched, "these answer one case and not the other: " + ", ".join(mismatched)
@pytest.mark.asyncio
async def test_a_registry_built_with_no_gates_casts_nothing_and_serves_as_before():
"""[seam-1 A5] The gate seam's absence pin: default construction casts no
gate, so the execute path every other test in this file covers IS the
no-gate path -- byte-parity with a registry that never heard of gates is
what this file keeps proving. The empty tuple is the whole of the new
state, and there is no way to cast one after construction.
"""
reg = _registry(_Plain())
assert reg._tool_gates == ()
assert reg.tool_gates == ()
with pytest.raises(AttributeError):
reg.tool_gates = ("late",) # type: ignore[misc]
assert await reg.execute("plain", {}) == "plain output"
class TestWhetherOneCallWentWrong:
"""`call_failed` is one rule with two halves, and each half answers a case
the other cannot. It was being copied inline before it was a function."""
def test_a_tool_own_verdict_is_read_even_when_the_text_reads_fine(self):
"""A policy refusal carries `ok=False` and says why in ordinary prose.
Reading only the leading word would call it a success."""
from raven.agent.tools.registry import call_failed
from raven.contracts.tool import ToolOutput
assert call_failed(ToolOutput("The reviewer declined this change.", ok=False)) is True
def test_a_bare_string_is_read_for_the_failure_text(self):
"""`execute` returns a plain string on the paths that refuse before
dispatch -- unparseable arguments, an invalid parameter set, a timeout.
There is no object to carry `ok`, and the leading `Error` is the whole of
the signal. Reading only `ok` calls every one of them a success."""
from raven.agent.tools.registry import call_failed
assert call_failed("Error: Tool 'sleeper' timed out after 600s.") is True
def test_a_call_that_worked_is_not_a_failure(self):
from raven.agent.tools.registry import call_failed
from raven.contracts.tool import ToolOutput
assert call_failed(ToolOutput("wrote 12 lines", ok=True)) is False
assert call_failed("wrote 12 lines") is False
class _Asks(Tool):
"""A nested schema shaped like ask_user's, so an error can sit inside an item."""
@property
def name(self) -> str:
return "asks"
@property
def description(self) -> str:
return "asks questions"
@property
def parameters(self) -> dict:
return {
"type": "object",
"properties": {
"questions": {
"type": "array",
"items": {
"type": "object",
"properties": {
"question": {"type": "string"},
"options": {"type": "array", "items": {"type": "string"}},
},
"required": ["question"],
},
}
},
"required": ["questions"],
}
async def execute(self, **kwargs) -> str:
return "asked"
class _Many(Tool):
"""Twelve required parameters, for a rejection longer than anyone should read."""
@property
def name(self) -> str:
return "many"
@property
def description(self) -> str:
return "needs a lot"
@property
def parameters(self) -> dict:
keys = [f"k{i}" for i in range(12)]
return {"type": "object", "properties": {k: {"type": "string"} for k in keys}, "required": keys}
async def execute(self, **kwargs) -> str:
return "done"
class TestHowAnInvalidCallIsReported:
"""The rejection is the only thing the model learns from a call it got wrong."""
@pytest.mark.asyncio
async def test_several_errors_are_listed_one_per_line(self):
out = await _registry(_NeedsPath()).execute("write_file", {})
assert out.startswith("Error: Invalid parameters for tool 'write_file':")
assert "\n- missing required path" in out
assert "\n- missing required content" in out
assert "try a different approach" in out
@pytest.mark.asyncio
async def test_a_lone_error_stays_on_the_first_line(self):
"""A single problem needs no list: the error stays one line."""
out = await _registry(_NeedsPath()).execute("write_file", {"path": "a.py"})
assert out.splitlines()[0] == "Error: Invalid parameters for tool 'write_file': missing required content"
@pytest.mark.asyncio
async def test_a_long_list_is_cut_with_a_count_of_what_was_left_out(self):
out = await _registry(_Many()).execute("many", {})
listed = [line for line in out.splitlines() if line.startswith("- missing required")]
assert len(listed) == 8
assert "4 more" in out
@pytest.mark.asyncio
async def test_an_error_inside_an_item_shows_what_that_item_should_look_like(self):
"""Live ask_user: the model knew the schema and still nested the question in
its options. The level it lost track of is the one to show."""
out = await _registry(_Asks()).execute("asks", {"questions": [{"options": [{"question": "A?"}]}]})
shape = [line for line in out.splitlines() if line.startswith("Expected questions[0]:")]
assert len(shape) == 1
assert "question: string" in shape[0]
assert "options?" in shape[0]
assert "string[]" in shape[0]
@pytest.mark.asyncio
async def test_an_error_at_the_top_level_shows_no_shape(self):
"""The whole schema is already in the tool definition the model called."""
out = await _registry(_NeedsPath()).execute("write_file", {"path": "a.py"})
assert "missing required content" in out
assert "Expected" not in out
@pytest.mark.asyncio
async def test_a_listed_rejection_still_counts_toward_the_same_streak(self):
"""`failure_class` reads the first 200 characters for the prefix; a layout
that pushed it further would split one repeated mistake into new ones."""
from raven.agent.loop.failure_streak import failure_class
out = await _registry(_Many()).execute("many", {})
assert failure_class(out) == "schema"
class _SchemaProbe(_Plain):
"""A real registry dispatch with a supplied schema and no external effects."""
def __init__(self, schema: dict) -> None:
self._schema = schema
@property
def parameters(self) -> dict:
return self._schema
@pytest.mark.slow
def test_a_server_pattern_does_not_block_the_registry_event_loop() -> None:
import os
import subprocess
import sys
import textwrap
script = textwrap.dedent("""
import asyncio, json, time
from raven.agent.tools.registry import ToolRegistry
from raven.contracts.tool import Tool
class Probe(Tool):
name = "pattern_probe"
description = "A schema-only probe."
timeout_seconds = 0.01
parameters = {"type": "object", "properties": {
"slug": {"type": "string", "pattern": "^([a-z0-9]+-?)+$"}}}
async def execute(self, **kwargs):
return "accepted"
async def main():
registry = ToolRegistry()
registry.register(Probe())
await registry.execute("pattern_probe", {"slug": "warmup"})
started = time.monotonic()
async def heartbeat():
await asyncio.sleep(0)
return time.monotonic() - started
beat = asyncio.create_task(heartbeat())
result = await registry.execute("pattern_probe", {"slug": "a" * 32 + "_"})
print(json.dumps({"result": result, "heartbeat": await beat}))
asyncio.run(main())
""")
env = {**os.environ, "PYTHONPATH": os.pathsep.join(map(str, sys.path))}
result = subprocess.run(
[sys._base_executable, "-c", script],
cwd=Path(__file__).resolve().parent.parent,
env=env,
capture_output=True,
text=True,
timeout=15,
check=True,
)
observed = json.loads(result.stdout)
assert observed["result"] == "accepted"
assert observed["heartbeat"] < 0.5
@pytest.mark.asyncio
async def test_an_invalid_array_bound_does_not_turn_a_call_into_an_execution_error():
schema = {"type": "object", "properties": {"items": {"type": "array", "minItems": "1"}}}
out = await _registry(_SchemaProbe(schema)).execute("plain", {"items": []})
assert out == "plain output"
@pytest.mark.asyncio
async def test_repeated_long_paths_keep_the_registry_rejection_bounded():
from raven.agent.loop.failure_streak import failure_class
keys = [f"k{i}" for i in range(25)]
schema = {"type": "object", "required": keys, "properties": {k: {"type": "integer"} for k in keys}}
key = "nested_" + "z" * 2000
out = await _registry(_SchemaProbe(schema)).execute("plain", {key: dict.fromkeys(keys, 1)})
assert out.startswith("Error: Invalid parameters for tool 'plain':")
assert len(out) <= 4096
assert "missing required k0" in out and "nested_" in out
assert key not in out and "..." in out
assert failure_class(out) == "schema"