## Summary
Rejected tool calls now report the argument path, the expected shape and
what arrived, so the model can correct a misplaced field or a JSON
string instead of repeating the same call. Generic string casting
preserves objects and arrays for validation, and the kernel's
model-facing `ask_user` cast preserves option objects it cannot render
under its accepted field policy.
The validation and compatibility boundaries are:
- Generic validation does not execute schema-supplied `pattern`
expressions. Tool-specific validation, including the DAG parser's
patterns, remains responsible for those checks.
- Suggestions and misplaced-key searches share one budget across an
entire validation, including nested objects: 16 suggestion attempts, 256
candidate comparisons and 256 queued search nodes. Candidates are
limited to 32 names of at most 64 characters per object. Diagnostic
exhaustion suppresses optional hints, while validation continues to
report invalid arguments.
- Displayed keys, paths, individual errors and expected shapes are
bounded at 64, 160, 512 and 300 characters respectively. The rendered
diagnostic body is capped at 3,500 characters. Actual parameter lookup
uses the original keys. The registry keeps the error prefix used by
failure classification.
- Objects with nonempty `patternProperties` leave additional-property
casting and validation to the tool until those rules can be checked
together safely. Explicit properties and required fields are still
checked. This is partial schema support, not an implementation of
`patternProperties`. Array bounds apply only when they are nonnegative
integers other than booleans; malformed bounds remain unchecked.
- `image_search` retains its single-query fallback when `queries` is
empty and `query` is valid. A nonempty batch still wins. The DAG tool
normalizes aliases declared by its parser, including required-field
aliases, and rejects simultaneous alias and canonical names instead of
overwriting either. Node `skills` and `mcps` remain playbook-owned
fields, accepted by the graph parser and rejected at the model-facing
registry boundary.
- Research round-trip questions use the shared public `option_text`
renderer. A choice such as `{"label": "EU only", "id": 1}` reaches the
broker as `EU only`, and a description is retained. The kernel's
model-facing entry continues to reject unknown option fields. These are
separate acceptance policies.
- A closed MCP schema still rejects unknown fields; an open schema
forwards them. This compatibility change is deliberate. A server's
willingness to ignore fields does not change its declared schema.
`const`, `uniqueItems`, valid array bounds and additional-property
constraints without `patternProperties` remain enforced. `pattern`,
`patternProperties`, `anyOf`, `oneOf` and `format` are not fully
implemented by the generic validator. Complete bounded pattern support
and the research renderer's pre-existing fallback for objects with no
label or description remain follow-up work. The research renderer can
still discard unknown fields; this PR does not claim every product cast
is lossless.
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
After resolving the test-file conflict with main, on Linux and Python
3.12.13:
```bash
uv run --frozen --no-sync pytest -q -n 0 \
tests/test_subagent_dag_core.py tests/test_subagent_dag_guard.py \
tests/test_subagent_dag_runner.py tests/test_subagent_dag_adjudication.py \
tests/test_subagent_dag_control_tools.py tests/test_subagent_dag_verdict.py \
tests/test_tool_params_cast.py tests/test_tool_registry_execute.py \
tests/test_ask_user_tool.py tests/test_mcp_client.py tests/test_agents_research_tools.py \
tests/test_agents_research_flow_ask_user_surface.py \
tests/test_agents_research_flow_ask_user.py tests/test_agent_tools_web_providers.py \
tests/test_security_untrusted_context.py tests/test_tool_admission.py \
tests/test_tool_refusal_continuation.py tests/test_l3_open_world.py --tb=short \
--cov=raven.agent.tools.params --cov=raven.agent.tools.ask_user \
--cov=raven.agent.tools.registry --cov=raven.agent.tools.web \
--cov=raven.agent.subagent.dag_tool --cov=research_flow.tools.ask_user \
--cov-branch --cov-report=json:coverage.json --cov-report=term
uv run --frozen --no-sync python scripts/coverage_gate.py \
--coverage-json coverage.json diff --base-ref origin/main --head HEAD --threshold 90
```
- 1629 passed, with 17 existing asyncio-marker warnings. HTTP(S) and ALL
proxy variables were unset. Coverage filenames above replace the
absolute local report paths used in the run; the diff gate used the
pinned main revision.
- Diff coverage: 98.08% (306/312 executable changed lines), above the
90% gate. This is coverage of the PR's changed production lines, not a
full-suite measurement.
- Conflict resolution preserves every top-level declaration from main
and all five additional PR regression functions. AST comparison reports
no duplicate declarations. All nine commits replayed; the only
range-diff change is the insertion context of the appended DAG tests.
- Through the real registry and graph parser, a synthetic branching
graph with `nodeSummary`, `promptTemplate` and `dependsOn` retains
`routes`, `select` and `when`, both as a node list and as JSON text.
Alias/canonical collisions and playbook-only `skills`/`mcps` remain
rejected. The probe uses a synthetic home and starts no workers.
- `uv run --frozen --no-sync ruff check raven evolver agents
plugins-dist tests scripts docs-site` and the matching `ruff format
--check`: pass.
- `uv run --frozen --no-sync ty check raven evolver agents plugins-dist
scripts`, `uv run --frozen --no-sync lint-imports` and `uv run --frozen
--no-sync deptry raven`: pass. Import contracts: 10 kept, 0 broken.
- Pre-commit on all changed files, `UV_NO_SYNC=1 make check-large-files
check-source-language COMMIT_RANGE=origin/main...HEAD`, and `uv run
--frozen --no-sync python scripts/check_commit_messages.py
origin/main..HEAD`: pass.
The full repository suite and native Windows checks run in CI for the
new head. The earlier head's full-suite counts, recorded-session replays
and native Windows review are not reused as measurements of this rebase.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed (no new
documentation in this conflict-resolution round)
## Risk
Closed schemas, including MCP schemas, and kernel `ask_user` option
objects with unknown fields have a stricter acceptance boundary. The
documented aliases and image-search fallback are preserved at their own
tool boundaries. Unsupported pattern rules remain delegated to tools;
this validator is not a complete JSON Schema implementation.
The budgets limit optional diagnostic work and rendered text, not the
total time required to validate arbitrarily large input. A missing-key
hint can be omitted when its search budget is exhausted. Bounded
previews can still contain a fragment of submitted text; length limits
are not a secret-redaction guarantee.
Four nonblocking re-review items remain outside this conflict-only
update: declared node aliases are not yet normalized at the replan
entry; some documentation still describes playbook-owned node fields as
model-facing; malformed MCP `properties: null` remains unsupported; and
the empty-batch fallback does not yet cover JSON-text empty batches or
`ppt_image_search`.
No new dependency, persisted state or migration. Rollback is a revert.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
807 lines
29 KiB
Python
807 lines
29 KiB
Python
"""Tests for the ToolRegistry execute boundary (raven/agent/tools/registry.py).
|
|
|
|
``Tool.execute`` may return ``str`` or :class:`ToolResult`, but every consumer
|
|
of the registry other than the agent loop -- the sentinel action executor, the
|
|
subagent manager, the context curator, tracing -- treats the return value as
|
|
model text and puts it straight into a message, a reply or an artifact. So the
|
|
registry unwraps here and hands back a ``str`` with the display string attached.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from raven.agent.tools.filesystem import WriteFileTool
|
|
from raven.agent.tools.registry import ToolRegistry
|
|
from raven.contracts.llm_provider import RunMeta, TruncationInfo
|
|
from raven.contracts.tool import Continuation, Tool, ToolOutput, ToolResult
|
|
|
|
|
|
class _Split(Tool):
|
|
def __init__(
|
|
self,
|
|
model_text: str,
|
|
display_text: str | None,
|
|
*,
|
|
retryable: bool = True,
|
|
blocks_call: bool = False,
|
|
continuation: Continuation = Continuation.CONTINUE,
|
|
) -> None:
|
|
self._model_text = model_text
|
|
self._display_text = display_text
|
|
self._retryable = retryable
|
|
self._blocks_call = blocks_call
|
|
self._continuation = continuation
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "split"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "returns ToolResult"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}, "required": []}
|
|
|
|
async def execute(self, **kwargs) -> ToolResult:
|
|
return ToolResult(
|
|
model_text=self._model_text,
|
|
display_text=self._display_text,
|
|
retryable=self._retryable,
|
|
blocks_call=self._blocks_call,
|
|
continuation=self._continuation,
|
|
)
|
|
|
|
|
|
class _Plain(Tool):
|
|
@property
|
|
def name(self) -> str:
|
|
return "plain"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "returns a bare str"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}, "required": []}
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
return "plain output"
|
|
|
|
|
|
def _registry(*tools: Tool) -> ToolRegistry:
|
|
reg = ToolRegistry()
|
|
for tool in tools:
|
|
reg.register(tool)
|
|
return reg
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_result_is_unwrapped_to_model_text_at_the_boundary():
|
|
reg = _registry(_Split("model sentence", "question -> answer"))
|
|
|
|
result = await reg.execute("split", {})
|
|
|
|
# A str, equal to the model text -- not a dataclass, and no repr anywhere.
|
|
assert isinstance(result, str)
|
|
assert result == "model sentence"
|
|
assert "ToolResult(" not in f"{result}"
|
|
assert json.loads(json.dumps({"result": result}))["result"] == "model sentence"
|
|
# The display string rides along for the agent loop.
|
|
assert result.display_text == "question -> answer" # type: ignore[attr-defined]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_plain_string_tools_carry_no_display_text():
|
|
reg = _registry(_Plain())
|
|
|
|
result = await reg.execute("plain", {})
|
|
|
|
assert result == "plain output"
|
|
assert getattr(result, "display_text", None) is None
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_error_prefixed_tool_result_keeps_hint_and_display():
|
|
reg = _registry(_Split("Error: broker unavailable", "asked -> nothing"))
|
|
|
|
result = await reg.execute("split", {})
|
|
|
|
assert result.startswith("Error: broker unavailable")
|
|
assert "try a different approach" in result
|
|
assert result.display_text == "asked -> nothing" # type: ignore[attr-defined]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_non_retryable_error_omits_hint_and_preserves_abort_signal():
|
|
reg = _registry(
|
|
_Split(
|
|
"Error: denied by safety policy",
|
|
None,
|
|
retryable=False,
|
|
blocks_call=True,
|
|
continuation=Continuation.ABORT_TURN,
|
|
)
|
|
)
|
|
|
|
result = await reg.execute("split", {})
|
|
|
|
assert "try a different approach" not in result
|
|
assert result.retryable is False # type: ignore[attr-defined]
|
|
assert result.blocks_call is True # type: ignore[attr-defined]
|
|
assert result.continuation is Continuation.ABORT_TURN # type: ignore[attr-defined]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_missing_tool_still_returns_a_plain_string():
|
|
reg = _registry(_Plain())
|
|
|
|
result = await reg.execute("nope", {})
|
|
|
|
assert isinstance(result, str)
|
|
assert "not available" in result
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_miss_names_both_readings_and_lists_nothing():
|
|
"""The registry cannot tell a hallucinated name from one unloaded mid-turn.
|
|
|
|
Both happen: a turn's prompt is assembled while an MCP server's tools
|
|
exist, and the server can go away before the model calls one. Saying only
|
|
"not found" accused the model of guessing, which is wrong half the time and
|
|
was the whole reason an earlier revision tracked tombstones to say
|
|
otherwise. Listing the catalog was worse -- 1470 tokens for a 210-tool
|
|
deploy, repeating schemas the request already carries, kept in history for
|
|
the rest of the run.
|
|
"""
|
|
reg = _registry(_Plain())
|
|
|
|
result = await reg.execute("nope", {})
|
|
|
|
assert "may have been unloaded" in result
|
|
assert "the name may be wrong" in result
|
|
assert _Plain().name not in result
|
|
assert "try a different approach" not in result
|
|
|
|
|
|
def test_tool_output_is_a_str_subclass():
|
|
out = ToolOutput("text", "display")
|
|
|
|
assert isinstance(out, str)
|
|
assert out == "text"
|
|
assert out.display_text == "display"
|
|
# Attribute is always present, so callers can getattr without a default dance.
|
|
assert ToolOutput("text").display_text is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Truncated arguments are reported as truncation, not as a missing field
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class _NeedsPath(Tool):
|
|
"""A tool with one required parameter, plus a record of what it received."""
|
|
|
|
def __init__(self) -> None:
|
|
self.received: dict | None = None
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "write_file"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "writes a file"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {
|
|
"type": "object",
|
|
"properties": {"path": {"type": "string"}, "content": {"type": "string"}},
|
|
"required": ["path", "content"],
|
|
}
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
self.received = kwargs
|
|
return "written"
|
|
|
|
|
|
class TestUnparsableArguments:
|
|
"""A call whose arguments were not JSON must be told so.
|
|
|
|
The loop parks the raw text under ``_raw_arguments`` and flags the call with
|
|
``run_meta.arguments_repaired`` when ``json.loads`` fails; the refusal reads
|
|
the flag and quotes the parked text back.
|
|
|
|
Falling through to schema validation instead would report it as "missing
|
|
required path" -- and a caller told it forgot a field it did send re-sends
|
|
the same malformed JSON forever. One observed session burned eighteen tool
|
|
calls in this loop and the model ended up reasoning about
|
|
``_raw_arguments``, an internal key it only ever saw because we invented it.
|
|
"""
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_error_names_the_parse_failure_not_a_missing_field(self):
|
|
reg = _registry(_NeedsPath())
|
|
|
|
result = await reg.execute(
|
|
"write_file",
|
|
{"_raw_arguments": '{"path": "a.py", "content": "x'},
|
|
run_meta=RunMeta(arguments_repaired=True),
|
|
)
|
|
|
|
assert "not valid JSON" in result
|
|
assert "missing required" not in result
|
|
# The raw text comes back so the caller can see what it actually sent.
|
|
assert '{"path": "a.py"' in result
|
|
# And the internal key never appears in what the caller is asked to fix.
|
|
assert "_raw_arguments" not in result
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_genuinely_missing_field_still_reports_as_missing(self):
|
|
reg = _registry(_NeedsPath())
|
|
|
|
result = await reg.execute("write_file", {"path": "a.py"})
|
|
|
|
assert "missing required content" in result
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_internal_key_is_never_named_even_without_run_meta(self):
|
|
"""A caller that forwards a repaired call without its ``run_meta`` still
|
|
reaches validation, and the key the provider invented must not be listed
|
|
as something the model sent."""
|
|
reg = _registry(_NeedsPath())
|
|
|
|
result = await reg.execute("write_file", {"_raw_arguments": '{"path": "a.py", "content": "x'})
|
|
|
|
assert "missing required path" in result
|
|
assert "_raw_arguments" not in result
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_long_argument_blob_is_capped(self):
|
|
reg = _registry(_NeedsPath())
|
|
|
|
result = await reg.execute(
|
|
"write_file",
|
|
{"_raw_arguments": "x" * 5000},
|
|
run_meta=RunMeta(arguments_repaired=True),
|
|
)
|
|
|
|
assert result.endswith("...")
|
|
assert len(result) < 1000
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_truncated_arguments_reported_as_truncation() -> None:
|
|
"""A cut-off call names the real cause instead of the missing field.
|
|
|
|
The schema is right that something is missing; it is wrong about why.
|
|
Telling a model that wrote a path it forgot the path sends it looking for a
|
|
mistake it did not make, and its most reasonable next move is to resend the
|
|
same oversized payload.
|
|
"""
|
|
reg = ToolRegistry()
|
|
reg.register(_NeedsPath())
|
|
|
|
out = await reg.execute(
|
|
"write_file",
|
|
{"_raw_arguments": '{"content": "def foo('},
|
|
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=4096)),
|
|
)
|
|
|
|
assert "[truncated]" in out
|
|
assert "4096-token output limit" in out
|
|
assert "missing required" not in out
|
|
# flag_truncation reaches its verdict from arguments_repaired and adds
|
|
# truncation to that same run_meta, so this pairing is the ordinary streamed
|
|
# cut -- the parked text has to come back on this path as well.
|
|
assert '{"content": "def foo(' in out
|
|
# The generic hint would still point at "try a different approach", which is
|
|
# the advice that produced the retry loop.
|
|
assert "different approach" not in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_truncated_but_parseable_arguments_also_reported_as_truncation() -> None:
|
|
"""Covers the shape where the transport closed the JSON for us.
|
|
|
|
The blob parses cleanly and simply lacks whatever the model had not reached
|
|
yet, so validation fails on a genuinely absent field -- indistinguishable
|
|
from a model error unless the truncation flag is honoured.
|
|
"""
|
|
reg = ToolRegistry()
|
|
reg.register(_NeedsPath())
|
|
|
|
out = await reg.execute(
|
|
"write_file",
|
|
{"content": "def foo(): ..."},
|
|
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=8192)),
|
|
)
|
|
|
|
assert "[truncated]" in out
|
|
assert "missing required" not in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_untruncated_invalid_arguments_keep_the_schema_message() -> None:
|
|
"""A genuine model mistake must still read as one."""
|
|
reg = ToolRegistry()
|
|
reg.register(_NeedsPath())
|
|
|
|
out = await reg.execute("write_file", {"content": "hello"})
|
|
|
|
assert "Invalid parameters" in out
|
|
assert "[truncated]" not in out
|
|
assert "different approach" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_truncated_call_is_never_dispatched() -> None:
|
|
"""Not even when every required field happens to have arrived.
|
|
|
|
A truncated call is not a smaller version of itself; it can be a different
|
|
call. `mode` is optional on the real write_file, so a cut landing before it
|
|
falls back to "overwrite" and silently replaces everything an earlier
|
|
append had written -- then reports success. Validation cannot see that: the
|
|
call is well-formed. Only the fact that it was cut says otherwise.
|
|
|
|
The cost of being wrong is one retry, since a turn can end with a complete
|
|
call and be cut in the prose after it. That is cheaper than the overwrite.
|
|
"""
|
|
tool = _NeedsPath()
|
|
reg = ToolRegistry()
|
|
reg.register(tool)
|
|
|
|
out = await reg.execute(
|
|
"write_file",
|
|
{"path": "a.py", "content": "x"},
|
|
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=4096)),
|
|
)
|
|
|
|
assert tool.received is None, "a truncated call must not reach the tool"
|
|
assert "[truncated]" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_untruncated_call_still_reaches_the_tool() -> None:
|
|
"""The refusal is keyed on truncation alone, not on anything else."""
|
|
tool = _NeedsPath()
|
|
reg = ToolRegistry()
|
|
reg.register(tool)
|
|
|
|
await reg.execute("write_file", {"path": "a.py", "content": "x"})
|
|
|
|
assert tool.received == {"path": "a.py", "content": "x"}
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The advice on a truncated call comes from the tool, not from one template
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class _Appendable(_NeedsPath):
|
|
"""A tool with a smaller form to fall back to."""
|
|
|
|
@property
|
|
def truncation_hint(self) -> str:
|
|
return "Send a smaller first chunk with mode=overwrite, then append the rest with mode=append."
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_tools_own_recovery_advice_reaches_the_model() -> None:
|
|
"""Observed live: the generic "send it in smaller pieces" produced thirty-two
|
|
iterations of the same call with the content field left out. The model had
|
|
worked out that it should split the file, and still did not use the append
|
|
mode -- it is declared in the tool schema, which a model in a retry loop
|
|
never re-reads. The error message is the one text it does read every turn.
|
|
"""
|
|
reg = ToolRegistry()
|
|
reg.register(_Appendable())
|
|
|
|
out = await reg.execute(
|
|
"write_file",
|
|
{"path": "snake.py"},
|
|
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=1200)),
|
|
)
|
|
|
|
assert "[truncated]" in out
|
|
assert "mode=append" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_tool_without_advice_still_gets_the_neutral_message() -> None:
|
|
"""And nothing more: a generic "split it up" was wrong for tools like exec,
|
|
whose argument is one command and has no smaller form.
|
|
"""
|
|
reg = ToolRegistry()
|
|
reg.register(_NeedsPath())
|
|
|
|
out = await reg.execute(
|
|
"write_file",
|
|
{"path": "snake.py"},
|
|
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=1200)),
|
|
)
|
|
|
|
assert "[truncated]" in out
|
|
assert "may have been cut short" in out
|
|
# No instruction at all: the facts stand, and nothing tells this tool to do
|
|
# something it may have no way of doing.
|
|
assert "smaller pieces" not in out
|
|
assert out.rstrip().endswith("It was not run. Send it again.")
|
|
|
|
|
|
def test_shipped_tools_that_can_be_split_say_how() -> None:
|
|
"""write_file and exec are the two the incident actually cycled through."""
|
|
from raven.agent.tools.filesystem import WriteFileTool
|
|
from raven.agent.tools.shell import ExecTool
|
|
|
|
write_hint = WriteFileTool(".").truncation_hint or ""
|
|
assert "mode=append" in write_hint
|
|
exec_hint = ExecTool().truncation_hint or ""
|
|
assert "horten the command" in exec_hint
|
|
|
|
# `truncation_hint` speaks only where the upstream confirmed the cut, so it
|
|
# states rather than hedges. The hedging moved to `incomplete_hint`, which
|
|
# is read under an "If it was the output limit:" heading -- lower-case and
|
|
# continuing a sentence rather than opening one.
|
|
for hint in (write_hint, exec_hint):
|
|
assert not hint.lstrip().startswith("If "), "no condition where the upstream was explicit"
|
|
for tool in (WriteFileTool("."), ExecTool()):
|
|
incomplete = tool.incomplete_hint or ""
|
|
assert incomplete and incomplete[0].islower(), "reads as the consequent of a condition"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_message_does_not_tell_the_model_to_withhold_the_content() -> None:
|
|
"""It used to say "do not resend the same content", and that was wrong.
|
|
|
|
Nothing past the cut is saved: the upstream drops the unfinished key, so
|
|
the call that reaches the tool carries only the fields that completed, and
|
|
the conversation history stores that same stub. The content genuinely has
|
|
to come again. Observed live: the model followed the old sentence exactly
|
|
-- forty turns of `write_file({"path": ...})` with the content withheld.
|
|
"""
|
|
reg = ToolRegistry()
|
|
reg.register(_NeedsPath())
|
|
|
|
out = await reg.execute(
|
|
"write_file",
|
|
{"path": "snake.py"},
|
|
run_meta=RunMeta(truncation=TruncationInfo(at_tokens=1200)),
|
|
)
|
|
|
|
assert "may have been cut short" in out
|
|
assert "Send it again" in out
|
|
assert "not resend" not in out.lower()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unparsed_earlier_call_is_told_it_is_bad_json() -> None:
|
|
"""Calls arrived after it, so a cut cannot explain it -- and the advice for
|
|
a cut would be wrong."""
|
|
registry = ToolRegistry()
|
|
registry.register(WriteFileTool(workspace=Path(".")))
|
|
|
|
text = await registry.execute(
|
|
"write_file", {"path": "a.py", "content": "x"}, run_meta=RunMeta(arguments_repaired=True)
|
|
)
|
|
|
|
assert "invalid arguments" in text
|
|
assert "well-formed arguments" in text
|
|
assert "mode=append" not in text, "splitting it up is not the fix here"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_ambiguous_refusal_never_asserts_that_a_cut_happened() -> None:
|
|
"""`truncation_hint` is advice for a turn that ran out of room, so it is
|
|
only ever attached when the upstream said so. Here nothing did: the
|
|
arguments failed to parse and no call followed, which a cut explains and
|
|
plain malformed JSON explains equally well.
|
|
|
|
Acting on truncation advice after writing bad JSON sends a model hunting
|
|
for a size problem it does not have, so both readings are offered, each
|
|
under its own condition, and the message says outright that nothing here
|
|
tells them apart. Longer is acceptable; asserting is not.
|
|
"""
|
|
registry = ToolRegistry()
|
|
registry.register(WriteFileTool(workspace=Path(".")))
|
|
|
|
text = await registry.execute(
|
|
"write_file",
|
|
{"path": "a.py", "content": "x"},
|
|
run_meta=RunMeta(arguments_repaired=True, last_of_turn=True),
|
|
)
|
|
|
|
assert "[incomplete arguments]" in text
|
|
assert "[truncated]" not in text, "that marker is the upstream's to earn"
|
|
assert "output token limit" in text, "the reading is named"
|
|
assert "tells the two apart" in text, "and disclaimed"
|
|
for branch in ("If it was the output limit", "If the arguments were malformed"):
|
|
assert branch in text, f"missing the {branch!r} branch"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_each_case_carries_its_own_tool_advice() -> None:
|
|
"""A certain cut and an ambiguous one are different situations, so a tool
|
|
speaks to each in its own words rather than one string doing double duty."""
|
|
registry = ToolRegistry()
|
|
registry.register(WriteFileTool(workspace=Path(".")))
|
|
|
|
certain = await registry.execute(
|
|
"write_file", {"path": "a.py", "content": "x"}, run_meta=RunMeta(truncation=TruncationInfo(at_tokens=8192))
|
|
)
|
|
ambiguous = await registry.execute(
|
|
"write_file", {"path": "a.py", "content": "x"}, run_meta=RunMeta(arguments_repaired=True, last_of_turn=True)
|
|
)
|
|
|
|
assert "[truncated]" in certain and "8192-token" in certain
|
|
assert "mode=append" in certain and "mode=append" in ambiguous
|
|
assert "If it was the output limit" not in certain, "no hedging where the upstream was explicit"
|
|
|
|
|
|
def test_a_tool_that_speaks_to_one_case_speaks_to_both() -> None:
|
|
"""The two hints say nearly the same thing in different frames, which is
|
|
exactly how they drift apart. A tool that answers one and not the other
|
|
leaves the ambiguous refusal with no way forward -- the case where the
|
|
upstream lied, which is the one this machinery exists for."""
|
|
from raven.agent.tools.filesystem import EditFileTool, ListDirTool, ReadFileTool
|
|
from raven.agent.tools.shell import ExecTool
|
|
|
|
tools = [
|
|
WriteFileTool(workspace=Path(".")),
|
|
ReadFileTool(workspace=Path(".")),
|
|
EditFileTool(workspace=Path(".")),
|
|
ListDirTool(workspace=Path(".")),
|
|
ExecTool(working_dir="."),
|
|
]
|
|
mismatched = [type(t).__name__ for t in tools if bool(t.truncation_hint) != bool(t.incomplete_hint)]
|
|
|
|
assert not mismatched, "these answer one case and not the other: " + ", ".join(mismatched)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_registry_built_with_no_gates_casts_nothing_and_serves_as_before():
|
|
"""[seam-1 A5] The gate seam's absence pin: default construction casts no
|
|
gate, so the execute path every other test in this file covers IS the
|
|
no-gate path -- byte-parity with a registry that never heard of gates is
|
|
what this file keeps proving. The empty tuple is the whole of the new
|
|
state, and there is no way to cast one after construction.
|
|
"""
|
|
reg = _registry(_Plain())
|
|
|
|
assert reg._tool_gates == ()
|
|
assert reg.tool_gates == ()
|
|
with pytest.raises(AttributeError):
|
|
reg.tool_gates = ("late",) # type: ignore[misc]
|
|
assert await reg.execute("plain", {}) == "plain output"
|
|
|
|
|
|
class TestWhetherOneCallWentWrong:
|
|
"""`call_failed` is one rule with two halves, and each half answers a case
|
|
the other cannot. It was being copied inline before it was a function."""
|
|
|
|
def test_a_tool_own_verdict_is_read_even_when_the_text_reads_fine(self):
|
|
"""A policy refusal carries `ok=False` and says why in ordinary prose.
|
|
Reading only the leading word would call it a success."""
|
|
from raven.agent.tools.registry import call_failed
|
|
from raven.contracts.tool import ToolOutput
|
|
|
|
assert call_failed(ToolOutput("The reviewer declined this change.", ok=False)) is True
|
|
|
|
def test_a_bare_string_is_read_for_the_failure_text(self):
|
|
"""`execute` returns a plain string on the paths that refuse before
|
|
dispatch -- unparseable arguments, an invalid parameter set, a timeout.
|
|
There is no object to carry `ok`, and the leading `Error` is the whole of
|
|
the signal. Reading only `ok` calls every one of them a success."""
|
|
from raven.agent.tools.registry import call_failed
|
|
|
|
assert call_failed("Error: Tool 'sleeper' timed out after 600s.") is True
|
|
|
|
def test_a_call_that_worked_is_not_a_failure(self):
|
|
from raven.agent.tools.registry import call_failed
|
|
from raven.contracts.tool import ToolOutput
|
|
|
|
assert call_failed(ToolOutput("wrote 12 lines", ok=True)) is False
|
|
assert call_failed("wrote 12 lines") is False
|
|
|
|
|
|
class _Asks(Tool):
|
|
"""A nested schema shaped like ask_user's, so an error can sit inside an item."""
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "asks"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "asks questions"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {
|
|
"type": "object",
|
|
"properties": {
|
|
"questions": {
|
|
"type": "array",
|
|
"items": {
|
|
"type": "object",
|
|
"properties": {
|
|
"question": {"type": "string"},
|
|
"options": {"type": "array", "items": {"type": "string"}},
|
|
},
|
|
"required": ["question"],
|
|
},
|
|
}
|
|
},
|
|
"required": ["questions"],
|
|
}
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
return "asked"
|
|
|
|
|
|
class _Many(Tool):
|
|
"""Twelve required parameters, for a rejection longer than anyone should read."""
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "many"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "needs a lot"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
keys = [f"k{i}" for i in range(12)]
|
|
return {"type": "object", "properties": {k: {"type": "string"} for k in keys}, "required": keys}
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
return "done"
|
|
|
|
|
|
class TestHowAnInvalidCallIsReported:
|
|
"""The rejection is the only thing the model learns from a call it got wrong."""
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_several_errors_are_listed_one_per_line(self):
|
|
out = await _registry(_NeedsPath()).execute("write_file", {})
|
|
|
|
assert out.startswith("Error: Invalid parameters for tool 'write_file':")
|
|
assert "\n- missing required path" in out
|
|
assert "\n- missing required content" in out
|
|
assert "try a different approach" in out
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_lone_error_stays_on_the_first_line(self):
|
|
"""A single problem needs no list: the error stays one line."""
|
|
out = await _registry(_NeedsPath()).execute("write_file", {"path": "a.py"})
|
|
|
|
assert out.splitlines()[0] == "Error: Invalid parameters for tool 'write_file': missing required content"
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_long_list_is_cut_with_a_count_of_what_was_left_out(self):
|
|
out = await _registry(_Many()).execute("many", {})
|
|
|
|
listed = [line for line in out.splitlines() if line.startswith("- missing required")]
|
|
assert len(listed) == 8
|
|
assert "4 more" in out
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_error_inside_an_item_shows_what_that_item_should_look_like(self):
|
|
"""Live ask_user: the model knew the schema and still nested the question in
|
|
its options. The level it lost track of is the one to show."""
|
|
out = await _registry(_Asks()).execute("asks", {"questions": [{"options": [{"question": "A?"}]}]})
|
|
|
|
shape = [line for line in out.splitlines() if line.startswith("Expected questions[0]:")]
|
|
assert len(shape) == 1
|
|
assert "question: string" in shape[0]
|
|
assert "options?" in shape[0]
|
|
assert "string[]" in shape[0]
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_error_at_the_top_level_shows_no_shape(self):
|
|
"""The whole schema is already in the tool definition the model called."""
|
|
out = await _registry(_NeedsPath()).execute("write_file", {"path": "a.py"})
|
|
|
|
assert "missing required content" in out
|
|
assert "Expected" not in out
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_listed_rejection_still_counts_toward_the_same_streak(self):
|
|
"""`failure_class` reads the first 200 characters for the prefix; a layout
|
|
that pushed it further would split one repeated mistake into new ones."""
|
|
from raven.agent.loop.failure_streak import failure_class
|
|
|
|
out = await _registry(_Many()).execute("many", {})
|
|
|
|
assert failure_class(out) == "schema"
|
|
|
|
|
|
class _SchemaProbe(_Plain):
|
|
"""A real registry dispatch with a supplied schema and no external effects."""
|
|
|
|
def __init__(self, schema: dict) -> None:
|
|
self._schema = schema
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return self._schema
|
|
|
|
|
|
@pytest.mark.slow
|
|
def test_a_server_pattern_does_not_block_the_registry_event_loop() -> None:
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
import textwrap
|
|
|
|
script = textwrap.dedent("""
|
|
import asyncio, json, time
|
|
from raven.agent.tools.registry import ToolRegistry
|
|
from raven.contracts.tool import Tool
|
|
|
|
class Probe(Tool):
|
|
name = "pattern_probe"
|
|
description = "A schema-only probe."
|
|
timeout_seconds = 0.01
|
|
parameters = {"type": "object", "properties": {
|
|
"slug": {"type": "string", "pattern": "^([a-z0-9]+-?)+$"}}}
|
|
async def execute(self, **kwargs):
|
|
return "accepted"
|
|
|
|
async def main():
|
|
registry = ToolRegistry()
|
|
registry.register(Probe())
|
|
await registry.execute("pattern_probe", {"slug": "warmup"})
|
|
started = time.monotonic()
|
|
async def heartbeat():
|
|
await asyncio.sleep(0)
|
|
return time.monotonic() - started
|
|
beat = asyncio.create_task(heartbeat())
|
|
result = await registry.execute("pattern_probe", {"slug": "a" * 32 + "_"})
|
|
print(json.dumps({"result": result, "heartbeat": await beat}))
|
|
asyncio.run(main())
|
|
""")
|
|
env = {**os.environ, "PYTHONPATH": os.pathsep.join(map(str, sys.path))}
|
|
result = subprocess.run(
|
|
[sys._base_executable, "-c", script],
|
|
cwd=Path(__file__).resolve().parent.parent,
|
|
env=env,
|
|
capture_output=True,
|
|
text=True,
|
|
timeout=15,
|
|
check=True,
|
|
)
|
|
observed = json.loads(result.stdout)
|
|
assert observed["result"] == "accepted"
|
|
assert observed["heartbeat"] < 0.5
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_invalid_array_bound_does_not_turn_a_call_into_an_execution_error():
|
|
schema = {"type": "object", "properties": {"items": {"type": "array", "minItems": "1"}}}
|
|
out = await _registry(_SchemaProbe(schema)).execute("plain", {"items": []})
|
|
assert out == "plain output"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_repeated_long_paths_keep_the_registry_rejection_bounded():
|
|
from raven.agent.loop.failure_streak import failure_class
|
|
|
|
keys = [f"k{i}" for i in range(25)]
|
|
schema = {"type": "object", "required": keys, "properties": {k: {"type": "integer"} for k in keys}}
|
|
key = "nested_" + "z" * 2000
|
|
out = await _registry(_SchemaProbe(schema)).execute("plain", {key: dict.fromkeys(keys, 1)})
|
|
assert out.startswith("Error: Invalid parameters for tool 'plain':")
|
|
assert len(out) <= 4096
|
|
assert "missing required k0" in out and "nested_" in out
|
|
assert key not in out and "..." in out
|
|
assert failure_class(out) == "schema"
|