## Summary
Rejected tool calls now report the argument path, the expected shape and
what arrived, so the model can correct a misplaced field or a JSON
string instead of repeating the same call. Generic string casting
preserves objects and arrays for validation, and the kernel's
model-facing `ask_user` cast preserves option objects it cannot render
under its accepted field policy.
The validation and compatibility boundaries are:
- Generic validation does not execute schema-supplied `pattern`
expressions. Tool-specific validation, including the DAG parser's
patterns, remains responsible for those checks.
- Suggestions and misplaced-key searches share one budget across an
entire validation, including nested objects: 16 suggestion attempts, 256
candidate comparisons and 256 queued search nodes. Candidates are
limited to 32 names of at most 64 characters per object. Diagnostic
exhaustion suppresses optional hints, while validation continues to
report invalid arguments.
- Displayed keys, paths, individual errors and expected shapes are
bounded at 64, 160, 512 and 300 characters respectively. The rendered
diagnostic body is capped at 3,500 characters. Actual parameter lookup
uses the original keys. The registry keeps the error prefix used by
failure classification.
- Objects with nonempty `patternProperties` leave additional-property
casting and validation to the tool until those rules can be checked
together safely. Explicit properties and required fields are still
checked. This is partial schema support, not an implementation of
`patternProperties`. Array bounds apply only when they are nonnegative
integers other than booleans; malformed bounds remain unchecked.
- `image_search` retains its single-query fallback when `queries` is
empty and `query` is valid. A nonempty batch still wins. The DAG tool
normalizes aliases declared by its parser, including required-field
aliases, and rejects simultaneous alias and canonical names instead of
overwriting either. Node `skills` and `mcps` remain playbook-owned
fields, accepted by the graph parser and rejected at the model-facing
registry boundary.
- Research round-trip questions use the shared public `option_text`
renderer. A choice such as `{"label": "EU only", "id": 1}` reaches the
broker as `EU only`, and a description is retained. The kernel's
model-facing entry continues to reject unknown option fields. These are
separate acceptance policies.
- A closed MCP schema still rejects unknown fields; an open schema
forwards them. This compatibility change is deliberate. A server's
willingness to ignore fields does not change its declared schema.
`const`, `uniqueItems`, valid array bounds and additional-property
constraints without `patternProperties` remain enforced. `pattern`,
`patternProperties`, `anyOf`, `oneOf` and `format` are not fully
implemented by the generic validator. Complete bounded pattern support
and the research renderer's pre-existing fallback for objects with no
label or description remain follow-up work. The research renderer can
still discard unknown fields; this PR does not claim every product cast
is lossless.
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
After resolving the test-file conflict with main, on Linux and Python
3.12.13:
```bash
uv run --frozen --no-sync pytest -q -n 0 \
tests/test_subagent_dag_core.py tests/test_subagent_dag_guard.py \
tests/test_subagent_dag_runner.py tests/test_subagent_dag_adjudication.py \
tests/test_subagent_dag_control_tools.py tests/test_subagent_dag_verdict.py \
tests/test_tool_params_cast.py tests/test_tool_registry_execute.py \
tests/test_ask_user_tool.py tests/test_mcp_client.py tests/test_agents_research_tools.py \
tests/test_agents_research_flow_ask_user_surface.py \
tests/test_agents_research_flow_ask_user.py tests/test_agent_tools_web_providers.py \
tests/test_security_untrusted_context.py tests/test_tool_admission.py \
tests/test_tool_refusal_continuation.py tests/test_l3_open_world.py --tb=short \
--cov=raven.agent.tools.params --cov=raven.agent.tools.ask_user \
--cov=raven.agent.tools.registry --cov=raven.agent.tools.web \
--cov=raven.agent.subagent.dag_tool --cov=research_flow.tools.ask_user \
--cov-branch --cov-report=json:coverage.json --cov-report=term
uv run --frozen --no-sync python scripts/coverage_gate.py \
--coverage-json coverage.json diff --base-ref origin/main --head HEAD --threshold 90
```
- 1629 passed, with 17 existing asyncio-marker warnings. HTTP(S) and ALL
proxy variables were unset. Coverage filenames above replace the
absolute local report paths used in the run; the diff gate used the
pinned main revision.
- Diff coverage: 98.08% (306/312 executable changed lines), above the
90% gate. This is coverage of the PR's changed production lines, not a
full-suite measurement.
- Conflict resolution preserves every top-level declaration from main
and all five additional PR regression functions. AST comparison reports
no duplicate declarations. All nine commits replayed; the only
range-diff change is the insertion context of the appended DAG tests.
- Through the real registry and graph parser, a synthetic branching
graph with `nodeSummary`, `promptTemplate` and `dependsOn` retains
`routes`, `select` and `when`, both as a node list and as JSON text.
Alias/canonical collisions and playbook-only `skills`/`mcps` remain
rejected. The probe uses a synthetic home and starts no workers.
- `uv run --frozen --no-sync ruff check raven evolver agents
plugins-dist tests scripts docs-site` and the matching `ruff format
--check`: pass.
- `uv run --frozen --no-sync ty check raven evolver agents plugins-dist
scripts`, `uv run --frozen --no-sync lint-imports` and `uv run --frozen
--no-sync deptry raven`: pass. Import contracts: 10 kept, 0 broken.
- Pre-commit on all changed files, `UV_NO_SYNC=1 make check-large-files
check-source-language COMMIT_RANGE=origin/main...HEAD`, and `uv run
--frozen --no-sync python scripts/check_commit_messages.py
origin/main..HEAD`: pass.
The full repository suite and native Windows checks run in CI for the
new head. The earlier head's full-suite counts, recorded-session replays
and native Windows review are not reused as measurements of this rebase.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed (no new
documentation in this conflict-resolution round)
## Risk
Closed schemas, including MCP schemas, and kernel `ask_user` option
objects with unknown fields have a stricter acceptance boundary. The
documented aliases and image-search fallback are preserved at their own
tool boundaries. Unsupported pattern rules remain delegated to tools;
this validator is not a complete JSON Schema implementation.
The budgets limit optional diagnostic work and rendered text, not the
total time required to validate arbitrarily large input. A missing-key
hint can be omitted when its search budget is exhausted. Bounded
previews can still contain a fragment of submitted text; length limits
are not a secret-redaction guarantee.
Four nonblocking re-review items remain outside this conflict-only
update: declared node aliases are not yet normalized at the replan
entry; some documentation still describes playbook-owned node fields as
model-facing; malformed MCP `properties: null` remains unsupported; and
the empty-batch fallback does not yet cover JSON-text empty batches or
`ppt_image_search`.
No new dependency, persisted state or migration. Rollback is a revert.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
1106 lines
43 KiB
Python
1106 lines
43 KiB
Python
"""Tests for the ask_user tool's display contract (raven/agent/tools/ask_user.py).
|
|
|
|
``ask_user`` is the first tool to return :class:`ToolResult`, so this pins the
|
|
split it introduces: ``model_text`` keeps the natural-language phrasing the
|
|
model reads, ``display_text`` carries the question/answer pairing the transcript
|
|
renders, and ``display_call`` labels the row with the question rather than the
|
|
raw arguments blob.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import copy
|
|
import json
|
|
|
|
import pytest
|
|
|
|
from raven.agent.loop.bundles import ToolWiring
|
|
from raven.agent.tools.ask_user import (
|
|
_MAX_JSON_LAYERS,
|
|
AskUserTool,
|
|
_normalize_options,
|
|
_normalize_questions,
|
|
)
|
|
from raven.agent.tools.registry import ToolRegistry
|
|
from raven.contracts.tool import ToolResult
|
|
|
|
|
|
class _StubBroker:
|
|
"""Stands in for QuestionBroker: replies from a scripted answer map."""
|
|
|
|
default_timeout_s = 600.0
|
|
|
|
def __init__(self, answers: dict[str, str], *, delay_s: float = 0.0) -> None:
|
|
self.answers = answers
|
|
self.asked: list[tuple[str, list[str]]] = []
|
|
self.calls: list[dict] = []
|
|
self._delay_s = delay_s
|
|
|
|
async def await_question(self, cid: str, *, prompt: str, choices: list[str], **kwargs) -> str:
|
|
self.asked.append((prompt, choices))
|
|
self.calls.append({"prompt": prompt, "choices": choices, **kwargs})
|
|
if self._delay_s:
|
|
await asyncio.sleep(self._delay_s)
|
|
return self.answers.get(prompt, "")
|
|
|
|
|
|
def _tool(answers: dict[str, str], **kw) -> tuple[AskUserTool, _StubBroker]:
|
|
broker = _StubBroker(answers, **kw)
|
|
tool = AskUserTool(broker=broker, conversation_id="tui:test") # type: ignore[arg-type]
|
|
return tool, broker
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_single_question_splits_model_and_display_text():
|
|
tool, broker = _tool({"Which package manager?": "uv"})
|
|
|
|
result = await tool.execute(questions=[{"question": "Which package manager?", "options": ["uv", "pip"]}])
|
|
|
|
assert isinstance(result, ToolResult)
|
|
# The model reads the full sentence, including the question it asked.
|
|
assert 'User answered: "Which package manager?" -> "uv".' in result.model_text
|
|
assert result.model_text.endswith("Continue.")
|
|
# A single question needs no pairing in the transcript; the row already
|
|
# shows the question via display_call.
|
|
assert result.display_text == "answered: uv"
|
|
assert broker.asked == [("Which package manager?", ["uv", "pip"])]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_batch_pairs_each_question_with_its_answer():
|
|
tool, _ = _tool({"Base branch?": "main", "Squash?": "yes"})
|
|
|
|
result = await tool.execute(questions=[{"question": "Base branch?"}, {"question": "Squash?"}])
|
|
|
|
assert isinstance(result, ToolResult)
|
|
# With several questions the display text must say which answer belongs to
|
|
# which question -- one line per pair, order preserved.
|
|
assert result.display_text == "Base branch? -> main\nSquash? -> yes"
|
|
assert 'User answered: "Base branch?" -> "main".' in result.model_text
|
|
assert 'User answered: "Squash?" -> "yes".' in result.model_text
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_unanswered_question_is_explicit_in_both_texts():
|
|
tool, _ = _tool({})
|
|
|
|
result = await tool.execute(questions=[{"question": "Ship it?"}])
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert "did not answer" in result.model_text
|
|
assert result.display_text == "answered: (no answer)"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unanswered_question_is_recorded_on_the_turn():
|
|
"""The model sentence is not the session's record of the question."""
|
|
from raven.permissions.turn import start_permission_turn
|
|
|
|
turn = start_permission_turn(None, conversation_id="tui:test", turn_id="t")
|
|
tool, _ = _tool({})
|
|
|
|
await tool.execute(questions=[{"question": "Ship it?"}, {"question": "Which branch?"}])
|
|
|
|
assert [item.question for item in turn.unanswered] == ["Ship it?", "Which branch?"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_answered_question_is_not_recorded_as_unanswered():
|
|
from raven.permissions.turn import start_permission_turn
|
|
|
|
turn = start_permission_turn(None, conversation_id="tui:test", turn_id="t")
|
|
tool, _ = _tool({"Ship it?": "yes"})
|
|
|
|
await tool.execute(questions=[{"question": "Ship it?"}])
|
|
|
|
assert turn.unanswered == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_call_with_no_broker_records_the_normalized_questions():
|
|
"""One-shot never wires a broker, so the call returns at that guard.
|
|
|
|
The questions that would have been asked are still the run's questions.
|
|
A blank one is dropped, a JSON string is read, and a call ``_prepare``
|
|
rejects is not a question anyone failed to answer.
|
|
"""
|
|
from raven.permissions.turn import start_permission_turn
|
|
|
|
turn = start_permission_turn(None, conversation_id="tui:test", turn_id="t")
|
|
tool = AskUserTool(broker=None, conversation_id="tui:test")
|
|
|
|
missing = await tool.execute(
|
|
questions='[{"question": " Which base branch? "}, {"question": " "}, {"question": "Ship it?"}]'
|
|
)
|
|
assert missing == "Error: ask_user not configured (no question broker)"
|
|
assert [item.question for item in turn.unanswered] == ["Which base branch?", "Ship it?"]
|
|
|
|
turn.unanswered.clear()
|
|
rejected = await tool.execute(questions=[{"question": "Only one?", "options": ["yes"]}])
|
|
assert rejected == "Error: ask_user not configured (no question broker)"
|
|
assert turn.unanswered == []
|
|
|
|
bare = AskUserTool(broker=object(), conversation_id="") # type: ignore[arg-type]
|
|
no_cid = await bare.execute(questions=[{"question": "Where?"}])
|
|
assert no_cid == "Error: ask_user has no conversation context"
|
|
assert [item.question for item in turn.unanswered] == ["Where?"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_error_paths_return_plain_strings():
|
|
# No broker / no conversation id / no questions predate ToolResult and stay
|
|
# bare strings, so the loop's str branch still has to work.
|
|
tool = AskUserTool(broker=None, conversation_id="tui:test")
|
|
assert await tool.execute(questions=[{"question": "hi"}]) == "Error: ask_user not configured (no question broker)"
|
|
|
|
tool, _ = _tool({})
|
|
assert await tool.execute(questions=[]) == "Error: ask_user requires at least one question"
|
|
assert await tool.execute(questions=[{"question": " "}]) == (
|
|
"Error: ask_user requires at least one non-empty question"
|
|
)
|
|
|
|
|
|
def test_display_call_labels_the_row_with_the_question():
|
|
tool, _ = _tool({})
|
|
|
|
assert tool.display_call({"questions": [{"question": "Which base branch?"}]}) == "Which base branch?"
|
|
assert tool.display_call({"questions": [{"question": "Base?"}, {"question": "Squash?"}]}) == "Base? | Squash?"
|
|
# Nothing worth showing falls back to the UI's generic preview.
|
|
assert tool.display_call({"questions": []}) is None
|
|
assert tool.display_call({"questions": [{"question": " "}]}) is None
|
|
assert tool.display_call({}) is None
|
|
|
|
|
|
def test_display_call_survives_a_json_encoded_questions_argument():
|
|
# Models emit the array as a JSON string often enough that display_call --
|
|
# which only labels a transcript row -- used to take the whole turn down
|
|
# with an AttributeError raised on a character of that string.
|
|
tool, _ = _tool({})
|
|
|
|
assert tool.display_call({"questions": '[{"question": "Base?"}]'}) == "Base?"
|
|
assert tool.display_call({"questions": '[{"question": "Base?"}, {"question": "Squash?"}]'}) == "Base? | Squash?"
|
|
|
|
|
|
def test_display_call_returns_none_for_unreadable_arguments():
|
|
tool, _ = _tool({})
|
|
|
|
assert tool.display_call({"questions": "not json at all"}) is None
|
|
assert tool.display_call({"questions": 42}) is None
|
|
assert tool.display_call({"questions": ["plain string", {"question": "Base?"}]}) == "Base?"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_execute_accepts_a_json_encoded_questions_argument():
|
|
# The same coercion has to reach execute: a non-empty string is truthy, so
|
|
# the empty-questions guard passed it straight through to entry.get().
|
|
tool, broker = _tool({"Which base branch?": "main"})
|
|
|
|
result = await tool.execute(questions='[{"question": "Which base branch?", "options": ["main", "develop"]}]')
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert broker.asked == [("Which base branch?", ["main", "develop"])]
|
|
assert result.display_text == "answered: main"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_execute_rejects_unreadable_questions_instead_of_raising():
|
|
tool, _ = _tool({})
|
|
|
|
assert await tool.execute(questions="not json at all") == (
|
|
"Error: ask_user questions is not valid JSON: Expecting value: line 1 column 1 (char 0)"
|
|
)
|
|
assert await tool.execute(questions=42) == "Error: ask_user requires at least one question"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_deeply_nested_json_does_not_kill_the_turn():
|
|
# json.loads answers deep nesting with RecursionError, which is not a
|
|
# ValueError -- so a normalizer that catches only TypeError/ValueError lets
|
|
# it escape by the exact route the normalizer exists to close.
|
|
tool, _ = _tool({})
|
|
payload = "[" * 20_000 + "]" * 20_000
|
|
|
|
assert tool.display_call({"questions": payload}) is None
|
|
assert await tool.execute(questions=payload) == "Error: ask_user requires at least one question"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_json_encoded_options_reach_the_user_as_options():
|
|
# options carries the same declared array type as questions and arrives as
|
|
# a JSON string just as often. Iterating that string offered the user one
|
|
# suggested answer per character.
|
|
tool, broker = _tool({"Which base?": "main"})
|
|
|
|
await tool.execute(questions=[{"question": "Which base?", "options": '["main", "develop"]'}])
|
|
|
|
assert broker.asked == [("Which base?", ["main", "develop"])]
|
|
|
|
|
|
def test_a_lone_option_string_is_one_option_not_its_letters():
|
|
"""Asserted on the normalizer, not through execute: upstream's contract
|
|
refuses a one-option question outright, so a lone string never reaches a
|
|
human either way. What must not happen is it arriving as three options.
|
|
"""
|
|
assert _normalize_options("yes") == ["yes"]
|
|
assert _normalize_options('["yes", "no"]') == ["yes", "no"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_absent_or_unusable_options_are_simply_empty():
|
|
tool, broker = _tool({"Q1?": "a", "Q2?": "b"})
|
|
|
|
await tool.execute(questions=[{"question": "Q1?"}, {"question": "Q2?", "options": None}])
|
|
|
|
assert broker.asked == [("Q1?", []), ("Q2?", [])]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_questions_written_as_plain_strings_still_reach_the_user():
|
|
# A list with nothing object-shaped in it is a model that wrote the
|
|
# questions as strings. Dropping them answered a perfectly clear question
|
|
# with "requires at least one question" and asked the user nothing.
|
|
tool, broker = _tool({"Which base branch?": "main"})
|
|
|
|
result = await tool.execute(questions=["Which base branch?"])
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert broker.asked == [("Which base branch?", [])]
|
|
assert tool.display_call({"questions": ["Which base branch?"]}) == "Which base branch?"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_object_entry_still_wins_over_a_loose_string():
|
|
# The mixed list keeps its old meaning: the objects are the questions and
|
|
# the loose string is noise, not a third question.
|
|
tool, broker = _tool({"Base?": "main"})
|
|
|
|
await tool.execute(questions=["noise", {"question": "Base?"}])
|
|
|
|
assert broker.asked == [("Base?", [])]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_twice_encoded_argument_is_unwrapped():
|
|
tool, broker = _tool({"Base?": "main"})
|
|
|
|
await tool.execute(questions=json.dumps(json.dumps([{"question": "Base?"}])))
|
|
|
|
assert broker.asked == [("Base?", [])]
|
|
|
|
|
|
def test_unwrapping_is_bounded():
|
|
# Past a couple of layers this is no longer a quirk to absorb, and the
|
|
# bound is what stops a crafted argument from spending the turn unwrapping.
|
|
payload = json.dumps([{"question": "Base?"}])
|
|
for _ in range(_MAX_JSON_LAYERS + 1):
|
|
payload = json.dumps(payload)
|
|
|
|
assert _normalize_questions(payload) == []
|
|
|
|
|
|
# --- through ToolRegistry, the path production actually takes -------------
|
|
#
|
|
# Everything above drives ``tool.execute`` directly, which is the function's
|
|
# contract but not the call chain. ``ToolRegistry.execute`` casts and validates
|
|
# against the declared schema first and returns the error without dispatching,
|
|
# so a coercion that lives only in ``execute`` never runs on the shapes that
|
|
# need it. These go through the registry.
|
|
|
|
|
|
def _registered(answers: dict[str, str]) -> tuple[ToolRegistry, _StubBroker]:
|
|
tool, broker = _tool(answers)
|
|
registry = ToolRegistry()
|
|
registry.register(tool)
|
|
return registry, broker
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"questions",
|
|
[
|
|
pytest.param('[{"question": "Which base branch?"}]', id="array-as-json-string"),
|
|
pytest.param(["Which base branch?"], id="entry-as-plain-string"),
|
|
pytest.param({"question": "Which base branch?"}, id="single-object-unwrapped"),
|
|
pytest.param(json.dumps(json.dumps([{"question": "Which base branch?"}])), id="twice-encoded"),
|
|
],
|
|
)
|
|
async def test_the_registry_path_asks_rather_than_rejecting(questions):
|
|
registry, broker = _registered({"Which base branch?": "main"})
|
|
|
|
out = await registry.execute("ask_user", {"questions": questions})
|
|
|
|
assert broker.asked == [("Which base branch?", [])], out
|
|
assert "Invalid parameters" not in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_registry_path_normalizes_options_too():
|
|
registry, broker = _registered({"Which base?": "main"})
|
|
|
|
out = await registry.execute(
|
|
"ask_user", {"questions": [{"question": "Which base?", "options": '["main", "develop"]'}]}
|
|
)
|
|
|
|
assert broker.asked == [("Which base?", ["main", "develop"])], out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_registry_path_reports_unreadable_input_in_the_tools_own_words():
|
|
registry, _ = _registered({})
|
|
|
|
out = await registry.execute("ask_user", {"questions": "not json at all"})
|
|
|
|
assert "questions is not valid JSON: Expecting value: line 1 column 1 (char 0)" in out
|
|
assert "should be array" not in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_registry_path_does_not_rewrite_the_callers_arguments():
|
|
"""Normalizing must leave the caller's dict alone.
|
|
|
|
The same `arguments` object the registry is handed also goes to the START
|
|
tool event and, on the assistant message, through `openai_tool_call`. Both
|
|
happen before `tools.execute` today, so an in-place edit could not reach
|
|
them -- but the object is shared, `cast_params` is the only thing standing
|
|
between the model's text and a rewrite of it, and nothing was watching:
|
|
dropping the copy leaves the whole suite green.
|
|
"""
|
|
registry, broker = _registered({"Which base?": "main"})
|
|
args = {"questions": [{"question": "Which base?", "options": '["main", "develop"]'}]}
|
|
frozen = copy.deepcopy(args)
|
|
|
|
await registry.execute("ask_user", args)
|
|
|
|
assert broker.asked == [("Which base?", ["main", "develop"])]
|
|
assert args == frozen
|
|
|
|
|
|
def test_schema_advertises_only_fields_the_tool_reads():
|
|
"""Every declared field must reach the broker; a decorative one misleads.
|
|
|
|
``multiple`` and ``custom`` were declared and never read, so a model that
|
|
asked for a multi-select got a single-select and never learned why.
|
|
"""
|
|
tool, _ = _tool({})
|
|
entry = tool.parameters["properties"]["questions"]["items"]["properties"]
|
|
|
|
assert "multiple" not in entry
|
|
assert "custom" not in entry
|
|
assert set(entry) == {"question", "header", "options", "recommended", "multi_select"}
|
|
assert tool.parameters["properties"]["questions"]["maxItems"] == 4
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_header_and_batch_position_reach_the_broker():
|
|
tool, broker = _tool({"Base?": "main", "Squash?": "yes"})
|
|
|
|
await tool.execute(
|
|
questions=[
|
|
{"question": "Base?", "header": "Base"},
|
|
{"question": "Squash?", "header": "Squash"},
|
|
]
|
|
)
|
|
|
|
assert [c["header"] for c in broker.calls] == ["Base", "Squash"]
|
|
assert [c["index"] for c in broker.calls] == [0, 1]
|
|
assert [c["total"] for c in broker.calls] == [2, 2]
|
|
# Every round-trip carries the whole batch so a surface can render the set
|
|
# and its progress while still collecting one answer at a time.
|
|
assert broker.calls[0]["batch"] == [
|
|
{"question": "Base?", "header": "Base", "choices": [], "recommended": "", "multi_select": False},
|
|
{"question": "Squash?", "header": "Squash", "choices": [], "recommended": "", "multi_select": False},
|
|
]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_multi_select_reaches_the_broker_and_the_batch_entry():
|
|
tool, broker = _tool({"Toppings?": "cheese, olives"})
|
|
|
|
await tool.execute(
|
|
questions=[{"question": "Toppings?", "options": ["cheese", "olives", "ham"], "multi_select": True}]
|
|
)
|
|
|
|
assert broker.calls[0]["multi_select"] is True
|
|
assert broker.calls[0]["batch"] == [
|
|
{
|
|
"question": "Toppings?",
|
|
"header": "",
|
|
"choices": ["cheese", "olives", "ham"],
|
|
"recommended": "",
|
|
"multi_select": True,
|
|
}
|
|
]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_multi_select_is_forced_false_without_options():
|
|
"""multi_select only means something alongside a choice list; a model that
|
|
sets it on a free-form question must not have it reach the broker as True."""
|
|
tool, broker = _tool({"Why?": "because"})
|
|
|
|
await tool.execute(questions=[{"question": "Why?", "multi_select": True}])
|
|
|
|
assert broker.calls[0]["multi_select"] is False
|
|
assert broker.calls[0]["batch"][0]["multi_select"] is False
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_batch_shares_one_deadline_instead_of_one_each():
|
|
"""N questions must not cost N full timeouts."""
|
|
tool, broker = _tool({"a?": "1", "b?": "2"}, delay_s=0.05)
|
|
broker.default_timeout_s = 1.0
|
|
|
|
await tool.execute(questions=[{"question": "a?"}, {"question": "b?"}])
|
|
|
|
first, second = broker.calls[0]["timeout_s"], broker.calls[1]["timeout_s"]
|
|
assert first <= 1.0
|
|
assert second < first, "the second question must inherit what the first left"
|
|
assert second > 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_single_option_question_is_rejected_without_asking_anyone():
|
|
tool, broker = _tool({})
|
|
|
|
result = await tool.execute(questions=[{"question": "Ship it?", "options": ["yes"]}])
|
|
|
|
assert isinstance(result, str)
|
|
assert "exactly one option" in result
|
|
assert "filler" in result
|
|
assert broker.asked == [], "a rejected call must not reach the user"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_free_form_question_with_no_options_is_still_allowed():
|
|
tool, broker = _tool({"Why?": "because"})
|
|
|
|
result = await tool.execute(questions=[{"question": "Why?"}])
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert broker.asked == [("Why?", [])]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_duplicate_question_text_is_rejected_before_prompting():
|
|
tool, broker = _tool({})
|
|
|
|
result = await tool.execute(questions=[{"question": "Ship it?"}, {"question": "Ship it?"}])
|
|
|
|
assert isinstance(result, str)
|
|
assert "duplicate question" in result
|
|
assert broker.asked == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_duplicate_option_labels_are_deduped_rather_than_rejected():
|
|
"""A repeated label is a typo with one obvious reading; dropping it costs
|
|
the model nothing, whereas rejecting the call costs a whole turn."""
|
|
tool, broker = _tool({"Which?": "uv"})
|
|
|
|
result = await tool.execute(questions=[{"question": "Which?", "options": ["uv", "pip", "uv"]}])
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert broker.asked == [("Which?", ["uv", "pip"])]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_batch_over_the_cap_is_rejected_with_the_cap_named():
|
|
tool, broker = _tool({})
|
|
|
|
result = await tool.execute(questions=[{"question": f"q{i}?"} for i in range(5)])
|
|
|
|
assert isinstance(result, str)
|
|
assert "at most 4" in result
|
|
assert broker.asked == []
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_unanswered_question_names_the_recommended_option():
|
|
"""Without a default the model only learns that nobody answered; naming the
|
|
option it recommended is what lets it proceed the way it intended to."""
|
|
tool, _ = _tool({})
|
|
|
|
result = await tool.execute(questions=[{"question": "Ship it?", "options": ["hold", "ship"], "recommended": 1}])
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert 'recommended option was "ship"' in result.model_text
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_recommendation_survives_a_duplicate_earlier_in_the_options():
|
|
"""The index counts the options as submitted, so a duplicate ahead of it must
|
|
not shift which label the recommendation resolves to."""
|
|
tool, broker = _tool({"Ship?": "ship"})
|
|
|
|
await tool.execute(questions=[{"question": "Ship?", "options": ["a", "a", "b", "c"], "recommended": 2}])
|
|
|
|
assert broker.calls[0]["choices"] == ["a", "b", "c"]
|
|
assert broker.calls[0]["recommended"] == "b"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_recommendation_past_the_deduped_length_still_resolves():
|
|
"""Dedup shortens the list, so an index valid against the submitted options
|
|
can fall outside it -- that must not silently drop the recommendation."""
|
|
tool, broker = _tool({"Ship?": "ship"})
|
|
|
|
await tool.execute(questions=[{"question": "Ship?", "options": ["a", "a", "b", "c"], "recommended": 3}])
|
|
|
|
assert broker.calls[0]["recommended"] == "c"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_out_of_range_recommended_index_is_ignored():
|
|
tool, _ = _tool({})
|
|
|
|
result = await tool.execute(questions=[{"question": "Ship it?", "options": ["a", "b"], "recommended": 9}])
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert "recommended option" not in result.model_text
|
|
assert "did not answer" in result.model_text
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_exhausted_budget_stops_asking_instead_of_starting_a_fresh_wait():
|
|
"""Once the shared deadline is spent, the remaining questions must not each
|
|
open a new wait -- that is the per-question timeout this replaces."""
|
|
tool, broker = _tool({"a?": "1", "b?": "2"}, delay_s=0.05)
|
|
broker.default_timeout_s = 0.01
|
|
|
|
result = await tool.execute(questions=[{"question": "a?"}, {"question": "b?"}])
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert [c["prompt"] for c in broker.calls] == ["a?"], "the second question must not be asked"
|
|
assert "did not answer" in result.model_text
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_overlong_header_is_truncated_rather_than_breaking_the_chip():
|
|
tool, broker = _tool({"Which?": "uv"})
|
|
|
|
await tool.execute(questions=[{"question": "Which?", "header": "a-very-long-header-label"}])
|
|
|
|
assert broker.calls[0]["header"] == "a-very-long-"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_recommended_option_reaches_the_broker_for_the_surface_to_mark():
|
|
tool, broker = _tool({"Ship?": "ship"})
|
|
|
|
await tool.execute(questions=[{"question": "Ship?", "options": ["hold", "ship"], "recommended": 1}])
|
|
|
|
assert broker.calls[0]["recommended"] == "ship"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tool_budget_overrides_whatever_the_surface_defaults_to():
|
|
"""The batch budget is configuration, so it has to reach the tool without
|
|
the tool re-reading config from under a transport that never had one."""
|
|
broker = _StubBroker({"a?": "1"})
|
|
broker.default_timeout_s = 600.0
|
|
tool = AskUserTool(broker=broker, conversation_id="tui:test", timeout_s=1.5) # type: ignore[arg-type]
|
|
|
|
await tool.execute(questions=[{"question": "a?"}])
|
|
|
|
assert broker.calls[0]["timeout_s"] <= 1.5
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_agent_loop_hands_the_configured_budget_to_the_tool(tmp_path):
|
|
"""The loop is where the config already is, so it is where the budget has
|
|
to be handed over -- the transports that wire the broker do not all have a
|
|
config to read."""
|
|
from raven.agent.loop.main import AgentLoop
|
|
from raven.config.schema import AskUserToolConfig
|
|
|
|
class _Provider:
|
|
"""AgentLoop construction reads the default model; no turn is run here."""
|
|
|
|
def get_default_model(self) -> str:
|
|
return "stub-model"
|
|
|
|
async def chat_with_retry(self, **kwargs): # pragma: no cover - never invoked
|
|
raise NotImplementedError
|
|
|
|
loop = AgentLoop(
|
|
provider=_Provider(), workspace=tmp_path, tools=ToolWiring(ask_user_config=AskUserToolConfig(timeout=42))
|
|
)
|
|
tool = loop.tools.get("ask_user")
|
|
assert tool is not None
|
|
|
|
broker = _StubBroker({"a?": "1"})
|
|
broker.default_timeout_s = 600.0
|
|
tool.set_broker(broker)
|
|
tool.set_context("tui:test")
|
|
|
|
await tool.execute(questions=[{"question": "a?"}])
|
|
|
|
assert broker.calls[0]["timeout_s"] <= 42
|
|
|
|
|
|
def test_description_states_the_cap_the_code_enforces():
|
|
"""The description is prompt text the model reads, so a cap named there and
|
|
a cap enforced here are two statements of one rule and will drift apart."""
|
|
from raven.agent.tools.ask_user import MAX_QUESTIONS
|
|
|
|
tool, _ = _tool({})
|
|
|
|
assert f"up to {MAX_QUESTIONS}" in tool.description
|
|
assert tool.parameters["properties"]["questions"]["maxItems"] == MAX_QUESTIONS
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_round_trip_through_the_real_broker():
|
|
"""Every other test here drives a stand-in, which cannot catch the tool and
|
|
the broker disagreeing about the keyword names they pass between them."""
|
|
from raven.rpc.question_broker import QuestionBroker
|
|
|
|
frames: list[dict] = []
|
|
|
|
async def send_frame(frame: dict) -> None:
|
|
frames.append(frame)
|
|
broker.reply(frame["params"]["conversation_id"], "uv")
|
|
|
|
broker = QuestionBroker(send_frame, timeout_s=5.0)
|
|
tool = AskUserTool(broker=broker, conversation_id="tui:test")
|
|
|
|
result = await tool.execute(
|
|
questions=[{"question": "Which?", "header": "Pkg", "options": ["uv", "pip"], "recommended": 0}]
|
|
)
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert 'User answered: "Which?" -> "uv".' in result.model_text
|
|
params = frames[0]["params"]
|
|
assert params["header"] == "Pkg"
|
|
assert params["recommended"] == "uv"
|
|
assert params["total"] == 1
|
|
assert params["timeout_s"] <= 5.0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_surface_answering_the_whole_batch_in_one_reply_emits_one_frame():
|
|
"""A page that renders the batch as one stepped form answers it in one
|
|
``clarify.respond`` carrying every question's answer. The broker must stash
|
|
the later ones rather than making the tool loop's later questions wait out
|
|
a frame that never comes."""
|
|
from raven.rpc.question_broker import QuestionBroker
|
|
|
|
frames: list[dict] = []
|
|
|
|
async def send_frame(frame: dict) -> None:
|
|
frames.append(frame)
|
|
params = frame["params"]
|
|
broker.reply(
|
|
params["conversation_id"],
|
|
"main",
|
|
answers=["main", "yes", "squash"],
|
|
)
|
|
|
|
broker = QuestionBroker(send_frame, timeout_s=5.0)
|
|
tool = AskUserTool(broker=broker, conversation_id="tui:test")
|
|
|
|
result = await tool.execute(
|
|
questions=[
|
|
{"question": "Base?"},
|
|
{"question": "Rebase?"},
|
|
{"question": "Squash?"},
|
|
]
|
|
)
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert len(frames) == 1, "the later questions must be answered from the stash, not a fresh frame each"
|
|
assert result.display_text == "Base? -> main\nRebase? -> yes\nSquash? -> squash"
|
|
assert 'User answered: "Base?" -> "main".' in result.model_text
|
|
assert 'User answered: "Rebase?" -> "yes".' in result.model_text
|
|
assert 'User answered: "Squash?" -> "squash".' in result.model_text
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_execute_holds_the_conversation_lock_against_a_relayed_asker():
|
|
"""A relayed ACP question (which takes ``question_lock`` itself, per
|
|
``acp_client.ask_user._ask``) must wait for this call's own batch to
|
|
finish rather than colliding with it on the broker's single pending slot.
|
|
|
|
Before the fix, the relayed call's ``await_question`` would evict this
|
|
call's still-pending question (``existing.future.set_result(existing.default)``),
|
|
which the tool then rendered as "did not answer" even though nobody had
|
|
seen the question yet.
|
|
|
|
The relayed side is the real route -- ``AskViaTool(tool).ask`` under the
|
|
lock, exactly what ``AskUserResponder._ask`` and ``Elicitor._one`` do --
|
|
not a direct ``broker.await_question``. A first version of this test took
|
|
the shortcut and so passed over ``ask_direct`` re-taking the same
|
|
non-reentrant lock, which stalled every relayed question for the whole
|
|
budget and answered it empty.
|
|
"""
|
|
from raven.acp_client.asker import AskViaTool, question_lock
|
|
from raven.rpc.question_broker import QuestionBroker
|
|
|
|
local_sent = asyncio.Event()
|
|
release_local = asyncio.Event()
|
|
events: list[str] = []
|
|
|
|
async def send_frame(frame: dict) -> None:
|
|
prompt = frame["params"]["question"]
|
|
events.append(f"sent:{prompt}")
|
|
if prompt == "Local?":
|
|
local_sent.set()
|
|
await release_local.wait()
|
|
broker.reply(frame["params"]["conversation_id"], f"answer-for-{prompt}")
|
|
|
|
broker = QuestionBroker(send_frame, timeout_s=5.0)
|
|
tool = AskUserTool(broker=broker, conversation_id="tui:test")
|
|
asker = AskViaTool(tool)
|
|
|
|
async def relayed_question() -> str | None:
|
|
# Waits for the local call to have a question in flight before even
|
|
# trying for the lock -- otherwise it could win the race to acquire
|
|
# first and this test would prove nothing about the collision.
|
|
await local_sent.wait()
|
|
async with question_lock("tui:test"):
|
|
events.append("relayed-acquired-lock")
|
|
return await asker.ask("Relayed?", None, "tui:test")
|
|
|
|
async def release_after_a_beat() -> None:
|
|
await asyncio.sleep(0.05)
|
|
release_local.set()
|
|
|
|
result, relayed_answer, _ = await asyncio.gather(
|
|
tool.execute(questions=[{"question": "Local?"}]),
|
|
relayed_question(),
|
|
release_after_a_beat(),
|
|
)
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert 'User answered: "Local?" -> "answer-for-Local?".' in result.model_text
|
|
assert relayed_answer == "answer-for-Relayed?"
|
|
# Strict order: the relayed call could not even send its frame until the
|
|
# local call's own round trip released the lock.
|
|
assert events == ["sent:Local?", "relayed-acquired-lock", "sent:Relayed?"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ask_direct_takes_no_lock_of_its_own():
|
|
"""Both relayed routes hold ``question_lock`` across their whole exchange and
|
|
reach the broker through ``ask_direct``; the lock is not re-entrant, so a
|
|
second acquire here would wait the budget out and answer empty with the
|
|
broker never having seen the question."""
|
|
from raven.acp_client.asker import question_lock
|
|
|
|
tool, broker = _tool({"Relayed?": "yes"})
|
|
async with question_lock("tui:test"):
|
|
answer = await asyncio.wait_for(tool.ask_direct("Relayed?", None, "tui:test"), timeout=1.0)
|
|
assert answer == "yes"
|
|
assert [prompt for prompt, _ in broker.asked] == ["Relayed?"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_busy_conversation_spends_the_calls_own_budget_not_a_second_one():
|
|
"""The wait for the lock is bounded by this call's deadline, so a conversation
|
|
another asker holds past it costs one budget, not the budget plus a separate
|
|
lock timeout -- and the question never reaches the broker."""
|
|
from raven.acp_client.asker import question_lock
|
|
|
|
broker = _StubBroker({"Q?": "a"})
|
|
tool = AskUserTool(broker=broker, conversation_id="tui:test", timeout_s=0.2) # type: ignore[arg-type]
|
|
lock = question_lock("tui:test")
|
|
await lock.acquire()
|
|
try:
|
|
started = asyncio.get_running_loop().time()
|
|
result = await tool.execute(questions=[{"question": "Q?"}])
|
|
elapsed = asyncio.get_running_loop().time() - started
|
|
finally:
|
|
lock.release()
|
|
|
|
assert isinstance(result, ToolResult)
|
|
assert "user did not answer" in result.model_text
|
|
assert broker.asked == []
|
|
assert 0.15 <= elapsed < 1.0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ask_direct_forwards_the_batch_to_the_broker():
|
|
tool, broker = _tool({"Which reviewer?": "chandler"})
|
|
batch = [{"question": "Which branch?"}, {"question": "Which reviewer?"}]
|
|
await tool.ask_direct("Which reviewer?", None, "tui:c1", index=1, total=2, batch=batch)
|
|
call = broker.calls[0]
|
|
assert (call["index"], call["total"], call["batch"]) == (1, 2, batch)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ask_direct_still_defaults_to_a_lone_question():
|
|
tool, broker = _tool({"Which branch?": "feat/x"})
|
|
await tool.ask_direct("Which branch?", None, "tui:c1")
|
|
call = broker.calls[0]
|
|
assert (call["index"], call["total"], call["batch"]) == (0, 1, None)
|
|
assert not call.get("default")
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_registry_dispatch_and_the_real_clarify_respond_route():
|
|
"""The production entry point is the registry, not ``execute`` directly, and
|
|
the answer arrives over the real ``clarify.respond`` handler. Neither layer
|
|
is exercised by the tests above, and both can reject a call the tool would
|
|
have accepted -- the schema validator runs in between.
|
|
"""
|
|
from raven.agent.tools.registry import ToolRegistry
|
|
from raven.rpc.dispatcher import Dispatcher
|
|
from raven.rpc.methods.question import register_question_methods
|
|
from raven.rpc.question_broker import QuestionBroker
|
|
|
|
dispatcher = Dispatcher()
|
|
|
|
async def send_frame(frame: dict) -> None:
|
|
params = frame["params"]
|
|
reply = await dispatcher.dispatch(
|
|
{
|
|
"jsonrpc": "2.0",
|
|
"id": 1,
|
|
"method": "clarify.respond",
|
|
"params": {"request_id": params["request_id"], "answer": "uv"},
|
|
}
|
|
)
|
|
assert reply["result"]["ok"] is True
|
|
|
|
broker = QuestionBroker(send_frame, timeout_s=5.0)
|
|
register_question_methods(dispatcher, question_broker=broker)
|
|
registry = ToolRegistry()
|
|
registry.register(AskUserTool(broker=broker, conversation_id="tui:test"))
|
|
|
|
answered = await registry.execute(
|
|
"ask_user",
|
|
{"questions": [{"question": "Which?", "header": "Pkg", "options": ["uv", "pip"], "recommended": 0}]},
|
|
)
|
|
assert 'User answered: "Which?" -> "uv".' in answered
|
|
|
|
# A rejection has to survive the registry intact -- it is the steer the model
|
|
# reads, and the registry appends to it rather than replacing it.
|
|
rejected = await registry.execute("ask_user", {"questions": [{"question": "Ship?", "options": ["yes"]}]})
|
|
assert "exactly one option" in rejected
|
|
assert "filler" in rejected
|
|
|
|
|
|
def test_an_option_written_as_an_object_reaches_the_user_as_its_words() -> None:
|
|
"""The schema says strings; a live deck run wrote objects -- option, description,
|
|
recommended -- and str() put each dict's repr on the screen as the thing to click.
|
|
The option is the text under the key that names the answer, its description
|
|
follows after a dash, and a flag on it becomes the recommendation unless the
|
|
model stated an index."""
|
|
from raven.agent.tools.ask_user import AskUserTool
|
|
|
|
payload = {
|
|
"questions": [
|
|
{
|
|
"question": "Dark or light cover?",
|
|
"options": [
|
|
{"option": "dark", "description": "navy ground, light type", "recommended": True},
|
|
{"option": "light", "description": "white ground, dark type"},
|
|
"plain string stays",
|
|
],
|
|
}
|
|
]
|
|
}
|
|
entry = AskUserTool().cast_params(payload)["questions"][0]
|
|
assert entry["options"] == [
|
|
"dark -- navy ground, light type",
|
|
"light -- white ground, dark type",
|
|
"plain string stays",
|
|
]
|
|
assert entry["recommended"] == 0
|
|
unnamed = {
|
|
"questions": [
|
|
{
|
|
"question": "Who is it for?",
|
|
"options": [
|
|
{"description": "the operations team, with checklists", "recommended": True},
|
|
{"description": "prospective vendors", "recommended": False},
|
|
],
|
|
}
|
|
]
|
|
}
|
|
described = AskUserTool().cast_params(unnamed)["questions"][0]
|
|
assert described["options"] == ["the operations team, with checklists", "prospective vendors"], (
|
|
"an object with only a description is that description"
|
|
)
|
|
assert described["recommended"] == 0
|
|
explicit = {
|
|
"questions": [{"question": "q", "options": [{"option": "a", "recommended": True}, "b"], "recommended": 1}]
|
|
}
|
|
assert AskUserTool().cast_params(explicit)["questions"][0]["recommended"] == 1
|
|
assert payload["questions"][0]["options"][0]["option"] == "dark", "the caller's payload is left alone"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize("encoded_layers", [0, 1, 2])
|
|
@pytest.mark.parametrize(
|
|
"malformed",
|
|
['[{"question": "Which base?", "options": ["main", "develop"]]', '[\n{"question": "Which base?"\n]'],
|
|
)
|
|
async def test_invalid_questions_json_reports_location_without_asking(malformed, encoded_layers):
|
|
payload = malformed
|
|
for _ in range(encoded_layers):
|
|
payload = json.dumps(payload)
|
|
with pytest.raises(json.JSONDecodeError) as error:
|
|
json.loads(malformed)
|
|
expected = f"questions is not valid JSON: {error.value}"
|
|
tool, broker = _tool({})
|
|
registry, registry_broker = _registered({})
|
|
args = {"questions": payload}
|
|
original = copy.deepcopy(args)
|
|
|
|
assert tool.display_call(args) is None
|
|
direct = await tool.execute(**args)
|
|
registered = await registry.execute("ask_user", args)
|
|
|
|
assert expected in direct
|
|
assert expected in registered
|
|
assert "requires at least one question" not in direct
|
|
assert "requires at least one question" not in registered
|
|
assert broker.asked == registry_broker.asked == []
|
|
assert args == original
|
|
|
|
|
|
# --- what the model is told when the shape is wrong ------------------------
|
|
#
|
|
# Re-created from live glm-5.3-flash calls. Each one either failed with only
|
|
# "missing required questions[0].question" -- the normalizer had already turned
|
|
# the option objects into strings, so the misplaced `question` was gone before
|
|
# the registry looked -- or ran and showed the user something the model never
|
|
# wrote.
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_question_written_inside_its_options_is_reported_where_it_went():
|
|
"""Twice in a row on one live turn: each option an object carrying its own
|
|
`question` and `description`, and no `question` on the entry itself."""
|
|
registry, broker = _registered({})
|
|
|
|
out = await registry.execute(
|
|
"ask_user",
|
|
{
|
|
"questions": [
|
|
{
|
|
"header": "Scope",
|
|
"options": [
|
|
{"description": "prepare only", "recommended": 0, "question": "Prepare now?"},
|
|
{"description": "prepare, then restart", "question": "Restart as well?"},
|
|
],
|
|
}
|
|
]
|
|
},
|
|
)
|
|
|
|
assert broker.asked == []
|
|
located = next(line for line in out.splitlines() if "missing required questions[0].question" in line)
|
|
assert "questions[0].options[0]" in located
|
|
assert "questions[0].options[1]" in located
|
|
assert "questions[0].options[0] should be string" in out
|
|
assert "questions[0].options[1] should be string" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_question_split_across_two_option_items_is_reported_not_rendered():
|
|
"""The real options nested in one item, the question alone in the next.
|
|
Rendered, the second item became an option reading `{'question': ...}`."""
|
|
registry, broker = _registered({})
|
|
|
|
out = await registry.execute(
|
|
"ask_user",
|
|
{
|
|
"questions": [
|
|
{
|
|
"header": "Create PR",
|
|
"options": [
|
|
{"description": "fix the wording first", "options": ["Fix, then create", "Create as is"]},
|
|
{"question": "Create the pull request now?"},
|
|
],
|
|
}
|
|
]
|
|
},
|
|
)
|
|
|
|
assert broker.asked == []
|
|
assert "{'question'" not in out
|
|
located = next(line for line in out.splitlines() if "missing required questions[0].question" in line)
|
|
assert "questions[0].options[1]" in located
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_choices_nested_inside_option_objects_are_not_silently_dropped():
|
|
"""The question was well formed and the call ran -- showing the user two
|
|
descriptions and never the choices that were nested inside them."""
|
|
registry, broker = _registered({"Which class is the ID number?": "anything"})
|
|
|
|
out = await registry.execute(
|
|
"ask_user",
|
|
{
|
|
"questions": [
|
|
{
|
|
"question": "Which class is the ID number?",
|
|
"options": [
|
|
{"description": "per the report", "options": ["non-sensitive", "sensitive"], "recommended": 0},
|
|
{"description": "per the task notes", "options": ["non-sensitive", "sensitive"]},
|
|
],
|
|
}
|
|
]
|
|
},
|
|
)
|
|
|
|
assert broker.asked == []
|
|
assert "questions[0].options[0] should be string" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ask_direct_still_hands_the_broker_only_strings():
|
|
"""Host-side questions never pass the registry's validation, so an option
|
|
object that the model-facing path now leaves for the validator must still
|
|
reach the broker as text here."""
|
|
tool, broker = _tool({"Proceed?": "yes"})
|
|
|
|
await tool.ask_direct("Proceed?", [{"question": "odd"}, "plain"], "tui:test")
|
|
|
|
assert all(isinstance(choice, str) for choice in broker.asked[0][1])
|
|
|
|
|
|
def test_an_option_object_with_no_words_is_left_for_the_validator():
|
|
"""A live sub-agent call sent `{"recommended": true}` as an option; rendered, the
|
|
user saw its repr. A name that is not text is no name either."""
|
|
from raven.agent.tools.ask_user import AskUserTool
|
|
|
|
payload = {"questions": [{"question": "Q?", "options": [{"recommended": True}, {"value": 3}, "plain"]}]}
|
|
|
|
options = AskUserTool().cast_params(payload)["questions"][0]["options"]
|
|
|
|
assert options == [{"recommended": True}, {"value": 3}, "plain"]
|
|
|
|
|
|
def test_options_written_as_one_object_are_one_option() -> None:
|
|
"""A bare object where the array belongs, on the model-facing path: it becomes
|
|
a one-element list rather than being iterated as its own keys, and its own
|
|
words are rendered. A `recommended` flag inside it is not read -- `_flagged`
|
|
reads flags from the list the schema declares, and this value is not one.
|
|
"""
|
|
from raven.agent.tools.ask_user import AskUserTool
|
|
|
|
payload = {"questions": [{"question": "Q?", "options": {"option": "only", "recommended": True}}]}
|
|
|
|
entry = AskUserTool().cast_params(payload)["questions"][0]
|
|
|
|
assert entry["options"] == ["only"]
|
|
|
|
|
|
async def test_kernel_options_with_unknown_metadata_are_rejected_before_the_broker():
|
|
tool, broker = _tool({"Region?": "Global"})
|
|
registry = ToolRegistry()
|
|
registry.register(tool)
|
|
result = await registry.execute(
|
|
"ask_user",
|
|
{
|
|
"questions": [{"question": "Region?", "options": [{"label": "EU only", "id": 1}, "Global"]}],
|
|
},
|
|
)
|
|
assert "Invalid parameters" in result
|
|
assert "options[0] should be string" in result and "id" in result
|
|
assert broker.asked == []
|