## Summary A failed turn's Web row can expand to the provider's full diagnostic text, up to 8,000 characters, while the existing bounded reason stays unchanged. The optional `full_detail` field travels through the runner, lane, RPC event and session marker, so live display and replay expose the same account. Copy includes the expanded account. Quiet runtime notices use "Show details" instead of the error label. Failure details and call-record bodies remove known credential shapes before the final cap. An unterminated private-key block is removed through the end of the input, including when an earlier scan or call record cut away its closing armour. Known cut boundaries also remove incomplete JWTs, vendor tokens, URL passwords and labelled credential tails; each of those patterns is searched only from behind the run of its own value's characters that ends the input and the label in front of it, so the scan stays linear whatever the text holds and still reads a label gap of any width. The authorization and JWT patterns avoid repeated backtracking, and the turn path and outer runner reuse the computed detail. Attachment and lane reads use the base exception's instance storage, so a foreign read-only property or throwing getter cannot replace the scrubbed value or swallow the terminal failure event. The optional fields are mirrored in the RPC schema and both generated clients. The design document separates this delivery from the subsequent Turn Continuation feature; Retry still starts an ordinary turn. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification The automated suites and static/build checks below were re-run after synchronizing with main, using the existing local dependencies. The independent behavioral and timing probes were measured before that synchronization. The last three commits change only `raven/security/redact.py`, two test files and the design document; on that final tree the Python suite, the ui-web tests, the type, generation and lint checks, `ty`, the import contracts, the range gates and pre-commit were run again, and the ui-web build checks were not, since none of their inputs changed. - `uv run --no-sync pytest -q tests/test_agent_loop_failure_text.py tests/test_llm_call_record.py tests/test_agent_loop_broken_turn.py tests/test_agent_loop_llm_error_patience.py tests/test_agent_loop_run_emit.py tests/test_spine_scheduler_lane.py tests/test_spine_events.py tests/test_rpc_spine.py tests/test_rpc_schema_match.py tests/test_kernel_budget.py tests/test_acp_redact.py tests/test_security_untrusted_context.py -x --tb=short --show-capture=no`: 904 passed. - From `ui-web`: `npm test -- --no-file-parallelism src/features/transcript/TranscriptPage.test.tsx src/state/session/pipeline.events.test.ts scripts/gates/i18n-keys.test.mjs`: 225 passed. - From `ui-web`: `npm run type-check`, `npm run gen:check`, `node ../ui-tui/scripts/gen-i18n.mjs --check`, `node ../ui-tui/scripts/gen-rpc-types.mjs --check`, and `node node_modules/eslint/bin/eslint.js src/features/transcript/TranscriptPage.tsx src/features/transcript/TranscriptPage.test.tsx`: passed. - `uv run --no-sync lint-imports`: 10 contracts kept. `uv run --no-sync ty check raven evolver agents plugins-dist scripts`: passed. Ruff check and format check passed on the 20 Python files this PR changes. - From `ui-web`: `npm run build`, `uv run --no-sync python build.py`, `node scripts/check-page.mjs`, `node scripts/check-css.mjs`, and `node scripts/check-class-namespace.mjs`: passed, including both boot snapshots. - `UV_NO_SYNC=1 make check-large-files check-source-language COMMIT_RANGE=origin/main..HEAD`: passed. `PYTHONPATH=. uv run --no-sync python scripts/check_commit_messages.py origin/main..HEAD`, commitlint over the same range and `git diff --check origin/main...HEAD`: passed. - `uv run --no-sync pre-commit run --from-ref origin/main --to-ref HEAD`: all applicable hooks passed except the whole-TUI Prettier hook, which reports `newInstancePicker.tsx` and `subagentsHub.tsx`. Both files and the formatter config are unchanged from main; an isolated main-file control with the same locked Prettier version reproduces both failures. `node ui-tui/node_modules/prettier/bin/prettier.cjs --check ui-tui/src/i18n/messages.generated.ts`: passed. - Regression coverage includes truncated private keys, incomplete JWT and credential tails, terminal-event delivery with foreign exception attributes, scan reuse and CPU cost, the cut-tail scan's cost on label-shaped runs and long gaps, its agreement with a whole-text search, a label gap of any width, the JWT boundary and private-key END rules, and quiet-notice expansion. The five independent boundary probes passed before the base synchronization; the committed regression suite above passed again afterwards. - On the same 32,267-character synthetic crash input, the queued event-loop callback delay fell from 15.351 seconds to 0.00951 seconds. The regression checks use thread CPU time with a 0.5-second ceiling. - Cut-tail scan: a failed turn whose crash text holds 31,000 spaces before its last two words took 32.6 seconds while the region came from a regex search for the last whitespace, and 0.016 seconds now; 64,000 characters of `pwd=` ending in a comma took 10.8 seconds through the whole-text tail search and 0.003 seconds now, and a 300,000-character call-record body of either shape takes under 0.3 seconds. These and the figures below were measured with the machine's load average between 36 and 132. Across 3,728 timed texts, every label or prefix unit repeated to 64,000 characters behind a cut and to 300,000 past the scan cap, the slowest measured alone took about 0.2 seconds against about 0.1 seconds on the base revision, and a text past the cap takes the same time at twice the size. Searching the whole text again turns six timing cases red; a smaller name room, a skipped gap or scheme step, or any value class narrowed below its pattern's turns the property or gap test red, as does removing the JWT boundary or restoring the END-only private-key rule. On 12,008 generated texts with gaps up to 5,000 characters, the output equals the whole-text version's. - Chromium with synthetic local data: full text retained and HTML escaped, expanded tooltip removed, internal scrolling and Retry preserved, neutral notice label displayed, no page errors. No live vendor call or native Windows rerun is claimed for these fixes. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - [x] Security impact considered: the long text is scrubbed of recognized credential shapes before persistence and publication. It remains diagnostic text and may contain hostnames or paths. Model history and chat-channel replies keep their existing bounded content. A session export can include the long text. `raven.security.redact` is also the scrubber for every ACP surface (tool titles, result previews, permission prompts, replay, error messages) and for the outbound client's permission log, so two of its rules reach those surfaces too: a private-key header without END hides the rest of that text, and a JWT glued to a preceding word character, such as `/verify-eyJ...`, is left unredacted. Tests pin both. - [x] Backward compatibility considered: the new fields are optional, older markers fall back to their bounded reason, and the TUI can ignore the extra field. Incomplete private-key blocks are conservatively removed to the end, which can also omit trailing diagnostic text. Custom runners attach long details in the exception's instance dictionary; descriptors are ignored. - [x] Rollback path is clear for risky changes: revert the squash commit. Older code ignores the optional marker field. ## Related Issues N/A --------- Co-authored-by: Codex <noreply@openai.com>
1626 lines
60 KiB
Python
1626 lines
60 KiB
Python
"""Tests pinning AgentLoop's spine-native ``run_turn(req, emit)`` output behavior.
|
|
|
|
run_turn fans the agent's output (streamed token deltas, reasoning, tool events,
|
|
notices, media) onto a single ``emit`` and returns a TurnOutcome. These pin that
|
|
observable behavior per category, plus origin gating and metadata reconstruction.
|
|
Driven against a real AgentLoop with only the LLM provider + sandbox edges faked
|
|
(never the output path itself).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import json
|
|
from dataclasses import dataclass, field
|
|
|
|
import pytest
|
|
|
|
from raven.agent.hook import AgentHook, HookDecision
|
|
from raven.agent.hook.adapters import DecisionConsumerAdapter
|
|
from raven.agent.loop import AgentLoop
|
|
from raven.agent.loop._shared import _MID_TURN_HEADER
|
|
from raven.agent.loop.bundles import HostWiring
|
|
from raven.contracts.llm_provider import ChatDelta, LLMResponse, ToolCallRequest
|
|
from raven.contracts.loop_hooks import AgentHook, HookDecision
|
|
from raven.contracts.tool import Tool, ToolResult
|
|
from raven.providers.base import ErrorClassification
|
|
from raven.sandbox import SandboxInitError
|
|
from raven.spine.events import EpisodeStart as EvEpisodeStart
|
|
from raven.spine.events import MediaOut as EvMediaOut
|
|
from raven.spine.events import Notice as EvNotice
|
|
from raven.spine.events import NoticeKind, ToolPhase
|
|
from raven.spine.events import Reasoning as EvReasoning
|
|
from raven.spine.events import StreamDelta as EvStreamDelta
|
|
from raven.spine.events import Text as EvText
|
|
from raven.spine.events import ToolEvent as EvToolEvent
|
|
from raven.spine.message import ChatType, Media, Source
|
|
from raven.spine.turn import Origin, TurnRequest
|
|
|
|
|
|
@dataclass
|
|
class _Reply:
|
|
channel: str
|
|
chat_id: str
|
|
content: str
|
|
media: list[str] = field(default_factory=list)
|
|
|
|
|
|
class _FakeTool(Tool):
|
|
"""Minimal no-sandbox tool so a tool-call turn can dispatch + fire events."""
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "faketool"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "characterization fake tool"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}, "required": []}
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
return "tool-ran"
|
|
|
|
|
|
class _SplitTool(Tool):
|
|
"""Returns ToolResult so the loop's unwrap branch is actually exercised."""
|
|
|
|
MODEL_TEXT = 'User answered: "Which base?" -> "main". Continue.'
|
|
DISPLAY_TEXT = "Which base? -> main"
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "splittool"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "fake tool with distinct model-facing and display-facing text"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}, "required": []}
|
|
|
|
def display_call(self, args: dict) -> str | None:
|
|
return "Which base?"
|
|
|
|
async def execute(self, **kwargs) -> ToolResult:
|
|
return ToolResult(model_text=self.MODEL_TEXT, display_text=self.DISPLAY_TEXT)
|
|
|
|
|
|
class _FakeChatProvider:
|
|
"""Non-streaming path: ``chat_with_retry`` returns scripted LLMResponses."""
|
|
|
|
def __init__(self, responses: list[LLMResponse]) -> None:
|
|
self._responses = list(responses)
|
|
self._i = 0
|
|
|
|
async def chat_with_retry(self, **kwargs) -> LLMResponse:
|
|
r = self._responses[min(self._i, len(self._responses) - 1)]
|
|
self._i += 1
|
|
return r
|
|
|
|
def get_default_model(self) -> str:
|
|
return "fake/model"
|
|
|
|
|
|
class _FakeStreamProvider:
|
|
"""Yields scripted stream chunks via ``chat_stream`` (the streaming path that
|
|
``run(req, emit)`` takes when emitting ChatDelta tokens)."""
|
|
|
|
def __init__(self, chunks: list[ChatDelta]) -> None:
|
|
self._chunks = chunks
|
|
|
|
async def chat_stream(self, **kwargs):
|
|
for chunk in self._chunks:
|
|
yield chunk
|
|
|
|
def get_default_model(self) -> str:
|
|
return "fake/model"
|
|
|
|
|
|
class _FakeStreamToolProvider:
|
|
"""``chat_stream`` yields a fresh scripted chunk-list per call, so a
|
|
tool-call iteration works under run() (call 1 -> tool_call_delta, call 2 ->
|
|
final content). run() always wires on_token_delta, so every turn streams."""
|
|
|
|
def __init__(self, scripts: list[list[ChatDelta]]) -> None:
|
|
self._scripts = scripts
|
|
self._i = 0
|
|
|
|
async def chat_stream(self, **kwargs):
|
|
script = self._scripts[min(self._i, len(self._scripts) - 1)]
|
|
self._i += 1
|
|
for chunk in script:
|
|
yield chunk
|
|
|
|
def get_default_model(self) -> str:
|
|
return "fake/model"
|
|
|
|
|
|
class _EmitCollector:
|
|
"""Records every RunnerEvent run() emits, in order."""
|
|
|
|
def __init__(self) -> None:
|
|
self.events: list = []
|
|
|
|
async def __call__(self, ev) -> None:
|
|
self.events.append(ev)
|
|
|
|
|
|
def _drain() -> list:
|
|
return []
|
|
|
|
|
|
def _req(text: str, *, media=(), origin: Origin = Origin.USER) -> TurnRequest:
|
|
return TurnRequest(
|
|
origin=origin,
|
|
source=Source(channel="cli", chat_id="c", sender_id="u", chat_type=ChatType.DM),
|
|
text=text,
|
|
media=media,
|
|
)
|
|
|
|
|
|
def _stub_edges(loop: AgentLoop) -> None:
|
|
"""No-op the sandbox/MCP bring-up so a text-only turn runs without a VM."""
|
|
|
|
async def _noop(**_kw) -> None:
|
|
return None
|
|
|
|
loop._start_executor = _noop
|
|
loop._connect_mcp = _noop
|
|
|
|
|
|
async def test_help_slash_returns_command_list_at_outbound_layer(tmp_path):
|
|
# Slash exit (:1294) pinned at the _process_message reply layer
|
|
# (media-capable; the old str-returning path would project metadata away).
|
|
loop = AgentLoop(provider=_FakeChatProvider([]), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
|
|
out = await loop._process_message(_req("/help"))
|
|
|
|
assert out is not None
|
|
content, _media = out
|
|
assert "Raven commands" in content
|
|
|
|
|
|
async def test_hook_short_circuit_preserves_media_at_outbound_layer(tmp_path):
|
|
# MediaOut category: a before_user_inbound short-circuit (:1199) returns a
|
|
# reply that can carry media. Pinned at the reply layer —
|
|
# the old str-returning path would project media away, so collapse-drops-media would
|
|
# be untestable otherwise.
|
|
async def _decision(req: TurnRequest):
|
|
return _Reply(
|
|
channel=req.source.channel,
|
|
chat_id=req.source.chat_id,
|
|
content="short",
|
|
media=["/tmp/x.png"],
|
|
)
|
|
|
|
loop = AgentLoop(
|
|
provider=_FakeChatProvider([]), workspace=tmp_path, host=HostWiring(hooks=[DecisionConsumerAdapter(_decision)])
|
|
)
|
|
_stub_edges(loop)
|
|
|
|
out = await loop._process_message(_req("hi"))
|
|
|
|
assert out is not None
|
|
content, media = out
|
|
assert media == ["/tmp/x.png"] # media survives the short-circuit return
|
|
assert content == "short"
|
|
|
|
|
|
# ── run(req, emit) collapse — emit sequence per category ────────────
|
|
|
|
|
|
async def test_run_streams_then_dissolves_main_response(tmp_path):
|
|
# Streaming main response: each non-empty chunk -> emit(ChatDelta); the
|
|
# return dissolves (b2) -> no trailing Text. Usage rides TurnOutcome.
|
|
chunks = [
|
|
ChatDelta(content="Hel"),
|
|
ChatDelta(content="lo", usage={"prompt_tokens": 3, "completion_tokens": 2, "total_tokens": 5}),
|
|
]
|
|
loop = AgentLoop(provider=_FakeStreamProvider(chunks), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
outcome = await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
# The turn opens with an episode boundary, then the two streamed deltas.
|
|
assert isinstance(sink.events[0], EvEpisodeStart) and sink.events[0].index == 0
|
|
deltas = [e for e in sink.events if isinstance(e, EvStreamDelta)]
|
|
assert [e.delta for e in deltas] == ["Hel", "lo"]
|
|
assert not any(isinstance(e, EvText) for e in sink.events) # dissolved, no double
|
|
assert outcome.usage.total_tokens == 5
|
|
assert outcome.explicit_reply is True
|
|
|
|
|
|
class _Announce(AgentHook):
|
|
"""An after_send hook of the kind a deck engine installs: it appends to the reply."""
|
|
|
|
async def after_send(self, ctx):
|
|
return HookDecision(modified_content=f"{ctx.outbound_content}\n\nDeck: /out/deck.pptx")
|
|
|
|
|
|
async def test_what_after_send_appends_reaches_a_streaming_sink_as_one_more_delta(tmp_path):
|
|
"""The reply left as deltas before the hook ran; the tail is sent after it.
|
|
|
|
Measured on five test homes: the deck engine's "Deck: / Preview: / MEDIA:" lines
|
|
never reached the host over ACP, because a streamed turn's returned text was
|
|
dropped at the emit boundary and the hook only ever changed that text.
|
|
"""
|
|
chunks = [ChatDelta(content="Hel"), ChatDelta(content="lo")]
|
|
loop = AgentLoop(provider=_FakeStreamProvider(chunks), workspace=tmp_path, host=HostWiring(hooks=[_Announce()]))
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
deltas = [e.delta for e in sink.events if isinstance(e, EvStreamDelta)]
|
|
assert deltas == ["Hel", "lo", "\n\nDeck: /out/deck.pptx"]
|
|
assert not any(isinstance(e, EvText) for e in sink.events)
|
|
|
|
|
|
class _Rewrite(AgentHook):
|
|
"""An after_send hook that replaces the reply wholesale rather than appending to it."""
|
|
|
|
async def after_send(self, ctx):
|
|
return HookDecision(modified_content="Completely different reply")
|
|
|
|
|
|
async def test_what_after_send_rewrites_wholesale_is_not_sent_after_a_streamed_reply(tmp_path):
|
|
"""The one branch of `_appended_by_hook` whose failure a user sees: the client has
|
|
the streamed "Hello" already, and a rewrite forwarded as if it were an append would
|
|
arrive as one more delta -- two replies on screen. Nothing is sent instead, and the
|
|
non-streamed path, which has sent nothing yet, carries the rewrite whole."""
|
|
from raven.agent.loop._shared import _appended_by_hook
|
|
|
|
assert _appended_by_hook("Hello", "Hello\n\nDeck: /out/d.pptx") == "\n\nDeck: /out/d.pptx"
|
|
assert _appended_by_hook("Hello", "Completely different reply") == ""
|
|
assert _appended_by_hook("Hello", "Hello") == ""
|
|
assert _appended_by_hook(None, "Deck: /out/d.pptx") == "Deck: /out/d.pptx"
|
|
|
|
chunks = [ChatDelta(content="Hel"), ChatDelta(content="lo")]
|
|
loop = AgentLoop(provider=_FakeStreamProvider(chunks), workspace=tmp_path, host=HostWiring(hooks=[_Rewrite()]))
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
assert [e.delta for e in sink.events if isinstance(e, EvStreamDelta)] == ["Hel", "lo"]
|
|
assert not any(isinstance(e, EvText) for e in sink.events)
|
|
|
|
|
|
async def test_what_after_send_appends_is_not_sent_twice_on_the_non_streamed_path(tmp_path):
|
|
loop = AgentLoop(
|
|
provider=_FakeChatProvider([LLMResponse(content="Hello", finish_reason="stop")]),
|
|
workspace=tmp_path,
|
|
host=HostWiring(hooks=[_Announce()]),
|
|
)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain, stream=False)
|
|
|
|
texts = [e.content for e in sink.events if isinstance(e, EvText)]
|
|
assert texts == ["Hello\n\nDeck: /out/deck.pptx"]
|
|
assert not any(isinstance(e, EvStreamDelta) for e in sink.events)
|
|
|
|
|
|
async def test_run_emits_reasoning_then_stream(tmp_path):
|
|
chunks = [
|
|
ChatDelta(content=None, reasoning_content="think"),
|
|
ChatDelta(content="answer"),
|
|
]
|
|
loop = AgentLoop(provider=_FakeStreamProvider(chunks), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
assert isinstance(sink.events[0], EvEpisodeStart)
|
|
reasoning = next(e for e in sink.events if isinstance(e, EvReasoning))
|
|
assert reasoning.content == "think"
|
|
assert any(isinstance(e, EvStreamDelta) and e.delta == "answer" for e in sink.events)
|
|
assert not any(isinstance(e, EvText) for e in sink.events)
|
|
|
|
|
|
async def test_run_tool_call_emits_tool_events_and_notice(tmp_path):
|
|
# Tool-call turn under run(): emit(ToolEvent start) + emit(Notice tool_hint) +
|
|
# emit(ToolEvent complete), then the final answer streams + dissolves.
|
|
# The ToolEvent schema: start carries tool_call_id/name/
|
|
# arguments, complete carries tool_call_id/result_preview/truncated.
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "t1", "function": {"name": "faketool", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
],
|
|
[ChatDelta(content="done")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_FakeTool())
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
tool_events = [e for e in sink.events if isinstance(e, EvToolEvent)]
|
|
start = next(e for e in tool_events if e.phase == ToolPhase.START)
|
|
assert start.tool_call_id == "t1" and start.name == "faketool" and start.arguments == {}
|
|
complete = next(e for e in tool_events if e.phase == ToolPhase.COMPLETE)
|
|
assert complete.tool_call_id == "t1" and complete.result_preview == "tool-ran"
|
|
assert complete.truncated is False
|
|
# The tool-call hint rides NoticeKind.TOOL_HINT (kept distinct from PROGRESS so
|
|
# an outlet gates it on send_tool_hints), not merged into PROGRESS.
|
|
assert any(isinstance(e, EvNotice) and e.kind is NoticeKind.TOOL_HINT for e in sink.events)
|
|
assert any(isinstance(e, EvStreamDelta) and e.delta == "done" for e in sink.events)
|
|
assert not any(isinstance(e, EvText) for e in sink.events) # streamed final dissolves
|
|
|
|
|
|
@pytest.mark.parametrize("blocking", [True, False])
|
|
async def test_tool_start_event_carries_the_registry_blocking_verdict(tmp_path, blocking):
|
|
"""ToolEvent(START).blocking must mirror the registered tool's own flag.
|
|
|
|
This is the whole chain the web channel depends on: registry -> loop payload
|
|
-> ToolEvent -> wire. A blocking tool runs a sub-agent with no automatic
|
|
deadline and emits nothing until it finishes, so a client that clocks the
|
|
stream has to learn from this field that the silence is expected.
|
|
"""
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "t1", "function": {"name": "faketool", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
],
|
|
[ChatDelta(content="done")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
tool = _FakeTool()
|
|
tool.blocking_interaction = blocking
|
|
loop.tools.register(tool)
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
start = next(e for e in sink.events if isinstance(e, EvToolEvent) and e.phase == ToolPhase.START)
|
|
assert start.blocking is blocking
|
|
|
|
|
|
async def test_tool_result_splits_model_text_from_display_preview(tmp_path):
|
|
# The ToolResult branch: the model's context gets model_text while the UI
|
|
# event's preview carries display_text. Every other fake tool in this file
|
|
# returns a bare str, so without this the unwrap branch never runs.
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "t9", "function": {"name": "splittool", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
],
|
|
[ChatDelta(content="done")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_SplitTool())
|
|
|
|
recorded: list[tuple[str, str, str]] = []
|
|
original_add = loop.context.add_tool_result
|
|
|
|
def _record(messages, tool_call_id, tool_name, result, *, trusted_note=""):
|
|
recorded.append((tool_call_id, tool_name, result))
|
|
return original_add(messages, tool_call_id, tool_name, result)
|
|
|
|
loop.context.add_tool_result = _record # type: ignore[method-assign]
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
# What the model reads.
|
|
assert recorded and recorded[0][:2] == ("t9", "splittool")
|
|
assert recorded[0][2] == _SplitTool.MODEL_TEXT
|
|
# What the transcript shows -- display_text, not the model sentence.
|
|
complete = next(e for e in sink.events if isinstance(e, EvToolEvent) and e.phase is ToolPhase.COMPLETE)
|
|
assert complete.result_preview == _SplitTool.DISPLAY_TEXT
|
|
assert complete.truncated is False
|
|
# display_call rides tool.start so the row can label itself.
|
|
start = next(e for e in sink.events if isinstance(e, EvToolEvent) and e.phase is ToolPhase.START)
|
|
assert start.display == "Which base?"
|
|
|
|
|
|
async def test_run_emits_one_episode_start_per_model_call(tmp_path):
|
|
# Episode boundary: run() emits one EpisodeStart per model call, 0-based and
|
|
# increasing. This turn has two calls (a tool-call iteration, then a stop
|
|
# iteration), so indices are [0, 1], and the first boundary precedes the
|
|
# first tool event of that call.
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "t1", "function": {"name": "faketool", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
],
|
|
[ChatDelta(content="done")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_FakeTool())
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
episodes = [e for e in sink.events if isinstance(e, EvEpisodeStart)]
|
|
assert [e.index for e in episodes] == [0, 1]
|
|
first_ep = next(i for i, e in enumerate(sink.events) if isinstance(e, EvEpisodeStart))
|
|
first_tool = next(i for i, e in enumerate(sink.events) if isinstance(e, EvToolEvent))
|
|
assert first_ep < first_tool
|
|
|
|
|
|
async def test_inject_message_merged_before_next_iteration(tmp_path):
|
|
# BusyPolicy.INJECT: a message injected mid-turn is drained at the next
|
|
# iteration's top and appended as a user message before that LLM call.
|
|
class _RecordingStreamToolProvider:
|
|
def __init__(self, scripts):
|
|
self._scripts = scripts
|
|
self._i = 0
|
|
self.calls: list[list[dict]] = []
|
|
|
|
async def chat_stream(self, **kwargs):
|
|
self.calls.append(list(kwargs.get("messages") or []))
|
|
script = self._scripts[min(self._i, len(self._scripts) - 1)]
|
|
self._i += 1
|
|
for chunk in script:
|
|
yield chunk
|
|
|
|
def get_default_model(self) -> str:
|
|
return "fake/model"
|
|
|
|
provider = _RecordingStreamToolProvider(
|
|
[
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "t1", "function": {"name": "faketool", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
],
|
|
[ChatDelta(content="done")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_FakeTool())
|
|
|
|
injects: list[list] = [[], [_req("also check the logs")]]
|
|
n = 0
|
|
|
|
def _drain_inject() -> list:
|
|
nonlocal n
|
|
out = injects[n] if n < len(injects) else []
|
|
n += 1
|
|
return out
|
|
|
|
await loop.run_turn(_req("start"), _EmitCollector(), _drain_inject, stream=True)
|
|
|
|
assert len(provider.calls) >= 2 # the tool call drove a second iteration
|
|
second = provider.calls[1]
|
|
assert any(m.get("role") == "user" and "also check the logs" in str(m.get("content", "")) for m in second), (
|
|
f"injected message not merged into the second iteration: {second}"
|
|
)
|
|
|
|
|
|
class _RecordingStreamProvider:
|
|
"""Plays one scripted stream per iteration and keeps every call's messages."""
|
|
|
|
def __init__(self, scripts):
|
|
self._scripts = scripts
|
|
self._i = 0
|
|
self.calls: list[list[dict]] = []
|
|
|
|
async def chat_stream(self, **kwargs):
|
|
self.calls.append(list(kwargs.get("messages") or []))
|
|
script = self._scripts[min(self._i, len(self._scripts) - 1)]
|
|
self._i += 1
|
|
for chunk in script:
|
|
yield chunk
|
|
|
|
def get_default_model(self) -> str:
|
|
return "fake/model"
|
|
|
|
|
|
def _stream_scripts(gaps: int) -> list[list[ChatDelta]]:
|
|
"""A tool call per gap -- each one drives another iteration -- then the answer."""
|
|
tool_calls = [
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": f"t{i}", "function": {"name": "faketool", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
]
|
|
for i in range(gaps)
|
|
]
|
|
return [*tool_calls, [ChatDelta(content="done")]]
|
|
|
|
|
|
def _labelled(messages: list[dict]) -> list[dict]:
|
|
return [m for m in messages if str(m.get("content") or "").startswith(_MID_TURN_HEADER)]
|
|
|
|
|
|
def _gap_drain(gaps: list[list]):
|
|
"""A drain that answers one gap's arrivals per iteration, in order."""
|
|
state = {"n": 0}
|
|
|
|
def _drain_inject() -> list:
|
|
out = gaps[state["n"]] if state["n"] < len(gaps) else []
|
|
state["n"] += 1
|
|
return out
|
|
|
|
return _drain_inject
|
|
|
|
|
|
async def test_mid_turn_messages_reach_the_model_as_one_labelled_message(tmp_path):
|
|
# Two corrections typed into one gap are one user message on the wire, under
|
|
# the header: as two bare user messages the model reads the second as a new
|
|
# question and answers that instead of steering the work it is doing.
|
|
provider = _RecordingStreamProvider(_stream_scripts(1))
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_FakeTool())
|
|
|
|
drain = _gap_drain([[], [_req("wait, skip the 2023 numbers"), _req("Q4 only")]])
|
|
await loop.run_turn(_req("summarise the report"), _EmitCollector(), drain, stream=True)
|
|
|
|
second = provider.calls[1]
|
|
labelled = _labelled(second)
|
|
assert len(labelled) == 1, second
|
|
assert labelled[0]["role"] == "user"
|
|
# In the order they arrived, joined by a blank line, under the one header.
|
|
assert labelled[0]["content"] == f"{_MID_TURN_HEADER}\n\nwait, skip the 2023 numbers\n\nQ4 only"
|
|
assert len([m for m in second if m.get("role") == "user"]) == 2, "the question, and the two arrivals as one"
|
|
|
|
|
|
async def test_a_single_mid_turn_message_is_labelled_too(tmp_path):
|
|
# The header is what says "this arrived while you worked", so it is not an
|
|
# artifact of there being two: one correction needs it just as much.
|
|
provider = _RecordingStreamProvider(_stream_scripts(1))
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_FakeTool())
|
|
|
|
drain = _gap_drain([[], [_req("only the last quarter")]])
|
|
await loop.run_turn(_req("summarise the report"), _EmitCollector(), drain, stream=True)
|
|
|
|
assert [m["content"] for m in _labelled(provider.calls[1])] == [f"{_MID_TURN_HEADER}\n\nonly the last quarter"]
|
|
|
|
|
|
async def test_a_mid_turn_attachment_the_message_already_names_is_not_named_twice(tmp_path):
|
|
# The page bakes its own attachment note into the text and derives `media`
|
|
# from it, so the drain's note repeated the path -- and the reader, whose
|
|
# live bubble showed the bare sentence, came back from a reload to an
|
|
# absolute path written into their own words.
|
|
provider = _RecordingStreamProvider(_stream_scripts(1))
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_FakeTool())
|
|
|
|
typed = "please look at the attached note\n\n[attachments]\n- /tmp/rt_note.txt"
|
|
drain = _gap_drain([[], [_req(typed, media=(Media(path="/tmp/rt_note.txt", mime="text/plain", kind="file"),))]])
|
|
await loop.run_turn(_req("summarise the report"), _EmitCollector(), drain, stream=True)
|
|
|
|
assert [m["content"] for m in _labelled(provider.calls[1])] == [f"{_MID_TURN_HEADER}\n\n{typed}"]
|
|
|
|
|
|
async def test_a_mid_turn_attachment_the_message_does_not_name_is_still_named(tmp_path):
|
|
# The other half: a sender that hands over a file without naming it -- a
|
|
# channel's own intake -- must still have it reach the model.
|
|
provider = _RecordingStreamProvider(_stream_scripts(1))
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_FakeTool())
|
|
|
|
drain = _gap_drain(
|
|
[[], [_req("look at this", media=(Media(path="/tmp/rt_note.txt", mime="text/plain", kind="file"),))]]
|
|
)
|
|
await loop.run_turn(_req("summarise the report"), _EmitCollector(), drain, stream=True)
|
|
|
|
assert [m["content"] for m in _labelled(provider.calls[1])] == [
|
|
f"{_MID_TURN_HEADER}\n\nlook at this\n[injected message; attached files: /tmp/rt_note.txt]"
|
|
]
|
|
|
|
|
|
async def test_mid_turn_messages_from_different_gaps_stay_apart(tmp_path):
|
|
# Folding is per adjacent run, not per turn: the work between two gaps is
|
|
# what the second correction is about, so merging across it would present
|
|
# the two as one thought and place them both before that work.
|
|
provider = _RecordingStreamProvider(_stream_scripts(2))
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_FakeTool())
|
|
|
|
drain = _gap_drain([[], [_req("skip the 2023 numbers")], [_req("Q4 only")]])
|
|
await loop.run_turn(_req("summarise the report"), _EmitCollector(), drain, stream=True)
|
|
|
|
third = provider.calls[2]
|
|
assert [m["content"] for m in _labelled(third)] == [
|
|
f"{_MID_TURN_HEADER}\n\nskip the 2023 numbers",
|
|
f"{_MID_TURN_HEADER}\n\nQ4 only",
|
|
]
|
|
|
|
|
|
async def test_autofill_rows_are_written_in_as_a_tool_call(tmp_path):
|
|
loop = AgentLoop(provider=_FakeChatProvider([]), workspace=tmp_path)
|
|
messages: list[dict] = []
|
|
loop._flush_autofill(
|
|
messages,
|
|
[
|
|
{
|
|
"agent": "raven-code",
|
|
"instance": "a1b2",
|
|
"summary": "Which branch? -> feat/x (answered for you)",
|
|
}
|
|
],
|
|
)
|
|
assert messages[0]["role"] == "assistant"
|
|
assert messages[0]["tool_calls"][0]["function"]["name"] == "answer_for_user"
|
|
assert messages[1]["role"] == "tool"
|
|
# The pairing the Chat Completions transport rejects when it is broken.
|
|
assert messages[1]["tool_call_id"] == messages[0]["tool_calls"][0]["id"]
|
|
assert "feat/x" in messages[1]["content"]
|
|
# Host-minted fields only. The sub-agent's own wording would arrive here
|
|
# unfenced -- nothing wraps an assistant message's arguments, while the
|
|
# summary above went through `add_tool_result` and is wrapped.
|
|
arguments = json.loads(messages[0]["tool_calls"][0]["function"]["arguments"])
|
|
assert arguments == {"agent": "raven-code", "instance": "a1b2"}
|
|
|
|
|
|
async def test_no_rows_writes_nothing(tmp_path):
|
|
loop = AgentLoop(provider=_FakeChatProvider([]), workspace=tmp_path)
|
|
messages: list[dict] = []
|
|
loop._flush_autofill(messages, [])
|
|
assert messages == []
|
|
|
|
|
|
class _RecordingAutofill:
|
|
"""Stands in for the turn's Autofill.
|
|
|
|
``set_snapshot`` keeps a copy rather than the live reference the real one
|
|
stores, so a test can tell what the list held at publish time.
|
|
"""
|
|
|
|
def __init__(self) -> None:
|
|
self._rows: list[dict] = []
|
|
self.snapshots: list[list[dict]] = []
|
|
|
|
def record(self, row: dict) -> None:
|
|
self._rows.append(row)
|
|
|
|
def pending_rows(self) -> list[dict]:
|
|
rows, self._rows = self._rows, []
|
|
return rows
|
|
|
|
def set_snapshot(self, messages: list[dict]) -> None:
|
|
self.snapshots.append(list(messages))
|
|
|
|
|
|
async def test_a_row_recorded_during_a_tool_call_reaches_the_next_llm_call(tmp_path):
|
|
# The seam: a sub-agent's question is answered while its spawn tool is still
|
|
# running, which is why the row can only be written at the next iteration's
|
|
# top -- splicing it in mid-batch would orphan the running tool_call.
|
|
from raven.acp_client.asker import start_ask_turn
|
|
|
|
auto = _RecordingAutofill()
|
|
|
|
class _AskingTool(_FakeTool):
|
|
"""A spawn stand-in: its sub-agent asks while the tool is still running."""
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
auto.record(
|
|
{
|
|
"agent": "raven-code",
|
|
"instance": "a1b2",
|
|
"summary": "Which branch? -> feat/x (answered for you)",
|
|
}
|
|
)
|
|
return "spawned"
|
|
|
|
class _RecordingStreamToolProvider:
|
|
def __init__(self, scripts):
|
|
self._scripts = scripts
|
|
self._i = 0
|
|
self.calls: list[list[dict]] = []
|
|
|
|
async def chat_stream(self, **kwargs):
|
|
self.calls.append(list(kwargs.get("messages") or []))
|
|
script = self._scripts[min(self._i, len(self._scripts) - 1)]
|
|
self._i += 1
|
|
for chunk in script:
|
|
yield chunk
|
|
|
|
def get_default_model(self) -> str:
|
|
return "fake/model"
|
|
|
|
provider = _RecordingStreamToolProvider(
|
|
[
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "t1", "function": {"name": "faketool", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
],
|
|
[ChatDelta(content="done")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_AskingTool())
|
|
start_ask_turn(None, auto, conversation_id="tui:c1")
|
|
|
|
await loop.run_turn(_req("start"), _EmitCollector(), _drain, stream=True)
|
|
|
|
def _row(messages: list[dict]) -> dict | None:
|
|
return next((m for m in messages if m.get("role") == "tool" and m.get("name") == "answer_for_user"), None)
|
|
|
|
assert _row(provider.calls[0]) is None # nothing had been answered yet
|
|
row = _row(provider.calls[1])
|
|
assert row is not None, f"autofill row not written into the second iteration: {provider.calls[1]}"
|
|
assert "feat/x" in row["content"]
|
|
# Published on every iteration, not once, and after the flush -- so the
|
|
# second publish carries what the first turn's questions were answered with.
|
|
assert len(auto.snapshots) == 2
|
|
assert _row(auto.snapshots[0]) is None
|
|
assert _row(auto.snapshots[1]) is not None
|
|
|
|
|
|
async def test_run_slash_emits_text_not_streamed(tmp_path):
|
|
# /help is an early return (no LLM stream) -> streamed=False -> emit(Text).
|
|
loop = AgentLoop(provider=_FakeChatProvider([]), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
outcome = await loop.run_turn(_req("/help"), sink, _drain)
|
|
|
|
texts = [e for e in sink.events if isinstance(e, EvText)]
|
|
assert len(texts) == 1 and "Raven commands" in texts[0].content
|
|
assert not any(isinstance(e, EvStreamDelta) for e in sink.events)
|
|
assert outcome.explicit_reply is True
|
|
|
|
|
|
async def test_run_short_circuit_emits_media_before_text(tmp_path):
|
|
# MediaOut category: a hook short-circuit returns media + content. MediaOut is
|
|
# independent of the stream and precedes Text.
|
|
async def _decision(req: TurnRequest):
|
|
return _Reply(
|
|
channel=req.source.channel,
|
|
chat_id=req.source.chat_id,
|
|
content="short",
|
|
media=["/tmp/x.png"],
|
|
)
|
|
|
|
loop = AgentLoop(
|
|
provider=_FakeChatProvider([]), workspace=tmp_path, host=HostWiring(hooks=[DecisionConsumerAdapter(_decision)])
|
|
)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
kinds = [type(e).__name__ for e in sink.events]
|
|
assert kinds == ["MediaOut", "Text"] # media first, then text
|
|
assert sink.events[0].media[0].path == "/tmp/x.png"
|
|
assert sink.events[1].content == "short"
|
|
|
|
|
|
async def test_run_propagates_sandbox_error_not_error_string(tmp_path):
|
|
# Unlike the old string-returning path (returned "[Sandbox error]"), run() lets the exception
|
|
# propagate so the lane turns it into TurnFailed.
|
|
loop = AgentLoop(provider=_FakeChatProvider([]), workspace=tmp_path)
|
|
|
|
async def _boom() -> None:
|
|
raise SandboxInitError("test: sandbox down")
|
|
|
|
loop._start_executor = _boom
|
|
sink = _EmitCollector()
|
|
|
|
with pytest.raises(SandboxInitError):
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
assert not any(isinstance(e, EvText) for e in sink.events)
|
|
|
|
|
|
async def test_run_propagates_mid_turn_error_not_sorry_text(tmp_path):
|
|
# N-TURNFAILED second source: a mid-turn provider error propagates out of
|
|
# _process_message (the "Sorry" catch lives in _dispatch, the bus wrapper run()
|
|
# does not use) -> run() re-raises -> the lane makes TurnFailed, not a Text.
|
|
class _BoomStreamProvider:
|
|
async def chat_stream(self, **kwargs):
|
|
raise RuntimeError("mid-turn boom")
|
|
yield # unreachable; makes this an async generator
|
|
|
|
def get_default_model(self) -> str:
|
|
return "fake/model"
|
|
|
|
loop = AgentLoop(provider=_BoomStreamProvider(), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
with pytest.raises(RuntimeError):
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
assert not any(isinstance(e, EvText) for e in sink.events)
|
|
|
|
|
|
def _message_tool_call(arguments: str) -> ChatDelta:
|
|
return ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "m1", "function": {"name": "message", "arguments": arguments}}]
|
|
},
|
|
)
|
|
|
|
|
|
async def test_run_message_tool_text_streams_and_dissolves(tmp_path):
|
|
# The message tool's reply routes through on_token -> ChatDelta (b2), then
|
|
# _process_message returns None -> no trailing Text. explicit_reply is still
|
|
# True (the agent did reply via the tool).
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[_message_tool_call('{"content": "hi via tool"}')],
|
|
[ChatDelta(content="")], # second iteration: nothing more, finish
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
outcome = await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
assert any(isinstance(e, EvStreamDelta) and e.delta == "hi via tool" for e in sink.events)
|
|
assert not any(isinstance(e, EvText) for e in sink.events) # tool reply dissolves
|
|
assert outcome.explicit_reply is True
|
|
|
|
|
|
async def test_run_message_tool_reply_outlives_a_failed_follow_up_call(tmp_path):
|
|
# The tool delivered the reply; the model call after it failed for good. The
|
|
# turn is answered, so it ends as it always has -- no AnswerlessTurnError, no
|
|
# error text delivered as a second reply.
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[_message_tool_call('{"content": "hi via tool"}')],
|
|
[
|
|
ChatDelta(
|
|
content="Error calling LLM (network@fake): boom",
|
|
finish_reason="error",
|
|
error_classification=ErrorClassification("network", retryable=False),
|
|
)
|
|
],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
outcome = await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
assert outcome.explicit_reply is True
|
|
assert not any(isinstance(e, EvText) for e in sink.events)
|
|
assert not any(isinstance(e, EvStreamDelta) and "Error calling LLM" in e.delta for e in sink.events)
|
|
|
|
|
|
async def test_run_message_tool_media_is_not_dropped(tmp_path):
|
|
# Regression guard: a message-tool reply carrying media must emit MediaOut
|
|
# (media is independent of the token stream). The tool path returns None from
|
|
# _process_message, so the return boundary never sees it — _route_to_stream
|
|
# must emit the media itself, matching what the bus path delivers.
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[_message_tool_call('{"content": "see this", "media": ["/tmp/pic.png"]}')],
|
|
[ChatDelta(content="")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
outcome = await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
media = [e for e in sink.events if isinstance(e, EvMediaOut)]
|
|
assert media and media[0].media[0].path == "/tmp/pic.png"
|
|
assert any(isinstance(e, EvStreamDelta) and e.delta == "see this" for e in sink.events)
|
|
assert outcome.explicit_reply is True
|
|
|
|
|
|
# ── stream=False (REPL assembly): reply is one Text, no ChatDelta ──
|
|
|
|
|
|
async def test_run_stream_false_main_reply_is_one_text(tmp_path):
|
|
# build_one_shot_spine wires stream=False -> non-streaming chat_with_retry -> the reply
|
|
# is one Text (CliOutlet renders it), never a ChatDelta.
|
|
provider = _FakeChatProvider([LLMResponse(content="full reply", finish_reason="stop")])
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
outcome = await loop.run_turn(_req("hi"), sink, _drain, stream=False)
|
|
|
|
texts = [e for e in sink.events if isinstance(e, EvText)]
|
|
assert len(texts) == 1 and texts[0].content == "full reply"
|
|
assert not any(isinstance(e, EvStreamDelta) for e in sink.events)
|
|
assert outcome.explicit_reply is True
|
|
|
|
|
|
async def test_run_stream_false_message_tool_emits_text(tmp_path):
|
|
# The message-tool reply under stream=False must emit Text, not ChatDelta —
|
|
# else a non-streaming outlet (CliOutlet) would eat the delta and the REPL
|
|
# would go silent for tool replies.
|
|
provider = _FakeChatProvider(
|
|
[
|
|
LLMResponse(
|
|
content=None,
|
|
tool_calls=[ToolCallRequest(id="m1", name="message", arguments={"content": "hi via tool"})],
|
|
finish_reason="tool_calls",
|
|
),
|
|
LLMResponse(content="", finish_reason="stop"),
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
outcome = await loop.run_turn(_req("hi"), sink, _drain, stream=False)
|
|
|
|
assert any(isinstance(e, EvText) and e.content == "hi via tool" for e in sink.events)
|
|
assert not any(isinstance(e, EvStreamDelta) for e in sink.events)
|
|
assert outcome.explicit_reply is True
|
|
|
|
|
|
# ── origin gates the user-inbound hook (engagement); magic-key fallback ──
|
|
|
|
|
|
def _hook_loop(tmp_path):
|
|
"""An AgentLoop whose decision_consumer short-circuits with 'hook-fired' when
|
|
the user-inbound hook runs; otherwise the (streaming) LLM reply 'llm' streams.
|
|
So the hook firing vs being skipped is observable from the output."""
|
|
|
|
async def _decision(req: TurnRequest):
|
|
return _Reply(
|
|
channel=req.source.channel,
|
|
chat_id=req.source.chat_id,
|
|
content="hook-fired",
|
|
)
|
|
|
|
loop = AgentLoop(
|
|
provider=_FakeStreamProvider([ChatDelta(content="llm")]),
|
|
workspace=tmp_path,
|
|
host=HostWiring(hooks=[DecisionConsumerAdapter(_decision)]),
|
|
)
|
|
_stub_edges(loop)
|
|
return loop
|
|
|
|
|
|
async def test_run_turn_user_origin_fires_user_inbound_hook(tmp_path):
|
|
sink = _EmitCollector()
|
|
await _hook_loop(tmp_path).run_turn(_req("hi", origin=Origin.USER), sink, _drain)
|
|
# USER -> hook fires -> short-circuits, the LLM is never reached.
|
|
assert any(isinstance(e, EvText) and e.content == "hook-fired" for e in sink.events)
|
|
assert not any(isinstance(e, EvStreamDelta) for e in sink.events)
|
|
|
|
|
|
async def test_run_turn_cron_origin_fires_hook_like_user(tmp_path):
|
|
# D-ENGAGE replicate: cron is NOT proactive-suppressed -> hook fires (as today).
|
|
sink = _EmitCollector()
|
|
await _hook_loop(tmp_path).run_turn(_req("tick", origin=Origin.CRON), sink, _drain)
|
|
assert any(isinstance(e, EvText) and e.content == "hook-fired" for e in sink.events)
|
|
|
|
|
|
async def test_run_turn_sentinel_origin_suppresses_user_inbound_hook(tmp_path):
|
|
sink = _EmitCollector()
|
|
await _hook_loop(tmp_path).run_turn(_req("nudge", origin=Origin.SENTINEL), sink, _drain)
|
|
# SENTINEL -> hook suppressed -> proceeds to the LLM (no short-circuit).
|
|
assert any(isinstance(e, EvStreamDelta) and e.delta == "llm" for e in sink.events)
|
|
assert not any(isinstance(e, EvText) and e.content == "hook-fired" for e in sink.events)
|
|
|
|
|
|
async def test_run_turn_subagent_origin_suppresses_user_inbound_hook(tmp_path):
|
|
sink = _EmitCollector()
|
|
await _hook_loop(tmp_path).run_turn(_req("result", origin=Origin.SUBAGENT), sink, _drain)
|
|
assert any(isinstance(e, EvStreamDelta) and e.delta == "llm" for e in sink.events)
|
|
assert not any(isinstance(e, EvText) and e.content == "hook-fired" for e in sink.events)
|
|
|
|
|
|
async def _process_via_chat(loop, msg):
|
|
# _process_message with no callbacks -> non-streaming chat_with_retry path.
|
|
return await loop._process_message(msg)
|
|
|
|
|
|
async def test_process_message_origin_none_plain_fires_hook(tmp_path):
|
|
# The legacy path passed origin=None; the safe default
|
|
# (origin not in _SKIP_*) treats it as a user inbound and fires the hook.
|
|
async def _decision(req: TurnRequest):
|
|
return _Reply(
|
|
channel=req.source.channel,
|
|
chat_id=req.source.chat_id,
|
|
content="hook-fired",
|
|
)
|
|
|
|
loop = AgentLoop(
|
|
provider=_FakeChatProvider([LLMResponse(content="llm", finish_reason="stop")]),
|
|
workspace=tmp_path,
|
|
host=HostWiring(hooks=[DecisionConsumerAdapter(_decision)]),
|
|
)
|
|
_stub_edges(loop)
|
|
out = await _process_via_chat(loop, _req("hi"))
|
|
assert out is not None
|
|
content, _media = out
|
|
assert content == "hook-fired"
|
|
|
|
|
|
async def test_run_turn_reconstructs_metadata_from_source_extras(tmp_path):
|
|
# channel metadata rides Source.extras; run_turn reconstructs it into
|
|
# the turn metadata so consumers (here _set_tool_context, which reads
|
|
# message_id for reply threading) still see it.
|
|
loop = AgentLoop(
|
|
provider=_FakeChatProvider([LLMResponse(content="ok", finish_reason="stop", tool_calls=[])]),
|
|
workspace=tmp_path,
|
|
)
|
|
_stub_edges(loop)
|
|
seen: dict = {}
|
|
real = loop._set_tool_context
|
|
|
|
def _spy(channel, chat_id, message_id=None, session_key=None):
|
|
seen["message_id"] = message_id
|
|
return real(channel, chat_id, message_id, session_key=session_key)
|
|
|
|
loop._set_tool_context = _spy
|
|
|
|
req = TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(
|
|
channel="tg",
|
|
chat_id="c",
|
|
sender_id="u",
|
|
chat_type=ChatType.DM,
|
|
extras={"message_id": "m1"},
|
|
),
|
|
text="hi",
|
|
)
|
|
await loop.run_turn(req, _EmitCollector(), _drain, stream=False)
|
|
assert seen.get("message_id") == "m1" # extras -> metadata -> _set_tool_context
|
|
|
|
|
|
# ── deliver_text: verbatim background delivery, skips the model (2b) ──
|
|
|
|
|
|
async def test_run_turn_deliver_text_emits_verbatim_skips_model_and_indexes(tmp_path):
|
|
# A background task pushes a finished result back via deliver_text: run_turn
|
|
# emits it as one Text verbatim, records it to the session, and hands it to
|
|
# the backend index — and never touches the model.
|
|
class _NoCallProvider:
|
|
async def chat_stream(self, **kwargs):
|
|
raise AssertionError("model must not be called for a deliver_text turn")
|
|
yield # unreachable; keeps this an async generator
|
|
|
|
async def chat_with_retry(self, **kwargs):
|
|
raise AssertionError("model must not be called for a deliver_text turn")
|
|
|
|
def get_default_model(self) -> str:
|
|
return "fake/model"
|
|
|
|
stored: list = []
|
|
|
|
class _FakeBackend:
|
|
async def store(self, key, messages, *, metadata=None) -> None:
|
|
stored.append((key, messages))
|
|
|
|
loop = AgentLoop(provider=_NoCallProvider(), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.backend = _FakeBackend()
|
|
sink = _EmitCollector()
|
|
|
|
req = TurnRequest(
|
|
origin=Origin.SUBAGENT,
|
|
source=Source(channel="weixin", chat_id="c", sender_id="subagent", chat_type=ChatType.DM),
|
|
text="",
|
|
conversation="weixin:c",
|
|
deliver_text="FULL REPORT [1]",
|
|
)
|
|
outcome = await loop.run_turn(req, sink, _drain, stream=False)
|
|
|
|
texts = [e for e in sink.events if isinstance(e, EvText)]
|
|
assert len(texts) == 1 and texts[0].content == "FULL REPORT [1]"
|
|
assert not any(isinstance(e, EvStreamDelta) for e in sink.events) # not streamed through the model
|
|
assert outcome.explicit_reply is True
|
|
|
|
session = loop.sessions.get_or_create("weixin:c")
|
|
assert any(m.get("role") == "assistant" and m.get("content") == "FULL REPORT [1]" for m in session.messages)
|
|
await loop.drain_backend_stores(timeout=5.0)
|
|
assert stored and stored[0][0] == "weixin:c"
|
|
assert stored[0][1][0]["content"] == "FULL REPORT [1]"
|
|
|
|
|
|
async def test_run_turn_deliver_text_persists_before_emit(tmp_path):
|
|
# save-then-send order: if the session save fails, nothing was emitted, so
|
|
# the user is never left with a delivered message that no turn recorded.
|
|
loop = AgentLoop(provider=_FakeChatProvider([]), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
|
|
def _boom(_session) -> None:
|
|
raise RuntimeError("save failed")
|
|
|
|
loop.sessions.save = _boom
|
|
sink = _EmitCollector()
|
|
|
|
req = TurnRequest(
|
|
origin=Origin.SUBAGENT,
|
|
source=Source(channel="weixin", chat_id="c", sender_id="subagent", chat_type=ChatType.DM),
|
|
text="",
|
|
conversation="weixin:c",
|
|
deliver_text="REPORT",
|
|
)
|
|
with pytest.raises(RuntimeError):
|
|
await loop.run_turn(req, sink, _drain, stream=False)
|
|
assert not any(isinstance(e, EvText) for e in sink.events) # save failed -> nothing delivered
|
|
|
|
|
|
async def test_run_turn_empty_extras_reconstructs_empty_metadata(tmp_path):
|
|
# No-regression: a host source carries no extras -> metadata={} ->
|
|
# _set_tool_context sees message_id=None.
|
|
loop = AgentLoop(
|
|
provider=_FakeChatProvider([LLMResponse(content="ok", finish_reason="stop", tool_calls=[])]),
|
|
workspace=tmp_path,
|
|
)
|
|
_stub_edges(loop)
|
|
seen: dict = {"message_id": "sentinel"}
|
|
|
|
def _spy(channel, chat_id, message_id=None, session_key=None):
|
|
seen["message_id"] = message_id
|
|
|
|
loop._set_tool_context = _spy
|
|
await loop.run_turn(_req("hi"), _EmitCollector(), _drain, stream=False)
|
|
assert seen["message_id"] is None # empty extras -> metadata={} -> no message_id
|
|
|
|
|
|
async def test_a_failing_display_call_costs_the_label_not_the_turn(tmp_path):
|
|
"""``display_call`` gets the model's raw arguments -- the registry's cast and
|
|
validation run later, on the execute path -- so it sees shapes the schema
|
|
forbids. It only labels a transcript row, yet an exception from it used to
|
|
leave the emit and end the turn with no reply at all: that is how a
|
|
JSON-encoded array argument took a turn down.
|
|
|
|
Driven through ``run_turn`` rather than against the helper, so that inlining
|
|
the guard back into the call site fails here instead of passing quietly.
|
|
"""
|
|
|
|
class _ExplodingLabel(Tool):
|
|
@property
|
|
def name(self) -> str:
|
|
return "exploding"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "raises from display_call"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}}
|
|
|
|
def display_call(self, args: dict) -> str | None:
|
|
raise AttributeError("'str' object has no attribute 'get'")
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
return "tool-ran"
|
|
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "t1", "function": {"name": "exploding", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
],
|
|
[ChatDelta(content="done")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_ExplodingLabel())
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
start = next(e for e in sink.events if isinstance(e, EvToolEvent) and e.phase is ToolPhase.START)
|
|
assert start.display is None # the label is what was lost
|
|
complete = next(e for e in sink.events if isinstance(e, EvToolEvent) and e.phase is ToolPhase.COMPLETE)
|
|
assert complete.result_preview == "tool-ran" # the tool still ran
|
|
assert any(isinstance(e, EvStreamDelta) and e.delta == "done" for e in sink.events) # the turn finished
|
|
|
|
|
|
async def test_a_working_display_call_still_labels_the_row(tmp_path):
|
|
class _LabelledTool(Tool):
|
|
@property
|
|
def name(self) -> str:
|
|
return "labelled"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "labels its row"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}}
|
|
|
|
def display_call(self, args: dict) -> str | None:
|
|
return "the label"
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
return "tool-ran"
|
|
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "t1", "function": {"name": "labelled", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
],
|
|
[ChatDelta(content="done")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_LabelledTool())
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
start = next(e for e in sink.events if isinstance(e, EvToolEvent) and e.phase is ToolPhase.START)
|
|
assert start.display == "the label"
|
|
|
|
|
|
async def test_run_turn_hands_the_mcp_connect_off_instead_of_awaiting_it(tmp_path):
|
|
"""The call site, not just the mechanism.
|
|
|
|
This replaces a test that asserted the turn passed a 90s bound. The bound is
|
|
gone with the wait: a server parked at the browser-authorization step blocks
|
|
on a human, and a turn that waits on it at all is the defect (measured once at
|
|
10m35s). The turn must reach ``prewarm_mcp`` and must not await a connect.
|
|
"""
|
|
prewarms = 0
|
|
awaited = 0
|
|
|
|
loop = AgentLoop(provider=_FakeStreamProvider([ChatDelta(content="hi")]), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
|
|
def _prewarm() -> None:
|
|
nonlocal prewarms
|
|
prewarms += 1
|
|
|
|
async def _connect() -> None: # pragma: no cover - reaching this is the failure
|
|
nonlocal awaited
|
|
awaited += 1
|
|
|
|
loop.prewarm_mcp = _prewarm
|
|
loop._connect_mcp = _connect
|
|
|
|
await loop.run_turn(_req("hi"), _EmitCollector(), _drain)
|
|
|
|
assert prewarms == 1
|
|
assert awaited == 0
|
|
|
|
|
|
# ── run_turn releases the conversation's foreground DAG runs when it ends ────────
|
|
|
|
|
|
class _ReleaseRecorder(Tool):
|
|
"""Stands in for run_subagent_dag: records release_turn calls."""
|
|
|
|
def __init__(self) -> None:
|
|
self.calls: list[tuple[str, bool]] = []
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "run_subagent_dag"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "records releases"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}, "required": []}
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
return "unused"
|
|
|
|
async def release_turn(self, conversation: str, *, flush: bool) -> None:
|
|
self.calls.append((conversation, flush))
|
|
|
|
|
|
def _loop_with_recorder(tmp_path):
|
|
from raven.agent.loop._shared import LoopOutcome
|
|
|
|
loop = AgentLoop(provider=_FakeChatProvider([]), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
recorder = _ReleaseRecorder()
|
|
loop.tools.register(recorder)
|
|
|
|
async def _ends_normally(req, emit, drain, **kwargs):
|
|
return LoopOutcome()
|
|
|
|
loop._run_turn = _ends_normally
|
|
return loop, recorder
|
|
|
|
|
|
async def test_run_turn_releases_the_conversations_dag_runs_when_it_ends(tmp_path):
|
|
loop, recorder = _loop_with_recorder(tmp_path)
|
|
|
|
await loop.run_turn(_req("hi"), _EmitCollector(), _drain)
|
|
|
|
assert recorder.calls == [("cli:c", True)]
|
|
|
|
|
|
async def test_a_cancelled_turn_releases_without_flushing(tmp_path):
|
|
import asyncio
|
|
|
|
loop, recorder = _loop_with_recorder(tmp_path)
|
|
|
|
async def _cancelled(req, emit, drain, **kwargs):
|
|
raise asyncio.CancelledError()
|
|
|
|
loop._run_turn = _cancelled
|
|
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await loop.run_turn(_req("hi"), _EmitCollector(), _drain)
|
|
|
|
assert recorder.calls == [("cli:c", False)]
|
|
|
|
|
|
async def test_a_turn_that_raises_still_releases_and_still_flushes(tmp_path):
|
|
"""The exit the other three left out, and the one the design doc names.
|
|
|
|
An ordinary error is not the user stopping the agent: the turn is over and its
|
|
graph's open questions are still worth re-sending, so this exit flushes exactly
|
|
as the normal one does. Only cancellation drops them.
|
|
|
|
Without it, an `except BaseException` that sets `flush = False` -- the plausible
|
|
way someone widens the cancellation arm -- turns every failed turn into a silent
|
|
drop with nothing red.
|
|
"""
|
|
loop, recorder = _loop_with_recorder(tmp_path)
|
|
|
|
async def _raises(req, emit, drain, **kwargs):
|
|
raise RuntimeError("the turn blew up")
|
|
|
|
loop._run_turn = _raises
|
|
|
|
with pytest.raises(RuntimeError, match="the turn blew up"):
|
|
await loop.run_turn(_req("hi"), _EmitCollector(), _drain)
|
|
|
|
assert recorder.calls == [("cli:c", True)], "an error end releases, and flushes: only a cancel drops"
|
|
|
|
|
|
async def test_a_direct_chat_turn_releases_nothing(tmp_path):
|
|
loop, recorder = _loop_with_recorder(tmp_path)
|
|
req = TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="cli", chat_id="c", sender_id="u", chat_type=ChatType.DM),
|
|
text="hi",
|
|
direct_target=("echo", "h1"),
|
|
)
|
|
|
|
await loop.run_turn(req, _EmitCollector(), _drain)
|
|
|
|
assert recorder.calls == []
|
|
|
|
|
|
# ── a rolled-back draft must not reach the stream ───────────────────
|
|
|
|
|
|
class _RollsBackTheFirstAnswer(AgentHook):
|
|
"""Sends the first tool-call-free response back once, as an engine hook does."""
|
|
|
|
rolls_back_iterations = True
|
|
|
|
def __init__(self) -> None:
|
|
self.rolled = 0
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "rollback_probe"
|
|
|
|
async def after_iteration(self, ctx):
|
|
response = ctx.response
|
|
if response is None or getattr(response, "tool_calls", None):
|
|
return HookDecision()
|
|
if self.rolled or not str(getattr(response, "content", "") or "").strip():
|
|
return HookDecision()
|
|
self.rolled += 1
|
|
return HookDecision(rollback=True, rollback_inject=[{"role": "user", "content": "try again"}])
|
|
|
|
|
|
async def test_a_rolled_back_draft_never_reaches_the_stream(tmp_path):
|
|
"""`rollback` pops history, and the stream has already left.
|
|
|
|
An `after_iteration` hook decides from state the response does not carry, so
|
|
it can only speak after the content has streamed. Popping the messages then
|
|
leaves the reader holding text the turn no longer has -- and on the ACP lane
|
|
the collector concatenates every chunk, so the rejected draft and its
|
|
replacement arrive as one answer. The deltas are held until something keeps
|
|
the response, which is what this pins.
|
|
"""
|
|
provider = _FakeStreamToolProvider([[ChatDelta(content="FIRST DRAFT")], [ChatDelta(content="FINAL REPLY")]])
|
|
hook = _RollsBackTheFirstAnswer()
|
|
loop = AgentLoop(
|
|
provider=provider,
|
|
workspace=tmp_path,
|
|
host=HostWiring(hooks=[hook]),
|
|
)
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
deltas = [e.delta for e in sink.events if isinstance(e, EvStreamDelta)]
|
|
assert hook.rolled == 1, "the hook never rolled anything back, so this proves nothing"
|
|
assert deltas == ["FINAL REPLY"], f"the caller saw the rejected draft: {deltas}"
|
|
|
|
|
|
class _PassThroughObserver(AgentHook):
|
|
"""`eval_engine`'s judge shape: overrides the phase, awaits inside it, never
|
|
sends anything back -- and says so, by leaving `rolls_back_iterations` alone."""
|
|
|
|
def __init__(self) -> None:
|
|
self.entered = 0
|
|
self.finished = 0
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "observer_probe"
|
|
|
|
async def after_iteration(self, ctx):
|
|
self.entered += 1
|
|
await asyncio.sleep(0) # an await inside the phase, as a judge call is
|
|
self.finished += 1
|
|
return HookDecision()
|
|
|
|
|
|
async def test_an_observer_that_cannot_roll_back_keeps_the_chunks(tmp_path):
|
|
"""The other half of holding, and the reason it is declared not inferred.
|
|
|
|
Overriding `after_iteration` says a hook watches the phase, not that it can
|
|
send the response back. `eval_engine`'s judge overrides it, awaits an LLM
|
|
call inside it, and its own contract says an evaluator never interrupts the
|
|
reply -- so inferring the capability from the override took the answer away
|
|
from the reader until the judge returned, for a draft that was never going
|
|
to be rejected.
|
|
"""
|
|
provider = _FakeStreamProvider([ChatDelta(content="Hel"), ChatDelta(content="lo")])
|
|
observer = _PassThroughObserver()
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path, host=HostWiring(hooks=[observer]))
|
|
_stub_edges(loop)
|
|
sink = _EmitCollector()
|
|
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
|
|
# Both counters, because CompositeHook swallows a hook exception: a body that
|
|
# raised past the await would leave the entered count alone and still pass.
|
|
assert observer.entered > 0, "the observer never ran, so this proves nothing"
|
|
assert observer.finished == observer.entered, "the observer raised inside the phase"
|
|
deltas = [e.delta for e in sink.events if isinstance(e, EvStreamDelta)]
|
|
assert deltas == ["Hel", "lo"], f"an observer that cannot roll back lost the reader the chunks: {deltas}"
|
|
|
|
|
|
class _LeakyTool(Tool):
|
|
"""Prints a key Raven holds, the way `cat ~/.raven/config.json` would."""
|
|
|
|
KEY = "sk-or-held-0123456789"
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "leakytool"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "fake tool whose output carries a held credential"
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}, "required": []}
|
|
|
|
async def execute(self, **kwargs) -> ToolResult:
|
|
return ToolResult(model_text=f'"apiKey": "{self.KEY}"', display_text=f"apiKey {self.KEY}")
|
|
|
|
|
|
async def test_a_held_key_reaches_neither_the_log_the_page_nor_the_model(tmp_path, monkeypatch):
|
|
"""The scrub used to run only where the model's message is built, after the
|
|
preview had been logged and sent to the page as `result_preview`."""
|
|
from loguru import logger
|
|
|
|
monkeypatch.setattr(
|
|
"raven.config.held_secrets.held_secrets", lambda: [(_LeakyTool.KEY, "providers.openrouter.apiKey")]
|
|
)
|
|
provider = _FakeStreamToolProvider(
|
|
[
|
|
[
|
|
ChatDelta(
|
|
content=None,
|
|
tool_call_delta={
|
|
"tool_calls": [{"index": 0, "id": "t7", "function": {"name": "leakytool", "arguments": "{}"}}]
|
|
},
|
|
)
|
|
],
|
|
[ChatDelta(content="done")],
|
|
]
|
|
)
|
|
loop = AgentLoop(provider=provider, workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
loop.tools.register(_LeakyTool())
|
|
recorded: list[str] = []
|
|
original_add = loop.context.add_tool_result
|
|
|
|
def _record(messages, tool_call_id, tool_name, result, *, trusted_note=""):
|
|
recorded.append(result)
|
|
return original_add(messages, tool_call_id, tool_name, result)
|
|
|
|
loop.context.add_tool_result = _record # type: ignore[method-assign]
|
|
logged: list[str] = []
|
|
handle = logger.add(lambda message: logged.append(str(message)), level="INFO")
|
|
try:
|
|
sink = _EmitCollector()
|
|
await loop.run_turn(_req("hi"), sink, _drain)
|
|
finally:
|
|
logger.remove(handle)
|
|
|
|
complete = next(e for e in sink.events if isinstance(e, EvToolEvent) and e.phase is ToolPhase.COMPLETE)
|
|
assert _LeakyTool.KEY not in complete.result_preview
|
|
assert "[redacted: providers.openrouter.apiKey]" in complete.result_preview
|
|
assert any("Tool result: leakytool" in line for line in logged)
|
|
assert not any(_LeakyTool.KEY in line for line in logged)
|
|
assert recorded and _LeakyTool.KEY not in recorded[0]
|
|
|
|
|
|
async def test_run_turn_hands_the_lane_the_long_account_of_a_crash(tmp_path):
|
|
"""Every agent turn's failure leaves through run_turn, so that is where the
|
|
exception gets the long account the kernel cannot build -- including a
|
|
crash raised before the turn path's own arms could file a marker."""
|
|
loop = AgentLoop(provider=_FakeChatProvider([]), workspace=tmp_path)
|
|
_stub_edges(loop)
|
|
message = "the setup fell over: " + "q" * 400
|
|
|
|
async def _crashes(req, emit, drain, **kwargs):
|
|
raise RuntimeError(message)
|
|
|
|
loop._run_turn = _crashes
|
|
with pytest.raises(RuntimeError) as raised:
|
|
await loop.run_turn(_req("hi"), _EmitCollector(), _drain)
|
|
|
|
assert raised.value.full_detail == f"RuntimeError: {message}"
|