Raven-1/tests/test_agent_loop_llm_error_patience.py
Dizhan Xue ccf1e2d08e
feat(*): let a failed turn's row open to its whole error (#978)
## Summary

A failed turn's Web row can expand to the provider's full diagnostic
text, up to 8,000 characters, while the existing bounded reason stays
unchanged. The optional `full_detail` field travels through the runner,
lane, RPC event and session marker, so live display and replay expose
the same account. Copy includes the expanded account. Quiet runtime
notices use "Show details" instead of the error label.

Failure details and call-record bodies remove known credential shapes
before the final cap. An unterminated private-key block is removed
through the end of the input, including when an earlier scan or call
record cut away its closing armour. Known cut boundaries also remove
incomplete JWTs, vendor tokens, URL passwords and labelled credential
tails; each of those patterns is searched only from behind the run of
its own value's characters that ends the input and the label in front of
it, so the scan stays linear whatever the text holds and still reads a
label gap of any width. The authorization and JWT patterns avoid
repeated backtracking, and the turn path and outer runner reuse the
computed detail. Attachment and lane reads use the base exception's
instance storage, so a foreign read-only property or throwing getter
cannot replace the scrubbed value or swallow the terminal failure event.

The optional fields are mirrored in the RPC schema and both generated
clients. The design document separates this delivery from the subsequent
Turn Continuation feature; Retry still starts an ordinary turn.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

The automated suites and static/build checks below were re-run after
synchronizing with main, using the existing local dependencies. The
independent behavioral and timing probes were measured before that
synchronization. The last three commits change only
`raven/security/redact.py`, two test files and the design document; on
that final tree the Python suite, the ui-web tests, the type, generation
and lint checks, `ty`, the import contracts, the range gates and
pre-commit were run again, and the ui-web build checks were not, since
none of their inputs changed.

- `uv run --no-sync pytest -q tests/test_agent_loop_failure_text.py
tests/test_llm_call_record.py tests/test_agent_loop_broken_turn.py
tests/test_agent_loop_llm_error_patience.py
tests/test_agent_loop_run_emit.py tests/test_spine_scheduler_lane.py
tests/test_spine_events.py tests/test_rpc_spine.py
tests/test_rpc_schema_match.py tests/test_kernel_budget.py
tests/test_acp_redact.py tests/test_security_untrusted_context.py -x
--tb=short --show-capture=no`: 904 passed.
- From `ui-web`: `npm test -- --no-file-parallelism
src/features/transcript/TranscriptPage.test.tsx
src/state/session/pipeline.events.test.ts
scripts/gates/i18n-keys.test.mjs`: 225 passed.
- From `ui-web`: `npm run type-check`, `npm run gen:check`, `node
../ui-tui/scripts/gen-i18n.mjs --check`, `node
../ui-tui/scripts/gen-rpc-types.mjs --check`, and `node
node_modules/eslint/bin/eslint.js
src/features/transcript/TranscriptPage.tsx
src/features/transcript/TranscriptPage.test.tsx`: passed.
- `uv run --no-sync lint-imports`: 10 contracts kept. `uv run --no-sync
ty check raven evolver agents plugins-dist scripts`: passed. Ruff check
and format check passed on the 20 Python files this PR changes.
- From `ui-web`: `npm run build`, `uv run --no-sync python build.py`,
`node scripts/check-page.mjs`, `node scripts/check-css.mjs`, and `node
scripts/check-class-namespace.mjs`: passed, including both boot
snapshots.
- `UV_NO_SYNC=1 make check-large-files check-source-language
COMMIT_RANGE=origin/main..HEAD`: passed. `PYTHONPATH=. uv run --no-sync
python scripts/check_commit_messages.py origin/main..HEAD`, commitlint
over the same range and `git diff --check origin/main...HEAD`: passed.
- `uv run --no-sync pre-commit run --from-ref origin/main --to-ref
HEAD`: all applicable hooks passed except the whole-TUI Prettier hook,
which reports `newInstancePicker.tsx` and `subagentsHub.tsx`. Both files
and the formatter config are unchanged from main; an isolated main-file
control with the same locked Prettier version reproduces both failures.
`node ui-tui/node_modules/prettier/bin/prettier.cjs --check
ui-tui/src/i18n/messages.generated.ts`: passed.
- Regression coverage includes truncated private keys, incomplete JWT
and credential tails, terminal-event delivery with foreign exception
attributes, scan reuse and CPU cost, the cut-tail scan's cost on
label-shaped runs and long gaps, its agreement with a whole-text search,
a label gap of any width, the JWT boundary and private-key END rules,
and quiet-notice expansion. The five independent boundary probes passed
before the base synchronization; the committed regression suite above
passed again afterwards.
- On the same 32,267-character synthetic crash input, the queued
event-loop callback delay fell from 15.351 seconds to 0.00951 seconds.
The regression checks use thread CPU time with a 0.5-second ceiling.
- Cut-tail scan: a failed turn whose crash text holds 31,000 spaces
before its last two words took 32.6 seconds while the region came from a
regex search for the last whitespace, and 0.016 seconds now; 64,000
characters of `pwd=` ending in a comma took 10.8 seconds through the
whole-text tail search and 0.003 seconds now, and a 300,000-character
call-record body of either shape takes under 0.3 seconds. These and the
figures below were measured with the machine's load average between 36
and 132. Across 3,728 timed texts, every label or prefix unit repeated
to 64,000 characters behind a cut and to 300,000 past the scan cap, the
slowest measured alone took about 0.2 seconds against about 0.1 seconds
on the base revision, and a text past the cap takes the same time at
twice the size. Searching the whole text again turns six timing cases
red; a smaller name room, a skipped gap or scheme step, or any value
class narrowed below its pattern's turns the property or gap test red,
as does removing the JWT boundary or restoring the END-only private-key
rule. On 12,008 generated texts with gaps up to 5,000 characters, the
output equals the whole-text version's.
- Chromium with synthetic local data: full text retained and HTML
escaped, expanded tooltip removed, internal scrolling and Retry
preserved, neutral notice label displayed, no page errors. No live
vendor call or native Windows rerun is claimed for these fixes.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- [x] Security impact considered: the long text is scrubbed of
recognized credential shapes before persistence and publication. It
remains diagnostic text and may contain hostnames or paths. Model
history and chat-channel replies keep their existing bounded content. A
session export can include the long text. `raven.security.redact` is
also the scrubber for every ACP surface (tool titles, result previews,
permission prompts, replay, error messages) and for the outbound
client's permission log, so two of its rules reach those surfaces too: a
private-key header without END hides the rest of that text, and a JWT
glued to a preceding word character, such as `/verify-eyJ...`, is left
unredacted. Tests pin both.
- [x] Backward compatibility considered: the new fields are optional,
older markers fall back to their bounded reason, and the TUI can ignore
the extra field. Incomplete private-key blocks are conservatively
removed to the end, which can also omit trailing diagnostic text. Custom
runners attach long details in the exception's instance dictionary;
descriptors are ignored.
- [x] Rollback path is clear for risky changes: revert the squash
commit. Older code ignores the optional marker field.

## Related Issues

N/A

---------

Co-authored-by: Codex <noreply@openai.com>
2026-10-17 14:28:35 +08:00

1168 lines
49 KiB
Python

"""A retryable model error outlasting the provider's ladder waits, then asks again.
The provider's own ladder is seconds long, which suits a dropped connection and
not a gateway that serves error pages for a few minutes. One measured deck build
had 62 minutes and 23.8M input tokens behind it when a 40-second OpenRouter outage
came back as a single error response and the turn ended on it. The loop now waits
out a second, longer ladder -- ``RecoveryLimits.llm_error_retry_delays`` -- before
giving the turn up, and the messages it asks with are the ones it asked with before.
"""
from __future__ import annotations
import tempfile
from pathlib import Path
import pytest
from raven.agent.loop import AgentLoop
from raven.agent.loop.bundles import ToolWiring, TurnPolicy
from raven.agent.loop.recovery import RecoveryLimits, limits_from_defaults
from raven.agent.window import shrink
from raven.contracts.tool import Tool, ToolResult
from raven.providers.base import ErrorClassification, LLMProvider, LLMResponse, ToolCallRequest
from raven.spine.events import NoticeKind
from raven.spine.message import ChatType, Source
from raven.spine.turn import AnswerlessTurnError, Origin, TurnRequest
from raven.utils.images import image_block, is_image_part, text_block
@pytest.fixture
def workspace():
with tempfile.TemporaryDirectory() as td:
yield Path(td)
class _FailsThenAnswers(LLMProvider):
"""Errors `failures` times with the given verdict, then answers."""
def __init__(self, failures: int, verdict: ErrorClassification):
super().__init__(api_key="test")
self.failures = failures
self.verdict = verdict
self.calls = 0
self.asked_with: list[int] = []
def get_default_model(self) -> str:
return "stub"
_CHAT_RETRY_DELAYS = () # the provider's own ladder, exhausted at once
async def chat(self, messages, tools=None, model=None, max_tokens=4096, temperature=0.7, **_):
self.calls += 1
self.asked_with.append(len(messages))
if self.calls <= self.failures:
return LLMResponse(
content="Error calling LLM (unparsable_response@ppt): Unable to get json response",
finish_reason="error",
error_classification=self.verdict,
)
return LLMResponse(content="real answer", finish_reason="stop")
async def chat_stream(self, *args, **kwargs): # pragma: no cover - the non-stream path is under test
raise NotImplementedError
def _agent(workspace: Path, provider: LLMProvider, delays: tuple[float, ...]) -> AgentLoop:
return AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
policy=TurnPolicy(max_iterations=10, empty_recovery=RecoveryLimits(llm_error_retry_delays=delays)),
tools=ToolWiring(restrict_to_workspace=True),
)
async def _turn(agent: AgentLoop, notices: list[tuple[NoticeKind, str]] | None = None):
async def on_notice(kind: NoticeKind, detail: str) -> None:
assert notices is not None
notices.append((kind, detail))
return await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="build the deck",
),
session_key="s1",
on_notice=None if notices is None else on_notice,
)
@pytest.mark.asyncio
async def test_a_retryable_error_is_waited_out_and_the_turn_finishes(workspace):
provider = _FailsThenAnswers(2, ErrorClassification("unparsable_response", retryable=True, should_fallback=True))
agent = _agent(workspace, provider, delays=(0.0, 0.0, 0.0))
out = await _turn(agent)
assert out is not None and out[0] == "real answer"
assert provider.calls == 3
assert len(set(provider.asked_with)) == 1, "the retried call asks with the same messages, nothing appended"
@pytest.mark.asyncio
async def test_the_ladder_is_the_budget(workspace):
provider = _FailsThenAnswers(3, ErrorClassification("server", retryable=True, should_fallback=True))
agent = _agent(workspace, provider, delays=(0.0, 0.0))
with pytest.raises(AnswerlessTurnError) as failed:
await _turn(agent)
assert provider.calls == 3, "two waits, three calls, then the turn fails on the third error"
assert "Error calling LLM" in str(failed.value)
@pytest.mark.asyncio
async def test_a_non_retryable_error_is_not_asked_again(workspace):
provider = _FailsThenAnswers(1, ErrorClassification("invalid_request", retryable=False))
agent = _agent(workspace, provider, delays=(0.0, 0.0, 0.0))
with pytest.raises(AnswerlessTurnError) as failed:
await _turn(agent)
assert provider.calls == 1
assert "Error calling LLM" in str(failed.value)
@pytest.mark.asyncio
async def test_each_wait_is_announced_with_the_error_category(workspace):
"""The wait is the longest silence a turn has, and nothing said so.
Worst case the ladder plus a first-byte budget per rung is minutes of a page
showing the last thing the model said, which reads as a wedged runtime. One
notice per wait -- carrying the category and not the vendor's own body, which
can hold a masked key and an account URL.
"""
provider = _FailsThenAnswers(2, ErrorClassification("server", retryable=True, should_fallback=True))
notices: list[tuple[NoticeKind, str]] = []
out = await _turn(_agent(workspace, provider, delays=(0.0, 0.0, 0.0)), notices)
assert out is not None and out[0] == "real answer"
assert notices == [(NoticeKind.LLM_RETRY, "server"), (NoticeKind.LLM_RETRY, "server")]
@pytest.mark.asyncio
async def test_an_error_nobody_will_ask_again_about_is_not_announced(workspace):
"""No wait, nothing to say: the turn is about to fail and its own report is
what the reader gets, so a "trying again" left on the status line would
contradict it."""
provider = _FailsThenAnswers(1, ErrorClassification("invalid_request", retryable=False))
notices: list[tuple[NoticeKind, str]] = []
with pytest.raises(AnswerlessTurnError):
await _turn(_agent(workspace, provider, delays=(0.0, 0.0, 0.0)), notices)
assert notices == []
class _WordsItsOwnFailure(LLMProvider):
"""Hands back a failure worded its own way -- the transport-failure account,
which is not the canonical sentence -- and is never asked twice."""
def __init__(self, detail: str) -> None:
super().__init__(api_key="test")
self._detail = detail
def get_default_model(self) -> str:
return "stub"
async def chat(self, *args, **kwargs):
return LLMResponse(
content=self._detail,
finish_reason="error",
error_classification=ErrorClassification("upstream_transport_failure", retryable=False),
)
async def chat_stream(self, *args, **kwargs): # pragma: no cover - the non-stream path is under test
raise NotImplementedError
@pytest.mark.asyncio
async def test_a_failure_the_provider_worded_itself_is_bounded_too(workspace):
"""Not every error response carries the canonical sentence: the
transport-failure account is the provider's own prose and is passed through
whole. It travels into a chat reply, a session marker and a cron job record
like any other, so the same ceiling applies to it."""
from raven.providers.base import LLM_ERROR_DETAIL_MAX
agent = _agent(workspace, _WordsItsOwnFailure("the upstream said it failed: " + "x" * 5000), delays=())
with pytest.raises(AnswerlessTurnError) as failed:
await _turn(agent)
detail = str(failed.value)
assert len(detail) == LLM_ERROR_DETAIL_MAX and detail.endswith("...")
assert detail.startswith("the upstream said it failed: ")
@pytest.mark.asyncio
async def test_a_canonical_sentence_is_not_cut_a_second_time(workspace):
"""Its detail was bounded where the sentence was built, so the whole
sentence is longer than the bound by the width of its head. Cutting it
again here would eat the detail the head promises a reader."""
from raven.providers.base import LLM_ERROR_DETAIL_MAX, canonical_llm_error
sentence = canonical_llm_error("invalid_request", "ppt", "y" * 5000)
assert len(sentence) > LLM_ERROR_DETAIL_MAX
agent = _agent(workspace, _WordsItsOwnFailure(sentence), delays=())
with pytest.raises(AnswerlessTurnError) as failed:
await _turn(agent)
assert str(failed.value) == sentence
class _StallsThenAnswers(LLMProvider):
"""Streams a word, then stalls (the idle cap's TimeoutError); answers whole next time."""
def __init__(self):
super().__init__(api_key="test")
self.calls = 0
def get_default_model(self) -> str:
return "stub"
async def chat(self, *args, **kwargs): # pragma: no cover - the stream path is under test
raise NotImplementedError
async def chat_stream(self, *args, **kwargs):
from raven.providers.base import ChatDelta
self.calls += 1
if self.calls == 1:
yield ChatDelta(content="partial")
raise TimeoutError
yield ChatDelta(content="answer")
def _streaming_agent(workspace: Path, provider: LLMProvider, *, retry_after_output: bool) -> AgentLoop:
return AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
policy=TurnPolicy(
max_iterations=10,
empty_recovery=RecoveryLimits(llm_error_retry_delays=(0.0,), llm_retry_after_output=retry_after_output),
),
tools=ToolWiring(restrict_to_workspace=True),
)
async def _streamed_turn(agent: AgentLoop, seen: list[str]):
async def on_delta(text: str) -> None:
seen.append(text)
return await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="build the deck",
),
session_key="s1",
on_token_delta=on_delta,
)
@pytest.mark.asyncio
async def test_a_stall_after_streamed_output_is_not_retried_by_the_outer_ladder_unless_asked(workspace):
"""The whole loop, not the helper alone: a stream that produced a word and then
stalled used to come back as a retryable error response, and the outer ladder
asked again regardless of `llm_retry_after_output` -- an interactive client
received the output of two attempts. Off, the turn fails on the first attempt;
on, the retry's answer is the answer and the caller saw both, as agreed."""
seen: list[str] = []
with pytest.raises(TimeoutError):
await _streamed_turn(_streaming_agent(workspace, _StallsThenAnswers(), retry_after_output=False), seen)
assert seen == ["partial"]
provider = _StallsThenAnswers()
seen = []
out = await _streamed_turn(_streaming_agent(workspace, provider, retry_after_output=True), seen)
assert out is not None and out[0] == "answer"
assert provider.calls == 2
assert seen == ["partial", "answer"]
class _StreamsThenReportsFailure(LLMProvider):
"""Streams a word, then reports the failure as its terminal delta; answers whole
next time. The shape the Anthropic adapter produces: it catches the exception
its own stream raised and hands it on as a delta."""
def __init__(self):
super().__init__(api_key="test")
self.calls = 0
def get_default_model(self) -> str:
return "stub"
async def chat(self, *args, **kwargs): # pragma: no cover - the stream path is under test
raise NotImplementedError
async def chat_stream(self, *args, **kwargs):
from raven.providers.base import ChatDelta
self.calls += 1
if self.calls == 1:
yield ChatDelta(content="partial")
yield ChatDelta(
content="Error calling LLM (server@stub): 503 upstream connect error",
finish_reason="error",
error_classification=ErrorClassification("server", retryable=True, should_fallback=True),
)
return
yield ChatDelta(content="answer")
@pytest.mark.asyncio
async def test_a_failure_delta_after_streamed_output_is_not_retried_unless_asked(workspace):
"""The twin of the stall above, in the shape the tree's own adapter produces:
the failure arrives as a delta rather than as a raise, so the rule that a
rendered stream is not asked again was never reached and the outer ladder
asked anyway -- the watcher saw the answer start over, up to four times. Off,
the turn fails on the first attempt and says why; on, the retry's answer is
the answer and the caller saw both, as agreed."""
provider = _StreamsThenReportsFailure()
seen: list[str] = []
with pytest.raises(AnswerlessTurnError) as failed:
await _streamed_turn(_streaming_agent(workspace, provider, retry_after_output=False), seen)
assert provider.calls == 1, "the reader had seen words, so the ladder did not ask again"
assert seen == ["partial"]
assert "503 upstream connect error" in str(failed.value)
provider = _StreamsThenReportsFailure()
seen = []
out = await _streamed_turn(_streaming_agent(workspace, provider, retry_after_output=True), seen)
assert out is not None and out[0] == "answer"
assert provider.calls == 2
assert seen == ["partial", "answer"]
class _ThinksThenStalls(LLMProvider):
"""Streams a long thought and a half-built tool call, then stalls; answers next time."""
def __init__(self):
super().__init__(api_key="test")
self.calls = 0
def get_default_model(self) -> str:
return "stub"
async def chat(self, *args, **kwargs): # pragma: no cover - the stream path is under test
raise NotImplementedError
async def chat_stream(self, *args, **kwargs):
from raven.providers.base import ChatDelta
self.calls += 1
if self.calls == 1:
yield ChatDelta(content=None, reasoning_content="weighing the layout")
yield ChatDelta(
content=None,
tool_call_delta={
"tool_calls": [{"index": 0, "id": "c1", "function": {"name": "write_file", "arguments": '{"pa'}}]
},
)
raise TimeoutError
yield ChatDelta(content="answer")
@pytest.mark.asyncio
async def test_a_stall_during_a_silent_think_is_asked_again(workspace):
"""Nothing reached the caller, so asking again repeats nothing. A thought is not
the reply unless the caller wired `on_reasoning_delta`, and the fragments of a
tool call are never rendered as one -- counting either as output made the long
quiet round, the one that writes a whole build script, the one round a dropped
stream could always end."""
provider = _ThinksThenStalls()
seen: list[str] = []
out = await _streamed_turn(_streaming_agent(workspace, provider, retry_after_output=False), seen)
assert out is not None and out[0] == "answer"
assert provider.calls == 2
assert seen == ["answer"], "the watcher saw the reply once"
assert "weighing" not in (out[0] or ""), "the abandoned thought is not part of the reply"
@pytest.mark.asyncio
async def test_what_the_failed_attempt_left_behind_is_dropped():
"""`stream_llm_call` directly, for the invariant the loop cannot show: every
retry starts from empty buffers. While a retry was only possible with all three
empty, the paths that ask again did not have to clear them; once a silent think
retries, a slot left standing merges with the next attempt's fragments into a
call the model never made, and a thinking block keeps the signature of a
generation that no longer exists."""
from raven.providers.base import ChatDelta
from raven.providers.streaming import stream_llm_call
class _HalfCallThenAnswers(LLMProvider):
def __init__(self):
super().__init__(api_key="test")
self.calls = 0
def get_default_model(self) -> str:
return "stub"
async def chat(self, *args, **kwargs): # pragma: no cover - the stream path is under test
raise NotImplementedError
async def chat_stream(self, *args, **kwargs):
self.calls += 1
if self.calls == 1:
yield ChatDelta(content=None, reasoning_content="first thought")
yield ChatDelta(
content=None,
tool_call_delta={
"tool_calls": [{"index": 0, "id": "c1", "function": {"name": "exec", "arguments": '{"comm'}}]
},
thinking_blocks=[{"type": "thinking", "thinking": "A", "signature": "sigA"}],
)
raise ConnectionError("upstream went away")
yield ChatDelta(content="here is the plan")
provider = _HalfCallThenAnswers()
out = await stream_llm_call(
provider,
messages=[{"role": "user", "content": "go"}],
tools=None,
model="stub",
on_token_delta=None,
max_reconnects=1,
)
assert provider.calls == 2
assert out.content == "here is the plan"
assert not out.tool_calls, "no call the second attempt did not make"
assert "first thought" not in (out.reasoning_content or "")
assert not out.thinking_blocks, "no signed block from a generation that is gone"
@pytest.mark.production_timing
def test_the_ladder_comes_from_agents_defaults():
class _Defaults:
llm_error_retry_delays = [30, 60, 120]
assert limits_from_defaults(_Defaults()).llm_error_retry_delays == (30.0, 60.0, 120.0)
assert limits_from_defaults(object()).llm_error_retry_delays == (15.0, 30.0, 60.0)
assert limits_from_defaults(object()).llm_retry_after_output is False
class _Off:
llm_error_retry_delays: list[float] = []
assert limits_from_defaults(_Off()).llm_error_retry_delays == (), "an explicit [] turns the ladder off"
class _Unattended:
llm_retry_after_output = True
assert limits_from_defaults(_Unattended()).llm_retry_after_output is True
def test_the_image_window_budget_is_a_setting() -> None:
"""``agents.defaults.imageWindowBudgetBytes`` rides on the limits; absent it is the
measured 12 MB, and 0 is a legal value that means no standing pass."""
class _Bounded:
image_window_budget_bytes = 4_000_000
class _ReactiveOnly:
image_window_budget_bytes = 0
assert limits_from_defaults(object()).image_window_budget_bytes == 12_000_000
assert limits_from_defaults(_Bounded()).image_window_budget_bytes == 4_000_000
assert limits_from_defaults(_ReactiveOnly()).image_window_budget_bytes == 0
def _picture(tag: str) -> dict:
return image_block(f"data:image/png;base64,{tag}")
def _with_pictures() -> list[dict]:
"""Three image-bearing messages on both transports: a tool result carrying its
captions and pictures interleaved, and two attached user messages (the shape a
Chat Completions endpoint gets), each stamped with where its pictures came from."""
return [
{"role": "user", "content": "build it"},
{
"role": "tool",
"tool_call_id": "1",
"name": "ppt_template",
"content": [
text_block("Template page 1 -- the cover"),
_picture("T1"),
text_block("Template page 2"),
_picture("T2"),
],
"_image_sources": [
{"tool": "ppt_template", "iteration": 1, "caption": "Template page 1 -- the cover"},
{"tool": "ppt_template", "iteration": 1, "caption": "Template page 2"},
],
},
{"role": "assistant", "content": "building"},
{
"role": "user",
"content": [_picture("B7")],
"_attached_image": True,
"_image_sources": [{"tool": "ppt_build", "iteration": 2, "caption": "Page 7 of 20: the claim"}],
},
{"role": "assistant", "content": "fixing"},
{
"role": "user",
"content": [_picture("B7b"), _picture("B8")],
"_attached_image": True,
"_image_sources": [
{"tool": "ppt_build", "iteration": 3, "caption": "Page 7 of 20: the claim"},
{"tool": "ppt_build", "iteration": 3, "caption": "Page 8 of 20"},
],
},
]
def _pictures_in(message: dict) -> int:
content = message.get("content")
return sum(1 for p in content if is_image_part(p)) if isinstance(content, list) else 0
def test_the_image_window_keeps_the_newest_pictures_and_says_what_the_rest_showed() -> None:
"""The picture the model has already looked at is replaced by a note that names
the tool, the round and the page, and says how to see it again -- not a bare
"elided" it has to guess about. The newest ``keep`` image-bearing messages are
left byte for byte as they were, whatever their role."""
messages = _with_pictures()
before_newest = [dict(messages[3]), dict(messages[5])]
changed, withdrawn = shrink.window_images(messages, 2)
assert (changed, withdrawn) == (1, 2)
assert [_pictures_in(m) for m in messages] == [0, 0, 0, 1, 0, 2]
assert messages[3] == before_newest[0] and messages[5] == before_newest[1]
notes = [p["text"] for p in messages[1]["content"] if p["type"] == "text"]
assert notes[0] == "Template page 1 -- the cover", "the caption beside the picture stays"
assert notes[1].startswith(
'[image no longer in context: "Template page 1 -- the cover" from ppt_template at iteration 1.'
)
assert "only the 2 newest image-bearing result(s) keep their pictures" in notes[1]
assert notes[1].endswith("To see it again, ask ppt_template for it again]")
assert '"Template page 2" from ppt_template' in notes[3]
assert messages[1]["name"] == "ppt_template" and messages[1]["tool_call_id"] == "1", "only the content changes"
def test_the_image_window_is_prefix_stable_between_iterations() -> None:
"""Between two calls only the message that just slid out of the window changes.
Everything before it is the same object it was, including messages withdrawn on
an earlier pass, so a cached prefix is invalidated from the slide-out point and
not rewritten from the top."""
messages = _with_pictures()
shrink.window_images(messages, 2)
first_pass = list(messages)
messages.append({"role": "assistant", "content": "one more"})
messages.append(
{
"role": "user",
"content": [_picture("B9")],
"_attached_image": True,
"_image_sources": [{"tool": "ppt_build", "iteration": 4, "caption": "Page 9 of 20"}],
}
)
changed, withdrawn = shrink.window_images(messages, 2)
assert (changed, withdrawn) == (1, 1)
same = [messages[i] is first_pass[i] for i in range(len(first_pass))]
assert same == [True, True, True, False, True, True], "exactly the message that slid out is new"
assert (
_pictures_in(messages[3]) == 0
and '"Page 7 of 20: the claim" from ppt_build at iteration 2' in messages[3]["content"][0]["text"]
)
assert shrink.window_images(messages, 2) == (0, 0), "a second pass over the same list is a no-op"
def _batch(tag: str, pictures: int, iteration: int) -> dict:
"""An attached batch of `pictures` renders, each 4 wire bytes of payload ("AAAA")."""
return {
"role": "user",
"content": [_picture("AAAA") for _ in range(pictures)],
"_attached_image": True,
"_image_sources": [
{"tool": "ppt_build", "iteration": iteration, "caption": f"Page {n} of 20"} for n in range(pictures)
],
}
def test_the_budget_window_collapses_once_when_the_pictures_outgrow_it_and_not_before() -> None:
"""Pictures stay while they fit; the moment they do not, everything but the newest
``keep`` goes in one collapse. The prefix breaks on that call and on no other, so a
deck that adds a batch per build breaks its cached prefix a handful of times
rather than once per build -- the count is the point, hence the assertion on it."""
messages: list[dict] = [{"role": "user", "content": "build it"}]
breaks = []
for arrival in range(1, 8):
messages.append({"role": "assistant", "content": f"round {arrival}"})
messages.append(_batch(f"b{arrival}", pictures=3, iteration=arrival)) # 12 wire bytes per batch
before = list(messages)
changed, withdrawn = shrink.window_images(messages, 2, budget=40, reason="budget")
breaks.append(changed)
untouched = [messages[i] is before[i] for i in range(len(before))]
if changed:
assert withdrawn == 3 * changed
assert sum(1 for same in untouched if not same) == changed, "only the collapsed messages are new objects"
else:
assert all(untouched), "a call under budget rewrites nothing"
# batches 1-3 fit (36 B); the 4th tips the total over 40 -> collapse to the newest 2;
# 5 fits again (36 B); the 6th tips it -> collapse; 7 fits. Two breaks in seven builds.
assert breaks == [0, 0, 0, 2, 0, 2, 0]
live = [i for i, m in enumerate(messages) if _pictures_in(m)]
assert len(live) == 3 and live[-1] == len(messages) - 1, "the newest batches are the ones still in view"
note = next(p["text"] for m in messages if m.get("_attached_image") for p in m["content"] if p["type"] == "text")
assert "outgrew their byte budget" in note and "only the 2 newest image-bearing result(s) keep theirs" in note
assert shrink.window_images(messages, 2, budget=40, reason="budget") == (0, 0), "idempotent under budget"
def test_the_budget_weighs_inline_bytes_only_and_the_users_picture_is_outside_it() -> None:
remote = {"type": "image_url", "image_url": {"url": "https://example.test/a.png"}}
sketch = {"role": "user", "content": [_picture("SKETCHSKETCH")]}
messages = [
sketch,
{"role": "tool", "tool_call_id": "1", "content": [remote]},
_batch("b1", 2, 1),
_batch("b2", 2, 2),
]
# 16 wire bytes across the two batches, the remote reference weighing nothing.
assert shrink.window_images(messages, 1, budget=16, reason="budget") == (0, 0)
assert shrink.window_images(messages, 1, budget=15, reason="budget") == (2, 3)
assert messages[0] is sketch, "the user's own picture is neither weighed nor withdrawn"
def test_the_budget_is_charged_the_base64_the_request_carries_not_the_bytes_it_decodes_to() -> None:
"""The budget bounds the request body, and a data URI travels encoded.
Counted on the decoded side, 11.24 MB of pictures passed the 12 MB default on a
request that put 16.8 MB on the wire; the gateway answered it with an empty 200
and zero usage, five times over on a byte-identical payload. The ratio is 4/3, so
the pin is a budget that sits between the two readings of the same picture: it has
to collapse the window, and would not have before.
"""
payload = "A" * 400 # 400 base64 characters, 300 bytes decoded
messages = [
{"role": "tool", "tool_call_id": "1", "content": [image_block(f"data:image/png;base64,{payload}")]},
{"role": "assistant", "content": "looked"},
{"role": "tool", "tool_call_id": "2", "content": [image_block(f"data:image/png;base64,{payload}")]},
]
assert shrink.window_images(messages, 1, budget=800, reason="budget") == (0, 0), "two pictures, 800 wire bytes"
assert shrink.window_images(messages, 1, budget=700, reason="budget") == (1, 1), (
"600 decoded would have fit 700; 800 on the wire does not"
)
def test_a_picture_without_provenance_is_still_accounted_for() -> None:
"""A message that arrived without a stamp (a transport the loop did not build,
an older transcript) gets a note that counts the picture rather than naming it,
and never a note that claims a tool it cannot know."""
messages = [
{"role": "tool", "tool_call_id": "1", "content": [_picture("a"), _picture("b")]},
{"role": "user", "content": [_picture("c")], "_attached_image": True},
]
assert shrink.window_images(messages, 1) == (1, 2)
notes = [p["text"] for p in messages[0]["content"]]
assert notes[0].startswith("[image no longer in context: picture 1 of 2.")
assert notes[1].startswith("[image no longer in context: picture 2 of 2.")
assert notes[0].endswith("To see it again, ask for it again]")
def test_a_picture_the_user_sent_is_not_part_of_the_window() -> None:
"""The user's own picture is the subject of the turn, not a result the model has
finished with: however many renders follow it, it stays -- until the endpoint has
refused the request and the window is closed with nothing else left to take out."""
sketch = {"role": "user", "content": [text_block("make it look like this"), _picture("SKETCH")]}
messages = [sketch, *_with_pictures()[1:]]
assert shrink.window_images(messages, 1) == (2, 3)
assert messages[0] is sketch and _pictures_in(messages[0]) == 1
assert shrink.window_images(messages, 0, reason="refused") == (1, 2), "the tool pictures, not the sketch"
assert _pictures_in(messages[0]) == 1
assert shrink.window_images(messages, 0, reason="refused", any_role=True) == (1, 1)
assert _pictures_in(messages[0]) == 0 and "refused this request's pictures" in messages[0]["content"][1]["text"]
def test_a_closed_window_withdraws_the_newest_picture_and_says_why() -> None:
"""Window zero is what a size refusal leaves behind: every picture comes out,
the newest included, and the note blames the endpoint rather than a newer batch."""
messages = _with_pictures()
assert shrink.window_images(messages, 0, reason="refused") == (3, 5)
assert not any(_pictures_in(m) for m in messages)
assert "The endpoint refused this request's pictures as too large" in messages[5]["content"][1]["text"]
assert "Page 8 of 20" in messages[5]["content"][1]["text"]
# The standing pass with the window already closed reads differently again: the
# picture was never shown, and the model is told none will be for this turn.
later = [{"role": "user", "content": [_picture("z")], "_attached_image": True}]
shrink.window_images(later, 0)
assert "so none are kept in context now" in later[0]["content"][0]["text"]
def test_the_overflow_path_uses_the_same_notes() -> None:
"""``shrink.emergency_shrink`` keeps its tighter count and its copy-and-return contract,
and the picture it drops gets the same kind of note, with the overflow as the reason."""
messages = _with_pictures()
out, elided = shrink.elide_older_images(messages)
assert elided == 2 and out is not messages
assert [_pictures_in(m) for m in messages] == [0, 2, 0, 1, 0, 2], "the input is left alone"
assert [_pictures_in(m) for m in out] == [0, 0, 0, 0, 0, 2]
assert "elided to fit the context window" in out[3]["content"][0]["text"]
assert shrink.elide_older_images(out) == (out, 0)
def test_the_overflow_path_sheds_the_users_own_pictures_last() -> None:
"""A turn that is nothing but pasted screenshots overflows on them alone, and the
overflow path could always reach them: they go after every result picture and
tool body, newest kept, with the same note. While a result picture is still in
the transcript it goes first and the user's stay."""
pasted = [
{"role": "user", "content": [text_block("first screenshot"), _picture("U1")]},
{"role": "assistant", "content": "seen"},
{"role": "user", "content": [text_block("second"), _picture("U2")]},
{"role": "assistant", "content": "seen"},
{"role": "user", "content": [text_block("third"), _picture("U3")]},
]
out, elided = shrink.emergency_shrink(pasted)
assert elided == 2 and out is not pasted
assert [_pictures_in(m) for m in pasted] == [1, 0, 1, 0, 1], "the input is left alone"
assert [_pictures_in(m) for m in out] == [0, 0, 0, 0, 1]
assert "elided to fit the context window" in out[0]["content"][1]["text"]
assert shrink.emergency_shrink(out) == (out, 0)
mixed = [*_with_pictures(), {"role": "user", "content": [_picture("U9")]}]
first, freed = shrink.emergency_shrink(mixed)
assert freed == 2 and _pictures_in(first[-1]) == 1, "result pictures go first, the user's stays"
again, freed_again = shrink.emergency_shrink(first)
assert freed_again == 1 and _pictures_in(again[5]) == 0, "then the older result picture goes"
assert _pictures_in(again[-1]) == 1, "the newest picture, the user's, is kept"
assert shrink.emergency_shrink(again) == (again, 0)
class _PageRender(Tool):
"""A tool that answers with a caption and a picture, the way the deck tools do."""
@property
def name(self) -> str:
return "page_render"
@property
def description(self) -> str:
return "renders one page"
@property
def parameters(self) -> dict:
return {"type": "object", "properties": {"page": {"type": "integer"}}}
async def execute(self, page: int = 7, **_) -> ToolResult:
caption = f"Page {page} of 20: the claim"
return ToolResult(model_text=caption, blocks=[text_block(caption), _picture(f"P{page}" * 40)])
class _ShowsThenRefuses(LLMProvider):
"""Asks for a render, refuses the request that carries it as too large, asks for
another render once that is settled, then answers. Records every request."""
def __init__(self):
super().__init__(api_key="test")
self.calls = 0
self.seen: list[list[dict]] = []
def get_default_model(self) -> str:
return "stub"
_CHAT_RETRY_DELAYS = ()
async def chat(self, messages, tools=None, model=None, max_tokens=4096, temperature=0.7, **_):
self.calls += 1
self.seen.append([dict(m) for m in messages])
renders = sum(1 for m in messages if m.get("role") == "tool")
pictures = sum(_pictures_in(m) for m in messages)
if renders == 0:
return LLMResponse(
content="",
finish_reason="tool_calls",
tool_calls=[ToolCallRequest(id="c1", name="page_render", arguments={"page": 7})],
)
if pictures:
return LLMResponse(
content="Error calling LLM (image_too_large): Downloaded image content cannot exceed 30MB",
finish_reason="error",
error_classification=ErrorClassification("image_too_large", strip_images=True),
)
if renders == 1:
return LLMResponse(
content="",
finish_reason="tool_calls",
tool_calls=[ToolCallRequest(id="c2", name="page_render", arguments={"page": 8})],
)
return LLMResponse(content="done", finish_reason="stop")
@pytest.mark.asyncio
async def test_a_size_refusal_closes_the_window_for_the_rest_of_the_turn(workspace):
"""The refusal is answered by withdrawing the pictures and asking again, and the
narrower window then stands: the next render is withdrawn before it is ever sent,
so the same refusal cannot come back a few calls later the way it did on both
measured runs. The pictures the loop attaches carry their provenance, and neither
that stamp nor the notes reach the session file."""
provider = _ShowsThenRefuses()
agent = _agent(workspace, provider, delays=(0.0,))
agent.tools.register(_PageRender())
out = await _turn(agent)
assert out is not None and out[0] == "done"
assert provider.calls == 4 # asks for page 7; refused; asked again, asks for page 8; answers
refused = provider.seen[1]
attached = refused[-1]
assert attached.get("_attached_image") and _pictures_in(attached) == 1
assert attached["_image_sources"] == [{"tool": "page_render", "iteration": 1, "caption": "Page 7 of 20: the claim"}]
retried = provider.seen[2]
assert not any(_pictures_in(m) for m in retried)
note = retried[-1]["content"][0]["text"]
assert note.startswith('[image no longer in context: "Page 7 of 20: the claim" from page_render at iteration 1.')
assert "refused this request's pictures as too large" in note
# Window at zero from here: the page 8 render was withdrawn before the call.
final = provider.seen[3]
assert not any(_pictures_in(m) for m in final)
assert "so none are kept in context now" in final[-1]["content"][0]["text"]
assert '"Page 8 of 20: the claim" from page_render at iteration 2' in final[-1]["content"][0]["text"]
persisted = " ".join(p.read_text(encoding="utf-8") for p in (workspace / "sessions").rglob("*.jsonl"))
assert persisted, "the turn was filed"
assert "_image_sources" not in persisted and "image no longer in context" not in persisted
class _ShowsThreePages(LLMProvider):
"""Asks for three renders, one per call, then answers. Records every request."""
def __init__(self):
super().__init__(api_key="test")
self.calls = 0
self.seen: list[list[dict]] = []
def get_default_model(self) -> str:
return "stub"
_CHAT_RETRY_DELAYS = ()
async def chat(self, messages, tools=None, model=None, max_tokens=4096, temperature=0.7, **_):
self.calls += 1
self.seen.append([dict(m) for m in messages])
if self.calls <= 3:
return LLMResponse(
content="",
finish_reason="tool_calls",
tool_calls=[
ToolCallRequest(id=f"c{self.calls}", name="page_render", arguments={"page": 6 + self.calls})
],
)
return LLMResponse(content="done", finish_reason="stop")
def _agent_with_budget(workspace: Path, provider: LLMProvider, budget: int) -> AgentLoop:
return AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
policy=TurnPolicy(
max_iterations=10,
empty_recovery=RecoveryLimits(llm_error_retry_delays=(), image_window_budget_bytes=budget),
),
tools=ToolWiring(restrict_to_workspace=True),
)
@pytest.mark.asyncio
async def test_the_configured_budget_collapses_the_window_and_zero_leaves_it_open(workspace):
"""The budget the loop enforces is the one on its limits. At one byte the third
render tips the total and the oldest picture is withdrawn with the budget note; at
zero no standing pass runs, so every picture stays until an endpoint refuses."""
bounded = _ShowsThreePages()
agent = _agent_with_budget(workspace, bounded, budget=1)
agent.tools.register(_PageRender())
out = await _turn(agent)
assert out is not None and out[0] == "done" and bounded.calls == 4
final = bounded.seen[3]
bearing = [m for m in final if m.get("_attached_image") or _pictures_in(m)]
assert [_pictures_in(m) for m in bearing] == [0, 1, 1], "the oldest collapsed, the newest two kept"
assert "outgrew their byte budget" in bearing[0]["content"][0]["text"]
open_window = _ShowsThreePages()
agent = _agent_with_budget(workspace, open_window, budget=0)
agent.tools.register(_PageRender())
out = await _turn(agent)
assert out is not None and out[0] == "done" and open_window.calls == 4
assert sum(_pictures_in(m) for m in open_window.seen[3]) == 3, "no standing pass: every render is still there"
assert not any("no longer in context" in str(m.get("content")) for m in open_window.seen[3])
class _ShowsOnce(LLMProvider):
"""Asks for one render, then answers. Records every request."""
def __init__(self):
super().__init__(api_key="test")
self.calls = 0
self.seen: list[list[dict]] = []
def get_default_model(self) -> str:
return "stub"
_CHAT_RETRY_DELAYS = ()
async def chat(self, messages, tools=None, model=None, max_tokens=4096, temperature=0.7, **_):
self.calls += 1
self.seen.append([dict(m) for m in messages])
if self.calls == 1:
return LLMResponse(
content="",
finish_reason="tool_calls",
tool_calls=[ToolCallRequest(id="c1", name="page_render", arguments={"page": 7})],
)
return LLMResponse(content="done", finish_reason="stop")
@pytest.mark.asyncio
async def test_the_tool_result_transport_stamps_the_same_provenance(workspace):
"""On a transport that carries a picture inside the ``tool`` message (the default
for the Anthropic family), the stamp is written on that message, one entry per
picture, the same entry the attached transport writes."""
provider = _ShowsOnce()
agent = _agent(workspace, provider, delays=())
agent._image_tool_result_ok["stub"] = True
agent.tools.register(_PageRender())
out = await _turn(agent)
assert out is not None and out[0] == "done" and provider.calls == 2
carried = [m for m in provider.seen[1] if m.get("role") == "tool" and _pictures_in(m)]
assert len(carried) == 1, "the picture rode inside the tool result"
assert carried[0]["_image_sources"] == [
{"tool": "page_render", "iteration": 1, "caption": "Page 7 of 20: the claim"}
]
assert not any(m.get("_attached_image") for m in provider.seen[1]), "nothing was attached to a user message"
@pytest.mark.asyncio
async def test_a_note_on_the_tool_result_transport_is_filed_with_its_facts_only(workspace):
"""The tool message survives the turn on this transport, so the note it carries
is filed: which picture, which tool, which round -- not why it left this turn or
how to ask for it again, which a resumed session would read as current."""
provider = _ShowsThenRefuses()
agent = _agent(workspace, provider, delays=(0.0,))
agent._image_tool_result_ok["stub"] = True
agent.tools.register(_PageRender())
out = await _turn(agent)
assert out is not None and out[0] == "done"
live = [m for m in provider.seen[2] if m.get("role") == "tool"][-1]["content"][1]["text"]
assert live.startswith(
'[image no longer in context: "Page 7 of 20: the claim" from page_render at iteration 1. The endpoint refused'
)
assert live.endswith("ask page_render for it again]")
persisted = " ".join(p.read_text(encoding="utf-8") for p in (workspace / "sessions").rglob("*.jsonl"))
assert "from page_render at iteration 1.]" in persisted and "Page 7 of 20: the claim" in persisted
assert "refused this request" not in persisted and "To see it again" not in persisted
assert "_image_sources" not in persisted
@pytest.mark.parametrize(
("reason", "keep"),
[("refused", 2), ("context", 2), ("budget", 2), ("superseded", 0), ("superseded", 2)],
)
@pytest.mark.parametrize(
"source",
[
{"caption": "Page 7 of 20. The claim, then the proof.", "tool": "page_render", "iteration": 3},
{"caption": None, "tool": None, "iteration": None},
],
)
def test_every_reason_the_note_can_give_files_down_to_its_facts(reason, keep, source):
"""The composer and the filed-form parser share the reasons' opening words; this
composes every reason the loop can pass and files it, so a reworded opening on
either side fails here instead of filing the turn's business into the session.
The caption carries the sentence break the parser must read past."""
from raven.agent.window.images import filed_image_note, withdrawn_image_note
live = withdrawn_image_note(source, index=1, total=2, reason=reason, keep=keep)
facts = live.partition(". ")[0] if source["caption"] is None else live[: live.index(". ", live.index("iteration"))]
assert filed_image_note(live) == f"{facts}.]"
assert facts.startswith("[image no longer in context: ") and "To see it again" not in facts
assert filed_image_note("plain tool text. It was elided by nobody]") == "plain tool text. It was elided by nobody]"
@pytest.mark.asyncio
async def test_an_image_refusal_with_nothing_to_withdraw_is_not_waited_out(workspace):
"""No picture in the conversation means the refusal was about something else, and
`image_too_large` is not retryable: the turn ends at once rather than after the
error ladder."""
provider = _FailsThenAnswers(5, ErrorClassification("image_too_large", strip_images=True))
agent = _agent(workspace, provider, delays=(0.0, 0.0, 0.0))
with pytest.raises(AnswerlessTurnError) as failed:
await _turn(agent)
assert provider.calls == 1
assert "Error calling LLM" in str(failed.value)
async def _watch(text: str) -> None:
"""A caller watching the reply form: what puts a spawn on the streamed path."""
class _DropsTwiceThenAnswers(LLMProvider):
"""The connection drops before the first delta twice, then the stream answers
whole: one failure more than the single reconnect the helper grants on its own,
so only a ladder handed down from the deployment reaches the answer."""
def __init__(self):
super().__init__(api_key="test")
self.calls = 0
def get_default_model(self) -> str:
return "stub"
async def chat(self, *args, **kwargs): # pragma: no cover - the fallback this test must not take
raise AssertionError("the spawn fell back to the waited-for call")
async def chat_stream(self, *args, **kwargs):
from raven.providers.base import ChatDelta
self.calls += 1
if self.calls <= 2:
raise ConnectionError("connection reset by peer")
yield ChatDelta(content="answer")
class _ShowsThenDrops(LLMProvider):
"""Streams a word, then the connection drops; answers whole next time."""
def __init__(self):
super().__init__(api_key="test")
self.calls = 0
def get_default_model(self) -> str:
return "stub"
async def chat(self, *args, **kwargs): # pragma: no cover - the fallback this test must not take
raise AssertionError("the spawn fell back to the waited-for call")
async def chat_stream(self, *args, **kwargs):
from raven.providers.base import ChatDelta
self.calls += 1
if self.calls == 1:
yield ChatDelta(content="partial")
raise ConnectionError("connection reset by peer")
yield ChatDelta(content="answer")
@pytest.mark.asyncio
async def test_a_spawned_subagents_dropped_stream_is_asked_again_on_the_deployments_ladder(workspace):
"""The subagent half of the wiring: `RavenLoopBackend` hands the ladder it was built
with to `stream_llm_call`. Left to that call's own defaults a spawn reconnects once
and waits never, so a stream dropped twice ended the run; with the ladder handed
over, the third call answers and its answer is the run's result."""
from raven.agent.subagent.backends.raven_loop import RavenLoopBackend
provider = _DropsTwiceThenAnswers()
backend = RavenLoopBackend(provider=provider, model="stub", agent_home=workspace / "home", retry_delays=(0.0,))
out = await backend.run("build the deck", task_id="t1", workspace=workspace, executor=None, on_delta=_watch)
assert provider.calls == 3
assert "answer" in (out or "")
@pytest.mark.asyncio
async def test_a_spawned_subagents_rendered_stream_is_retried_only_when_the_deployment_says_so(workspace):
"""`llmRetryAfterOutput` reaches a spawn the same way. Built with it on, a stream
that showed a word and then dropped is asked again and the retry's word is the
answer; built without it, the default, the drop ends the run as the rule has
always read."""
from raven.agent.subagent.backends.raven_loop import RavenLoopBackend
seen: list[str] = []
async def watch(text: str) -> None:
seen.append(text)
provider = _ShowsThenDrops()
backend = RavenLoopBackend(
provider=provider, model="stub", agent_home=workspace / "home", retry_delays=(0.0,), retry_after_output=True
)
out = await backend.run("build the deck", task_id="t1", workspace=workspace, executor=None, on_delta=watch)
assert provider.calls == 2
assert "answer" in (out or "")
assert seen == ["partial", "answer"]
provider = _ShowsThenDrops()
backend = RavenLoopBackend(provider=provider, model="stub", agent_home=workspace / "home", retry_delays=(0.0,))
with pytest.raises(ConnectionError):
await backend.run("build the deck", task_id="t1", workspace=workspace, executor=None, on_delta=_watch)
assert provider.calls == 1
class _RefusedWithABody(LLMProvider):
"""One error response whose call record kept the provider's whole body."""
def __init__(self, body: str):
super().__init__(api_key="test")
self._body = body
def get_default_model(self) -> str:
return "stub"
_CHAT_RETRY_DELAYS = () # the provider's own ladder, exhausted at once
async def chat(self, *args, **kwargs):
from raven.contracts.llm_provider import CallRecord
return LLMResponse(
content="Error calling LLM (rate_limit@openrouter): Provider returned error",
finish_reason="error",
error_classification=ErrorClassification("rate_limit", retryable=False),
call_record=CallRecord(http_status=429, body=self._body),
)
async def chat_stream(self, *args, **kwargs): # pragma: no cover - the non-stream path is under test
raise NotImplementedError
@pytest.mark.asyncio
async def test_a_call_the_ladder_gave_up_on_keeps_the_body_its_record_kept(workspace):
"""The canonical sentence is the bounded report; the body behind it -- which
provider refused, from which pool, and what to do about it -- rides the
failure as its long account."""
body = (
'{"error":{"code":429,"metadata":{"raw":"z-ai/glm-5.3-flash is temporarily rate-limited upstream",'
'"provider_name":"Z.AI","limit_source":"upstream_provider_shared_pool"}}}'
)
agent = _agent(workspace, _RefusedWithABody(body), delays=())
with pytest.raises(AnswerlessTurnError) as failed:
await _turn(agent)
assert str(failed.value) == "Error calling LLM (rate_limit@openrouter): Provider returned error"
assert failed.value.full_detail == f"{failed.value}\n\n{body}"