Raven-1/tests/test_agent_loop_session_effort.py
ypflll 869ad78abd
refactor(agent): read the charter and the window's shrinking through the roles (#468)
## Summary

The agent loop already names four strategy roles (Memory, Planning,
Capability, Action), but two judgements a turn makes still lived in the
shell around them. This moves both onto the roles, in two dependent
commits, with no behaviour change intended.

**1. The charter reads through Memory and Action.** A dispatched
sub-agent's playbook (`_meta["raven.playbook"]`) was read from four
places inside the loop and the tool registry. `prompt` and `stopWhen`
are now read by `DefaultMemory.assemble()`, which fills `task_brief` and
`task_done_when` on the `TurnContext` the identity segment renders.
`checks` and `code` are read by `DefaultAction.judge()`, which
`ToolRegistry.execute` asks once per proposed call, the way it already
asks the permission gate. `tools` stays in `withheld_names()` and the
registry still decides what a refusal does, so a replaced role withholds
nothing it could not already withhold.

**2. Mid-turn window shrinking seats on Memory.** The five recoveries
the loop ran inline go through one method,
`MemoryModule.shrink(messages, pressure, state, model)`, with
`WindowPressure`, `WindowState` and `ShrinkResult` as the carriers:

- proactive compaction, once the last billed reading crosses the trigger
- the standing image window
- a provider's overflow error
- a picture refused inside a tool result
- pictures refused for size

The retry mechanism stays in the shell. The six `continue` sites and the
iteration rollback did not move, so a hook sees a retried call exactly
as it saw the first. `WindowState` is a per-turn object the shell owns
and hands to each call, because the role outlives the turn;
`image_window` has no default, since 0 is a real value meaning "no
picture stays".

The pure pieces both sides need move to a new package,
`raven/agent/window/`, which the import contracts treat as cargo that
may not import the loop shell. `REASONING_EFFORT_LADDER` moves to
`contracts/llm_provider.py` because the loop's empty-response descent
and the window's head summary both read it and neither package may
import the other.

Contract papers: `CONTRACTS_VERSION` 26 to 28, `CONTRACTS_LINE_CEILING`
3,040 to 3,200, each bump with its own review paragraph in
`tests/test_kernel_budget.py`.

A follow-up PR carries the third step, which gives a sub-agent nine
verbs in place of the six hook phases. It is kept separate because it
touches every bundled plugin.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [x] Refactor
- [ ] Other

## Verification

Gates, run at each of the two commits:

```
uv run ruff check --no-cache raven evolver agents plugins-dist tests scripts
uv run ruff format --check --no-cache raven evolver agents plugins-dist tests scripts
uv run lint-imports
uv run pytest -q
npx commitlint --from origin/main --to HEAD --config commitlint.config.cjs
uv run python scripts/check_commit_messages.py origin/main..HEAD
uv run python scripts/check_source_language.py origin/main..HEAD
uv run python scripts/check_large_files.py origin/main..HEAD
```

Import contracts: 10 kept, 0 broken at both commits. The suite is green
at both; the failures this machine shows are the same ones it shows on
an unmodified `main` (missing browser and optional packages), and
neither commit adds one. The loop's own e2e files, which the default
selection leaves out, were run too (charter, playbook, truncation,
direct chat, fallback chain, ACP stdio): no failure that `main` does not
also show.

Behaviour preservation. Each commit was run against a worktree of its
own parent under the same scripted provider, outputs normalised for
nonces and temp paths, then byte-compared. Four scenarios, eight
comparisons, all identical:

- a whole turn with no playbook: host turn, spawned sub-agent, tool
calls and refusals
- a playbook-on worker lane in process, including two charter refusals
- six window-shrink scenarios: proactive compaction, the standing image
window, overflow, a refused picture, oversized pictures, the effort
ladder
- the five-plugin hook seam, every phase's decision and metadata per
iteration

Mutation check on the second commit: breaking each of the five moves
inside `DefaultMemory.shrink` turns exactly its own scenario red, so the
comparison is not vacuous.

Real model, Sonnet 4.5 through OpenRouter, at both commits:

- playbook on, in-process lane: both workers receive the charter, write
their files and finish
- playbook on, ACP fork lane: the charter prompt crosses the process
boundary, and a charter check blocks a `write_file` outside the fence

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

No user-visible behaviour change is intended, and the comparisons above
are byte-identical on every path they cover.

For anyone replacing a harness role: `MemoryModule` gains `shrink`, so a
replacement written against the old protocol needs it.
`DefaultMemory.bind()` names the missing member at assembly rather than
failing inside a turn.

Rollback is a revert of the merge commit. Nothing persisted on disk
changes shape.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: yao pengfei <yaopengfei@shanda.com>
Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 10:52:48 +08:00

269 lines
10 KiB
Python

"""A session's mode may move the reasoning effort of its turns' model calls.
The loop reads its per-session policy once at the turn's start; when the policy
names a reasoning effort, every model call of that turn -- the ReAct steps, the
wrap-up after an exhausted budget and the head summary a compaction pays for --
carries it as an explicit argument.
A policy that names none passes nothing, so the provider's configured default
applies exactly as before, and another session's turns are untouched.
"""
from __future__ import annotations
import pytest
from raven.agent.loop import AgentLoop
from raven.agent.loop.bundles import EngineWiring, ToolWiring, TurnPolicy
from raven.agent.window import compaction
from raven.config.raven import CheckpointConfig, RuntimeConfig
from raven.config.schema import CompactionConfig
from raven.contracts.llm_provider import LLMResponse, ToolCallRequest
from raven.providers.base import LLMProvider
from raven.spine import ChatType, Origin, Source, TurnRequest
class _ScriptedProvider:
"""chat_with_retry answers from a script and records every call's kwargs."""
def __init__(self, responses: list[LLMResponse]) -> None:
self._responses = list(responses)
self.calls: list[dict] = []
async def chat_with_retry(self, **kwargs) -> LLMResponse:
snapshot = dict(kwargs)
snapshot["messages"] = [dict(m) for m in kwargs.get("messages") or []]
self.calls.append(snapshot)
index = min(len(self.calls) - 1, len(self._responses) - 1)
return self._responses[index]
def get_default_model(self) -> str:
return "fake/model"
def _text(content: str) -> LLMResponse:
return LLMResponse(content=content, finish_reason="stop")
def _tool_call(name: str, arguments: dict) -> LLMResponse:
return LLMResponse(content="", tool_calls=[ToolCallRequest(id="c1", name=name, arguments=arguments)])
def _req(chat_id: str = "c") -> TurnRequest:
return TurnRequest(
origin=Origin.USER,
source=Source(channel="cli", chat_id=chat_id, sender_id="u", chat_type=ChatType.DM),
text="hi",
)
def _loop(tmp_path, provider, max_iterations: int = 6) -> AgentLoop:
loop = AgentLoop(
provider=provider,
workspace=tmp_path,
model="fake/model",
policy=TurnPolicy(max_iterations=max_iterations),
tools=ToolWiring(restrict_to_workspace=True),
)
async def _noop(**_kw) -> None:
return None
loop._start_executor = _noop
loop._connect_mcp = _noop
return loop
@pytest.mark.asyncio
async def test_a_session_policys_effort_reaches_every_model_call(tmp_path):
provider = _ScriptedProvider([_tool_call("list_dir", {"path": "."}), _text("done")])
loop = _loop(tmp_path, provider)
loop.set_session_policy("cli:c", mode="medium", reasoning_effort="medium")
out = await loop._process_message(_req())
assert out is not None
assert len(provider.calls) == 2
assert [call["reasoning_effort"] for call in provider.calls] == ["medium", "medium"]
@pytest.mark.asyncio
async def test_a_policy_naming_no_effort_passes_none_so_the_configured_default_stands(tmp_path):
provider = _ScriptedProvider([_text("done")])
loop = _loop(tmp_path, provider)
loop.set_session_policy("cli:c", mode="high")
await loop._process_message(_req())
assert "reasoning_effort" not in provider.calls[0], "an explicit None would override the provider's sentinel"
@pytest.mark.asyncio
async def test_another_session_runs_on_the_defaults(tmp_path):
provider = _ScriptedProvider([_text("done")])
loop = _loop(tmp_path, provider)
loop.set_session_policy("cli:c", mode="max", reasoning_effort="max")
await loop._process_message(_req(chat_id="d"))
assert "reasoning_effort" not in provider.calls[0]
@pytest.mark.asyncio
async def test_the_exhaustion_wrap_up_call_runs_at_the_sessions_effort(tmp_path):
"""The wrap-up after an exhausted iteration budget is a model call of the
same turn, so it pays the turn's effort too."""
provider = _ScriptedProvider([_tool_call("list_dir", {"path": "."})] * 3 + [_text("summary")])
loop = _loop(tmp_path, provider, max_iterations=1)
loop.set_session_policy("cli:c", mode="max", reasoning_effort="max")
out = await loop._process_message(_req())
assert out is not None
wrap_up = provider.calls[-1]
assert wrap_up["tools"] is None, "the wrap-up withholds the tools"
assert wrap_up["reasoning_effort"] == "max"
class _CompactingProvider(LLMProvider):
"""A main loop that blows past the compaction line, and a summary desk that
records the effort every head summary was asked for."""
def __init__(self) -> None:
super().__init__(api_key="test")
self.main_efforts: list[str | None] = []
self.summary_efforts: list[str | None] = []
async def chat(
self,
messages,
tools=None,
model=None,
max_tokens=4096,
temperature=0.7,
reasoning_effort=None,
tool_choice=None,
):
if messages and messages[0].get("content") == compaction.SUMMARY_INSTRUCTIONS:
self.summary_efforts.append(reasoning_effort)
return LLMResponse(content="Compacted handoff brief. " * 60, finish_reason="stop")
self.main_efforts.append(reasoning_effort)
if len(self.main_efforts) == 1:
return LLMResponse(
content="",
tool_calls=[ToolCallRequest(id="c1", name="no_such_tool", arguments={})],
finish_reason="tool_calls",
usage={"prompt_tokens": 4990, "completion_tokens": 10},
)
return LLMResponse(content="answer after compaction", finish_reason="stop")
def get_default_model(self) -> str:
return "fake/model"
def _seeded_transcript(rounds: int = 6) -> list[dict]:
messages: list[dict] = [
{"role": "system", "content": "sys prompt"},
{"role": "user", "content": "the task statement"},
]
for i in range(rounds):
messages.append(
{
"role": "assistant",
"content": "",
"tool_calls": [
{"id": f"s{i}", "type": "function", "function": {"name": "no_such_tool", "arguments": "{}"}}
],
}
)
messages.append({"role": "tool", "tool_call_id": f"s{i}", "content": f"seed evidence {i} " * 40})
return messages
@pytest.mark.asyncio
async def test_the_compaction_summary_opts_out_of_the_sessions_effort(tmp_path):
"""The one model call of a turn that does not take the session's effort.
It used to, on the reasoning that a head summary is a model call of the
same turn like the ReAct steps around it. They are not alike: a step thinks
in order to decide, a handoff brief is a transcript read back, and a
reasoning model pays for thinking out of the same budget as the brief.
Measured, two summary calls at the session's effort came back with an empty
body having spent the whole budget before the brief began, and compaction
degraded to blind elision for the rest of that run.
Deliberately narrow: the ReAct steps still run at the session's effort, and
so does the watch-work judgement below, which is a judgement rather than a
transcription.
"""
provider = _CompactingProvider()
loop = AgentLoop(
provider=provider,
workspace=tmp_path,
model="fake/model",
policy=TurnPolicy(max_iterations=12),
tools=ToolWiring(restrict_to_workspace=True),
engine=EngineWiring(
context_window_tokens=1000,
compaction_config=CompactionConfig(enabled=True, reserved_tokens=100, preserve_recent_tokens=300),
runtime_config=RuntimeConfig(checkpoint=CheckpointConfig(policy="never")),
),
)
loop.set_session_policy("acp:s1", mode="max", reasoning_effort="max")
final, _used, _messages, outcome = await loop._run_agent_loop(_seeded_transcript(), session_key="acp:s1")
assert final == "answer after compaction" and outcome.status == "completed"
assert provider.summary_efforts == [compaction.SUMMARY_REASONING_EFFORT], (
"a max session must not spend the summary's budget on thinking"
)
assert set(provider.main_efforts) == {"max"}, "only the summary opts out"
@pytest.mark.asyncio
async def test_the_compaction_summary_names_the_floor_even_with_no_policy_effort(tmp_path):
"""Unstated is not the same as little. The run this fix comes from named no
effort at all, so the backend picked -- and what it picked spent the whole
summary budget. The floor is stated outright rather than left open, which
does override a provider default configured for the turn's own calls.
"""
provider = _CompactingProvider()
loop = AgentLoop(
provider=provider,
workspace=tmp_path,
model="fake/model",
policy=TurnPolicy(max_iterations=12),
tools=ToolWiring(restrict_to_workspace=True),
engine=EngineWiring(
context_window_tokens=1000,
compaction_config=CompactionConfig(enabled=True, reserved_tokens=100, preserve_recent_tokens=300),
runtime_config=RuntimeConfig(checkpoint=CheckpointConfig(policy="never")),
),
)
loop.set_session_policy("acp:s1", mode="high")
await loop._run_agent_loop(_seeded_transcript(), session_key="acp:s1")
assert provider.summary_efforts == [compaction.SUMMARY_REASONING_EFFORT]
@pytest.mark.asyncio
async def test_the_watch_work_judgement_is_handed_the_sessions_effort(tmp_path):
"""The turn path hands the pinned effort to the watch-work helper for
every tool result it judges, so the in-turn classifier is not the one
model call of a max session that runs at the default."""
provider = _ScriptedProvider([_tool_call("list_dir", {"path": "."}), _text("done")])
loop = _loop(tmp_path, provider)
loop.set_session_policy("cli:c", mode="max", reasoning_effort="max")
handed: list[str | None] = []
async def spy(state, name, args, result, message, reasoning_effort=None):
handed.append(reasoning_effort)
return ""
loop._note_watch_work = spy
out = await loop._process_message(_req())
assert out is not None
assert handed == ["max"], handed