Raven-1/tests/test_cli_agent_commands.py
Zuyi Zhou bcbc295c0d
fix: record unanswered questions an unattended turn could not ask (#807)
## Summary

An unattended turn could ask a question and leave no trace a person can
read.

`raven agent -m` already lists calls the permission gate refused (#708).
It said nothing about a question nobody could answer. On the web, a
sub-agent turn with no page subscribed to the session binds no asker
(`start_ask_turn(None)` in `raven/rpc/spine.py`). The host `ask_user`
tool still tells the model "user did not answer; proceed with best
judgment". That sentence is the tool result the model reads. Opening the
session later showed no event for the question.

Questions that were never put to anyone are now recorded on the turn: a
sub-agent `ask_user` or elicitation with no asker, a question that gives
up waiting for the conversation lock, a round trip that is structurally
unavailable, and a host `ask_user` call whose answer comes back empty. A
one-shot call has no broker at all, so it returns at that guard; the
questions `_prepare` accepts are recorded there too, and the same error
string is what the model still reads. A call `_prepare` rejects is not
recorded. The turn's save writes one assistant entry with a
`question_unanswered` notice, which is what a reopened session draws. A
one-shot prints the same questions after the reply. That list does not
change the exit status: the turn continued, which is what the model was
told to do. A refusal is still the run that exits 3.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

```
uv run --frozen --python 3.12 --extra dev ruff check raven/agent/loop/turn_path.py raven/agent/tools/ask_user.py tests/test_ask_user_tool.py tests/test_cli_one_shot_spine.py tests/test_agents_research_flow_report_shape.py
  All checks passed

uv run --frozen --python 3.12 --extra dev ruff format --check <same files>
  already formatted

uv run --frozen --python 3.12 --extra dev pytest -q \
  tests/test_ask_user_tool.py::test_a_call_with_no_broker_records_the_normalized_questions \
  tests/test_ask_user_tool.py::test_error_paths_return_plain_strings \
  tests/test_cli_one_shot_spine.py::test_a_one_shot_ask_user_call_with_no_broker_is_what_gets_reported \
  tests/test_agents_research_flow_report_shape.py::test_the_bounce_scaffolding_never_becomes_history \
  tests/test_agent_loop_session_stamps.py::test_an_unanswered_question_is_stored_as_a_notice
  5 passed

cd ui-tui && npm run lint:i18n
  i18n: generated catalogue is up to date

PYTHONPATH=. uv run --frozen --python 3.12 --extra dev python scripts/check_source_language.py origin/main...HEAD
  exit 0

make check-commits && PR_TITLE="fix: record unanswered questions an unattended turn could not ask" make check-pr-title
  exit 0

git diff --check origin/main...HEAD
  exit 0

git merge-tree --write-tree HEAD origin/main
  clean
```

Reverting `raven/agent/tools/ask_user.py` to its parent makes both new
no-broker tests fail with an empty record. Reverting
`raven/agent/loop/turn_path.py` to the parent, where filing was a
method, makes `test_the_bounce_scaffolding_never_becomes_history` raise
`AttributeError: '_Loop' object has no attribute '_file_unanswered'`.
Both files were restored afterwards. The earlier reverts (notice not
stored, ACP ask and elicitation not recorded, one-shot report not
printed) still stand for those paths.

The full suite was not run. A run of the touched CLI modules in this
terminal also failed existing assertions that search Rich output for a
plain substring (help flags, and `1 action(s) were refused`). Those
assertions fail the same way with this diff's refusal printer left
unchanged. The type checker was not rerun locally; CI's python lint job
runs it on the head.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

The model is still told to continue. Nothing new is approved. The
one-shot exit status is unchanged: unanswered questions are reported and
do not by themselves exit 3. The notice is one more assistant entry on
the session, with `notice.kind` `question_unanswered`, so a client that
already draws runtime notices draws this one. Rollback is reverting the
commit.

## Related Issues

#708
2026-09-27 10:30:48 +08:00

1021 lines
39 KiB
Python

"""CLI tests for ``raven agent``.
The ``agent`` command is a one-shot ``-m`` single-turn runner (the
interactive REPL was removed; ``raven tui`` is the interactive front-end).
Smoke-level coverage: ``--help`` works, options are surfaced, the
``no-API-key`` path exits cleanly, bare invocation points at the TUI.
"""
from __future__ import annotations
import re
from collections.abc import Callable
from pathlib import Path
from types import SimpleNamespace
import pytest
from typer.testing import CliRunner
from raven.cli.commands import app
from raven.config.loader import set_config_path
runner = CliRunner()
@pytest.fixture
def tmp_config(tmp_path: Path) -> Path:
cfg = tmp_path / "config.json"
set_config_path(cfg)
yield cfg
set_config_path(None) # type: ignore[arg-type]
def test_agent_help_works() -> None:
"""``raven agent --help`` lists the key options."""
r = runner.invoke(app, ["agent", "--help"])
assert r.exit_code == 0
assert "one-shot agent turn" in r.stdout
# core options surfaced
assert "--message" in r.stdout
assert "--session" in r.stdout
assert "--workspace" in r.stdout
assert "--config" in r.stdout
assert "--markdown" in r.stdout
def test_home_flag_moves_agent_home(tmp_config: Path, tmp_path: Path):
from raven.core.config_stack import load_runtime_config
config = load_runtime_config(None, home=str(tmp_path / "elsewhere"))
assert config.agents.defaults.workspace == str(tmp_path / "elsewhere")
def test_agent_help_documents_both_directories():
r = runner.invoke(app, ["agent", "--help"])
assert r.exit_code == 0
assert "--home" in r.output
assert "Working directory" in r.output
def test_agent_help_omits_removed_skill_extract_flags() -> None:
"""The inert skill-extraction flags stay removed: the mechanism their
help text described was replaced by the MemoryBackend plugin, so the
flags never had any effect."""
r = runner.invoke(app, ["agent", "--help"])
assert r.exit_code == 0
assert "--wait-skill-extract" not in r.stdout
assert "--flush-skill-buffer" not in r.stdout
def test_agent_without_message_prints_pointer_and_exits_nonzero(tmp_config: Path) -> None:
"""Bare ``raven agent`` no longer enters an interactive loop: it points
at ``raven tui`` / ``agent -m`` and exits non-zero."""
r = runner.invoke(app, ["agent"])
assert r.exit_code != 0
assert "REPL was removed" in r.stdout
assert "raven tui" in r.stdout
assert "-m" in r.stdout
def test_agent_without_api_key_exits_cleanly(tmp_config: Path) -> None:
"""With no provider configured, the command must exit non-zero — and
crucially must not raise a *crash* exception (NameError / AttributeError /
ImportError). ``typer.testing.CliRunner`` captures the exception, so the
only reliable way to detect a regression like a missing import is to
inspect ``r.exception`` directly.
"""
from raven.config.loader import save_config
from raven.config.schema import Config
save_config(Config()) # default config, no keys
r = runner.invoke(app, ["agent", "-m", "hello"])
# Reject any crash-class exception: those signal a refactor regression,
# not user error. typer.Exit(...) is fine (intentional non-zero exit).
if r.exception is not None:
assert not isinstance(r.exception, (NameError, AttributeError, ImportError)), (
f"Crash-class exception leaked through: {r.exception!r}"
)
assert r.exit_code != 0
# ============================================================================
# Session binding flags (task 3.3)
# ============================================================================
def test_agent_help_shows_continue_flag() -> None:
"""--continue flag appears in agent --help."""
r = runner.invoke(app, ["agent", "--help"])
assert r.exit_code == 0
assert "--continue" in r.stdout
def test_agent_help_shows_resume_flag() -> None:
"""--resume flag appears in agent --help."""
r = runner.invoke(app, ["agent", "--help"])
assert r.exit_code == 0
assert "--resume" in r.stdout
async def _skip_background_grace(*_args, **_kwargs) -> None:
"""Stand-in for ``_wait_for_background_work``: the stub subagents/scheduler
below never go busy, so the real 2 s sub-agent hand-off grace has nothing
to wait for and only adds wall time."""
return None
def _invoke_agent_capturing_session(
monkeypatch: pytest.MonkeyPatch,
home: Path,
extra_args: list[str],
*,
backend: object | None = None,
drain_outcome: object | None = None,
message_args: list[str] | None = None,
during_turn: Callable[[], None] | None = None,
) -> tuple[object, dict[str, str]]:
"""Run ``agent -m`` with the provider and AgentLoop stubbed out, capturing
the session_id that reaches the spine turn (req.conversation is the session
key, mirroring the old session_key arg). ``message_args`` replaces the
default ``-m hi`` so the --message-file path can be exercised too.
``during_turn`` runs inside the turn's own task, where the gate would."""
from raven.config.loader import save_config
from raven.config.schema import Config
from raven.spine import Text, TurnOutcome, Usage
cfg = Config()
cfg.providers.openrouter.api_key = "stub-test-key"
save_config(cfg)
captured: dict[str, str] = {}
class _StubSubagents:
def set_submit(self, _submit) -> None:
pass
def get_running_count(self) -> int:
return 0
class _StubAgentLoop:
def __init__(self, **kwargs):
self.channels_config = kwargs.get("channels_config")
self.subagents = _StubSubagents()
# The teardown stops the loop's skill watcher, so the double
# carries the context the real loop always has.
self.context = SimpleNamespace(skills=SimpleNamespace(stop_file_watcher=lambda: None))
def configure_personalization(self, *_args) -> None:
pass
async def run_turn(self, req, emit, drain, *, stream, **_kw) -> TurnOutcome:
captured["session_id"] = req.conversation
captured["text"] = str(getattr(req, "text", ""))
if during_turn is not None:
during_turn()
await emit(Text(content="stub-response", source=req.source))
return TurnOutcome(usage=Usage(0, 0, 0), explicit_reply=True)
async def await_pending_extractions(self, **_kw) -> None:
pass
async def close_mcp(self) -> None:
pass
async def drain_backend_stores(self, *_a, **_kw):
from raven.memory_engine import DrainOutcome
return drain_outcome if drain_outcome is not None else DrainOutcome(lost=0, in_flight=0)
monkeypatch.setattr("raven.cli.agent_commands.make_provider", lambda _: object())
monkeypatch.setattr("raven.agent.loop.AgentLoop", _StubAgentLoop)
monkeypatch.setattr("raven.cli.agent_commands._wait_for_background_work", _skip_background_grace)
# This test exercises session keying, not memory: don't boot the real
# (bundled) everos backend / plugin tools inside the CliRunner (the
# embedded everos runtime is heavy and not under test here).
monkeypatch.setattr(
"raven.core.plugin_stack.maybe_build_memory_backend",
lambda *a, **k: backend,
)
monkeypatch.setattr(
"raven.core.plugin_stack.build_plugin_tools",
lambda *a, **k: [],
)
message = message_args if message_args is not None else ["-m", "hi"]
r = runner.invoke(app, ["agent", *message, "--home", str(home), *extra_args])
return r, captured
def test_agent_default_mints_fresh_session(tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Bare ``agent -m`` mints a fresh ``cli:{chat_id}`` per invocation."""
import re
ws = tmp_path / "chanwork"
ws.mkdir()
r1, cap1 = _invoke_agent_capturing_session(monkeypatch, ws, [])
assert r1.exit_code == 0, r1.stdout
assert re.fullmatch(r"cli:\d{8}_\d{6}_[0-9a-f]{6}", cap1["session_id"]), (
f"expected freshly minted cli session key, got {cap1['session_id']!r}"
)
r2, cap2 = _invoke_agent_capturing_session(monkeypatch, ws, [])
assert r2.exit_code == 0
assert cap1["session_id"] != cap2["session_id"], "each bare invocation must mint a NEW session"
def test_a_one_shot_shutdown_tells_the_user_what_the_drain_left(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``agent -m`` renders the drain outcome; it does not merely compute it.
The one-shot exit is the only place a CLI user learns a turn did not
settle -- there is no next screen to carry the notice. The same teardown
in ``tui_commands`` is driven by its own test and this copy was driven by
none, so dropping the render here would have cost nothing red.
"""
from raven.memory_engine import DrainOutcome
class _Backend:
def __init__(self) -> None:
self.stopped = False
async def start(self) -> None:
pass
async def stop(self) -> None:
self.stopped = True
ws = tmp_path / "chanwork"
ws.mkdir()
backend = _Backend()
r, _ = _invoke_agent_capturing_session(
monkeypatch,
ws,
[],
backend=backend,
drain_outcome=DrainOutcome(lost=2, in_flight=0),
)
assert r.exit_code == 0, r.stdout
assert "2 turn(s) were not written" in r.stdout
# The drain has to precede the stop: stopping closes the HTTP client the
# queued writes still need.
assert backend.stopped
def test_agent_continue_binds_most_recent_cli_session(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``-c`` binds the agent to the most-recent persisted cli session."""
from raven.session.manager import SessionManager
ws = tmp_path / "chanwork"
ws.mkdir()
mgr = SessionManager(ws)
seeded = "20990101_000000_aaaaaa"
s = mgr.get_or_create(f"cli:{seeded}")
s.add_message("user", "earlier turn")
mgr.save(s)
r, captured = _invoke_agent_capturing_session(monkeypatch, ws, ["-c"])
assert r.exit_code == 0, r.stdout
assert captured["session_id"] == f"cli:{seeded}"
def test_agent_resume_binds_resolved_session(tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""``--resume <prefix>`` resolves and binds that cli session."""
from raven.session.manager import SessionManager
ws = tmp_path / "chanwork"
ws.mkdir()
mgr = SessionManager(ws)
seeded = "20990101_000000_bbbbbb"
s = mgr.get_or_create(f"cli:{seeded}")
s.add_message("user", "earlier turn")
mgr.save(s)
r, captured = _invoke_agent_capturing_session(monkeypatch, ws, ["--resume", seeded[:20]])
assert r.exit_code == 0, r.stdout
assert captured["session_id"] == f"cli:{seeded}"
def test_agent_session_key_passthrough(tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""``--session <key>`` passes a full key through unchanged (any channel)."""
ws = tmp_path / "chanwork"
ws.mkdir()
r, captured = _invoke_agent_capturing_session(monkeypatch, ws, ["--session", "feishu:ou_xyz"])
assert r.exit_code == 0, r.stdout
assert captured["session_id"] == "feishu:ou_xyz"
def test_agent_bare_session_resolves_cross_channel(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``--session <bare id>`` resolves to an existing session on a non-cli
channel — it must NOT be mis-routed to a colon-less/malformed key."""
from raven.session.manager import SessionManager
ws = tmp_path / "chanwork"
ws.mkdir()
mgr = SessionManager(ws)
cid = "20990101_000000_cccccc"
s = mgr.get_or_create(f"tui:{cid}")
s.add_message("user", "earlier turn")
mgr.save(s)
r, captured = _invoke_agent_capturing_session(monkeypatch, ws, ["--session", cid])
assert r.exit_code == 0, r.stdout
assert captured["session_id"] == f"tui:{cid}"
def test_agent_unknown_bare_session_falls_back_to_cli(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``--session <bare id>`` with no matching session falls back to a proper
``cli:<id>`` key — never a colon-less/malformed path."""
ws = tmp_path / "chanwork"
ws.mkdir()
cid = "20990101_000000_dddddd"
r, captured = _invoke_agent_capturing_session(monkeypatch, ws, ["--session", cid])
assert r.exit_code == 0, r.stdout
assert captured["session_id"] == f"cli:{cid}"
@pytest.mark.parametrize(
"args",
[
["-c", "--resume", "x"],
["--session", "cli:abc", "-c"],
["--session", "cli:abc", "--resume", "x"],
["--session", "cli:abc", "-c", "--resume", "x"],
],
)
def test_agent_session_binding_flags_mutually_exclusive(tmp_config: Path, args: list[str]) -> None:
"""More than one of --session/--continue/--resume exits with usage error."""
r = runner.invoke(app, ["agent", "-m", "hi", *args])
assert r.exit_code == 2, f"expected usage error, got {r.exit_code}: {r.stdout}"
assert "mutually exclusive" in r.stdout
def test_agent_continue_without_prior_session_starts_fresh(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``-c`` with no stored cli session prints a notice and mints fresh."""
import re
ws = tmp_path / "chanwork"
ws.mkdir()
r, captured = _invoke_agent_capturing_session(monkeypatch, ws, ["-c"])
assert r.exit_code == 0, r.stdout
assert re.fullmatch(r"cli:\d{8}_\d{6}_[0-9a-f]{6}", captured["session_id"])
assert "no previous cli session" in r.stdout
@pytest.mark.asyncio
async def test_wait_for_background_work_covers_subagent_follow_up(monkeypatch: pytest.MonkeyPatch) -> None:
"""One-shot teardown waits for a sub-agent and its submitted follow-up."""
from raven.cli import agent_commands
state = {"subagent_running": True, "follow_up_running": False}
sleeps: list[float] = []
class _Subagents:
def get_running_count(self) -> int:
return int(state["subagent_running"])
class _Scheduler:
def has_inflight(self, conversation: str) -> bool:
return conversation == "cli:direct" and state["follow_up_running"]
async def _sleep(delay: float) -> None:
sleeps.append(delay)
if delay == 1.0 and state["subagent_running"]:
state["subagent_running"] = False
state["follow_up_running"] = True
elif delay == 1.0:
state["follow_up_running"] = False
elif delay == 2.0:
state["follow_up_running"] = False
monkeypatch.setattr(agent_commands.asyncio, "sleep", _sleep)
from types import SimpleNamespace
await agent_commands._wait_for_background_work(SimpleNamespace(subagents=_Subagents()), _Scheduler(), "cli:session")
assert sleeps == [1.0, 1.0, 2.0]
# ============================================================================
# No cron on the one-shot path
# ============================================================================
def test_agent_message_mode_constructs_no_cron_service(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``agent -m`` must not build a CronService (and hence never registers
CronTool): with the REPL gone this process is never a cron runner, so a
cli-bound job it created would have nothing to fire it."""
import raven.proactive_engine.schedulers.cron.service as cron_mod
ws = tmp_path / "chanwork"
ws.mkdir()
constructed: list[object] = []
orig_init = cron_mod.CronService.__init__
def _spy_init(self, *args, **kwargs) -> None:
constructed.append(self)
orig_init(self, *args, **kwargs)
monkeypatch.setattr(cron_mod.CronService, "__init__", _spy_init)
r, _ = _invoke_agent_capturing_session(monkeypatch, ws, [])
assert r.exit_code == 0, r.stdout
assert constructed == [], "agent -m must not construct a CronService"
# ============================================================================
# Pure helper functions
# ============================================================================
def test_print_agent_response_with_markdown(capsys: pytest.CaptureFixture) -> None:
"""``_print_agent_response`` renders the body — markdown mode."""
from raven.cli.agent_commands import _print_agent_response
_print_agent_response("# hi", render_markdown=True)
out = capsys.readouterr().out
# rich's Markdown renderer typically prints the heading text
assert "hi" in out
def test_print_agent_response_plain(capsys: pytest.CaptureFixture) -> None:
"""``_print_agent_response`` renders plain text — markdown disabled."""
from raven.cli.agent_commands import _print_agent_response
_print_agent_response("hello world", render_markdown=False)
out = capsys.readouterr().out
assert "hello world" in out
# ============================================================================
# agent -m single-shot mode (mocked)
# ============================================================================
def test_agent_message_mode_mocked_provider(tmp_config: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""``agent -m 'hi'`` with a mocked provider must reach a clean exit
(no traceback). We mock ``make_provider`` so the agent loop builds
without contacting any LLM."""
from raven.config.loader import save_config
from raven.config.schema import Config
# Save a config with the openrouter key so the gateway-validation passes.
cfg = Config()
cfg.providers.openrouter.api_key = "stub-test-key"
save_config(cfg)
monkeypatch.setattr(
"raven.cli.agent_commands.make_provider",
lambda _: (_ for _ in ()).throw(RuntimeError("mock-no-provider")),
)
# We can't drive the full agent loop without a real provider; we only
# assert that the CLI exits cleanly (no uncaught traceback) when the
# provider build raises a controlled error.
r = runner.invoke(app, ["agent", "-m", "hello"])
if r.exception is not None:
assert not isinstance(r.exception, (NameError, AttributeError, ImportError)), (
f"Crash-class exception leaked through: {r.exception!r}"
)
# ============================================================================
# Provider error rendering
# ============================================================================
def test_agent_auth_error_exit_nonzero_with_guidance(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A 401 from the provider must exit non-zero, show a fix hint, and must
not dump the duplicated exception name or the raw JSON error body.
The provider is a real ``LiteLLMProvider`` whose ``acompletion`` is stubbed
to raise the same exception shape litellm raises on an OpenRouter 401 —
no network involved.
"""
from raven.config.loader import save_config
from raven.config.schema import Config
from raven.spine import Text, TurnOutcome, Usage
cfg = Config()
cfg.providers.openrouter.api_key = "sk-or-v1-stub-invalid"
save_config(cfg)
class AuthenticationError(Exception):
status_code = 401
async def _raise_401(**_kwargs):
raise AuthenticationError(
"litellm.AuthenticationError: AuthenticationError: OpenrouterException - "
'{"error":{"message":"User not found.","code":401}}'
)
# Pre-register the env var with monkeypatch so the provider's _setup_env
# write is rolled back with the test instead of leaking a fake key.
monkeypatch.setenv("OPENROUTER_API_KEY", "sk-or-v1-stub-invalid")
monkeypatch.setattr("raven.providers.litellm_provider.acompletion", _raise_401)
monkeypatch.setattr("raven.cli.agent_commands._wait_for_background_work", _skip_background_grace)
class _StubSubagents:
def set_submit(self, _submit) -> None:
pass
def get_running_count(self) -> int:
return 0
class _AuthFailAgentLoop:
def __init__(self, **kwargs):
self.channels_config = kwargs.get("channels_config")
self.subagents = _StubSubagents()
self.context = SimpleNamespace(skills=SimpleNamespace(stop_file_watcher=lambda: None))
def configure_personalization(self, *_args) -> None:
pass
async def run_turn(self, req, emit, drain, *, stream, **_kw) -> TurnOutcome:
from raven.providers.litellm_provider import LiteLLMProvider
provider = LiteLLMProvider(
api_key="sk-or-v1-stub-invalid",
default_model="anthropic/claude-opus-4-5",
provider_name="openrouter",
)
resp = await provider.chat_with_retry(messages=[{"role": "user", "content": req.text}])
await emit(Text(content=resp.content or "", source=req.source))
return TurnOutcome(usage=Usage(0, 0, 0), explicit_reply=True)
async def await_pending_extractions(self, **_kw) -> None:
pass
async def close_mcp(self) -> None:
pass
monkeypatch.setattr("raven.cli.agent_commands.make_provider", lambda _: object())
monkeypatch.setattr("raven.agent.loop.AgentLoop", _AuthFailAgentLoop)
monkeypatch.setattr("raven.core.plugin_stack.maybe_build_memory_backend", lambda *a, **k: None)
monkeypatch.setattr("raven.core.plugin_stack.build_plugin_tools", lambda *a, **k: [])
ws = tmp_path / "ws"
ws.mkdir()
result = runner.invoke(app, ["agent", "-m", "hi", "-w", str(ws)])
assert result.exit_code != 0
assert result.output.count("AuthenticationError") <= 1
assert "raven provider" in result.output
assert '{"error"' not in result.output
def test_print_turn_failure_draws_a_failure_not_a_reply(capsys: pytest.CaptureFixture) -> None:
"""A failed turn's words are printed as a failure and the command exits
non-zero. The canonical error sentence still takes the diagnosis path; any
other text is marked as an error rather than drawn under the answer banner."""
from raven.cli import agent_commands
agent_commands._ONE_SHOT_EXIT["code"] = 0
try:
agent_commands._print_turn_failure("APIError: OpenrouterException - Cannot connect to host")
out = capsys.readouterr().out
assert "Error: turn failed: APIError: OpenrouterException - Cannot connect to host" in out
assert "Raven" not in out
assert agent_commands._ONE_SHOT_EXIT["code"] == 1
# Provider text is not markup: a bracketed path must print, not raise.
agent_commands._print_turn_failure("APIError: bad value [/v1/chat] rejected")
out = capsys.readouterr().out
assert "[/v1/chat]" in out
agent_commands._ONE_SHOT_EXIT["code"] = 0
agent_commands._print_turn_failure("Error calling LLM (network@openrouter): connection refused")
out = capsys.readouterr().out
assert "Error: turn failed" not in out
assert "connection refused" in out
assert agent_commands._ONE_SHOT_EXIT["code"] == 1
finally:
agent_commands._ONE_SHOT_EXIT["code"] = 0
def test_print_llm_error_renders_diagnosis_and_marks_exit(capsys: pytest.CaptureFixture) -> None:
"""``_print_llm_error`` turns the canonical error content into a
diagnosis + fix hint and marks the one-shot exit code; ordinary reply
content is left for the normal renderer."""
from raven.cli import agent_commands
from raven.providers.base import LLMProvider, format_llm_error
class _AuthError(Exception):
status_code = 401
exc = _AuthError(
"litellm.AuthenticationError: AuthenticationError: OpenrouterException - "
'{"error":{"message":"User not found.","code":401}}'
)
content = format_llm_error(exc, LLMProvider.classify_error(exc), provider="openrouter")
agent_commands._ONE_SHOT_EXIT["code"] = 0
try:
assert agent_commands._print_llm_error(content) is True
out = capsys.readouterr().out
assert "provider rejected the credentials" in out
assert "openrouter" in out
assert "User not found." in out
assert "raven provider test openrouter" in out
assert '{"error"' not in out
assert agent_commands._ONE_SHOT_EXIT["code"] == 1
agent_commands._ONE_SHOT_EXIT["code"] = 0
assert agent_commands._print_llm_error("just a normal reply") is False
assert agent_commands._ONE_SHOT_EXIT["code"] == 0
finally:
agent_commands._ONE_SHOT_EXIT["code"] = 0
def test_print_llm_error_auth_does_not_fabricate_a_status_code(capsys: pytest.CaptureFixture) -> None:
"""The auth bucket also fires on 403 / PermissionDeniedError, so the
rendered line must not claim 401; the provider's own reason is what the
user needs to tell an invalid key from a key without access."""
from raven.cli import agent_commands
from raven.providers.base import LLMProvider, format_llm_error
class _PermissionDeniedError(Exception):
status_code = 403
exc = _PermissionDeniedError("permission denied: your key has no access to anthropic/claude-opus-4-5")
classification = LLMProvider.classify_error(exc)
assert classification.category == "auth"
content = format_llm_error(exc, classification, provider="openrouter")
agent_commands._ONE_SHOT_EXIT["code"] = 0
try:
assert agent_commands._print_llm_error(content) is True
out = capsys.readouterr().out
assert "401" not in out
assert "provider rejected the credentials (openrouter)" in out
assert "no access to anthropic/claude-opus-4-5" in out
assert "raven provider test openrouter" in out
assert agent_commands._ONE_SHOT_EXIT["code"] == 1
finally:
agent_commands._ONE_SHOT_EXIT["code"] = 0
def test_print_llm_error_non_auth_categories_get_apt_hint_not_key_guidance(
capsys: pytest.CaptureFixture,
) -> None:
"""Non-auth categories must not print the credential fix line: a rate
limit or a network drop is not fixed by re-checking the key. Each gets a
category-apt hint instead; categories with no apt one-liner render the
error line alone."""
from raven.cli import agent_commands
cases = {
"rate_limit": "retry",
"network": "connectivity",
"context_overflow": "shorten",
"server": "retry",
"model_unavailable": "provider use",
"billing": "account",
}
try:
for category, expected in cases.items():
agent_commands._ONE_SHOT_EXIT["code"] = 0
content = f"Error calling LLM ({category}@openrouter): something went wrong"
assert agent_commands._print_llm_error(content) is True, category
out = capsys.readouterr().out
assert "raven provider test" not in out, category
assert "raven onboard" not in out, category
assert expected in out, f"{category}: missing apt hint in {out!r}"
assert agent_commands._ONE_SHOT_EXIT["code"] == 1, category
agent_commands._ONE_SHOT_EXIT["code"] = 0
assert agent_commands._print_llm_error("Error calling LLM (unknown): boom") is True
out = capsys.readouterr().out
assert "raven provider test" not in out
assert "Fix:" not in out
assert "Hint:" not in out
assert agent_commands._ONE_SHOT_EXIT["code"] == 1
finally:
agent_commands._ONE_SHOT_EXIT["code"] = 0
# ---------------------------------------------------------------------------
# Workspace template sync prints one summary line, not one per file, and
# the one-shot path renders a per-turn tokens/cost summary line.
# ---------------------------------------------------------------------------
def test_workspace_sync_prints_single_summary(tmp_path: Path, capsys: pytest.CaptureFixture) -> None:
from raven.utils.workspace import sync_workspace_templates
ws = tmp_path / "workspace"
added = sync_workspace_templates(ws)
first = capsys.readouterr()
first_text = first.out + first.err
assert added
assert "Initialized workspace" in first_text
assert first_text.count("Created") == 0
added_again = sync_workspace_templates(ws)
second = capsys.readouterr()
assert added_again == []
assert (second.out + second.err).strip() == ""
(ws / added[0]).unlink()
re_added = sync_workspace_templates(ws)
third = capsys.readouterr()
assert len(re_added) == 1
assert "(1 file)" in (third.out + third.err)
def test_workspace_sync_debug_detail_lifts_with_raven_logging(tmp_path: Path) -> None:
"""The module-level logger.disable in helpers yields to a later
logger.enable('raven'): loguru drops descendant rules whenever a parent
rule is set, so callers that enable logging before syncing get the
per-file detail. Freezes the behavior the helpers comment relies on."""
from loguru import logger
from raven.utils.workspace import sync_workspace_templates
records: list[str] = []
sink_id = logger.add(lambda m: records.append(str(m)), level="DEBUG")
try:
logger.enable("raven")
sync_workspace_templates(tmp_path / "ws", silent=True)
finally:
logger.remove(sink_id)
assert any("workspace sync: created" in m for m in records)
def _invoke_agent_with_usage(monkeypatch: pytest.MonkeyPatch, tmp_path: Path, *, turn_summary_off: bool = False):
"""Run ``agent -m`` with a stub AgentLoop that reports LLM usage through
the TokenWise after-hook, mirroring how the real loop feeds UsageTracker."""
from raven.config.loader import save_config
from raven.config.schema import Config
from raven.contracts.token_strategy import UsageSnapshot
from raven.spine import Text, TurnOutcome, Usage
from raven.token_wise.registry import StrategyRegistry
cfg = Config()
cfg.providers.openrouter.api_key = "stub-test-key"
if turn_summary_off:
cfg.cli.turn_summary = False
save_config(cfg)
class _StubSubagents:
def set_submit(self, _submit) -> None:
pass
def get_running_count(self) -> int:
return 0
class _StubAgentLoop:
def __init__(self, **kwargs):
self.channels_config = kwargs.get("channels_config")
self.subagents = _StubSubagents()
self.context = SimpleNamespace(skills=SimpleNamespace(stop_file_watcher=lambda: None))
self.strategies = StrategyRegistry([])
def configure_personalization(self, *_args) -> None:
pass
async def run_turn(self, req, emit, drain, *, stream, **_kw) -> TurnOutcome:
await self.strategies.after_llm_call(
{"content": "stub"},
UsageSnapshot(
model="stub-model",
input_tokens=1200,
output_tokens=340,
cost_usd=0.004,
session_key=req.conversation,
),
)
await emit(Text(content="stub-response", source=req.source))
return TurnOutcome(usage=Usage(1200, 340, 1540), explicit_reply=True)
async def await_pending_extractions(self, **_kw) -> None:
pass
async def close_mcp(self) -> None:
pass
monkeypatch.setattr("raven.cli.agent_commands.make_provider", lambda _: object())
monkeypatch.setattr("raven.agent.loop.AgentLoop", _StubAgentLoop)
monkeypatch.setattr("raven.cli.agent_commands._wait_for_background_work", _skip_background_grace)
monkeypatch.setattr("raven.core.plugin_stack.maybe_build_memory_backend", lambda *a, **k: None)
monkeypatch.setattr("raven.core.plugin_stack.build_plugin_tools", lambda *a, **k: [])
return runner.invoke(app, ["agent", "-m", "hi", "-w", str(tmp_path / "ws")])
def test_turn_summary_line_present(tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
r = _invoke_agent_with_usage(monkeypatch, tmp_path)
assert re.search(r"\d[\d.,k]*\s*(in|tokens)", r.output)
def test_turn_summary_respects_config_off(tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
r = _invoke_agent_with_usage(monkeypatch, tmp_path, turn_summary_off=True)
assert not re.search(r"\d[\d.,k]*\s*(in|tokens)", r.output)
def test_one_shot_teardown_stops_the_skill_file_watcher() -> None:
"""The one-shot teardown must stop the skill watcher before exit.
The loop's ``ContextBuilder`` starts ``SkillFileWatcher``, a daemon thread
parked inside ``watchfiles``' Rust ``watch()``. Nothing here stopped it, so
``Py_FinalizeEx`` tore the interpreter down under that native call:
measured 2026-09-18, a one-shot that rendered its whole turn and its own
teardown still exited 139.
This command used to carry its own ``os._exit(0)`` for that hazard. It was
traded for a hard-exit gate in ``raven.cli.commands.run``, and that gate has
since been removed, so the exit is an ordinary one again and the thread has
to actually be stopped.
The teardown is nested in the command with no import seam, so this pins the
call in the source.
"""
import inspect
from raven.cli import agent_commands
src = inspect.getsource(agent_commands.register)
teardown = src.split("finally:", 1)[1]
assert "stop_file_watcher()" in teardown
def test_agent_help_shows_the_message_file_and_permission_mode_flags() -> None:
r = runner.invoke(app, ["agent", "--help"])
assert r.exit_code == 0
assert "--message-file" in r.stdout
assert "--permission-mode" in r.stdout
def test_message_file_is_read_instead_of_argv(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""An unattended driver passes the prompt in a file: it must reach the turn.
The point of the flag is that the prompt is NOT in this process's argv,
where every other process in the sandbox can read it.
"""
ws = tmp_path / "chanwork"
ws.mkdir()
prompt = tmp_path / "prompt.md"
prompt.write_text("judge the gates and write the report\n", encoding="utf-8")
r, captured = _invoke_agent_capturing_session(monkeypatch, ws, [], message_args=["--message-file", str(prompt)])
assert r.exit_code == 0, r.stdout
assert "judge the gates" in captured["text"]
def test_message_and_message_file_are_mutually_exclusive(tmp_config: Path, tmp_path: Path) -> None:
prompt = tmp_path / "prompt.md"
prompt.write_text("hi\n", encoding="utf-8")
r = runner.invoke(app, ["agent", "-m", "hi", "--message-file", str(prompt)])
assert r.exit_code != 0
assert "mutually exclusive" in r.stdout
def test_an_empty_message_file_is_a_usage_error(tmp_config: Path, tmp_path: Path) -> None:
prompt = tmp_path / "prompt.md"
prompt.write_text(" \n", encoding="utf-8")
r = runner.invoke(app, ["agent", "--message-file", str(prompt)])
assert r.exit_code != 0
assert "is empty" in r.stdout
def test_a_missing_message_file_is_a_usage_error(tmp_config: Path, tmp_path: Path) -> None:
r = runner.invoke(app, ["agent", "--message-file", str(tmp_path / "nope.md")])
assert r.exit_code != 0
assert "--message-file" in r.stdout
def test_permission_mode_binds_the_mode_to_this_turns_session(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``--permission-mode full`` is how an unattended run says it has no reader."""
from raven.permissions import session_mode
ws = tmp_path / "chanwork"
ws.mkdir()
r, captured = _invoke_agent_capturing_session(monkeypatch, ws, ["--permission-mode", "full"])
assert r.exit_code == 0, r.stdout
assert session_mode(captured["session_id"]) == "full"
def test_an_unknown_permission_mode_names_the_ones_that_exist(tmp_config: Path) -> None:
r = runner.invoke(app, ["agent", "-m", "hi", "--permission-mode", "yolo"])
assert r.exit_code != 0
assert "ask" in r.stdout and "full" in r.stdout
def test_a_plain_reply_is_not_rewrapped_by_the_console(capsys: pytest.CaptureFixture) -> None:
"""A driver parses what the one-shot printed, so the console adds no newlines.
Hard-wrapping at the terminal width lands a line break inside a JSON string
literal, and the reply the caller has to parse stops being JSON.
"""
from raven.cli.agent_commands import _print_agent_response
line = '{"text": "' + "word " * 60 + '"}'
_print_agent_response(line, render_markdown=False)
printed = capsys.readouterr().out
assert line in printed
def test_a_one_shot_whose_mutations_were_refused_says_so_and_exits_3(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Observed: an unattended ``-m`` under the ask tier had every write, spawn
and mutating exec refused, was told to carry on without them, and exited 0
with nothing on screen to say so. A driver reading the status code could not
tell that run from one that did the work."""
from raven.permissions.turn import note_refusal
ws = tmp_path / "chanwork"
ws.mkdir()
def refuse_twice() -> None:
reason = "This call requires user approval (ask tier), but this turn is not interactive"
note_refusal("write_file", "write_file path=report.md", reason, "unattended")
note_refusal("write_file", "write_file path=report.md", "denied earlier", "denied_earlier")
r, _ = _invoke_agent_capturing_session(monkeypatch, ws, [], during_turn=refuse_twice)
assert r.exit_code == 3, r.stdout
assert "stub-response" in r.stdout
# One action, reported once, however many times the model repeated it.
assert "1 action(s) were refused" in r.stdout
assert "write_file path=report.md" in r.stdout
assert "--permission-mode full" in r.stdout
def test_a_one_shot_refused_only_by_a_deny_rule_reports_it_and_exits_0(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A deny rule is the operator's own policy doing what it was set to do:
worth saying, not a failed run, and no advice to widen permissions."""
from raven.permissions.turn import note_refusal
ws = tmp_path / "chanwork"
ws.mkdir()
def deny() -> None:
note_refusal("exec", "git push origin main", "blocked by a deny rule", "user_deny")
r, _ = _invoke_agent_capturing_session(monkeypatch, ws, [], during_turn=deny)
assert r.exit_code == 0, r.stdout
assert "1 action(s) were refused" in r.stdout
assert "git push origin main" in r.stdout
assert "--permission-mode" not in r.stdout
def test_a_one_shot_whose_questions_went_unanswered_says_so_and_exits_0(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A question nobody could answer used to vanish into the model's context.
The run still succeeded: the model was told to continue."""
from raven.permissions.turn import note_unanswered
ws = tmp_path / "chanwork"
ws.mkdir()
def ask_twice() -> None:
note_unanswered("Which base branch?")
note_unanswered("Which base branch?")
r, _ = _invoke_agent_capturing_session(monkeypatch, ws, [], during_turn=ask_twice)
plain = re.sub(r"\x1b\[[0-9;]*m", "", r.stdout)
assert r.exit_code == 0, plain
assert "stub-response" in plain
assert "1 question(s) went unanswered" in plain
assert plain.count("Which base branch?") == 1
assert "best judgment" in plain
assert "were refused" not in plain
assert "--permission-mode" not in plain
def test_a_one_shot_with_nothing_refused_prints_no_report(
tmp_config: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
ws = tmp_path / "chanwork"
ws.mkdir()
r, _ = _invoke_agent_capturing_session(monkeypatch, ws, [])
assert r.exit_code == 0, r.stdout
assert "were refused" not in r.stdout