Raven-1/tests/test_agent_loop_first_call_guard.py
0xKT 8716ec525c feat(docker): run raven in a docker container and make a first run configurable from the page
## Summary

Two things arrive together because the second is what made the first usable.

**Run raven from a container.** A Dockerfile builds the page, the Python
environment and the engine into one image; `cd docker && docker compose up`
serves it. nginx serves the built page and proxies the RPC socket to the engine
over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a
second container would have no address to dial. Compose publishes on 127.0.0.1,
and the entrypoint prints a one-time sign-in link on boot.

**A first run is no longer a dead end.** The gateway used to refuse to start
until a provider was configured, while being the thing that serves the page
where a provider gets configured. It now starts, says what is missing, and picks
up a key saved in Settings on the next turn without a restart. The onboarding
overlay no longer takes the page either: New chat, the model chip and Send send
the reader to Settings > Models instead, and an empty model list is marked on
the control that fills it.

Also carried: the provider catalogue registry (capabilities, modalities and
context windows per model), the reworked Settings > Models pane, and four fixes
found in review of that work -- a sibling catalogue answering 200 with a
non-catalogue body no longer crashes the whole fetch; adding a tag no longer
erases the label and description already stored for that model; `authenticated`
no longer demands a bearer token from a local provider reached by address; and
registry rows are keyed by the spelling every lookup produces, which recovers
593 rows that no reader could reach.

## Type

- [ ] Fix
- [x] Feature
- [x] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Commands run on the branch head, with their results:

- `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped
- `npx vitest run` in `ui-web` -- 1573 passed (101 files)
- `npx vitest run` in `ui-tui` -- 2037 passed (142 files)
- `uv run ruff check raven scripts tests` and `ruff format --check` -- clean
- `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean
- `npm run lint:rpc` in `ui-tui` -- generated.ts in sync
- `make check-source-language`, `make check-large-files` -- clean

Each of the four review fixes was checked by reverting it and watching a named
test go red, then restoring it.

The container image itself has not been built by CI or by me -- no Docker daemon
was available on the machine this was written on. It has been built and run by
the author of the branch.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

Two deliberate security trade-offs in the container deployment, both documented
in `docker/README.md`:

- nginx rewrites the `Origin` header, because the engine accepts only an Origin
  naming its own loopback port and no browser reaching a published port can send
  one. This drops a defence-in-depth layer. The session cookie is
  SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite
  ignores ports, so another page on localhost counts as same-site.
- `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to
  every page load, so anyone who can reach the published port is signed in. It
  is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any
  deployment reachable from elsewhere. These are two settings that must agree,
  and nothing warns when they do not.

Behaviour changes outside the container: the gateway now starts without a
configured provider rather than exiting 1, and the first-run onboarding overlay
no longer opens by default (`?onboard=1` still forces it).

Rollback is by revert; nothing here migrates state or changes an on-disk format.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-11 03:27:52 +00:00

246 lines
8.6 KiB
Python

"""The loop's own pre-request awaits are bounded, and a stall there is reported.
Every timeout a provider holds can only fire once the call has been entered.
The deck stall of 2026-09-10 never got that far: ``Iteration 1/600`` opened,
``memory.recall`` and ``skill.inject`` finished, and then fifteen minutes passed
with no ``llm.call`` span, no ``llm.input`` artefact, no retry line and no
established outbound socket -- the turn never issued its first request, so
nothing at the provider could have caught it. A retry of the identical turn was
healthy.
Covers:
- a hook that never returns does not hold the turn: the stage is abandoned at
``llmFirstByteTimeout`` and the model is asked anyway
- a strategy that never returns leaves the request un-hooked rather than unsent
- a ``TimeoutError`` the stage raised itself is not this guard's deadline and
propagates, so a bad pre-process still cannot send an unprocessed request
- the record names the stage, the bound and how long it waited
- a budget of 0 switches the guard off, and a stage that answers in time is
untouched
"""
from __future__ import annotations
import asyncio
import tempfile
from pathlib import Path
from typing import Any
import pytest
from raven.agent.loop import AgentLoop
from raven.agent.loop.bundles import EngineWiring, HostWiring, ToolWiring, TurnPolicy
from raven.agent.loop.first_call import FirstCallGuard
from raven.contracts.llm_provider import GenerationSettings
from raven.contracts.loop_hooks import AgentHook, AgentHookContext, HookDecision
from raven.contracts.token_strategy import TokenStrategy
from raven.providers.base import LLMProvider, LLMResponse
from raven.spine.message import ChatType, Source
from raven.spine.turn import Origin, TurnRequest
from raven.token_wise.registry import StrategyRegistry
@pytest.fixture
def workspace():
with tempfile.TemporaryDirectory() as td:
yield Path(td)
class _Answers(LLMProvider):
"""Answers at once, and records what it was asked with."""
def __init__(self, first_byte_timeout: float = 0.05):
super().__init__(api_key="test")
self.generation = GenerationSettings(timeout=600.0, first_byte_timeout=first_byte_timeout)
self.calls = 0
self.asked_tools: list[Any] = []
def get_default_model(self) -> str:
return "stub"
_CHAT_RETRY_DELAYS = ()
async def chat(self, messages, tools=None, model=None, max_tokens=4096, temperature=0.7, **_):
self.calls += 1
self.asked_tools.append(tools)
return LLMResponse(content="real answer", finish_reason="stop")
async def chat_stream(self, *args, **kwargs): # pragma: no cover - non-stream path under test
raise NotImplementedError
class _StallingHook(AgentHook):
"""A ``before_iteration`` that never comes back."""
def __init__(self) -> None:
self.entered = 0
async def before_iteration(self, ctx: AgentHookContext) -> HookDecision:
self.entered += 1
await asyncio.sleep(30)
return HookDecision(short_circuit_result="the hook answered") # pragma: no cover
class _StallingStrategy(TokenStrategy):
"""A ``before_llm_call`` that never comes back."""
name = "stalls"
def __init__(self) -> None:
self.entered = 0
async def before_llm_call(self, messages, tools, model):
self.entered += 1
await asyncio.sleep(30)
return [], None, model # pragma: no cover
def _agent(workspace: Path, provider: LLMProvider, *, hooks=None, strategies=None) -> AgentLoop:
return AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
policy=TurnPolicy(max_iterations=4),
tools=ToolWiring(restrict_to_workspace=True),
host=HostWiring(hooks=hooks or []),
engine=EngineWiring(strategies=StrategyRegistry(strategies or [])),
)
async def _turn(agent: AgentLoop):
return await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="build the deck",
),
session_key="s1",
)
@pytest.mark.asyncio
async def test_a_hook_that_never_returns_does_not_hold_the_turn(workspace):
hook = _StallingHook()
provider = _Answers()
agent = _agent(workspace, provider, hooks=[hook])
out = await asyncio.wait_for(_turn(agent), 20)
assert hook.entered >= 1, "the stalling hook really was entered"
assert provider.calls >= 1, "the model was asked even though the hook never answered"
assert out is not None and out[0] == "real answer"
@pytest.mark.asyncio
async def test_a_stalled_strategy_leaves_the_request_unhooked(workspace):
strategy = _StallingStrategy()
provider = _Answers()
agent = _agent(workspace, provider, strategies=[strategy])
out = await asyncio.wait_for(_turn(agent), 20)
assert strategy.entered >= 1
assert provider.calls >= 1
assert provider.asked_tools[0], "asked with the loop's own tool schemas, not the strategy's empty result"
assert out is not None and out[0] == "real answer"
@pytest.mark.asyncio
async def test_a_stages_own_timeout_error_is_not_this_guards_deadline():
"""``asyncio.wait_for`` re-raises a child's ``TimeoutError`` as its own.
So an ``except TimeoutError`` around it read a strategy's immediate failure
as a 0.0-second first-call stall and answered the fallback -- which around
``StrategyRegistry.before_llm_call`` means sending the original messages,
tools and model. The TokenStrategy contract has before-call errors propagate
exactly so a bad pre-process cannot send an unprocessed request.
"""
raised: list[str] = []
async def refuses():
raised.append("ran")
raise TimeoutError("the strategy's own bound, not the guard's")
guard = FirstCallGuard(30.0, model="stub")
with pytest.raises(TimeoutError, match="the strategy's own bound"):
await guard.stage("the before_llm_call strategies", refuses(), iteration=1, fallback=("fallback",))
assert raised == ["ran"]
@pytest.mark.asyncio
async def test_a_stages_own_timeout_error_is_not_recorded_as_a_stall():
"""And it leaves no first-call-guard record: nothing stalled."""
from loguru import logger
lines: list[str] = []
sink = logger.add(lambda m: lines.append(m), level="ERROR")
async def refuses():
raise TimeoutError("mine, not yours")
try:
guard = FirstCallGuard(30.0, model="stub")
with pytest.raises(TimeoutError):
await guard.stage("the before_iteration hooks", refuses(), iteration=1, fallback=None)
finally:
logger.remove(sink)
assert not [ln for ln in lines if "First call guard" in ln]
@pytest.mark.asyncio
async def test_the_record_names_the_stage_and_the_wait(workspace):
"""Today's failure left nothing; a stall must say which bound was hit and
how long it waited."""
from loguru import logger
lines: list[str] = []
sink = logger.add(lambda m: lines.append(m), level="ERROR")
try:
agent = _agent(workspace, _Answers(), hooks=[_StallingHook()])
await asyncio.wait_for(_turn(agent), 20)
finally:
logger.remove(sink)
stalls = [ln for ln in lines if "First call guard" in ln]
assert stalls, f"no first-call-guard record in {lines}"
record = stalls[0]
assert "the before_iteration hooks" in record
assert "llmFirstByteTimeout" in record
assert "quiet for 0.1s" in record or "quiet for 0.0s" in record
@pytest.mark.asyncio
async def test_a_stage_that_answers_in_time_is_untouched():
guard = FirstCallGuard(5.0, model="stub")
assert await guard.stage("x", asyncio.sleep(0, result="value"), iteration=1, fallback="fallback") == "value"
@pytest.mark.asyncio
async def test_a_zero_budget_switches_the_guard_off():
"""0 restores what the loop did before: no bound at all on these awaits."""
async def slow():
await asyncio.sleep(0.15)
return "late but delivered"
guard = FirstCallGuard(0, model="stub")
assert guard.budget == 0.0
assert await guard.stage("x", slow(), iteration=1, fallback="fallback") == "late but delivered"
@pytest.mark.asyncio
async def test_the_stalled_awaitable_is_cancelled_not_left_running():
"""A guard that abandoned a stage must not leave its coroutine alive to
finish later and mutate the turn behind the loop's back."""
finished: list[str] = []
async def slow():
await asyncio.sleep(0.3)
finished.append("ran on") # pragma: no cover
guard = FirstCallGuard(0.05, model="stub")
assert await guard.stage("x", slow(), iteration=1, fallback=None) is None
await asyncio.sleep(0.4)
assert finished == []