## Summary The web page gains two things a conversation could only get from the terminal before: exporting the selected turns as a download, and reporting a problem through the bug report pipeline. The trajectory view is now on for everyone. - **Trajectory view without `--dev` (R0).** `raven`, `raven web`, `raven serve` and `raven gateway` no longer take `--dev`; the trajectory policy starts enabled. `raven tui --dev` (running the TUI from source) is unchanged. - **Turn selection.** From this version each turn's opening user message records its trace id (the Turn Trace Mark, never sent to the model). `raven/trajectory/selection.py` lists a session's main-line turns, closes a selection over the sub-agent and delegated traces it dispatched, collects their spans, and cuts the session record down to the selected turns; older records are matched conservatively and an unmatched turn stays in the trajectory only. - **Export.** The Trajectory Export package v1 (`schema: raven_trajectory_export`, `format_version: 1`) is the input contract of the coming `web-traj-import`; it is defined in `CONTEXT.md` and pinned by tests. It holds the raw spans, artifacts and v2 message blobs of the selected traces plus the cut session record, unredacted. Exports run as background jobs (at most two) and the browser downloads them from the same-origin `/trajectory/export/download` route with a one-time token. New RPCs: `trajectory.turns`, `trajectory.export`, `trajectory.export_status`, `trajectory.export_cancel`. - **Bug reports.** `freeze_export` is split into staging and finishing so a report can be prepared from selected turns; the CLI's behaviour is unchanged. A Report Session holds a web report in the gateway between page requests: at most two at once, all blocking I/O on worker threads, a cancel checked after every step (a report cancelled while packaging is never sent), a failed or uploaded report releases its slot and keeps its last status for ten minutes, and an idle one is cancelled after thirty minutes. Seven `bugreport.*` RPCs, the `bugreport-v1` capability (announced only while the receiver URL is valid), and error codes -32026 / -32027 / -32028. Every decision made on the web page, defaults and bulk buttons included, is remembered in the redaction history with `web` as its source, unlike the CLI's bulk flags. - **Web page.** A three-dot conversation menu in the header's corner (the desk launcher moves one slot in) opens the new `export` wizard. Export: pick turns, name the file under an "unredacted" warning, follow the job, download once automatically with a link to download again. Report a problem: pick turns, describe the problem, decide the uncertain values in two columns (hits highlighted at code points), upload, and get a receipt. Closing before the upload starts discards the report being prepared; once the upload starts the report is saved on the machine, a failure marked retryable offers a retry, and the saved report can be sent later from the command line by its id. Every answer is checked against the wizard's epoch and the job or report it was asked about, and a write step whose answer is lost is settled from the report's status instead of being sent again. Layouts were measured at 360 and 390 px in English and Chinese. - **Docs and tests.** The webui and trajectory-debugging pages (English and Chinese) describe both flows. Two integration tests drive them end to end: the export through the real download route, and the bug report against a real receiver process, including a stopped receiver, a retry, the redaction history's replay and a cancel at each step. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - The branch merges the current `main` (one merge commit, no conflicts); everything below ran on the merged tree. - `uv run pytest tests/ -q` (full suite): 57 failed / 31491 passed / 177 skipped. Every one of the 57 also fails on the current `main` on this machine (environment-specific, pre-existing); none is introduced by this branch. - `uv run pytest -m integration tests/integration/test_web_traj_export_e2e.py tests/integration/test_web_bugreport_e2e.py tests/integration/test_bug_report_upload_e2e.py -q`: 10 passed (three consecutive runs of the two new files). - `uv run ruff check`, `uv run ruff format --check`, `uv run ty check` on the changed Python: pass. - ui-web: `npm run gen:check` (239 methods), `npm run lint`, `npm run type-check`, `npx vitest run` (246 files / 3868 passed), `npm run build` and `python3 ui-web/build.py` (both boot snapshots match), `check-page.mjs`, `check-css.mjs`, `check-class-namespace.mjs`: pass. - ui-tui: `npm run lint:rpc`, `npm run lint:i18n`: pass. - `uv run --group docs mkdocs build -f docs-site/mkdocs.yml --strict`: pass. - `make check-source-language check-large-files check-commits`: pass. - Headless Chromium on the built page (offline fixtures): both flows walked to the download and to the receipt; the header corner measured in five layouts (desk panel open and closed, workspace panel open, shell mode, narrow split) and the wizard at 360 and 390 px in both languages. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - A process started with the old `--dev` flag that runs `system.upgrade` relaunches itself with `--dev`, which the new version refuses; the user runs `raven web` once by hand. This was accepted as the cost of dropping the flag. - The trajectory view is on by default: each open page calls the light, in-memory `trajectory.state` every five seconds, and the index is built only when the view is opened. No switch to turn it off is added here; the policy keeps a replaceable source for a settings entry later. - The export is the raw trajectory and may contain secrets. The page says so in a warning; the download is served only on the same origin, behind the session guard, by an unguessable one-time token, and no path ever reaches the page. - Rollback: revert the squash commit. Nothing is migrated; the `trace_id` field already written into session records is ignored by older versions. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
788 lines
32 KiB
Python
788 lines
32 KiB
Python
"""The episodic annotate step that runs after a user turn.
|
|
|
|
Pins the arithmetic the step does around ``MemoryConsolidator`` -- when it
|
|
seeds, when it counts enough, where it slices, what it does with the
|
|
consolidator's answer -- against a fake consolidator that records every call.
|
|
The consolidator itself (annotate -> ``episodes.md``, hot tags -> ``user.md``)
|
|
is exercised on a live gateway, not here.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
from loguru import logger
|
|
|
|
from raven.agent.loop import AgentLoop
|
|
from raven.agent.loop.bundles import EngineWiring, ToolWiring, TurnPolicy
|
|
from raven.config.raven import MemoryConfig
|
|
from raven.contracts.llm_provider import LLMResponse
|
|
from raven.spine.message import ChatType, Source
|
|
from raven.spine.turn import Origin, TurnRequest
|
|
|
|
|
|
class _StubProvider:
|
|
api_key = "test"
|
|
|
|
def get_default_model(self) -> str:
|
|
return "stub"
|
|
|
|
async def chat(self, *args: Any, **kwargs: Any) -> Any:
|
|
raise NotImplementedError
|
|
|
|
async def chat_with_retry(self, *args: Any, **kwargs: Any) -> Any:
|
|
raise NotImplementedError
|
|
|
|
|
|
class _FakeChatProvider:
|
|
def __init__(self, responses: list[LLMResponse]) -> None:
|
|
self._responses = list(responses)
|
|
self._i = 0
|
|
|
|
async def chat_with_retry(self, **kwargs: Any) -> LLMResponse:
|
|
r = self._responses[min(self._i, len(self._responses) - 1)]
|
|
self._i += 1
|
|
return r
|
|
|
|
def get_default_model(self) -> str:
|
|
return "fake/model"
|
|
|
|
|
|
class _FakeConsolidator:
|
|
"""Records what the step hands it; answers whatever the test scripted."""
|
|
|
|
def __init__(self, *, result: bool = True, hold: asyncio.Event | None = None, window: int = 1_000_000) -> None:
|
|
self.result = result
|
|
self.hold = hold
|
|
self.tails: list[list[dict[str, Any]]] = []
|
|
self.refreshes = 0
|
|
self.context_window_tokens = window
|
|
self._locks: dict[str, asyncio.Lock] = {}
|
|
|
|
def get_lock(self, session_key: str) -> asyncio.Lock:
|
|
return self._locks.setdefault(session_key, asyncio.Lock())
|
|
|
|
async def consolidate_messages(self, messages: list[dict[str, Any]]) -> bool:
|
|
self.tails.append(list(messages))
|
|
if self.hold is not None:
|
|
await self.hold.wait()
|
|
return self.result
|
|
|
|
async def maybe_refresh_hot_tags(self) -> int:
|
|
self.refreshes += 1
|
|
return 0
|
|
|
|
|
|
class _FakeBackend:
|
|
async def start(self) -> None: ...
|
|
|
|
async def stop(self) -> None: ...
|
|
|
|
async def feedback(self, signals: Any) -> None: ...
|
|
|
|
async def recall(self, query: Any, *, user_id: Any = None, agent_id: Any = None, top_k: int = 5) -> list:
|
|
return []
|
|
|
|
async def store(self, session_id: Any, messages: Any, *, metadata: Any = None) -> None: ...
|
|
|
|
|
|
def _make_loop(workspace: Path, *, provider: Any | None = None, every: int = 3, backend: Any = None) -> AgentLoop:
|
|
loop = AgentLoop(
|
|
provider=provider or _StubProvider(),
|
|
workspace=workspace,
|
|
model="stub",
|
|
policy=TurnPolicy(max_iterations=2),
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
|
engine=EngineWiring(backend=backend, memory_config=MemoryConfig(annotate_every_turns=every)),
|
|
)
|
|
return loop
|
|
|
|
|
|
def _turn(i: int) -> list[dict[str, Any]]:
|
|
return [{"role": "user", "content": f"q{i}"}, {"role": "assistant", "content": f"a{i}"}]
|
|
|
|
|
|
def _session_with(loop: AgentLoop, key: str, turns: int):
|
|
session = loop.sessions.get_or_create(key)
|
|
for i in range(turns):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
return session
|
|
|
|
|
|
async def _settle(loop: AgentLoop) -> None:
|
|
if loop._consolidation_tasks:
|
|
await asyncio.gather(*list(loop._consolidation_tasks), return_exceptions=True)
|
|
|
|
|
|
@pytest.fixture
|
|
def log_lines() -> Any:
|
|
lines: list[str] = []
|
|
handle = logger.add(lambda m: lines.append(m), level="DEBUG", format="{level} {message}")
|
|
yield lines
|
|
logger.remove(handle)
|
|
|
|
|
|
def _cursor_on_disk(loop: AgentLoop, key: str) -> Any:
|
|
"""The cursor as the next process would read it: drop the cached copy, reload."""
|
|
loop.sessions._cache.pop(key, None)
|
|
return loop.sessions.get_or_create(key).metadata.get("annotated_through")
|
|
|
|
|
|
class TestSeedAndCount:
|
|
async def test_first_sight_seeds_the_cursor_and_writes_nothing(self, tmp_path: Path) -> None:
|
|
loop = _make_loop(tmp_path)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:seed", turns=6)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=10)
|
|
await _settle(loop)
|
|
|
|
assert fake.tails == []
|
|
assert session.metadata["annotated_through"] == 10
|
|
assert _cursor_on_disk(loop, "cli:seed") == 10
|
|
|
|
async def test_the_fifth_turn_hands_the_tail_to_the_annotator(self, tmp_path: Path) -> None:
|
|
loop = _make_loop(tmp_path, every=5)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:five", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(5):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 1
|
|
tail = fake.tails[0]
|
|
assert tail[0] == {"role": "user", "content": "q0"}
|
|
assert tail[-1] == {"role": "assistant", "content": "a4"}
|
|
assert session.metadata["annotated_through"] == 10
|
|
assert _cursor_on_disk(loop, "cli:five") == 10
|
|
assert fake.refreshes == 1
|
|
|
|
async def test_three_turns_since_the_cursor_call_nothing(self, tmp_path: Path) -> None:
|
|
loop = _make_loop(tmp_path, every=5)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:three", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(3):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
assert fake.tails == []
|
|
assert session.metadata["annotated_through"] == 0
|
|
|
|
async def test_zero_turns_the_step_off(self, tmp_path: Path) -> None:
|
|
loop = _make_loop(tmp_path, every=0)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:off", turns=20)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
assert fake.tails == []
|
|
assert "annotated_through" not in session.metadata
|
|
assert _cursor_on_disk(loop, "cli:off") is None
|
|
|
|
|
|
class TestCursorEdges:
|
|
async def test_undo_past_the_cursor_is_clamped(self, tmp_path: Path) -> None:
|
|
loop = _make_loop(tmp_path, every=3)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:undo", turns=4)
|
|
session.metadata["annotated_through"] = 8
|
|
loop.sessions.save(session)
|
|
before = session.last_consolidated
|
|
|
|
assert session.undo_last_turn(2) == 4
|
|
for i in (4, 5, 6):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 1
|
|
assert fake.tails[0][0] == {"role": "user", "content": "q4"}
|
|
assert session.metadata["annotated_through"] == 10
|
|
assert session.last_consolidated == before == 0
|
|
|
|
async def test_a_declined_annotation_keeps_the_cursor(self, tmp_path: Path, log_lines) -> None:
|
|
loop = _make_loop(tmp_path, every=3)
|
|
fake = _FakeConsolidator(result=False)
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:declined", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(3):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 1
|
|
assert session.metadata["annotated_through"] == 0
|
|
assert fake.refreshes == 0
|
|
assert any("WARNING" in line and "declined" in line for line in log_lines)
|
|
|
|
async def test_a_failing_patch_is_logged_not_raised(self, tmp_path: Path, log_lines, monkeypatch) -> None:
|
|
loop = _make_loop(tmp_path, every=3)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:boom", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(3):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
def _boom(*_a: Any, **_k: Any) -> bool:
|
|
raise OSError("disk says no")
|
|
|
|
monkeypatch.setattr(loop.sessions, "append_metadata_patch", _boom)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
assert any("ERROR" in line and "failed" in line for line in log_lines)
|
|
# the same loop still takes the next call
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
|
|
class TestConcurrency:
|
|
async def test_the_reply_does_not_wait_for_the_annotator(self, tmp_path: Path) -> None:
|
|
loop = _make_loop(tmp_path, every=3)
|
|
gate = asyncio.Event()
|
|
fake = _FakeConsolidator(hold=gate)
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:async", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(3):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await asyncio.sleep(0)
|
|
assert len(fake.tails) == 1
|
|
assert session.metadata["annotated_through"] == 0
|
|
|
|
gate.set()
|
|
await _settle(loop)
|
|
assert session.metadata["annotated_through"] == 6
|
|
|
|
async def test_a_waiting_task_recomputes_and_does_nothing(self, tmp_path: Path) -> None:
|
|
loop = _make_loop(tmp_path, every=3)
|
|
gate = asyncio.Event()
|
|
fake = _FakeConsolidator(hold=gate)
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:lock", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(3):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await asyncio.sleep(0)
|
|
session.messages.extend(_turn(3))
|
|
loop.sessions.save(session)
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await asyncio.sleep(0)
|
|
|
|
gate.set()
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 1
|
|
assert session.metadata["annotated_through"] == 6
|
|
|
|
|
|
class TestOrigins:
|
|
@pytest.mark.parametrize("origin", [Origin.SENTINEL, Origin.CRON, Origin.HEARTBEAT, Origin.SUBAGENT])
|
|
async def test_machine_turns_are_ignored(self, tmp_path: Path, origin: Origin) -> None:
|
|
loop = _make_loop(
|
|
tmp_path, provider=_FakeChatProvider([LLMResponse(content="ok", finish_reason="stop")]), every=1
|
|
)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
_stub_edges(loop)
|
|
|
|
await loop._process_message(_req("hello", origin=origin), origin=origin)
|
|
await _settle(loop)
|
|
|
|
assert fake.tails == []
|
|
session = loop.sessions.get_or_create("cli:c")
|
|
assert "annotated_through" not in session.metadata
|
|
|
|
async def test_a_user_turn_reaches_the_step(self, tmp_path: Path) -> None:
|
|
loop = _make_loop(
|
|
tmp_path, provider=_FakeChatProvider([LLMResponse(content="ok", finish_reason="stop")]), every=1
|
|
)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
_stub_edges(loop)
|
|
|
|
await loop._process_message(_req("hello", origin=Origin.USER), origin=Origin.USER)
|
|
await _settle(loop)
|
|
session = loop.sessions.get_or_create("cli:c")
|
|
# a new session seeds at 0 and counts its first turn in the same pass
|
|
assert len(fake.tails) == 1
|
|
assert fake.tails[0][0]["role"] == "user"
|
|
assert session.metadata.get("annotated_through") == len(session.messages)
|
|
|
|
await loop._process_message(_req("again", origin=Origin.USER), origin=Origin.USER)
|
|
await _settle(loop)
|
|
assert len(fake.tails) == 2
|
|
assert fake.tails[1][0] == {
|
|
k: v for k, v in fake.tails[1][0].items() if k in ("role", "content", "timestamp", "trace_id")
|
|
}
|
|
assert fake.tails[1][0]["content"] == "again"
|
|
assert fake.tails[1][0]["trace_id"].startswith("trace-")
|
|
|
|
|
|
def _req(text: str, *, origin: Origin = Origin.USER) -> TurnRequest:
|
|
return TurnRequest(
|
|
origin=origin,
|
|
source=Source(channel="cli", chat_id="c", sender_id="u", chat_type=ChatType.DM),
|
|
text=text,
|
|
media=(),
|
|
)
|
|
|
|
|
|
def _stub_edges(loop: AgentLoop) -> None:
|
|
async def _noop(**_kw: Any) -> None:
|
|
return None
|
|
|
|
loop._start_executor = _noop
|
|
loop._connect_mcp = _noop
|
|
|
|
|
|
class TestABackendChangesNothingHere:
|
|
async def test_a_wired_backend_still_gets_the_tail_annotated(self, tmp_path: Path) -> None:
|
|
"""The host's log is written whether or not a plugin owns memory.
|
|
|
|
Every acceptance run for this step drove a gateway with no backend,
|
|
while the shipped default wires one. The step reads no backend, so the
|
|
claim is structural -- pinned here so a later gate on one reddens
|
|
rather than silently stopping the log on the configuration almost
|
|
everyone runs.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=3, backend=_FakeBackend())
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:wired", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(3):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
assert loop.backend is not None
|
|
assert len(fake.tails) == 1
|
|
assert session.metadata["annotated_through"] == 6
|
|
|
|
|
|
class TestWhatHappensDuringTheAwait:
|
|
async def test_a_truncation_during_the_await_does_not_claim_the_turns_that_replaced_it(
|
|
self, tmp_path: Path
|
|
) -> None:
|
|
"""The reader can undo mid-annotation and then keep talking.
|
|
|
|
The task sliced against indices that no longer mean anything:
|
|
``undo_last_turn`` moved the cursor itself, and the messages the
|
|
write-back would claim are the new turns that arrived after it. A
|
|
clamp to the live length is not enough -- the session is longer than
|
|
the stale arithmetic, so the patch lands inside the replacements and
|
|
they are never annotated.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=3)
|
|
gate = asyncio.Event()
|
|
fake = _FakeConsolidator(hold=gate)
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:race", turns=0)
|
|
for i in range(6):
|
|
session.messages.extend(_turn(i))
|
|
session.metadata["annotated_through"] = 6
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=10)
|
|
await asyncio.sleep(0)
|
|
assert len(fake.tails) == 1 # sliced messages 6..12, now holding
|
|
|
|
assert session.undo_last_turn(4) == 8
|
|
assert session.metadata["annotated_through"] == 4 # pulled back to the cut
|
|
for i in range(10, 13):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
gate.set()
|
|
await _settle(loop)
|
|
|
|
assert session.metadata["annotated_through"] == 4, "the stale write-back claimed the new turns"
|
|
|
|
fake.tails.clear()
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
assert [m["content"] for m in fake.tails[0]] == ["q10", "a10", "q11", "a11", "q12", "a12"]
|
|
|
|
async def test_a_truncation_the_cursor_cannot_see_is_still_detected(self, tmp_path: Path) -> None:
|
|
"""The cut can sit above the cursor, which then does not move at all.
|
|
|
|
``undo_last_turn`` only pulls a cursor back when the cut is below it,
|
|
so a session annotated from zero is truncated with its cursor left at
|
|
zero. Turns arriving afterwards take the length back past the stale
|
|
write-back, which then lands inside them. Comparing the cursor's value
|
|
sees nothing; the truncation counter does.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=3)
|
|
gate = asyncio.Event()
|
|
fake = _FakeConsolidator(hold=gate)
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:silent", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(3):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=4)
|
|
await asyncio.sleep(0)
|
|
assert len(fake.tails) == 1 # sliced 0..6, now holding
|
|
|
|
assert session.undo_last_turn(2) == 4
|
|
assert session.metadata["annotated_through"] == 0, "the cut is above the cursor, so it does not move"
|
|
for i in range(10, 13):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
gate.set()
|
|
await _settle(loop)
|
|
|
|
assert session.metadata["annotated_through"] == 0, "the stale write-back claimed the new turns"
|
|
|
|
fake.tails.clear()
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
assert [m["content"] for m in fake.tails[0]] == ["q0", "a0", "q10", "a10", "q11", "a11", "q12", "a12"]
|
|
|
|
async def test_the_tail_ends_on_the_turn_that_asked_for_it(self, tmp_path: Path) -> None:
|
|
"""A task that waited must not slice into the turn that overtook it.
|
|
|
|
The task is held on the lock, not inside the annotator, so the slice
|
|
happens after the next turn's question has landed. Sliced to the live
|
|
end it would hand over a user message whose answer has not arrived.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=3)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:boundary", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(3):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
held = fake.get_lock(session.key)
|
|
await held.acquire()
|
|
loop._maybe_annotate_tail(session, prev_len=4)
|
|
await asyncio.sleep(0)
|
|
assert fake.tails == [] # still waiting on the lock
|
|
session.messages.append({"role": "user", "content": "still unanswered"})
|
|
held.release()
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 1
|
|
assert fake.tails[0][-1] == {"role": "assistant", "content": "a2"}
|
|
|
|
|
|
class TestMachineTurnsInsideTheTail:
|
|
async def test_a_sentinel_turn_between_two_user_turns_is_not_annotated(self, tmp_path: Path) -> None:
|
|
"""Sentinel reads this file; none of its turn may come back to it.
|
|
|
|
The mark rides on the one entry that opens a turn and on nothing else
|
|
in it -- the reply, the tool calls and their results are all saved
|
|
unmarked (`initial_messages[-1]` at the stamp site). A filter that read
|
|
each message on its own would drop Sentinel's question and keep its
|
|
answer, which is the half that reaches `episodes.md`.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=2)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:mixed", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
session.messages.extend(_turn(0))
|
|
# The shape _save_turn writes: the mark on the opening entry only.
|
|
session.messages.extend(
|
|
[
|
|
{"role": "user", "content": "SENTINEL NOTICE", "origin": "sentinel"},
|
|
{"role": "assistant", "content": "reminder handled, nudged the user"},
|
|
{"role": "assistant", "content": "", "tool_calls": [{"id": "t1"}]},
|
|
{"role": "tool", "tool_call_id": "t1", "content": "nudge sent"},
|
|
]
|
|
)
|
|
session.messages.extend(_turn(1))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 1
|
|
contents = [m.get("content") for m in fake.tails[0]]
|
|
assert "SENTINEL NOTICE" not in contents
|
|
assert "reminder handled, nudged the user" not in contents, "the reply is the half that reaches the file"
|
|
assert "nudge sent" not in contents
|
|
assert contents == ["q0", "a0", "q1", "a1"]
|
|
|
|
async def test_a_persons_turn_keeps_its_unmarked_parts(self, tmp_path: Path) -> None:
|
|
"""The same rule must not eat a person's tool calls, which are unmarked too."""
|
|
loop = _make_loop(tmp_path, every=2)
|
|
fake = _FakeConsolidator()
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:tools", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
session.messages.extend(
|
|
[
|
|
{"role": "user", "content": "q0"},
|
|
{"role": "assistant", "content": "", "tool_calls": [{"id": "t1"}]},
|
|
{"role": "tool", "tool_call_id": "t1", "content": "ran it"},
|
|
{"role": "assistant", "content": "a0"},
|
|
]
|
|
)
|
|
session.messages.extend(_turn(1))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
|
|
assert [m.get("content") for m in fake.tails[0]] == ["q0", "", "ran it", "a0", "q1", "a1"]
|
|
|
|
|
|
class TestTheNewCommandDoesNotRedoTheWork:
|
|
async def test_new_starts_from_whichever_cursor_ran_last(self, tmp_path: Path) -> None:
|
|
"""``/new`` and the after-turn step write the same file.
|
|
|
|
``last_consolidated`` only moves in the token path, which the shipped
|
|
context engine owns and so never runs. Sliced from it alone, ``/new``
|
|
hands the annotator every message the step already wrote: the same
|
|
episodes a second time, and every tag counted twice toward the
|
|
threshold that rewrites the profile.
|
|
"""
|
|
from raven.memory_engine.consolidate.consolidator import MemoryConsolidator
|
|
|
|
loop = _make_loop(tmp_path, every=3)
|
|
session = _session_with(loop, "cli:new", turns=0)
|
|
for i in range(3):
|
|
session.messages.extend(_turn(i))
|
|
session.metadata["annotated_through"] = 6
|
|
loop.sessions.save(session)
|
|
assert session.last_consolidated == 0
|
|
|
|
seen: list[list[dict[str, Any]]] = []
|
|
|
|
class _Recording(MemoryConsolidator):
|
|
def __init__(self) -> None:
|
|
self._locks: dict[str, asyncio.Lock] = {}
|
|
|
|
def get_lock(self, key: str) -> asyncio.Lock:
|
|
return self._locks.setdefault(key, asyncio.Lock())
|
|
|
|
async def consolidate_messages(self, messages: list[dict[str, Any]]) -> bool:
|
|
seen.append(list(messages))
|
|
return True
|
|
|
|
async def maybe_refresh_hot_tags(self) -> int:
|
|
return 0
|
|
|
|
assert await _Recording().consolidate_unconsolidated(session) is True
|
|
assert seen == [], "nothing is left unannotated, so nothing is handed over"
|
|
|
|
session.messages.extend(_turn(3))
|
|
await _Recording().consolidate_unconsolidated(session)
|
|
assert [m["content"] for m in seen[-1]] == ["q3", "a3"]
|
|
|
|
|
|
class TestATruncationWhileTheTaskIsQueued:
|
|
async def test_a_cut_before_the_lock_is_taken_is_still_seen(self, tmp_path: Path) -> None:
|
|
"""The counter is read with the boundary, not after the wait.
|
|
|
|
A second task can sit queued behind the first. A truncation that lands
|
|
while it waits has already moved the counter by the time it takes the
|
|
lock, so a counter read there agrees with itself and the task goes on
|
|
to slice the changed list against the boundary it captured.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=2)
|
|
gate = asyncio.Event()
|
|
fake = _FakeConsolidator(hold=gate)
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:queued", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
for i in range(2):
|
|
session.messages.extend(_turn(i))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=2) # task 1 takes the lock and holds
|
|
await asyncio.sleep(0)
|
|
assert len(fake.tails) == 1
|
|
|
|
session.messages.extend(_turn(2))
|
|
loop._maybe_annotate_tail(session, prev_len=4) # task 2 queues with end=6
|
|
await asyncio.sleep(0)
|
|
|
|
assert session.undo_last_turn(1) == 2 # the first half of a retry
|
|
loop.sessions.save(session)
|
|
|
|
gate.set()
|
|
await _settle(loop)
|
|
|
|
# task 2 saw the cut and stood down; nothing was annotated twice
|
|
assert len(fake.tails) == 1
|
|
assert session.metadata["annotated_through"] <= len(session.messages)
|
|
|
|
|
|
class TestOneCallIsBounded:
|
|
async def test_an_oversized_tail_is_cut_at_a_turn_and_the_rest_rides_the_next_pass(self, tmp_path: Path) -> None:
|
|
"""A declined tail must not be retried larger every turn.
|
|
|
|
Without a bound, a tail refused for its size comes back one turn
|
|
longer each time and the log stops being written for that session.
|
|
The cut lands on a turn boundary and the cursor moves to it, so the
|
|
next pass starts after what was sent.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=2)
|
|
fake = _FakeConsolidator(window=400) # budget is half of this
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:big", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
big = "x" * 4000
|
|
for i in range(4):
|
|
session.messages.extend([{"role": "user", "content": f"q{i}"}, {"role": "assistant", "content": big}])
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=6)
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 1
|
|
sent = fake.tails[0]
|
|
assert len(sent) < 8, "the whole tail went out unbounded"
|
|
assert sent[-1]["role"] == "assistant", "cut mid-turn"
|
|
assert session.metadata["annotated_through"] == len(sent)
|
|
|
|
fake.tails.clear()
|
|
loop._maybe_annotate_tail(session, prev_len=len(session.messages) - 2)
|
|
await _settle(loop)
|
|
assert fake.tails[0][0] == {"role": "user", "content": f"q{len(sent) // 2}"}
|
|
|
|
async def test_a_refused_oversized_turn_moves_the_cursor_past_it(self, tmp_path: Path, log_lines) -> None:
|
|
"""The one refusal the cursor absorbs, because retrying it is repeating it.
|
|
|
|
A turn larger than the whole budget is sent whole, and nothing records
|
|
that it was. Keeping the cursor on a refusal therefore does not retry
|
|
the call later under better conditions: every qualifying turn after it
|
|
rebuilds the same tail and sends the same oversized prompt again.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=1)
|
|
fake = _FakeConsolidator(result=False, window=100)
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:refused-huge", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
session.messages.extend([{"role": "user", "content": "q0"}, {"role": "assistant", "content": "y" * 40000}])
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=0)
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 1
|
|
assert session.metadata["annotated_through"] == 2
|
|
assert fake.refreshes == 0, "nothing was written, so no section may refresh"
|
|
assert any("WARNING" in line and "larger than the bound" in line for line in log_lines)
|
|
|
|
# and the next pass starts after it rather than sending it again
|
|
session.messages.extend(_turn(1))
|
|
loop.sessions.save(session)
|
|
loop._maybe_annotate_tail(session, prev_len=2)
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 2
|
|
assert fake.tails[1] == _turn(1)
|
|
|
|
async def test_an_ordinary_decline_still_keeps_the_cursor(self, tmp_path: Path) -> None:
|
|
"""The twin of the case above, so the exemption cannot widen quietly.
|
|
|
|
A tail that fits the budget and is refused is retried, because the
|
|
refusal may have been the provider rather than the size.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=1)
|
|
fake = _FakeConsolidator(result=False, window=1_000_000)
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:refused-small", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
session.messages.extend(_turn(0))
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=0)
|
|
await _settle(loop)
|
|
|
|
assert session.metadata["annotated_through"] == 0
|
|
|
|
async def test_the_cut_is_located_by_position_not_by_equality(self, tmp_path: Path) -> None:
|
|
"""Two messages in one slice can be equal, and the cursor must not care.
|
|
|
|
A search by value answers with the earlier twin, which leaves the
|
|
cursor short of what was actually sent and re-annotates the overlap on
|
|
the next pass. Repeated answers are ordinary -- a tool that prints the
|
|
same thing twice, an agent that replies "done" -- and messages written
|
|
under a frozen clock carry equal timestamps as well.
|
|
"""
|
|
loop = _make_loop(tmp_path, every=2)
|
|
fake = _FakeConsolidator(window=400) # budget 200
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:twins", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
same = {"role": "assistant", "content": "x" * 200}
|
|
session.messages.extend(
|
|
[
|
|
{"role": "user", "content": "q0"},
|
|
dict(same),
|
|
{"role": "user", "content": "q1"},
|
|
dict(same), # equal to index 1, and the last message sent
|
|
{"role": "user", "content": "q2"},
|
|
{"role": "assistant", "content": "y" * 4000},
|
|
]
|
|
)
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=4)
|
|
await _settle(loop)
|
|
|
|
sent = fake.tails[0]
|
|
assert sent == session.messages[:4], "the cut did not land where this case needs it"
|
|
assert session.metadata["annotated_through"] == 4, "the cursor answered with the earlier twin"
|
|
|
|
async def test_one_turn_larger_than_the_budget_still_goes_out(self, tmp_path: Path) -> None:
|
|
"""Otherwise the cursor stalls on it forever and nothing after it is ever seen."""
|
|
loop = _make_loop(tmp_path, every=1)
|
|
fake = _FakeConsolidator(window=100)
|
|
loop.memory_consolidator = fake
|
|
session = _session_with(loop, "cli:huge", turns=0)
|
|
session.metadata["annotated_through"] = 0
|
|
session.messages.extend([{"role": "user", "content": "q0"}, {"role": "assistant", "content": "y" * 40000}])
|
|
loop.sessions.save(session)
|
|
|
|
loop._maybe_annotate_tail(session, prev_len=0)
|
|
await _settle(loop)
|
|
|
|
assert len(fake.tails) == 1
|
|
assert len(fake.tails[0]) == 2
|
|
assert session.metadata["annotated_through"] == 2
|