Raven-1/tests/test_runtime_checkpoint_deep.py
KT a7f0502be0
ci(*): make the idle ceiling a gate and take the last waits out (#428)
## Summary

Follow-up to #413, from a measured look at what was left: a CI shard now
spends 72 s on fixed cost (22 s environment, 48 s collection and worker
start) against 82 s of tests, the four shards are within 15 s of each
other, and 70 unmarked tests were still over the 2 s wall-clock ceiling
on CI against 42 on a laptop because CPU-bound work runs three to four
times slower there. This PR makes the ceiling a gate that is portable
across machines, takes the remaining waits out, precompiles bytecode so
the workers stop doing it four times, and fixes two product waits the
same investigation surfaced.

- `test(*)`: the ceiling judges idle time instead of wall clock, where
idle is the wall clock minus every CPU second spent on the test's
behalf, the reaped children's included (`os.times`). A test waiting on a
production sleep or a process that answers nothing waits the same
seconds on every machine; a test that computes for 2 s here and 7 s on
CI, or drives a browser or eight spawned workers, is not the problem.
The numbers ride on the teardown report so the xdist controller can
judge them; options are `--idle-ceiling` / `--idle-ceiling-strict`. The
fixture that shortens production ladders takes in the ACP cancel settle
(5 s) and the gateway delivery grace (2 s); the CLI agent tests stub the
one-shot command's unconditional 2 s hand-off grace; the rpc subagent
tests no longer spawn a login shell or probe a real endpoint over the
network; three literal timeouts shrink where the stub never answers
anyway. What is left over the ceiling carries a marker: seven real
LibreOffice renders and two tests that wait on a process answering
nothing carry `slow`, two whose assertion is the wait itself carry
`production_timing`.
- `ci(*)`: every shard runs with `--idle-ceiling-strict` (on the make
line, not in `addopts`: the Windows job runs `--noconftest`). Bytecode
precompilation was tried and dropped, the numbers are below; the
workflow is otherwise unchanged from #413.
- `test(agent)`: a heartbeat test asked for five beats of the fifteen
its intervals should produce in a 0.6 s window, and a loaded runner
delivered three. It reads the two gaps instead, which the load cannot
invert: the wait before the first beat is one interval, the wait after
it is the other. Older than the gate that surfaced it; nothing here
touches the registry.
- `perf(ppt-engine)`: `MeasuredWidth.width` remembers measured widths
(bounded, oldest key out); a 15-slide template's QA gates made 1848
freetype calls over 162 distinct keys. Importing `text_metrics` first
raised on a circular import through `template.capacity`; capacity now
imports the module and resolves the name at call time.
- `perf(*)`: `raven gateway` and `raven web` start the litellm import on
a daemon thread at boot, so saving a key from the settings page no
longer waits three to five seconds on it.
- `fix(providers)`: one thread at a time initialises litellm. It raises
three logger levels for the duration of the import and puts the captured
values back after, which holds only while one thread is doing it; a
second caller arriving inside that window captures WARNING as the level
to restore, and litellm's DEBUG and INFO stop reaching the file sink
until a restart. The window was narrow while every import sat on a
startup path and is ordinary now that a server warms it in the
background, so the two callers share a module lock. The strict-mode
warm-up also runs before the fixtures that redirect the home, and
publishing the OAuth token directories created them, so it warms under a
temporary home.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [x] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Whole suite, `uv run --frozen --all-extras pytest -q` (macOS, 4
workers), before and after the waits came out and with the ceiling
strict:

| run | result | idle list |
|---|---|---|
| main after #413 | 22052 passed, 43 skipped in 219.41 s | 39 over 2 s
wall clock |
| this branch, `--idle-ceiling` warning | 22057 passed, 43 skipped in
210.20 s | 14 over 2 s idle, all real render or subprocess, now marked |
| this branch, `--idle-ceiling-strict` | 22058 passed, 43 skipped in
204.89 s, exit 0 | none |

The ceiling is three seconds, placed from the suite's own distribution
rather than guessed. Unmarked tests over one second of idle, measured
across the whole suite:

| idle | tests | what they are |
|---|---|---|
| 1.00-1.19 s | 19 | a test that sets a one second timeout and lets it
expire |
| 1.2-1.3 s | 3 | a subprocess handshake |
| 1.5-1.6 s | 3 | an ACP turn interrupted after 0.8 s, and two like it |
| above 1.6 s | 0 | the renders above this line all carry a marker |

A runner stretches that 1.6 s to about 2.2 s, which is how a two second
line failed two shards over tests waiting exactly as long as they meant
to. Three seconds sits in the gap the population leaves, and stays under
every ladder this guards against: the shortest constant the audit found
was a three second grace, the rest run 5, 15, 30 and 60 seconds.

What the metric separates, measured on this suite: a test that sleeps 5
s reports `4.96s idle of 5.02s`; a test that drives a headless browser
for 13 s reports under 0.8 s; six `raven doctor` subprocesses report 1.2
s where wall clock said 4.2 s; eight spawned processes contending for a
lock report under 0.8 s where wall clock said 3.6 s.

Files touched by the product changes and the new markers (`uv run
--frozen --all-extras pytest -q tests/test_litellm_setup.py
tests/test_cli_gateway_commands.py tests/test_cli_serve_commands.py
tests/test_ppt_engine_assets_fonts.py
tests/test_ppt_engine_template_capacity.py
tests/test_cli_doctor_commands.py tests/test_ppt_engine_end_to_end.py
tests/test_ppt_engine_render_service.py tests/test_rpc_model.py`): `434
passed, 2 skipped in 33.39s`, none over 1 s idle. `python -c "import
raven_ppt.services.assets.text_metrics"` in a fresh interpreter:
ImportError before, imports after.

Test-side speedups, measured per test: the ten CLI agent tests 2.0-4.4 s
to 0.05-0.36 s; the three rpc subagent tests 1.4-2.6 s to 0.01-0.02 s;
the rpc save-key test 2.2-2.8 s to 0.03 s; the two ACP probe tests 2.1 s
to 0.2 s.

Bytecode precompilation, seconds from the start of the pytest step to
the first test, per shard, over three runs of this workflow:

| what was compiled | shard 1 | shard 2 | shard 3 | shard 4 | mean |
install step |
|---|---|---|---|---|---|---|
| nothing (#413) | 52.7 | 46.5 | 45.9 | 46.0 | 47.8 | 3-4 s |
| site-packages and checkout | 39.8 | 39.3 | 35.6 | 45.5 | 40.1 | 12-17
s |
| checkout only | 55.5 | 50.1 | 44.4 | 53.8 | 51.0 | 3-4 s |

Compiling site-packages is the half that moved the number and it cost
more in the install step than it returned. Compiling the checkout alone
is inside the noise of doing nothing: the four workers share one `.pyc`
cache, so the first to import a module writes it for the rest, and most
of what a laptop measures as cold compilation is pytest rewriting test
files through a cache `compileall` cannot fill. Both are gone; the
comment in the workflow records it.

The reworked heartbeat assertion was checked against the bug it guards
by making the registry keep its first interval for every beat: one beat
arrives in the window and the assertion fails on it.

The diff gate is reproduced locally against `origin/main` after a full
`make coverage`: `Diff coverage: 100.00% (12/12 executable changed
lines)`. The two warm-up calls in the serving boot coroutines carry a
no-cover pragma naming the test that pins them, because those coroutines
bind a port and serve until they are killed.

The lock is checked against the race it closes: with it taken away, a
test that holds the first caller inside the import and starts a second
fails on `[30, 30, 30] == [20, 20, 20]`. A strict session under a clean
HOME now leaves no `.raven` behind.

`make lint-python` clean.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed (none)

## Risk

- The ceiling is now a gate on CI. A new test that waits more than 2 s
without `slow` or `production_timing` turns its shard red; the summary
names the test and the marker to use. A local run stays warning-only.
- A subprocess that does real work is charged to the test that reaped
it, so only a real wait trips the gate. A strict session imports litellm
up front, because its 1.7 s of file opens would otherwise land on
whichever test in the shard imports it first.
- The litellm warm-up runs on a daemon thread at server boot only; unit
tests never reach either boot path (their source is pinned instead), and
a failed import is logged, not raised.
- The width memo returns the same floats; the cap evicts the oldest key.
The capacity import change resolves the same name one step later.
- Rollback: revert the PR; every change is additive or a marker.

- [x] Security impact considered (no security surface touched)
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-16 14:51:50 +08:00

734 lines
31 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""Deep tests - pathological inputs, edge state, and concurrency.
The base file ``test_runtime_checkpoint.py`` validates the happy path and
core regressions; this file is the fail-safe hardening tier:
- D1: filesystem pathology (unicode names, symlinks, deep nesting, file<->dir
transitions, chmod, binary).
- D2: edge state (missing/read-only workspace, corrupted/blocked shadow dir,
pre-existing files where dirs are expected).
- D3: concurrency (interleaved sessions, repeated stash, async-parallel
commits sharing one shadow git).
"""
from __future__ import annotations
import asyncio
import os
import shutil
import tempfile
from pathlib import Path
import pytest
from raven.agent import workdir
from raven.agent.loop import AgentLoop
from raven.agent.loop.bundles import EngineWiring, ToolWiring, TurnPolicy
from raven.agent.loop.checkpoint import CheckpointService
from raven.agent.loop.recovery import RecoveryLimits
from raven.config.raven import CheckpointConfig, RuntimeConfig
from raven.providers.base import LLMProvider, LLMResponse
@pytest.fixture
def workspace():
with tempfile.TemporaryDirectory() as td:
yield Path(td)
async def _run_turn_body(agent: AgentLoop, workspace: Path):
"""Drive ``_run_agent_loop`` the way ``run_turn`` does: inside the
working-directory binding. The per-turn checkpoint reads that binding, so
an unbound call would snapshot nothing at all.
"""
with workdir.bind(workspace):
return await agent._run_agent_loop([{"role": "user", "content": "go"}])
# =============================================================================
# D1 — Filesystem pathological inputs
# =============================================================================
async def test_d1_unicode_and_space_in_filename(workspace):
"""add -A is name-agnostic — unicode + spaces still snapshot cleanly."""
svc = CheckpointService(workspace)
(workspace / "测试 文件.py").write_text("# hi\n", encoding="utf-8")
cid, changed = await svc.commit_turn("t1")
assert cid is not None
# git uses UTF-8 for paths; the file shows up by its actual name.
assert any("测试" in p for p in changed), f"got: {changed!r}"
async def test_d1_symlink_snapshotted_as_symlink(workspace):
"""Symlinks are first-class in git (mode 120000). add -A stages the link
itself, not the target — so the snapshot doesn't accidentally inline a
large target or pull in out-of-workspace files."""
(workspace / "target.txt").write_text("real\n", encoding="utf-8")
(workspace / "link.txt").symlink_to("target.txt")
svc = CheckpointService(workspace)
cid, changed = await svc.commit_turn("t1")
assert cid is not None
assert "link.txt" in changed and "target.txt" in changed
async def test_d1_deeply_nested_paths(workspace):
p = workspace / "a" / "b" / "c" / "d" / "e.py"
p.parent.mkdir(parents=True)
p.write_text("ok\n", encoding="utf-8")
svc = CheckpointService(workspace)
cid, changed = await svc.commit_turn("t1")
assert cid is not None
assert "a/b/c/d/e.py" in changed
async def test_d1_file_to_directory_transition(workspace):
"""Turn 1: file ``thing``. Turn 2: thing is now a directory holding x.py.
The transition must snapshot — this is a realistic agent edit pattern when
refactoring a single-file module into a package."""
svc = CheckpointService(workspace)
(workspace / "thing").write_text("a = 1\n", encoding="utf-8")
cid1, _ = await svc.commit_turn("t1")
assert cid1 is not None
(workspace / "thing").unlink()
(workspace / "thing").mkdir()
(workspace / "thing" / "x.py").write_text("b = 2\n", encoding="utf-8")
cid2, changed = await svc.commit_turn("t2")
assert cid2 is not None and cid2 != cid1
# Both the deletion of the old file and the addition of the new path appear.
assert "thing" in changed or "thing/x.py" in changed
async def test_d1_chmod_only_change_captured(workspace):
"""Mode-bit change (e.g. chmod +x on a script) without content change is
still a recoverable diff — git tracks executable bit and we want recovery
to know about it."""
p = workspace / "script.sh"
p.write_text("#!/bin/sh\necho hi\n", encoding="utf-8")
svc = CheckpointService(workspace)
await svc.commit_turn("t1")
os.chmod(p, 0o755) # executable bit flipped
cid2, changed = await svc.commit_turn("t2")
# git records the mode change → file shows in diff --cached.
assert cid2 is not None
assert "script.sh" in changed
async def test_d1_binary_file(workspace):
"""Binary blobs (small, ~64 KiB) shouldn't choke the snapshot pipeline."""
(workspace / "blob.bin").write_bytes(os.urandom(64 * 1024))
svc = CheckpointService(workspace)
cid, changed = await svc.commit_turn("t1")
assert cid is not None
assert "blob.bin" in changed
# =============================================================================
# D2 — Edge state / fault injection
# =============================================================================
async def test_d2_workspace_does_not_exist():
"""A bogus workspace path must degrade — no crash, no half-built shadow."""
bogus = Path(tempfile.gettempdir()) / "raven_nonexistent_xyz"
if bogus.exists():
shutil.rmtree(bogus)
svc = CheckpointService(bogus)
cid, changed = await svc.commit_turn("t1")
assert cid is None
assert changed == []
async def test_d2_shadow_path_blocked_by_a_regular_file(workspace):
"""If something pre-existing sits at the shadow git-dir path as a file
(not a directory), init must fail gracefully — checkpoint is best-effort."""
blocker = workspace / ".raven"
blocker.write_text("not a dir", encoding="utf-8") # blocks mkdir
svc = CheckpointService(workspace)
cid, changed = await svc.commit_turn("t1")
assert cid is None
assert changed == []
async def test_d2_corrupted_shadow_repo(workspace):
"""A pre-existing but corrupted shadow .git → commit_turn doesn't raise."""
gd = workspace / ".raven" / "shadow.git"
gd.mkdir(parents=True)
(gd / "HEAD").write_text("garbage\n", encoding="utf-8") # malformed HEAD
# Some other dirs left half-formed
(gd / "objects").mkdir(exist_ok=True)
svc = CheckpointService(workspace)
(workspace / "x.py").write_text("ok\n", encoding="utf-8")
cid, _changed = await svc.commit_turn("t1")
# Either git recovers (rare) and returns a sha, or it fails gracefully.
# The contract is: never raise into the caller.
assert cid is None or isinstance(cid, str)
async def test_d2_very_long_label(workspace):
svc = CheckpointService(workspace)
(workspace / "x.py").write_text("ok\n", encoding="utf-8")
cid, changed = await svc.commit_turn("L" * 4000) # 4 KB label
assert cid is not None
assert "x.py" in changed
# =============================================================================
# D3 — Concurrency
# =============================================================================
class _NoopProvider(LLMProvider):
def __init__(self) -> None:
super().__init__(api_key="test")
async def chat(
self,
messages,
tools=None,
model=None,
max_tokens=4096,
temperature=0.7,
reasoning_effort=None,
tool_choice=None,
):
return LLMResponse(content="", finish_reason="stop")
def get_default_model(self) -> str:
return "stub"
def _agent_with_checkpoint(workspace: Path) -> AgentLoop:
return AgentLoop(
provider=_NoopProvider(),
workspace=workspace,
model="stub",
# These checkpoint tests use a no-op model only as a "turn ends" stub;
# empty-recovery is an orthogonal feature they don't exercise, so
# disable it (a plain-empty response then completes immediately).
policy=TurnPolicy(max_iterations=2, empty_recovery=RecoveryLimits(enabled=False)),
tools=ToolWiring(restrict_to_workspace=True),
engine=EngineWiring(runtime_config=RuntimeConfig(checkpoint=CheckpointConfig(policy="always"))),
)
def test_d3_interleaved_sessions_keep_recovery_isolated(workspace):
"""Two sessions A and B both interrupted; assembling for B must NOT
inject A's recovery (and vice-versa). Key isolation in _pending_recovery."""
agent = _agent_with_checkpoint(workspace)
agent._pending_recovery["A"] = {"checkpoint_id": "aaa", "files": ["a.py"]}
agent._pending_recovery["B"] = {"checkpoint_id": "bbb", "files": ["b.py"]}
msgs_b = [{"role": "user", "content": "bbb-turn"}]
agent._inject_recovery_block("B", msgs_b)
text_b = msgs_b[-1]["content"]
assert "b.py" in text_b and "bbb" in text_b
assert "a.py" not in text_b and "aaa" not in text_b
msgs_a = [{"role": "user", "content": "aaa-turn"}]
agent._inject_recovery_block("A", msgs_a)
text_a = msgs_a[-1]["content"]
assert "a.py" in text_a and "aaa" in text_a
assert "b.py" not in text_a
def test_d3_repeated_stash_latest_wins(workspace):
"""If a session is interrupted twice before consumption, the latter stash
must replace the former — recovery prompt should reflect *current* state,
not stale files."""
agent = _agent_with_checkpoint(workspace)
from raven.agent.loop import LoopOutcome
out1 = LoopOutcome(status="interrupted", checkpoint_id="old", edited_files=["stale.py"])
out2 = LoopOutcome(status="interrupted", checkpoint_id="new", edited_files=["fresh.py"])
with workdir.bind(workspace):
agent._stash_recovery("s", out1)
agent._stash_recovery("s", out2)
pending = agent._pending_recovery["s"]
assert pending["checkpoint_id"] == "new"
assert pending["files"] == ["fresh.py"]
async def test_d3_concurrent_commits_serialize_or_degrade(workspace):
"""Two concurrent commit_turn calls share one shadow .git. Git's index lock
means at most one can hold the lock at a time. Either both succeed (one
waits) or one degrades to (None, []) — but neither must raise."""
svc = CheckpointService(workspace)
# Seed an initial commit so both racers can detect changes.
(workspace / "seed.py").write_text("0\n", encoding="utf-8")
await svc.commit_turn("seed")
(workspace / "a.py").write_text("1\n", encoding="utf-8")
(workspace / "b.py").write_text("2\n", encoding="utf-8")
r1, r2 = await asyncio.gather(
svc.commit_turn("race-1"),
svc.commit_turn("race-2"),
return_exceptions=False, # the service must not raise
)
# At least one snapshot must have captured the two new files; if one was
# locked out we accept (None, []) for it but never an exception.
ids = [r1[0], r2[0]]
assert any(cid is not None for cid in ids), f"at least one concurrent commit should land; got {r1!r}, {r2!r}"
# =============================================================================
# D2 bonus — loop wraps degrade
# =============================================================================
# =============================================================================
# D5 — Performance
# =============================================================================
@pytest.mark.slow
async def test_d5_perf_1k_files_commit(workspace, capsys):
"""Measure commit_turn latency on a 1000-file workspace:
- cold commit (full snapshot)
- warm commit after modifying 10 files (typical-turn diff)
Generous bounds: this is a sanity check, not a strict perf gate.
"""
import time
for i in range(1000):
(workspace / f"f{i:04d}.py").write_text(f"# file {i}\n", encoding="utf-8")
svc = CheckpointService(workspace)
t0 = time.perf_counter()
cid1, changed1 = await svc.commit_turn("cold")
t_cold = time.perf_counter() - t0
assert cid1 is not None
assert len(changed1) == 1000
for i in range(10):
(workspace / f"f{i:04d}.py").write_text(f"# mod {i}\n", encoding="utf-8")
t0 = time.perf_counter()
cid2, changed2 = await svc.commit_turn("warm")
t_warm = time.perf_counter() - t0
assert cid2 is not None and cid2 != cid1
assert len(changed2) == 10
with capsys.disabled():
print(f"\n[D5 perf] 1000-file cold commit: {t_cold:.2f}s | 10-file diff after 1000 baseline: {t_warm:.2f}s")
# Generous upper bound — catches catastrophic regressions only.
assert t_cold < 30.0, f"cold commit took {t_cold:.2f}s, too slow"
assert t_warm < 10.0, f"warm commit took {t_warm:.2f}s, too slow"
async def test_d2_loop_unaffected_when_shadow_blocked(workspace):
"""Tying D2 back to the loop: if shadow is unusable (blocked by a file),
the loop still returns cleanly — the safety net never breaks the turn."""
(workspace / ".raven").write_text("blocker", encoding="utf-8")
agent = _agent_with_checkpoint(workspace)
final, _u, _m, outcome = await _run_turn_body(agent, workspace)
# _NoopProvider returns no tool calls → natural completion.
assert outcome.status == "completed"
assert outcome.checkpoint_id is None # shadow blocked → degrade
# =============================================================================
# D6 — Policy gating (S3): policy × interactive determines activation
# =============================================================================
def _agent(workspace: Path, *, policy: str, interactive: bool) -> AgentLoop:
return AgentLoop(
provider=_NoopProvider(),
workspace=workspace,
model="stub",
# These checkpoint tests use a no-op model only as a "turn ends" stub;
# empty-recovery is an orthogonal feature they don't exercise, so
# disable it (a plain-empty response then completes immediately).
policy=TurnPolicy(max_iterations=2, empty_recovery=RecoveryLimits(enabled=False), interactive=interactive),
tools=ToolWiring(restrict_to_workspace=True),
engine=EngineWiring(runtime_config=RuntimeConfig(checkpoint=CheckpointConfig(policy=policy))),
)
def test_d6_policy_never_disables_checkpoint(workspace):
"""``policy="never"`` is the kill switch — no shadow git regardless of
interactive. Loop is byte-identical to the pre-checkpoint baseline."""
a_inter = _agent(workspace, policy="never", interactive=True)
a_one_shot = _agent(workspace, policy="never", interactive=False)
with workdir.bind(workspace):
assert a_inter._turn_checkpoint() is None
assert a_one_shot._turn_checkpoint() is None
def test_d6_policy_always_overrides_interactive(workspace):
"""``policy="always"`` keeps the checkpoint on even for ``-m`` one-shots.
Useful for users who want the safety net everywhere."""
a_inter = _agent(workspace, policy="always", interactive=True)
a_one_shot = _agent(workspace, policy="always", interactive=False)
with workdir.bind(workspace):
assert a_inter._turn_checkpoint() is not None
assert a_one_shot._turn_checkpoint() is not None
def test_d6_policy_interactive_gated_by_call_site(workspace):
"""Default ``policy="interactive"`` follows the caller's signal:
REPL/TUI/gateway get the snapshot, ``-m`` one-shot doesn't (no next turn
to inject recovery into anyway)."""
a_inter = _agent(workspace, policy="interactive", interactive=True)
a_one_shot = _agent(workspace, policy="interactive", interactive=False)
with workdir.bind(workspace):
assert a_inter._turn_checkpoint() is not None
assert a_one_shot._turn_checkpoint() is None
def test_d6_default_policy_is_interactive(workspace):
"""Sanity check on the rollout decision: default is ``"interactive"`` so
out-of-the-box behavior matches the table the team approved."""
cfg = CheckpointConfig()
assert cfg.policy == "interactive"
# =============================================================================
# D7 — Safety hardening (S4-A/B/C): excludes, .gitignore respect, GC
# =============================================================================
async def test_d7_ephemeral_dirs_never_snapshotted(workspace):
"""S4-B: built-in excludes keep build artifacts out of the shadow git
even when the user has no ``.gitignore``. node_modules, venv, build/,
*.log all share the same pattern; we sample three categories."""
(workspace / "node_modules").mkdir()
(workspace / "node_modules" / "react").mkdir()
(workspace / "node_modules" / "react" / "index.js").write_text("// huge", encoding="utf-8")
(workspace / "venv").mkdir()
(workspace / "venv" / "bin").mkdir()
(workspace / "venv" / "bin" / "python").write_text("#!/bin/sh", encoding="utf-8")
(workspace / "app.log").write_text("startup\n", encoding="utf-8")
(workspace / "real.py").write_text("x=1\n", encoding="utf-8")
svc = CheckpointService(workspace)
cid, changed = await svc.commit_turn("t1")
assert cid is not None
# Real source file is captured...
assert "real.py" in changed
# ...ephemeral noise is not.
assert not any(p.startswith("node_modules/") for p in changed), changed
assert not any(p.startswith("venv/") for p in changed), changed
assert "app.log" not in changed, changed
async def test_d7_dotenv_and_secrets_never_snapshotted(workspace):
"""S4-B: credential-shaped files (``.env``, ``*.key``, ``*.pem``) ship
in the default exclude list. Defense in depth — the user's own
``.gitignore`` usually covers these, but a fresh workspace might not."""
(workspace / ".env").write_text("API_KEY=hunter2\n", encoding="utf-8")
(workspace / ".env.production").write_text("API_KEY=prod\n", encoding="utf-8")
(workspace / "deploy.pem").write_text("-----BEGIN PRIVATE KEY-----\n", encoding="utf-8")
(workspace / "main.py").write_text("ok\n", encoding="utf-8")
svc = CheckpointService(workspace)
cid, changed = await svc.commit_turn("t1")
assert cid is not None
assert "main.py" in changed
assert ".env" not in changed, changed
assert ".env.production" not in changed, changed
assert "deploy.pem" not in changed, changed
async def test_d7_respects_user_gitignore_in_workspace(workspace):
"""S4-A: when the workspace itself has a ``.gitignore``, the shadow git
honors it automatically (standard ``add -A`` semantics on the work-tree).
This is the load-bearing safety property for credential-leak protection
— verified end-to-end here so a future change can't quietly remove it."""
(workspace / ".gitignore").write_text("private_data.yml\nlocal_cache/\n", encoding="utf-8")
(workspace / "private_data.yml").write_text("token: hunter2\n", encoding="utf-8")
(workspace / "local_cache").mkdir()
(workspace / "local_cache" / "blob.bin").write_text("xxx", encoding="utf-8")
(workspace / "code.py").write_text("ok\n", encoding="utf-8")
svc = CheckpointService(workspace)
cid, changed = await svc.commit_turn("t1")
assert cid is not None
assert "code.py" in changed
# .gitignore itself IS snapshotted (it's a real file the user wants tracked)...
assert ".gitignore" in changed
# ...but the patterns inside it are honored.
assert "private_data.yml" not in changed, changed
assert not any(p.startswith("local_cache/") for p in changed), changed
async def test_d7_notice_txt_written_next_to_shadow(workspace):
"""S4 discoverability: a user noticing ``.raven/`` should be able to
identify what it is without grepping our codebase."""
svc = CheckpointService(workspace)
(workspace / "x.py").write_text("ok\n", encoding="utf-8")
await svc.commit_turn("t1")
notice = workspace / ".raven" / "NOTICE.txt"
assert notice.exists()
text = notice.read_text()
assert "Raven" in text and "checkpoint" in text
assert "policy" in text # tells them how to turn it off
async def test_d7_gc_auto_config_set_on_shadow_repo(workspace):
"""S4-C: ``gc.auto`` is set in the shadow repo's git config at init so
long-lived sessions can prune loose objects. We don't measure GC firing
(slow + flaky); the config bit is the load-bearing piece."""
svc = CheckpointService(workspace)
(workspace / "x.py").write_text("ok\n", encoding="utf-8")
await svc.commit_turn("t1")
rc, out, _ = await svc._git("config", "--get", "gc.auto")
assert rc == 0
assert out.strip() == "256"
async def test_d7_gc_invoked_every_n_commits(workspace, monkeypatch):
"""S4-C heartbeat: every N successful commits the service fires
``git gc --auto``. We spy on ``_git`` to count invocations rather than
pay for a real GC (which is cheap but still noisier than necessary)."""
import raven.agent.loop.checkpoint as cp_module
# Make the heartbeat fire every 3 commits to keep this test fast.
monkeypatch.setattr(cp_module, "_GC_EVERY_N_COMMITS", 3)
svc = CheckpointService(workspace)
real_git = svc._git
gc_calls: list[tuple] = []
async def _spy(*args):
if args[:1] == ("gc",):
gc_calls.append(args)
return await real_git(*args)
monkeypatch.setattr(svc, "_git", _spy)
for i in range(7):
(workspace / f"f{i}.py").write_text(f"{i}\n", encoding="utf-8")
cid, _ = await svc.commit_turn(f"t{i}")
assert cid is not None
# Commits 3 and 6 should each have triggered one gc --auto.
assert len(gc_calls) == 2, f"expected 2 gc fires, got {gc_calls}"
assert all(call[1] == "--auto" for call in gc_calls)
# =============================================================================
# D8 — End-to-end: the full safety story under a multi-turn AgentLoop
# =============================================================================
async def test_d8_one_shot_mode_creates_no_shadow_dir(workspace):
"""S3 end-to-end: ``interactive=False`` + default policy means
``raven agent -m "..."`` leaves no ``.raven/`` artifact on disk —
one-shot commands don't pay the shadow-git cost."""
agent = _agent(workspace, policy="interactive", interactive=False)
_f, _u, _m, outcome = await _run_turn_body(agent, workspace)
assert outcome.status == "completed"
assert outcome.checkpoint_id is None
assert not (workspace / ".raven").exists()
# =============================================================================
# D9 — Containment safety: shadow_dir must stay strictly under the workspace
# =============================================================================
#
# Without this check the per-workspace recovery isolation this suite depends on
# silently breaks: a misconfigured ``shadow_dir`` can put the shadow git in
# a sibling/global path and let a second AgentLoop on a different workspace
# share the repo, cross-contaminating ``edited_files`` in the recovery
# prompt. Tests pin the four shapes of bad input so a future refactor can't
# remove the guard without an alarm.
def test_d9_parent_escape_rejected(workspace):
"""``shadow_dir="../foo"`` resolves outside the workspace → rejected."""
with pytest.raises(ValueError, match="strictly under the workspace"):
CheckpointService(workspace, shadow_dir="../escape")
def test_d9_absolute_path_rejected(workspace):
"""``shadow_dir`` as an absolute path replaces the workspace prefix
entirely (standard ``Path /`` semantics) → also rejected."""
with pytest.raises(ValueError, match="strictly under the workspace"):
CheckpointService(workspace, shadow_dir="/tmp/some_global_path")
def test_d9_empty_string_rejected(workspace):
"""Empty ``shadow_dir`` collapses to the workspace itself — a degenerate
case where the shadow's git-dir would equal the work-tree and a normal
``git init`` could clobber the user's files."""
with pytest.raises(ValueError, match="strictly under the workspace"):
CheckpointService(workspace, shadow_dir="")
def test_d9_dot_path_rejected(workspace):
"""``"."`` is the same degenerate case as the empty string — git-dir
would equal the workspace."""
with pytest.raises(ValueError, match="strictly under the workspace"):
CheckpointService(workspace, shadow_dir=".")
def test_d9_nested_relative_path_accepted(workspace):
"""Sanity: a normal nested relative path is fine."""
svc = CheckpointService(workspace, shadow_dir="deep/nested/shadow.git")
# ``workspace`` is an unresolved tmp path; CheckpointService resolves it
# (on macOS /var -> /private/var), so compare against the resolved form.
assert svc._git_dir.is_relative_to(workspace.resolve())
assert svc._git_dir.name == "shadow.git"
async def test_d9_agent_loop_degrades_when_shadow_dir_invalid(workspace):
"""AgentLoop must not crash when the user config contains a bad
``shadow_dir`` — checkpoint silently disables so the turn still runs."""
bad_cfg = RuntimeConfig(
checkpoint=CheckpointConfig(
policy="always",
shadow_dir="../escape_via_config",
)
)
agent = AgentLoop(
provider=_NoopProvider(),
workspace=workspace,
model="stub",
# These checkpoint tests use a no-op model only as a "turn ends" stub;
# empty-recovery is an orthogonal feature they don't exercise, so
# disable it (a plain-empty response then completes immediately).
policy=TurnPolicy(max_iterations=2, empty_recovery=RecoveryLimits(enabled=False), interactive=True),
tools=ToolWiring(restrict_to_workspace=True),
engine=EngineWiring(runtime_config=bad_cfg),
)
with workdir.bind(workspace):
assert agent._turn_checkpoint() is None, "bad config should degrade to no checkpoint"
final, _u, _m, outcome = await _run_turn_body(agent, workspace)
assert outcome.status == "completed"
assert outcome.checkpoint_id is None # no shadow → no checkpoint id
# And no escaped directory was created either.
escaped = workspace.parent / "escape_via_config"
assert not escaped.exists(), f"shadow leaked to {escaped}"
# =============================================================================
# D10 — Contract atomicity: recovery is never silently dropped, git never hangs
# =============================================================================
def test_d10_recovery_kept_when_content_unknown_type(workspace):
"""Regression: previously ``_inject_recovery_block`` ``pop()``'d the
pending entry BEFORE checking whether ``content`` was a shape it could
actually write to (str / list). A user message with ``content=None``
(or any other shape) would therefore consume the recovery silently —
the user got a "Please continue" prompt with no clue what was lost.
Now the pop only happens after a successful mutation; an unknown
content shape keeps the recovery pending for the next assembly."""
agent = _agent_with_checkpoint(workspace)
agent._pending_recovery["s"] = {"checkpoint_id": "abc123", "files": ["x.py"]}
# First attempt: content=None → injection can't safely happen.
msgs_bad = [{"role": "user", "content": None}]
agent._inject_recovery_block("s", msgs_bad)
assert msgs_bad[-1]["content"] is None, "content must not be mutated"
assert "s" in agent._pending_recovery, "recovery must NOT be dropped"
# Subsequent attempt with normal str content lands cleanly.
msgs_ok = [{"role": "user", "content": "go"}]
agent._inject_recovery_block("s", msgs_ok)
assert "abc123" in msgs_ok[-1]["content"]
assert "x.py" in msgs_ok[-1]["content"]
assert "s" not in agent._pending_recovery
async def test_d10_git_subprocess_times_out_and_kills_proc(workspace, monkeypatch):
"""If the git child hangs (NFS lock / index-lock / disk full), ``_git``
must time out instead of blocking the whole turn. Verified by stubbing
``asyncio.create_subprocess_exec`` to return a fake proc whose
``communicate`` never returns, plus shrinking the timeout so the test
runs in milliseconds. The hanging proc must also be killed (no zombie)."""
import raven.agent.loop.checkpoint as cp_module
monkeypatch.setattr(cp_module, "_GIT_TIMEOUT_SECONDS", 0.05)
class _HangingProc:
returncode = None
def __init__(self) -> None:
self.killed = False
async def communicate(self):
await asyncio.sleep(60) # never reached — wait_for fires first
return b"", b""
def kill(self) -> None:
self.killed = True
async def wait(self) -> int:
return -9
fake = _HangingProc()
async def _fake_create(*_args, **_kwargs):
return fake
monkeypatch.setattr(asyncio, "create_subprocess_exec", _fake_create)
svc = CheckpointService(workspace)
rc, out, err = await svc._git("status")
assert rc == -1, f"timeout must surface as non-zero rc, got {rc}"
assert err == "timeout"
assert out == ""
assert fake.killed, "hanging proc must be killed to avoid zombies"
async def test_d10_commit_turn_degrades_on_git_hang(workspace, monkeypatch):
"""End-to-end: if any git call hangs, ``commit_turn`` returns
``(None, [])`` and does not raise — preserving the "never break a turn"
contract that the rest of the safety net relies on."""
import raven.agent.loop.checkpoint as cp_module
monkeypatch.setattr(cp_module, "_GIT_TIMEOUT_SECONDS", 0.05)
class _HangingProc:
returncode = None
async def communicate(self):
await asyncio.sleep(60)
return b"", b""
def kill(self) -> None:
pass
async def wait(self) -> int:
return -9
async def _fake_create(*_args, **_kwargs):
return _HangingProc()
monkeypatch.setattr(asyncio, "create_subprocess_exec", _fake_create)
svc = CheckpointService(workspace)
(workspace / "a.py").write_text("ok\n", encoding="utf-8")
cid, changed = await svc.commit_turn("t1")
assert cid is None and changed == [], f"hang must degrade to no-op, got {(cid, changed)!r}"
async def test_d8_interactive_mode_with_user_gitignore_end_to_end(workspace):
"""S3 + S4-A together: an interactive session in a workspace with a
``.gitignore`` snapshots the source files but never the user-marked
private ones. This is the load-bearing user-facing guarantee."""
(workspace / ".gitignore").write_text(".env\n", encoding="utf-8")
(workspace / ".env").write_text("SECRET=abc\n", encoding="utf-8")
(workspace / "src.py").write_text("ok\n", encoding="utf-8")
agent = _agent(workspace, policy="interactive", interactive=True)
_f, _u, _m, outcome = await _run_turn_body(agent, workspace)
# The (no-op provider) turn still completes; the workspace files get
# snapshotted by the end-of-turn commit_turn.
assert outcome.status == "completed"
assert outcome.checkpoint_id is not None
# Verify the snapshot's contents directly via the shadow git.
with workdir.bind(workspace):
svc = agent._turn_checkpoint()
rc, out, _ = await svc._git("ls-tree", "-r", "--name-only", "HEAD")
assert rc == 0
tracked = out.splitlines()
assert "src.py" in tracked
assert ".gitignore" in tracked
assert ".env" not in tracked, f"secret leaked into shadow: {tracked}"