## Summary The research, code, oncall and ppt agents keep prompt files in their workspace that the model reads on every turn: `agent_memory/profile/soul.md`, `agent_memory/profile/agent.md` and `TOOLS.md`. They were written only when absent (`seed_once`), so after a raven upgrade or a config change the model kept reading the text of the first launch while the gates enforced the current release. A Raven-Research install seeded before the reader layout still asked the model for the `## Answer` / `## Findings` / `## Limitations` report while its shape gate checked the reader layout. These files are now generated output: - `write_generated()` in `raven/config/product_render.py` rewrites a product-owned file from the release and the rendered config on every launch. - An operator's own lines go in the sibling `*.local.md` file (`agent_memory/profile/agent.local.md`, `soul.local.md`, `TOOLS.local.md`) and are appended after the product text at the next launch. They must be UTF-8: one in another encoding (an editor's legacy code page, GBK for example) raises a decode error that names the `*.local.md` and asks for UTF-8, rather than one naming the generated file. - A receipt beside each file (`.<name>.generated.sha256`) records `<owner> <digest>` for the last text written. A file the caller's own receipt lines do not name (edited in place, written by another product, or left by an older release) is backed up to `<name>.bak-<time>-<pid>` before it is replaced, and the log says which of those it is when the receipt can tell. A backup is never readable by more accounts than the original: on POSIX an owner-only copy; on Windows, where a new file takes its folder's inherited ACL whatever mode it is created with, a hard link to the file itself, so it keeps the file's security descriptor and no bytes are written. A Windows volume that cannot hard-link refuses the backup, and a failed backup leaves the file untouched. While the target changes, the receipt names both texts, so a failure between the two writes never leaves a file the next launch cannot recognise. - `product_text=None` (raven-code's `CODE_CONDUCT=off`) removes only a file this product wrote; an edited one, or another product's file and receipt, is left alone. - The launchers pass `adopt=True`, since their home is the product's own ACP home, and refuse to start with the file named when a prompt cannot be written. On a raven without `write_generated` they fall back to seeding once and log that the file will not be refreshed. - The ppt engine writes into whatever home its plugin is granted, which can be a host's or another product's, so it passes `adopt=False`: it replaces only a seat its own receipt, or the digest of a prompt it has shipped, shows it wrote. A test fails when a ppt prompt changes without its digest being added. It marks its identity written only once the write lands and retries on the next turn otherwise. A seat that fails does not stop the other seats; the host's template sync waits for the retry, since it would fill the failed seat and a seat the engine did not write is never replaced. A seat it leaves alone is logged at DEBUG, not WARNING: in a host's home those are the host's own, every process, with nothing to act on. The notice says how to hand a seat back (move the lines into `*.local.md`, delete the file), which is what a seat edited in Raven-PPT's own home before this release needs; the docs name that exception. - raven-code's earlier per-file receipts and legacy digests are still honoured, so untouched copies are refreshed without a backup. - `atomic_replace()` takes a `newline` argument so generated files stay LF on every platform. It also removes its temp file when the replace fails, and removes a temp file already present before it writes, clearing the read-only attribute first: Windows refuses to replace a read-only target once the temp file has taken its mode, and the read-only temp file left behind used to fail every later launch even after the target was made writable again. - The temp file `atomic_replace()` writes is created no more open than the target from its first byte. On Windows a new file takes its folder's inherited ACL and `os.replace` (`MoveFileExW`) moves that descriptor onto the target, so a prompt kept private in a folder other accounts may read came back readable by them, `*.local.md` lines included. When the target exists, the temp file is now made by `CreateFileW` with the target's DACL as `GetFileSecurityW` reads it, and the save is refused when that DACL cannot be read. On POSIX it is created with the target's mode rather than the umask default, closing the span in which the new text sat in a wider file until the chmod. This holds for every caller that passes no explicit mode, `config.json` included. The Win32 bindings behind it are declared once per process, behind a cache: a structure defined per call left a new ctypes pointer type, about 6 KB, alive for every save. Docs: `agents/README.md` (the seed-once contract becomes "Regenerated prompts", with rollback steps), `docs-site` agent integrations (en and zh), `CONTEXT-MAP.md`, and `CHANGELOG.md` (an unreleased entry that promised the old behaviour, plus a Fixed entry). Tests whose meaning changed: seven existing tests asserted that an edited prompt is never overwritten; they now assert it is backed up and regenerated. The raven-code launcher test fixture now pins `CODE_CONDUCT` and `CODE_CONDUCT_DISCIPLINE` to `on`: a deleted switch fell back to the developer's `.env`, and two conduct tests on main already failed with `CODE_CONDUCT_DISCIPLINE=off` for that reason. Checked and deliberately not changed (documented in `agents/README.md`): - The processes of one product (one per model binding) share these files, so a launch can change what an older process still running reads on its next turn. Closing that needs a per-process snapshot or in-process injection of the prompt, which is a separate change. - Backups are never deleted automatically. - Host workspace templates (`USER.md`, `HEARTBEAT.md` and the rest) are the user's files and stay write-if-missing. - A file removed from a release is not removed from an installed `~/.raven/agents` copy. Nothing loads such a leftover today: modes are read by label, tools are imported by name, and no release has removed a skill directory. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run at the head of this branch, rebased on the current main: - `uv run pytest -q`: 32739 passed, 17 failed, 147 skipped, 2 xfailed. The same 17 test IDs fail on the current main in a clean worktree with the same interpreter and the same fetched tokenizer dictionary (proxy redaction, tokenizer data, network-safety resolution, install script, document converters, node runtime, trajectory time zone); none is introduced here. - `uv run pytest tests/test_agents_code_launcher.py -q` with `CODE_CONDUCT` and `CODE_CONDUCT_DISCIPLINE` each set to `off` in the environment, alone and together: 79 passed in all four runs. - `make lint-python lint-imports lint-types lint-deps`: all pass, 10 import contracts kept, no dependency issues. - `scripts/check_commit_messages.py`, `scripts/check_source_language.py` and `scripts/check_large_files.py` over `origin/main..HEAD`, and `git diff --check`: pass. - Each fix was reverted on its own and its test went red: the receipt written after the target, a failed target write, the backup mode, the ppt `adopt=False` guard, receipts without an owner, an undeletable earlier receipt, the `*.local.md` read-back from the assembled system prompt, the temp file a failed replace leaves behind, and the temp file's mode while it holds the new text. - Windows review findings: `test_a_windows_backup_is_the_edited_file_itself_so_its_acl_comes_with_it` and `test_a_windows_backup_the_volume_cannot_link_is_refused_not_copied` force the link path on Linux; `test_a_failed_replace_takes_its_tmp_with_it` and `test_a_read_only_tmp_a_dead_writer_left_does_not_block_the_replace` cover the temp file on every platform; `test_the_tmp_is_never_more_open_than_the_target_while_it_holds_the_new_bytes` pins the POSIX mode from creation; `test_a_read_only_prompt_regenerates_once_it_is_writable_again` (read-only retry) and `test_a_private_windows_prompt_keeps_its_own_dacl_when_regenerated_and_backed_up` (a private prompt in a folder every local user may read keeps its own DACL through regeneration and backup, read back with `icacls /save`) run on Windows only. - The `Windows ACP stdio` CI job gains a step that runs `tests/test_config_product_render.py` and the two temp-file tests natively. On the Windows runner: 158 passed, 2 skipped (the two POSIX-mode tests), the DACL, read-only retry and hard-link backup tests among the passes. - Diff coverage over `raven/` with the related suites: 94.12%, above the 90% gate. - Second-round review findings, each test red without its fix: `test_an_addendum_that_is_not_utf8_is_named_and_nothing_is_written`, `test_a_local_file_that_is_not_utf8_is_named_when_the_launch_stops` and `test_an_addendum_that_is_not_utf8_names_itself_and_holds_back_no_other_seat` (the decode error names the addendum; the ppt engine writes the other seats and holds the sync); `test_a_hosts_own_seats_are_left_alone_without_a_warning_in_every_process` and `test_a_seat_left_alone_says_how_to_hand_it_back` (DEBUG level, and the notice's instructions); `test_the_windows_bindings_are_declared_once_not_per_save` (Win32 stand-ins on Linux, two saves load each DLL once). Windows is covered by the CI step above, not by a local run. The CRLF fold of an older copy is reasoned, not run. `npx commitlint` is not installed on the machine this was run on; the Python commit check above was run instead. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - User-visible: on the first launch after upgrading, each agent rewrites its prompt files. A file edited in place, or left by an older release without a receipt, is copied to `<name>.bak-*` and the launcher log names the copy; an in-place edit stops taking effect until its lines are moved into `*.local.md`. A `*.local.md` that is not UTF-8 stops the launch with the file named. In Raven-PPT's own home a seat edited before this release keeps the edit, and its `*.local.md` is not applied, until the seat is deleted. A Raven-Research install seeded before the reader layout switches to it at that launch. - Security: a backup is never readable by more accounts than the file it keeps (an owner-only copy on POSIX, a hard link keeping the file's ACL on Windows), and a replaced file keeps its own mode, and on Windows its DACL, from the temp file's first byte; receipts hold product names and digests only; `*.local.md` is operator-owned and carries the same trust as editing `agent.md` did before. - Compatibility: a launcher running on a raven without `write_generated` seeds once and logs it; the ppt engine on such a raven keeps write-if-missing. On a Windows volume that cannot hard-link (FAT32, exFAT, some network shares), an edited prompt file stops the launch with the file named, until its lines move to `*.local.md` and the file is deleted. On Windows, a save through `atomic_replace()` whose target's DACL cannot be read is refused rather than made with the folder's ACL. - Rollback: revert this change, then stop the product's processes, delete each generated file whose `.<name>.generated.sha256` receipt still names its current text, and relaunch so the older code seeds its own. `*.local.md` files, backups and edited files are left as they are. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: userName20260323 <270284818+userName20260323@users.noreply.github.com> Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
302 lines
11 KiB
Python
302 lines
11 KiB
Python
"""Tests for raven.utils.atomic_io."""
|
|
|
|
import multiprocessing
|
|
import os
|
|
import stat
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from raven.utils.atomic_io import atomic_replace, atomic_update, locked_append, write_transaction
|
|
|
|
WRITERS = 2
|
|
CALLS_PER_WRITER = 50
|
|
LINES_PER_CALL = 5
|
|
|
|
|
|
def _append_worker(path_str: str, writer_id: int) -> None:
|
|
for call_idx in range(CALLS_PER_WRITER):
|
|
block = [f"{writer_id}:{call_idx}:{line_idx}" for line_idx in range(LINES_PER_CALL)]
|
|
locked_append(Path(path_str), block)
|
|
|
|
|
|
def test_locked_append_appends_lines(tmp_path: Path):
|
|
"""Sequential calls accumulate lines in order."""
|
|
path = tmp_path / "s.jsonl"
|
|
locked_append(path, ["a", "b"])
|
|
locked_append(path, ["c"])
|
|
assert path.read_text(encoding="utf-8") == "a\nb\nc\n"
|
|
|
|
|
|
def test_lock_lives_in_hidden_lock_subdir(tmp_path: Path):
|
|
"""The advisory lock sidecar lives in a hidden ``.lock/`` dir derived from
|
|
the target's own parent — never beside the target file."""
|
|
path = tmp_path / "s.jsonl"
|
|
locked_append(path, ["a"])
|
|
beside = [p.name for p in tmp_path.iterdir() if p.is_file() and p.name.endswith(".lock")]
|
|
assert beside == []
|
|
assert (tmp_path / ".lock" / "s.jsonl.lock").exists()
|
|
|
|
|
|
def test_locked_append_concurrent_writers_lose_nothing(tmp_path: Path):
|
|
"""Two processes appending concurrently: every line lands, and the
|
|
lines of one locked_append call stay contiguous (turn-block invariant)."""
|
|
path = tmp_path / "s.jsonl"
|
|
# spawn, not the Linux default fork: pytest leaves the parent multi-threaded,
|
|
# and forking from there segfaults the interpreter at exit.
|
|
ctx = multiprocessing.get_context("spawn")
|
|
procs = [ctx.Process(target=_append_worker, args=(str(path), w)) for w in range(WRITERS)]
|
|
for p in procs:
|
|
p.start()
|
|
for p in procs:
|
|
p.join(timeout=60)
|
|
assert p.exitcode == 0
|
|
|
|
lines = path.read_text(encoding="utf-8").splitlines()
|
|
assert len(lines) == WRITERS * CALLS_PER_WRITER * LINES_PER_CALL
|
|
assert len(set(lines)) == len(lines)
|
|
|
|
block_positions: dict[tuple[str, str], list[int]] = {}
|
|
for pos, line in enumerate(lines):
|
|
writer_id, call_idx, _ = line.split(":")
|
|
block_positions.setdefault((writer_id, call_idx), []).append(pos)
|
|
for positions in block_positions.values():
|
|
assert positions == list(range(positions[0], positions[0] + LINES_PER_CALL))
|
|
|
|
|
|
def test_locked_append_repairs_missing_trailing_newline(tmp_path: Path):
|
|
"""Appending after a crashed partial line starts on a fresh line, so
|
|
the new record is not merged into the partial one."""
|
|
path = tmp_path / "s.jsonl"
|
|
path.write_text('{"partial": "tru', encoding="utf-8")
|
|
locked_append(path, ["next"])
|
|
assert path.read_text(encoding="utf-8") == '{"partial": "tru\nnext\n'
|
|
|
|
|
|
def test_atomic_replace_swaps_content(tmp_path: Path):
|
|
"""atomic_replace replaces the whole file and leaves no temp residue."""
|
|
path = tmp_path / "s.jsonl"
|
|
path.write_text("old\n", encoding="utf-8")
|
|
atomic_replace(path, "new1\nnew2\n")
|
|
assert path.read_text(encoding="utf-8") == "new1\nnew2\n"
|
|
residue = [p.name for p in tmp_path.iterdir() if p.name not in ("s.jsonl", ".lock")]
|
|
assert residue == []
|
|
|
|
|
|
def test_atomic_replace_creates_missing_file(tmp_path: Path):
|
|
path = tmp_path / "fresh.jsonl"
|
|
atomic_replace(path, "data\n")
|
|
assert path.read_text(encoding="utf-8") == "data\n"
|
|
|
|
|
|
def test_helpers_work_cross_platform(tmp_path: Path):
|
|
"""Both helpers work on every platform. Locking is cross-platform via
|
|
portalocker (POSIX fcntl + Windows LockFileEx) — there is no longer an
|
|
fcntl-absent 'degrade to unlocked' path."""
|
|
path = tmp_path / "s.jsonl"
|
|
locked_append(path, ["x"])
|
|
atomic_replace(path, "y\n")
|
|
assert path.read_text(encoding="utf-8") == "y\n"
|
|
|
|
|
|
def test_atomic_replace_preserves_the_targets_mode(tmp_path: Path):
|
|
"""os.replace swaps the inode: a config chmod 600 (api keys) must not come
|
|
back world-readable after a save — the leak _persist_migrations documented
|
|
and five sibling _write_atomic copies never fixed."""
|
|
path = tmp_path / "config.json"
|
|
path.write_text("{}", encoding="utf-8")
|
|
os.chmod(path, 0o600)
|
|
|
|
atomic_replace(path, '{"k": 1}')
|
|
|
|
assert (path.stat().st_mode & 0o777) == 0o600
|
|
|
|
|
|
def test_atomic_update_concurrent_increments_lose_nothing(tmp_path: Path):
|
|
"""The locked RMW transaction is what the unlocked tmp+replace sites lack:
|
|
two writers read-modify-write concurrently and neither update is swallowed."""
|
|
import threading
|
|
|
|
path = tmp_path / "counter.txt"
|
|
|
|
def bump(_: int) -> None:
|
|
def update(current: str | None) -> tuple[str, None]:
|
|
value = int(current or "0")
|
|
return str(value + 1), None
|
|
|
|
atomic_update(path, update)
|
|
|
|
threads = [threading.Thread(target=bump, args=(i,)) for i in range(16)]
|
|
for t in threads:
|
|
t.start()
|
|
for t in threads:
|
|
t.join()
|
|
|
|
assert path.read_text(encoding="utf-8") == "16"
|
|
|
|
|
|
def test_atomic_replace_with_explicit_mode_holds_from_creation(tmp_path: Path):
|
|
"""Secret-bearing files (serve.json's token) need their mode from the very
|
|
first byte; an explicit mode wins over both umask and a leftover tmp's
|
|
stale wide mode."""
|
|
path = tmp_path / "serve.json"
|
|
stale_tmp = tmp_path / "serve.json.tmp"
|
|
stale_tmp.write_text("crashed writer residue", encoding="utf-8")
|
|
os.chmod(stale_tmp, 0o644)
|
|
|
|
atomic_replace(path, '{"token": "s3cret"}', mode=0o600)
|
|
|
|
assert (path.stat().st_mode & 0o777) == 0o600
|
|
assert path.read_text(encoding="utf-8") == '{"token": "s3cret"}'
|
|
|
|
|
|
@pytest.mark.skipif(os.name == "nt", reason="POSIX file modes are not available on Windows")
|
|
def test_the_tmp_is_never_more_open_than_the_target_while_it_holds_the_new_bytes(tmp_path: Path, monkeypatch):
|
|
"""A chmod after the write left a private target's next text in a
|
|
umask-default tmp for the span in between."""
|
|
path = tmp_path / "agent.md"
|
|
path.write_text("private", encoding="utf-8")
|
|
os.chmod(path, 0o600)
|
|
seen: list[int] = []
|
|
real_fsync = os.fsync
|
|
|
|
def watched(fd: int) -> None:
|
|
seen.append(stat.S_IMODE(os.fstat(fd).st_mode))
|
|
real_fsync(fd)
|
|
|
|
monkeypatch.setattr(os, "fsync", watched)
|
|
previous = os.umask(0o022)
|
|
try:
|
|
atomic_replace(path, "still private")
|
|
finally:
|
|
os.umask(previous)
|
|
assert seen == [0o600]
|
|
assert path.read_text(encoding="utf-8") == "still private"
|
|
|
|
|
|
def test_the_windows_bindings_are_declared_once_not_per_save(tmp_path: Path, monkeypatch):
|
|
"""ctypes keeps every pointer type it is asked for: a structure declared per
|
|
call pinned about 6 KB more on every save, for the process's life."""
|
|
import ctypes
|
|
import sys
|
|
import types
|
|
|
|
from raven.utils import atomic_io
|
|
|
|
class _Fn:
|
|
def __init__(self, impl):
|
|
self.impl = impl
|
|
|
|
def __call__(self, *args):
|
|
return self.impl(*args)
|
|
|
|
def get_security(_path, _info, buffer, _size, needed):
|
|
needed._obj.value = 16
|
|
return buffer is not None
|
|
|
|
def create(path, *_rest):
|
|
return os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
|
|
|
loaded: list[str] = []
|
|
|
|
def win_dll(name, **_kwargs):
|
|
loaded.append(name)
|
|
return types.SimpleNamespace(
|
|
GetFileSecurityW=_Fn(get_security), CreateFileW=_Fn(create), CloseHandle=_Fn(lambda _h: True)
|
|
)
|
|
|
|
monkeypatch.setattr(ctypes, "WinDLL", win_dll, raising=False)
|
|
monkeypatch.setitem(sys.modules, "msvcrt", types.SimpleNamespace(open_osfhandle=lambda handle, _flags: handle))
|
|
atomic_io._win32_security.cache_clear()
|
|
try:
|
|
target = tmp_path / "agent.md"
|
|
target.write_text("old", encoding="utf-8")
|
|
for attempt in range(2):
|
|
tmp = tmp_path / f"agent.md.{attempt}.tmp"
|
|
os.close(atomic_io._create_with_dacl_of(tmp, target))
|
|
assert tmp.is_file()
|
|
assert loaded == ["advapi32", "kernel32"]
|
|
finally:
|
|
atomic_io._win32_security.cache_clear()
|
|
|
|
|
|
def test_a_failed_replace_takes_its_tmp_with_it(tmp_path: Path, monkeypatch):
|
|
"""Windows refuses to replace a read-only target, by which point the tmp
|
|
has taken the target's read-only mode; left behind, it refused the next
|
|
attempt's open even after the target was made writable again."""
|
|
path = tmp_path / "agent.md"
|
|
path.write_text("old", encoding="utf-8")
|
|
|
|
def refused(_src, dst):
|
|
raise PermissionError(13, "Access is denied", str(dst))
|
|
|
|
monkeypatch.setattr(os, "replace", refused)
|
|
with pytest.raises(PermissionError):
|
|
atomic_replace(path, "new")
|
|
assert not (tmp_path / "agent.md.tmp").exists()
|
|
|
|
monkeypatch.undo()
|
|
atomic_replace(path, "new")
|
|
assert path.read_text(encoding="utf-8") == "new"
|
|
|
|
|
|
def test_a_read_only_tmp_a_dead_writer_left_does_not_block_the_replace(tmp_path: Path):
|
|
path = tmp_path / "agent.md"
|
|
path.write_text("old", encoding="utf-8")
|
|
stale_tmp = tmp_path / "agent.md.tmp"
|
|
stale_tmp.write_text("dead writer residue", encoding="utf-8")
|
|
os.chmod(stale_tmp, stat.S_IREAD)
|
|
|
|
atomic_replace(path, "new")
|
|
|
|
assert path.read_text(encoding="utf-8") == "new"
|
|
assert not stale_tmp.exists()
|
|
|
|
|
|
def test_write_transaction_is_reentrant_for_the_helpers(tmp_path: Path):
|
|
"""atomic_replace inside write_transaction on the same path must not
|
|
deadlock on the sidecar (a second flock in one process blocks)."""
|
|
path = tmp_path / "reg.json"
|
|
path.write_text("old", encoding="utf-8")
|
|
|
|
with write_transaction(path):
|
|
current = path.read_text(encoding="utf-8")
|
|
atomic_replace(path, current + "+new")
|
|
|
|
assert path.read_text(encoding="utf-8") == "old+new"
|
|
# The lock is genuinely released afterwards: a plain helper call proceeds.
|
|
atomic_replace(path, "after")
|
|
assert path.read_text(encoding="utf-8") == "after"
|
|
|
|
|
|
def _probe_lock(lock_path_str: str, q) -> None:
|
|
from raven.utils.portable_lock import LockTimeoutError, file_lock
|
|
|
|
try:
|
|
with file_lock(Path(lock_path_str), blocking=False):
|
|
q.put("acquired")
|
|
except LockTimeoutError:
|
|
q.put("blocked")
|
|
|
|
|
|
def test_write_transaction_excludes_other_processes(tmp_path: Path):
|
|
"""Deterministic mutual exclusion: while a transaction is held, another
|
|
process's non-blocking acquire is refused — the lost-update window between
|
|
a registry's read and its flush is closed, not narrowed."""
|
|
path = tmp_path / "reg.json"
|
|
lock_path = path.parent / ".lock" / (path.name + ".lock")
|
|
|
|
with write_transaction(path):
|
|
q = multiprocessing.Queue()
|
|
proc = multiprocessing.Process(target=_probe_lock, args=(str(lock_path), q))
|
|
proc.start()
|
|
verdict = q.get(timeout=10)
|
|
proc.join(10)
|
|
assert verdict == "blocked"
|
|
|
|
q2 = multiprocessing.Queue()
|
|
proc2 = multiprocessing.Process(target=_probe_lock, args=(str(lock_path), q2))
|
|
proc2.start()
|
|
assert q2.get(timeout=10) == "acquired"
|
|
proc2.join(10)
|