## Summary
A file an `exec` command wrote reached the desk diff as a bare row. The
loop listed the working directory on either side of the command (#692),
so it knew a file had changed but never what it changed from: a rewrite
showed a bare `M` with no counts, and opening any such row fell back to
the file viewer instead of the diff pane. File-tool edits were
unaffected; this only concerned commands (`python3 gen.py > out.json`,
`echo x >> notes.md`, `sed -i ...`).
This supersedes #826. That PR measured the change in the loop, around a
call named `exec`. Here the exec tool measures it itself, around the
command it runs, and hands it back the way it already hands back the
files a command removed; the loop only forwards it.
How it works:
- **The tool measures.** `raven/agent/tools/command_writes.py` lists the
directory before and after the command and, where the checkpoint's
shadow repo covers it, stages the tree first, so a rewritten or removed
file is shown against what it held. The result travels on a new
`ToolResult.written` / `ToolOutput.written` (`FileWrite` on the tool
paper, contract version 32 -> 33, and 33 -> 34 for
`FileRemoval.withheld`), beside the existing `removed`.
- **The repo is handed in.** Tools do not import the loop shell
(import-linter), so `ExecTool` holds the shadow repo through a
`ShadowTree` protocol and a resolver the loop passes in
(`AgentLoop._command_shadow`: the turn's repo, only where it covers the
command's directory). The built-in is built with it, and a plugin's
same-name `exec` replacement (raven-code's `CodeExecTool`) is handed it
through `measure_writes` before it is registered, so it measures as the
built-in does, as the loop-level listing it replaces did for any tool
named `exec`. Both happen before the registry admits the tool, because
the registry reads the ceiling off the spec it admits.
- **Every command stages afresh inside its own call**, into a
per-process index of its own in the shadow repo (never the index the
per-turn commit reads), and waits up to 120 s. Whenever there is no
snapshot to measure against -- the tree cannot be staged at all
(checkpoint off, `working_dir` outside the repo, git error) or it did
not finish within the wait -- the command runs all the same and is still
listed, so its files are reported without the text they held; where a
repo covers the directory, its rules still decide what text may be
shown. A late staging is left running, so a later command usually finds
it done. What follows the command is bounded too: the shadow repo's
reads get 60 s (past it every file goes out without text) and the whole
post-command step 120 s (past it the call returns the command's output
with no record of its files), so a slow measurement never costs the
output. `measure_writes` raises the tool's registry ceiling by what
measuring may take (the 120 s wait and the 120 s bound), so the
built-in's goes from 660 s to 900 s and a replacement's own ceiling
keeps its margin. A git read whose caller stops waiting kills its
process.
- **Warm-up as a session opens.** The first staging of a directory
hashes every file. `session.create` / `session.resume` call
`ExecTool.warm` for the directory the session resolves to, and a session
opened by its first message (a channel's) is warmed at the start of that
turn. `warm` returns at once and runs on the staging thread, so it
overlaps the model's first reply.
- **One rule for what text may be shown.** Every file a command created,
rewrote or removed, including the ones its own removal watch caught by
name, carries text only where the shadow repo's rules would store it
(`git check-ignore --no-index` against its excludes and the user's
`.gitignore`, the rules alone, not what the index holds). So a `.env` or
a key never enters the event or the stored session, even one the repo
staged before an ignore rule named it, and even when the staging failed:
the rules need no tree. Where no shadow repo can vouch for a file, a
created file carries its counts only.
- **Loop.** The exec-only listing branch in `turn_path.py` is gone; the
loop turns `result.written` into `file_written`. Besides that it has the
resolver above and the one-line warm-up at turn start. The registry now
keeps `removed` / `written` on a result it rewrites as an error (a
command whose output begins with "Error" still wrote what it wrote). A
removal whose body the rules kept back goes out marked `withheld` (a new
defaulted field on `FileRemoval`), and the turn's removal watch fills in
a bare removal from what the turn wrote only when it is not: with the
checkpoint off every bare removal stays fillable, as before, and a
withheld body is never put back.
- **Wire and page** (unchanged from #826). `file_written` entries gain
optional `added` / `removed` / `diff`; the diffs share a 512 KiB budget
per call, past which the counts still go. Contract updated in
`rpc-schema/openrpc.json`, `raven/rpc/models.py` and both generated
clients. ui-web draws these rows with their hunks, so they open in the
diff pane with +/- in the desk list, and `fromUnified` now skips only a
real `---`/`+++` header pair, so a removed `-- note` line is no longer
dropped.
- **Robustness.** Stagings run on daemon threads with blocking git calls
(an asyncio subprocess still starting when its loop closes hangs the
close on CPython 3.12 macOS). `add` and `write-tree` run under one lock
per index; the staging `add` has its own 600 s ceiling; the turn commit
waits for a warm-up's repo setup instead of racing it on the config
lock. The setup's future and every staging future are running from
creation, so a cancelled waiter cannot cancel them, and the warm-up and
staging threads settle them whatever happens, so no staging is left
published for nobody to finish (the cancellation 0xKT and gloryfromca
measured on #826). Git queries now encode paths with `surrogateescape`,
so a POSIX file name that is not UTF-8 no longer raises after the
command has run (the follow-up gloryfromca raised on #826).
Behaviour that moved with the measurement: staging now happens after the
permission gate has ruled, so a denied command is never staged; a
background command (`run_in_background`) takes no listing, since its
files land after the call returns; a sub-agent's `exec` does not list
(its runner lists every call itself, as before).
Not covered: files the shadow repo excludes (`*.log`, `.env`, the user's
`.gitignore`) get no diff, text files over 256 KB get no counts, a
same-name `exec` that is not an `ExecTool` (none ships) reports no
files, and existing sessions keep the rows they stored.
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
- `make test-python` -- 27077 passed, 108 skipped, 3 failed (on the
previous head; the focused set below is on this one). The 3 failures
(`test_rpc_files.py::test_a_host_without_libreoffice_says_so`,
`test_simulation_scenario.py::test_a_fresh_trial_refreshes_the_subagent_homes_before_it_runs`,
`test_simulation_suite.py::test_a_stopped_run_is_marked_in_its_record_and_finished_with_its_evidence`)
fail the same way on `github/main` on this machine.
- `uv run --frozen --all-extras pytest tests/test_runtime_checkpoint.py
tests/test_runtime_checkpoint_deep.py
tests/test_agent_loop_session_stamps.py tests/test_agent_loop_workdir.py
tests/test_rpc_session.py tests/test_shell_command_writes.py
tests/test_shell_file_removals.py tests/test_tool_registry_execute.py
tests/test_tool_registry_channels.py
tests/test_tool_registry_session_overlay.py tests/test_rpc_spine.py
tests/test_workdir_snapshot.py tests/test_shell_approval.py
tests/test_contracts_two_tier_ledger.py tests/test_kernel_budget.py
tests/test_removal_watch.py tests/test_agents_code_flow_exec.py
tests/test_agents_code_tools_plugin.py
tests/test_agents_code_launcher.py tests/test_subagent_dag_runner.py
tests/test_agents_research_ask_user_text.py tests/test_living_docs.py
tests/test_tracing_compact.py -q --idle-ceiling-strict` -- 1231 passed.
- `make lint-python lint-types lint-imports` -- clean (10 import
contracts kept). `npm run gen:check --prefix ui-web`, `npm run
type-check --prefix ui-web`, `npm run lint --prefix ui-web` (0 errors, 4
pre-existing warnings), `npm run lint:rpc --prefix ui-tui` -- clean.
`npm test --prefix ui-web` -- 3107 passed.
- Mutation checks, each restored afterwards and each turning at least
one test red (48): a staging timeout not falling back to running the
command without a diff; the wiring not asking the tool to record writes,
or not handing it the repo; the resolver dropping its `covers` check, or
ignoring the turn's binding; the registry dropping `written` on unwrap
or on an error result; the loop ignoring `written`; no turn-start
warm-up; `session.create` / `session.resume` not warming; a failing
warm-up failing the open; created-file text ignoring `trackable`; the
diff budget ignored; removals losing the staged text; staging a
directory too large to list; reading the written files on the event
loop; a named removal reported twice; the ceiling under the sum of the
wait, the command cap and the measuring bound; no bound on the shadow
repo's reads after the command, or on the whole post-command step; a
late read keeping a named removal's text; a cancelled git read left
running; the checkpoint's timeout not being a `TimeoutError`; `warm`
ignoring `record_writes`; a background command measured; the setup
future not marked running; the warm-up or a command's staging not
settling its future on an unexpected error; a rewritten or removed file
read without the rule; a named removal's text kept without the rule; the
repo dropped when the staging failed, or its rules applied only where
there is a tree; a tree-less staging still read for blobs; the removal
watch refilling a withheld body, or never refilling an unknown one; a
named, listed or timed-out removal not marked withheld where the repo
kept its body back, or a listed one marked where no repo ruled;
measuring switched on only for the built-in, so a plugin's same-name
`exec` reports nothing; `measure_writes` not raising the ceiling, or
raising it on every call; the built-in or the replacement measured only
after the registry admitted it, so the admitted ceiling stays at 660 s;
a failed staging not warned about, or warned about on every command.
- Real gateway (`raven serve` from this branch, isolated `RAVEN_HOME`,
real model), in a cold 12,775-file / 428 MB directory on a machine
running endpoint protection at load average 13-14. The first message
opened the session and the warm-up started with it; the model's first
`exec` (append to a `notes.md` Raven had never touched, and create
`made_by_exec.txt`) waited out the cold staging and took 37.8 s in
total. The second `exec` took 3.7 s, with the staging under the 1 s
sampling interval. The desk lists `M notes.md +2` and `A
made_by_exec.txt +3`; `notes.md` opens in the diff pane with both
commands' hunks, and a reload draws the same rows.
## Risk
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
User-visible changes:
- An `exec` now runs only after the working directory has been staged
(or the 120 s wait has passed). On a warm directory that is well under a
second; the first command in a large directory the shadow repo has never
indexed waits for the first staging (tens of seconds on a slow disk),
which the session-open warm-up usually hides. Past 120 s the command
runs without a diff.
- Opening a session starts one background `git add` per directory per
process, writing into the same `.raven/shadow.git` the checkpoint
already uses. Each process keeps its own `exec-<pid>.index` there; files
older than 7 days are pruned.
- No file content is stored beyond what the checkpoint already commits,
and the same excludes apply. With `runtime.checkpoint.policy = "never"`
a command's rewrite is reported bare, as today, and a created file
carries counts only.
- The new `FileWritten` fields are optional, so older clients ignore
them. `ToolResult` / `ToolOutput` gain a defaulted `written` field and
`FileRemoval` a defaulted `withheld`; contract version 34.
Rollback: revert the commit. The staging indexes in `.raven/shadow.git`
are inert without the code and are pruned by age.
## Related Issues
Supersedes #826
---------
Co-authored-by: arelchan <204152633+arelchan@users.noreply.github.com>
Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>