Raven-1/tests/test_shell_approval.py
KT 5854582b8e
feat(*): the files a main-conversation command wrote reach the desk diff (#692)
## Summary

The desk builds the main conversation's changed-files rows on the
client, from the file tools' own arguments and results. A file a shell
command created (`python3 build.py > out.txt`), rewrote (`echo more >>
notes.md`) or removed through a glob left no row, because a command
returns its output and names no file. #670 closed that gap for every
sub-agent lane by listing the working directory on either side of a
command; this does the same for the main conversation, which needs a
wire field because its rows are drawn on the page rather than read from
a run record.

- `raven/agent/loop/turn_path.py`: around a call named `exec` and only
that, the directory the command runs in is listed before and after the
call with `snapshot.take`, off the event loop. The tool says where that
is (`ExecTool.listing_root`, resolved the way `execute` resolves its
cwd): the turn's working directory unless the call names a
`working_dir`, and no listing at all for a command sent to a registered
`machine`, whose files are not on this disk. The in-process sub-agent
lane asks the same question through `snapshot.root_for`, so a
sub-agent's `working_dir` command is recorded too. Created and modified
paths the call did not name itself (its `file_change`, its removals)
become the new `tool.complete.file_written` list of `{path, created,
size, lines}`; a deletion the listing finds joins `file_removed` with no
body, since the file was gone before anything read it. Stored on the
tool entry under the same name and passed through `session.resume`, so a
reloaded page draws what the live one drew.
- Wire: `FileWritten` in `raven/rpc/models.py`, on `ToolCompletePayload`
and `TranscriptMessage`; `rpc-schema/openrpc.json` and both generated
clients regenerated. `lines` is an integer for a created text file of
256 KB or less and `null` otherwise, including every rewrite, whose old
contents the listing never held.
- Page: a created file is an `A` row with its line count, a rewritten
one a bare `M`, neither with a patch; a row a file tool already made
this turn is left alone (its arguments say more than a listing can); the
reload path replays the stored field. A change row with no patch to show
opens the file itself instead of an empty patch pane (a removal keeps
its pane; a task file's row, keyed by node rather than path, is
untouched). The offline fixture's GTM run gains one such command so the
demo desk shows both rows and the fixture-shape gate sees the field
sent.

Not in this PR: the ACP translator still surfaces no file changes to an
ACP client, the same as for `file_removed`.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run from the worktree on top of `0133c67c` (main with #656, #657 and
#670; the branch was rebased onto it when main's own contract additions
collided with the generated client, which was regenerated from the
merged schema):

- `.venv/bin/ruff check raven tests`; `.venv/bin/ruff format --check
raven tests` -- clean
- `.venv/bin/python -m pytest tests/test_agent_loop_session_stamps.py
tests/test_rpc_spine.py tests/test_rpc_session.py
tests/test_rpc_schema_match.py tests/test_turn_wire_shape_conformance.py
tests/test_spine_events.py tests/test_contracts_two_tier_ledger.py
tests/test_kernel_budget.py tests/test_workdir_snapshot.py
tests/test_subagent_activity.py tests/test_shell_file_removals.py
tests/test_acp_updates.py -q` -- 952 passed
- `make test-python` (`uv run --frozen --all-extras pytest -q`) -- 25798
passed, 109 skipped
- `npm test --prefix ui-web` (the whole suite, including the
fixture-shape gate) -- 200 files, 2767 tests passed; `npm run lint
--prefix ui-web` -- 0 errors (4 pre-existing hook-dependency warnings in
files this change does not touch); `npm run type-check --prefix ui-web`,
`npm run gen:check --prefix ui-web`, `npm run lint:rpc --prefix ui-tui`
-- clean
- Mutation checks, each restored afterwards: the listing taken around
every tool; a created file reported as a rewrite; the held-out result
path dropped; the storage rename disabled; `file_written` emitted when
empty; the key dropped from `session.resume`; `file_written=None` on the
served `ToolEvent`; a count of 0 on a decode failure; the 256 KB cap
dropped; the created row drawn as `M`; the existing-row guard dropped;
the ninth argument dropped in the stage; the file door removed, ignoring
the kind, ignoring a task key; the fixture no longer sending the field
(stored, then live); the row takeover removed and its removal guard
dropped; `.raven` back in the walk; the count back on the loop; the
shell ignoring its bound directory -- every one turned at least one new
test red.
- Review round two (reviewer's blocker): a command with a `working_dir`
of its own is listed there, in both lanes, and a `machine` command takes
no listing -- `tests/test_shell_approval.py` (the root follows the
arguments), `tests/test_workdir_snapshot.py` (`root_for`),
`tests/test_agent_loop_session_stamps.py` (the real `ExecTool` with
`working_dir`; a remote command takes no listing) and
`tests/test_subagent_manager.py` (a sub-agent command outside its
workspace); each red with the root pinned back to the lane's directory
or the machine guard dropped.
- Review round: one row per file when a file tool follows a command that
already left the file behind (the listing keys by the resolved path, the
tool by the path the model typed); the listing skips `.raven`, where the
checkpoint keeps a shadow git repo inside the working directory; the
line counting moved into the worker beside the walks; a test through the
real `ExecTool` (its cwd is the directory the listing walks), the served
`ToolEvent` hop, a non-text and an over-cap created file, and the live
fixture frame.
- Real gateway (`raven serve` from this branch, isolated `RAVEN_HOME`,
default model `deepseek/deepseek-v4-flash`): the main agent runs `printf
"gen one\ngen two\n" > work/main_generated.txt` and `echo more >>
work/main_shell_mod.md` beside its file-tool steps. Before this change
neither command left a row. After: the desk diff tab draws `A
work/main_generated.txt +2` and `M work/main_shell_mod.md` beside `M
work/main_keep.md +1-1` (edit_file), a red `D work/main_old.md -2`
(`rm`) and `A work/e2e_main_new.md +2` (write_file); a file one command
created and the next removed draws nothing; reopening the page
(`session.resume`) draws the identical five rows, and the stored tool
entries carry `file_written` with `created: true, lines: 2` and
`created: false, lines: null`. The same run repeated on the final tree
(after the review round) and its reload draw the same rows.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

Behaviour change: every main-conversation `exec` now walks the turn's
working directory twice, bounded at 20000 entries with the machinery
directories (`.git`, `node_modules`, `.venv`, caches) skipped and run
off the event loop; a tree past the bound records nothing from listings,
and the file tools' own rows stand as before. A created file's line
count reads the file (up to 256 KB). Wire: one new optional key on
`tool.complete` and on `session.resume` tool entries, absent on every
call that is not a command; a client that does not read it sees what it
saw before. Rollback is reverting the commits.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-23 13:35:55 +08:00

977 lines
43 KiB
Python

"""ShellCommandPolicy: safe, hard-denied, and approval-required command decisions."""
from __future__ import annotations
from pathlib import Path
import pytest
from raven.agent.tools.registry import ToolRegistry
from raven.agent.tools.shell import ExecTool
from raven.config.schema import PermissionsConfig
from raven.contracts.permissions import ApprovalChoice, ApprovalOutcome
from raven.contracts.tool import Continuation
from raven.permissions.builtin import BuiltinRulings
from raven.permissions.gate import PermissionGate
from raven.permissions.shell_policy import CommandDecision, ShellCommandPolicy
from raven.permissions.turn import start_permission_turn
from raven.sandbox import ExecResult, SandboxExecutor
@pytest.fixture
def policy() -> ShellCommandPolicy:
# Mirrors the shipped default in that `rm` is classified from tokens, not
# matched here -- a fixture that still pattern-denies it would test a
# policy no user runs.
return ShellCommandPolicy(deny_patterns=[r"\b(mkfs|diskpart)\b"])
@pytest.fixture
def asking_policy() -> ShellCommandPolicy:
"""The ACP editor's shape: the deletion family declared, so deletes ask.
The terminal ships no asking families -- deletes answer to the tiers -- but
the matcher's reach still matters wherever a surface declares it.
"""
from raven.permissions.shell_policy import DELETE_MATCHERS
policy = ShellCommandPolicy(deny_patterns=[r"\b(mkfs|diskpart)\b"])
for name, matcher in DELETE_MATCHERS:
policy.register_approval_matcher(name, matcher)
return policy
@pytest.mark.parametrize(
"command",
[
"pwd",
"ls -la",
"echo hello",
"find . -name '*.py'",
"printf 'rm file.txt'",
"echo 'find . -delete'",
"git grep -n shutdown",
"grep -rn reboot /var/log",
"man shutdown",
"systemctl show reboot.target",
"grep -rn 'systemctl poweroff' docs/",
"bash -lc 'ls'",
],
)
def test_safe_commands_are_allowed(policy: ShellCommandPolicy, command: str) -> None:
assert policy.evaluate(command) is CommandDecision.ALLOW
@pytest.mark.parametrize(
"command",
[
"rm -rf /",
"rm -rf /*",
"rm -rf ~",
"rm -r $HOME",
"echo ready && rm -rf /",
"mkfs /dev/test",
"shutdown now",
"halt",
"sudo -n reboot",
'bash -c "poweroff"',
"systemctl poweroff",
"systemctl reboot",
"sudo systemctl reboot",
"busybox poweroff",
"loginctl poweroff",
"systemctl -i poweroff",
"init 0",
"init 6",
"telinit 0",
"telinit 6",
],
)
def test_hard_denied_commands_cannot_be_approved(policy: ShellCommandPolicy, command: str) -> None:
assert policy.evaluate(command) is CommandDecision.HARD_DENY
@pytest.mark.parametrize(
"command",
[
"rm file.txt",
"rm -f file.txt",
"rm file1 file2",
"/bin/rm file.txt",
"rm --force file.txt",
"sudo rm file.txt",
"sudo -u root rm file.txt",
"sudo --user=root unlink file.txt",
"command unlink file.txt",
"MODE=test rm file.txt",
"env MODE=test rm file.txt",
"env -u MODE rm file.txt",
"unlink file.txt",
"find ./tmp -delete",
"echo ready && rm file.txt",
"printf done | unlink file.txt",
"cd /tmp\nrm file.txt",
"cd /tmp\r\nunlink file.txt",
"(rm file.txt)",
"{ rm file.txt; }",
"echo $(rm file.txt)",
"echo `rm file.txt`",
"nohup rm file.txt &",
'bash -c "rm file.txt"',
"bash -c'rm file.txt'",
'bash -c"rm file.txt"',
'bash --rcfile setup.sh -c "rm file.txt"',
'sh -lc "find tmp -delete"',
'find . -name "*.log" -exec rm {} \\;',
'find . -name "*.log" -execdir unlink {} \\;',
'find . -exec sh -c "rm \\"$1\\"" _ {} \\;',
],
)
def test_delete_commands_require_approval(asking_policy: ShellCommandPolicy, command: str) -> None:
assert asking_policy.evaluate(command) is CommandDecision.REQUIRE_APPROVAL
def test_ordinary_deletes_answer_to_the_tiers_by_default(policy: ShellCommandPolicy) -> None:
"""The shipped terminal policy declares no asking family: an ordinary
delete is a mutation like any other, decided by the permission tiers."""
assert policy.evaluate("rm file.txt") is CommandDecision.ALLOW
assert policy.evaluate("rm -rf build/") is CommandDecision.ALLOW
def test_hard_deny_wins_when_command_also_matches_approval(asking_policy: ShellCommandPolicy) -> None:
assert asking_policy.evaluate("unlink old.txt && rm -rf /") is CommandDecision.HARD_DENY
def test_matcher_failure_is_fail_closed(policy: ShellCommandPolicy) -> None:
def broken_matcher(command: str) -> bool:
raise RuntimeError("broken")
policy.register_approval_matcher("broken", broken_matcher)
assert policy.evaluate("echo harmless") is CommandDecision.HARD_DENY
@pytest.mark.parametrize("command", ["echo 'unterminated", "echo trailing\\"])
def test_shell_parse_failure_is_fail_closed(policy: ShellCommandPolicy, command: str) -> None:
assert policy.evaluate(command) is CommandDecision.HARD_DENY
class _RecordingExecutor(SandboxExecutor):
def __init__(self, *, sandboxed: bool) -> None:
self._sandboxed = sandboxed
self.commands: list[str] = []
@property
def is_sandboxed(self) -> bool:
return self._sandboxed
async def exec(self, command: str, **kwargs) -> ExecResult:
self.commands.append(command)
return ExecResult(stdout="ok", stderr="", exit_code=0)
class _ApprovalResponder:
def __init__(self, answers: list[bool]) -> None:
self.answers = answers
self.requests: list[dict] = []
async def await_approval(self, **request) -> ApprovalOutcome:
self.requests.append(request)
allowed = self.answers.pop(0)
return ApprovalOutcome(choice=ApprovalChoice.ALLOW if allowed else ApprovalChoice.DENY)
def _gated_exec(executor, responder, tmp_path, *, mode: str = "ask", families=()):
"""The real dispatch path: gate at the registry door, ExecTool behind it."""
builtin = BuiltinRulings()
for name, matcher in families:
builtin._policy.register_approval_matcher(name, matcher)
gate = PermissionGate(
config_source=lambda: PermissionsConfig(mode=mode),
builtin=builtin,
allow_ask=True,
)
registry = ToolRegistry(permission_gate=gate)
registry.register(ExecTool(executor=executor, working_dir=str(tmp_path)))
start_permission_turn(responder, conversation_id="session-a", turn_id="turn-a")
return registry
async def test_direct_delete_executes_once_after_approval(tmp_path) -> None:
executor = _RecordingExecutor(sandboxed=False)
responder = _ApprovalResponder([True])
registry = _gated_exec(executor, responder, tmp_path)
result = await registry.execute("exec", {"command": "rm file.txt"})
assert "Exit code: 0" in result
assert executor.commands == ["rm file.txt"]
assert responder.requests == [
{
"conversation_id": "session-a",
"turn_id": "turn-a",
"tool_call_id": "",
"command": "rm file.txt",
"description": "Approve this action: rm file.txt",
"suggested_pattern": "",
"kind": "shell.exec",
"family": "",
"origin": "",
"origin_name": "",
"evidence": {"command": "rm file.txt", "cwd": str(tmp_path)},
}
]
async def test_direct_delete_without_responder_is_denied(tmp_path) -> None:
executor = _RecordingExecutor(sandboxed=False)
registry = _gated_exec(executor, None, tmp_path)
result = await registry.execute("exec", {"command": "unlink file.txt"})
assert result.retryable is False
assert result.blocks_call is True
assert result.continuation is Continuation.CONTINUE
assert "requires user approval" in str(result)
assert "not interactive" in str(result)
assert "Do not retry" in str(result)
assert executor.commands == []
async def test_denied_command_is_not_prompted_again_in_same_turn(tmp_path) -> None:
executor = _RecordingExecutor(sandboxed=False)
responder = _ApprovalResponder([False])
registry = _gated_exec(executor, responder, tmp_path)
first = await registry.execute("exec", {"command": "find tmp -delete"})
second = await registry.execute("exec", {"command": "find tmp -delete"})
assert first.retryable is False
assert first.blocks_call is True
assert first.continuation is Continuation.CONTINUE
assert "denied" in str(first).lower()
assert "denied" in str(second).lower()
assert executor.commands == []
assert len(responder.requests) == 1
async def test_allow_once_does_not_cover_a_second_execution(tmp_path) -> None:
executor = _RecordingExecutor(sandboxed=False)
responder = _ApprovalResponder([True, False])
registry = _gated_exec(executor, responder, tmp_path)
first = await registry.execute("exec", {"command": "rm file.txt"})
second = await registry.execute("exec", {"command": "rm file.txt"})
assert "Exit code: 0" in first
assert "denied" in str(second).lower()
assert executor.commands == ["rm file.txt"]
assert len(responder.requests) == 2
async def test_new_turn_can_prompt_for_a_previously_denied_command(tmp_path) -> None:
executor = _RecordingExecutor(sandboxed=False)
responder = _ApprovalResponder([False, True])
registry = _gated_exec(executor, responder, tmp_path)
first = await registry.execute("exec", {"command": "unlink file.txt"})
start_permission_turn(responder, conversation_id="session-a", turn_id="turn-b")
second = await registry.execute("exec", {"command": "unlink file.txt"})
assert "denied" in str(first).lower()
assert "Exit code: 0" in second
assert executor.commands == ["unlink file.txt"]
assert len(responder.requests) == 2
async def test_hard_denied_command_never_requests_approval(tmp_path) -> None:
executor = _RecordingExecutor(sandboxed=False)
responder = _ApprovalResponder([True])
registry = _gated_exec(executor, responder, tmp_path)
result = await registry.execute("exec", {"command": "rm -rf /"})
assert result.retryable is False
assert result.blocks_call is True
assert result.continuation is Continuation.CONTINUE
assert "blocked" in str(result)
assert "Do not retry" in str(result)
assert responder.requests == []
assert executor.commands == []
async def test_a_sandboxed_executor_earns_no_relaxation(tmp_path) -> None:
# Boxlite mounts the real workspace read-write into the VM, so a
# filesystem catastrophe inside it reaches host data. The deny list
# holds through a sandboxed executor exactly as through a direct one.
executor = _RecordingExecutor(sandboxed=True)
responder = _ApprovalResponder([False])
registry = _gated_exec(executor, responder, tmp_path, mode="full")
result = await registry.execute("exec", {"command": "rm -rf /"})
assert result.blocks_call is True
assert "blocked" in str(result)
assert responder.requests == []
assert executor.commands == []
class TestTheTargetIsWhatMakesADeleteUnconditional:
"""Only a recursive delete aimed at ``/`` or the home tree is unconditional.
Everything the machine holds lives under those two paths, so no mode, rule
or click can rescue them. An ordinary recursive delete (``rm -rf build/``)
is daily work: it answers to the tiers, and to the deletion family on a
surface that declares one.
"""
@pytest.mark.parametrize(
"command",
[
"rm -f parse_turns.py turns.json stats.json",
"rm scratch.txt",
"rm -f build.py app_css.css",
"cd /work && rm -f a.py",
],
)
def test_naming_the_files_asks_on_a_declaring_surface(
self, asking_policy: ShellCommandPolicy, command: str
) -> None:
assert asking_policy.evaluate(command) is CommandDecision.REQUIRE_APPROVAL
@pytest.mark.parametrize(
"command",
[
"sudo rm -rf /",
"bash -c 'rm -rf /'",
"rm -fr /",
"rm -R ~/",
"rm --recursive ~",
# A separated recursive flag must not walk past the check.
"rm -f -r /",
],
)
def test_catastrophic_targets_stay_unconditional(self, policy: ShellCommandPolicy, command: str) -> None:
assert policy.evaluate(command) is CommandDecision.HARD_DENY
@pytest.mark.parametrize(
"command",
["rm -rf build", "rm -fr build", "rm -R build", "rm --recursive build", "rm -f -r build"],
)
def test_an_ordinary_recursive_delete_answers_to_the_tiers(self, policy: ShellCommandPolicy, command: str) -> None:
assert policy.evaluate(command) is CommandDecision.ALLOW
def test_a_recursive_delete_quoted_inside_another_command_is_still_text(self, policy: ShellCommandPolicy) -> None:
"""The old regexp searched the raw string, so mentioning the command
was as forbidden as running it."""
assert policy.evaluate("echo rm -rf nope") is CommandDecision.ALLOW
@pytest.mark.parametrize(
"command",
[
"timeout 5 rm -rf /",
"timeout -s KILL 5 rm -rf /",
"timeout --kill-after=2 5s rm -rf ~",
"nice rm -rf /",
"nice -n 10 rm -rf /",
"ionice -c 3 rm -rf ~",
"time rm -rf /",
"setsid rm -rf /",
"stdbuf -oL rm -rf /*",
"sudo timeout 5 rm -rf /",
"env FOO=1 timeout 5 rm -rf /",
"xargs sh -c 'rm -rf /'",
"find / -name '*' -exec rm -rf {} +",
],
)
def test_a_wrapper_does_not_launder_a_catastrophic_delete(self, policy: ShellCommandPolicy, command: str) -> None:
"""A program that runs another program must not hide the delete behind
itself: the wrapper stripping reads through to the target."""
assert policy.evaluate(command) is CommandDecision.HARD_DENY
@pytest.mark.parametrize(
"command",
["xargs rm -rf < list.txt", "git ls-files -o | xargs rm -rf", "xargs -0 -n1 rm -rf", "xargs -I{} rm -rf {}"],
)
def test_stdin_fed_targets_are_past_the_tripwire(self, policy: ShellCommandPolicy, command: str) -> None:
"""The unconditional list is a lexical tripwire, not a boundary: targets
arriving on stdin are invisible to it, exactly like a delete written in
Python. Those runs answer to the tiers (and the sandbox), which is the
actual boundary."""
assert policy.evaluate(command) is CommandDecision.ALLOW
@pytest.mark.parametrize(
"command",
[
"xargs rm -f a.py",
"timeout 5 rm a.py",
],
)
def test_a_wrapped_named_delete_still_only_asks(self, asking_policy: ShellCommandPolicy, command: str) -> None:
assert asking_policy.evaluate(command) is CommandDecision.REQUIRE_APPROVAL
@pytest.mark.parametrize(
"command",
[
# The search path is benign, so path inspection alone would clear it;
# the catastrophic target rides the -exec command, which has to be
# read as its own argv.
"find . -exec rm -rf --no-preserve-root / {} +",
"find . -type f -execdir rm -rf / {} ;",
'find . -exec sh -c "rm -rf /" {} +',
# The delete rides a shell the carried command spawns: what find
# walks under / is deleted all the same, one wrapper deeper.
"find / -exec sh -c 'rm -rf \"$1\"' sh {} +",
# Or a recognized command runner: timeout/nice hand the walk on.
"find / -exec timeout 5 rm -rf --no-preserve-root {} +",
"find / -execdir nice -n 5 rm -rf {} +",
"find . -exec timeout 5 rm -rf / {} +",
# Multiple search roots: the catastrophic one is not first, so
# reading only paths[0] would clear it.
"find /home / -delete",
"find /tmp/scratch ~ -exec rm {} +",
],
)
def test_a_find_exec_carrying_a_root_delete_is_unconditional(
self, policy: ShellCommandPolicy, command: str
) -> None:
assert policy.evaluate(command) is CommandDecision.HARD_DENY
@pytest.mark.parametrize(
"command",
[
# -exec of an ordinary delete: the carried command targets what find
# matched under a benign path, not the root, so it is daily work.
'find . -name "*.log" -exec rm -rf {} +',
"find build -execdir rm {} ;",
# / here is -path's argument, not a search root: the walk starts
# at . and deletes nothing outside it.
"find . -path / -delete",
],
)
def test_an_ordinary_find_exec_answers_to_the_tiers(self, policy: ShellCommandPolicy, command: str) -> None:
assert policy.evaluate(command) is CommandDecision.ALLOW
@pytest.mark.parametrize(
"command",
[
"rmdir /s C:\\",
"rd /s /q C:\\",
"del /f /s /q C:\\*",
"rmdir /s %USERPROFILE%",
"del /s C:",
"cmd /c rmdir /s C:\\",
"cmd.exe /d /c del /f /s /q C:\\*",
'cmd /c "rmdir /s %USERPROFILE%"',
"echo done\nrmdir /s %USERPROFILE%",
" rmdir /s %USERPROFILE%",
# Quoting the target is ordinary cmd syntax, not a disguise.
'rmdir /s "%USERPROFILE%"',
'del /f /s /q "C:\\*"',
],
)
def test_a_recursive_windows_delete_of_a_root_is_unconditional(
self, policy: ShellCommandPolicy, command: str
) -> None:
assert policy.evaluate(command) is CommandDecision.HARD_DENY
@pytest.mark.parametrize(
"command",
[
# Ordinary Windows cleanup: no /s, or /s over a named subdir. The
# Unix parallel is rm -f a.txt and rm -rf build -- both go to tiers.
"del /f scratch.txt",
"del /q old.log",
"rmdir /s build",
"rmdir /s C:\\Users\\me\\project",
"del /f /q C:\\proj\\build",
# cmd in argument position is text, not a wrapper, and a quoted
# ampersand separates nothing.
"echo cmd /c rmdir /s C:",
'echo "safe & rmdir /s C: "',
# A quoted named subdir keeps its backslashes and stays ordinary.
'rmdir /s "C:\\Users\\me\\my project"',
],
)
def test_ordinary_windows_deletes_answer_to_the_tiers(self, policy: ShellCommandPolicy, command: str) -> None:
assert policy.evaluate(command) is CommandDecision.ALLOW
@pytest.mark.parametrize(
"command",
[
# `rm` appears, but never in the command position -- reading the
# whole argv for the word would refuse all three, and a hard deny
# is the one decision no approval can rescue.
"timeout 5 grep -r rm .",
"xargs grep -r foo",
"nice -n 5 pytest -q",
],
)
def test_a_wrapper_does_not_invent_a_delete(self, policy: ShellCommandPolicy, command: str) -> None:
assert policy.evaluate(command) is CommandDecision.ALLOW
class TestExternalEffectFamilies:
"""The opt-in group, and where its line is drawn.
The built-in policy asks about exactly one family -- deletion -- which fits a
terminal the reader is already watching. Behind an editor nothing is on
screen, so ``git push``, ``npm install`` and ``curl -o`` would run unannounced.
The group registered by ``raven acp`` closes that, and the tests below are as
much about what it does *not* ask for: a prompt on every build and every
documentation fetch trains the reader to approve without looking, which costs
more than it buys.
"""
@pytest.fixture
def asking(self) -> ShellCommandPolicy:
from raven.permissions.shell_policy import DELETE_MATCHERS, EXTERNAL_EFFECT_MATCHERS
policy = ShellCommandPolicy(deny_patterns=[])
for name, matcher in DELETE_MATCHERS:
policy.register_approval_matcher(name, matcher)
for name, matcher in EXTERNAL_EFFECT_MATCHERS:
policy.register_approval_matcher(name, matcher)
return policy
@pytest.mark.parametrize(
("command", "family"),
[
("git push origin main", "publish_command"),
("gh pr create --fill", "publish_command"),
("gh pr merge 3 --squash", "publish_command"),
("glab mr merge 5", "publish_command"),
# Naming an organisation's secrets is itself worth asking about, so
# the read-verb relief does not reach this group.
("gh secret list", "publish_command"),
# And the word after a publishing verb is usually an operand, not a
# verb: these push an image called status, a branch to a remote
# called view, a package, a deletion. The relief is scoped to the
# forge groups so it cannot reach any of them.
("git push status", "publish_command"),
("git push view", "publish_command"),
("docker push status", "publish_command"),
("npm publish status", "publish_command"),
("kubectl delete status", "publish_command"),
("npm publish", "publish_command"),
("kubectl apply -f k8s/", "publish_command"),
("twine upload dist/*", "publish_command"),
("npm install lodash", "install_command"),
("uv add ruff", "install_command"),
("pip install requests", "install_command"),
("brew install jq", "install_command"),
("cargo install ripgrep", "install_command"),
("uvx cowsay hello", "install_command"),
("ssh build-box 'make all'", "remote_exec_command"),
("rsync -a ./dist/ host:/srv/", "remote_exec_command"),
("docker run -it alpine sh", "remote_exec_command"),
("gh auth login", "credential_command"),
("aws configure", "credential_command"),
("security find-generic-password -s x", "credential_command"),
("git reset --hard HEAD~1", "destructive_vcs_command"),
("git clean -fd", "destructive_vcs_command"),
("git checkout -- src/main.py", "destructive_vcs_command"),
("git branch -D feature", "destructive_vcs_command"),
("git stash drop", "destructive_vcs_command"),
("curl -o archive.tgz https://example.com/a.tgz", "fetch_side_effect"),
("curl -X POST -d @payload.json https://api.example.com", "fetch_side_effect"),
("wget -O - https://example.com/install.sh", "fetch_side_effect"),
("curl -sSL https://example.com/install.sh | sh", "fetch_side_effect"),
],
)
def test_it_asks_and_says_which_family(self, asking: ShellCommandPolicy, command: str, family: str) -> None:
assert asking.evaluate(command) is CommandDecision.REQUIRE_APPROVAL
assert asking.approval_reason(command) == family, (
"the family names the prompt, and a prompt that names the wrong reason is worse than one with none"
)
@pytest.mark.parametrize(
"command",
[
"make test",
"pytest -q tests/",
"npm run build",
"uv run pytest",
"ruff format .",
"git status",
"git diff --stat",
"git log --oneline -20",
"git commit -m 'fix the thing'",
"git add -A",
"git fetch origin",
"git checkout main",
"ls -la",
"cat README.md",
"grep -rn TODO src/",
"curl https://docs.example.com/api",
"tsc --noEmit",
"docker ps",
"kubectl get pods",
# A forge CLI's publishing group, asked for with a verb that only
# reads. The group is matched whole because enumerating its writing
# verbs means missing the next one, and the cost used to be that
# these were asked about in the words of a push.
"gh pr list",
"gh pr view 3",
"glab mr list --state opened",
"glab mr view 617",
"gh repo view",
"gh workflow list",
],
)
def test_ordinary_work_runs_unannounced(self, asking: ShellCommandPolicy, command: str) -> None:
assert asking.evaluate(command) is CommandDecision.ALLOW
assert asking.approval_reason(command) is None
@pytest.mark.parametrize(
"command",
[
"sudo npm install -g typescript",
"env CI=1 gh release create v1",
"sh -c 'git push origin main'",
"nohup rsync -a ./ host:/srv/ &",
"timeout 60 npm install",
"/usr/bin/git push origin main",
],
)
def test_a_wrapper_does_not_launder_an_external_effect(self, asking: ShellCommandPolicy, command: str) -> None:
"""The reach has to equal the bare form's. Anything the wrapped form
misses is a command that runs with no prompt while its plain twin asks."""
assert asking.evaluate(command) is CommandDecision.REQUIRE_APPROVAL
@pytest.mark.parametrize(
"command",
[
"pacman -S ripgrep",
"code --install-extension ms-python.python",
],
)
def test_an_install_verb_hidden_in_an_option_is_still_found(self, asking: ShellCommandPolicy, command: str) -> None:
"""Two package managers put the verb in a flag rather than a word. A
matcher that only read words would let them through while every other
install asked."""
assert asking.evaluate(command) is CommandDecision.REQUIRE_APPROVAL
assert asking.approval_reason(command) == "install_command"
def test_a_git_subcommand_destructive_with_no_flag_at_all(self, asking: ShellCommandPolicy) -> None:
"""``filter-branch`` rewrites history unconditionally -- no flag makes it
safe, so the family carries no flag list for it and the bare form fires."""
assert asking.evaluate("git filter-branch --msg-filter cat") is CommandDecision.REQUIRE_APPROVAL
assert asking.approval_reason("git filter-branch --msg-filter cat") == "destructive_vcs_command"
def test_a_destructive_flag_without_its_subcommand_does_not_fire(self, asking: ShellCommandPolicy) -> None:
"""``--hard`` belongs to ``reset``. Matching the flag alone would ask about
anything else that happens to carry it."""
assert asking.evaluate("git log --hard") is CommandDecision.ALLOW
assert asking.evaluate("git status -f") is CommandDecision.ALLOW
@pytest.mark.parametrize("command", ["rm -rf /", "shutdown -h now", "reboot"])
def test_a_token_classified_hard_deny_has_no_reason_to_explain(
self, asking: ShellCommandPolicy, command: str
) -> None:
"""The other hard-deny path: these are refused by token inspection rather
than by a deny pattern, and a refusal has no prompt to describe."""
assert asking.evaluate(command) is CommandDecision.HARD_DENY
assert asking.approval_reason(command) is None
def test_an_empty_segment_does_not_break_the_walk(self, asking: ShellCommandPolicy) -> None:
"""A wrapper with nothing after it, and an empty compound segment. Both
occur in real command strings and neither names an executable."""
assert asking.evaluate("sudo") is CommandDecision.ALLOW
assert asking.evaluate("env") is CommandDecision.ALLOW
assert asking.evaluate(";; git push") is CommandDecision.REQUIRE_APPROVAL
def test_a_command_inside_a_nested_shell_is_still_classified(self, asking: ShellCommandPolicy) -> None:
assert asking.evaluate("""sh -c "sh -c 'git push'" """) is CommandDecision.REQUIRE_APPROVAL
def test_unparseable_quoting_closes_the_gate(self, asking: ShellCommandPolicy) -> None:
"""``shlex`` raises "No closing quotation" on an unbalanced quote, which
reaches the policy's fail-closed branch. Refusing is the right direction:
a command string the classifier cannot read is one whose effect it cannot
bound, and the alternative is running it unexamined."""
assert asking.evaluate("git status 'unbalanced") is CommandDecision.HARD_DENY
assert asking.approval_reason("git status 'unbalanced") is None
def test_the_walk_stops_at_a_fixed_depth(self) -> None:
"""Called directly with the depth already at the bound, because reaching
it through real shell quoting takes five alternating quote levels that no
command has. What the bound buys is termination: without it a crafted
string could recurse until the stack ran out."""
from raven.permissions.shell_policy import _MAX_EMBEDDED_SHELL_DEPTH, _iter_argv
shallow = list(_iter_argv("sh -c 'git push'"))
at_bound = list(_iter_argv("sh -c 'git push'", _depth=_MAX_EMBEDDED_SHELL_DEPTH))
assert ["git", "push"] in shallow, "the inner command is reached below the bound"
assert ["git", "push"] not in at_bound, "and not descended into at it"
assert at_bound == [["sh", "-c", "git push"]], "the outer argv is still yielded"
def test_hard_deny_still_outranks_the_new_families(
self,
) -> None:
"""Ordering is security-sensitive: a matcher must never turn an
unconditionally forbidden command into an approvable one."""
from raven.permissions.shell_policy import EXTERNAL_EFFECT_MATCHERS
policy = ShellCommandPolicy(deny_patterns=[r"\bmkfs\b"])
for name, matcher in EXTERNAL_EFFECT_MATCHERS:
policy.register_approval_matcher(name, matcher)
assert policy.evaluate("mkfs.ext4 /dev/sda1 && git push") is CommandDecision.HARD_DENY
assert policy.approval_reason("mkfs.ext4 /dev/sda1") is None, "a refusal has no prompt to explain"
assert policy.evaluate("rm -rf / && npm publish") is CommandDecision.HARD_DENY
def test_the_default_policy_asks_about_none_of_them(self, policy: ShellCommandPolicy) -> None:
"""The group is opt-in. A terminal user watching their own shell does not
need a prompt before ``git push``, and adding one would change behaviour
for every existing surface."""
for command in ("git push origin main", "npm install lodash", "ssh box ls"):
assert policy.evaluate(command) is CommandDecision.ALLOW
def test_a_faulty_matcher_closes_the_gate_and_explains_nothing(self, policy: ShellCommandPolicy) -> None:
def _broken(command: str) -> bool:
raise RuntimeError("matcher is wrong")
policy.register_approval_matcher("broken", _broken)
assert policy.evaluate("echo hi") is CommandDecision.HARD_DENY
assert policy.approval_reason("echo hi") is None
async def test_the_prompt_names_the_family_that_fired(tmp_path) -> None:
"""The description was a constant before the families existed -- it read
"Delete files using a shell command" for whatever was being asked about,
which was accurate only while deletion was the one registered family."""
from raven.permissions.shell_policy import EXTERNAL_EFFECT_MATCHERS
executor = _RecordingExecutor(sandboxed=False)
responder = _ApprovalResponder([True])
registry = _gated_exec(executor, responder, tmp_path, families=EXTERNAL_EFFECT_MATCHERS)
await registry.execute("exec", {"command": "git push origin main"})
assert responder.requests[0]["description"] == "Publish or push work to a remote"
assert executor.commands == ["git push origin main"]
async def test_an_unregistered_family_still_gets_a_usable_prompt(tmp_path) -> None:
"""A surface can register a matcher this table has no description for. The
fallback is deliberately vague rather than a guess: naming the wrong reason
is worse than naming none."""
executor = _RecordingExecutor(sandboxed=False)
responder = _ApprovalResponder([False])
families = (("house_style", lambda command: command.startswith("weird")),)
registry = _gated_exec(executor, responder, tmp_path, families=families)
result = await registry.execute("exec", {"command": "weird --thing"})
assert responder.requests[0]["description"] == "Run a command that needs your approval"
assert "denied" in str(result).lower()
assert executor.commands == []
class TestAGlobalOptionValueIsNotASubcommand:
"""The gap that let a hand-written command through the boundary added here.
``_subcommands`` skipped options but not their values, so a value was counted
as one of the words it was looking for and the budget ran out before the
verb. ``git --git-dir X --work-tree Y push`` therefore read as the two paths
and never saw ``push``: an ALLOW for the exact command that bare ``git push``
prompts about. Nothing below is adversarial -- every shape is one a person
types, and two of them (``aws --profile``, ``git --git-dir``) are the normal
way to drive those tools from outside their own tree.
"""
@pytest.fixture
def asking(self) -> ShellCommandPolicy:
from raven.permissions.shell_policy import DELETE_MATCHERS, EXTERNAL_EFFECT_MATCHERS
policy = ShellCommandPolicy(deny_patterns=[])
for name, matcher in DELETE_MATCHERS:
policy.register_approval_matcher(name, matcher)
for name, matcher in EXTERNAL_EFFECT_MATCHERS:
policy.register_approval_matcher(name, matcher)
return policy
@pytest.mark.parametrize(
("command", "family"),
[
("git --git-dir /tmp/repo/.git --work-tree /tmp/repo push origin main", "publish_command"),
("git -C /repo --no-pager push", "publish_command"),
("aws --profile prod --region us-east-1 s3 cp ./x s3://bucket/x", "publish_command"),
("kubectl --namespace kube-system --context prod apply -f x.yaml", "publish_command"),
("gh --repo owner/name pr create --fill", "publish_command"),
("docker --host tcp://build:2375 push registry/image", "publish_command"),
("npm --prefix /srv/app install lodash", "install_command"),
("git --git-dir /tmp/r/.git reset --hard HEAD~1", "destructive_vcs_command"),
],
)
def test_the_verb_is_found_past_its_global_options(
self, asking: ShellCommandPolicy, command: str, family: str
) -> None:
assert asking.evaluate(command) is CommandDecision.REQUIRE_APPROVAL
assert asking.approval_reason(command) == family
def test_an_attached_value_consumes_nothing_extra(self, asking: ShellCommandPolicy) -> None:
"""``--git-dir=X`` carries its value in the same word. Consuming a
following token for it would eat the verb instead of the value, which is
the same bug pointing the other way."""
assert asking.approval_reason("git --git-dir=/tmp/r/.git push") == "publish_command"
def test_a_double_dash_ends_the_options(self, asking: ShellCommandPolicy) -> None:
"""After ``--`` a word that looks like an option is an argument, so the
table must not keep consuming past it."""
assert asking.approval_reason("git -C /repo push -- --not-an-option") == "publish_command"
@pytest.mark.parametrize(
"command",
[
# The two options here are documented AWS globals that a table of
# "options certain to take a value" did not happen to list. Review
# found this shape against the table version, and it is the reason
# the word budget is gone rather than the table extended: no table of
# every option of every tool can be complete, so nothing that decides
# whether to ask may depend on one being complete.
"aws --query '{}' --cli-binary-format raw-in-base64-out s3 cp ./x s3://bucket/x",
"aws --profile p --region r --output json --query x s3 cp a b",
"kubectl -n ns --context c --kubeconfig k --as u apply -f x.yaml",
"git -c a.b=c -c d.e=f --git-dir /r/.git --work-tree /r push origin main",
],
)
def test_no_number_of_option_values_can_hide_the_verb(self, asking: ShellCommandPolicy, command: str) -> None:
assert asking.evaluate(command) is CommandDecision.REQUIRE_APPROVAL
assert asking.approval_reason(command) is not None
def test_a_quoted_argument_of_metacharacters_is_not_a_command_boundary(self, asking: ShellCommandPolicy) -> None:
"""Found while fixing the case above, and it is the more serious half.
The segmenter decided a token was a boundary when every character in it
was a shell metacharacter, which a *quoted* argument can satisfy.
``aws --query '{}' ... s3 cp`` was therefore cut in two at its own
argument, and the second piece began with an option -- so ``argv[0]``,
which every family matcher keys on to find its table, was an option
rather than an executable, and no family could fire at all. Extending the
option table would not have touched this: the command never reached the
table as one command.
"""
from raven.permissions.shell_policy import _iter_argv
argvs = list(_iter_argv("aws --query '{}' --cli-binary-format raw s3 cp ./x s3://b/x"))
assert len(argvs) == 1, f"one command, not {len(argvs)}: {argvs}"
assert argvs[0][0] == "aws", "the executable has to survive segmentation"
@pytest.mark.parametrize(
("command", "decision"),
[
# The shapes that rely on a bare ``{}`` being an ordinary word, and
# the real operators that must still split. Both directions, because
# the fix moves the line between them.
("{ rm file.txt; }", CommandDecision.REQUIRE_APPROVAL),
("xargs -I{} rm -rf {}", CommandDecision.REQUIRE_APPROVAL),
(r'find . -name "*.log" -exec rm {} \;', CommandDecision.REQUIRE_APPROVAL),
("rm -rf / && git push", CommandDecision.HARD_DENY),
],
)
def test_the_operators_that_must_still_split_still_split(
self, asking: ShellCommandPolicy, command: str, decision: CommandDecision
) -> None:
assert asking.evaluate(command) is decision
@pytest.mark.parametrize(
"command",
[
# A repository directory literally named ``|``. Review initialized one
# and confirmed ``git -C '|' status`` works, so this is a command a
# person can run, not a contrivance. Under the whole-token rule this
# segmented as [['git', '-C'], ['push', 'origin', 'main']]: the quoted
# argument read as a pipeline operator, and the piece holding ``push``
# started at a word that is not an executable.
"git -C '|' push origin main",
# zsh accepts ``|&`` and runs the right-hand side as the pipeline. It
# was on no operator list, so the whole thing stayed one token inside
# the ``-c`` string and nothing looked past it.
"zsh -c 'echo ignored |& git push origin main'",
"git -C '&&' push",
"sh -c 'true; git push origin main'",
],
)
def test_a_quoted_operator_is_an_argument_and_a_real_one_is_a_boundary(
self, asking: ShellCommandPolicy, command: str
) -> None:
"""The reason segmentation now reads the raw text instead of tokens.
``shlex`` strips quote provenance, so once ``git -C '|' push`` has been
tokenised there is no information left that distinguishes the argument
from the operator. No rule written over tokens can tell them apart; the
fix had to move to where the quoting is still visible.
"""
assert asking.evaluate(command) is CommandDecision.REQUIRE_APPROVAL
assert asking.approval_reason(command) == "publish_command"
@pytest.mark.parametrize(
"command",
["echo a|grep b", "make test && echo done", "echo '|' ", "echo 'a && b'"],
)
def test_the_quoting_rules_do_not_invent_boundaries_or_lose_them(
self, asking: ShellCommandPolicy, command: str
) -> None:
"""Both directions of the same scanner: a real operator still splits, and
a quoted one still does not, without either turning ordinary work into a
prompt."""
assert asking.evaluate(command) is CommandDecision.ALLOW
def test_an_unknown_option_over_reads_rather_than_under_reads(self, asking: ShellCommandPolicy) -> None:
"""No table lists every option of every tool. An unconsumed value becomes
a candidate word, which can only make the policy ask about more than it
must -- the direction a security boundary is allowed to fail in. This one
passes before the fix too; it is here to pin the fallback, because the
obvious "consume the next token after any option" would break it."""
assert asking.evaluate("git --future-flag somevalue push") is CommandDecision.REQUIRE_APPROVAL
class TestSandboxingDoesNotRelaxClassification:
"""The classifier has no sandbox input: the Boxlite VM mounts the real
workspace read-write, so the filesystem rules a sandbox supposedly
contains still reach host data through that mount."""
def _asking(self) -> ShellCommandPolicy:
from raven.permissions.shell_policy import EXTERNAL_EFFECT_MATCHERS
policy = ShellCommandPolicy(deny_patterns=[r"\bmkfs\b"])
for name, matcher in EXTERNAL_EFFECT_MATCHERS:
policy.register_approval_matcher(name, matcher)
return policy
@pytest.mark.parametrize(
"command",
["git push origin main", "npm install lodash", "ssh host ls", "curl -o out https://example.com"],
)
def test_external_effects_ask(self, command: str) -> None:
assert self._asking().evaluate(command) is CommandDecision.REQUIRE_APPROVAL
assert self._asking().approval_reason(command) is not None
@pytest.mark.parametrize("command", ["rm -rf /", "shutdown now", "mkfs.ext4 /dev/sda1"])
def test_the_deny_list_holds(self, command: str) -> None:
assert self._asking().evaluate(command) is CommandDecision.HARD_DENY
def test_the_exec_prompt_shows_the_command_where_it_would_run(tmp_path) -> None:
tool = ExecTool(working_dir=str(tmp_path))
assert tool.approval_kind == "shell.exec"
assert tool.approval_evidence({"command": "rm a"}) == {"command": "rm a", "cwd": str(tmp_path)}
assert tool.approval_evidence({"command": "rm a", "working_dir": "/srv"}) == {"command": "rm a", "cwd": "/srv"}
assert tool.approval_evidence({"command": "rm a", "machine": "prod"}) == {"command": "rm a", "machine": "prod"}
def test_the_listing_root_is_where_the_command_would_run(tmp_path) -> None:
"""The directory listed around a command has to be the one the command ran
in, or the listing describes a tree the command never touched: a per-call
``working_dir`` moves it, and a registered machine takes it off this disk."""
tool = ExecTool(working_dir=str(tmp_path))
assert tool.listing_root({"command": "make"}) == tmp_path
assert tool.listing_root({"command": "make", "working_dir": "/srv"}) == Path("/srv")
assert tool.listing_root({"command": "make", "machine": "prod"}) is None