Raven-1/tests/test_shell_comments.py
0xKT 8716ec525c feat(docker): run raven in a docker container and make a first run configurable from the page
## Summary

Two things arrive together because the second is what made the first usable.

**Run raven from a container.** A Dockerfile builds the page, the Python
environment and the engine into one image; `cd docker && docker compose up`
serves it. nginx serves the built page and proxies the RPC socket to the engine
over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a
second container would have no address to dial. Compose publishes on 127.0.0.1,
and the entrypoint prints a one-time sign-in link on boot.

**A first run is no longer a dead end.** The gateway used to refuse to start
until a provider was configured, while being the thing that serves the page
where a provider gets configured. It now starts, says what is missing, and picks
up a key saved in Settings on the next turn without a restart. The onboarding
overlay no longer takes the page either: New chat, the model chip and Send send
the reader to Settings > Models instead, and an empty model list is marked on
the control that fills it.

Also carried: the provider catalogue registry (capabilities, modalities and
context windows per model), the reworked Settings > Models pane, and four fixes
found in review of that work -- a sibling catalogue answering 200 with a
non-catalogue body no longer crashes the whole fetch; adding a tag no longer
erases the label and description already stored for that model; `authenticated`
no longer demands a bearer token from a local provider reached by address; and
registry rows are keyed by the spelling every lookup produces, which recovers
593 rows that no reader could reach.

## Type

- [ ] Fix
- [x] Feature
- [x] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Commands run on the branch head, with their results:

- `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped
- `npx vitest run` in `ui-web` -- 1573 passed (101 files)
- `npx vitest run` in `ui-tui` -- 2037 passed (142 files)
- `uv run ruff check raven scripts tests` and `ruff format --check` -- clean
- `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean
- `npm run lint:rpc` in `ui-tui` -- generated.ts in sync
- `make check-source-language`, `make check-large-files` -- clean

Each of the four review fixes was checked by reverting it and watching a named
test go red, then restoring it.

The container image itself has not been built by CI or by me -- no Docker daemon
was available on the machine this was written on. It has been built and run by
the author of the branch.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

Two deliberate security trade-offs in the container deployment, both documented
in `docker/README.md`:

- nginx rewrites the `Origin` header, because the engine accepts only an Origin
  naming its own loopback port and no browser reaching a published port can send
  one. This drops a defence-in-depth layer. The session cookie is
  SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite
  ignores ports, so another page on localhost counts as same-site.
- `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to
  every page load, so anyone who can reach the published port is signed in. It
  is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any
  deployment reachable from elsewhere. These are two settings that must agree,
  and nothing warns when they do not.

Behaviour changes outside the container: the gateway now starts without a
configured provider rather than exiting 1, and the first-run onboarding overlay
no longer opens by default (`?onboard=1` still forces it).

Rollback is by revert; nothing here migrates state or changes an on-disk format.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-11 03:27:52 +00:00

259 lines
10 KiB
Python

"""A comment is text the shell will not run, and must not decide safety.
The policy turns `#` off in its lexer, so an English comment keeps taking part
in quote parsing. Three read-only commands were refused in one session because
their comments said `there's`, `what's` and `raven's`: the apostrophe opened a
quote that never closed, `shlex` raised, and the fail-closed branch answered
`hard_deny`. The user could only see "policy evaluation failed", asked four
times why, and the model -- given nothing to go on -- guessed wrong twice.
Fail-closed is right and stays: an executable region nobody can parse is still
refused. What changes is that the region the shell would ignore stops being
read as if it would run.
Both gates are tested here, but they no longer answer the same question. The
deny list has one owner now; what `ExecTool._guard_command` still decides on
its own is the allowlist and the workspace boundary. A comment could reach
both -- so those are what it is asked about here, rather than a second deny
read that no longer exists.
"""
from __future__ import annotations
import pytest
from raven.agent.tools.shell import ExecTool
from raven.permissions.shell_policy import CommandDecision, ShellCommandPolicy
@pytest.fixture
def policy() -> ShellCommandPolicy:
from raven.permissions.shell_policy import DELETE_MATCHERS
policy = ShellCommandPolicy(deny_patterns=[r"\b(mkfs|diskpart)\b", r"\bdd\s+if="])
for name, matcher in DELETE_MATCHERS:
policy.register_approval_matcher(name, matcher)
return policy
@pytest.fixture
def allowlisted(tmp_path) -> ExecTool:
"""The other one. `\\bls\\b` matches inside English prose on purpose: that is
the whole question -- whether a command can be talked onto the allowlist by
text the shell will not run."""
return ExecTool(
working_dir=str(tmp_path),
allow_patterns=[r"\bls\b"],
restrict_to_workspace=False,
)
# ---------- the session, verbatim --------------------------------------------
#: The three commands from the incident, each with the apostrophe that broke it.
INTERCEPTED = [
(
"there's",
"# Check if there's a raven-code specific config somewhere\n"
'find /tmp/state -name "*.json" -path "*code*" -not -path "*/traces/*" 2>/dev/null | head -20',
),
(
"what's",
"# Check what's listening on common ports\n"
'curl -s -o /dev/null -w "%{http_code}" http://localhost:18791/ 2>/dev/null; echo " :18791"',
),
(
"raven's",
"# Check how the main raven's exec tool gets initialized - look at the tool registry\n"
'grep -n "ExecTool" /tmp/raven/tool_index.py 2>/dev/null | head -20',
),
]
@pytest.mark.parametrize(("name", "command"), INTERCEPTED, ids=[n for n, _ in INTERCEPTED])
def test_a_read_only_command_survives_its_own_comment(name: str, command: str, policy: ShellCommandPolicy) -> None:
"""Each of the three, and the control the ledger ran beside them: the same
command without the comment was allowed every time."""
assert policy.evaluate(command) is CommandDecision.ALLOW
assert policy.evaluate(command.split("\n", 1)[1]) is CommandDecision.ALLOW
# ---------- what a comment must not be able to do ----------------------------
def test_a_comment_cannot_smuggle_a_denied_pattern_into_a_refusal(policy: ShellCommandPolicy) -> None:
"""The other direction of the same fault: text the shell ignores deciding
that a safe command is dangerous."""
assert policy.evaluate("ls -la # unlike dd if=/dev/zero, this one only lists") is CommandDecision.ALLOW
async def test_the_real_thing_is_still_denied() -> None:
"""Asserted through the permission gate, the deny list's one owner now."""
from raven.config.schema import PermissionsConfig
from raven.permissions.builtin import BuiltinRulings
from raven.permissions.gate import PermissionGate
gate = PermissionGate(
config_source=PermissionsConfig,
builtin=BuiltinRulings(extra_deny_patterns=[r"\b(mkfs|diskpart)\b", r"\bdd\s+if="]),
allow_ask=False,
)
result = await gate.enforce("exec", {"command": "dd if=/dev/zero of=/tmp/x"})
assert result is not None and not result.ok
assert "blocked by safety guard" in result.model_text
def test_a_denied_pattern_outside_the_comment_still_denies(policy: ShellCommandPolicy) -> None:
assert policy.evaluate("dd if=/dev/zero of=/tmp/x # make a file") is CommandDecision.HARD_DENY
@pytest.mark.parametrize(
"command",
[
"# tidy up\nrm -rf /",
"rm -rf / # tidy up",
"# shutdown notes\nshutdown -h now",
"shutdown -h now # going down",
],
)
def test_a_comment_does_not_excuse_the_command_beside_it(command: str, policy: ShellCommandPolicy) -> None:
"""The direction that matters most. Stripping comments must not become a
way to hide the executable half from the classifier."""
assert policy.evaluate(command) is CommandDecision.HARD_DENY
def test_a_dangerous_command_inside_a_comment_is_not_run_and_not_refused(policy: ShellCommandPolicy) -> None:
"""`rm -rf /` written in a comment is a sentence about `rm -rf /`."""
assert policy.evaluate("ls # never run rm -rf / on a live host") is CommandDecision.ALLOW
# ---------- what is not a comment --------------------------------------------
@pytest.mark.parametrize(
"command",
[
"grep -n 'foo#bar' file.txt",
'echo "a # b"',
"grep -rn '#!/bin/sh' scripts/",
"echo ${#PATH}",
"echo $#",
"git show HEAD:file#1 2>/dev/null",
],
)
def test_a_hash_that_is_not_a_comment_is_left_alone(command: str, policy: ShellCommandPolicy) -> None:
"""A `#` starts a comment only at the start of a word and outside quotes.
Cutting on every `#` would silently shorten commands, which is how a
classifier stops seeing the operative half."""
assert policy.evaluate(command) is CommandDecision.ALLOW
@pytest.mark.parametrize(
"command",
[
# Checked against real bash, not from memory:
# $ bash -c 'echo X$(echo b)#; echo PWNED' -> Xb# then PWNED
# $ bash -c 'echo X`echo b`#; echo PWNED' -> Xb# then PWNED
# The `#` stays part of the word and the `;` still separates a command,
# so the delete after it runs.
"echo $(echo b)#; rm -rf /",
"echo `echo b`#; rm -rf /",
],
ids=["command-substitution", "backtick"],
)
def test_a_hash_after_a_substitution_does_not_open_a_comment(command: str, policy: ShellCommandPolicy) -> None:
"""The comment-open set is not the operator set.
`)` and a backtick end an operator's reach but not a word, so cutting there
dropped the rest of the line out of every check while the shell went on
running it. Both of these are `hard_deny` on `main`.
"""
assert policy.evaluate(command) is CommandDecision.HARD_DENY
def test_a_hash_glued_to_a_word_does_not_hide_what_follows(policy: ShellCommandPolicy) -> None:
"""The adversarial version of the case above: if `foo#` were read as a
comment, everything after it would leave the classifier's view."""
assert policy.evaluate("echo a#b; rm -rf /") is CommandDecision.HARD_DENY
# ---------- fail-closed, still --------------------------------------------
@pytest.mark.parametrize(
"command",
[
"echo 'unterminated",
'grep "no closing quote /tmp',
"# a comment\necho 'still unterminated",
],
)
def test_an_executable_region_that_cannot_be_parsed_is_still_refused(command: str, policy: ShellCommandPolicy) -> None:
"""The invariant that must survive the fix. The last case is the pointed
one: a well-formed comment does not license a broken command after it."""
assert policy.evaluate(command) is CommandDecision.HARD_DENY
def test_a_wrapper_cannot_launder_a_denied_command_through_a_comment(policy: ShellCommandPolicy) -> None:
"""`bash -c` carries its own script, comments and all."""
assert policy.evaluate("bash -c '# tidy\nrm -rf /'") is CommandDecision.HARD_DENY
# ---------- one command, one classification ----------------------------------
def test_the_approval_prompt_describes_the_command_not_the_comment(policy: ShellCommandPolicy) -> None:
"""`evaluate` and `approval_reason` answer about the same text.
Read from the raw command, a comment mentioning a recursive delete tripped
the short-circuit that returns no reason at all -- so the user was asked to
approve a deletion under the generic "needs your approval" line, decided
entirely by prose the shell will not run.
"""
commented = "rm /tmp/x # careful, not rm -rf /"
assert policy.evaluate(commented) is CommandDecision.REQUIRE_APPROVAL
assert policy.approval_reason(commented) == policy.approval_reason("rm /tmp/x") == "delete_command"
# ---------- what the guard still decides on its own -------------------------
@pytest.mark.parametrize(
"command",
["ls -la # see ../notes for why", "ls -la # config lives in /etc/raven"],
ids=["traversal", "absolute"],
)
def test_a_comment_is_not_a_path_the_command_reaches_for(command: str, tmp_path) -> None:
"""Same shape as the incident: a read-only command refused for English
written beside it, and a message just as opaque about why."""
fenced = ExecTool(working_dir=str(tmp_path), restrict_to_workspace=True)
assert fenced._guard_command(command, cwd=str(tmp_path)) is None
@pytest.mark.parametrize(
"command",
["ls ../outside", "cat /etc/passwd"],
ids=["traversal", "absolute"],
)
def test_a_real_path_outside_the_workspace_is_still_refused(command: str, tmp_path) -> None:
fenced = ExecTool(working_dir=str(tmp_path), restrict_to_workspace=True)
assert fenced._guard_command(command, cwd=str(tmp_path)) is not None
def test_a_comment_cannot_talk_a_command_onto_the_allowlist(allowlisted: ExecTool, tmp_path) -> None:
"""The guard's other half, and the one that costs something if it is wrong.
A surface with an allowlist runs only what it named, so text the shell
discards satisfying that list is a command running which was never allowed
-- the opposite direction from the incident, and the reason this gate reads
the executable view rather than being handed a stripped string.
"""
refusal = allowlisted._guard_command("cat /etc/passwd # this is basically ls", cwd=str(tmp_path))
assert "not in allowlist" in (refusal or "")
assert allowlisted._guard_command("ls -la # unlike cat, this only lists", cwd=str(tmp_path)) is None