feat(*): a Raven daemon on a registered machine, reached over one watched ssh link (#920)
## Summary
First of three changes that let a Raven dispatch work to the machines in
its machine registry on a design where the machine owns the work: a
Raven
daemon runs there, the ssh is only the way to it, and an ssh that goes
mid-task does not end the task. This one puts the daemon on a machine
and
keeps the link to it; nothing dispatches to a machine yet. The next
change
runs acp sub-agents through it, the one after reads and writes files
there
by address.
It follows #893 (the ssh line and the remote working directory) and
replaces #897 and #901, closed when the design changed: their code is in
here, reworked to the daemon.
After the review's second round, four more things hold: a page read
(`read_file`, here and on a machine) reads the file a chunk at a time
and decodes only the page's own lines, so no one step holds the
interpreter for the whole file and the file is never whole in memory;
the whole-file calls on a machine go through one gate (readers
together, a writer alone, a waiting writer first), so a read never
answers with the old page and the new digest; the install reads the
index pip is configured with on the machine the way pip does, names it
in the plan and hands it to `uv pip install`; and a machine without uv
gets one pinned version as an unmanaged install, which writes no
receipt under the home directory and changes no PATH, while a node
directory that was there keeps its mode.
What a daemon is (`raven/node/`):
- This Raven's own code, running as a daemon on a registered machine.
The host's ssh runs a relay there (`python -I -m raven.node --attach
--dir <node_dir>`, `raven/node/relay.py`): it bridges the ssh session
to the daemon's unix socket under the node directory, starts the daemon
when none answers (`--serve`, detached, its words in `log/serve.log`),
and leaves with the ssh. The daemon leaves on its own after ten minutes
with no connection and no running agent; the next relay starts it again,
so nothing is registered as a service and a reboot costs one connect.
It opens no network port; the socket is the registered user's own
(mode 600) in `run/` (0700), or, when that path is too long for a
socket, in a directory of the user's own under the temp directory
(0700, checked before use: a directory of the user's, never a link,
never another user's, narrowed when wider). The relay forwards only to
a socket of the user's and starts no daemon beside anything else; a
daemon whose socket's place holds a thing it cannot remove says so in
one line and leaves rather than listen beside it. ssh login is the
authentication.
- It speaks newline-framed JSON-RPC 2.0, the framing ACP uses, so the
host drives it with `AcpClient` (over an `Endpoint` seam this change
adds: a process, or a channel of a link). Protocol 3.
- File calls (`fs/read`, `list`, `find`, `grep`, `stat`, `write`,
`edit`,
`pull`) run Raven's own file tools there, fenced to the `roots` each
call names, so a remote read comes back exactly as a local one would.
A write may carry `expect`, the digest the file had when it was read,
and is refused with `CHANGED` when it no longer matches. A page read
with its digest, a stat with one, a chunk and a checked write run in a
thread of the daemon's, writers one at a time under a lock, so the
loop that answers the heartbeat and forwards every agent's frames is
not held for the length of a large file.
- Agents (`agent/launch`, `frame`, `stderr`, `exit`, `cancel`, `close`):
the command named is started in the machine user's login environment
(captured once by the daemon itself, under `bash` or `zsh` `-lic`),
in its own process group, under a task id of the host's. Its ACP
frames travel on a numbered channel of the connection that holds it,
and are kept in the task's record on disk (`<node_dir>/tasks/<task>/`:
`task.json`, `frames.jsonl`, `stderr.log`; the directories 0700 and the
files 0600, as the host's own journal is) as they arrive. A connection
that goes leaves the agent running; `agent/attach` on a later
connection replays the record from the frame the host last read and
takes the task over, and a finished task is replayed from its record
alone, even by a daemon that never ran it. `agent/cancel` ends the
whole process tree: the tree is read off `ps` before the first signal,
so a command the agent put in a session of its own (Claude Code's shell
tool does) goes too. `task/list` names the tasks running or remembered;
records are kept seven days. `node/shutdown` ends agents only under
`--stdio`, the one mode that ties them to the connection (tests, and a
fallback).
- Probes (`probe/machine`, `probe/agent`) and `session/dir`, which makes
a session directory and says where it is as the machine resolves it.
- Small on purpose: it imports the file tools and nothing that talks to
a
model, reads a config or opens a port; beside Raven's code the machine
needs uv (or curl or wget, to fetch one with), a Python 3.12 if it has
one (else uv brings one), loguru and pillow (about 55 MB, more with a
fetched uv or Python), ripgrep-bin when its index has it. A test runs
the node with nothing importable beyond the standard library, its own
code and those two packages, and asks every file call, so the next
import added under `raven/agent/tools` is caught here rather than on
every installed node.
The same code at both ends (`raven/node/bundle.py`): the code is named
by
a digest of the package's files, and `node/hello` compares it, so a
daemon of other code is refused with the command that updates it. An
install lives in `<node_dir>/<version>-<digest>`.
The host side (`raven/node/client.py`):
- One link per machine, shared by every caller in the process; a link
idle for ten minutes with no agent on it and no call on its way to it
is closed the next time any link is asked for. A heartbeat every 5 s;
one unanswered for 15 s means the link is lost.
- A lost link no longer fails the agents on it. Each endpoint waits,
holding what the host writes meanwhile (up to 1000 frames); the link is
tried again on a backoff (2, 4, 8, 16 s) for as long as any agent
waits; a link that comes back attaches every task again from the frame
the host last read, sends what waited and any cancel asked in between.
Only a task the machine no longer knows ends an endpoint, in one
sentence. A link closed on purpose just closes: the daemon keeps its
agents, and `tasks()` on a later link lists them. The listing says
"link lost: ...; N tasks continue on the machine; reconnecting".
- Every failure is one sentence that names the machine and what to do,
never its address: not installed (with the install command),
unreachable, login refused, other code, a refused handshake, a call
that ran out of time, or the daemon's own last word.
The registry's package (`raven/ops/remote_path.py`) tells a remote path
from a local one (a UNC or extended Windows path, and on a Windows host
a
drive-relative one, are this computer's), resolves it against the
registry, and lends the tools the pool through one seam (`use_nodes`,
`nodes`) -- the tools must not import the node package, which imports
them. `raven/acp_client/remote.py` keeps the ssh command, the machine
lookup, the posix wrapping of a remote command and the one-sentence
reading of an ssh failure from #893, and loses the launch command, the
env preamble and the session-directory ssh that #897 would have used:
the daemon does those. The ssh command is handed to the connection as an
argv, and `AcpClient.launch` takes one as it is, so no host's own
grammar
reads it again (a Windows host's parser split the line into twelve
words). What reaches a model is a path on the machine written the way
scp writes it, address included, because that is how a file there is
named to the tools; the way in -- the key path, ssh's options, ssh's own
messages -- reaches no config file, log line or model, and an error
names
a machine by id and name only. The lane stays
debug-only (`RAVEN_DEBUG_MACHINES=1`, ruled on #897): with it off no
daemon is started and no install is planned, each saying why.
Installing (`raven ops connection install-node <id>`,
`raven/node/install.py`): one look at the machine (the install, its uv
or
a curl or wget to fetch one with, a Python 3.12 if it has one, the free
space), then a question, then everything through uv, as the repository's
dependency rule asks: a uv fetched under the node directory from its
installer when the machine has none, a virtualenv uv makes of the
machine's Python 3.12 or of one it brings, the pinned packages with
`uv pip`, the code as a tarball on ssh's stdin, each step one ssh call
and a failed step removing the half-made install; the node directory is
made 0700. A machine with neither uv nor a way to fetch one is refused
with the installer's address. `--dir` keeps installs elsewhere and
records it on the row; `--prune` removes other versions. Nothing else on
the machine changes. `raven ops connection doctor` reports the registry
as before.
Also: `CONTEXT.md` defines the Raven node and seats `raven.node` inner
in
the import contract (`pyproject.toml`, `lint-imports`); the repo layout
table gets its row; the gateway, the TUI and the RPC entrance close the
node pool with the ACP pool on shutdown.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
- `uv run pytest tests/test_node_bundle.py tests/test_node_server.py
tests/test_node_client.py tests/test_node_relay.py
tests/test_node_install.py tests/test_acp_client_teardown.py
tests/test_acp_remote.py tests/test_ops_remote_path.py
tests/test_cli_ops_connection_commands.py
tests/test_ops_connection_add.py tests/test_ops_transport.py
tests/test_subagent_acp.py tests/test_acp_client*.py
tests/test_cli_gateway*.py`
-- 859 passed on this tree; diff coverage of this change's lines from
these groups alone passes the 90% gate. Covered, beside what #901's
tests covered (parity of every file call with the tool's own answer,
the fence, each malformed field, the handshake, the stdio loop, the
install's scripts and one real install on this computer, through uv):
- The socket's directory is the user's own or not used (a file, a
link and another user's directory refused; wider bits narrowed); a
daemon refused its socket's place says so and leaves; the relay
forwards to nothing that is not the user's and starts no daemon
beside it; a task's records are 0700 and 0600; a ping is answered
while a file's digest is taken; a call on its way keeps its link;
the ssh argv reaches the connection as it is and `AcpClient.launch`
takes an argv; the installer fetches uv when the machine has none
(curl or wget) and refuses a machine with neither; a drive-relative
path is this computer's on a Windows host and scp's elsewhere, an
extended or UNC path this computer's anywhere; the node answers
every file call with only its requirements importable.
- A task is recorded as it runs and goes on when its connection ends;
a second connection attaches from the frame the host had not read and
hears the exit; a finished task is replayed from its record by a
daemon that never ran it; a record left "running" by a daemon that
died reads as lost; an attach for a task nobody knows is refused by
code; the task list and the sweep of old records; shutdown from a host
leaves a daemon's tasks running.
- The daemon on its socket: two connections in turn, the agent surviving
the first, the daemon leaving when idle and leaving at once when
another serves the same directory; `--serve` and `--attach` need
`--dir`.
- The link, over a stand-in `ssh` that hands the line to a real login
shell and starts the real relay and daemon: a request made while the
link is down is answered after the link is back, on the same daemon;
a daemon told to stop takes its agents and the turn says so; a link
closed on purpose leaves the agent for the next link to list, attach
and cancel; a heartbeat lost parks the endpoint and the reconnect
attaches it again; a task the machine no longer knows ends its
endpoint in one sentence; an attach the link cannot carry parks every
waiting endpoint again; held frames are capped; a file fetched whole
and checked, and refused outside the roots.
- The relay in this process: it starts a daemon when none answers,
bridges bytes both ways, reaches the same daemon again without
starting another, and gives up in one sentence when none comes up.
- Cancel ends a child the agent put in a session of its own; the tree
is read off `ps`; the daemon's own group is never signalled.
- `uv run pytest --collect-only` -- no import error; `lint-imports` --
10 kept, 0 broken. The daemon's two test modules skip on win32 before
importing it (fcntl, unix sockets), and the two POSIX teardown tests
lend a SIGKILL where the platform has none.
- `uv run pytest` on the final tree of the three changes, which includes
this one -- 29217 passed, 121 skipped, 3 failed:
`test_rpc_files.py::test_a_host_without_libreoffice_says_so`,
`test_simulation_scenario.py::test_a_fresh_trial_refreshes_the_subagent_homes_before_it_runs`
and
`test_simulation_suite.py::test_a_stopped_run_is_marked_in_its_record_and_finished_with_its_evidence`,
which fail the same way on main on this machine. The groups above
also pass under `--idle-ceiling-strict`, CI's idle ceiling.
- Real machines (the final tree of the three changes, two registered
Linux machines, a scratch registry and `RAVEN_HOME`):
- Install through uv on both: on the machine without uv, uv fetched
with the machine's curl under the node directory, then a virtualenv
of the machine's own Python 3.12, 16 s in all; on the machine whose
home disk was full, under `/tmp`, with a Python uv brought, 11 s.
Both nodes answered the handshake with this code's digest.
- Connect and handshake 0.7 to 0.9 s; `session/dir` 11 to 18 ms;
`agent/launch` 13 ms; a request's round trip 12 ms.
- With a real Claude Code running a 25 s shell loop: the host's ssh
killed mid-turn, the link read as lost in 5 s, the turn finished 27 s
after the kill with the agent's reply, 25 of 25 lines written, the
same daemon on both sides of the gap; a cancel left 0 processes 3 s
later; three sessions at once answered in 8 s; with every link closed
the daemon stayed with its records and no agent process remained.
- The second round's four notes (`tests/test_filesystem_tools.py
tests/test_node_install.py tests/test_node_server.py
tests/test_node_client.py tests/test_read_file_image.py` -- 337
passed, 100% diff coverage of that commit's lines): a page is the
lines `splitlines` gives whatever the chunk edge, over every break it
knows; a 120 MiB file pages with every ping answered within 0.25 s; a
read with a slowed digest beside a write of the same file answers with
the old page and the old digest and the write lands after; a waiting
writer goes before new readers; the probe reads `pip.conf` and
`PIP_INDEX_URL` as pip does under a real shell, the plan names the
index and the install hands it to uv; the uv step fetches one pinned
version unmanaged; a node directory that was there keeps its mode and
one the install makes is owner-only. On a registered Linux machine
with the node reinstalled: a page read of a 1 GiB file there beside
pings every 50 ms kept the longest wait of a ping at 0.34 s; the reads
sent during a write all answered with a page and a digest of one
version; the probe found the mirror that machine's pip is configured
with; an install into a fresh node directory, forced down the fetched
path, put uv 0.13.0 under it unmanaged in 24 s, with the receipt under
the home directory untouched and the directory at 700.
- The full review's five findings (`tests/test_node_client.py
tests/test_node_server.py tests/test_node_relay.py` -- 223 passed, the
seven new tests failing on the code before the fix, diff coverage of
that commit's lines past the gate): a link the heartbeat has not yet
seen die, replaced by an ordinary call, brings its parked agents along
(`agent/attach` on the new link, nothing left parked); a replay the new
link cannot carry keeps every frame it did not take, parks the endpoint
being replayed and every one not yet reached, and the link is tried
again; a cancel asked on a dead link, or failing on the new one, stays
wanted until a link delivers it, and a refusal clears it; two installs
on one long node directory get sockets apart; a launch still capturing
the login environment is counted, so a second one at the limit is
refused before the first spawns. On a registered Linux machine, one
script run before and after the fix (the local relay killed with the
heartbeat kept out of the way, then a plain call; a cancel asked in
that gap; eleven launches at once against the limit of eight): before,
the agent stayed parked and never answered, the cancelled task ran on
with its record saying running, nine launches started and ten tasks ran
there; after, the call took 0.3 s and the agent answered through the
new link, the task ended within the reattach with its record saying
so, eight started and three were refused with eight running, and
nothing was left running on the machine. The other half of the first
finding, a relaunch that fails under the call with agents parked: on the
machine, the relay killed and the relaunch made to fail once, before the
agent was still parked 30 s after the machine was reachable again and no
reconnect was under way; after, the reconnect the heartbeat loss
schedules runs here too, the agent was attached again on the first retry
after the machine answered and answered through the restored link.
- `scripts/check_source_language.py`, `scripts/check_commit_messages.py`
and `scripts/check_large_files.py` on the branch range, `commitlint` on
the commit and the prospective squash header, `ruff` -- all clean.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
The command describes itself in `--help`; user docs follow with the
change that runs sub-agents on a machine.
## Risk
Opt-in and inert: nothing starts a daemon unless the debug lane is on
and
a caller asks for a machine, and an install runs only when the owner
asks,
after saying what it will do. On a machine it adds one directory under
`~/.raven-node` (or the row's `node_dir`): the install, a `run/` with
the
socket, lock and pid of this install, a `tasks/` with the records, a
`log/`; removing the directory uninstalls it. The daemon runs as the
registered user, listens on a socket in a directory only that user can
open (and the relay forwards to a socket of that user's only), reads and
writes only inside the roots each call names, and leaves when idle. A
task's record holds the agent's ACP frames, which may quote what the
agent read: it is the user's alone (0700, 0600) and is removed after
seven days. The way in stays in the ssh argv, in memory; no error names
the address, and a model sees it only as part of a path on the machine,
written the way scp writes it. Rollback is a plain revert; an installed
daemon is then an inert directory that leaves once idle.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
Follows #893; replaces #897 and #901.
---------
Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-10-14 13:54:16 +08:00
|
|
|
"""raven.node.bundle: the code a Raven node runs, and the digest that names it.
|
|
|
|
|
|
|
|
|
|
The digest is how a host and a node tell they hold the same code, so these pin
|
|
|
|
|
what it covers -- every file's name and bytes, not the bytecode each Python
|
|
|
|
|
writes for itself -- and that what is sent unpacks to the same digest on the
|
|
|
|
|
other side.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
|
|
|
|
|
import io
|
|
|
|
|
import tarfile
|
|
|
|
|
from importlib.metadata import version
|
|
|
|
|
from pathlib import Path
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
|
|
from raven.node import bundle
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _tree(root: Path, files: dict[str, bytes]) -> Path:
|
|
|
|
|
for rel, data in files.items():
|
|
|
|
|
path = root / rel
|
|
|
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
|
|
|
path.write_bytes(data)
|
|
|
|
|
return root
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture
|
|
|
|
|
def pkg(tmp_path: Path) -> Path:
|
|
|
|
|
return _tree(
|
|
|
|
|
tmp_path / "raven",
|
|
|
|
|
{
|
|
|
|
|
"__init__.py": b"x = 1\n",
|
|
|
|
|
"sub/mod.py": b"y = 2\n",
|
|
|
|
|
"sub/data.json": b"{}\n",
|
|
|
|
|
"sub/__pycache__/mod.cpython-312.pyc": b"\x00bytecode",
|
|
|
|
|
"stale.pyc": b"\x00",
|
|
|
|
|
"old.pyo": b"\x00",
|
|
|
|
|
},
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_the_files_are_the_source_in_a_fixed_order_without_bytecode(pkg):
|
|
|
|
|
assert [p.as_posix() for p in bundle.files(pkg)] == ["__init__.py", "sub/data.json", "sub/mod.py"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_the_digest_follows_the_code_and_nothing_else(pkg):
|
|
|
|
|
first = bundle.tree_digest(pkg)
|
|
|
|
|
assert len(first) == 12 and first == bundle.tree_digest(pkg)
|
|
|
|
|
|
|
|
|
|
(pkg / "sub" / "__pycache__" / "other.cpython-313.pyc").write_bytes(b"another Python's bytecode")
|
|
|
|
|
assert bundle.tree_digest(pkg) == first
|
|
|
|
|
|
|
|
|
|
(pkg / "sub" / "mod.py").write_bytes(b"y = 3\n")
|
|
|
|
|
edited = bundle.tree_digest(pkg)
|
|
|
|
|
assert edited != first
|
|
|
|
|
|
|
|
|
|
(pkg / "sub" / "mod.py").rename(pkg / "sub" / "moved.py")
|
|
|
|
|
assert bundle.tree_digest(pkg) != edited, "a file's name is part of the code"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_two_trees_whose_bytes_only_line_up_differently_do_not_share_a_digest(tmp_path):
|
|
|
|
|
# Each name and each file's bytes are closed off, so moving a boundary --
|
|
|
|
|
# bytes from one file into the next, or from a name into its contents --
|
|
|
|
|
# never reads as the same code.
|
|
|
|
|
assert bundle.tree_digest(_tree(tmp_path / "a", {"a": b"x", "b": b"y"})) != bundle.tree_digest(
|
|
|
|
|
_tree(tmp_path / "b", {"a": b"xb\x00y"})
|
|
|
|
|
)
|
|
|
|
|
assert bundle.tree_digest(_tree(tmp_path / "c", {"a": b"bc"})) != bundle.tree_digest(
|
|
|
|
|
_tree(tmp_path / "d", {"ab": b"c"})
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_what_is_sent_unpacks_under_raven_to_the_same_digest(pkg, tmp_path):
|
|
|
|
|
try:
|
|
|
|
|
(pkg / "linked.py").symlink_to(pkg / "sub" / "mod.py")
|
|
|
|
|
except (OSError, NotImplementedError):
|
|
|
|
|
pytest.skip("making a symlink needs a privilege this account lacks")
|
|
|
|
|
|
|
|
|
|
data = bundle.tarball(pkg)
|
|
|
|
|
|
|
|
|
|
out = tmp_path / "unpacked"
|
|
|
|
|
with tarfile.open(fileobj=io.BytesIO(data), mode="r:gz") as archive:
|
|
|
|
|
names = archive.getnames()
|
|
|
|
|
archive.extractall(out, filter="data")
|
|
|
|
|
assert sorted(names) == ["raven/__init__.py", "raven/linked.py", "raven/sub/data.json", "raven/sub/mod.py"]
|
|
|
|
|
assert not (out / "raven" / "linked.py").is_symlink(), "the node gets the file a link points at"
|
|
|
|
|
assert bundle.tree_digest(out / "raven") == bundle.tree_digest(pkg)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_this_raven_is_named_by_its_version_and_the_digest_of_its_own_package():
|
|
|
|
|
from raven import __version__
|
|
|
|
|
|
|
|
|
|
assert bundle.package_dir().joinpath("node", "bundle.py").is_file()
|
|
|
|
|
assert bundle.digest() == bundle.tree_digest(bundle.package_dir())
|
|
|
|
|
assert bundle.install_name() == f"{__version__}-{bundle.digest()}"
|
|
|
|
|
names = {p.as_posix() for p in bundle.files()}
|
|
|
|
|
assert {"__init__.py", "node/server.py", "agent/tools/filesystem.py"} <= names
|
|
|
|
|
assert not any(name.endswith((".pyc", ".pyo")) or "__pycache__" in name for name in names)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_requirements_are_pinned_to_the_versions_this_raven_runs_with():
|
|
|
|
|
assert bundle.pinned(("loguru", "raven-no-such-package")) == [
|
|
|
|
|
f"loguru=={version('loguru')}",
|
|
|
|
|
"raven-no-such-package",
|
|
|
|
|
]
|