## Summary Registered machines are now available without a debug environment switch. `RAVEN_DEBUG_MACHINES` previously defaulted to off: it kept machine-backed agents out of the dispatch table, refused node/installer access, hid remote preset destinations, and treated file paths locally while off. It was a trial feature gate, not a verbose-logging option. At the owner's request, this PR removes the variable and all of those checks. Normal startup and restart recovery use the configured machine entries; even a leftover value of `0` is ignored. Machine registration, SSH authentication, file permission prompts, and concurrency limits still apply. Fourth change in the series that lets a Raven dispatch work to the machines in its machine registry. #920, #921 (merged through #922) and #922 are on main; this change stands on main alone now, and the diff is its own. What this does: a turn running on a registered machine survives this computer stopping, and the next start picks it up and reports it. - While a turn runs on a machine, this computer remembers what the machine does not know (`raven/acp_client/remote_runs.py`): one owner-only file per run under `<raven home>/machines/runs/`, with the machine's registry id, the task id its daemon runs the agent under, the agent, the session key, the instance handle, the dispatch id, the ACP session, the id of the prompt whose answer is awaited, the ids this side has used, how many of the agent's frames were read here, the agent's answer to `initialize` and the connection's launch key. Kept current after every frame the connection has dealt with, removed when the turn ends here. No address is in it. - When this process stops -- the gateway, `serve`, the TUI, the page stack -- the agents on machines stay with their daemons (`remote.leave_agents_running()` before the drain): a cancelled turn on a machine sends no `session/cancel`, closing its connection ends the endpoint only, and the manager leaves the dispatch's record open rather than finishing it as cancelled. A hard kill or `/restart` leaves the same record, written as the turn ran. - At the next start -- the gateway once its generation is bound; the `serve`, TUI and page stacks once -- `raven/agent/subagent/resume_remote.py` picks every run up: `task/list` on the machine, `agent/attach` from the frame last read (`NodePool.attach`), the connection adopted into the pool with request ids carried on past the other process's (`AcpConnectionPool.adopt`, `AcpClient.continue_ids_from`), the prompt's answer awaited (`AcpClient.await_response`), the reply built as a turn ended here is -- the partial notice when the agent stopped early, the machine note saying it was picked up after this computer restarted -- then the dispatch's record finished, the registry row and the node claim closed, and the result announced into the conversation that asked, as the manager announces a turn that ended here (`raven/acp_client/remote_resume.py`). A machine that cannot be asked keeps its record for a later start; a run the machine no longer has, one whose agent ended while this computer was away, one whose agent left the agent table, and one whose turn says nothing are announced as errors that say so. The adopted agent process serves later dispatches of its entry, as one started here would. - Seams: `AcpClient.request(sent=)` hears the id of what goes out; `ChannelEndpoint.watchers` hear the frames the connection has dealt with (`consumed`, called by the read loop); `ChannelEndpoint.stays` reads the leaving flag. - A cli entry's run on a machine is recorded while it runs (kind `cli`, no ACP part) and left the same way, the manager reading the dispatch as left; the entry picks it up (`CliAgentBackend.pick_up`): the daemon replays the command's whole output from its first line, the exit code ends it, and the reply closes with the machine note. - The record carries the task as the model wrote it, and -- once the manager has read the run as left -- the dispatch's origin and summary (`remote_runs.annotate`): the announce after a restart names the instance, node and tool call as the announce here does and quotes the task as authored, never the rendered prompt on disk; a host killed outright gets an origin made up from the spawn record and the run. - The record goes last, once the outcome has reached its conversation: a conclusion that fails, a stop under it, or an announce the scheduler refuses (`_announce_result` says so now) leaves the record, its claim released, for the next start to try again. - A run that returned here keeps its record the same way, until the manager's announce of the reply was taken: the manager dispatches with `reported_by_caller=True`, the backend then leaves the record behind with the reply instead of removing it as it returns, and the manager removes it after the announce (`remote_runs.forget_dispatch`) -- or, when the draining scheduler refused the announce, annotates it with the dispatch's origin and summary for the next start to report. A host killed between the agent's answer and the announce therefore still has the way back to the run. A caller that takes the reply as its own result (a graph node, a probe, a direct chat) dispatches without the keyword, and the record goes as it did. The record's frame count stops at the prompt: the turn's own frames, the chunks before the answer as much as the answer, are never checkpointed, so a pick-up replays the turn whole and the reply it reports is complete; a record that had moved past the answer would have resumed after it and waited forever. Not in this change: a run from the DAG lane has no spawn record to finish; it is announced from the origin the manager added to its record when it was left, or, for a host killed outright, from the run's record alone. The Tasks page reads the finished record on its next load rather than live. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Machine availability and restart checks after removing the trial switch: - New default-behavior regressions were run before the removal: 8 failed, with the enabled-value control passing. After removal, all 9 passed, including ACP/CLI recovery through the real AgentRegistry with the variable absent or set to `0`. - `uv run --no-sync pytest tests/test_acp_remote.py tests/test_node_client.py tests/test_node_server.py tests/test_node_install.py tests/test_ops_remote_path.py tests/test_permissions_remote_read.py tests/test_remote_file_tools.py tests/test_deliver_files_tool.py tests/test_raven_config_tool.py tests/test_rpc_subagents.py tests/test_shell_machine_channel.py tests/test_subagent_acp_remote.py tests/test_subagent_cli_remote.py tests/test_resume_remote.py tests/test_remote_resume.py tests/test_remote_runs.py tests/test_subagent_manager.py tests/test_cli_tui_commands.py tests/test_rpc_schema_match.py tests/test_config_schema.py -n 4 --idle-ceiling-strict --cov=raven --cov-branch --cov-report=json:<coverage-json> --cov-report= --tb=short` -- 1799 passed on the combined tree, including the latest published result-retention fix. An earlier run had a missing stdio response in the minimal-node dependency test and an idle-gate hit; both passed a focused recheck, and this final complete selection passed with the strict gate. - `uv run --no-sync python scripts/coverage_gate.py --coverage-json <coverage-json> diff --base-ref origin/main --threshold 90` -- 92.92% (656/706 changed executable lines) for this PR; the gate-removal commit alone covers 2/2 new executable lines (the remainder deletes gates and updates descriptions). - `UV_NO_SYNC=1 make lint-python lint-imports lint-deps lint-types`, the commit/source-language/large-file scope gates and `git diff --check` pass. Both RPC generators' `--check` runs pass; `npm test -- scripts/gates/fixture-shape.test.mjs` from `ui-web` passes all five tests. - Real Linux machine, isolated node installations, separate dispatch and restart processes: before removal, a task started with the trial switch on was reported as an absent agent and its recovery record deleted when restarted without it. After removal, both absent and old `0` settings recover the full CLI result and read its output file by address. The absent-variable case kills the relay before host shutdown. The combined final tree was reinstalled and both scenarios repeated successfully. - With the variable absent, 12 further timing/boundary probes pass: early CLI exit and blank lines, relay loss before heartbeat with normal-call recovery, tunnel loss, ACP/CLI/OpenAI cancellation during disconnection, 11 launches admitting 8 and refusing 3, occupied tunnel port, and an unwritable prompt file. Three attachment, POSIX-command and IPv6 checks also pass; Windows command parsing is simulated, not native Windows. No test task or tunnel remains; the dedicated node directories are removed. - CONTEXT.md, configuration and RPC descriptions, transport/path docstrings, both collaboration guides and the owner's design notes were checked against the code. The guides explain the former trial switch and normal startup without it. The existing SSH, permission and limit tests continue to exercise their checks with no test-only switch setup. - `env -u RAVEN_DEBUG_MACHINES uv run --no-sync pytest -n 4 --tb=short --cov=raven --cov-branch --cov-report=json:<full-coverage-json> --cov-report=` -- 31345 passed, 230 skipped, 2 xfailed, 5 failed. A separate current-main checkout reproduces four failures: CLI color rendering on this host, LibreOffice detection, and the two simulation cases. The remaining failure was the direct-chat test stub rejecting `reported_by_caller` from the latest result-retention change; the published wrapper fix now forwards only keywords declared by its fake backend. The full suite was not rerun for that test-only correction. Final direct-chat and manager recheck: 277 passed with strict idle accounting. Full-PR changed-line coverage from this run is 96.18% (679/706). Earlier verification: - `uv run pytest tests/test_remote_runs.py tests/test_remote_resume.py tests/test_resume_remote.py tests/test_acp_client_resume.py tests/test_node_client.py tests/test_subagent_acp_remote.py tests/test_subagent_cli_remote.py tests/test_cli_tui_commands.py tests/test_subagent_manager.py tests/test_rpc_bootstrap.py tests/test_cli_gateway_commands.py tests/test_acp_client_launch.py tests/test_subagent_acp.py tests/test_scope_canon.py tests/test_l4_entrances.py` -- all passed (605 passed). Covered: the record written owner-only, kept current per frame and per prompt, removed when the turn ends here and kept when this process leaves; a dispatch left at shutdown neither finished nor announced by the manager; an endpoint ended while leaving leaves the agent with the daemon (real daemon); a task picked up from the frame last read, with the watchers hearing each frame; a pick-up the machine refuses or cannot carry; an answer awaited for a request another process sent, ids carried on, no `session/cancel` while leaving; the reply read off replayed frames with the other sessions' frames ignored, a turn cut short marked partial, an unreachable machine keeping the record, a run the machine no longer has, a prompt never sent, an agent that ended while this computer was away, an answer waited out, a turn that said nothing, a pick-up beside a live connection closing its own; the manager-side conclusion: record finished and announced, an error announced and the record failed, an unreachable machine keeping the record, an agent off the table, a failure inside one pick-up not stopping the next, a run with no conversation, the once-per-process scheduling; a cli run recorded while it runs and left at shutdown, picked up whole on the real daemon behind the stand-in ssh, its pick-up's errors said (a non-zero exit, a machine that no longer has it, a link down, this computer leaving), a cancel that is not a leave ending the command with no record left; the TUI's stop saying the agents stay before the cancel and its start scheduling the pick-up; the record annotated with the dispatch's origin and summary when the manager reads it as left, the announce quoting the task as authored and naming instance, node and tool call, with and without the manager's fields; the record outliving a refused announce and a failed conclusion, claim released; a cli record read without an ACP session, an older record read with defaults; the record moving with the frames before the prompt and not past it, a dispatch's record forgotten by the manager's id with its claim, the manager dispatching with `reported_by_caller` and forgetting the record after an announce that was taken and annotating it after one that was refused; on the real daemon behind the stand-in ssh, an acp reply and a cli output returned with the record kept and the host "stopped" before the announce, the next start reading the same reply again whole, and a dispatch without the keyword leaving no record. End to end on the real daemon behind the stand-in ssh: a turn dispatched, this process "stopped" the way the gateway stops, the agent still running on the daemon, then a fresh pick-up reading its answer, finishing the record and announcing it, with the agent's process the pool's connection again. - `uv run pytest` on this tree, at an earlier revision of this change -- 29270 passed, 121 skipped, 3 failed: `test_rpc_files.py::test_a_host_without_libreoffice_says_so`, `test_simulation_scenario.py::test_a_fresh_trial_refreshes_the_subagent_homes_before_it_runs` and `test_simulation_suite.py::test_a_stopped_run_is_marked_in_its_record_and_finished_with_its_evidence`, which fail the same way on main on this machine; diff coverage of this change against its base: 97.45% (153 of 157 executable changed lines), and 94.67% for the four changes against main; `lint-imports` 10 kept, 0 broken; `ty` clean. - Real machine (a registered Linux machine with Claude Code signed in, node reinstalled with this tree): process A dispatched a 25 s task, saw the prompt out after 2.4 s and the record keep up with the frames (6, then 10), then stopped the way the gateway stops 5.4 s after the dispatch, leaving the record and the agent running there; process B, started fresh, found the record and the running task, picked it up in 2.4 s, announced the agent's reply to the conversation with the machine note, finished the record, removed the run's record, held the agent's process as the pool's connection, and closing the pool ended the agent on the machine. This revision, the cli lane with the code before and after, the node reinstalled for each: a cli entry `sh -c 'echo $$ > pid.txt; sleep 20; echo done; cat'` was dispatched and the host stopped the gateway's way 2.7 s later. Before: no record, the command left running on the machine, and the next start found nothing to pick up. After: the record kept with the task as authored and the manager's origin, the command running on, and the next start picked it up in 16.8 s (the sleep), announcing `done` and `hi` with the instance, node and tool call and the machine note. The same with the link killed before the stop: recorded, left, picked up. With the machine's daemon ended between the stop and the pick-up, it ended its groups on its way out, the task's record there said exit -9, and the pick-up announced that as the run's error in 0.1 s, leaving nothing on the machine. - `scripts/check_source_language.py`, `scripts/check_commit_messages.py` and `scripts/check_large_files.py` on the branch range, `commitlint`, `ruff` -- all clean. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed The English and Chinese agent-collaboration guides explain normal startup, registered-machine setup and recovery, and the removed trial switch. ## Risk Configured machine entries are available by default; entries without a `machine` remain local. A remote-looking file path is resolved against the registry even without an environment variable; use an explicit local path such as `./a:b` for a local filename containing a colon. A formerly hidden machine entry is now available for dispatch, and an old variable set to `0` no longer disables it. Stopping Raven no longer cancels a turn running on a machine; it goes on there until the agent finishes or is cancelled, and its answer arrives in the conversation after the next start, as a sub-agent result that triggers a turn. One record per run in the raven home, owner-only, removed when the run is concluded; a machine that cannot be reached keeps it, so a run is reported at most once and never lost quietly. The adopted connection answers requests with ids past the ones the stopped process used, so a replayed frame resolves nothing sent after the restart. Rollback is a plain revert; a record left behind is then an inert file. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues #920 and #922 (merged); #921 was carried by #922. --------- Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com>
959 lines
36 KiB
Python
959 lines
36 KiB
Python
"""The exec tool's machine channel: registry-addressed, capped, kill-transitive.
|
|
|
|
The channel exists so the address never has to enter the model's context: a
|
|
field run measured every remote look typed as a raw ``ssh -p <port> root@<ip>``
|
|
because the task statement had to carry the address for anything to work at
|
|
all. Naming a connection id instead keeps host/port/key below the model, makes
|
|
the cap reach both ends of the wire, and gives the detach refusal one place to
|
|
stand.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import time
|
|
|
|
import pytest
|
|
|
|
from raven.agent.tools import machine_exec
|
|
from raven.agent.tools.shell import ExecTool
|
|
|
|
ROW = {
|
|
"id": "conn_gpu",
|
|
"display_name": "GPU box",
|
|
"host": "203.0.113.7",
|
|
"port": 58717,
|
|
"user": "root",
|
|
"key": "~/.ssh/id_rsa",
|
|
}
|
|
|
|
|
|
@pytest.fixture
|
|
def registry(monkeypatch):
|
|
monkeypatch.setattr("raven.ops.connections.load", lambda: [dict(ROW)])
|
|
|
|
|
|
def _capturing_runner(record: list[str], rc: int = 0, out: str = "ok"):
|
|
def factory(row, *, cap_seconds=None):
|
|
def run(cmd: str) -> tuple[int, str]:
|
|
record.append(cmd)
|
|
return rc, out
|
|
|
|
return run
|
|
|
|
return factory
|
|
|
|
|
|
# ---- the schema is honest about what this install can reach ----
|
|
|
|
|
|
def test_no_registry_means_no_machine_parameter(monkeypatch, tmp_path):
|
|
monkeypatch.setattr("raven.ops.connections.load", lambda: [])
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
assert "machine" not in tool.parameters["properties"]
|
|
assert "name a machine" not in tool.description
|
|
|
|
|
|
def test_a_registered_machine_advertises_the_channel(registry, tmp_path):
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
prop = tool.parameters["properties"]["machine"]
|
|
assert "connection id" in prop["description"]
|
|
assert "exactly as listed" in prop["description"], "the id form is taught before the first mistake"
|
|
assert "name a machine" in tool.description
|
|
|
|
|
|
def test_a_broken_registry_leaves_the_plain_shell_working(monkeypatch, tmp_path):
|
|
def boom():
|
|
raise ValueError("mangled json")
|
|
|
|
monkeypatch.setattr("raven.ops.connections.load", boom)
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
assert "machine" not in tool.parameters["properties"]
|
|
|
|
|
|
# ---- routing: a named machine bypasses the local guards, not the rules ----
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_named_machine_routes_to_the_channel(registry, monkeypatch, tmp_path):
|
|
seen: dict = {}
|
|
|
|
async def fake(command, *, connection, cwd=None):
|
|
seen.update(command=command, connection=connection, cwd=cwd)
|
|
return "routed"
|
|
|
|
monkeypatch.setattr("raven.agent.tools.machine_exec.run_on_machine", fake)
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="nvidia-smi", machine="conn_gpu", working_dir="/root/ws")
|
|
|
|
assert out == "routed"
|
|
assert seen == {"command": "nvidia-smi", "connection": "conn_gpu", "cwd": "/root/ws"}
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unknown_id_answers_with_the_list(registry):
|
|
out = await machine_exec.run_on_machine("ls", connection="conn-typo")
|
|
|
|
assert "No connection with id 'conn-typo'" in out
|
|
assert "conn_gpu" in out, "the refusal carries the registry, so the next call can be right"
|
|
|
|
|
|
# ---- the channel's own rules ----
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_detaching_command_is_refused_before_anything_is_sent(registry, monkeypatch):
|
|
def never(*a, **kw):
|
|
raise AssertionError("a refused command must not build a runner")
|
|
|
|
monkeypatch.setattr("raven.ops.transport.runner_from", never)
|
|
|
|
out = await machine_exec.run_on_machine("nohup python train.py &", connection="conn_gpu")
|
|
|
|
assert "Refusing this command" in out
|
|
assert "ops_submit" in out
|
|
assert "Nothing was run" in out
|
|
assert "stopped when the call returns" in out, "the refusal says the fence behind it"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"command",
|
|
[
|
|
"sleep 300 &",
|
|
"sleep 300 & echo started",
|
|
"python train.py > log 2>&1 & echo ok",
|
|
"(sleep 300 &)",
|
|
"bash -c 'sleep 300 &'",
|
|
"sleep 300 &; echo",
|
|
],
|
|
)
|
|
def test_every_backgrounding_ampersand_is_refused_not_only_a_trailing_one(command):
|
|
"""Reviewer 2026-09-07: the guard caught a bare trailing & and let the other
|
|
five shapes through to the machine."""
|
|
assert machine_exec._detaching(command) == "a backgrounding &"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"command", ["make -j8 && make test", "python x.py 2>&1 | tail", "cmd &> log", "a |& b", "x <&0"]
|
|
)
|
|
def test_a_chain_a_redirection_or_a_pipe_is_not_a_backgrounding_ampersand(command):
|
|
assert machine_exec._detaching(command) is None
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_remote_command_runs_as_one_capped_group_on_the_remote_end(registry, monkeypatch):
|
|
sent: list[str] = []
|
|
monkeypatch.setattr("raven.ops.transport.runner_from", _capturing_runner(sent))
|
|
|
|
out = await machine_exec.run_on_machine("nvidia-smi", connection="conn_gpu", cwd="/root/ws")
|
|
|
|
assert len(sent) == 1
|
|
assert sent[0].startswith("cd /root/ws")
|
|
assert "set -m" in sent[0] and "kill -TERM -- -$pid" in sent[0], "the group kill must live on the remote end"
|
|
assert f"sleep {machine_exec._TIMEOUT_S};" in sent[0], "so must the cap"
|
|
assert sent[0].rstrip().endswith("raven-look nvidia-smi")
|
|
assert "on GPU box in /root/ws (exit 0)" in out
|
|
|
|
|
|
def _gone(pid: int) -> bool:
|
|
"""Whether ``pid`` is dead or a zombie nobody has reaped yet.
|
|
|
|
No ``pgrep``/``ps``: the CI image ships neither. A signal probe answers
|
|
"alive" for a zombie, and a swept child whose parent is gone sits as one
|
|
until init gets to it, so on Linux the state is read from /proc.
|
|
"""
|
|
import os
|
|
from pathlib import Path
|
|
|
|
try:
|
|
os.kill(pid, 0)
|
|
except ProcessLookupError:
|
|
return True
|
|
status = Path(f"/proc/{pid}/status")
|
|
if status.exists():
|
|
return "State:\tZ" in status.read_text(errors="replace")
|
|
return False
|
|
|
|
|
|
def _wait_gone(pid: int, seconds: float = 5.0) -> bool:
|
|
deadline = time.monotonic() + seconds
|
|
while time.monotonic() < deadline:
|
|
if _gone(pid):
|
|
return True
|
|
time.sleep(0.05)
|
|
return _gone(pid)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_work_left_running_in_the_background_is_stopped_when_the_look_returns(registry, monkeypatch, tmp_path):
|
|
"""The reviewer's reproduction, against a real shell: a child the guard
|
|
cannot see (spawned from python, no & in the command) is reparented to
|
|
init and runs on unbounded unless the group is swept. Local transport, so
|
|
the real runner and the real wrapper run. The child's pid is written by
|
|
the spawner before the foreground returns, so the check needs no process
|
|
listing tool."""
|
|
import sys
|
|
|
|
monkeypatch.setattr("raven.ops.connections.load", lambda: [{**ROW, "transport": "local"}])
|
|
pidfile = tmp_path / "child.pid"
|
|
spawn = f"import subprocess; p = subprocess.Popen(['sleep', '2718']); open({str(pidfile)!r}, 'w').write(str(p.pid))"
|
|
command = f"{sys.executable} -c {spawn!r}; echo ok"
|
|
assert machine_exec._detaching(command) is None, "the shape the guard cannot see"
|
|
|
|
started = time.monotonic()
|
|
out = await machine_exec.run_on_machine(command, connection="conn_gpu")
|
|
|
|
assert time.monotonic() - started < 10, "the sweep ends the call; nothing waits for the cap"
|
|
assert "(exit 0)" in out and "ok" in out
|
|
assert "left work running in the background; it was stopped" in out
|
|
assert "ops_submit" in out
|
|
child = int(pidfile.read_text())
|
|
assert _wait_gone(child), "the child must not outlive the call"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_cap_fires_on_the_machine_and_takes_the_whole_group(registry, monkeypatch, tmp_path):
|
|
monkeypatch.setattr("raven.ops.connections.load", lambda: [{**ROW, "transport": "local"}])
|
|
monkeypatch.setattr(machine_exec, "_TIMEOUT_S", 1)
|
|
pidfile = tmp_path / "sleeper.pid"
|
|
# The sleeper writes its own pid, then becomes the sleep, so the pid on
|
|
# file is the process the cap has to reach.
|
|
sleeper = f"bash -c 'echo $$ > {pidfile}; exec sleep 2719'"
|
|
|
|
started = time.monotonic()
|
|
out = await machine_exec.run_on_machine(f"echo before; {sleeper}; echo after", connection="conn_gpu")
|
|
|
|
assert time.monotonic() - started < 4, "a cap kill returns when the group is gone, not after the KILL grace"
|
|
assert "killed at the 1s limit" in out and "before" in out and "after" not in out
|
|
assert _wait_gone(int(pidfile.read_text()))
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_local_transport_is_not_wrapped_in_timeout(registry, monkeypatch):
|
|
sent: list[str] = []
|
|
monkeypatch.setattr("raven.ops.transport.runner_from", _capturing_runner(sent))
|
|
monkeypatch.setattr("raven.ops.connections.load", lambda: [{**ROW, "transport": "local"}])
|
|
|
|
await machine_exec.run_on_machine("ls", connection="conn_gpu")
|
|
|
|
assert "timeout " not in sent[0], "macOS ships no GNU timeout; the local runner caps the call itself"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_cap_kill_names_the_job_door(registry, monkeypatch):
|
|
from raven.ops.transport import TIMED_OUT_RC
|
|
|
|
monkeypatch.setattr("raven.ops.transport.runner_from", _capturing_runner([], rc=TIMED_OUT_RC, out="partial"))
|
|
|
|
out = await machine_exec.run_on_machine("python train.py", connection="conn_gpu")
|
|
|
|
assert f"killed at the {machine_exec._TIMEOUT_S}s limit" in out
|
|
assert "ops_submit" in out, "a cap kill is a routing signal, not a transient failure"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_missing_directory_is_named_not_guessed(registry, monkeypatch):
|
|
monkeypatch.setattr("raven.ops.transport.runner_from", _capturing_runner([], rc=66, out=""))
|
|
|
|
out = await machine_exec.run_on_machine("ls", connection="conn_gpu", cwd="/no/such/dir")
|
|
|
|
assert "no such directory" in out
|
|
assert "Nothing was run" in out
|
|
|
|
|
|
# ---- the runner half ----
|
|
|
|
|
|
def test_runner_from_refuses_a_row_with_no_address():
|
|
from raven.ops.transport import TransportError, runner_from
|
|
|
|
with pytest.raises(TransportError, match="no address"):
|
|
runner_from({"id": "conn_lost"})
|
|
|
|
|
|
def test_a_local_row_runs_here_in_the_ssh_answer_shape():
|
|
from raven.ops.transport import runner_from
|
|
|
|
run = runner_from({"id": "here", "transport": "local"}, cap_seconds=30)
|
|
rc, out = run("echo hello && echo oops 1>&2; exit 3")
|
|
|
|
assert rc == 3
|
|
assert "hello" in out
|
|
assert "oops" in out, "stderr rides along on failure, the way the ssh runner answers"
|
|
|
|
|
|
def test_a_local_row_cap_kills_the_command():
|
|
from raven.ops.transport import TIMED_OUT_RC, runner_from
|
|
|
|
run = runner_from({"id": "here", "transport": "local"}, cap_seconds=0.2)
|
|
rc, _ = run("sleep 5")
|
|
|
|
assert rc == TIMED_OUT_RC
|
|
|
|
|
|
def test_a_local_row_cap_kills_the_commands_children_too(tmp_path):
|
|
"""subprocess.run's timeout reached the shell alone: a python that had
|
|
forked a sleep kept both alive past the 124 that said they were stopped
|
|
(reviewed 2026-09-04). The command runs in its own session and the cap
|
|
kills that session."""
|
|
import os
|
|
import sys
|
|
import time
|
|
|
|
from raven.ops.transport import TIMED_OUT_RC, runner_from
|
|
|
|
pid_file = tmp_path / "child.pid"
|
|
script = (
|
|
"import subprocess, sys, time, pathlib; "
|
|
"p = subprocess.Popen(['sleep', '30']); "
|
|
f"pathlib.Path({str(pid_file)!r}).write_text(str(p.pid)); "
|
|
"time.sleep(30)"
|
|
)
|
|
run = runner_from({"id": "here", "transport": "local"}, cap_seconds=0.5)
|
|
rc, _ = run(f"{sys.executable} -c {script!r}")
|
|
|
|
assert rc == TIMED_OUT_RC
|
|
child = int(pid_file.read_text())
|
|
deadline = time.monotonic() + 3
|
|
while time.monotonic() < deadline:
|
|
try:
|
|
os.kill(child, 0)
|
|
except ProcessLookupError:
|
|
break
|
|
time.sleep(0.05)
|
|
else:
|
|
os.kill(child, 9)
|
|
raise AssertionError(f"the command's child (pid {child}) outlived the cap")
|
|
|
|
|
|
def test_without_killpg_the_cap_falls_back_to_a_tree_kill_by_pid(monkeypatch):
|
|
"""Native Windows has neither ``killpg`` nor a working ``start_new_session``.
|
|
The runner must still return 124 and still reach the command's children:
|
|
the platform's tree kill is ``taskkill /T /F`` by pid."""
|
|
import os
|
|
from types import SimpleNamespace
|
|
|
|
from raven.ops import transport
|
|
|
|
monkeypatch.delattr(os, "killpg", raising=False)
|
|
calls: list[list[str]] = []
|
|
|
|
def fake_run(argv, **kwargs):
|
|
calls.append(list(argv))
|
|
return SimpleNamespace(returncode=0)
|
|
|
|
monkeypatch.setattr(transport.subprocess, "run", fake_run)
|
|
monkeypatch.setattr(transport.shutil, "which", lambda name: "C:/Windows/System32/taskkill.exe")
|
|
killed: list[str] = []
|
|
proc = SimpleNamespace(pid=4242, poll=lambda: None, kill=lambda: killed.append("kill"))
|
|
|
|
assert transport._own_group_kwargs() == {}
|
|
transport._kill_process_tree(proc)
|
|
|
|
assert calls == [["C:/Windows/System32/taskkill.exe", "/T", "/F", "/PID", "4242"]]
|
|
assert killed == ["kill"], "and the shell itself is still killed when the tree kill left it"
|
|
|
|
|
|
# ---- the plain shell refuses typed ssh to a machine the registry knows ----
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_typed_ssh_to_a_registered_machine_is_refused(registry, tmp_path):
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(command=f"ssh -p 58717 root@203.0.113.7 'nohup python train.py &'; touch {marker}")
|
|
|
|
assert "Error:" in out
|
|
assert "conn_gpu" in out and "ops_submit" in out
|
|
assert not marker.exists()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ssh_to_an_unregistered_host_still_runs(registry, tmp_path):
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="echo ssh 198.51.100.9 would be typed here")
|
|
|
|
assert "would be typed here" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_naming_the_host_without_ssh_is_not_a_bypass(registry, tmp_path):
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="echo the box is 203.0.113.7")
|
|
|
|
assert "203.0.113.7" in out and "Error:" not in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_broken_registry_refuses_nothing_in_the_plain_shell(monkeypatch, tmp_path):
|
|
def boom():
|
|
raise ValueError("mangled json")
|
|
|
|
monkeypatch.setattr("raven.ops.connections.load", boom)
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="echo ssh 203.0.113.7 with a broken registry")
|
|
|
|
assert "with a broken registry" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"written",
|
|
["ssh", "/usr/bin/ssh", "\\ssh"],
|
|
ids=["bare", "path-qualified", "backslash-escaped"],
|
|
)
|
|
async def test_the_ssh_client_is_recognised_however_it_is_written(registry, tmp_path, written):
|
|
"""All three spellings run the client, so all three reach the far side.
|
|
|
|
A leading backslash suppresses alias lookup and an absolute path skips
|
|
PATH; neither changes what executes. Matching only the bare word left both
|
|
on the plain path, past the cap, the process-group sweep and the ledger.
|
|
"""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / f"ran-{written.strip('\\/').replace('usr', '')}"
|
|
|
|
out = await tool.execute(command=f"{written} -o ConnectTimeout=1 -p 58717 root@203.0.113.7 true; touch {marker}")
|
|
|
|
assert "Error:" in out and "conn_gpu" in out
|
|
assert not marker.exists()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_different_port_at_the_same_address_is_a_different_machine(monkeypatch, tmp_path):
|
|
"""The registry holds several machines at one address on different ports.
|
|
|
|
Matching the address alone returns whichever row is listed first, so the
|
|
refusal names a machine the command was not reaching and the recovery it
|
|
proposes would run the work somewhere else.
|
|
"""
|
|
other = dict(ROW) | {"id": "conn_gpu_b", "display_name": "GPU box B", "port": 64101}
|
|
monkeypatch.setattr("raven.ops.connections.load", lambda: [dict(ROW), other])
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="ssh -o ConnectTimeout=1 -p 64101 root@203.0.113.7 true")
|
|
|
|
assert "conn_gpu_b" in out, "the refusal names the machine on the port that was typed"
|
|
assert "conn_gpu'" not in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_longer_address_that_starts_with_a_registered_one_still_runs(registry, tmp_path):
|
|
"""203.0.113.70 is not 203.0.113.7.
|
|
|
|
A substring test made every address with a registered one as its prefix
|
|
unreachable from here, which breaks the guarantee that ssh to a machine
|
|
the registry does not hold keeps working.
|
|
"""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="echo ssh root@203.0.113.70 would be typed here")
|
|
|
|
assert "would be typed here" in out and "Error:" not in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_line_that_cannot_be_tokenised_is_not_judged(registry, tmp_path):
|
|
"""An unbalanced quote is not a shell line this can read.
|
|
|
|
The shell rejects it too, so nothing reaches a machine either way, and
|
|
guessing at a half-parsed line would refuse commands that never ran.
|
|
"""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="ssh root@203.0.113.7 'unterminated")
|
|
|
|
assert "conn_gpu" not in out, "no refusal is issued for a line nobody can parse"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_flag_that_takes_no_value_does_not_hide_the_destination(registry, tmp_path):
|
|
"""-v takes no argument, so the destination is the next word, not the one after it."""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(command=f"ssh -v -o ConnectTimeout=1 -p 58717 root@203.0.113.7 true; touch {marker}")
|
|
|
|
assert "Error:" in out and "conn_gpu" in out
|
|
assert not marker.exists()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ssh_with_no_destination_at_all_is_left_alone(registry, tmp_path):
|
|
"""Flags and nothing else reaches no machine; ssh prints its usage and stops."""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="ssh -V")
|
|
|
|
assert "conn_gpu" not in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_row_whose_port_is_not_a_number_is_read_as_the_default(monkeypatch, tmp_path):
|
|
"""A hand-edited registry must not make the guard give up on the row.
|
|
|
|
22 is what ssh itself would use, so a row with no usable port is compared
|
|
against the port a command that names none would reach.
|
|
"""
|
|
monkeypatch.setattr("raven.ops.connections.load", lambda: [dict(ROW) | {"port": "not-a-number"}])
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="ssh -o ConnectTimeout=1 root@203.0.113.7 true")
|
|
|
|
assert "Error:" in out and "conn_gpu" in out
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unspaced_operator_still_separates_the_commands(registry, tmp_path):
|
|
"""A shell reads "true&&ssh" as two commands; splitting on whitespace reads one word.
|
|
|
|
That word is neither "true" nor "ssh", so the guard saw no ssh at all and
|
|
the registered machine was reached from the plain shell.
|
|
"""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(command=f"true&&ssh -o ConnectTimeout=1 -p 58717 root@203.0.113.7 true; touch {marker}")
|
|
|
|
assert "Error:" in out and "conn_gpu" in out
|
|
assert not marker.exists()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_every_ssh_in_the_line_is_read_not_just_the_first(registry, tmp_path):
|
|
"""A compound line reaches each of its commands, so each destination counts.
|
|
|
|
Stopping at the first let a line through on the strength of the half that
|
|
was allowed, and the registered machine in the second half still ran.
|
|
"""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(
|
|
command=(
|
|
"ssh -o ConnectTimeout=1 root@198.51.100.9 true; "
|
|
f"ssh -o ConnectTimeout=1 -p 58717 root@203.0.113.7 true; touch {marker}"
|
|
)
|
|
)
|
|
|
|
assert "Error:" in out and "conn_gpu" in out
|
|
assert not marker.exists()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_port_given_as_an_option_counts_like_the_flag(registry, tmp_path):
|
|
"""OpenSSH honours "-o Port=N" exactly as it honours "-p N" (checked with ssh -G).
|
|
|
|
Consuming -o as a value-taking flag and dropping its value left the guard
|
|
on port 22, so a machine registered elsewhere read as unregistered.
|
|
"""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(command=f"ssh -o ConnectTimeout=1 -o Port=58717 root@203.0.113.7 true; touch {marker}")
|
|
|
|
assert "Error:" in out and "conn_gpu" in out
|
|
assert not marker.exists()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"option",
|
|
["-o 'Port 58717'", '-o "Port 58717"', "-o 'Port = 58717'"],
|
|
ids=["single-quoted", "double-quoted", "spaced-equals"],
|
|
)
|
|
async def test_the_port_option_is_read_when_a_space_separates_it(registry, tmp_path, option):
|
|
"""OpenSSH takes the option written either way, so the guard has to read both.
|
|
|
|
`ssh -G -o "Port 58717" root@203.0.113.7` prints `port 58717`, the same as
|
|
the equals spelling. Splitting the value on the equals sign alone discarded
|
|
the spaced one, left the guard on port 22, and the line ran on the plain
|
|
shell path -- past the cap, the process-group sweep and the ledger.
|
|
"""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(command=f"ssh -o ConnectTimeout=1 {option} root@203.0.113.7 true; touch {marker}")
|
|
|
|
assert "Error:" in out and "conn_gpu" in out
|
|
assert not marker.exists()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"redirection",
|
|
["2>&1", "&>/dev/null", "2>/dev/null"],
|
|
ids=["descriptor-dup", "both-streams", "descriptor-to-file"],
|
|
)
|
|
async def test_a_redirection_before_the_destination_does_not_hide_it(registry, tmp_path, redirection):
|
|
"""A redirection is plumbing; the ssh's own words continue after it.
|
|
|
|
`2>&1` arrives as the three tokens 2, >& and 1, so the bare descriptor was
|
|
taken for the destination, and `&>` -- matching no operator the reader knew
|
|
-- became one itself. Either way the registered host further along the line
|
|
was never read and the command ran on the plain shell path.
|
|
"""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(
|
|
command=f"ssh {redirection} -o ConnectTimeout=1 -p 58717 root@203.0.113.7 true; touch {marker}"
|
|
)
|
|
|
|
assert "Error:" in out and "conn_gpu" in out
|
|
assert not marker.exists()
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("command", "expected"),
|
|
[
|
|
("ssh -p 58717 root@h true 2>&1 | tee log", [("h", 58717)]),
|
|
("ssh -p 58717 root@h >", [("h", 58717)]),
|
|
("ssh root@h >; ssh -p 9 root@k", [("h", 22), ("k", 9)]),
|
|
("ssh root@h <<EOF\nfoo\nEOF", [("h", 22)]),
|
|
("ssh -o Port=58717 -o 'Port 22' root@h", [("h", 58717)]),
|
|
("ssh 2>&1", []),
|
|
],
|
|
ids=["after-host", "dangling", "dangling-then-separator", "heredoc", "first-port-wins", "no-host"],
|
|
)
|
|
def test_redirections_are_stepped_over_wherever_they_fall(command, expected):
|
|
"""A redirection is neither a destination nor the end of the command.
|
|
|
|
Before the destination it is skipped so the host after it is read; after
|
|
the destination it changes nothing; dangling before a separator it does not
|
|
swallow the separator, so the next ssh in the line is still found.
|
|
"""
|
|
assert machine_exec._ssh_destinations(command) == expected
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("command", "expected"),
|
|
[
|
|
("ssh -o Port=58717 -o 'Port 22' root@h", 58717),
|
|
("ssh -o 'Port 22' -o Port=58717 root@h", 22),
|
|
("ssh -p 2222 -o Port=58717 root@h", 2222),
|
|
("ssh -o Port=58717 -p 2222 root@h", 58717),
|
|
("ssh -p 58717 -p 22 root@h", 58717),
|
|
("ssh -p 22 -p 58717 root@h", 22),
|
|
],
|
|
ids=["option-then-option", "reversed", "flag-then-option", "option-then-flag", "flag-twice", "flag-twice-reversed"],
|
|
)
|
|
def test_a_repeated_port_keeps_the_first_value_as_ssh_does(command, expected):
|
|
"""ssh takes the first obtained value for every option, `-p` and `-o Port`
|
|
queueing together: `ssh -G -p 2222 -o Port=58717 host` prints 2222 and the
|
|
two reversed prints 58717 (measured with OpenSSH 9.9p2).
|
|
|
|
Overwriting instead read `-p 58717 -p 22` as port 22, so a command that
|
|
really reaches the registered machine on 58717 read as unregistered and
|
|
ran on the plain shell path -- the bypass this guard exists to close.
|
|
"""
|
|
assert machine_exec._ssh_destinations(command) == [("h", expected)]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_option_that_is_not_the_port_leaves_the_port_alone(registry, tmp_path):
|
|
"""-o carries many settings; only Port changes where the command lands."""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
|
|
out = await tool.execute(command="echo ssh -o StrictHostKeyChecking=no root@203.0.113.7 true")
|
|
|
|
assert "conn_gpu" not in out, "port 22 is not the registered 58717"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_value_taking_flag_does_not_stand_in_for_the_destination(registry, tmp_path):
|
|
"""-l takes the login name, so the bare host after it is still the destination.
|
|
|
|
Also the form with no "user@": the destination is the whole word, not
|
|
whatever follows an at sign that is not there.
|
|
"""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(command=f"ssh -l root -o ConnectTimeout=1 -p 58717 203.0.113.7 true; touch {marker}")
|
|
|
|
assert "Error:" in out and "conn_gpu" in out
|
|
assert not marker.exists()
|
|
|
|
|
|
# Each spelling beside what OpenSSH itself resolves it to. The expectations were
|
|
# measured with `ssh -G` (OpenSSH_9.9p2 here, and OpenSSH_8.9p1 in review for
|
|
# the ones that apply to both), and the test below re-measures them wherever an
|
|
# ssh client is installed, so a wrong expectation cannot hide in this table.
|
|
GETOPT_SPELLINGS = [
|
|
("ssh -vp 58717 root@h true", ("h", 58717)),
|
|
("ssh -4p 58717 h", ("h", 58717)),
|
|
("ssh -vvvp58717 h", ("h", 58717)),
|
|
("ssh -vo Port=58717 h", ("h", 58717)),
|
|
("ssh -o Port=58717 -vp 22 h", ("h", 58717)),
|
|
("ssh root@h -p 58717 true", ("h", 58717)),
|
|
("ssh root@h -o Port=58717 true", ("h", 58717)),
|
|
("ssh root@h -vp 58717 ls -la", ("h", 58717)),
|
|
("ssh -p 58717 root@h -p 22", ("h", 58717)),
|
|
("ssh root@h true -p 58717", ("h", 22)),
|
|
("ssh root@h -- -p 58717", ("h", 22)),
|
|
("ssh -- root@h -p 58717", ("h", 22)),
|
|
("ssh -B lo -p 58717 root@h", ("h", 58717)),
|
|
("ssh -P tag -p 58717 root@h", ("h", 58717)),
|
|
]
|
|
GETOPT_IDS = [
|
|
"bundled",
|
|
"bundled-after-a-number-flag",
|
|
"bundled-attached",
|
|
"bundled-option",
|
|
"bundled-after-a-port",
|
|
"port-after-host",
|
|
"option-after-host",
|
|
"bundled-after-host",
|
|
"first-port-wins-across-host",
|
|
"after-the-remote-command",
|
|
"after-double-dash",
|
|
"double-dash-before-host",
|
|
"bind-interface-value",
|
|
"tag-value",
|
|
]
|
|
|
|
|
|
@pytest.mark.parametrize(("command", "expected"), GETOPT_SPELLINGS, ids=GETOPT_IDS)
|
|
def test_options_are_read_the_way_getopt_and_ssh_read_them(command, expected):
|
|
"""Two readings the reviewer showed let a real connection past the guard
|
|
(2026-09-21): a bundled group (`-vp 58717`) was skipped as if it took no
|
|
argument, so the port became the destination; and the scan stopped at the
|
|
host, while OpenSSH goes back to reading options after it until the first
|
|
word that is not one. `-B` takes a value too; the set that said which flags
|
|
do was kept by hand and had lost it, so it is read from ssh's own getopt
|
|
string now."""
|
|
assert machine_exec._ssh_destinations(command) == [expected]
|
|
|
|
|
|
@pytest.mark.parametrize(("command", "expected"), GETOPT_SPELLINGS, ids=GETOPT_IDS)
|
|
def test_the_table_agrees_with_the_ssh_on_this_computer(command, expected):
|
|
import shutil
|
|
import subprocess
|
|
|
|
if not shutil.which("ssh"):
|
|
pytest.skip("no ssh on this computer")
|
|
words = command.split()[1:]
|
|
|
|
def resolved(args: list[str]) -> tuple[str, int]:
|
|
out = subprocess.run(["ssh", "-G", *args], capture_output=True, text=True, check=False, timeout=10).stdout
|
|
seen = dict(line.split(" ", 1) for line in out.splitlines() if line.startswith(("hostname ", "port ")))
|
|
return seen.get("hostname", ""), int(seen.get("port", 0) or 0)
|
|
|
|
if "tag" in words and resolved(["-P", "tag", "host.invalid"])[0] != "host.invalid":
|
|
# ``-P`` takes no value before OpenSSH 9.2, where ``tag`` would BE the
|
|
# host: this row's expectation is the newer client's reading, so it is
|
|
# skipped rather than failed. Asked as a question rather than read off a
|
|
# version string, and asked as its own invocation so it cannot perturb
|
|
# the row's own arguments (reviewed 2026-09-24: the earlier check looked
|
|
# for the literal "tag" in the output, which an older ssh prints as the
|
|
# hostname -- so it never skipped).
|
|
pytest.skip("this ssh predates `-P tag` (OpenSSH < 9.2), where P takes no value")
|
|
assert resolved(words) == expected
|
|
|
|
|
|
def test_the_value_taking_flags_are_ssh_s_own():
|
|
"""The set is derived from the getopt string, not listed beside it: the
|
|
hand-kept list this replaces had lost `B`."""
|
|
assert machine_exec._SSH_VALUE_FLAGS == frozenset("bceilmopBDEFIJLOPQRSWw")
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_port_written_after_the_host_is_still_refused(registry, tmp_path):
|
|
"""Through ExecTool, the shape the reviewer ran: the whole line used to run
|
|
on the plain shell path and reached 203.0.113.7:58717."""
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
for command in (
|
|
f"ssh -o ConnectTimeout=1 root@203.0.113.7 -p 58717 true; touch {marker}",
|
|
f"ssh -o ConnectTimeout=1 -vp 58717 root@203.0.113.7 true; touch {marker}",
|
|
):
|
|
out = await tool.execute(command=command)
|
|
assert "Error:" in out and "conn_gpu" in out, command
|
|
assert not marker.exists()
|
|
|
|
|
|
def test_the_machine_lane_appears_once_a_first_machine_is_added(tmp_path, monkeypatch):
|
|
"""The reviewed shape (2026-09-24): a process that starts with no machine,
|
|
then gets its first one from ``ops_connection_add``. The registry used to
|
|
serve exec from the copy it took at registration, so ``machine`` stayed
|
|
missing until a restart -- while every text after the add sent the model
|
|
to ``exec(machine=...)``. Written through the real writer and read back
|
|
through the real registry, not a patched ``load``."""
|
|
from raven.agent.tools.registry import ToolRegistry
|
|
from raven.ops import connection_add, connections
|
|
|
|
store = tmp_path / "connections.json"
|
|
monkeypatch.setattr(connections, "store_path", lambda: store)
|
|
registry = ToolRegistry()
|
|
registry.register(ExecTool(working_dir=str(tmp_path)))
|
|
|
|
def served() -> dict:
|
|
(exec_def,) = [d for d in registry.get_definitions() if d["function"]["name"] == "exec"]
|
|
return exec_def["function"]
|
|
|
|
before = served()
|
|
assert "machine" not in before["parameters"]["properties"]
|
|
assert "machine" not in before["description"]
|
|
|
|
connection_add.write(dict(ROW, transport="ssh"))
|
|
|
|
after = served()
|
|
assert "machine" in after["parameters"]["properties"], "the lane the reply names is in the served schema"
|
|
assert "pass 'machine'" in after["description"]
|
|
|
|
|
|
# ---- a copy to a registered machine is the plain shell's, as on main ----
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"command",
|
|
[
|
|
"scp -P 58717 a.txt root@203.0.113.7:/tmp/",
|
|
"sftp -P 58717 root@203.0.113.7",
|
|
"rsync -av -e 'ssh -p 58717' ./dir root@203.0.113.7:/data/",
|
|
"scp.exe -P 58717 a.txt root@203.0.113.7:/tmp/",
|
|
],
|
|
)
|
|
async def test_a_copy_to_a_registered_machine_is_not_refused(registry, tmp_path, command):
|
|
"""A copy starts nothing on the far side, and passes exec's own question like any command.
|
|
|
|
The file tools cannot carry what a copy carries -- a directory, a binary
|
|
file, a file over the fetch budget -- so a refusal left those with no
|
|
route, and taught the very spelling it objected to. Python stands for
|
|
the client: nothing here connects anywhere.
|
|
"""
|
|
import sys
|
|
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
# Python stands for the client on any host (``true`` and ``touch`` are not
|
|
# on a Windows one): the copy's words are its arguments, and the marker
|
|
# says the line ran.
|
|
client = f'"{sys.executable}" -c "import pathlib, sys; pathlib.Path(sys.argv[-1]).touch()"'
|
|
|
|
out = await tool.execute(command=f'{client} {command} "{marker}"')
|
|
|
|
assert "Error:" not in out
|
|
assert marker.exists()
|
|
|
|
|
|
# ---- the ssh client is read in its Windows spellings too ----
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("word", "program"),
|
|
[
|
|
("ssh", "ssh"),
|
|
("/usr/bin/ssh", "ssh"),
|
|
("\\ssh", "ssh"),
|
|
("ssh.exe", "ssh"),
|
|
("SSH.EXE", "ssh"),
|
|
("C:\\Windows\\System32\\OpenSSH\\ssh.exe", "ssh"),
|
|
("ssh.cmd", "ssh"),
|
|
("sshx", "sshx"),
|
|
("my-ssh.exe.bak", "my-ssh.exe.bak"),
|
|
],
|
|
)
|
|
def test_the_program_a_word_runs_is_read_with_windows_spellings_too(word, program):
|
|
assert machine_exec._program(word) == program
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"command",
|
|
[
|
|
"C:\\Windows\\System32\\OpenSSH\\ssh.exe -p 58717 root@203.0.113.7 ls",
|
|
'"C:\\Program Files\\OpenSSH\\ssh.exe" -p 58717 root@203.0.113.7 ls',
|
|
"SSH.EXE -p 58717 root@203.0.113.7 ls",
|
|
],
|
|
)
|
|
async def test_on_a_windows_host_a_program_s_whole_path_keeps_its_backslashes_and_its_name(
|
|
registry, tmp_path, monkeypatch, command
|
|
):
|
|
# The shell there reads a backslash as a path's own; read as POSIX escapes, the name was lost.
|
|
monkeypatch.setattr(machine_exec, "_WINDOWS", True)
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(command=f"{command}; touch {marker}")
|
|
|
|
assert out.startswith("Error:") and "conn_gpu" in out and "Nothing was run." in out
|
|
assert "203.0.113.7" not in out
|
|
assert not marker.exists()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"command",
|
|
[
|
|
'ssh -p 58717 "root@203.0.113.7" true',
|
|
'ssh -p "58717" root@203.0.113.7 true',
|
|
'ssh -p"58717" root@203.0.113.7 true',
|
|
'ssh -p 58717 "root@"203.0.113.7 true',
|
|
'ssh -p "587"17 root@203.0.113.7 true',
|
|
],
|
|
)
|
|
async def test_on_a_windows_host_a_quoted_destination_or_port_is_matched_like_a_bare_one(
|
|
registry, tmp_path, monkeypatch, command
|
|
):
|
|
# That shell takes the quotes off before the program sees the word; read with them on, the
|
|
# host and the port went unmatched and the command reached the executor.
|
|
monkeypatch.setattr(machine_exec, "_WINDOWS", True)
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
marker = tmp_path / "ran"
|
|
|
|
out = await tool.execute(command=f"{command}; touch {marker}")
|
|
|
|
assert out.startswith("Error:") and "conn_gpu" in out and "Nothing was run." in out
|
|
assert "203.0.113.7" not in out
|
|
assert not marker.exists()
|
|
|
|
|
|
def test_on_a_windows_host_the_words_are_read_without_their_quotes(monkeypatch):
|
|
monkeypatch.setattr(machine_exec, "_WINDOWS", True)
|
|
assert machine_exec._words('scp -P "58717" a.txt "root@h:/x" && -p"2222" \'two words\'') == [
|
|
"scp",
|
|
"-P",
|
|
"58717",
|
|
"a.txt",
|
|
"root@h:/x",
|
|
"&&",
|
|
"-p2222",
|
|
"two words",
|
|
]
|
|
# A quoted piece and a bare one next to it are one word, as that shell's argv has them; a backslash is kept.
|
|
assert machine_exec._words('ssh -p "587"17 "root@"h true 2>&1 C:\\dir\\scp.exe "C:\\Program Files\\x"') == [
|
|
"ssh",
|
|
"-p",
|
|
"58717",
|
|
"root@h",
|
|
"true",
|
|
"2",
|
|
">&",
|
|
"1",
|
|
"C:\\dir\\scp.exe",
|
|
"C:\\Program Files\\x",
|
|
]
|
|
assert machine_exec._words("scp 'unbalanced") == []
|
|
|
|
monkeypatch.setattr(machine_exec, "_WINDOWS", False)
|
|
assert machine_exec._words('scp -P "58717" a.txt "root@h:/x"') == ["scp", "-P", "58717", "a.txt", "root@h:/x"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_typed_ssh_s_refusal_names_file_tools_without_a_debug_switch(registry, tmp_path, monkeypatch):
|
|
tool = ExecTool(working_dir=str(tmp_path))
|
|
monkeypatch.delenv("RAVEN_DEBUG_MACHINES", raising=False)
|
|
said = await tool.execute(command="ssh -p 58717 root@203.0.113.7 ls")
|
|
assert "scp and rsync to it are still fine here" in said
|
|
assert "read_file, write_file and edit_file on the path the way scp writes it" in said
|