## Summary Add a Machines tab to the Agent Hub and let owners register an SSH machine with either a key or a password. - The Machines tab reuses the existing machine cards, details and registration form. The hub remembers the last selected module. Machine details show the login method, and removal explains that the locally stored password is deleted. - Password registrations keep a 0600 credential file under the Raven home's 0700 keys directory. Registration, node installation, file tools, tunnels and Oncall use the shared SSH askpass transport. Passwords are excluded from command lines, registry rows and responses, and included in held-secret scrubbing. - Password connections authenticate independently, disable inherited jump hosts, and answer only the registered account's password prompts. Oncall cannot redirect a stored password to another endpoint. Windows rejects this mode before creating credentials because its OpenSSH cannot execute Raven's POSIX helper. Key authentication remains supported as before. - Concurrent registrations preserve the successful request's password. Each probe uses its own temporary 0600 credential file, which is scrubbed and removed afterwards. Password writes receive a file identity; cleanup checks ownership under the writer's lock. Successful registration checks duplicates again and publishes its password with the registry row under that lock. Both completion orders, RPC cancellation and CLI aborts are covered. The RPC contract adds optional `MachineRow.auth` and password registration parameters. English and Chinese machine documentation, the runtime context and changelog describe the resulting behavior. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed After rebasing onto the latest main: ```bash uv run --no-sync pytest tests/test_ops_connection_add.py tests/test_ops_connections.py tests/test_ops_transport.py tests/test_cli_ops_connection_commands.py tests/test_rpc_machines.py tests/test_held_secrets_machine_passwords.py tests/test_acp_remote.py tests/test_node_client.py tests/test_rpc_schema_match.py tests/test_rpc_registration.py tests/test_import_cycle_budget.py -q -n 0 --tb=short --cov=raven.ops.connection_add --cov=raven.ops.transport --cov=raven.cli.ops_connection_commands --cov=raven.rpc.methods.machines --cov-report=json:/tmp/pr1005-fix-coverage.json ``` Result: 932 passed, 6 skipped. The skipped cases require zsh, tcsh or csh, which are absent on this Linux host. - `uv run --no-sync python scripts/coverage_gate.py --coverage-json /tmp/pr1005-fix-coverage.json diff --base-ref HEAD^ --threshold 99`: 100% (94/94 executable changed lines) for the concurrency fix. - `uv run --no-sync pre-commit run --from-ref HEAD^ --to-ref HEAD`: passed, including formatting, file checks and private-key detection. - `uv run --no-sync ty check raven/ops/connection_add.py raven/ops/transport.py raven/cli/ops_connection_commands.py raven/rpc/methods/machines.py`: passed. - `uv run --no-sync lint-imports`: all 10 contracts passed. - `uv run --no-sync python scripts/check_source_language.py origin/main..HEAD`, `uv run --no-sync python scripts/check_commit_messages.py origin/main..HEAD`, `make check-large-files`, and `git diff --check origin/main...HEAD`: passed. The original RPC failure/cancellation and CLI-abort reproductions failed before the fix. Additional regressions cover the reverse completion order, probes with competing passwords, temporary-secret scrubbing, duplicate registration and failed registry writes. The complete rebased source was compared with the original PR plus main, and the concurrency fix retained its patch identity. ## Risk - [x] Security impact considered: stored and temporary passwords are owner-readable and scrubbed from tool output. Unknown SSH host keys are accepted on the first connection, as before; the guides explain verifying the host from a terminal first on an untrusted network. Passwords shorter than six characters or containing CR/LF are rejected. - [x] Backward compatibility considered: key registration remains the default and `MachineRow.auth` is optional. Password publication and cleanup now use file locks and atomic replacement; temporary probe files are removed when the probe ends. - [x] Rollback path is clear: revert this PR to restore the earlier registration flow. Removing a machine deletes the password stored under its id. Existing password rows require migration to key authentication if reverting the feature. ## Related Issues N/A --------- Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com> Co-authored-by: Dizhan Xue <xuedizhan17@mails.ucas.ac.cn> Co-authored-by: MiniMax (minimax-m3.1-flash-free) <noreply@minimax.io> Co-authored-by: Codex <noreply@openai.com>
9.3 KiB
Manage long-running work with Oncall
Model training, solver runs, and parameter sweeps often need repeated status checks after submission, decisions about whether to continue or adjust the work, and a final report backed by measured results.
This guide shows how to use Raven-Oncall to manage that workflow within a declared budget and allowed actions: prepare a machine, delegate the task, follow its progress, stop when needed, and verify the outcome. Start with the bounded local-job example below before entrusting a longer run.
The initial delegation returns after the first turn, not when the job finishes. A DAG node returning from Oncall is therefore not proof that a remote artifact or final report is ready.
When to use Oncall
When a job needs multiple rounds of observation or adjustment, Raven-Oncall can execute it, inspect outputs, decide the next step within the declared budget and allowed actions, and report the measured outcome.
A campaign records that goal, target machine, execution setup, baseline, budget, trials, and follow-up state. The agent's scheduler can arrange another look; a scheduled look is not the job itself.
Prepare the agent and machine
Check Raven-Oncall through Agent Integrations, including its model credentials. Keep the host and agent runtime/scheduler available for follow-up work. Closing a browser is different from stopping the resident runtime.
The host manages a machine registry:
raven ops connection list
raven ops connection add
raven ops connection doctor
add is interactive and contacts the selected local/SSH machine by default.
It records its identity, access parameters, installed software, working paths,
budget unit, and concurrency. A non-interactive local example, after replacing
the executable and project paths:
raven ops connection add --id local-lab --name "Local lab" \
--transport local --software "Python at /absolute/path/to/python" \
--budget-unit minute --concurrency 1 --path /absolute/path/to/project \
--non-interactive
The registry is the path selected by RAVEN_CONNECTIONS. Otherwise a
sub-agent reads the one in the raven home (RAVEN_HOME, or ~/.raven) that the
host hands it, and any other instance -- the host itself, including one started
with --config -- reads the connections.json beside its config when there is
one, else the home. A first registration lands in the home. Sub-agents run on a
rendered config in a state directory of their own and inherit the home, so
they read the owner's registry rather than a copy of it. SSH registration can
also write a managed alias in ~/.ssh/config; the key path is recorded, not
the private key bytes. A machine registered with a password
(Working on registered machines) runs
campaigns too: their ssh and their rsync log in with the password Raven keeps
for it, through the same options and askpass hook as every other connection,
and a campaign carries the connection id and auth, never the password. The
address and account are the registry's: a campaign meta that gives such a
connection another host, port or user is refused before any ssh starts, since
the kept password goes only to the machine it was registered for. Do not put
keys or passwords in a task message.
An agent can register a machine too. The ops_connection_add tool (on in
raven-oncall and raven-code through tools.connectionAdd; off elsewhere) takes
what the owner said in conversation -- this computer or another, what they
call it, an address if another -- fills a port, user or key they left out from
their own ssh config, reaches the machine, and only then writes the row. When
the registry lists no machine that fits, the agent asks the owner rather than
looking for a way in with a raw ssh command. It never asks for a password: a
machine that takes one is the owner's to register, from the Agent Hub's
Machines tab or with raven ops connection add --auth password.
doctor validates registry usability; it is not a fresh execution test of
every remote environment. --skip-probe on add skips contacting the machine
and explicitly leaves connectivity unverified. The Oncall agent, not the
host's DAG registry, decides how to run a campaign on that machine.
Give a bounded work order
Start with a disposable local job and provide the following information:
| Field | Example |
|---|---|
| Machine and directory | Registered “Local lab”; a scratch project directory |
| Goal | Run the supplied benchmark and report whether the target was reached |
| Command and environment | Exact existing script, interpreter, and required inputs |
| Baseline and metric | Previous measured value, units, and whether higher/lower is better |
| Resource budget | Total machine minutes and allowed concurrency |
| Allowed actions | Read outputs and retry only the listed parameter changes |
| Stop/escalation conditions | Stop on budget exhaustion; ask before changing dependencies |
| Delivery | Return evidence, output locations, and remaining uncertainty |
Example request:
Delegate this to Raven-Oncall on the registered Local lab machine. Use the scratch project and existing smoke script I attached. Inspect and declare the setup before submitting. Allow at most ten machine minutes and one job at a time. Do not install packages or alter the baseline. Watch the result, stop if the budget is exhausted, and report the measured outcome and logs.
These are user requirements to translate into a campaign, not literal CLI flags. Inspect the declared setup: a budget mentioned only in prose is not evidence that a configured meter will enforce it.
What the agent does
Declare goal/setup -> submit or observe -> inspect status and outputs
-> schedule another look / adjust allowed parameters / ask the owner
-> stop or finish -> deliver the evidence
Relevant model tools inside Raven-Oncall, not host shell subcommands:
| Tool | Role |
|---|---|
ops_declare |
Record the campaign's target, setup, baseline, and budget |
ops_submit |
Submit a trial under the declared setup and checks |
ops_tune_status / ops_outputs |
Inspect campaign state and output evidence |
ops_check_later |
Schedule the next observation |
ops_campaigns / ops_note |
Find campaigns and record decisions |
ops_ask_owner |
Evaluate a request for human intervention |
ops_kill / ops_finish |
Stop work or close the campaign with a report |
The next-look key is the campaign: a new scheduled look replaces the pending one rather than adding unlimited polling jobs. If scheduling is unavailable or has no usable route, the tool reports that manual checking is needed. Do not interpret that notice as a successful unattended watch.
Budgets and approval boundaries
Campaigns can meter compute, wall-clock watch time, or observation count. Units and concurrency matter: ten GPU-minutes is not necessarily ten elapsed minutes. A campaign with no declared budget has no total-budget stop. These meters do not cap model-provider bills or every external cloud charge.
Oncall's declaration, execution checks, and escalation policies are distinct
from host tool permissions. In particular, an allowed
ops_ask_owner returns text for the model to deliver using messaging; it does
not prove the owner received it. General messaging remains a separate tool.
The shipped Oncall config does not enable a sandbox by default and does not confine all execution to the project directory. Use scoped machine accounts, scratch paths, and an isolated environment before enabling unattended writes. Host approval does not imply a separate human review of every child command.
Follow up, stop, and verify
Use instance collaboration to ask the same agent for campaign status, the pending observation, and the latest output evidence. When requesting a stop, name the campaign and ask it to check running jobs, cancel further observation, and report what actually stopped.
Closing a UI, canceling a host turn, or forgetting an instance is not a remote job kill. After a runtime interruption, inspect the saved ledger, remote processes, and scheduling route before resubmitting. Persisted campaign files do not guarantee every transport automatically resumes its watch.
Accept a final report only when it names the goal, baseline where applicable, actual measurements, termination reason, and retrievable output locations. A “done” message without evidence is not a benchmark result.
Troubleshooting
| Symptom | Check |
|---|---|
| Agent cannot select a machine | Registry id/name, access fields, and software paths |
| Submission refused | Declared setup, resource availability, budget, and immutable baseline checks |
| First reply arrives but the job continues | Expected first-turn behavior; inspect the campaign |
| No later update | Resident runtime, scheduler, wake route, and delivery channel |
| Budget seems different from elapsed time | Meter, unit, concurrency, and recorded consumption |
| Restart left uncertain state | Read ledger and remote status before retrying or killing anything |
Implementation evidence lives in agents/raven-oncall/plugins/oncall-flow/,
raven/ops/connections.py, raven/ops/connection_add.py,
raven/agent/tools/connection_add.py, and raven/cli/ops_connection_commands.py.