## Summary Adds `experimental/`, a first version of worker RSI: a Curator that shapes a Raven worker's harness from the task, the materials handed to it and feedback, the way a new employee is trained. The worker is a harness-of-harnesses. A root Raven harness faces the user and calls child harnesses, each a full Raven agent hosted over ACP, through the nodes of its playbooks. Every harness has the same four strategy facets (memory, planning, capability, action). The Curator reads the actual harnesses and generates code and config on those facets at both levels: the root candidate sets the playbooks and writes per-node requirements, each child is curated against the requirements of its nodes, and the set is checked on copies and activated together. Layout: - `curator/`: staged generation (understand, select, design, implement, repair), two-level composition, harness contracts, and `raven_adapter/`, the only layer that touches a worker's Raven runtime - `analyst/`: turns an evaluator's words and execution records into observable behavior requirements - `iteration/`: the generic loop (onboarding curation, then trial, review and curation each round) and a CLI for a human evaluator - `simulation/`: a simulated travel agency that runs the loop, plus one showcase case (`simulation/cases/s0925c/`: case notes, the exported transcript split by phase, and the round 1 and round 3 decks) - `docs/design.md` (structure, curation flow, composition sequence, the Raven connection) and `docs/rsi-iteration.md` (the loop and the Analyst) Key decisions: - No change under `raven/`, and `raven/` does not import `experimental/`. Generated artifacts land on Raven's existing extension points. The few Raven internals the adapter reads are listed in `docs/design.md` section 5.4. - The Curator explores a read-only source snapshot; the caller names the evaluation side to withhold from it, so the generic layers name no scenario. - The single-layer owner analysis is kept only to reproduce the showcase case and is marked for removal; the two-layer Analyst is the default. - The two decks are `.pptx` files of about 230 KB each. They pass `check-large-files`; flagging them in case they count as report assets under AGENTS.md section 7, in which case they can move out of git behind a link. - One follow-up for Raven 0.2.2: playbooks in prompt and rounds modes ship no fixed nodes, so node requirements are read from dag playbooks only. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification On the branch rebased onto the latest `main`: - `uv run pytest tests/test_harness_curator_*.py tests/test_analyst_*.py tests/test_iteration_*.py tests/test_simulation_*.py -q`: 368 passed - `uv run pytest -m integration tests/integration/test_harness_curator_*.py --deselect tests/integration/test_harness_curator_real_llm.py -q` (offline, stub providers): 57 passed - `uv run pytest -q` (whole suite, with the host proxy variables unset and `PYTHONTZPATH` set): 26692 passed, 2 failed. One was an order-dependent test of this PR and is fixed here; the other, `tests/test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing`, fails only because the machine runs as root and is unrelated. - `uv run ruff check` and `uv run ruff format --check` on the touched files: clean - `scripts/check_large_files.py origin/main..HEAD`, `scripts/check_source_language.py origin/main..HEAD`, `scripts/check_commit_messages.py origin/main..HEAD` and `npx commitlint --from origin/main --to HEAD`: all exit 0 - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk No user-visible change for Raven: `experimental/` is not imported by `raven/` and is not packaged in the wheel (`packages = ["raven"]`). `CONTEXT.md` gains one glossary entry. The Curator's exploration offers a shell only under an OS sandbox executor. Rollback is a revert of the squash commit; nothing else depends on `experimental/`. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Yuqi Yang <yiqiyang524@gmail.com> Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
462 lines
21 KiB
Python
462 lines
21 KiB
Python
"""Mine for the value case: run cultivation runs over disclosure plans within a budget, judge each, rank them.
|
|
|
|
A candidate is one disclosure plan: a chain preset, a partition given as JSON, or a random partition of the scenario's
|
|
materials (the owner still knows everything from the start and has handed everything over before the last round; only
|
|
the partition differs, see `experimental.simulation.__main__`). Each finished run is judged by
|
|
`experimental.simulation.value`; with `--stop-on-value` the first run that qualifies stops the others.
|
|
|
|
Each run gets its own Raven home under `--state-root` (the only place a configuration with credentials is written,
|
|
readable by its owner only) and its own record directory under `--runs-dir`. While runs go, their spend is read from
|
|
the model calls their Raven homes record; when the suite's total passes `--budget`, the running runs are stopped. After
|
|
a run ends, credentials are removed from the configuration copies the worker left in its records, and everything the
|
|
run left in its Raven home except that configuration is copied beside the records: the traces, the employee's working
|
|
directory (tickets, briefs, research, deck projects), the run log and the command line, so a run's records hold its
|
|
whole process. Then the cultivation record is exported and judged, and a summary is written.
|
|
"""
|
|
|
|
import argparse
|
|
import json
|
|
import os
|
|
import random
|
|
import shutil
|
|
import signal
|
|
import subprocess
|
|
import sys
|
|
import time
|
|
from dataclasses import dataclass, field
|
|
from pathlib import Path
|
|
|
|
from .__main__ import CHAINS
|
|
from .employee import FORWARDED
|
|
from .record import REDACTED, SUMMARY, build_record, export_record, redact
|
|
from .scenario import Scenario
|
|
|
|
KEPT_FILES = ("run.log", "command.json")
|
|
VALUE = "value.json"
|
|
|
|
|
|
# USD per million tokens (fresh input, output, cache-read input) for models whose provider reports no cost.
|
|
PRICES = {"deepseek/deepseek-flash": (0.30, 1.20, 0.006)}
|
|
|
|
|
|
def call_cost(attributes: dict, prices=PRICES) -> float:
|
|
"""A model call's cost: what the provider reported, else its tokens at the model's price (0 when unpriced)."""
|
|
try:
|
|
reported = float(attributes.get("llm.usage.cost_total") or 0)
|
|
except (TypeError, ValueError):
|
|
reported = 0.0
|
|
model = str(attributes.get("llm.model") or "")
|
|
price = prices.get(model) or next((p for name, p in prices.items() if name.split("/")[-1] == model), None)
|
|
if reported or not price:
|
|
return reported
|
|
tokens = [attributes.get(f"llm.usage.{key}") or 0 for key in ("input_tokens", "output_tokens", "cache_read_tokens")]
|
|
return sum(float(count) * rate for count, rate in zip(tokens, price, strict=True)) / 1_000_000
|
|
|
|
|
|
def spend(state: Path, records: Path | None = None, prices=PRICES) -> dict[str, float]:
|
|
"""Model spend of one run from the calls its Raven home recorded, split by who made them.
|
|
|
|
A child harness hosted beside the employee or one of its replicas keeps its own log under the run's records
|
|
(`children/<id>/native/traces/logs`, or under `replicas/<label>/`); every call there counts as a subagent's.
|
|
Replicas' own calls land in the run's Raven home with the employee's; once the home is gone, its logs are read
|
|
from the copy the records keep under `traces/`.
|
|
"""
|
|
parts = {"simulation": 0.0, "employee": 0.0, "subagents": 0.0}
|
|
logs = [(log, None) for log in (Path(state) / "traces" / "logs").glob("audit-spans*.log*")]
|
|
if not logs and records is not None:
|
|
logs = [(log, None) for log in (Path(records) / "traces").glob("audit-spans*.log*")]
|
|
if records is not None:
|
|
for owner in ("", "replicas/*/"):
|
|
found = Path(records).glob(f"{owner}children/*/native/traces/logs/audit-spans*.log*")
|
|
logs += [(log, "subagents") for log in found]
|
|
for log, fixed in logs:
|
|
for line in log.open(errors="replace"):
|
|
if '"llm.call"' not in line:
|
|
continue
|
|
try:
|
|
attributes = json.loads(line).get("attributes", {})
|
|
except ValueError:
|
|
continue
|
|
session = str(attributes.get("session.id") or "")
|
|
session = "" if session in ("None", "null") else session
|
|
who = fixed or ("subagents" if session.startswith("acp:") else ("employee" if session else "simulation"))
|
|
parts[who] += call_cost(attributes, prices)
|
|
return {**parts, "total": sum(parts.values())}
|
|
|
|
|
|
def _scrub(value) -> tuple[object, int]:
|
|
if isinstance(value, dict):
|
|
count = 0
|
|
out = {}
|
|
for key, item in value.items():
|
|
if (
|
|
isinstance(item, str)
|
|
and item
|
|
and (key.lower() in ("apikey", "api_key") or key.endswith("_API_KEY") or redact(item) != item)
|
|
):
|
|
out[key], count = REDACTED, count + 1
|
|
else:
|
|
out[key], found = _scrub(item)
|
|
count += found
|
|
return out, count
|
|
if isinstance(value, list):
|
|
pairs = [_scrub(item) for item in value]
|
|
return [item for item, _ in pairs], sum(found for _, found in pairs)
|
|
return value, 0
|
|
|
|
|
|
CONFIG_COPIES = ("config.json", "deployment.json", ".config.rendered.*.json")
|
|
|
|
|
|
def scrub(records: Path) -> int:
|
|
"""Redact credentials in every configuration copy the worker wrote anywhere under a run's records (the root and
|
|
each child harness keep their own, down to version folders); returns how many."""
|
|
total = 0
|
|
for pattern in CONFIG_COPIES:
|
|
for path in Path(records).rglob(pattern):
|
|
if path.is_symlink() or not path.is_file():
|
|
continue
|
|
try:
|
|
data, count = _scrub(json.loads(path.read_text()))
|
|
except ValueError:
|
|
continue
|
|
if not count:
|
|
continue
|
|
mode = path.stat().st_mode & 0o777
|
|
path.write_text(json.dumps(data, ensure_ascii=False, indent=2))
|
|
os.chmod(path, mode)
|
|
total += count
|
|
return total
|
|
|
|
|
|
def random_partition(materials: list[str], rounds: int, rng: random.Random, onboard=()) -> list[list[str]]:
|
|
"""A partition of every material into 2 to `rounds` non-empty steps, each material in its scenario order.
|
|
|
|
Materials in `onboard` always come at onboarding, so every round can reach the stages they serve.
|
|
"""
|
|
rest = [name for name in materials if name not in onboard]
|
|
if not rest:
|
|
raise ValueError("a partition needs a material outside the onboarding set")
|
|
while True:
|
|
count = rng.randint(2, max(2, min(rounds, len(rest) + 1)))
|
|
places = {name: 0 if name in onboard else rng.randrange(count) for name in materials}
|
|
steps = [[name for name in materials if places[name] == step] for step in range(count)]
|
|
if all(steps):
|
|
return steps
|
|
|
|
|
|
def candidates(args) -> list[tuple[str, list[str]]]:
|
|
"""Each disclosure plan to try, as a label and the simulation arguments that select it."""
|
|
found = [(chain, ["--chain", chain]) for chain in args.chains or ()]
|
|
given = json.loads(Path(args.partitions).read_text()) if args.partitions else []
|
|
found += [
|
|
(f"p{index}", ["--partition", json.dumps(steps, ensure_ascii=False)]) for index, steps in enumerate(given, 1)
|
|
]
|
|
if args.random_partitions:
|
|
scenario = Scenario.load(args.scenario)
|
|
materials = list(
|
|
scenario.without(args.without).materials if getattr(args, "without", None) else scenario.materials
|
|
)
|
|
onboard = list(getattr(args, "onboard", None) or ())
|
|
unknown = set(onboard) - set(materials)
|
|
if unknown:
|
|
raise ValueError(f"the scenario has no materials named {sorted(unknown)}")
|
|
rng = random.Random(args.seed)
|
|
for index in range(1, args.random_partitions + 1):
|
|
steps = random_partition(materials, args.rounds, rng, onboard)
|
|
found.append((f"r{args.seed}-{index}", ["--partition", json.dumps(steps, ensure_ascii=False)]))
|
|
return found
|
|
|
|
|
|
def command(plan: list[str], args, *, config: Path, workdir: Path, records: Path) -> list[str]:
|
|
"""The simulation command line for one plan; every model and size setting is explicit so the run is reproducible."""
|
|
argv = [sys.executable, "-m", "experimental.simulation", "--config", str(config), "--workdir", str(workdir)]
|
|
argv += ["--home", str(args.home), "--state-dir", str(records), "--scenario", str(args.scenario), *plan]
|
|
argv += ["--rounds", str(args.rounds), "--turns", str(args.turns), "--repeats", str(args.repeats)]
|
|
argv += ["--timeout", str(args.timeout)]
|
|
for flag, value in (
|
|
("--curator-model", args.curator_model),
|
|
("--analysis", getattr(args, "analysis", None)),
|
|
("--analyst-model", args.analyst_model),
|
|
("--simulation-model", args.simulation_model),
|
|
("--traveller-model", args.traveller_model),
|
|
("--curator-calls", getattr(args, "curator_calls", None)),
|
|
("--curator-queries", getattr(args, "curator_queries", None)),
|
|
("--subagent-model", args.subagent_model),
|
|
("--curator-effort", getattr(args, "curator_effort", None)),
|
|
("--traveller-effort", getattr(args, "traveller_effort", None)),
|
|
):
|
|
if value:
|
|
argv += [flag, str(value)]
|
|
if args.cards:
|
|
argv += ["--cards", *args.cards]
|
|
if getattr(args, "concurrent_drills", 1) > 1:
|
|
argv += ["--concurrent-drills", str(args.concurrent_drills)]
|
|
if getattr(args, "without", None):
|
|
argv += ["--without", *args.without]
|
|
return [*argv, "--seed", str(args.seed)]
|
|
|
|
|
|
@dataclass
|
|
class Run:
|
|
label: str
|
|
plan: list[str]
|
|
name: str
|
|
state: Path
|
|
records: Path
|
|
process: subprocess.Popen | None = None
|
|
started: float = 0.0
|
|
ended: float | None = None
|
|
stopped: str | None = None
|
|
notes: list[str] = field(default_factory=list)
|
|
|
|
@property
|
|
def log(self) -> Path:
|
|
return self.state / "run.log"
|
|
|
|
|
|
def start(run: Run, args) -> None:
|
|
run.state.mkdir(parents=True, exist_ok=False)
|
|
(run.state / "workdir").mkdir()
|
|
config = run.state / "config.json"
|
|
shutil.copyfile(args.config, config)
|
|
os.chmod(config, 0o600)
|
|
env = {**os.environ, "RAVEN_HOME": str(run.state)}
|
|
env.update({name: os.environ[name] for name in FORWARDED if os.environ.get(name)})
|
|
argv = command(run.plan, args, config=config, workdir=run.state / "workdir", records=run.records)
|
|
(run.state / "command.json").write_text(json.dumps(argv[1:], indent=2))
|
|
with run.log.open("w") as log:
|
|
run.process = subprocess.Popen(
|
|
argv, cwd=args.repository, env=env, stdout=log, stderr=subprocess.STDOUT, start_new_session=True
|
|
)
|
|
run.started = time.time()
|
|
|
|
|
|
def descendants(pid: int) -> list[int]:
|
|
"""Every process below `pid`, read from /proc: subagents start their own sessions, so a process group misses them."""
|
|
children: dict[int, list[int]] = {}
|
|
for stat in Path("/proc").glob("[0-9]*/stat"):
|
|
try:
|
|
fields = stat.read_text().rsplit(")", 1)[1].split()
|
|
except OSError:
|
|
continue
|
|
children.setdefault(int(fields[1]), []).append(int(stat.parent.name))
|
|
found, queue = [], [pid]
|
|
while queue:
|
|
for child in children.get(queue.pop(), []):
|
|
found.append(child)
|
|
queue.append(child)
|
|
return found
|
|
|
|
|
|
def _signal(pids, sig) -> None:
|
|
for pid in pids:
|
|
try:
|
|
os.kill(pid, sig)
|
|
except ProcessLookupError:
|
|
pass
|
|
|
|
|
|
def stop(run: Run, reason: str, grace: float = 20) -> None:
|
|
"""Stop a run's process group and every process below it, worker and subagents included; say why in its record."""
|
|
if run.process is None or run.process.poll() is not None:
|
|
return
|
|
run.stopped = reason
|
|
below = descendants(run.process.pid)
|
|
try:
|
|
os.killpg(run.process.pid, signal.SIGTERM)
|
|
except ProcessLookupError:
|
|
pass
|
|
_signal(below, signal.SIGTERM)
|
|
try:
|
|
run.process.wait(timeout=grace)
|
|
except subprocess.TimeoutExpired:
|
|
try:
|
|
os.killpg(run.process.pid, signal.SIGKILL)
|
|
except ProcessLookupError:
|
|
pass
|
|
_signal([pid for pid in below if Path(f"/proc/{pid}").exists()], signal.SIGKILL)
|
|
for record in (run.records / "iteration").glob("*.json"):
|
|
data = json.loads(record.read_text())
|
|
if data.get("status") == "running":
|
|
data["status"], data["error"] = "error", f"stopped by the suite: {reason}"
|
|
record.write_text(json.dumps(data, ensure_ascii=False, indent=2))
|
|
|
|
|
|
def finish(run: Run, args) -> dict:
|
|
"""Leave a run's records with its evidence and without credentials; returns its summary."""
|
|
run.ended = run.ended or time.time()
|
|
redacted = scrub(run.records) if run.records.is_dir() else 0
|
|
traces, workdir = run.state / "traces" / "logs", run.state / "workdir"
|
|
if run.records.is_dir():
|
|
if traces.is_dir():
|
|
shutil.copytree(traces, run.records / "traces", dirs_exist_ok=True)
|
|
if workdir.is_dir():
|
|
shutil.copytree(workdir, run.records / "workdir", dirs_exist_ok=True, symlinks=True)
|
|
for name in KEPT_FILES:
|
|
if (run.state / name).is_file():
|
|
text = (run.state / name).read_text(errors="replace")
|
|
(run.records / name).write_text(redact(text))
|
|
isolation = None
|
|
try:
|
|
from .isolation import write as check_isolation
|
|
|
|
isolation = check_isolation(run.records, run.state, args.forbid) if run.records.is_dir() else None
|
|
except Exception as exc: # noqa: BLE001 -- an unscanned run is judged not isolated, never lost
|
|
run.notes.append(f"isolation scan failed: {exc!r}")
|
|
from .files import page_images
|
|
|
|
decks = sorted(run.records.rglob("*.pptx")) if run.records.is_dir() else []
|
|
for deck in (deck for deck in decks if "deliverables" in deck.relative_to(run.records).parts):
|
|
try:
|
|
page_images(deck)
|
|
except Exception as exc: # noqa: BLE001 -- a deck without page images is still recorded
|
|
run.notes.append(f"no page images for {deck.name}: {exc!r}")
|
|
exported = None
|
|
try:
|
|
exported = str(export_record(run.records, run.records / "record", state_root=run.state.parent))
|
|
except Exception as exc: # noqa: BLE001 -- a failed export must not lose the run's summary
|
|
run.notes.append(f"record export failed: {exc!r}")
|
|
value = None
|
|
try:
|
|
value = build_record(run.records, state_root=run.state.parent)["value"]
|
|
(run.records / VALUE).write_text(json.dumps(value, ensure_ascii=False, indent=2))
|
|
except Exception as exc: # noqa: BLE001 -- an unjudged run still gets its summary
|
|
run.notes.append(f"value judgement failed: {exc!r}")
|
|
status, rounds = "unknown", 0
|
|
for record in (run.records / "iteration").glob("*.json"):
|
|
data = json.loads(record.read_text())
|
|
status, rounds = data.get("status", "unknown"), len(data.get("rounds") or [])
|
|
summary = {
|
|
"label": run.label,
|
|
"plan": run.plan,
|
|
"name": run.name,
|
|
"status": status,
|
|
"rounds": rounds,
|
|
"stopped": run.stopped,
|
|
"exit": run.process.returncode if run.process else None,
|
|
"seconds": round(run.ended - run.started),
|
|
"spend": spend(run.state, run.records),
|
|
"credentials_redacted": redacted,
|
|
"record": exported,
|
|
"value": {key: value[key] for key in ("verdict", "score")}
|
|
| {"cultivated": len(value.get("cultivated", [])), "cases": len(value["cases"])}
|
|
if value
|
|
else None,
|
|
"isolation": {key: isolation[key] for key in ("valid", "breaches")} if isolation else None,
|
|
"notes": run.notes,
|
|
}
|
|
if run.records.is_dir():
|
|
(run.records / SUMMARY).write_text(json.dumps(summary, ensure_ascii=False, indent=2))
|
|
return summary
|
|
|
|
|
|
def over_budget(runs: list[Run], budget: float | None) -> bool:
|
|
return budget is not None and sum(spend(run.state, run.records)["total"] for run in runs if run.started) >= budget
|
|
|
|
|
|
def main(args) -> list[dict]:
|
|
stamp = time.strftime("%m%d-%H%M")
|
|
runs = [
|
|
Run(
|
|
label,
|
|
plan,
|
|
f"{args.prefix}{label}-{stamp}",
|
|
args.state_root / f"{args.prefix}{label}-{stamp}",
|
|
args.runs_dir / f"{args.prefix}{label}-{stamp}",
|
|
)
|
|
for label, plan in candidates(args)
|
|
]
|
|
if not runs:
|
|
raise ValueError("name at least one chain, partition file or random partition")
|
|
for run in runs:
|
|
if run.records.exists() or run.state.exists():
|
|
raise ValueError(f"a run named {run.name} already exists")
|
|
pending, active, done = list(runs), [], []
|
|
while pending or active:
|
|
while pending and len(active) < args.parallel and not over_budget(runs, args.budget):
|
|
run = pending.pop(0)
|
|
start(run, args)
|
|
active.append(run)
|
|
print(f"started {run.name}: records {run.records}", flush=True)
|
|
if over_budget(runs, args.budget):
|
|
for run in active:
|
|
stop(run, f"the suite reached its budget of ${args.budget}")
|
|
pending.clear()
|
|
for run in list(active):
|
|
if run.process.poll() is not None:
|
|
active.remove(run)
|
|
summary = finish(run, args)
|
|
done.append(summary)
|
|
print(json.dumps(summary, ensure_ascii=False), flush=True)
|
|
if args.stop_on_value and (summary["value"] or {}).get("verdict") == "qualifies":
|
|
for other in active:
|
|
stop(other, f"{run.name} already showed the value case")
|
|
pending.clear()
|
|
time.sleep(args.poll)
|
|
ranking = sorted(done, key=lambda summary: -((summary["value"] or {}).get("score") or 0))
|
|
total = sum(summary["spend"]["total"] for summary in done)
|
|
(args.runs_dir / f"{args.prefix}mining-{stamp}.json").write_text(
|
|
json.dumps({"spend": total, "ranking": ranking}, ensure_ascii=False, indent=2)
|
|
)
|
|
print(f"mining finished: {len(done)} runs, ${total:.2f}", flush=True)
|
|
return ranking
|
|
|
|
|
|
def cli():
|
|
parser = argparse.ArgumentParser(description="Mine for the value case over disclosure plans within a budget.")
|
|
parser.add_argument("--config", type=Path, required=True, help="Raven JSON configuration with provider credentials")
|
|
parser.add_argument("--home", type=Path, required=True, help="The employee's baseline agent home")
|
|
parser.add_argument("--runs-dir", type=Path, required=True, help="Where each run's record directory is created")
|
|
parser.add_argument("--state-root", type=Path, required=True, help="Where each run's Raven home is created")
|
|
parser.add_argument("--scenario", type=Path, default=Path("travel_agency"))
|
|
parser.add_argument("--chains", nargs="+", choices=sorted(CHAINS), default=[])
|
|
parser.add_argument("--partitions", type=Path, help="A JSON file holding a list of partitions to try")
|
|
parser.add_argument("--random-partitions", type=int, default=0, help="How many random partitions to try")
|
|
parser.add_argument(
|
|
"--seed", type=int, default=1, help="Seed of the random partitions and of the values drawn for the drill cards"
|
|
)
|
|
parser.add_argument("--without", nargs="+", help="Leave these materials out of the scenario in every run")
|
|
parser.add_argument(
|
|
"--onboard",
|
|
nargs="+",
|
|
help="Materials every random partition gives at onboarding; the rest are staged at random",
|
|
)
|
|
parser.add_argument("--stop-on-value", action="store_true", help="Stop the other runs once one qualifies")
|
|
parser.add_argument("--prefix", default="", help="Prefix for run names")
|
|
parser.add_argument("--rounds", type=int, default=4)
|
|
parser.add_argument("--turns", type=int, default=14)
|
|
parser.add_argument("--repeats", type=int, default=1)
|
|
parser.add_argument("--cards", nargs="+")
|
|
parser.add_argument("--concurrent-drills", type=int, default=1, help="Drills each run plays at once in a round")
|
|
parser.add_argument("--timeout", type=float, default=3600)
|
|
parser.add_argument("--curator-model")
|
|
parser.add_argument("--analysis", choices=("owner", "analyst"))
|
|
parser.add_argument("--analyst-model")
|
|
parser.add_argument("--simulation-model")
|
|
parser.add_argument("--traveller-model")
|
|
parser.add_argument("--curator-calls", type=int)
|
|
parser.add_argument("--curator-queries", type=int)
|
|
parser.add_argument("--subagent-model")
|
|
parser.add_argument("--curator-effort", choices=("low", "medium", "high"))
|
|
parser.add_argument("--traveller-effort", choices=("low", "medium", "high"))
|
|
parser.add_argument(
|
|
"--budget", type=float, help="Stop every running run once the suite has spent this many dollars"
|
|
)
|
|
parser.add_argument(
|
|
"--forbid", type=Path, nargs="*", default=[], help="Places the employee must not reach (see isolation.py)"
|
|
)
|
|
parser.add_argument("--parallel", type=int, default=1)
|
|
parser.add_argument("--poll", type=float, default=60)
|
|
args = parser.parse_args()
|
|
args.config = args.config.expanduser().resolve()
|
|
args.home, args.runs_dir, args.state_root = (
|
|
path.expanduser().resolve() for path in (args.home, args.runs_dir, args.state_root)
|
|
)
|
|
args.repository = Path(__file__).resolve().parents[2]
|
|
main(args)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
cli()
|