Handsome-wzw 7b45527b5b fix(tools): an array sent as its own JSON is still an array
## Summary

Reported as "deliver_files is broken". It is not: **the tool never ran.**

Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:

```
Error: Invalid parameters for tool 'deliver_files': files should be array
```

What the model sent was the array, spelled as a string:

```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```

`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.

## Type

- [x] Fix
- [ ] Feature

## The edge of the forgiving is the design

When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.

**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.

**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.

## Not just deliver_files

Six built-in tools declare an array or object parameter and were all exposed:

| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |

The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.

## Verification

```
make lint-python    ruff check + ruff format --check, 1850 files
make lint-imports   9 contracts kept, 0 broken
pytest tests/       19,699 passed, 7 failed (all pre-existing, see below)
```

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string

**End to end, through the real tool**, with the exact payload that failed five times in production:

```
files sent as:            str -> [{"path": ".../poster.png", "title": "..."}]
returned:                 Delivered 1 file(s): poster.png (72 B).
still "should be array":  False
registered in the store:  True
```

Twelve cases, five mutations, each killing at least one:

| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |

The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.

**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.

## Risk

- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert

## Related Issues

N/A

## Swept

`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.

- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.

Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).

## Deliberately not fixed

The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.

## Moved from !594

Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.

Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00

410 lines
16 KiB
Python

"""The run/status/finalize state machine over durable artifacts.
Resume model: work is proven by artifacts, not by process state.
- phase 1 (cold start): proof = vanilla trial result files. Resume fills
only the missing trials.
- phase 2 (rounds): proof = journal/rounds.jsonl. Resume replays completed
rounds (loop.run's built-in journal replay) and continues.
- phase 3 (unseal): proof = the ``unsealed_at`` stamp in run_meta.json.
Unsealing is one-way — a stamped run refuses to resume (--force overrides,
marking any further rounds as retention-invalid is on the caller).
``status`` never reads the sealed directory: while a run is resumable, test
numbers must stay invisible (SOP §0) — the only path to them is natural
termination or an explicit ``finalize``.
"""
from __future__ import annotations
import shutil
import subprocess
import sys
from pathlib import Path
from evolver.launch.config import RunSpec, RunSpecError, load_run_spec
from evolver.launch.contract import BenchBundle, LaunchContext
from evolver.launch.models import build_role_call_fns, describe_models
from evolver.launch.registry import load_bench
from evolver.launch.state import RunMeta, atomic_write_json, load_json_or
from evolver.tree import git_ops
def _say(msg: str) -> None:
print(f"[evolve] {msg}", flush=True)
def _load_spec(config_path: str, smoke: bool) -> RunSpec:
try:
return load_run_spec(config_path, smoke=smoke)
except RunSpecError as exc:
print(f"config error: {exc}", file=sys.stderr)
raise SystemExit(2) from exc
def _build_bundle(spec: RunSpec, *, with_models: bool) -> BenchBundle:
try:
models = (
build_role_call_fns(spec.models)
if with_models
else {
"driver": None,
"design": None,
"verdict": None,
}
)
except ValueError as exc:
print(f"models error: {exc}", file=sys.stderr)
raise SystemExit(2) from exc
try:
build = load_bench(spec.bench, repo_root=spec.repo_root)
return build(LaunchContext(spec=spec, models=models))
except ValueError as exc:
print(f"bench setup error ({spec.bench}): {exc}", file=sys.stderr)
raise SystemExit(2) from exc
def _note_defaulted_base(spec: RunSpec) -> None:
if not spec.base_sha_defaulted:
return
_say(
f"base_sha not set — using repo HEAD {spec.base_sha[:12]} "
"(pin base_sha in the yaml to freeze the root explicitly)"
)
proc = subprocess.run(
["git", "-C", str(spec.repo_root), "status", "--porcelain", "--untracked-files=no"],
capture_output=True,
text=True,
)
if proc.stdout.strip():
_say(
"warning: repo_root has uncommitted changes — they are NOT part "
"of the root node (evaluations check out commits, not the "
"working tree)"
)
def _claim_ephemeral_root(spec: RunSpec) -> None:
"""Park ephemeral git worktrees under work_dir; sweep hard-kill leftovers.
Normal exits (including Ctrl-C) clean them via context managers — only a
SIGKILL'd run leaves them behind, still registered against the subject
repo. work_dir is per-run, so the sweep can only ever touch this run's
own garbage, never a concurrent run's live worktrees.
"""
tmp_root = Path(spec.work_dir) / "tmp"
if tmp_root.exists():
shutil.rmtree(tmp_root, ignore_errors=True)
subprocess.run(
["git", "-C", str(spec.repo_root), "worktree", "prune"],
capture_output=True,
text=True,
)
git_ops.set_ephemeral_root(tmp_root)
def _meta_guard(spec: RunSpec, *, force: bool) -> RunMeta:
"""Config-drift + one-way-unseal guards; returns the (possibly new) meta."""
snapshot = spec.snapshot()
meta = RunMeta.load(spec.work_dir)
if meta is None:
return RunMeta.create(spec.work_dir, snapshot)
if meta.unsealed_at and not force:
print(
f"this run was unsealed at {meta.unsealed_at} "
f"(reason: {meta.finalize_reason}); resuming would leak test "
"numbers into decisions. Start a fresh work_dir, or pass --force "
"to continue with retention marked invalid.",
file=sys.stderr,
)
raise SystemExit(2)
if not meta.check_config(snapshot) and not force:
print(
"config drift: the effective configuration differs from the one "
"this work_dir was started with (same-regime rule, SOP §0). "
"Candidate and control arms would no longer be comparable. "
"Use a fresh work_dir for the new config, or --force to override.",
file=sys.stderr,
)
if spec.base_sha_defaulted:
print(
"note: base_sha is omitted in the yaml, so it re-resolved to "
f"the current HEAD ({spec.base_sha[:12]}); if the repo gained "
"commits since the run started, pin base_sha to the original "
f"root ({meta.config_snapshot.get('base_sha', '?')[:12]}) to "
"resume.",
file=sys.stderr,
)
raise SystemExit(2)
return meta
def _run_rounds(spec: RunSpec, bundle: BenchBundle):
from evolver.orchestrator.state.journal import RoundJournal
orch = bundle.build_orchestrator()
journal = RoundJournal(bundle.journal_path)
result = orch.run(bundle.root_node_id, journal=journal, root_node=bundle.root_node)
return orch, journal, result
def _unseal_and_report(
spec: RunSpec, bundle: BenchBundle, orch, records: list[dict], meta: RunMeta, reason: str
) -> bool:
"""Returns True on success; False when unseal scoring failed (not stamped)."""
if bundle.unseal is None:
_say("no sealed test set configured; skipping unseal")
if not meta.unsealed_at:
meta.stamp_unsealed(reason=f"{reason} (no sealed test)")
return True
_say("unsealing: blind-scoring deliverables on the sealed test set …")
try:
report = bundle.unseal(records, orch)
except (RuntimeError, OSError, subprocess.CalledProcessError) as exc:
print(f"unseal scoring failed: {exc}", file=sys.stderr)
print(
"the run is NOT stamped; fix the environment and re-run the same command to retry the unseal",
file=sys.stderr,
)
return False
# Stamp before writing the report: the stamp is what stops a resume, so
# no crash window may leave test numbers on disk in a still-resumable
# run. A crash after the stamp leaves a final run with retention.json
# missing — `finalize` detects that and recomputes.
if not meta.unsealed_at:
meta.stamp_unsealed(reason=reason)
atomic_write_json(Path(spec.work_dir) / "retention.json", report)
_say(f"retention report -> {Path(spec.work_dir) / 'retention.json'}")
for key in (
"best_round",
"best_node_id",
"best_train",
"best_test",
"vanilla_train",
"vanilla_test",
"retention",
"sealed_credited_2sigma",
):
if isinstance(report, dict) and key in report:
_say(f" {key}: {report[key]}")
return True
def cmd_run(config_path: str, *, smoke: bool = False, force: bool = False) -> int:
spec = _load_spec(config_path, smoke)
_note_defaulted_base(spec)
# Build (validate) the bundle before creating run_meta: a first launch
# that dies on a config mistake must not leave a fingerprint behind, or
# the corrected config would be refused as drift on the next attempt.
bundle = _build_bundle(spec, with_models=True)
spec.work_dir.mkdir(parents=True, exist_ok=True)
_claim_ephemeral_root(spec)
meta = _meta_guard(spec, force=force)
meta.config_snapshot.setdefault("resolved_models", describe_models(spec.models))
meta.save()
done, total = bundle.cold_start_done(), bundle.cold_start_total
if done < total:
_say(f"phase 1/3 cold start: {done}/{total} trials present, running the rest …")
else:
_say(f"phase 1/3 cold start: {total} trials present, verifying the infra-rerun ladder …")
# Always invoked: run_cold_start is idempotent (fills missing trials only)
# and owns the infra-rerun ladder, which may have salvage work to do even
# when every base trial file exists.
try:
bundle.run_cold_start()
except KeyboardInterrupt:
_say(
f"interrupted during cold start "
f"({bundle.cold_start_done()}/{total} trials done and kept); "
"re-run the same command to continue"
)
return 130
except (subprocess.CalledProcessError, OSError, RuntimeError) as exc:
print(f"cold start failed: {exc}", file=sys.stderr)
print(
f"completed trials are kept "
f"({bundle.cold_start_done()}/{total} present); fix the "
"environment and re-run the same command to resume",
file=sys.stderr,
)
return 1
done = bundle.cold_start_done()
if done < total:
_say(f"cold start still incomplete ({done}/{total}); re-run to retry the missing trials")
return 1
_say(f"phase 1/3 cold start complete ({total} trials)")
_say("phase 2/3 evolution rounds (interrupt any time; same command resumes)")
_say(
f" live progress: {spec.work_dir}/findings.md (per-round log), "
f"{spec.work_dir}/journal/rounds.jsonl (checkpoints)"
)
try:
orch, journal, result = _run_rounds(spec, bundle)
except KeyboardInterrupt:
_say(
"interrupted — completed rounds are journaled; "
"re-run the same command to resume, `status` to inspect, "
"`finalize` to stop here and unseal"
)
return 130
except RuntimeError as exc:
# Environment-shaped failures (Gate0 precheck, dead endpoint) are
# actionable messages, not tracebacks; completed work is durable.
print(f"run stopped: {exc}", file=sys.stderr)
print("fix the environment and re-run the same command to resume", file=sys.stderr)
return 1
for rr in result.rounds:
_say(f"round {rr.round_index}: parent={rr.parent_id} promoted={rr.promoted} -> {rr.next_parent_id}")
_say(f"stopped: {result.stop_reason}; final parent: {result.final_parent_id}")
_say("phase 3/3 unseal")
ok = _unseal_and_report(spec, bundle, orch, journal.load(), meta, reason=result.stop_reason or "terminated")
return 0 if ok else 1
def cmd_check(config_path: str, *, smoke: bool = False) -> int:
"""Validate everything cheap before spending: config, models, bench setup.
Builds the model call_fns (catches a missing claude binary / bad spec) and
the bench bundle (catches dead whitelist prefixes, missing task files or
subject config, absent AppWorld install) without running anything.
"""
spec = _load_spec(config_path, smoke)
_note_defaulted_base(spec)
bundle = _build_bundle(spec, with_models=True)
_say(f"bench: {spec.bench} (root {spec.base_sha[:12]} @ {spec.repo_root})")
_say(f"work_dir: {spec.work_dir}")
_say(f"models: {describe_models(spec.models)}")
_say(
f"funnel: k_screen={spec.funnel.k_screen} k_confirm={spec.funnel.k_confirm} "
f"budget={spec.funnel.budget.max_why_per_round}x"
f"{spec.funnel.budget.candidates_per_why} "
f"rounds<={spec.funnel.termination.max_rounds}"
)
_say(f"cold start: {bundle.cold_start_done()}/{bundle.cold_start_total} trials present")
_say(f"sealed test: {'configured' if bundle.unseal else 'not configured'}")
if bundle.precheck is not None:
_say("bench precheck: probing environment + subject endpoint …")
try:
bundle.precheck()
except RuntimeError as exc:
print(f"bench precheck failed: {exc}", file=sys.stderr)
return 1
_say("bench precheck: OK")
_say("check OK — ready to run")
return 0
def _node_status_counts(work_dir: Path) -> dict[str, int]:
counts: dict[str, int] = {}
for path in sorted(Path(work_dir).glob("nodes/*.json")):
rec = load_json_or(path, {})
status = rec.get("status", "?")
counts[status] = counts.get(status, 0) + 1
return counts
def cmd_status(config_path: str, *, smoke: bool = False) -> int:
spec = _load_spec(config_path, smoke)
meta = RunMeta.load(spec.work_dir)
if meta is None:
_say(f"no run state under {spec.work_dir} (phase 0: not started)")
return 0
_say(f"work_dir: {spec.work_dir}")
_say(f"started: {meta.created_at} config: {meta.config_hash}")
if meta.unsealed_at:
_say(f"UNSEALED at {meta.unsealed_at} ({meta.finalize_reason}) — run is final")
report = load_json_or(Path(spec.work_dir) / "retention.json", None)
if report:
_say(f"retention report: {Path(spec.work_dir) / 'retention.json'}")
return 0
bundle = _build_bundle(spec, with_models=False)
done, total = bundle.cold_start_done(), bundle.cold_start_total
if done < total:
_say(f"phase 1: cold start {done}/{total} trials — no results yet")
return 0
from evolver.orchestrator.state.journal import RoundJournal
records = RoundJournal(bundle.journal_path).load()
if not records:
_say("phase 2: cold start done, no completed rounds yet")
return 0
_say(f"phase 2: {len(records)} completed round(s); test stays sealed until termination or `finalize`")
counts = _node_status_counts(spec.work_dir)
if counts:
_say("candidates by status: " + ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
promoted = []
for rec in records:
_say(
f" round {rec.get('round_index')}: promoted={rec.get('promoted')} "
f"beat_vanilla={rec.get('beat_vanilla')} "
f"parent -> {rec.get('next_parent_id')}"
)
if rec.get("promoted") and rec.get("next_parent_sha"):
promoted.append((rec.get("next_parent_id"), rec.get("next_parent_sha"), rec.get("next_parent_train")))
if promoted:
_say("promoted commits (train-side numbers only):")
for node_id, sha, train in promoted:
_say(f" {node_id} @ {sha} train={train}")
return 0
def cmd_finalize(config_path: str, *, smoke: bool = False, yes: bool = False) -> int:
spec = _load_spec(config_path, smoke)
meta = RunMeta.load(spec.work_dir)
if meta is None:
print("nothing to finalize: run was never started", file=sys.stderr)
return 2
recompute = bool(meta.unsealed_at)
if recompute and (Path(spec.work_dir) / "retention.json").is_file():
_say(f"already unsealed at {meta.unsealed_at}; see retention.json")
return 0
bundle = _build_bundle(spec, with_models=False)
if recompute:
if bundle.unseal is None:
# Finalized without a sealed test: there is no report to rebuild.
_say(f"already finalized at {meta.unsealed_at} ({meta.finalize_reason}); no sealed test was configured")
return 0
_say(
"unseal stamp present but retention.json is missing (interrupted "
"unseal); recomputing the report — the run stays final"
)
from evolver.orchestrator.state.journal import RoundJournal
records = RoundJournal(bundle.journal_path).load()
if not records:
print("nothing to finalize: no completed rounds in the journal", file=sys.stderr)
return 2
if not yes and not recompute:
what = (
"and unseal the test set"
if bundle.unseal is not None
else "(no sealed test configured — no test numbers exist)"
)
print(
f"finalize will END this run after {len(records)} round(s) {what} "
"— it cannot be resumed afterwards. Re-run with --yes.",
file=sys.stderr,
)
return 2
# Unsealing scores nodes on test: the orchestrator (hence models for its
# construction path) is not needed, but the bench's eval is — the bundle
# closures carry it. vanilla_train comes from the built orchestrator, so
# build it without LLM roles (they are only called during rounds).
_claim_ephemeral_root(spec)
orch = bundle.build_orchestrator()
ok = _unseal_and_report(spec, bundle, orch, records, meta, reason="user_finalized")
return 0 if ok else 1
__all__ = ["cmd_run", "cmd_status", "cmd_check", "cmd_finalize"]