Raven-1/evolver/README.md
Handsome-wzw 7b45527b5b fix(tools): an array sent as its own JSON is still an array
## Summary

Reported as "deliver_files is broken". It is not: **the tool never ran.**

Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:

```
Error: Invalid parameters for tool 'deliver_files': files should be array
```

What the model sent was the array, spelled as a string:

```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```

`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.

## Type

- [x] Fix
- [ ] Feature

## The edge of the forgiving is the design

When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.

**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.

**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.

## Not just deliver_files

Six built-in tools declare an array or object parameter and were all exposed:

| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |

The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.

## Verification

```
make lint-python    ruff check + ruff format --check, 1850 files
make lint-imports   9 contracts kept, 0 broken
pytest tests/       19,699 passed, 7 failed (all pre-existing, see below)
```

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string

**End to end, through the real tool**, with the exact payload that failed five times in production:

```
files sent as:            str -> [{"path": ".../poster.png", "title": "..."}]
returned:                 Delivered 1 file(s): poster.png (72 B).
still "should be array":  False
registered in the store:  True
```

Twelve cases, five mutations, each killing at least one:

| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |

The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.

**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.

## Risk

- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert

## Related Issues

N/A

## Swept

`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.

- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.

Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).

## Deliberately not fixed

The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.

## Moved from !594

Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.

Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00

12 KiB

evolver — harness self-evolution

Status: retirement of this tree is planned alongside the vendored fork trees (the step-8 retirement wave), pending partner sign-off -- the C3 confirmation item recorded in the oncall migration closure (2026-09-01). Until that wave lands, this README describes the tool as it stands.

A budget-bounded loop that improves an agent harness against a benchmark: diagnose failing trajectories, design candidate patches, screen them cheaply, confirm survivors at K=3, and promote only what beats the baseline through three verification gates — with a sealed test set for an honest generalisation number. The methodology is specified in docs/specs/self-evolution-loop-sop.md (the SOP) and mapped to this codebase in docs/specs/self-evolution-loop-raven-mapping.md.

Two ways to run self-evolution

The SOP is the methodology; this package is one executor of it. There are two:

  1. The evolver pipeline (this package) — the automated loop below. The orchestrator executes the whole funnel unattended: diagnosis, candidate design, screening, K=3 confirmation, the three gates, promotion, and the sealed-test unseal, with resumability and the guardrails (path whitelist, fixed denominators, config fingerprint) enforced in code. Use this for long multi-round runs where the numbers have to be trustworthy.
  2. An agent driving the SOP directly — hand docs/specs/self-evolution-loop-sop.md to a coding agent (e.g. Claude Code) inside the subject repo and have it walk the steps itself: read failing trajectories, pick a WHY, write a candidate as a git commit, run the benchmark scorer for the K evaluations, and apply the gate arithmetic by hand. Nothing enforces the protocol — sealed-test discipline, fixed denominators, and honest gate math are only as good as the agent's adherence — so treat results as exploratory. It is the fastest way to prototype on a benchmark that has no BenchBundle plugin yet, and what it learns (WHY taxonomy, whitelist, scorer wiring) feeds directly into writing one (docs/specs/evolve-bench-contract.md).

The rest of this README covers way 1.

Glossary (terms used below)

  • WHY — a failure-cause class (e.g. "stops before verifying"); each round targets one or two WHYs. WHERE — the code location a patch touches.
  • K — attempts per task per evaluation; K=3 confirm is the SOP standard.
  • The funnel — cheap screen (K=1 on a small anchor task set) before the expensive full-train K=3 confirmation; most candidates die cheap.
  • Three gates — Gate-f: enough valid (non-infra) measurements; Gate-b: the candidate's code actually executed (beacon fired); Gate2: paired statistical significance vs the baseline on the same tasks.
  • Sealed test — the test split is scored but stored where no decision step can read it; opened once, at the end (retention.json).
  • BenchBundle — the plugin object a benchmark implements to become evolvable (see "Adding a benchmark").

Quickstart

All commands run from the repo checkout root (they invoke the evolver module; from elsewhere, add --project /path/to/checkout to uv run).

# 1. Write a run spec (copy docs/examples/evolve_appworld.yaml and edit paths)
# 2. Validate everything cheap — config, models, bench setup, plus one small
#    probe completion against the subject endpoint; no trials run:
uv run python -m evolver check  --config my_run.yaml

# 3. Tiny wiring run (isolated <work_dir>_smoke; minutes, not hours):
uv run python -m evolver run    --config my_run.yaml --smoke

# 4. The real run — one command does all three phases:
#    cold-start baseline (if missing) -> evolution rounds -> unseal + retention
uv run python -m evolver run    --config my_run.yaml

A real run takes hours to days, and the cost is dominated by subject-agent evaluations, not LLM calls: cold start = train x K trials (e.g. 90 x 3), plus roughly a full train x K evaluation per candidate that survives screening. The loop's own calls per round are modest — tens of driver calls (diagnosis), one design call per candidate (2-6, on your design model, typically the expensive one), a handful of verdict calls. Interrupt freely — re-running the same command resumes from the last durable artifact: completed trials are never re-run (trial-level idempotency), completed rounds are replayed from the journal.

uv run python -m evolver status   --config my_run.yaml   # progress; never
                                                               # reveals test numbers
uv run python -m evolver finalize --config my_run.yaml --yes
                                                               # end now + unseal (one-way)

What you get at the end: the console summary plus, under work_dir/, retention.json (the sealed-test verdict: best round, train/test curve, retention rate, paired significance), nodes/*.json (every candidate's ledger: git commit, final status, gate stats), findings.md (human-readable per-round log), and the promoted harness as a real git commit in your subject repo.

Interruption semantics

Interrupted during What exists status shows
phase 1 (cold start) partial baseline trials (kept) trial count, "no results yet"
phase 2 (rounds) commit list: promoted nodes + train scores rounds, candidates by status — test stays sealed
phase 3 (unseal) everything the final report

While a run is resumable, test numbers are physically withheld (they sit in a directory nothing in the decision path reads). finalize is the explicit trade: end the run now, see the result, never resume — the one-way stamp in run_meta.json enforces it. The same file records a config fingerprint: a run refuses to resume under a changed configuration (candidate and control arms must stay comparable).

Config file

See docs/examples/evolve_appworld.yaml for the annotated schema. The sections: bench (registered benchmark name), repo_root/base_sha (the subject repo being evolved and its root commit; omit base_sha to use the repo's current HEAD, resolved once at launch and pinned for the whole run — uncommitted changes are never part of the root), models (the loop's driver/design/verdict brains — omit entirely to use Raven's own configured model; note this is distinct from the subject agent's model, which is pinned inside the bench config for the whole run), funnel (K values, per-round budget, termination), bench_config (bench-owned), and smoke (overlay applied by --smoke).

Bootstrap: AppWorld

The shipped example bench. One-time setup:

  1. Install AppWorld into a directory of its own — it is the subject environment, kept out of this repo's uv env. In that directory: create a venv at appworld-venv/, pip install appworld && appworld install, then appworld download data — it must end up with a data/ folder. Point bench_config.appworld_data_root at the directory.

  2. Write the subject runtime config JSON (the benchmarked agent's model endpoint — not the loop's models). Copy docs/examples/subject_runtime.json and fill in your endpoint; check validates it against the schema. Point bench_config.config_path at it.

  3. Put train (and optionally test) task ids in text files, one id per line. Valid ids come from AppWorld's official splits:

    APPWORLD_ROOT=<install> <install>/appworld-venv/bin/python -c \
      "from appworld import load_task_ids; print('\n'.join(load_task_ids('train')))" > train.txt
    

    (APPWORLD_ROOT matters: appworld resolves data/ relative to the current directory otherwise. The bench exports it automatically at run time, but this manual step happens before any bench code runs.)

    (splits: train, dev, test_normal, test_challenge; using dev as the sealed test file is the usual setup). Placeholder ids like <failing-task-1> are refused at check time.

  4. check, then run --smoke, then run. For the smoke: section's pinned tasks, any 2-3 train ids work for a wiring test; ids your agent fails make the smoke round exercise the full design path (you learn which ones from the cold start of a real run, or a quick manual batch).

Adding a benchmark

Implement one build(ctx) -> BenchBundle function and register it — the contract (scorer subprocess, result reader with an infra marker, trajectory renderer, splits, editable-path whitelist) is documented in docs/specs/evolve-bench-contract.md. benchmarks/appworld/evolve/entry.py is the reference implementation.

Documentation map

Document What it is
this README user-facing: quickstart, config, bootstrap, security
docs/specs/self-evolution-loop-sop.md the methodology spec (SOP; English translation of the upstream document)
docs/specs/self-evolution-loop-raven-mapping.md SOP clause -> Raven code, deviations, unwired parts
docs/specs/evolve-bench-contract.md what a new benchmark implements
orchestrator/DESIGN.md design notes: inversion of control, SOP<->module table, cross-module conventions

Status and known limitations (2026-07)

  • Verified: the unit/e2e suite (126 tests: gate math, the infra-rerun ladder, git commit/worktree primitives, sandbox whitelist capture, driver transports, config, guards, idempotency, interrupt -> resume -> finalize with a fake bench) plus a real AppWorld smoke through the unified entry: cold start -> resume -> one full round (real diagnose/design, a real candidate commit, focused probe + confirm, promoted with an honest credited=False Gate-b verdict) -> termination.
  • Not yet exercised at scale: a full-size run (90 tasks x K=3, multi-round, hours) and the sealed-test unseal/retention path on a real benchmark have only been exercised via tests, not a production run.
  • One bench example ships today (AppWorld, built-in scorer line); a framework-line example (external scorer) is planned.

Security notes

This system has an LLM edit and execute code. Understand the boundaries before running it anywhere sensitive:

  • Candidate edits are constrained to a per-bench path whitelist (path_guard / sandbox capture); everything else the designer touches is reverted, and each whitelist prefix is validated at startup. On top of the whitelist, an immutable kernel (applier/path_guard.py) carves out the measurement surface — the evolver itself, and for AppWorld the scorer (evolve/ including grade.py, and batch.py) — so a candidate can rewrite the agent (prompt, loop wiring, tools) but never the code that grades it or records infra failures.
  • Threat model: these guards stop an LLM designer from taking shortcuts, not a determined adversary. Candidate code runs in the same process as the (immutable-on-disk) grader and the benchmark oracle is reachable over HTTP, so in-memory tampering or oracle probing is possible in principle. Promoted candidates are a handful of small commits — diff-audit them before citing numbers (nodes/*.json records every commit).
  • Candidate code runs during evaluation with the same privileges as your benchmark scorer, and the design-step sandbox is not filesystem/network jailed. Run evolutions in an isolated environment (container/VM, scoped credentials), not on a workstation with live secrets.
  • Python candidates are required to carry an activation_beacon() call, and credit is only assigned when the beacon actually fired (Gate-b) — this is an attribution mechanism (presence-level, see the mapping doc), not a sandbox.
  • The default per-round baseline is frozen at cold start (cost-bound); if your subject endpoint's throughput drifts across a long run, set bench_config.baseline_mode: same_session (~2x eval cost) so control and candidate arms are always measured in the same window.
  • The design step sends your failing trajectories to whatever model you configure; treat trajectory content accordingly.