Raven-1/evolver/__init__.py

35 lines
1.8 KiB
Python
Raw Permalink Normal View History

feat(*): self-evolution stack: evolver, appworld bench, unified entry (#140) ## Summary Port the harness self-evolution framework onto Raven and grow it into a runnable, benchmark-pluggable stack: a budget-bounded loop that diagnoses failing trajectories, designs candidate patches as real git commits, screens them cheaply, confirms survivors, and promotes only what passes three verification gates -- with a sealed test set for an honest generalisation number. - **Evolution machinery** (`raven/evolver/`): orchestrator state machine (seven-step funnel), three verification gates (infra validity, beacon attribution, paired significance), sealed-test runner with retention, git-backed candidate tree, judge, taxonomy, activation ledger. - **AppWorld bench** (`benchmarks/appworld/`): agent_cli, batch scorer with trial-level resume and an infra-rerun ladder, plus the evolve glue (designer, diagnosis, adapter, Gate0 precheck with real-generation probing). - **Unified entry**: `python -m raven.evolver {check,run,status,finalize}` drives cold start -> rounds -> terminate -> unseal as a resumable state machine. Config drift and unseal one-wayness are mechanized in run meta; benches plug in via a documented contract (`docs/specs/evolve-bench-contract.md`). - **Designer feedback**: inert-death loop (pruned_inert status, per-WHY history, gentle decay) so unreachable triggers teach the next attempt. - **Launch hygiene**: `check` runs the bench Gate0 precheck, so a dead subject endpoint fails at check time instead of at cold start; ephemeral eval/edit worktrees live under `work_dir/tmp` and hard-kill leftovers are swept on the next launch of the same run. - **Docs**: SOP translation (internal details redacted), SOP-to-code map, bench plugin contract, and a user README with security notes and known limitations. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed Commands run: - `uv run pytest tests/test_evolver_launch.py tests/test_evolver_launch_e2e.py -q` -> 56 passed (config, guards, trial-level idempotency, interrupt -> resume -> finalize with a fake bench, check precheck pass/fail, worktree sweep). - `uv run ruff check` on all touched files -> no new findings versus base (one S607 partial-path git call matching the file's existing idiom). - Real AppWorld end-to-end through the unified entry, from a fresh config written by following the README user path: `check` -> `run --smoke` (~16 min: cold start on 3 tasks, one full round with real diagnose/design, a real 1-line candidate commit, focused probe + confirm, promoted with an honest `credited=False` Gate-b verdict) -> termination. - Real `check` both ways: live subject endpoint -> `bench precheck: OK`; config pointed at a dead (NXDOMAIN) endpoint -> exit 1 at check time with an actionable `subject endpoint unreachable` message. - The bench placement (repo-root `benchmarks/`, outside the wheel, matching this directory's decoupling convention) was verified with a second real smoke: all three phases ran against a checkout of this layout, worktree evals resolved `python -m benchmarks.appworld.batch` from the checkout, and the designer wrote its candidate to the new whitelist path (`benchmarks/appworld/agent_cli.py`) as a real commit. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes Security: this system has an LLM edit and execute code. Candidate edits are constrained to a per-bench path whitelist, an immutable kernel protects the measurement surface (the evolver itself and the scorer), and the README documents the threat model plus the recommendation to run evolutions in an isolated environment. Backward compatibility: purely additive -- new packages plus one optional `BenchBundle.precheck` field; nothing existing changes behavior. Rollback: revert the branch; no migrations, no config changes to existing features. ## Related Issues N/A --- Reviewer notes: - Exercised at smoke scale only: a full-size run (90 tasks x K=3, multi-round, hours) and the sealed-test unseal/retention path on a real benchmark are covered by tests but not yet by a production run; the README states this under "Status and known limitations". - One bench ships today (AppWorld, built-in scorer line); a framework-line example (external scorer) is planned as a follow-up. --------- Co-authored-by: Ethan Wang <ethan.wang@evermind.ai> Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com> Co-authored-by: Elliot Chen <2340896+cyfyifanchen@users.noreply.github.com>
2026-07-17 19:37:05 +08:00
"""Harness self-evolution subsystem.
A budget-bounded loop that improves an agent harness against a benchmark:
diagnose failing trajectories, design candidate patches as real git commits,
screen them cheaply, confirm survivors at K=3, and promote only what beats
the baseline through three verification gates — with a sealed test set for
an honest generalisation number. Methodology:
``docs/specs/self-evolution-loop-sop.md``; user-facing entry point and
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
quickstart: ``evolver/README.md`` (``python -m evolver``).
feat(*): self-evolution stack: evolver, appworld bench, unified entry (#140) ## Summary Port the harness self-evolution framework onto Raven and grow it into a runnable, benchmark-pluggable stack: a budget-bounded loop that diagnoses failing trajectories, designs candidate patches as real git commits, screens them cheaply, confirms survivors, and promotes only what passes three verification gates -- with a sealed test set for an honest generalisation number. - **Evolution machinery** (`raven/evolver/`): orchestrator state machine (seven-step funnel), three verification gates (infra validity, beacon attribution, paired significance), sealed-test runner with retention, git-backed candidate tree, judge, taxonomy, activation ledger. - **AppWorld bench** (`benchmarks/appworld/`): agent_cli, batch scorer with trial-level resume and an infra-rerun ladder, plus the evolve glue (designer, diagnosis, adapter, Gate0 precheck with real-generation probing). - **Unified entry**: `python -m raven.evolver {check,run,status,finalize}` drives cold start -> rounds -> terminate -> unseal as a resumable state machine. Config drift and unseal one-wayness are mechanized in run meta; benches plug in via a documented contract (`docs/specs/evolve-bench-contract.md`). - **Designer feedback**: inert-death loop (pruned_inert status, per-WHY history, gentle decay) so unreachable triggers teach the next attempt. - **Launch hygiene**: `check` runs the bench Gate0 precheck, so a dead subject endpoint fails at check time instead of at cold start; ephemeral eval/edit worktrees live under `work_dir/tmp` and hard-kill leftovers are swept on the next launch of the same run. - **Docs**: SOP translation (internal details redacted), SOP-to-code map, bench plugin contract, and a user README with security notes and known limitations. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed Commands run: - `uv run pytest tests/test_evolver_launch.py tests/test_evolver_launch_e2e.py -q` -> 56 passed (config, guards, trial-level idempotency, interrupt -> resume -> finalize with a fake bench, check precheck pass/fail, worktree sweep). - `uv run ruff check` on all touched files -> no new findings versus base (one S607 partial-path git call matching the file's existing idiom). - Real AppWorld end-to-end through the unified entry, from a fresh config written by following the README user path: `check` -> `run --smoke` (~16 min: cold start on 3 tasks, one full round with real diagnose/design, a real 1-line candidate commit, focused probe + confirm, promoted with an honest `credited=False` Gate-b verdict) -> termination. - Real `check` both ways: live subject endpoint -> `bench precheck: OK`; config pointed at a dead (NXDOMAIN) endpoint -> exit 1 at check time with an actionable `subject endpoint unreachable` message. - The bench placement (repo-root `benchmarks/`, outside the wheel, matching this directory's decoupling convention) was verified with a second real smoke: all three phases ran against a checkout of this layout, worktree evals resolved `python -m benchmarks.appworld.batch` from the checkout, and the designer wrote its candidate to the new whitelist path (`benchmarks/appworld/agent_cli.py`) as a real commit. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes Security: this system has an LLM edit and execute code. Candidate edits are constrained to a per-bench path whitelist, an immutable kernel protects the measurement surface (the evolver itself and the scorer), and the README documents the threat model plus the recommendation to run evolutions in an isolated environment. Backward compatibility: purely additive -- new packages plus one optional `BenchBundle.precheck` field; nothing existing changes behavior. Rollback: revert the branch; no migrations, no config changes to existing features. ## Related Issues N/A --- Reviewer notes: - Exercised at smoke scale only: a full-size run (90 tasks x K=3, multi-round, hours) and the sealed-test unseal/retention path on a real benchmark are covered by tests but not yet by a production run; the README states this under "Status and known limitations". - One bench ships today (AppWorld, built-in scorer line); a framework-line example (external scorer) is planned as a follow-up. --------- Co-authored-by: Ethan Wang <ethan.wang@evermind.ai> Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com> Co-authored-by: Elliot Chen <2340896+cyfyifanchen@users.noreply.github.com>
2026-07-17 19:37:05 +08:00
Package layout:
- ``launch`` — the unified entry: run spec, bench plugin contract,
run/check/status/finalize state machine
- ``orchestrator`` — the SOP round loop: diagnose, design, screen, confirm,
gates, sealed test (see ``orchestrator/DESIGN.md``)
- ``tree`` — git-backed harness version tree (commit-per-candidate)
- ``activation`` — beacon ledger: which candidate code actually fired
- ``applier`` — path whitelist + beacon guards on candidate edits
- ``judge`` — LLM judge with L1/L2/L3 + (WHERE, WHY) output
- ``scheduler`` — task/anchor selection and bandit utilities
- ``analysis`` — trial-ledger readers and stability bucketing
- ``compressor`` — trajectory compression for diagnosis prompts
Provenance notes for readers of module docstrings: citations of the form
``spec §NN`` refer to the upstream project's internal design document (not
shipped; the shipped methodology spec is
``docs/specs/self-evolution-loop-sop.md``) — the numbers are kept as
provenance for the constants and enums they justify. ``TB2`` names the
upstream terminal-agent benchmark line this system was first developed
against; the example benchmark shipped here is AppWorld. ``GSME`` (Gated
Semantic MAP-Elites) is the quality-diversity elite archive
(``orchestrator/archive.py``).
"""