# Evolver: design and implementation New to the tool? Start with [Usage and experiments](evolver.md) for status, setup, costs, and the run/resume/finalize workflow. This page is the implementation reference. **Companion:** the authoritative methodology spec is `docs/specs/self-evolution-loop-sop.md`, kept in the repository beside the code rather than published here. This document answers: which code implements each SOP clause in Raven, which deviations are deliberate, and which parts are present but unwired. **How to use:** before changing `evolver/**` or `benchmarks/appworld/evolve/**`, locate the SOP clause here; if your change alters a correspondence, update this document in the same PR. ## 0. The fundamental architectural difference (read first) The SOP's loop (§3 / §8.3) is **Claude-driven**: a human opens Claude and walks the seven-step funnel by hand. Raven implements the same methodology as a **program-driven** loop: `evolver/orchestrator/loop.py::EvolutionOrchestrator` drives the funnel, and the SOP's CLI parts became functions inside the loop. The SOP §8.0 division of labor (semantics to the model, deterministic arithmetic to code) is preserved unchanged: diagnosis / design / verdict are still LLM calls, wrapped in `orchestrator/nodes/semantic.py::SemanticNode` (parsing errors are fed back to the model, with a limited number of repair retries). Only "who presses the next-step button" changed. The launcher automates the funnel and persists completed-round checkpoints, so an operator can resume without reconstructing the experiment by hand. ## 1. SOP §0 general rules -> implementation | SOP clause | Raven implementation | Evidence (file :: symbol) | |---|---|---| | Gate0 pre-run environment health check | precheck injected per round, forced once before cold start | `orchestrator/production.py::build_evolution_orchestrator` (`run_gate0`); `benchmarks/appworld/evolve/precheck.py::make_appworld_precheck` | | Infra-failure ladder: detect -> rerun <=2 -> score 0 if still broken | `eval_with_infra_rerun`, `max_reruns=2`, produces the `_infra_rerun{1,2}` directory ladder; the KEPT rule takes the measurement with the fewest infra trials | `orchestrator/scoring.py::eval_with_infra_rerun`; `benchmarks/appworld/evolve/adapter.py::ladder_out_dirs / read_kept_out_dir` | | ★ Denominator = total tasks; infra tasks never excluded, scored 0 in place | Gate-f only reports the contamination list, never shrinks the denominator | `orchestrator/gates/pipeline.py::run_gates` (comment cites SOP §0 directly) | | Division of labor: semantics = model, determinism = code | see §0 above | `orchestrator/nodes/semantic.py::SemanticNode` | | Two verdicts: navigation = K3 mean > vanilla (bank); credited = paired 2σ (paper) | `PairedResult` keeps them as two independent fields: `promoted` (navigator) and `credited_2sigma` (label); promotion reads only the former | `orchestrator/gates/paired.py::PairedResult / paired_lift` | | Paired σ: std of per-task paired diffs, removing between-task difficulty | `d_i = rate_c,i − rate_v,i`, `se = stdev(d)/√n`, `z = lift/se` | `orchestrator/gates/paired.py::paired_lift` | | Sealed test: keep test results out of selection, then evaluate retention | The AppWorld launcher scores the baseline and recorded round deliverables at finalization through `unseal_retention`. `SealedTestRunner.score` returns `None`; results are stored separately, but filesystem access is not isolated from arbitrary code. | `orchestrator/sealed/runner.py::SealedTestRunner / unseal_retention` | | Test never enters anchor/train | stronger than the SOP's discipline: a mechanized assertion — leakage raises at startup | `orchestrator/sealed/runner.py::assert_no_test_leak`, wired in `loop.py` construction | | Discipline: diagnose from train trajectories only | the diagnosis corpus source hangs off train only; test trajectories have no read path (the sealed runner stores scores only) | `orchestrator/scoring.py::EvalBackend.trajectories` | | Discipline: configuration identical throughout | `run_meta.json` records a configuration fingerprint; the launcher normally refuses a changed configuration or a finalized run. `--force` bypasses these guards and invalidates the sealed-experiment assumption. | `evolver/launch/state.py::RunMeta.check_config`; `evolver/launch/runner.py::_meta_guard` | ## 2. SOP §1 cold start -> implementation | SOP clause | Raven implementation | Evidence | |---|---|---| | Vanilla train full set x K=3 thick ledger | `backend.cold_start()` must return a non-empty stability ledger; the initial baseline is built from the evaluation results in the vanilla output directory and then frozen. | `orchestrator/loop.py` (construction); `benchmarks/appworld/evolve/run.py` (`seed_label="van0"`, `cold_start_k`) | | Failure map covers >= 7 WHY classes | `diagnose_round(min_why_classes=7)` | `orchestrator/nodes/diagnose.py::diagnose_round` | | WHY x WHERE taxonomy; inducible for a new bench | `TaxonomySpec` + two-stage `induce_taxonomy`; induction failure raises loudly, never silently borrowing another bench's table | `orchestrator/nodes/taxonomy.py` | ## 3. SOP §2 seven-step funnel -> implementation | SOP step | Raven implementation | Evidence | |---|---|---| | ① Round 1 uses the cold-start map; round 2+ re-diagnoses the child (fired / flipped / still-failing), appending to the continuously updated failure map | `_diagnosed_parents` prevents duplicate diagnosis; `merge_failure_maps` accumulates cross-round and persists; flips recorded in `_flips` with a harm-replay excerpt showing the regressed task's actual failure | `orchestrator/loop.py` (round body); `orchestrator/production.py::outcome_hook` | | ② 1-2 WHYs x 2-3 candidates, budget-capped; env-gated default-off | WHY selection defaults to driver mode (model picks; formula fallback + shadow log); `Budget(max_why_per_round x candidates_per_why)` enforced in code; the appworld line passes per-node `activation_env` | `benchmarks/appworld/evolve/editor.py::driver_select_whys / rerank_whys`; `orchestrator/config.py::Budget`; `benchmarks/appworld/evolve/adapter.py` | | ③ Free pruning: beacon_guard + preflight | beacon: the editor rejects python edits without `activation_beacon` (hard); preflight: `make_zero_hit_preflight` zero-hit prune, **off by default** (see §6 ①) | `benchmarks/appworld/evolve/editor.py`; `orchestrator/production.py::make_zero_hit_preflight`; `benchmarks/appworld/evolve/run.py` (`zero_hit_preflight=False`) | | ④ Apply, create child node; path_guard shields the kernel; git persistence | edit-then-commit: edits land as a real git child commit of the parent, the working tree remains unchanged, and the program determines and returns the changed file paths | `evolver/tree/git_ops.py::commit_files_as_child`; `evolver/applier/path_guard.py` | | ⑤a K=1 anchor generous-pass screen; σ_screen computed; cull at 1.5σ; three tiers | three buckets clear_win / within_band / cull, only cull is blocked; σ formula identical to the SOP, emitted by `select_anchor` from the ledger; the AppWorld line uses a focused-subset Fisher probe variant (also generous-pass: culls only significantly-worse) | `orchestrator/nodes/screen.py::screen_candidate`; `evolver/scheduler/anchor_selection.py::select_anchor / simple_anchor`; `orchestrator/gates/strategies.py::FocusedFisherGate` | | ⑤a anchor composition: affinity majority + icebreakers + sentinels, ⊂ train | all three roles implemented; the sentinel guard adds stratification the SOP does not have (stable = mean guard, fragile = Fisher, avoiding noise kills); the affinity data source is unwired (§6 ③) | `evolver/scheduler/anchor_selection.py`; `orchestrator/gates/strategies.py` (sentinel guard) | | ⑤ Borrowing (SOP tags [defer]) | module present (byte-identical to upstream), unwired (§6 ②) | `evolver/scheduler/tree_aware_bandit.py` | | ⑤b Survivors: full set x K=3 confirm | `k_confirm=3`, confirm runs the full train set | `orchestrator/gates/strategies.py` | | ⑥ Three shields in order Gate-f -> Gate-b -> Gate2 | `run_gates` is exactly this order; Gate-b fails OPEN without instrumentation data (never condemns an uninstrumented-but-honest candidate); the reported score always uses the full-set fixed denominator, so a Gate-b subset mean can never masquerade as the score | `orchestrator/gates/pipeline.py::run_gates`; `orchestrator/production.py` (beacon-aware `fired_source`) | | Gate-b data chain | write side: the editor forces inline beacons -> per-attempt beacon directories; read side: union over the confirm dir + infra-ladder siblings. Honesty note: attribution is presence-level — Gate-b proves *a* beacon in the candidate's code executed on a task, not that the beacon sat inside the mechanism's trigger condition; an unconditionally-placed beacon degrades Gate-b to a no-op (promotion still requires the full-train win) | `evolver/activation/ledger.py::beacon_workspace / mark_beacons_enabled` (called from `benchmarks/appworld/batch.py`); `evolver/activation/ledger.py::read_fired_tasks` | | ⑦ Best of bank becomes parent; if no candidate passes, retain the existing parent and diagnose again | `beat_vanilla` patience signal + greedy parent selection | `orchestrator/loop.py` | | Termination: 10 rounds with nobody above vanilla (vs vanilla, not the previous parent) or 20-round cap; never consult test | `TerminationTracker(patience=10, max_rounds=20)`, the signal defined as beating the FIXED vanilla; plus a protection the SOP lacks: errored rounds do not burn patience (`max_consecutive_errors` is a separate backstop) | `orchestrator/termination.py` | | Four node statuses | superset: `pruned_inert / pruned_at_screen / pruned_at_confirm / promoted_to_baseline / errored / blocked_l1 / archived-methodology-failure`; inert deaths additionally feed the per-WHY history so the designer learns from them | `evolver/tree/node.py::NodeStatus`; `orchestrator/production.py::inert_hook` | | WHERE bound mechanically from the artifact, never by self-declaration | `bind_where` derives the lever from the actually-touched files; the self-declared `patch_where` stays in the ledger for audit and never decides the archive coordinate (4-tier granularity, see §6 ④) | `orchestrator/archive.py::bind_where / cell_of` | ## 4. SOP §3 persistence -> implementation | SOP layer | Raven counterpart | |---|---| | findings work log | `/findings.md` (one section per round, driver verdict) | | Cross-session state | journal (`orchestrator/state/journal.py`, crash-resume replays completed rounds) + `history.json` (per-WHY attempt history) | | Box durable artifacts | `failure_map.json` (failure map updated across rounds) / `nodes/.json` (node ledger: identity + git anchor + final status + gate stats) / per-round out-dirs (per-task results) | The SOP §3.1 recording scheme is retained: Git commits store code versions (`commit_files_as_child` produces commit SHAs), and `nodes/*.json` stores the evolution tree structure. The two are linked by `git_commit_sha`. ## 5. Where we exceed the SOP - **Sealed-test evaluation is automated.** `assert_no_test_leak` checks that task splits do not overlap. At finalization, `unseal_retention` scores recorded commits and selects the deliverable by training score. These controls separate evaluation from selection; they do not sandbox candidate code or prevent an operator from reading files. - **A single candidate's crash cannot sink a round:** the `errored` status + errored rounds not burning patience (`max_consecutive_errors` as its own backstop); not covered by the SOP. - **QD archive and recombination:** a (WHERE x WHY) per-cell elite bank + cross-cell recombinant candidates (`orchestrator/archive.py`) — an exploration mechanism beyond the SOP; the gates and evaluation criteria remain unchanged. - **The inert-death feedback loop** (2026-07): a distinct `pruned_inert` status + inert deaths recorded into history + the design prompt distinguishing "the trigger never fired" from "the mechanism was rejected" + gentle WHY decay for inert deaths (`0.55^n_fail x 0.85^n_inert`). - **Harm replay:** when a candidate causes a task to regress, an excerpt of that task's actual execution trajectory under the candidate is provided to the next candidate-design attempt; the SOP only requires flip counts. ## 6. Deliberate deviations and unwired parts { #6-deliberate-deviations-and-unwired-parts-honest-list } 1. **zero-hit preflight is off by default** (`zero_hit_preflight=False`, decided 2026-07). With instrumentation supplied, Gate-b excludes tasks where no beacon fired; without it, Gate-b is skipped. Preflight is a separate budget-saving check with a false-prune risk. TRIGGER_REGEX declaration is opt-in and the actual prune rate is unknown. Gather data first (enable on a run, inspect `pruned_inert` entries in history), then decide the default. SOP §2 tags ③ as [now]; this is a deliberate deviation. 2. **Borrowing unwired** (SOP §2 ⑤, tagged [defer]): `tree_aware_bandit` is present, but the orchestrator does not call it. The AppWorld train set is currently run in full; add borrowing when larger task sets justify it. 3. **Affinity anchor lacks a data source:** `select_anchor(affinity)` accepts it; the appworld side has no trigger-density source (upstream's `affinity_picker.py` was not ported). Anchors are currently icebreakers, sentinels, and borderline tasks; the impact of omitting affinity selection has not been measured here. 4. **WHERE mechanical binding has 4 tiers** (prompt/runtime/mixed/edit), coarser than the judge schema's 14 classes. Sufficient for QD cells; refine `_lever_of_path` for fine-grained lever statistics. Self-declared and mechanical values are both in the ledger; no reconciliation alarm (an optional observability add-on). 5. **`pruned_inert` matches the SOP's status semantically** but differs in implementation: the SOP judges it via a preflight CLI, we via the embedded loop preflight (default off, see ①). ## 6.5 The unified entry (added 2026-07) SOP §8.3's "manual orchestration" is superseded by `python -m evolver run --config `: a single-command state machine runs cold start -> rounds -> termination -> unseal. Before finalization, it resumes from saved trials and completed-round checkpoints, not from an arbitrary instruction inside an interrupted call. Configuration drift and finalization are checked against `run_meta.json` unless `--force` overrides the guards. Benches plug in via the contract in `docs/specs/evolve-bench-contract.md`, kept in the repository beside the code; implementation in `evolver/launch/` + `evolver/cli.py`. ## 7. SOP parts <-> Raven parts quick reference | Part cited in SOP §8.1 | Raven counterpart | Shape difference | |---|---|---| | `analysis/proxy_features.py` | `evolver/analysis/proxy_features.py` | byte-identical | | `analysis/failure_map_builder.py` | `evolver/analysis/failure_map_builder.py` | byte-identical | | `activation/preflight.py` (CLI) | `orchestrator/production.py::make_zero_hit_preflight` | CLI -> embedded; regex variant, default off | | `analysis/stability_bucket.py` | `evolver/analysis/stability_bucket.py` | byte-identical | | `scheduler/bandit_tasks.py` | `evolver/scheduler/bandit_tasks.py` | byte-identical | | `scheduler/anchor_selection.py` | `evolver/scheduler/anchor_selection.py` | same lineage + our `simple_anchor` | | `scheduler/affinity_picker.py` | **no counterpart** (§6 ③) | — | | `scheduler/tree_aware_bandit.py` | `evolver/scheduler/tree_aware_bandit.py` | same lineage, unwired (§6 ②) | | `activation/gate_audit.py` (CLI) | `orchestrator/gates/pipeline.py::run_gates` (Gate-b embedded) | CLI -> embedded | | `analysis/paired_significance.py` (CLI) | `orchestrator/gates/paired.py::paired_lift`; retention in `sealed/runner.py::unseal_retention` | CLI -> embedded | | `aggregate_keq3` / `gate0_ctrf_audit` (external eval engine) | `benchmarks/appworld/evolve/eval.py / adapter.py` (K=3 aggregation + infra ladder) | cross-repo contract -> same-repo module | | `tree/*` | `evolver/tree/` (node/store/git_ops) | same lineage + our `commit_files_as_child` / `read_file_at` |