## Summary Reorganize the Raven documentation landing page and expand the bilingual documentation site. - Add bilingual guidance for agent collaboration, ACP and A2A protocols, DAG orchestration, channels, browser collaboration, knowledge bases, Playbooks, proactive follow-ups, Oncall, permissions, tracing, trajectory replay, and Evolver. - Group daily usage under `Use Raven` and `Using Raven`. - Group `Technical Reference` and `Developer Guide` under `Explore the documentation` while preserving separate cards. ## Type - [ ] Fix - [ ] Feature - [x] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - [x] Relevant tests pass locally - [ ] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed Exact checks: - `uv run --frozen --group docs mkdocs build -f docs-site/mkdocs.yml --strict` - passed. - `uv run --frozen --group docs pytest tests/test_docs_site.py tests/test_living_docs.py tests/integration/test_docs_site_controls_e2e.py -q -o addopts=''` - 77 passed. - `uv run --frozen python scripts/check_commit_messages.py origin/main..HEAD` - passed. - `git diff --check origin/main...HEAD` - passed. - `make check-large-files` - passed. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes This is a documentation-only change. It does not change runtime behavior, configuration semantics, or public APIs. Reverting the documentation commits restores the previous site content and navigation. ## Related Issues N/A --------- Co-authored-by: userName20260323 <270284818+userName20260323@users.noreply.github.com> Co-authored-by: Claude (claude) <noreply@anthropic.com>
16 KiB
Evolver: design and implementation
New to the tool? Start with Usage and experiments for status, setup, costs, and the run/resume/finalize workflow. This page is the implementation reference.
Companion: the authoritative methodology spec is
docs/specs/self-evolution-loop-sop.md, kept in the repository beside the code
rather than published here. This document answers: which code
implements each SOP clause in Raven, which deviations are deliberate, and
which parts are present but unwired.
How to use: before changing evolver/** or
benchmarks/appworld/evolve/**, locate the SOP clause here; if your
change alters a correspondence, update this document in the same PR.
0. The fundamental architectural difference (read first)
The SOP's loop (§3 / §8.3) is Claude-driven: a human opens Claude and
walks the seven-step funnel by hand. Raven implements the same methodology as a
program-driven loop: evolver/orchestrator/loop.py::EvolutionOrchestrator
drives the funnel, and the SOP's CLI parts became functions inside the loop.
The SOP §8.0 division of labor (semantics to the model, deterministic
arithmetic to code) is preserved unchanged: diagnosis / design / verdict are
still LLM calls, wrapped in orchestrator/nodes/semantic.py::SemanticNode
(parsing errors are fed back to the model, with a limited number of repair
retries). Only "who presses the next-step button" changed.
The launcher automates the funnel and persists completed-round checkpoints, so an operator can resume without reconstructing the experiment by hand.
1. SOP §0 general rules -> implementation
| SOP clause | Raven implementation | Evidence (file :: symbol) |
|---|---|---|
| Gate0 pre-run environment health check | precheck injected per round, forced once before cold start | orchestrator/production.py::build_evolution_orchestrator (run_gate0); benchmarks/appworld/evolve/precheck.py::make_appworld_precheck |
| Infra-failure ladder: detect -> rerun <=2 -> score 0 if still broken | eval_with_infra_rerun, max_reruns=2, produces the _infra_rerun{1,2} directory ladder; the KEPT rule takes the measurement with the fewest infra trials |
orchestrator/scoring.py::eval_with_infra_rerun; benchmarks/appworld/evolve/adapter.py::ladder_out_dirs / read_kept_out_dir |
| ★ Denominator = total tasks; infra tasks never excluded, scored 0 in place | Gate-f only reports the contamination list, never shrinks the denominator | orchestrator/gates/pipeline.py::run_gates (comment cites SOP §0 directly) |
| Division of labor: semantics = model, determinism = code | see §0 above | orchestrator/nodes/semantic.py::SemanticNode |
| Two verdicts: navigation = K3 mean > vanilla (bank); credited = paired 2σ (paper) | PairedResult keeps them as two independent fields: promoted (navigator) and credited_2sigma (label); promotion reads only the former |
orchestrator/gates/paired.py::PairedResult / paired_lift |
| Paired σ: std of per-task paired diffs, removing between-task difficulty | d_i = rate_c,i − rate_v,i, se = stdev(d)/√n, z = lift/se |
orchestrator/gates/paired.py::paired_lift |
| Sealed test: keep test results out of selection, then evaluate retention | The AppWorld launcher scores the baseline and recorded round deliverables at finalization through unseal_retention. SealedTestRunner.score returns None; results are stored separately, but filesystem access is not isolated from arbitrary code. |
orchestrator/sealed/runner.py::SealedTestRunner / unseal_retention |
| Test never enters anchor/train | stronger than the SOP's discipline: a mechanized assertion — leakage raises at startup | orchestrator/sealed/runner.py::assert_no_test_leak, wired in loop.py construction |
| Discipline: diagnose from train trajectories only | the diagnosis corpus source hangs off train only; test trajectories have no read path (the sealed runner stores scores only) | orchestrator/scoring.py::EvalBackend.trajectories |
| Discipline: configuration identical throughout | run_meta.json records a configuration fingerprint; the launcher normally refuses a changed configuration or a finalized run. --force bypasses these guards and invalidates the sealed-experiment assumption. |
evolver/launch/state.py::RunMeta.check_config; evolver/launch/runner.py::_meta_guard |
2. SOP §1 cold start -> implementation
| SOP clause | Raven implementation | Evidence |
|---|---|---|
| Vanilla train full set x K=3 thick ledger | backend.cold_start() must return a non-empty stability ledger; the initial baseline is built from the evaluation results in the vanilla output directory and then frozen. |
orchestrator/loop.py (construction); benchmarks/appworld/evolve/run.py (seed_label="van0", cold_start_k) |
| Failure map covers >= 7 WHY classes | diagnose_round(min_why_classes=7) |
orchestrator/nodes/diagnose.py::diagnose_round |
| WHY x WHERE taxonomy; inducible for a new bench | TaxonomySpec + two-stage induce_taxonomy; induction failure raises loudly, never silently borrowing another bench's table |
orchestrator/nodes/taxonomy.py |
3. SOP §2 seven-step funnel -> implementation
| SOP step | Raven implementation | Evidence |
|---|---|---|
| ① Round 1 uses the cold-start map; round 2+ re-diagnoses the child (fired / flipped / still-failing), appending to the continuously updated failure map | _diagnosed_parents prevents duplicate diagnosis; merge_failure_maps accumulates cross-round and persists; flips recorded in _flips with a harm-replay excerpt showing the regressed task's actual failure |
orchestrator/loop.py (round body); orchestrator/production.py::outcome_hook |
| ② 1-2 WHYs x 2-3 candidates, budget-capped; env-gated default-off | WHY selection defaults to driver mode (model picks; formula fallback + shadow log); Budget(max_why_per_round x candidates_per_why) enforced in code; the appworld line passes per-node activation_env |
benchmarks/appworld/evolve/editor.py::driver_select_whys / rerank_whys; orchestrator/config.py::Budget; benchmarks/appworld/evolve/adapter.py |
| ③ Free pruning: beacon_guard + preflight | beacon: the editor rejects python edits without activation_beacon (hard); preflight: make_zero_hit_preflight zero-hit prune, off by default (see §6 ①) |
benchmarks/appworld/evolve/editor.py; orchestrator/production.py::make_zero_hit_preflight; benchmarks/appworld/evolve/run.py (zero_hit_preflight=False) |
| ④ Apply, create child node; path_guard shields the kernel; git persistence | edit-then-commit: edits land as a real git child commit of the parent, the working tree remains unchanged, and the program determines and returns the changed file paths | evolver/tree/git_ops.py::commit_files_as_child; evolver/applier/path_guard.py |
| ⑤a K=1 anchor generous-pass screen; σ_screen computed; cull at 1.5σ; three tiers | three buckets clear_win / within_band / cull, only cull is blocked; σ formula identical to the SOP, emitted by select_anchor from the ledger; the AppWorld line uses a focused-subset Fisher probe variant (also generous-pass: culls only significantly-worse) |
orchestrator/nodes/screen.py::screen_candidate; evolver/scheduler/anchor_selection.py::select_anchor / simple_anchor; orchestrator/gates/strategies.py::FocusedFisherGate |
| ⑤a anchor composition: affinity majority + icebreakers + sentinels, ⊂ train | all three roles implemented; the sentinel guard adds stratification the SOP does not have (stable = mean guard, fragile = Fisher, avoiding noise kills); the affinity data source is unwired (§6 ③) | evolver/scheduler/anchor_selection.py; orchestrator/gates/strategies.py (sentinel guard) |
| ⑤ Borrowing (SOP tags [defer]) | module present (byte-identical to upstream), unwired (§6 ②) | evolver/scheduler/tree_aware_bandit.py |
| ⑤b Survivors: full set x K=3 confirm | k_confirm=3, confirm runs the full train set |
orchestrator/gates/strategies.py |
| ⑥ Three shields in order Gate-f -> Gate-b -> Gate2 | run_gates is exactly this order; Gate-b fails OPEN without instrumentation data (never condemns an uninstrumented-but-honest candidate); the reported score always uses the full-set fixed denominator, so a Gate-b subset mean can never masquerade as the score |
orchestrator/gates/pipeline.py::run_gates; orchestrator/production.py (beacon-aware fired_source) |
| Gate-b data chain | write side: the editor forces inline beacons -> per-attempt beacon directories; read side: union over the confirm dir + infra-ladder siblings. Honesty note: attribution is presence-level — Gate-b proves a beacon in the candidate's code executed on a task, not that the beacon sat inside the mechanism's trigger condition; an unconditionally-placed beacon degrades Gate-b to a no-op (promotion still requires the full-train win) | evolver/activation/ledger.py::beacon_workspace / mark_beacons_enabled (called from benchmarks/appworld/batch.py); evolver/activation/ledger.py::read_fired_tasks |
| ⑦ Best of bank becomes parent; if no candidate passes, retain the existing parent and diagnose again | beat_vanilla patience signal + greedy parent selection |
orchestrator/loop.py |
| Termination: 10 rounds with nobody above vanilla (vs vanilla, not the previous parent) or 20-round cap; never consult test | TerminationTracker(patience=10, max_rounds=20), the signal defined as beating the FIXED vanilla; plus a protection the SOP lacks: errored rounds do not burn patience (max_consecutive_errors is a separate backstop) |
orchestrator/termination.py |
| Four node statuses | superset: pruned_inert / pruned_at_screen / pruned_at_confirm / promoted_to_baseline / errored / blocked_l1 / archived-methodology-failure; inert deaths additionally feed the per-WHY history so the designer learns from them |
evolver/tree/node.py::NodeStatus; orchestrator/production.py::inert_hook |
| WHERE bound mechanically from the artifact, never by self-declaration | bind_where derives the lever from the actually-touched files; the self-declared patch_where stays in the ledger for audit and never decides the archive coordinate (4-tier granularity, see §6 ④) |
orchestrator/archive.py::bind_where / cell_of |
4. SOP §3 persistence -> implementation
| SOP layer | Raven counterpart |
|---|---|
| findings work log | <work_dir>/findings.md (one section per round, driver verdict) |
| Cross-session state | journal (orchestrator/state/journal.py, crash-resume replays completed rounds) + history.json (per-WHY attempt history) |
| Box durable artifacts | failure_map.json (failure map updated across rounds) / nodes/<id>.json (node ledger: identity + git anchor + final status + gate stats) / per-round out-dirs (per-task results) |
The SOP §3.1 recording scheme is retained: Git commits store code versions
(commit_files_as_child produces commit SHAs), and nodes/*.json stores the
evolution tree structure. The two are linked by git_commit_sha.
5. Where we exceed the SOP
- Sealed-test evaluation is automated.
assert_no_test_leakchecks that task splits do not overlap. At finalization,unseal_retentionscores recorded commits and selects the deliverable by training score. These controls separate evaluation from selection; they do not sandbox candidate code or prevent an operator from reading files. - A single candidate's crash cannot sink a round: the
erroredstatus + errored rounds not burning patience (max_consecutive_errorsas its own backstop); not covered by the SOP. - QD archive and recombination: a (WHERE x WHY) per-cell elite bank +
cross-cell recombinant candidates (
orchestrator/archive.py) — an exploration mechanism beyond the SOP; the gates and evaluation criteria remain unchanged. - The inert-death feedback loop (2026-07): a distinct
pruned_inertstatus + inert deaths recorded into history + the design prompt distinguishing "the trigger never fired" from "the mechanism was rejected"- gentle WHY decay for inert deaths (
0.55^n_fail x 0.85^n_inert).
- gentle WHY decay for inert deaths (
- Harm replay: when a candidate causes a task to regress, an excerpt of that task's actual execution trajectory under the candidate is provided to the next candidate-design attempt; the SOP only requires flip counts.
6. Deliberate deviations and unwired parts
- zero-hit preflight is off by default (
zero_hit_preflight=False, decided 2026-07). With instrumentation supplied, Gate-b excludes tasks where no beacon fired; without it, Gate-b is skipped. Preflight is a separate budget-saving check with a false-prune risk. TRIGGER_REGEX declaration is opt-in and the actual prune rate is unknown. Gather data first (enable on a run, inspectpruned_inertentries in history), then decide the default. SOP §2 tags ③ as [now]; this is a deliberate deviation. - Borrowing unwired (SOP §2 ⑤, tagged [defer]):
tree_aware_banditis present, but the orchestrator does not call it. The AppWorld train set is currently run in full; add borrowing when larger task sets justify it. - Affinity anchor lacks a data source:
select_anchor(affinity)accepts it; the appworld side has no trigger-density source (upstream'saffinity_picker.pywas not ported). Anchors are currently icebreakers, sentinels, and borderline tasks; the impact of omitting affinity selection has not been measured here. - WHERE mechanical binding has 4 tiers (prompt/runtime/mixed/edit),
coarser than the judge schema's 14 classes. Sufficient for QD cells;
refine
_lever_of_pathfor fine-grained lever statistics. Self-declared and mechanical values are both in the ledger; no reconciliation alarm (an optional observability add-on). pruned_inertmatches the SOP's status semantically but differs in implementation: the SOP judges it via a preflight CLI, we via the embedded loop preflight (default off, see ①).
6.5 The unified entry (added 2026-07)
SOP §8.3's "manual orchestration" is superseded by
python -m evolver run --config <yaml>: a single-command state machine runs
cold start -> rounds -> termination -> unseal. Before finalization, it resumes
from saved trials and completed-round checkpoints, not from an arbitrary
instruction inside an interrupted call. Configuration drift and finalization
are checked against run_meta.json unless --force overrides the guards.
Benches plug in via the contract in
docs/specs/evolve-bench-contract.md, kept in the repository beside the code;
implementation in evolver/launch/ + evolver/cli.py.
7. SOP parts <-> Raven parts quick reference
| Part cited in SOP §8.1 | Raven counterpart | Shape difference |
|---|---|---|
analysis/proxy_features.py |
evolver/analysis/proxy_features.py |
byte-identical |
analysis/failure_map_builder.py |
evolver/analysis/failure_map_builder.py |
byte-identical |
activation/preflight.py (CLI) |
orchestrator/production.py::make_zero_hit_preflight |
CLI -> embedded; regex variant, default off |
analysis/stability_bucket.py |
evolver/analysis/stability_bucket.py |
byte-identical |
scheduler/bandit_tasks.py |
evolver/scheduler/bandit_tasks.py |
byte-identical |
scheduler/anchor_selection.py |
evolver/scheduler/anchor_selection.py |
same lineage + our simple_anchor |
scheduler/affinity_picker.py |
no counterpart (§6 ③) | — |
scheduler/tree_aware_bandit.py |
evolver/scheduler/tree_aware_bandit.py |
same lineage, unwired (§6 ②) |
activation/gate_audit.py (CLI) |
orchestrator/gates/pipeline.py::run_gates (Gate-b embedded) |
CLI -> embedded |
analysis/paired_significance.py (CLI) |
orchestrator/gates/paired.py::paired_lift; retention in sealed/runner.py::unseal_retention |
CLI -> embedded |
aggregate_keq3 / gate0_ctrf_audit (external eval engine) |
benchmarks/appworld/evolve/eval.py / adapter.py (K=3 aggregation + infra ladder) |
cross-repo contract -> same-repo module |
tree/* |
evolver/tree/ (node/store/git_ops) |
same lineage + our commit_files_as_child / read_file_at |