## Summary Nothing in this repo could see a signature change reach its callers. Ruff reads one file at a time, the import contracts read edges rather than calls, and the suite only covers lines it executes. The consequence was concrete: a required `sandbox_home` argument added to `BoxliteExecutor` reached `plugins-dist/design-engine`'s render path with CI green, and a reviewer -- not a gate -- found it. `ty` answers that question. This adds it as a pinned dev dependency, a `make lint-types` target and a CI step, with **110 of its 133 rules live**: nine switched off with their measured backlog recorded beside each, fourteen shipped disabled upstream and untouched. Getting there meant clearing 81 diagnostics. This PR folds what were three stacked PRs (#422 and #423, now closed); each rule's story is a commit of its own. ### 1. The gate, at 108 rules (`chore(*): gate call correctness...`) 21 diagnostics to clear, five of them real: | Where | What | |---|---| | `evolver/orchestrator/production.py` | `design_of` annotated as taking two parameters while its docstring, its only implementation and its one call site all use three | | `raven/cli/provider_commands.py` | `dict[str, callable]` -- the builtin function, not a type, so the annotation asserted nothing | | `raven/config/schema.py` | `Config` is a `BaseSettings` but declared `env_prefix` / `env_nested_delimiter` through pydantic's `ConfigDict`, which does not carry those keys | | `raven/spine/delivery.py` | `StreamDelta.conversation_id` is optional and was used as a `dict[str, str]` key; a delta without one stored a `None` entry the close path then read as "no open stream", losing that stream's done marker | | `research_flow/support/harness_text.py` | a predicate table typed its matchers as `object`, so nothing checked they are callable | The rest were unproven invariants and annotations wider than the code: three late-bound callables asserted where a comment used to carry the claim, one closure binding the callable it guarded instead of re-reading it, four `hasattr` probes that now check what the call actually needs, and a contextmanager yielding the value it just set. ### 2. `unresolved-import` (`fix(*): resolve the imports the type gate could not`) 21 diagnostics; three were stale module paths nothing had run into: - `HarnessNode.current_core_version()` read `raven.__core_version__`, a module that exists neither here nor in the wheel `scripts/build_core_wheel.py` produces. The read sat inside `try/except ImportError`, so the except branch was the only branch and **every node `store.py` saved recorded `core_version="unknown"`**. It reads `raven.__version__` now and returns `0.1.13`. - `raven/agent/loop/wiring.py` imported `SkillForgeRouterConfig` from `raven.config.schema`; it lives in `raven.config.raven`. Under `TYPE_CHECKING`, so the annotation simply meant nothing. - three scripts imported `raven.context_engine.base` and `raven.memory_engine.base`, neither of which exists. **Two failed at module import** -- running them was not possible. The other eighteen were the environment: the three agent products keep their plugin package beside them, so their roots are declared in `[tool.ty.environment]`; `lxml` ships a compiled `etree` with no type information, so `lxml-stubs` is a dev dependency (real types rather than five silenced lines); `uno` and `com.sun.star.*` are suppressed per line, since that worker runs inside LibreOffice's own interpreter. ### 3. `invalid-return-type` (`fix(*): make the declared return types true`) 39 diagnostics, and **one stale annotation held eight of them**: `run_turn` and the `_run_turn` behind it were declared `-> LoopOutcome` while all three of their returns build a `TurnOutcome` -- a different dataclass with no field in common. Nothing broke, because the spine worker reads what it actually gets and `LoopOutcome` never leaves `_run_agent_loop`. The annotation was the only thing wrong. The other 31: nine annotations narrower than the code (five `return tuple(<genexpr>)` typed as fixed-arity tuples, an inner `one()` still declaring the two-tuple it returned before a third element was added, `image_block`'s TypedDict), five declarations refusing a value the function returns (four tool `execute` overrides saying `-> str` while returning `ToolResult`), seven invariants that lived in a `# type: ignore` comment, and ten honest widenings. ### 4. What the gates then caught in this PR's own work - The qq factory guard passed locally only because an earlier test had left an event loop behind. Reproduced the CI state (`asyncio.set_event_loop(None)`), then made the test async so the loop is its own. - `invalid-return-type`, while still disabled, hid a regression introduced here: the qq factory returning the class instead of an instance. Found by diffing the rule histogram, which is why that rule is on now. - The diff-coverage gate named four lines in `raven/routing/cache.py` -- the asserts this work had just added, sitting inside a `try` whose `except Exception` would have swallowed the `AssertionError` and served the stale cache. `_do_refresh` returns its data now; the asserts, their noqa and the swallowing all go with it. `child_scope`, `QQChannel.start()` and `BenchmarkCache` had no tests at all and have them now. Every one was mutation-checked. ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [x] CI / tooling - [ ] Refactor - [ ] Other ## Verification - `make lint-types` -> All checks passed; 81 diagnostics cleared across the three rule sets - Gate proven live three times, once per rule: a missing argument, an import of a module that does not exist, and a function returning a str under `-> int` each turn it red, and removing the probe turns it green - `uv run ruff check` + `ruff format --check` over all six trees -> clean - `uv run lint-imports` -> 10 kept, 0 broken; `deptry raven` (all extras) -> no issues; `uv lock --check` -> resolved - `make check-source-language` -> exit 0; `commitlint` -> exit 0 - `make coverage` locally: **22050 passed, 45 skipped, 4 failed**; diff coverage 92.59% - All four failures are **pre-existing on `main`**, checked by a real checkout in the same environment where they fail identically: two `test_the_products_tool_face_*` launcher tests, `test_a_ceiling_as_large_as_the_window_still_leaves_room_for_history` (`reserved_tools = 6044 > 6_000`), and `test_a_missing_key_is_said_once_rather_than_per_query` (environment-dependent: it asserts `SERPER_API_KEY` is absent) - Rule counts measured with both `--extra dev` and `--all-extras` -- identical, so this gate's verdict does not depend on which optional packages are installed - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk Behaviour changes are confined to the fixes listed above, and each replaces an error path with a correct one: nodes are stamped with the real raven version instead of `"unknown"`, two scripts that could not start now start, the stream bookkeeping no longer stores a `None` key, and a cache refresh returns what it fetched. Everything else is a declaration moved to match what the function already returned. `tests/` is out of the gate's scope (113 diagnostics under the same rules). Tests are executed, so a broken call there is already loud; this gate is for code CI does not run. Bringing them in is sized, separate work. Rollback: revert the whole PR to drop the gate and its fixes together, or delete the `[tool.ty]` block, the `lint-types` target and its CI step to keep the fixes and drop only the gate. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
evolver — harness self-evolution
Status: retirement of this tree is planned alongside the vendored fork trees (the step-8 retirement wave), pending partner sign-off -- the C3 confirmation item recorded in the oncall migration closure (2026-09-01). Until that wave lands, this README describes the tool as it stands.
A budget-bounded loop that improves an agent harness against a benchmark:
diagnose failing trajectories, design candidate patches, screen them cheaply,
confirm survivors at K=3, and promote only what beats the baseline through
three verification gates — with a sealed test set for an honest
generalisation number. The methodology is specified in
docs/specs/self-evolution-loop-sop.md (the SOP) and mapped to this codebase
in docs/specs/self-evolution-loop-raven-mapping.md.
Two ways to run self-evolution
The SOP is the methodology; this package is one executor of it. There are two:
- The evolver pipeline (this package) — the automated loop below. The orchestrator executes the whole funnel unattended: diagnosis, candidate design, screening, K=3 confirmation, the three gates, promotion, and the sealed-test unseal, with resumability and the guardrails (path whitelist, fixed denominators, config fingerprint) enforced in code. Use this for long multi-round runs where the numbers have to be trustworthy.
- An agent driving the SOP directly — hand
docs/specs/self-evolution-loop-sop.mdto a coding agent (e.g. Claude Code) inside the subject repo and have it walk the steps itself: read failing trajectories, pick a WHY, write a candidate as a git commit, run the benchmark scorer for the K evaluations, and apply the gate arithmetic by hand. Nothing enforces the protocol — sealed-test discipline, fixed denominators, and honest gate math are only as good as the agent's adherence — so treat results as exploratory. It is the fastest way to prototype on a benchmark that has noBenchBundleplugin yet, and what it learns (WHY taxonomy, whitelist, scorer wiring) feeds directly into writing one (docs/specs/evolve-bench-contract.md).
The rest of this README covers way 1.
Glossary (terms used below)
- WHY — a failure-cause class (e.g. "stops before verifying"); each round targets one or two WHYs. WHERE — the code location a patch touches.
- K — attempts per task per evaluation; K=3 confirm is the SOP standard.
- The funnel — cheap screen (K=1 on a small anchor task set) before the expensive full-train K=3 confirmation; most candidates die cheap.
- Three gates — Gate-f: enough valid (non-infra) measurements; Gate-b: the candidate's code actually executed (beacon fired); Gate2: paired statistical significance vs the baseline on the same tasks.
- Sealed test — the test split is scored but stored where no decision
step can read it; opened once, at the end (
retention.json). - BenchBundle — the plugin object a benchmark implements to become evolvable (see "Adding a benchmark").
Quickstart
All commands run from the repo checkout root (they invoke the
evolver module; from elsewhere, add --project /path/to/checkout
to uv run).
# 1. Write a run spec (copy docs/examples/evolve_appworld.yaml and edit paths)
# 2. Validate everything cheap — config, models, bench setup, plus one small
# probe completion against the subject endpoint; no trials run:
uv run python -m evolver check --config my_run.yaml
# 3. Tiny wiring run (isolated <work_dir>_smoke; minutes, not hours):
uv run python -m evolver run --config my_run.yaml --smoke
# 4. The real run — one command does all three phases:
# cold-start baseline (if missing) -> evolution rounds -> unseal + retention
uv run python -m evolver run --config my_run.yaml
A real run takes hours to days, and the cost is dominated by subject-agent
evaluations, not LLM calls: cold start = train x K trials (e.g. 90 x 3),
plus roughly a full train x K evaluation per candidate that survives
screening. The loop's own calls per round are modest — tens of driver calls
(diagnosis), one design call per candidate (2-6, on your design model,
typically the expensive one), a handful of verdict calls. Interrupt freely —
re-running the same command resumes from the last durable artifact:
completed trials are never re-run (trial-level idempotency), completed
rounds are replayed from the journal.
uv run python -m evolver status --config my_run.yaml # progress; never
# reveals test numbers
uv run python -m evolver finalize --config my_run.yaml --yes
# end now + unseal (one-way)
What you get at the end: the console summary plus, under work_dir/,
retention.json (the sealed-test verdict: best round, train/test curve,
retention rate, paired significance), nodes/*.json (every candidate's
ledger: git commit, final status, gate stats), findings.md (human-readable
per-round log), and the promoted harness as a real git commit in your
subject repo.
Interruption semantics
| Interrupted during | What exists | status shows |
|---|---|---|
| phase 1 (cold start) | partial baseline trials (kept) | trial count, "no results yet" |
| phase 2 (rounds) | commit list: promoted nodes + train scores | rounds, candidates by status — test stays sealed |
| phase 3 (unseal) | everything | the final report |
While a run is resumable, test numbers are physically withheld (they sit in a
directory nothing in the decision path reads). finalize is the explicit
trade: end the run now, see the result, never resume — the one-way stamp in
run_meta.json enforces it. The same file records a config fingerprint: a
run refuses to resume under a changed configuration (candidate and control
arms must stay comparable).
Config file
See docs/examples/evolve_appworld.yaml for the annotated schema. The
sections: bench (registered benchmark name), repo_root/base_sha (the
subject repo being evolved and its root commit; omit base_sha to use the
repo's current HEAD, resolved once at launch and pinned for the whole run —
uncommitted changes are never part of the root), models (the loop's
driver/design/verdict brains — omit entirely to use Raven's own configured
model; note this is distinct from the subject agent's model, which is
pinned inside the bench config for the whole run), funnel (K values,
per-round budget, termination), bench_config (bench-owned), and smoke
(overlay applied by --smoke).
Bootstrap: AppWorld
The shipped example bench. One-time setup:
-
Install AppWorld into a directory of its own — it is the subject environment, kept out of this repo's uv env. In that directory: create a venv at
appworld-venv/,pip install appworld && appworld install, thenappworld download data— it must end up with adata/folder. Pointbench_config.appworld_data_rootat the directory. -
Write the subject runtime config JSON (the benchmarked agent's model endpoint — not the loop's models). Copy
docs/examples/subject_runtime.jsonand fill in your endpoint;checkvalidates it against the schema. Pointbench_config.config_pathat it. -
Put train (and optionally test) task ids in text files, one id per line. Valid ids come from AppWorld's official splits:
APPWORLD_ROOT=<install> <install>/appworld-venv/bin/python -c \ "from appworld import load_task_ids; print('\n'.join(load_task_ids('train')))" > train.txt(
APPWORLD_ROOTmatters: appworld resolvesdata/relative to the current directory otherwise. The bench exports it automatically at run time, but this manual step happens before any bench code runs.)(splits:
train,dev,test_normal,test_challenge; usingdevas the sealed test file is the usual setup). Placeholder ids like<failing-task-1>are refused atchecktime. -
check, thenrun --smoke, thenrun. For thesmoke:section's pinned tasks, any 2-3 train ids work for a wiring test; ids your agent fails make the smoke round exercise the full design path (you learn which ones from the cold start of a real run, or a quick manual batch).
Adding a benchmark
Implement one build(ctx) -> BenchBundle function and register it — the
contract (scorer subprocess, result reader with an infra marker, trajectory
renderer, splits, editable-path whitelist) is documented in
docs/specs/evolve-bench-contract.md. benchmarks/appworld/evolve/entry.py
is the reference implementation.
Documentation map
| Document | What it is |
|---|---|
| this README | user-facing: quickstart, config, bootstrap, security |
docs/specs/self-evolution-loop-sop.md |
the methodology spec (SOP; English translation of the upstream document) |
docs/specs/self-evolution-loop-raven-mapping.md |
SOP clause -> Raven code, deviations, unwired parts |
docs/specs/evolve-bench-contract.md |
what a new benchmark implements |
orchestrator/DESIGN.md |
design notes: inversion of control, SOP<->module table, cross-module conventions |
Status and known limitations (2026-07)
- Verified: the unit/e2e suite (126 tests: gate math, the infra-rerun
ladder, git commit/worktree primitives, sandbox whitelist capture, driver
transports, config, guards, idempotency, interrupt -> resume -> finalize
with a fake bench) plus a real AppWorld smoke through the unified entry: cold start -> resume -> one full round
(real diagnose/design, a real candidate commit, focused probe + confirm,
promoted with an honest
credited=FalseGate-b verdict) -> termination. - Not yet exercised at scale: a full-size run (90 tasks x K=3, multi-round, hours) and the sealed-test unseal/retention path on a real benchmark have only been exercised via tests, not a production run.
- One bench example ships today (AppWorld, built-in scorer line); a framework-line example (external scorer) is planned.
Security notes
This system has an LLM edit and execute code. Understand the boundaries before running it anywhere sensitive:
- Candidate edits are constrained to a per-bench path whitelist
(
path_guard/ sandbox capture); everything else the designer touches is reverted, and each whitelist prefix is validated at startup. On top of the whitelist, an immutable kernel (applier/path_guard.py) carves out the measurement surface — the evolver itself, and for AppWorld the scorer (evolve/includinggrade.py, andbatch.py) — so a candidate can rewrite the agent (prompt, loop wiring, tools) but never the code that grades it or records infra failures. - Threat model: these guards stop an LLM designer from taking shortcuts,
not a determined adversary. Candidate code runs in the same process as
the (immutable-on-disk) grader and the benchmark oracle is reachable over
HTTP, so in-memory tampering or oracle probing is possible in principle.
Promoted candidates are a handful of small commits — diff-audit them
before citing numbers (
nodes/*.jsonrecords every commit). - Candidate code runs during evaluation with the same privileges as your benchmark scorer, and the design-step sandbox is not filesystem/network jailed. Run evolutions in an isolated environment (container/VM, scoped credentials), not on a workstation with live secrets.
- Python candidates are required to carry an
activation_beacon()call, and credit is only assigned when the beacon actually fired (Gate-b) — this is an attribution mechanism (presence-level, see the mapping doc), not a sandbox. - The default per-round baseline is frozen at cold start (cost-bound); if
your subject endpoint's throughput drifts across a long run, set
bench_config.baseline_mode: same_session(~2x eval cost) so control and candidate arms are always measured in the same window. - The design step sends your failing trajectories to whatever model you configure; treat trajectory content accordingly.