zhanghui a018351f95
chore(*): gate call correctness with a pinned type checker (#416)
## Summary

Nothing in this repo could see a signature change reach its callers.
Ruff reads one file at a time, the import contracts read edges rather
than calls, and the suite only covers lines it executes. The consequence
was concrete: a required `sandbox_home` argument added to
`BoxliteExecutor` reached `plugins-dist/design-engine`'s render path
with CI green, and a reviewer -- not a gate -- found it.

`ty` answers that question. This adds it as a pinned dev dependency, a
`make lint-types` target and a CI step, with **110 of its 133 rules
live**: nine switched off with their measured backlog recorded beside
each, fourteen shipped disabled upstream and untouched.

Getting there meant clearing 81 diagnostics. This PR folds what were
three stacked PRs (#422 and #423, now closed); each rule's story is a
commit of its own.

### 1. The gate, at 108 rules (`chore(*): gate call correctness...`)

21 diagnostics to clear, five of them real:

| Where | What |
|---|---|
| `evolver/orchestrator/production.py` | `design_of` annotated as taking
two parameters while its docstring, its only implementation and its one
call site all use three |
| `raven/cli/provider_commands.py` | `dict[str, callable]` -- the
builtin function, not a type, so the annotation asserted nothing |
| `raven/config/schema.py` | `Config` is a `BaseSettings` but declared
`env_prefix` / `env_nested_delimiter` through pydantic's `ConfigDict`,
which does not carry those keys |
| `raven/spine/delivery.py` | `StreamDelta.conversation_id` is optional
and was used as a `dict[str, str]` key; a delta without one stored a
`None` entry the close path then read as "no open stream", losing that
stream's done marker |
| `research_flow/support/harness_text.py` | a predicate table typed its
matchers as `object`, so nothing checked they are callable |

The rest were unproven invariants and annotations wider than the code:
three late-bound callables asserted where a comment used to carry the
claim, one closure binding the callable it guarded instead of re-reading
it, four `hasattr` probes that now check what the call actually needs,
and a contextmanager yielding the value it just set.

### 2. `unresolved-import` (`fix(*): resolve the imports the type gate
could not`)

21 diagnostics; three were stale module paths nothing had run into:

- `HarnessNode.current_core_version()` read `raven.__core_version__`, a
module that exists neither here nor in the wheel
`scripts/build_core_wheel.py` produces. The read sat inside `try/except
ImportError`, so the except branch was the only branch and **every node
`store.py` saved recorded `core_version="unknown"`**. It reads
`raven.__version__` now and returns `0.1.13`.
- `raven/agent/loop/wiring.py` imported `SkillForgeRouterConfig` from
`raven.config.schema`; it lives in `raven.config.raven`. Under
`TYPE_CHECKING`, so the annotation simply meant nothing.
- three scripts imported `raven.context_engine.base` and
`raven.memory_engine.base`, neither of which exists. **Two failed at
module import** -- running them was not possible.

The other eighteen were the environment: the three agent products keep
their plugin package beside them, so their roots are declared in
`[tool.ty.environment]`; `lxml` ships a compiled `etree` with no type
information, so `lxml-stubs` is a dev dependency (real types rather than
five silenced lines); `uno` and `com.sun.star.*` are suppressed per
line, since that worker runs inside LibreOffice's own interpreter.

### 3. `invalid-return-type` (`fix(*): make the declared return types
true`)

39 diagnostics, and **one stale annotation held eight of them**:
`run_turn` and the `_run_turn` behind it were declared `-> LoopOutcome`
while all three of their returns build a `TurnOutcome` -- a different
dataclass with no field in common. Nothing broke, because the spine
worker reads what it actually gets and `LoopOutcome` never leaves
`_run_agent_loop`. The annotation was the only thing wrong.

The other 31: nine annotations narrower than the code (five `return
tuple(<genexpr>)` typed as fixed-arity tuples, an inner `one()` still
declaring the two-tuple it returned before a third element was added,
`image_block`'s TypedDict), five declarations refusing a value the
function returns (four tool `execute` overrides saying `-> str` while
returning `ToolResult`), seven invariants that lived in a `# type:
ignore` comment, and ten honest widenings.

### 4. What the gates then caught in this PR's own work

- The qq factory guard passed locally only because an earlier test had
left an event loop behind. Reproduced the CI state
(`asyncio.set_event_loop(None)`), then made the test async so the loop
is its own.
- `invalid-return-type`, while still disabled, hid a regression
introduced here: the qq factory returning the class instead of an
instance. Found by diffing the rule histogram, which is why that rule is
on now.
- The diff-coverage gate named four lines in `raven/routing/cache.py` --
the asserts this work had just added, sitting inside a `try` whose
`except Exception` would have swallowed the `AssertionError` and served
the stale cache. `_do_refresh` returns its data now; the asserts, their
noqa and the swallowing all go with it.

`child_scope`, `QQChannel.start()` and `BenchmarkCache` had no tests at
all and have them now. Every one was mutation-checked.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [x] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- `make lint-types` -> All checks passed; 81 diagnostics cleared across
the three rule sets
- Gate proven live three times, once per rule: a missing argument, an
import of a module that does not exist, and a function returning a str
under `-> int` each turn it red, and removing the probe turns it green
- `uv run ruff check` + `ruff format --check` over all six trees ->
clean
- `uv run lint-imports` -> 10 kept, 0 broken; `deptry raven` (all
extras) -> no issues; `uv lock --check` -> resolved
- `make check-source-language` -> exit 0; `commitlint` -> exit 0
- `make coverage` locally: **22050 passed, 45 skipped, 4 failed**; diff
coverage 92.59%
- All four failures are **pre-existing on `main`**, checked by a real
checkout in the same environment where they fail identically: two
`test_the_products_tool_face_*` launcher tests,
`test_a_ceiling_as_large_as_the_window_still_leaves_room_for_history`
(`reserved_tools = 6044 > 6_000`), and
`test_a_missing_key_is_said_once_rather_than_per_query`
(environment-dependent: it asserts `SERPER_API_KEY` is absent)
- Rule counts measured with both `--extra dev` and `--all-extras` --
identical, so this gate's verdict does not depend on which optional
packages are installed

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

Behaviour changes are confined to the fixes listed above, and each
replaces an error path with a correct one: nodes are stamped with the
real raven version instead of `"unknown"`, two scripts that could not
start now start, the stream bookkeeping no longer stores a `None` key,
and a cache refresh returns what it fetched. Everything else is a
declaration moved to match what the function already returned.

`tests/` is out of the gate's scope (113 diagnostics under the same
rules). Tests are executed, so a broken call there is already loud; this
gate is for code CI does not run. Bringing them in is sized, separate
work.

Rollback: revert the whole PR to drop the gate and its fixes together,
or delete the `[tool.ty]` block, the `lint-types` target and its CI step
to keep the fixes and drop only the gate.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-16 07:33:14 -07:00

706 lines
28 KiB
Python

"""Harness node schema + JSON round-trip.
Each evolver iteration produces a :class:`HarnessNode` — one version of
the harness with metadata about how it was made, how it performed, and
what next iterations might do.
The structure (spec §12.2) has five blocks:
1. **Identity**: ``node_id`` / ``parent_id`` / Git pointers / created_at.
2. **AppliedPatch**: what was changed relative to ``parent_id`` to create
this node — ``target_file``, ``patch_where``, ``patch_why``, diff,
reasoning, source-evidence pointers back to trajectories.
3. **EvalResult**: what was measured (bandit task subset, per-task
pass/fail, subset pass rate, dense signals, L1 alert count).
4. **JudgeAnalysis**: what the judge inferred from this node's
trajectories — counts of L1/L2/L3 seen + candidate patches the judge
proposes for future child nodes.
5. **Status**: where this node stands in the evolver lifecycle.
Storage layout (spec §12.3):
- Git commit handles the physical code state of the node (we don't
duplicate it here).
- One JSON file per node in ``evolver/nodes/<node_id>.json`` holds the
metadata defined here.
- Trajectory data lives in ``evolver/trajectories/`` keyed by id; this
schema only carries trajectory *pointers*, never inline bodies.
Why explicit ``to_dict`` / ``from_dict`` instead of stdlib ``dataclasses.asdict``?
Two reasons:
- We need to deserialise enums (``PatchWhere``, ``PatchWhy``,
``NodeStatus``) from their string values, which ``asdict`` doesn't help
with on load.
- Tuples like ``turn_range = (start, end)`` round-trip through JSON as
lists — we want to coerce back to tuple on load for type stability.
Both are surface-level concerns, but the explicit code keeps the failure
modes loud (missing fields raise, unknown enums raise).
"""
from __future__ import annotations
import json
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from pathlib import Path
from typing import Any, Optional, Union
from evolver.judge.schema import PatchWhere, PatchWhy
# ---------------------------------------------------------------------------
# Status enum
# ---------------------------------------------------------------------------
class NodeStatus(str, Enum):
"""Where a node stands in the evolver lifecycle.
- ``active``: live in the tree, eligible for bandit-on-nodes selection.
- ``pruned_low_score``: still in archive (Git keeps the commit) but
bandit deprioritises it; rare for it to be re-selected.
- ``pruned_inert``: culled at the zero-GPU preflight (SOP §2 ③) — its
trigger predicate had zero hits over the historical trajectory corpus,
so the mechanism was never applied or evaluated. Distinct from
``pruned_at_screen``: an inert death indicts the trigger's
reachability, not the mechanism body.
- ``pruned_at_screen``: culled at the K=1 anchor screen — its anchor-mean
pass@1 fell more than ``cull_threshold`` below vanilla, so it never ran
the full-set confirm (SOP §2 ⑤a: screen on the anchor, then return).
- ``pruned_at_confirm``: ran the full-set K=3 confirm but did not beat
vanilla / pass the gates (SOP §2 ⑥).
- ``blocked_l1``: judge flagged an L1 infra bug while evaluating this
node; evolver paused until human review.
- ``promoted_to_baseline``: this node became a new baseline (rare;
happens at major lift inflection points).
- ``archived_methodology_failure``: rewards from this node are
contaminated by a methodology defect; excluded from the scheduler
at load time so the bandit never learns from tainted outcomes.
- ``errored``: apply or evaluation raised for this candidate; the round
recorded the reason and skipped it rather than aborting (a single
candidate's crash must not sink the round).
"""
active = "active"
pruned_low_score = "pruned_low_score"
pruned_inert = "pruned_inert"
pruned_at_screen = "pruned_at_screen"
pruned_at_confirm = "pruned_at_confirm"
blocked_l1 = "blocked_l1"
promoted_to_baseline = "promoted_to_baseline"
archived_methodology_failure = "archived-methodology-failure"
errored = "errored"
# ---------------------------------------------------------------------------
# Patch-related substructures
# ---------------------------------------------------------------------------
@dataclass
class SourceEvidence:
"""A pointer into a trajectory used as evidence for a patch.
A single applied patch may cite multiple trajectories (e.g., five
trajectories all hit ``repetition_breaker`` at turns 20-30); each is
one ``SourceEvidence``.
"""
trajectory_id: str
turn_range: tuple[int, int]
finding: str # one-line summary of what the evidence shows
def to_dict(self) -> dict[str, Any]:
return {
"trajectory_id": self.trajectory_id,
"turn_range": list(self.turn_range),
"finding": self.finding,
}
@classmethod
def from_dict(cls, d: dict[str, Any]) -> "SourceEvidence":
tr = d.get("turn_range")
if not isinstance(tr, (list, tuple)) or len(tr) != 2:
raise ValueError(f"SourceEvidence.turn_range must be 2-element list/tuple, got {tr!r}")
return cls(
trajectory_id=_require(d, "trajectory_id", "SourceEvidence"),
turn_range=(int(tr[0]), int(tr[1])),
finding=_require(d, "finding", "SourceEvidence"),
)
@dataclass
class PatchComponent:
"""One independently-rollbackable piece of a multi-file patch.
A patch may bundle several file edits that together implement one
semantic improvement (e.g., add a new hook file + register it in a
config + reference it from a prompt template). Each such file edit
is one ``PatchComponent``. The bundle hangs together via the
``depends_on`` graph: if ``comp_3`` lists ``comp_1`` as dependency,
``comp_3`` MUST be applied after ``comp_1`` (and cannot survive in
isolation when ``comp_1`` is dropped during bisect).
Component-level bisect (spec §18.5.1.x): when an applied patch
causes regression, the evolver tries dropping subsets of components
(respecting ``depends_on``) and keeping the highest-quality subset
that still beats the parent.
For the simple "1 file, 1 fix" case, an ``AppliedPatch`` carries
exactly one ``PatchComponent`` — the multi-component machinery
becomes a no-op.
"""
component_id: str # "comp_1" / "comp_2" — unique within one patch
target_file: str # repo-relative path
diff: str # unified diff for THIS file only
rationale: str # what this component does
depends_on: list[str] = field(default_factory=list) # other component_ids
def to_dict(self) -> dict[str, Any]:
return {
"component_id": self.component_id,
"target_file": self.target_file,
"diff": self.diff,
"rationale": self.rationale,
"depends_on": list(self.depends_on),
}
@classmethod
def from_dict(cls, d: dict[str, Any]) -> "PatchComponent":
return cls(
component_id=_require(d, "component_id", "PatchComponent"),
target_file=_require(d, "target_file", "PatchComponent"),
diff=_require(d, "diff", "PatchComponent"),
rationale=_require(d, "rationale", "PatchComponent"),
depends_on=list(d.get("depends_on") or []),
)
@dataclass
class AppliedPatch:
"""The actual patch that was applied to the parent to create this node.
Multi-component design (spec §12.2, updated 2026-05-28):
A patch is a non-empty ordered list of ``PatchComponent``.
Most patches have exactly one component (single-file edit). Multi-file
coupled patches use ``components`` with explicit ``depends_on`` so the
evolver can do component-level bisect on regression (§18.5.1.x).
Convenience properties for the common single-component case:
- ``target_file`` returns the first component's file
- ``diff`` returns the concatenated diff across components
The canonical code state still lives in the node's Git commit;
components describe *how* the diff is logically structured.
"""
patch_where: PatchWhere
patch_why: PatchWhy
components: list[PatchComponent] # >= 1
overall_reasoning: str # judge's reasoning at the whole-patch level
source_evidence: list[SourceEvidence] = field(default_factory=list)
patch_why_extra: Optional[str] = None # only when patch_why=other
# Bisect bookkeeping (spec §18.5.1.x). Set when a patch was rescued
# from regression by dropping some components.
partial: bool = False
dropped_components: list[str] = field(default_factory=list)
def __post_init__(self) -> None:
if self.patch_why == PatchWhy.other and not self.patch_why_extra:
raise ValueError("AppliedPatch with patch_why=other requires non-empty patch_why_extra")
if not self.components and self.patch_where != PatchWhere.control:
raise ValueError("AppliedPatch requires at least one component")
# Component ids must be unique within the patch
ids = [c.component_id for c in self.components]
if len(ids) != len(set(ids)):
raise ValueError(f"AppliedPatch components have duplicate component_id: {ids}")
# depends_on references must resolve to existing component_ids
valid_ids = set(ids)
for c in self.components:
for dep in c.depends_on:
if dep not in valid_ids:
raise ValueError(
f"PatchComponent {c.component_id!r} depends_on {dep!r} which is not a sibling component"
)
@property
def target_file(self) -> Optional[str]:
"""Primary file = first component's file (single-component default).
Returns None for control-arm nodes that carry no components.
"""
return self.components[0].target_file if self.components else None
@property
def diff(self) -> str:
"""Concatenated diff across all components (Git-applicable as a whole)."""
return "\n".join(c.diff for c in self.components)
def to_dict(self) -> dict[str, Any]:
return {
"patch_where": self.patch_where.value,
"patch_why": self.patch_why.value,
"patch_why_extra": self.patch_why_extra,
"overall_reasoning": self.overall_reasoning,
"components": [c.to_dict() for c in self.components],
"source_evidence": [e.to_dict() for e in self.source_evidence],
"partial": self.partial,
"dropped_components": list(self.dropped_components),
}
@classmethod
def from_dict(cls, d: dict[str, Any]) -> "AppliedPatch":
components_raw = _require(d, "components", "AppliedPatch")
patch_where_raw = d.get("patch_where", "")
is_control = patch_where_raw == PatchWhere.control.value
if not isinstance(components_raw, list) or (not components_raw and not is_control):
raise ValueError("AppliedPatch.components must be a non-empty list")
return cls(
patch_where=_coerce_enum(
_require(d, "patch_where", "AppliedPatch"),
PatchWhere,
"patch_where",
),
patch_why=_coerce_enum(
_require(d, "patch_why", "AppliedPatch"),
PatchWhy,
"patch_why",
),
components=[PatchComponent.from_dict(c) for c in components_raw],
overall_reasoning=_require(d, "overall_reasoning", "AppliedPatch"),
source_evidence=[SourceEvidence.from_dict(e) for e in (d.get("source_evidence") or [])],
patch_why_extra=d.get("patch_why_extra"),
partial=bool(d.get("partial", False)),
dropped_components=list(d.get("dropped_components") or []),
)
# ---------------------------------------------------------------------------
# Evaluation substructure
# ---------------------------------------------------------------------------
@dataclass
class PerTaskResult:
"""One task's outcome under this node.
``pass_outcome`` rather than ``pass`` to avoid the Python keyword.
``turns`` and ``elapsed_sec`` are optional because the eval driver
may or may not surface them depending on the task.
"""
pass_outcome: bool
turns: Optional[int] = None
elapsed_sec: Optional[float] = None
def to_dict(self) -> dict[str, Any]:
d: dict[str, Any] = {"pass": self.pass_outcome}
if self.turns is not None:
d["turns"] = self.turns
if self.elapsed_sec is not None:
d["elapsed_sec"] = self.elapsed_sec
return d
@classmethod
def from_dict(cls, d: dict[str, Any]) -> "PerTaskResult":
if "pass" not in d:
raise ValueError("PerTaskResult missing required 'pass'")
return cls(
pass_outcome=bool(d["pass"]),
turns=d.get("turns"),
elapsed_sec=d.get("elapsed_sec"),
)
@dataclass
class EvalResult:
"""Aggregated evaluation of a single node.
``dense_signals`` is a flexible bag — current planned keys
(avg_token_usage / avg_turn_count / redundancy_rate /
test_run_frequency) are documented but not enforced, so future signals
can be added without schema migration.
"""
bandit_tasks_chosen: list[str]
per_task_results: dict[str, PerTaskResult]
subset_pass_rate: float
dense_signals: dict[str, float] = field(default_factory=dict)
l1_alert_count: int = 0
def __post_init__(self) -> None:
if not 0.0 <= self.subset_pass_rate <= 1.0:
raise ValueError(f"subset_pass_rate must be in [0,1], got {self.subset_pass_rate}")
if self.l1_alert_count < 0:
raise ValueError(f"l1_alert_count must be >= 0, got {self.l1_alert_count}")
def to_dict(self) -> dict[str, Any]:
return {
"bandit_tasks_chosen": list(self.bandit_tasks_chosen),
"per_task_results": {k: v.to_dict() for k, v in self.per_task_results.items()},
"subset_pass_rate": self.subset_pass_rate,
"dense_signals": dict(self.dense_signals),
"l1_alert_count": self.l1_alert_count,
}
@classmethod
def from_dict(cls, d: dict[str, Any]) -> "EvalResult":
return cls(
bandit_tasks_chosen=list(_require(d, "bandit_tasks_chosen", "EvalResult")),
per_task_results={
k: PerTaskResult.from_dict(v) for k, v in _require(d, "per_task_results", "EvalResult").items()
},
subset_pass_rate=float(_require(d, "subset_pass_rate", "EvalResult")),
dense_signals=dict(d.get("dense_signals") or {}),
l1_alert_count=int(d.get("l1_alert_count", 0)),
)
# ---------------------------------------------------------------------------
# Judge analysis substructure
# ---------------------------------------------------------------------------
@dataclass
class ProposedComponent:
"""A proposed component of a future patch (judge output, no diff yet).
Counterpart to :class:`PatchComponent`: the judge describes WHAT to
change (target_file + natural-language summary) but doesn't write
the actual diff. The mutation operator (e.g., GEPA library) later
turns each ``ProposedComponent`` into a concrete ``PatchComponent``
with a unified diff.
"""
component_id: str
target_file: str
summary: str # natural language description of the proposed change
depends_on: list[str] = field(default_factory=list)
def to_dict(self) -> dict[str, Any]:
return {
"component_id": self.component_id,
"target_file": self.target_file,
"summary": self.summary,
"depends_on": list(self.depends_on),
}
@classmethod
def from_dict(cls, d: dict[str, Any]) -> "ProposedComponent":
return cls(
component_id=_require(d, "component_id", "ProposedComponent"),
target_file=_require(d, "target_file", "ProposedComponent"),
summary=_require(d, "summary", "ProposedComponent"),
depends_on=list(d.get("depends_on") or []),
)
@dataclass
class CandidatePatch:
"""A patch the judge proposes for a *future* child node.
These are not yet applied — the bandit-on-nodes / bandit-on-WHY will
later pick one of these from this node's pool to spawn an actual child
(with its own ``AppliedPatch``). The bridge from candidate to applied
is the mutation operator producing the unified diff.
Multi-component design (spec §12.2, updated 2026-05-28): a candidate
patch carries one or more :class:`ProposedComponent`. The judge is
encouraged to keep N=1 for simple single-file fixes, and to use N>1
only when the change is structurally coupled (e.g., new hook file +
config registration). ``depends_on`` between components is honored
by both the mutation operator and any component-level bisect during
evolution.
"""
patch_where: PatchWhere
patch_why: PatchWhy
components: list[ProposedComponent] # >= 1
overall_reasoning: str
source_trajectory_id: str
patch_why_extra: Optional[str] = None # only when patch_why=other
def __post_init__(self) -> None:
if self.patch_why == PatchWhy.other and not self.patch_why_extra:
raise ValueError("CandidatePatch with patch_why=other requires non-empty patch_why_extra")
if not self.components:
raise ValueError("CandidatePatch requires at least one component")
ids = [c.component_id for c in self.components]
if len(ids) != len(set(ids)):
raise ValueError(f"CandidatePatch components have duplicate component_id: {ids}")
valid_ids = set(ids)
for c in self.components:
for dep in c.depends_on:
if dep not in valid_ids:
raise ValueError(
f"ProposedComponent {c.component_id!r} depends_on {dep!r} which is not a sibling component"
)
@property
def target_file(self) -> str:
"""Primary file = first component's file (single-component default)."""
return self.components[0].target_file
def to_dict(self) -> dict[str, Any]:
return {
"patch_where": self.patch_where.value,
"patch_why": self.patch_why.value,
"patch_why_extra": self.patch_why_extra,
"overall_reasoning": self.overall_reasoning,
"components": [c.to_dict() for c in self.components],
"source_trajectory_id": self.source_trajectory_id,
}
@classmethod
def from_dict(cls, d: dict[str, Any]) -> "CandidatePatch":
components_raw = _require(d, "components", "CandidatePatch")
if not isinstance(components_raw, list) or not components_raw:
raise ValueError("CandidatePatch.components must be a non-empty list")
return cls(
patch_where=_coerce_enum(
_require(d, "patch_where", "CandidatePatch"),
PatchWhere,
"patch_where",
),
patch_why=_coerce_enum(
_require(d, "patch_why", "CandidatePatch"),
PatchWhy,
"patch_why",
),
components=[ProposedComponent.from_dict(c) for c in components_raw],
overall_reasoning=_require(d, "overall_reasoning", "CandidatePatch"),
source_trajectory_id=_require(d, "source_trajectory_id", "CandidatePatch"),
patch_why_extra=d.get("patch_why_extra"),
)
@dataclass
class JudgeAnalysis:
"""The judge's verdict on this node's trajectories.
``issue_types_seen`` counts L1 / L2 / L3 occurrences across the
node's evaluated trajectories — used by the L1 routing layer to
decide whether to pause the evolver (high L1 count) and by
bandit-on-WHY to rebalance pathology coverage.
``candidate_patches`` is the menu the evolver chooses from for the
next iteration of child-node creation.
"""
issue_types_seen: dict[str, int] = field(default_factory=dict)
candidate_patches: list[CandidatePatch] = field(default_factory=list)
def to_dict(self) -> dict[str, Any]:
return {
"issue_types_seen": dict(self.issue_types_seen),
"candidate_patches": [p.to_dict() for p in self.candidate_patches],
}
@classmethod
def from_dict(cls, d: dict[str, Any]) -> "JudgeAnalysis":
return cls(
issue_types_seen=dict(d.get("issue_types_seen") or {}),
candidate_patches=[CandidatePatch.from_dict(p) for p in (d.get("candidate_patches") or [])],
)
# ---------------------------------------------------------------------------
# Top-level node
# ---------------------------------------------------------------------------
SCHEMA_VERSION = 1 # bump when breaking change to JSON layout
@dataclass
class HarnessNode:
"""One harness version in the evolver tree.
Three sub-blocks (``patch`` / ``eval`` / ``judge_analysis``) are all
``Optional`` — they get populated as the node moves through its
lifecycle:
- Root node: all three are None (no parent, no eval yet).
- Right after creation: ``patch`` set; ``eval`` and ``judge_analysis``
still None.
- After evaluation: ``eval`` populated.
- After judge runs on trajectories: ``judge_analysis`` populated.
"""
node_id: str
parent_id: Optional[str]
git_commit_sha: str
git_branch: str
created_at: str # ISO 8601 UTC
created_at_iter: int
# core_version (spec §22.5.4): the immutable-kernel version this node was
# created under. Used for cohort-controlled cross-experiment comparison —
# nodes with different ``core_version`` must not be directly compared on
# eval numbers. Read from ``raven.__version__`` at
# evolver startup and injected into each child node. Defaults to
# ``"unknown"`` so old JSON files without this field deserialise (with a
# warning logged at the call site, not here).
core_version: str = "unknown"
status: NodeStatus = NodeStatus.active
patch: Optional[AppliedPatch] = None
eval: Optional[EvalResult] = None
judge_analysis: Optional[JudgeAnalysis] = None
def __post_init__(self) -> None:
if self.created_at_iter < 0:
raise ValueError(f"created_at_iter must be >= 0, got {self.created_at_iter}")
if not self.node_id:
raise ValueError("node_id must be non-empty")
if not self.git_commit_sha:
raise ValueError("git_commit_sha must be non-empty")
if not self.core_version:
# Empty string isn't valid — caller must pass either a real
# version like "1.0.0" or the sentinel "unknown".
raise ValueError("core_version must be non-empty (use 'unknown' if not available)")
# Root invariant: no parent → no applied patch
if self.parent_id is None and self.patch is not None:
raise ValueError("root node (parent_id=None) must not carry an AppliedPatch")
@staticmethod
def utc_now() -> str:
"""ISO 8601 UTC timestamp for ``created_at`` fields."""
return datetime.now(timezone.utc).isoformat()
# -- serialisation ------------------------------------------------------
def to_dict(self) -> dict[str, Any]:
return {
"_schema_version": SCHEMA_VERSION,
"node_id": self.node_id,
"parent_id": self.parent_id,
"git_commit_sha": self.git_commit_sha,
"git_branch": self.git_branch,
"created_at": self.created_at,
"created_at_iter": self.created_at_iter,
"core_version": self.core_version,
"status": self.status.value,
"patch": self.patch.to_dict() if self.patch else None,
"eval": self.eval.to_dict() if self.eval else None,
"judge_analysis": self.judge_analysis.to_dict() if self.judge_analysis else None,
}
@classmethod
def from_dict(cls, d: dict[str, Any]) -> "HarnessNode":
ver = d.get("_schema_version", 1)
if ver != SCHEMA_VERSION:
raise ValueError(
f"HarnessNode schema version mismatch: file has {ver!r}, "
f"code understands {SCHEMA_VERSION}. Migration not yet implemented."
)
patch_raw = d.get("patch")
eval_raw = d.get("eval")
judge_raw = d.get("judge_analysis")
# Missing ``core_version`` is tolerated for backwards compat with
# JSON files written before §22.5 — the node falls back to
# ``"unknown"`` and downstream cohort analysis must exclude or
# specially mark these nodes (spec §22.5.4).
return cls(
node_id=_require(d, "node_id", "HarnessNode"),
parent_id=d.get("parent_id"),
git_commit_sha=_require(d, "git_commit_sha", "HarnessNode"),
git_branch=_require(d, "git_branch", "HarnessNode"),
created_at=_require(d, "created_at", "HarnessNode"),
created_at_iter=int(_require(d, "created_at_iter", "HarnessNode")),
core_version=d.get("core_version", "unknown"),
status=_coerce_enum(
d.get("status", NodeStatus.active.value),
NodeStatus,
"status",
),
patch=AppliedPatch.from_dict(patch_raw) if patch_raw else None,
eval=EvalResult.from_dict(eval_raw) if eval_raw else None,
judge_analysis=JudgeAnalysis.from_dict(judge_raw) if judge_raw else None,
)
@staticmethod
def current_core_version() -> str:
"""Read the current immutable-kernel version from
:mod:`raven` version (spec §22.5).
Evolver code should call this when constructing new
:class:`HarnessNode` instances so the node carries the kernel
version it was created under. Falls back to ``"unknown"`` when raven
is not importable at all, which a bench run outside the install is.
"""
try:
from raven import __version__
return __version__
except ImportError:
return "unknown"
def save(self, path: Union[str, Path]) -> None:
"""Write this node's metadata to ``path`` as pretty JSON.
Parent dirs are created on demand so callers can pass
``evolver/nodes/<node_id>.json`` without pre-mkdir.
"""
p = Path(path)
p.parent.mkdir(parents=True, exist_ok=True)
# Write through a temp + rename so an interrupted write doesn't
# leave a half-formed JSON file — common pattern for crash-safe
# file replacement on POSIX.
tmp = p.with_suffix(p.suffix + ".tmp")
tmp.write_text(
json.dumps(self.to_dict(), indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
tmp.replace(p)
@classmethod
def load(cls, path: Union[str, Path]) -> "HarnessNode":
p = Path(path)
raw = json.loads(p.read_text(encoding="utf-8"))
if not isinstance(raw, dict):
raise ValueError(f"node file {p} top-level must be a JSON object, got {type(raw).__name__}")
return cls.from_dict(raw)
# ---------------------------------------------------------------------------
# Internal helpers
# ---------------------------------------------------------------------------
def _require(d: dict[str, Any], key: str, what: str) -> Any:
"""Pull a required key from ``d`` or raise with a contextual message."""
if key not in d:
raise ValueError(f"{what} missing required field {key!r}")
return d[key]
def _coerce_enum(value: Any, enum_cls: type, field_name: str) -> Any:
"""Coerce a string into ``enum_cls`` or raise with valid options."""
if isinstance(value, enum_cls):
return value
if not isinstance(value, str):
raise ValueError(f"field {field_name!r} must be a string, got {type(value).__name__}")
try:
return enum_cls(value)
except ValueError as exc:
valid = [m.value for m in enum_cls]
raise ValueError(f"field {field_name!r}={value!r} not one of {valid}") from exc
__all__ = [
"AppliedPatch",
"CandidatePatch",
"EvalResult",
"HarnessNode",
"JudgeAnalysis",
"NodeStatus",
"PatchComponent",
"PerTaskResult",
"ProposedComponent",
"SCHEMA_VERSION",
"SourceEvidence",
]