Raven-1/evolver/scheduler/tree_loader.py
Handsome-wzw 7b45527b5b fix(tools): an array sent as its own JSON is still an array
## Summary

Reported as "deliver_files is broken". It is not: **the tool never ran.**

Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:

```
Error: Invalid parameters for tool 'deliver_files': files should be array
```

What the model sent was the array, spelled as a string:

```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```

`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.

## Type

- [x] Fix
- [ ] Feature

## The edge of the forgiving is the design

When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.

**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.

**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.

## Not just deliver_files

Six built-in tools declare an array or object parameter and were all exposed:

| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |

The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.

## Verification

```
make lint-python    ruff check + ruff format --check, 1850 files
make lint-imports   9 contracts kept, 0 broken
pytest tests/       19,699 passed, 7 failed (all pre-existing, see below)
```

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string

**End to end, through the real tool**, with the exact payload that failed five times in production:

```
files sent as:            str -> [{"path": ".../poster.png", "title": "..."}]
returned:                 Delivered 1 file(s): poster.png (72 B).
still "should be array":  False
registered in the store:  True
```

Twelve cases, five mutations, each killing at least one:

| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |

The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.

**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.

## Risk

- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert

## Related Issues

N/A

## Swept

`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.

- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.

Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).

## Deliberately not fixed

The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.

## Moved from !594

Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.

Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00

344 lines
12 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""Load an evolver tree (HarnessNode JSON files) into a
:class:`TreeAwareTaskScheduler`.
Bridge between the durable tree representation (
:mod:`evolver.tree.store`) and the in-memory bandit
(:mod:`evolver.scheduler.tree_aware_bandit`). This is the
canonical loader used for paper §15 #2 tree-aware bandit experiments
and for any future round's task subset selection.
Reads three kinds of outcome data from each :class:`HarnessNode`:
1. **Primary eval** — ``node.eval.per_task_results`` (single pass/fail
per task).
2. **Multi-attempt replay** — when ``node.eval.dense_signals`` carries
``k_attempt_replay_dir``, the loader walks that trial directory and
replays each per-attempt outcome (used for the v7 root's k=3 paired
baseline so the bandit sees three independent observations per
task instead of a single union).
3. **Secondary evals** — when ``dense_signals`` carries
``secondary_eval_<i>_path``, the loader reads the referenced JSON
file and adds those outcomes (used for round-1 validation runs:
11-borderline + rescue-retest-k=3).
Secondary eval formats supported (selected by
``secondary_eval_<i>_format``):
- ``task_dict_passed`` — ``{task_id: {passed: bool, ...}}`` (11-borderline)
- ``rescue_k3_attempts`` — ``{task: ..., attempts: [{passed: bool, ...}, ...]}`` (rescue retest)
- ``per_task_passed_block`` — ``{task: {passed: bool, ...}}`` (round 1 first_round_eval child block)
"""
from __future__ import annotations
import json
import logging
from collections import deque
from pathlib import Path
from typing import Optional
from evolver.scheduler.tree_aware_bandit import TreeAwareTaskScheduler
from evolver.tree.node import HarnessNode
from evolver.tree.store import EvolverTreeStore
logger = logging.getLogger(__name__)
def load_tree_into_scheduler(
nodes_dir: Path,
*,
repo_root: Optional[Path] = None,
all_task_ids: Optional[list[str]] = None,
ancestry_lambda: float = 0.7,
exploration_weight: float = 1.0,
rng_seed: Optional[int] = 42,
skip_multi_attempt_replay: bool = False,
) -> TreeAwareTaskScheduler:
"""Build a fully populated :class:`TreeAwareTaskScheduler` by reading
every :class:`HarnessNode` JSON in ``nodes_dir`` + their referenced
outcome data files.
Parameters
----------
nodes_dir
Directory containing ``<node_id>.json`` files (e.g.
``REPO_ROOT/evolver/nodes/``).
repo_root
Repository root. Used to resolve relative paths in
``dense_signals``. Defaults to ``nodes_dir.parent.parent``.
all_task_ids
Task pool the scheduler manages. Defaults to ``bandit_tasks_chosen``
from the root node (typically all 89 TB2 tasks).
ancestry_lambda / exploration_weight / rng_seed
Forwarded to :class:`TreeAwareTaskScheduler`.
skip_multi_attempt_replay
If True, don't walk ``k_attempt_replay_dir`` even if present. The
primary union outcomes get added instead. Used for fast tests.
Returns
-------
A scheduler with the full tree topology registered and all known
outcomes replayed. Ready for ``scheduler.choose(v_new_id, K=...)``.
"""
repo_root = repo_root or nodes_dir.parent.parent
store = EvolverTreeStore(repo_root=repo_root, nodes_dir=nodes_dir)
nodes = store.load_all_nodes()
if not nodes:
raise ValueError(f"No HarnessNode JSONs found in {nodes_dir}")
archived = [n for n in nodes if n.status.value.startswith("archived")]
if archived:
logger.info(
"Excluded %d archived node(s) from scheduler: %s",
len(archived),
[n.node_id for n in archived],
)
nodes = [n for n in nodes if not n.status.value.startswith("archived")]
if not nodes:
raise ValueError(f"No non-archived HarnessNode JSONs found in {nodes_dir}")
# ── Determine the task pool ──
if all_task_ids is None:
# Use the root's bandit_tasks_chosen as the canonical pool
roots = [n for n in nodes if n.parent_id is None]
if len(roots) != 1:
raise ValueError(f"Expected exactly one root node, found {len(roots)}: {[r.node_id for r in roots]}")
root = roots[0]
if root.eval is None:
raise ValueError(
f"Root node {root.node_id!r} must carry an eval to define "
f"the task pool. Pass all_task_ids explicitly if intentional."
)
all_task_ids = list(root.eval.bandit_tasks_chosen)
scheduler = TreeAwareTaskScheduler(
all_task_ids=all_task_ids,
ancestry_lambda=ancestry_lambda,
exploration_weight=exploration_weight,
rng_seed=rng_seed,
)
# ── Topological order — parents before children ──
for node in _topological_order(nodes):
scheduler.add_node(node.node_id, parent_id=node.parent_id)
_record_outcomes(
scheduler=scheduler,
node=node,
repo_root=repo_root,
skip_multi_attempt_replay=skip_multi_attempt_replay,
)
logger.info(
"Loaded tree from %s: %d nodes, %d observations",
nodes_dir,
scheduler.n_nodes(),
scheduler.n_observations(),
)
return scheduler
def _topological_order(nodes: list[HarnessNode]) -> list[HarnessNode]:
"""Return nodes ordered so every node's ``parent_id`` appears before
it. Uses BFS from roots; raises on cycles or orphan parents."""
by_id = {n.node_id: n for n in nodes}
# Pre-check: every non-root node must reference a parent that exists
# in the node set. (Otherwise BFS would silently skip it and fall
# through to the "possible cycle" branch with a less useful message.)
for n in nodes:
if n.parent_id is not None and n.parent_id not in by_id:
raise ValueError(f"Node {n.node_id!r} references parent_id {n.parent_id!r} which is not in the node set")
children_of: dict[Optional[str], list[HarnessNode]] = {}
for n in nodes:
children_of.setdefault(n.parent_id, []).append(n)
roots = children_of.get(None, [])
if not roots:
raise ValueError("No root node (parent_id=None) found")
seen: set[str] = set()
ordered: list[HarnessNode] = []
queue: deque[HarnessNode] = deque(roots)
while queue:
n = queue.popleft()
if n.node_id in seen:
continue
seen.add(n.node_id)
ordered.append(n)
for child in children_of.get(n.node_id, []):
queue.append(child)
if len(ordered) != len(nodes):
missing = {n.node_id for n in nodes} - seen
raise ValueError(f"Topological sort incomplete — possible cycle. Missing: {missing}")
return ordered
def _record_outcomes(
*,
scheduler: TreeAwareTaskScheduler,
node: HarnessNode,
repo_root: Path,
skip_multi_attempt_replay: bool,
) -> None:
"""Replay all known outcomes for ``node`` into ``scheduler``."""
if node.eval is None:
return
dense = node.eval.dense_signals or {}
# ── 1. Multi-attempt replay (root k=3 case) ──
replay_dir_rel = dense.get("k_attempt_replay_dir")
if isinstance(replay_dir_rel, str) and replay_dir_rel and not skip_multi_attempt_replay:
replay_dir = repo_root / replay_dir_rel
if replay_dir.is_dir():
_replay_k_attempts(
scheduler=scheduler,
node_id=node.node_id,
trial_dir=replay_dir,
task_pool=set(scheduler._task_ids),
)
# Skip the per_task_results union — we just replayed every attempt
else:
logger.warning(
"node %s: k_attempt_replay_dir does not exist: %s — falling back to per_task_results union",
node.node_id,
replay_dir,
)
_replay_primary(scheduler, node)
else:
# ── 1b. Primary eval (single attempt per task) ──
_replay_primary(scheduler, node)
# ── 2. Secondary evals ──
n_secondary = int(dense.get("secondary_eval_count", 0) or 0)
for i in range(n_secondary):
label = dense.get(f"secondary_eval_{i}_label")
path_rel = dense.get(f"secondary_eval_{i}_path")
fmt = dense.get(f"secondary_eval_{i}_format")
if not (isinstance(path_rel, str) and isinstance(fmt, str)):
logger.warning(
"node %s secondary_eval_%d missing path/format — skipping",
node.node_id,
i,
)
continue
path = repo_root / path_rel
if not path.is_file():
logger.warning(
"node %s secondary_eval_%d path missing: %s — skipping",
node.node_id,
i,
path,
)
continue
try:
data = json.loads(path.read_text())
except json.JSONDecodeError:
logger.warning(
"node %s secondary_eval_%d not valid JSON: %s — skipping",
node.node_id,
i,
path,
)
continue
_apply_secondary_format(scheduler, node.node_id, label, fmt, data)
def _replay_primary(scheduler: TreeAwareTaskScheduler, node: HarnessNode) -> None:
"""Add the primary eval's per_task_results as one outcome each."""
assert node.eval is not None
pool = set(scheduler._task_ids)
for task_id, result in node.eval.per_task_results.items():
if task_id not in pool:
logger.debug(
"node %s: per_task task %s not in task pool, skipping",
node.node_id,
task_id,
)
continue
scheduler.add_outcome(node.node_id, task_id, result.pass_outcome)
def _replay_k_attempts(
*,
scheduler: TreeAwareTaskScheduler,
node_id: str,
trial_dir: Path,
task_pool: set[str],
) -> None:
"""Walk a legacy-runner trial dir (each subdir = one trial = one task ×
attempt) and replay each attempt as a separate outcome."""
n_replayed = 0
for result_json in trial_dir.glob("*/result.json"):
try:
d = json.loads(result_json.read_text())
except json.JSONDecodeError:
continue
task_name = d.get("task_name")
if not task_name or task_name not in task_pool:
continue
reward = (d.get("verifier_result") or {}).get("rewards", {}).get("reward", 0.0)
passed = bool(reward) and reward > 0
scheduler.add_outcome(node_id, task_name, passed)
n_replayed += 1
logger.info(
"node %s: replayed %d k-attempt outcomes from %s",
node_id,
n_replayed,
trial_dir,
)
def _apply_secondary_format(
scheduler: TreeAwareTaskScheduler,
node_id: str,
label: Optional[str],
fmt: str,
data: dict,
) -> None:
pool = set(scheduler._task_ids)
n_added = 0
if fmt == "task_dict_passed":
# {task_id: {passed: bool, ...}}
for task_id, v in data.items():
if task_id not in pool:
continue
if isinstance(v, dict) and "passed" in v:
scheduler.add_outcome(node_id, task_id, bool(v["passed"]))
n_added += 1
elif fmt == "rescue_k3_attempts":
# {task: <task_id>, attempts: [{passed: bool, ...}, ...]}
task_id = data.get("task")
if task_id and task_id in pool:
for attempt in data.get("attempts", []):
if isinstance(attempt, dict) and "passed" in attempt:
scheduler.add_outcome(node_id, task_id, bool(attempt["passed"]))
n_added += 1
elif fmt == "per_task_passed_block":
# {task_id: {passed: bool, ...}} (same shape as task_dict_passed)
for task_id, v in data.items():
if task_id not in pool:
continue
if isinstance(v, dict) and "passed" in v:
scheduler.add_outcome(node_id, task_id, bool(v["passed"]))
n_added += 1
else:
logger.warning(
"node %s secondary_eval %r unknown format: %r — skipping",
node_id,
label,
fmt,
)
return
logger.info(
"node %s secondary_eval %r (%s): added %d outcomes",
node_id,
label,
fmt,
n_added,
)
__all__ = ["load_tree_into_scheduler"]