Raven-1/tests/test_analyst_run.py
tzteyang ad2f054dd4
feat(*): add an experimental Curator that cultivates harness-of-harnesses workers (#802)
## Summary

Adds `experimental/`, a first version of worker RSI: a Curator that
shapes a
Raven worker's harness from the task, the materials handed to it and
feedback,
the way a new employee is trained.

The worker is a harness-of-harnesses. A root Raven harness faces the
user and
calls child harnesses, each a full Raven agent hosted over ACP, through
the
nodes of its playbooks. Every harness has the same four strategy facets
(memory, planning, capability, action). The Curator reads the actual
harnesses
and generates code and config on those facets at both levels: the root
candidate sets the playbooks and writes per-node requirements, each
child is
curated against the requirements of its nodes, and the set is checked on
copies and activated together.

Layout:
- `curator/`: staged generation (understand, select, design, implement,
repair), two-level composition, harness contracts, and `raven_adapter/`,
  the only layer that touches a worker's Raven runtime
- `analyst/`: turns an evaluator's words and execution records into
  observable behavior requirements
- `iteration/`: the generic loop (onboarding curation, then trial,
review and
  curation each round) and a CLI for a human evaluator
- `simulation/`: a simulated travel agency that runs the loop, plus one
  showcase case (`simulation/cases/s0925c/`: case notes, the exported
  transcript split by phase, and the round 1 and round 3 decks)
- `docs/design.md` (structure, curation flow, composition sequence, the
Raven
  connection) and `docs/rsi-iteration.md` (the loop and the Analyst)

Key decisions:
- No change under `raven/`, and `raven/` does not import
`experimental/`.
Generated artifacts land on Raven's existing extension points. The few
Raven
internals the adapter reads are listed in `docs/design.md` section 5.4.
- The Curator explores a read-only source snapshot; the caller names the
evaluation side to withhold from it, so the generic layers name no
scenario.
- The single-layer owner analysis is kept only to reproduce the showcase
case
  and is marked for removal; the two-layer Analyst is the default.
- The two decks are `.pptx` files of about 230 KB each. They pass
  `check-large-files`; flagging them in case they count as report assets
under AGENTS.md section 7, in which case they can move out of git behind
a
  link.
- One follow-up for Raven 0.2.2: playbooks in prompt and rounds modes
ship no
  fixed nodes, so node requirements are read from dag playbooks only.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

On the branch rebased onto the latest `main`:

- `uv run pytest tests/test_harness_curator_*.py tests/test_analyst_*.py
tests/test_iteration_*.py tests/test_simulation_*.py -q`: 368 passed
- `uv run pytest -m integration
tests/integration/test_harness_curator_*.py --deselect
tests/integration/test_harness_curator_real_llm.py -q`
  (offline, stub providers): 57 passed
- `uv run pytest -q` (whole suite, with the host proxy variables unset
and
`PYTHONTZPATH` set): 26692 passed, 2 failed. One was an order-dependent
test
  of this PR and is fixed here; the other,

`tests/test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing`,
  fails only because the machine runs as root and is unrelated.
- `uv run ruff check` and `uv run ruff format --check` on the touched
files:
  clean
- `scripts/check_large_files.py origin/main..HEAD`,
  `scripts/check_source_language.py origin/main..HEAD`,
  `scripts/check_commit_messages.py origin/main..HEAD` and
  `npx commitlint --from origin/main --to HEAD`: all exit 0

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

No user-visible change for Raven: `experimental/` is not imported by
`raven/` and is not packaged in the wheel (`packages = ["raven"]`).
`CONTEXT.md` gains one glossary entry. The Curator's exploration offers
a
shell only under an OS sandbox executor. Rollback is a revert of the
squash
commit; nothing else depends on `experimental/`.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Yuqi Yang <yiqiyang524@gmail.com>
Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-09-26 21:00:58 +08:00

179 lines
8.3 KiB
Python

"""Program-side triage, one bounded model exchange and the recorded feedback stay faithful to the judgements."""
import json
from types import SimpleNamespace
import pytest
from experimental.analyst.feedback import Feedback
from experimental.analyst.materials import RECORDS, read_records
from experimental.analyst.run import NAME, Limits, analyse, triage
from experimental.curator.harness import Plan, Task
from experimental.curator.raven_adapter.worker import Execution
from experimental.iteration.exchange import ExchangeError
from experimental.iteration.protocols import Exchange, Item, Signal
from raven.contracts.llm_provider import LLMResponse, ToolCallRequest
class Provider:
def __init__(self, *responses):
self.responses = list(responses)
self.requests = []
async def chat_with_retry(self, **kwargs):
self.requests.append(kwargs)
return self.responses.pop(0)
class FakeWorker:
def __init__(self, root, plan=None):
self.baseline = SimpleNamespace(task=Task(text="Serve the agency's travellers"))
self.root = root
self.last_plan = plan
async def inspect(self):
return SimpleNamespace(sources={"skill.local/pricing": {}, "reference.index": {}, "skill.local/sop": {}})
def response(name, arguments):
return LLMResponse(content=None, tool_calls=[ToolCallRequest(name, name, arguments)])
def execution(turn_id, reply, *extra):
records = [{"kind": "runner.event", "event_type": "Text", "event": {"content": reply}}, *extra]
return Execution(turn_id, [], records, {}, "artifact-1")
@pytest.fixture
def sessions():
return {
"student": [
Exchange("Plan me a trip", execution("t1", "Here is a three-day itinerary.")),
Exchange("How much?", execution("t2", "About 3000.", {"kind": "tool.error", "error": "no pricing tool"})),
]
}
REQUIREMENT = {
"behavior": "Before proposing an itinerary, ask for budget, dates, party size and departure city.",
"observed": "In turn t1 the assistant proposed a three-day itinerary knowing only the destination.",
"evidence": ["t1", "Here is a three-day itinerary."],
"expectation": "new",
"acceptance": "No itinerary appears before those four items have been asked for.",
"strength": "must_hold",
}
def test_triage_decides_only_when_nothing_needs_interpretation():
assert triage(()).decision == "continue"
passed = Signal("verifier", items=(Item("asks budget", "pass"),), satisfied=True)
assert triage((passed,)).decision == "continue"
assert triage((Signal("verifier", items=(Item("asks budget", "fail"),), satisfied=False),)) is None
assert triage((Signal("verifier", "Too pushy.", satisfied=True),)) is None
assert triage((Signal("dataset", metrics={"pass_rate": 1.0}),)) is None
assert triage((Signal("human", "Fine."),)) is None
def test_feedback_requires_requirements_exactly_for_curate():
with pytest.raises(ValueError, match="curate requires"):
Feedback(decision="curate", reason="r")
with pytest.raises(ValueError, match="carry none"):
Feedback(decision="continue", reason="r", requirements=(REQUIREMENT,))
@pytest.mark.asyncio
async def test_model_reads_records_then_submits_and_the_round_is_recorded(tmp_path, sessions):
plan = Plan(
understanding="u",
design="d",
changes=[{"target": "planning.strategy", "reason": "r", "expected": "asks first", "verification": "v"}],
)
worker = FakeWorker(tmp_path, plan)
signals = (Signal("human", "It never asked my budget before recommending."),)
provider = Provider(
response(RECORDS, {"turn_id": "t2", "kind": "tool."}),
response(NAME, {"decision": "curate", "reason": "The intake gap recurs.", "requirements": [REQUIREMENT]}),
)
previous = (Signal("dataset", metrics={"pass_rate": 0.5}),)
feedback = await analyse(worker, provider, signals, sessions, previous_signals=previous, model="analyst-model")
assert feedback.decision == "curate" and feedback.requirements[0].expectation == "new"
materials = json.loads(provider.requests[0]["messages"][1]["content"])
assert materials["signals"][0]["text"] == "It never asked my budget before recommending."
assert materials["previous_signals"][0]["metrics"] == {"pass_rate": 0.5}
assert materials["previous_expectations"][0]["expected"] == "asks first"
assert materials["skills"] == ["skill.local/pricing", "skill.local/sop"]
assert materials["sessions"]["student"][1]["errors"] == ["no pricing tool"]
assert materials["sessions"]["student"][1]["record_kinds"] == {"runner.event": 1, "tool.error": 1}
tools = {entry["function"]["name"] for entry in provider.requests[0]["tools"]}
assert tools == {RECORDS, NAME}
assert provider.requests[0]["tools"][0]["function"]["parameters"]["properties"]["turn_id"]["enum"] == ["t1", "t2"]
tool_replies = [row for row in provider.requests[1]["messages"] if row["role"] == "tool"]
query_result = json.loads(tool_replies[0]["content"])
assert query_result["turn_id"] == "t2" and "no pricing tool" in query_result["text"]
assert "runner.event" not in query_result["text"]
assert all(request["model"] == "analyst-model" for request in provider.requests)
record = json.loads(next((tmp_path / "analysis").glob("*.json")).read_text())
assert record["feedback"]["decision"] == "curate" and record["materials"]["task"] == worker.baseline.task.text
assert [row["event"] for row in record["trace"]] == ["model.call", "query", "model.call", NAME]
@pytest.mark.asyncio
async def test_invalid_submissions_are_returned_and_the_budget_is_final(tmp_path, sessions):
worker = FakeWorker(tmp_path)
signals = (Signal("human", "Not good."),)
provider = Provider(
response(NAME, {"decision": "curate", "reason": "r"}),
LLMResponse(content="Let me think."),
)
with pytest.raises(ExchangeError, match="analyst: submission budget exhausted") as info:
await analyse(worker, provider, signals, sessions, limits=Limits(max_calls=2))
assert [row["event"] for row in info.value.trace] == [
"model.call",
"output.rejected",
"model.call",
"output.missing",
]
rejection = json.loads(next(row for row in provider.requests[1]["messages"] if row["role"] == "tool")["content"])
assert "curate requires" in rejection["error"]
record = json.loads(next((tmp_path / "analysis").glob("*.json")).read_text())
assert record["error"].startswith("analyst: submission budget exhausted") and "feedback" not in record
def test_record_query_filters_by_kind_and_bounds_offsets(sessions):
result = read_records(sessions, {"turn_id": "t2", "kind": "runner", "length": 40})
assert result["next_offset"] == 40 and result["total_characters"] > 40
assert read_records(sessions, {"turn_id": "t1"})["next_offset"] is None
with pytest.raises(ValueError, match="offset exceeds"):
read_records(sessions, {"turn_id": "t1", "offset": 10**6})
with pytest.raises(ValueError, match="no turn"):
read_records(sessions, {"turn_id": "missing"})
def test_limits_reject_non_positive_bounds():
for bad in ({"max_calls": 0}, {"call_timeout": 0}, {"call_timeout": float("inf")}):
with pytest.raises(ValueError, match="finite positive"):
Limits(**bad)
def test_read_records_rejects_an_unknown_turn_as_a_parameter_error(sessions):
with pytest.raises(ValueError, match="no turn in this round has the id 'ghost'"):
read_records(sessions, {"turn_id": "ghost"})
assert json.loads(read_records(sessions, {"turn_id": "t2", "kind": "tool."})["text"])[0]["kind"] == "tool.error"
@pytest.mark.asyncio
async def test_the_last_call_offers_only_the_submission_tool(tmp_path, sessions):
worker = FakeWorker(tmp_path)
provider = Provider(
response(RECORDS, {"turn_id": "t1"}),
response(NAME, {"decision": "continue", "reason": "Nothing to change."}),
)
feedback = await analyse(worker, provider, (Signal("human", "Hm."),), sessions, limits=Limits(max_calls=2))
assert feedback.decision == "continue"
assert [entry["function"]["name"] for entry in provider.requests[0]["tools"]] == [RECORDS, NAME]
assert [entry["function"]["name"] for entry in provider.requests[1]["tools"]] == [NAME]
nudges = [
row for row in provider.requests[1]["messages"] if row["role"] == "user" and "last call" in str(row["content"])
]
assert len(nudges) == 1