Raven-1/tests/test_simulation_contracts.py
tzteyang ad2f054dd4
feat(*): add an experimental Curator that cultivates harness-of-harnesses workers (#802)
## Summary

Adds `experimental/`, a first version of worker RSI: a Curator that
shapes a
Raven worker's harness from the task, the materials handed to it and
feedback,
the way a new employee is trained.

The worker is a harness-of-harnesses. A root Raven harness faces the
user and
calls child harnesses, each a full Raven agent hosted over ACP, through
the
nodes of its playbooks. Every harness has the same four strategy facets
(memory, planning, capability, action). The Curator reads the actual
harnesses
and generates code and config on those facets at both levels: the root
candidate sets the playbooks and writes per-node requirements, each
child is
curated against the requirements of its nodes, and the set is checked on
copies and activated together.

Layout:
- `curator/`: staged generation (understand, select, design, implement,
repair), two-level composition, harness contracts, and `raven_adapter/`,
  the only layer that touches a worker's Raven runtime
- `analyst/`: turns an evaluator's words and execution records into
  observable behavior requirements
- `iteration/`: the generic loop (onboarding curation, then trial,
review and
  curation each round) and a CLI for a human evaluator
- `simulation/`: a simulated travel agency that runs the loop, plus one
  showcase case (`simulation/cases/s0925c/`: case notes, the exported
  transcript split by phase, and the round 1 and round 3 decks)
- `docs/design.md` (structure, curation flow, composition sequence, the
Raven
  connection) and `docs/rsi-iteration.md` (the loop and the Analyst)

Key decisions:
- No change under `raven/`, and `raven/` does not import
`experimental/`.
Generated artifacts land on Raven's existing extension points. The few
Raven
internals the adapter reads are listed in `docs/design.md` section 5.4.
- The Curator explores a read-only source snapshot; the caller names the
evaluation side to withhold from it, so the generic layers name no
scenario.
- The single-layer owner analysis is kept only to reproduce the showcase
case
  and is marked for removal; the two-layer Analyst is the default.
- The two decks are `.pptx` files of about 230 KB each. They pass
  `check-large-files`; flagging them in case they count as report assets
under AGENTS.md section 7, in which case they can move out of git behind
a
  link.
- One follow-up for Raven 0.2.2: playbooks in prompt and rounds modes
ship no
  fixed nodes, so node requirements are read from dag playbooks only.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

On the branch rebased onto the latest `main`:

- `uv run pytest tests/test_harness_curator_*.py tests/test_analyst_*.py
tests/test_iteration_*.py tests/test_simulation_*.py -q`: 368 passed
- `uv run pytest -m integration
tests/integration/test_harness_curator_*.py --deselect
tests/integration/test_harness_curator_real_llm.py -q`
  (offline, stub providers): 57 passed
- `uv run pytest -q` (whole suite, with the host proxy variables unset
and
`PYTHONTZPATH` set): 26692 passed, 2 failed. One was an order-dependent
test
  of this PR and is fixed here; the other,

`tests/test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing`,
  fails only because the machine runs as root and is unrelated.
- `uv run ruff check` and `uv run ruff format --check` on the touched
files:
  clean
- `scripts/check_large_files.py origin/main..HEAD`,
  `scripts/check_source_language.py origin/main..HEAD`,
  `scripts/check_commit_messages.py origin/main..HEAD` and
  `npx commitlint --from origin/main --to HEAD`: all exit 0

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

No user-visible change for Raven: `experimental/` is not imported by
`raven/` and is not packaged in the wheel (`packages = ["raven"]`).
`CONTEXT.md` gains one glossary entry. The Curator's exploration offers
a
shell only under an OS sandbox executor. Rollback is a revert of the
squash
commit; nothing else depends on `experimental/`.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Yuqi Yang <yiqiyang524@gmail.com>
Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-09-26 21:00:58 +08:00

400 lines
16 KiB
Python

"""Information-flow contracts, hop by hop, on record shapes cut from recorded runs (texts translated to English).
customer -> employee -> owner (the scenario's Analyst, or a speaker the base Analyst reads) -> Curator -> next round:
each hop gets what it needs and nothing it must not see.
"""
import json
from types import SimpleNamespace
import pytest
from experimental.analyst.activity import activity
from experimental.analyst.run import NAME as FEEDBACK
from experimental.curator.harness import Task
from experimental.curator.raven_adapter.worker import Execution
from experimental.iteration.conversation import Conversation
from experimental.iteration.protocols import Exchange
from experimental.iteration.run import Limits, history_entry, run
from experimental.simulation.agency import NAME as REVIEW
from experimental.simulation.agency import Agency, reports
from experimental.simulation.channel import seen
from experimental.simulation.scenario import BUNDLED, Scenario
from raven.contracts.llm_provider import LLMResponse, ToolCallRequest
TRAVEL = BUNDLED / "travel_agency"
CAPTION = "Your plan deck is ready: quote HL-Q-1026-4P, prices and dates match the quote sheet."
DECK = "Harbourlight-HL-Q-1026-4P.pptx"
GATE_REASON = (
"This customer-visible message carries internal notes (hit '[internal': '[internal material request] this "
"consultation...'). Rewrite it with only what the customer should read."
)
def delivery_turn():
"""A short reply, then the deck handed over with its caption."""
return [
{"kind": "runner.event", "event_type": "Text", "event": {"content": "One moment."}},
{
"kind": "runner.event",
"event_type": "ToolEvent",
"event": {
"phase": "start",
"tool_call_id": "F8IMCHIqZ",
"name": "deliver_files",
"arguments": {
"files": [{"path": f"decks/20260925_124631/{DECK}", "title": "Plan"}],
"message": CAPTION,
},
},
},
{
"kind": "runner.event",
"event_type": "ToolEvent",
"event": {
"phase": "complete",
"tool_call_id": "F8IMCHIqZ",
"name": "",
"ok": True,
"result_preview": f"Delivered 1 file(s): {DECK} (241.4 KB).",
"metadata": {"raven_delivery": {"message": CAPTION, "files": [{"name": DECK}]}},
},
},
{"kind": "runner.event", "event_type": "Progress", "event": {"content": "Checking the price list now"}},
]
def gate_turn():
"""The review gate passes four times, then sends one message back."""
rows = []
for iteration in range(1, 5):
for phase in ("execute_tools", "after_iteration"):
rows += [
{"kind": "participant.call", "target": "action.review", "phase": phase, "iteration": iteration},
{"kind": "participant.result", "target": "action.review", "phase": phase, "result": None},
]
rows += [
{
"kind": "participant.result",
"target": "action.review",
"phase": "execute_tools",
"result": {"verdict": "resample", "reason": GATE_REASON},
},
{"kind": "loop.control", "rollbacks": 1, "rollbacks_refused": 0, "mode": "high"},
{"kind": "runner.event", "event_type": "Text", "event": {"content": "Let me check that with a colleague."}},
]
return rows
def played(records, deliverables=()):
return Exchange("I accept the quote.", Execution("t1", [], records, {}, "artifact-1", tuple(deliverables)))
def test_the_customer_and_the_owner_read_the_delivery_caption_and_never_progress_lines():
text = seen(delivery_turn(), [DECK])
assert "One moment." in text and CAPTION in text
assert "Checking the price list" not in text
assert CAPTION not in seen(delivery_turn(), [])
def test_the_owner_reads_what_each_colleague_was_asked_and_answered_once():
"""A delegation result, and a playbook step's account after a skill catalog that quotes the start of the
report's marker."""
result = (
"The task the sub-agent was given:\n[BEGIN UNTRUSTED subagent #a9392399 - data]\nResearch Xi'an for the deck."
"\n[END UNTRUSTED subagent #a9392399]\n\nWhat it returned as its answer:\n[BEGIN UNTRUSTED subagent #d58d2594"
" - data]\nG1234 Hefei to Xi'an, 4h (source: 12306).\n[END UNTRUSTED subagent #d58d2594]\n\nTail of what the "
"sub-agent did, one message per line:\n[BEGIN UNTRUSTED subagent #3da887c5 - data]\nopened 12306\n"
"[END UNTRUSTED subagent #3da887c5]"
)
step = (
"[Runtime Context - metadata only]\nChannel: curator\n[BEGIN UNTRUSTED skill catalog #94b82ee1 - data]\n"
"Possibly relevant skills (matched for: '[BEGIN UNTRUSTED subagent #73781a37 - everyth'):\n- local/weather\n"
"[END UNTRUSTED skill catalog #94b82ee1]\n\nDAG run r1: node 'design-brief' needs your decision.\n\n"
"[BEGIN UNTRUSTED subagent #73781a37 - data]\nThe brief is filed as brief.md.\n[END UNTRUSTED subagent #73781a37]"
)
request = {"kind": "provider.request", "parameters": {"messages": [{"role": "user", "content": result}]}}
later = {
"kind": "provider.request",
"parameters": {"messages": [{"role": "user", "content": result}, {"role": "user", "content": step}]},
}
found = reports([played([request, later])])
assert found == [
{"task": "Research Xi'an for the deck.", "report": "G1234 Hefei to Xi'an, 4h (source: 12306)."},
{"task": "DAG run r1: node 'design-brief' needs your decision.", "report": "The brief is filed as brief.md."},
]
def test_a_gate_that_sent_work_back_reaches_the_curator_once_with_its_reason():
rows = activity({"family": [played(gate_turn())]})
acted = [row for row in rows if row["acted"]]
assert acted == [
{
"session": "family",
"scope": "root",
"target": "action.review",
"decision": "resample",
"count": 1,
"acted": True,
"reasons": [GATE_REASON],
}
]
assert history_entry(1, (), None, None, rows)["mechanisms_acted"] == acted
class Worker:
def __init__(self, root):
self.baseline = SimpleNamespace(task=Task(text="Serve the agency's travellers"))
self.root, self.last_plan, self.artifact_id = root, None, "artifact-0"
async def run(self, text, *, session_key):
return Execution("t1", [], gate_turn(), {}, self.artifact_id)
async def inspect(self):
return SimpleNamespace(sources={"skill.builtin/weather": {}}, facts={})
class Traveller:
name = "family"
def __init__(self):
self.said = ["We are four, going to Xi'an.", None]
async def speak(self, exchanges):
if not exchanges:
self.said = ["We are four, going to Xi'an.", None]
return self.said.pop(0) if self.said else None
class Owner:
def __init__(self, *arguments, names=()):
names = [*names, *[REVIEW] * (len(arguments) - len(names))]
self.responses = [
LLMResponse(content=None, tool_calls=[ToolCallRequest(name, name, a)]) for name, a in zip(names, arguments)
]
self.requests = []
async def chat_with_retry(self, **kwargs):
self.requests.append(kwargs)
return self.responses.pop(0)
@pytest.mark.asyncio
async def test_the_curator_hears_the_owner_and_the_facts_but_never_the_conversation_or_the_criteria(tmp_path):
scenario = Scenario.load(TRAVEL)
rule = next(criterion for criterion in scenario.criteria if criterion.id == "no-internal-chatter")
verdicts = [
{"id": c.id, "result": "fail" if c.id == rule.id else "pass", "session": "family"} for c in scenario.criteria
]
shortfall = {
"criteria": [rule.id],
"cause": "not_held",
"behavior": "Nothing internal ever reaches a customer.",
"observed": "In the family drill the employee wrote an internal note to the customer.",
"evidence": ["family: [internal material request]"],
"acceptance": "No customer-visible message carries internal notes.",
"strength": "must_hold",
}
said = {"verdicts": verdicts, "shortfalls": [shortfall], "remark": "It leaked an internal note."}
owner = Owner(said, said)
home = tmp_path / "home"
agency = Agency(
scenario,
owner,
home / "skills",
workdir=tmp_path / "work",
deliver="dialog",
uploads=home / "uploads",
analysis="owner",
)
agency.prepare()
heard = []
async def curator(worker, provider, *, feedback=None, **options):
heard.append(feedback)
worker = Worker(tmp_path / "run")
await run(worker, owner, [Conversation(Traveller(), max_turns=2)], agency, curator=curator, limits=Limits(2))
packet = json.loads(owner.requests[0]["messages"][1]["content"])
assert packet["conversations"]["family"] == [
{"traveller": "We are four, going to Xi'an.", "assistant": "Let me check that with a colleague."}
]
assert "resample" not in json.dumps(packet) and "action.review" not in json.dumps(packet)
feedback = heard[-1]
assert set(feedback) == {
"signals",
"history",
"mechanism_activity",
"decision",
"reason",
"requirements",
"filtered",
"task_updates",
}
assert feedback["decision"] == "curate" and feedback["requirements"][0]["behavior"] == shortfall["behavior"]
assert [(row["target"], row["decision"]) for row in feedback["mechanism_activity"] if row["acted"]] == [
("action.review", "resample")
]
told = json.dumps(feedback, ensure_ascii=False)
assert "We are four" not in told and "Let me check that" not in told
assert rule.check not in told and feedback["signals"][0]["items"] == []
def test_the_owner_sees_a_failed_write_as_failed_and_knows_when_a_text_was_cut():
from experimental.simulation.agency import back_office, cut
rows = [
{
"kind": "runner.event",
"event_type": "ToolEvent",
"event": {
"phase": "start",
"tool_call_id": "w1",
"name": "write_file",
"arguments": {"path": "handover/hold.md"},
},
},
{
"kind": "runner.event",
"event_type": "ToolEvent",
"event": {
"phase": "complete",
"tool_call_id": "w1",
"name": "",
"ok": False,
"result_preview": "Error: Path handover/hold.md is outside allowed directories",
},
},
{
"kind": "runner.event",
"event_type": "ToolEvent",
"event": {
"phase": "start",
"tool_call_id": "s1",
"name": "spawn",
"arguments": {"agent": "Raven-Research"},
},
},
{
"kind": "runner.event",
"event_type": "ToolEvent",
"event": {"phase": "complete", "tool_call_id": "s1", "name": "", "ok": True},
},
{
"kind": "runner.event",
"event_type": "ToolEvent",
"event": {
"phase": "start",
"tool_call_id": "w2",
"name": "write_file",
"arguments": {"path": "handover/hold.md"},
},
},
]
timeline = back_office([played(rows)])
assert [(row["action"], row["ok"]) for row in timeline] == [
("write_file", False),
("spawn", True),
("write_file", None),
]
assert "outside allowed directories" in timeline[0]["error"]
assert cut("x" * 10, 4) == "xxxx\n[... 6 more characters not shown]" and cut("short", 10) == "short"
def test_the_owner_reads_what_the_employee_filed_and_not_the_hosts_bookkeeping(tmp_path):
"""Raven's `.raven` checkpoint files and a deck engine's dot files are bookkeeping, not something filed."""
from experimental.simulation.agency import written
before, after = tmp_path / "before", tmp_path / "after"
before.mkdir()
for path, text in {
"handover/hold-1022.md": "ticket",
".raven/shadow.git/HEAD": "ref",
"decks/run/.deck-mode.json": "{}",
}.items():
(after / path).parent.mkdir(parents=True, exist_ok=True)
(after / path).write_text(text)
assert written(after, before, label="workdir") == {"workdir/handover/hold-1022.md": "ticket"}
@pytest.mark.asyncio
async def test_with_an_analyst_the_owner_only_speaks_and_its_scorecard_reaches_neither_analyst_nor_curator(tmp_path):
scenario = Scenario.load(TRAVEL)
rule = next(criterion for criterion in scenario.criteria if criterion.id == "no-internal-chatter")
verdicts = [
{
"id": c.id,
"result": "fail" if c.id == rule.id else "pass",
"session": "family",
"actual": "[internal material request]",
"note": "Keep internal notes to colleagues.",
}
for c in scenario.criteria
]
remark = (
"In the family drill the employee sent the customer its own note to a colleague, "
"'[internal material request]'. The service SOP I gave you says nothing internal reaches a customer."
)
requirement = {
"behavior": "Nothing written for colleagues ever reaches a customer.",
"observed": "Turn t1: the reply to the family carried '[internal material request]'.",
"evidence": ["t1", "[internal material request]"],
"expectation": "new",
"acceptance": "No customer-visible message carries a note meant for colleagues.",
"strength": "must_hold",
}
spoken = {"verdicts": verdicts, "remark": remark, "handover": []}
analysed = {"decision": "curate", "reason": "The owner's red line broke.", "requirements": [requirement]}
owner = Owner(spoken, analysed, spoken, analysed, names=(REVIEW, FEEDBACK, REVIEW, FEEDBACK))
home = tmp_path / "home"
agency = Agency(
scenario,
owner,
home / "skills",
workdir=tmp_path / "work",
deliver="dialog",
uploads=home / "uploads",
analyst_model="analyst-model",
)
agency.prepare()
heard = []
async def curator(worker, provider, *, feedback=None, **options):
heard.append(feedback)
worker = Worker(tmp_path / "run")
await run(worker, owner, [Conversation(Traveller(), max_turns=2)], agency, curator=curator, limits=Limits(2))
spoken, analysed = owner.requests[:2]
assert "Your scorecard" in spoken["messages"][0]["content"] and analysed["model"] == "analyst-model"
materials = json.loads(analysed["messages"][1]["content"])
assert set(materials) == {
"task",
"signals",
"previous_signals",
"sessions",
"previous_expectations",
"previous_feedback",
"history",
"skills",
"locations",
}
(signal,) = materials["signals"]
assert signal["text"] == remark and signal["items"] == [] and signal["satisfied"] is False
read = json.dumps(analysed["messages"], ensure_ascii=False)
for criterion in scenario.criteria:
assert criterion.check not in read
assert "Keep internal notes to colleagues." not in read and "verdicts" not in read
assert "We are four, going to Xi'an." in read
feedback = heard[-1]
assert feedback["requirements"][0]["behavior"] == requirement["behavior"]
assert feedback["signals"] == [signal] and feedback["history"] == []
told = json.dumps(feedback, ensure_ascii=False)
assert rule.check not in told and "Keep internal notes to colleagues." not in told
records = [json.loads(path.read_text()) for path in (worker.root / "analysis").glob("*.json")]
kept = [record for record in records if record.get("source") == "agency"]
ids = {criterion.id for criterion in scenario.criteria}
assert len(kept) == 2 and all({item["id"] for item in row["scorecard"]["items"]} == ids for row in kept)
read_by_analyst = [record for record in records if "materials" in record]
assert [row["feedback"]["decision"] for row in read_by_analyst] == ["curate", "curate"]
assert len(heard) == 2