tzteyang ad2f054dd4
feat(*): add an experimental Curator that cultivates harness-of-harnesses workers (#802)
## Summary

Adds `experimental/`, a first version of worker RSI: a Curator that
shapes a
Raven worker's harness from the task, the materials handed to it and
feedback,
the way a new employee is trained.

The worker is a harness-of-harnesses. A root Raven harness faces the
user and
calls child harnesses, each a full Raven agent hosted over ACP, through
the
nodes of its playbooks. Every harness has the same four strategy facets
(memory, planning, capability, action). The Curator reads the actual
harnesses
and generates code and config on those facets at both levels: the root
candidate sets the playbooks and writes per-node requirements, each
child is
curated against the requirements of its nodes, and the set is checked on
copies and activated together.

Layout:
- `curator/`: staged generation (understand, select, design, implement,
repair), two-level composition, harness contracts, and `raven_adapter/`,
  the only layer that touches a worker's Raven runtime
- `analyst/`: turns an evaluator's words and execution records into
  observable behavior requirements
- `iteration/`: the generic loop (onboarding curation, then trial,
review and
  curation each round) and a CLI for a human evaluator
- `simulation/`: a simulated travel agency that runs the loop, plus one
  showcase case (`simulation/cases/s0925c/`: case notes, the exported
  transcript split by phase, and the round 1 and round 3 decks)
- `docs/design.md` (structure, curation flow, composition sequence, the
Raven
  connection) and `docs/rsi-iteration.md` (the loop and the Analyst)

Key decisions:
- No change under `raven/`, and `raven/` does not import
`experimental/`.
Generated artifacts land on Raven's existing extension points. The few
Raven
internals the adapter reads are listed in `docs/design.md` section 5.4.
- The Curator explores a read-only source snapshot; the caller names the
evaluation side to withhold from it, so the generic layers name no
scenario.
- The single-layer owner analysis is kept only to reproduce the showcase
case
  and is marked for removal; the two-layer Analyst is the default.
- The two decks are `.pptx` files of about 230 KB each. They pass
  `check-large-files`; flagging them in case they count as report assets
under AGENTS.md section 7, in which case they can move out of git behind
a
  link.
- One follow-up for Raven 0.2.2: playbooks in prompt and rounds modes
ship no
  fixed nodes, so node requirements are read from dag playbooks only.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

On the branch rebased onto the latest `main`:

- `uv run pytest tests/test_harness_curator_*.py tests/test_analyst_*.py
tests/test_iteration_*.py tests/test_simulation_*.py -q`: 368 passed
- `uv run pytest -m integration
tests/integration/test_harness_curator_*.py --deselect
tests/integration/test_harness_curator_real_llm.py -q`
  (offline, stub providers): 57 passed
- `uv run pytest -q` (whole suite, with the host proxy variables unset
and
`PYTHONTZPATH` set): 26692 passed, 2 failed. One was an order-dependent
test
  of this PR and is fixed here; the other,

`tests/test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing`,
  fails only because the machine runs as root and is unrelated.
- `uv run ruff check` and `uv run ruff format --check` on the touched
files:
  clean
- `scripts/check_large_files.py origin/main..HEAD`,
  `scripts/check_source_language.py origin/main..HEAD`,
  `scripts/check_commit_messages.py origin/main..HEAD` and
  `npx commitlint --from origin/main --to HEAD`: all exit 0

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

No user-visible change for Raven: `experimental/` is not imported by
`raven/` and is not packaged in the wheel (`packages = ["raven"]`).
`CONTEXT.md` gains one glossary entry. The Curator's exploration offers
a
shell only under an OS sandbox executor. Rollback is a revert of the
squash
commit; nothing else depends on `experimental/`.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Yuqi Yang <yiqiyang524@gmail.com>
Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-09-26 21:00:58 +08:00

333 lines
14 KiB
Python

"""What the agency's own materials imply for one drill, computed rather than judged, for the owner to judge with.
For the card that was played: what the price list and booking policy give for each product (season and party band,
per-person and child prices, extra nights, party total, insurance) and the budget per person per night the selection
matrix works on. For a delivered deck: facts read from the file's structure (its name against the deck spec's naming
rule, slide shape, fonts against the template's, first and last layouts against the template's cover and contact
pages, motion, template placeholders left in its text, quote numbers in it). Nothing here reads the employee's
messages or pages for meaning: the owner, a model, reads the conversation and the files and decides every verdict,
using these figures the way a person uses a calculator and a ruler. The materials and words they come from are the
scenario's `reference.md`, a table whose second column gives each key's values in backticks.
"""
import re
import string
import zipfile
from dataclasses import dataclass
from pathlib import Path
from pptx import Presentation
from .cards import Trip
from .channel import said
from .files import _MOTION, deck_slides
CONFIG = "reference.md"
NUMBER = r"(?:\d{1,3}(?:,\d{3})+|\d+)(?:\.\d+)?"
_FACE = re.compile(r'typeface="([^"]*)"')
_SLIDE = re.compile(r"ppt/slides/slide\d+\.xml$")
@dataclass(frozen=True)
class Product:
name: str
base: int
bands: tuple[tuple[int, int], ...]
prices: dict[str, tuple[int, ...]]
extra: int
child: bool
@dataclass(frozen=True)
class Expected:
product: str
season: str
band: tuple[int, int]
unit: int
children: int
child_unit: int
extra_nights: int
extra: int
total: int
def _number(text: str) -> float:
return float(text.replace(",", ""))
def _ints(text: str) -> list[int]:
return [int(_number(found)) for found in re.findall(NUMBER, text)]
def _tables(text: str) -> list[list[list[str]]]:
tables, rows = [], []
for line in [*text.splitlines(), ""]:
if line.strip().startswith("|"):
cells = [cell.strip() for cell in line.strip().strip("|").split("|")]
if not all(re.fullmatch(r":?-{3,}:?", cell) for cell in cells):
rows.append(cells)
elif rows:
tables.append(rows)
rows = []
return tables
def _sections(text: str) -> list[tuple[str, str]]:
return [(part.partition("\n")[0], part.partition("\n")[2]) for part in re.split(r"^## ", text, flags=re.M)[1:]]
def _body(text: str) -> str:
return text.split("---", 2)[2] if text.startswith("---") else text
def words(path: Path) -> dict[str, tuple[str, ...]]:
"""The scenario's reference words: each table row's key and the backticked values in its second cell."""
found = {}
for table in _tables(Path(path).read_text()):
for row in table:
values = re.findall(r"`([^`]+)`", row[1]) if len(row) > 1 else []
if values and re.fullmatch(r"[a-z_]+", row[0]):
found[row[0]] = tuple(values)
return found
def _pattern(template: str) -> re.Pattern:
parts = []
for literal, field, spec, _ in string.Formatter().parse(template):
parts.append(re.escape(literal))
if field:
width = re.search(r"\d+", spec or "")
parts.append(rf"(?P<{field}>\d{{{int(width.group())}}})" if width else rf"(?P<{field}>\d+)")
return re.compile("".join(parts))
@dataclass(frozen=True)
class Rules:
"""Everything read from a scenario's materials, loaded once; `load` returns None without a `reference.md`."""
words: dict[str, tuple[str, ...]]
products: dict[str, Product]
seasons: dict[str, tuple[tuple[tuple[int, int], tuple[int, int]], ...]]
insurance: int
child_age: int
child_ratio: int
deck_name: str
fonts: frozenset[str]
cover: str
contact: str
quote: re.Pattern
@classmethod
def load(cls, scenario) -> "Rules | None":
config = scenario.root / CONFIG
if not config.is_file():
return None
found = words(config)
prices = _body(scenario.text(found["price_list"][0]))
products, seasons, insurance = {}, {}, None
for heading, body in _sections(prices):
tables = _tables(body)
if not tables:
continue
header, *rows = tables[0]
if len(header) >= 3 and rows and all(_ints(row[0]) for row in rows):
if not all(re.fullmatch(NUMBER, cell) for row in rows for cell in row[1:]):
continue
name = re.match(r"\w+", heading).group()
notes = [line for line in body.splitlines() if not line.strip().startswith("|") and _ints(line)]
products[name] = Product(
name=name,
base=_ints(heading)[0],
bands=tuple((_ints(row[0])[0], _ints(row[0])[-1]) for row in rows),
prices={
season: tuple(int(_number(row[i])) for row in rows) for i, season in enumerate(header[1:], 1)
},
extra=_ints(notes[0])[0],
child=name in found["child_products"],
)
names = {season for product in products.values() for season in product.prices}
for table in _tables(prices):
for row in table:
if row[0] in names and len(row) == 2:
spans = []
for chunk in re.split(r"[^\w\s]+", row[1]):
ints = _ints(chunk)
if len(ints) in (3, 4):
spans.append(((ints[0], ints[1]), (ints[2], ints[3] if len(ints) == 4 else 31)))
seasons[row[0]] = tuple(spans)
elif any(word in row[0] for word in found["insurance"]) and len(row) > 1 and _ints(row[1]):
insurance = _ints(row[1])[0]
child = next(
line
for line in _body(scenario.text(found["booking_policy"][0])).splitlines()
if "%" in line and any(word in line for word in found["child"])
)
spec = _body(scenario.text(found["deck_spec"][0]))
material, _, file = found["template"][0].partition("/")
template = scenario.materials[material] / file
deck = Presentation(str(template))
with zipfile.ZipFile(template) as package:
fonts = {face for name in package.namelist() if _SLIDE.match(name) for face in _faces(package.read(name))}
fonts |= set(_theme(package).values())
if not products or set(seasons) != names or insurance is None:
raise ValueError(f"{CONFIG}: the price list gave no products, seasons or insurance line")
return cls(
words=found,
products=products,
seasons=seasons,
insurance=insurance,
child_age=_ints(child)[0],
child_ratio=int(re.search(r"(\d+)\s*%", child).group(1)),
deck_name=re.search(r'"([^"\s]+?)-[^"\s]*\.pptx"', spec).group(1),
fonts=frozenset(face for face in fonts if face and not face.startswith("+")),
cover=deck.slides[0].slide_layout.name,
contact=deck.slides[-1].slide_layout.name,
quote=_pattern(found["quote"][0]),
)
def season(self, day: tuple[int, int]) -> str | None:
for name, spans in self.seasons.items():
for first, last in spans:
if (first <= day <= last) if first <= last else (day >= first or day <= last):
return name
return None
def expected(self, product: str, trip: Trip) -> Expected | None:
"""The quote the price list gives this card for `product`; None off the list, out of season or over the cap."""
found, season = self.products.get(product), self.season(trip.start)
if found is None or season is None:
return None
band = next((i for i, (low, high) in enumerate(found.bands) if low <= trip.persons <= high), None)
if band is None:
return None
unit = found.prices[season][band]
children = sum(age < self.child_age for age in trip.children) if found.child else 0
child_unit = unit * self.child_ratio // 100
extra_nights = max(0, trip.nights - found.base)
extra = extra_nights * found.extra * trip.persons
total = unit * (trip.persons - children) + child_unit * children + extra
return Expected(product, season, found.bands[band], unit, children, child_unit, extra_nights, extra, total)
def number(self, trip: Trip) -> str:
return self.words["quote"][0].format(month=trip.start[0], day=trip.start[1], persons=trip.persons)
def _faces(xml: bytes) -> set[str]:
return set(_FACE.findall(xml.decode("utf-8", errors="replace")))
def _theme(package: zipfile.ZipFile) -> dict[str, str]:
"""The theme's heading and body fonts, latin and East Asian."""
names = sorted(name for name in package.namelist() if re.match(r"ppt/theme/theme\d+\.xml$", name))
xml = package.read(names[0]).decode("utf-8", errors="replace") if names else ""
fonts = {}
for role in ("majorFont", "minorFont"):
block = re.search(rf"<a:{role}>(.*?)</a:{role}>", xml, re.S)
for script, face in re.findall(r'<a:(latin|ea) typeface="([^"]*)"', block.group(1) if block else ""):
fonts[f"{role}.{script}"] = face
return fonts
def quoted(exchanges, pattern: re.Pattern) -> list[str]:
"""Quote numbers the assistant used in a conversation, first use first."""
return list(dict.fromkeys(found.group(0) for turn in exchanges for found in pattern.finditer(said(turn))))
def prices(rules: Rules, trip: Trip) -> dict:
"""What the price list and booking policy give this card for each product."""
by_product = {}
for name, product in rules.products.items():
expected = rules.expected(name, trip)
if expected is None:
by_product[name] = "no price: the party is over this product's cap or the date is in no season"
continue
row = {
"base_nights": product.base,
"party_band": f"{expected.band[0]}-{expected.band[1]}",
"per_person": expected.unit,
"extra_nights": expected.extra_nights,
"extra_night_price_per_person": product.extra,
"extra_nights_total": expected.extra,
"party_total": expected.total,
}
if product.child:
row["children_at_child_price"] = expected.children
row["child_price"] = expected.child_unit
if trip.nights < product.base:
row["nights_short_of_base"] = product.base - trip.nights
by_product[name] = row
return {
"quote_number": rules.number(trip),
"persons": trip.persons,
"nights": trip.nights,
"departure_season": rules.season(trip.start),
"budget_per_person_per_night": round(trip.budget / trip.persons / trip.nights),
"insurance_per_person": rules.insurance,
"insurance_total": rules.insurance * trip.persons,
"by_product": by_product,
}
def deck(rules: Rules, exchanges) -> dict | None:
"""Facts read from the structure of the last deck a conversation handed over; None when it handed none over."""
delivered = [
Path(path)
for turn in exchanges
for path in turn.execution.deliverables
if Path(path).suffix.lower() == ".pptx" and Path(path).is_file()
]
if not delivered:
return None
path = delivered[-1]
facts = {
"file": path.name,
"names_per_spec": [f"{rules.deck_name}-{number}.pptx" for number in quoted(exchanges, rules.quote)],
}
try:
opened = Presentation(str(path))
with zipfile.ZipFile(path) as package:
slides = [package.read(name) for name in package.namelist() if _SLIDE.match(name)]
theme = _theme(package)
except Exception as exc: # noqa: BLE001 -- a deck that does not open is itself the fact
return {**facts, "opens": False, "error": repr(exc)}
text = _deck_text(path)
used = {face for xml in slides for face in _faces(xml) if face and not face.startswith("+")} | {
face for face in theme.values() if face
}
return {
**facts,
"opens": True,
"slides": len(opened.slides),
"aspect": round(opened.slide_width / opened.slide_height, 3),
"fonts_outside_template": sorted(used - rules.fonts),
"template_fonts": sorted(rules.fonts),
"first_slide_layout": opened.slides[0].slide_layout.name if len(opened.slides) else None,
"last_slide_layout": opened.slides[-1].slide_layout.name if len(opened.slides) else None,
"template_cover_layout": rules.cover,
"template_contact_layout": rules.contact,
"slides_with_motion": sum(bool(_MOTION.search(xml.decode("utf-8", errors="replace"))) for xml in slides),
"template_placeholders_left": [
marker for marker in rules.words["placeholders"] if re.search(rf"(?<![\d,.]){re.escape(marker)}", text)
],
"quote_numbers_in_deck": sorted({found.group(0) for found in rules.quote.finditer(text)}),
}
def _deck_text(path: Path) -> str:
try:
return deck_slides(path)
except Exception: # noqa: BLE001 -- unreadable text leaves no placeholders to report
return ""
def references(rules: Rules, exchanges, trip: Trip | None) -> dict:
"""Everything computed for one drill: the card's prices when the card holds a trip, the deck's facts when one was
handed over."""
found = {}
if trip is not None:
found["prices"] = prices(rules, trip)
facts = deck(rules, exchanges)
if facts is not None:
found["deck"] = facts
return found