Raven-1/tests/test_stint_bootstrap.py
Ethan Wang 78e41b2f8d
feat(*): a playbook mode that takes many rounds (#499)
## Summary

A third playbook mode. A `mode: stint` playbook declares roles instead
of nodes, and the driver compiles one sub-agent graph per round, so a
stint is not a sibling of a `dag` run but a layer over it: one fan-out
is one graph, and many rounds are one graph each. Thirty rounds is
thirty graphs on the shared dispatch path, which is what makes the
validation, the dispatch budget, the approval gate and the detached-run
bookkeeping the ones already there rather than a second set.

The stint is a file. Nothing depends on the process that started it
still being alive: a gateway restart, a machine restart, or a `raven
playbook stints resume` all pick it up, and the resume is node-granular,
so a round that died with two roles done re-runs the third.

**One noun, singular and plural.** The mode is `mode: stint`; one run of
such a
playbook is a stint, with its own id, checkout and record; the plural is
the
command family and the RPC namespace. `round` keeps its own meaning as
the unit
inside -- one pass of every role -- and `stage` as one role's turn in
it, so
"the stint is on round 7 of 30" says three different things with three
different
words. The whole surface sits under `raven playbook`: `stints` answers
what has
run on this machine, `stint` acts on one project's layout and the
backlog its
roles share.

**Handover is files, not a conversation.** Every role opens a fresh
conversation each round, which is a decision rather than a limitation:
which instance serves which role changes, processes restart, transcripts
get compacted. What a role learned this round reaches the next one only
by being written down. That makes the journal the contract -- an
append-only file, windowed by recent rounds and by characters, because a
forty-round journal would otherwise fill the prompt it exists to inform.

**The boundary is three layers.** `owns` and `appends` are declared in
the playbook, rendered into the role's prompt from the same declaration
the enforcement reads, measured by the role's own checks, and undone
afterwards with the file the role wrote kept under `violations/`.
Declaring without telling is a trap; telling without undoing is a
suggestion; and the undo reads the stage's own commits as well as the
worktree, because a role that commits a stray write disappears from the
working tree and a boundary any role can step around by committing is
not one. Nothing refuses a write before it lands: a charter narrows a
role's tools only where the playbook declares a `playbook:` block, and
`owns` is not derived into one, so the shipped playbook's roles have no
tool gate and the undo is the boundary rather than its backstop.

Two shapes the boundary cannot hold are refused where they are written
rather than left to surface as violations in round four:

- **Two roles that can run at once where either is enforced.** They
share one checkout, so the first judged is graded against the difference
the whole tree shows and quarantines the other's work. Observed before
it was refused: two roles with disjoint `owns`, each writing only what
it owned, and the first judged came back as `dev-a wrote 1 path(s) it
may not write: src/b/work.py`, with that file gone. The message names
both ways out, because which one is right is the author's call. What
would make it work is a checkout per concurrent role, which is a merge
problem, not a validation rule.
- **`enforce.read: hard`.** It reaches the guard text, which then tells
the role that reads outside its paths are refused at the tool gate, and
no gate refuses one. The wording stays in `rounds_prompt` for whoever
wires the gate; until then the grade is not offered.

**Checks come before any model's opinion.** `verify[]` runs real
commands, and a failure goes back to the role that caused it with the
failure text, up to `maxHandbacks`. When that budget runs out the round
records the failure and carries on rather than failing the node: a
failed node skips every role downstream, and a failed build is exactly
what the reviewer downstream has to see.

**One approval, and it names what it is approving.** The whole stint is
confirmed once, at the start, and the text states the round budget and
every command the stint may run -- a playbook is a distribution unit, so
a file somebody handed you does not get to run shell thirty times on one
unexamined yes. Every round is charged against the hourly dispatch
budget like any graph; a round the budget turns down pauses the stint
rather than failing it, and `stints resume` takes it up once the
allowance recovers. From the CLI the same approval is put to the
terminal, and with no terminal to ask at the run is refused rather than
approved on nobody's behalf.

A stint works a checkout of its own, cut from the project's head, so
hours of its commits do not collide with the conversation that started
it. `stop.maxRounds` defaults to 10 and is capped at 99. `stop.until`
lets a role end a stint early by reporting a marker on a line of its own
-- a whole line, so a role quoting the marker mid-sentence does not end
the plan -- and only a terminal role is told the marker, because a role
that knows how to stop the stint has a reason to reach for it.

**Surfaces.** `raven playbook stints
list/get/extend/pause/stop/resume/sweep/answer`, where `resume` and
`extend` are relayed to the raven serving the page when one is up, so a
round runs in the process that owns the conversation;
`playbooks.stints.*` RPC methods, the read-only ones reading the record
off disk so opening the page does not cost what a dispatch costs; a
Stint tab on the playbook page showing the roles, what each owns, the
checks with their full commands, and what carries between rounds; and a
Runs tab listing every stint with its standing, how far into its budget
it is, each round's checks and what was undone, with pause, stop and
resume.

`raven playbook stint` lays a project out for one. The shipped playbook
is `rounds`: a planner, a builder and a verifier, on any project that
has a specification. `init` writes `.stint/`: a guard file per role, the
shared prose, and a link to the project's own specification, chosen by
two sieves -- the file name first, then how much of the document says
something is required, so an empty `SPEC.md` stub does not send thirty
rounds off a blank page. `task`, `ask` and `confirm` work the backlog
the roles share.

New package `raven/stint/` carries the parts a stint needs that are not
the playbook: the record, its checkout, the ownership grades, the
journal window and the check runner. `CONTEXT.md`, `CHANGELOG.md` and
the two `docs-site` repo-layout tables gain their rows; `README.md` is
not touched.

Three unrelated changes ride along. Two are the same shape -- a web
change that does not show:

- `index.html` went out with no caching header, so browsers fell back to
heuristic freshness and served a rebuilt page from cache without asking,
which reads exactly like a server that never picked the change up.
`no-cache` keeps the entry and revalidates; the ETag still saves the
transfer.
- `make ui` is one target for the two-step page build, because running
only vite leaves the served one-file page unchanged with nothing to say
so.

The third is `raven agent` (`raven/cli/agent_commands.py`):
`--message-file` reads the prompt from a file, `--permission-mode` picks
the tier for the run, and `--no-markdown` prints a soft-wrapped `Text`
rather than a markdown render; 92 lines of tests in
`tests/test_cli_agent_commands.py`.

Designed and deliberately not built, so a reviewer knows where the edges
are: a checkout per concurrent role with each role merging itself in
when it is judged (`docs/plans/2026-09-18-stint-acceptance.md`), and the
backlog-writing round zero. `raven playbook stint confirm` records a
person's word on the plan and nothing in the rounds path reads the flag
yet, which that document also says.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

```
uv run ruff check --no-cache raven evolver agents plugins-dist tests scripts
uv run ruff format --check --no-cache raven evolver agents plugins-dist tests scripts
uv run lint-imports
make lint-types
make lint-ui
make lint-tui
uv run pytest -q
uv run python scripts/check_commit_messages.py main..HEAD
uv run python scripts/check_source_language.py main..HEAD
uv run python scripts/check_large_files.py main..HEAD
```

Ruff clean over 2,004 files. Import contracts: 10 kept, 0 broken. `ty`
clean, which caught one real defect on the way in: `StintStore.list`
shadowed the builtin for every annotation after it in the class body, so
`-> list[StintRecord]` on it and on `live()` resolved to the method. The
UI gates include the generated-file drift checks (`gen:check`,
`lint:rpc`, `lint:i18n`) and both type-checks; eslint reports 23
warnings and 0 errors, none of them in a file this branch touches. `make
lint-bridge` was not run: this checkout has no `bridge/node_modules`,
and nothing here touches `bridge/`.

Suite: the whole `uv run pytest -q` runs in CI as four shards, and every
shard is green at this head, with the import-cycle budget (44 modules,
the same as `main`), the coverage gate and the three content gates over
the commit range. Locally the suites this branch touches are run per
change; the full run is CI's.

Behaviour, driven rather than asserted:

- A stray write by a concurrent role was reproduced end to end before
the validation refused the shape: `dev-b's own file still there after
dev-a was judged: False`, `violations recorded against dev-a: ['dev-a
wrote 1 path(s) it may not write: src/b/work.py']`.
- The check runner's timeout kills the process group, not just the
direct child. Proved as a regression guard: red with `process.kill()`,
green with the group teardown.
- The round budget refuses rather than wrapping: extending a real stint
past the ceiling answered `that would give it a budget of 104, and 99 is
the most any stint may have`, and the file was not written.
- The spec picker's two sieves, over a real project: `KEPT docs/PRD.md |
weight 12`, `dropped SPEC.md | len 12 | weight 0`, `dropped notes.md |
len 868 | weight 0`.
- Only a terminal role is told the stop marker: `planner=False
builder=False verifier=True` on the shipped example.
- The shipped example is discoverable with no user layer: `listed:
['rounds']`, `origin: builtin`.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

Additive. `mode: stint` is a new value; `dag` and `prompt` playbooks
load, validate, dispatch and report exactly as before, and the sections
the new mode reads are withheld from the generator's emit schema, so no
generated playbook can reach them. That withholding is pinned by a test
that fails when a section is added to the spec without being added to
the refusal.

Nothing outside the new mode changes behaviour: `spawn` and the DAG path
keep the per-session in-memory dispatch window they have on `main`.

A plan runs shell from `verify[]`. It is gated on the approval, the
approval text lists every command, and a playbook carrying `verify`
cannot be generated -- only written by hand or shipped. The builtin
layer ships exactly one playbook, `rounds`, pinned by name in a test,
for that reason: withheld from the generator and not shipped either, the
mode would be reachable only by transcribing YAML out of prose.

Rollback is a revert of the merge commit. A plan's on-disk record is
under the sub-agent history root and is read by nothing else; nothing
else's on-disk shape changes.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Ethan Wang <ethan.wang@evermind.ai>
Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>
2026-09-23 12:24:08 +08:00

281 lines
12 KiB
Python

"""Turning a project into one a run can work in: the scaffolding, and the plan's shape."""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from raven.stint import bootstrap
from raven.stint import roster as roster_mod
from raven.stint.bootstrap import InitError
PRD = """# Arena
## 7. Acceptance gates
| # | name | criterion |
|---|------|-----------|
| 7.1 | clean_boot | it boots with an empty stderr |
| 7.2 | feel_log | every event carries a timestamp |
| 7.3 | input | the shot leaves within 60 ms |
"""
def _project(tmp_path: Path, *, docs: dict[str, str] | None = None) -> Path:
project = tmp_path / "game"
(project / "docs").mkdir(parents=True)
for name, body in (docs or {"PRD-arena.md": PRD}).items():
(project / "docs" / name).write_text(body, encoding="utf-8")
(project / "README.md").write_text("# readme\n", encoding="utf-8")
return project
def test_init_lays_out_the_directory_and_links_the_specification(tmp_path: Path) -> None:
project = _project(tmp_path)
result = bootstrap.init(project)
home = project / ".stint"
assert (home / "SPEC.md").is_symlink()
assert (home / "SPEC.md").resolve() == (project / "docs" / "PRD-arena.md").resolve()
for name in ("HUMAN_DECISIONS.md", "AGENT_DECISIONS.md", "FIXLOG.md", "PLAYBOOK.md"):
assert (home / name).is_file(), name
for role in ("planner", "builder", "verifier"):
assert (home / f"{role}.md").is_file(), role
assert (project / "reports").is_dir()
assert result.spec.endswith("PRD-arena.md")
def test_a_link_rather_than_a_copy_so_a_revision_is_one_edit(tmp_path: Path) -> None:
project = _project(tmp_path)
bootstrap.init(project)
(project / "docs" / "PRD-arena.md").write_text(PRD + "\n| 7.4 | new | added later |\n", encoding="utf-8")
assert "7.4" in (project / ".stint" / "SPEC.md").read_text(encoding="utf-8")
def test_the_guard_files_it_writes_are_a_roster_the_runtime_can_read(tmp_path: Path) -> None:
"""The point of the exercise: init's output is the runtime's input, with nothing in between."""
project = _project(tmp_path)
bootstrap.init(project)
roster = roster_mod.load(project)
assert [role.name for role in roster.in_round()] == ["planner", "builder", "verifier"]
assert roster.get("builder").session == roster_mod.CONTINUE
assert roster.get("planner").session == roster_mod.CONTINUE, "the Planner holds the whole project across rounds"
assert roster.conflicts() == []
assert roster.grade("planner", "reports/brief_01.md", round_index=1) == roster_mod.OWNS
assert roster.grade("verifier", "reports/brief_01.md", round_index=1) == roster_mod.NEVER
def test_no_slot_is_left_unfilled_in_a_generated_guard(tmp_path: Path) -> None:
project = _project(tmp_path)
bootstrap.init(project)
for role in ("planner", "builder", "verifier"):
assert "{{" not in (project / ".stint" / f"{role}.md").read_text(encoding="utf-8"), role
def test_the_enforcement_grade_is_the_operator_s_to_choose(tmp_path: Path) -> None:
project = _project(tmp_path)
bootstrap.init(project, enforce_read="hard", enforce_write="soft")
role = roster_mod.load(project).get("verifier")
assert role.enforce_read == roster_mod.HARD
assert role.enforce_write == roster_mod.SOFT
def test_running_init_twice_keeps_what_a_person_edited(tmp_path: Path) -> None:
project = _project(tmp_path)
bootstrap.init(project)
edited = project / ".stint" / "planner.md"
mine = edited.read_text(encoding="utf-8") + "\n## My own rule\n\nAlways ask me first.\n"
edited.write_text(mine, encoding="utf-8")
again = bootstrap.init(project)
assert edited.read_text(encoding="utf-8") == mine
assert ".stint/planner.md" in again.kept
assert bootstrap.init(project, force=True) and "My own rule" not in edited.read_text(encoding="utf-8")
def test_several_candidate_specifications_are_reported_rather_than_guessed(tmp_path: Path) -> None:
project = _project(tmp_path, docs={"PRD-a.md": PRD, "spec-b.md": PRD, "design-c.md": PRD})
result = bootstrap.init(project)
assert any("more than one document" in note for note in result.notes)
def test_a_project_with_no_specification_says_so(tmp_path: Path) -> None:
project = tmp_path / "bare"
project.mkdir()
result = bootstrap.init(project)
assert any("no specification document" in note for note in result.notes)
assert not (project / ".stint" / "SPEC.md").exists()
def test_a_named_specification_that_is_not_there_is_refused(tmp_path: Path) -> None:
with pytest.raises(InitError):
bootstrap.init(_project(tmp_path), spec=tmp_path / "nope.md")
def test_the_facts_section_is_drafted_for_a_person_to_check(tmp_path: Path) -> None:
project = _project(tmp_path)
bootstrap.init(project)
text = (project / ".stint" / "HUMAN_DECISIONS.md").read_text(encoding="utf-8")
assert "## Facts about this project" in text
assert "## Awaiting a person" in text
assert "- Machine:" in text and "- Display:" in text
# Not the game engine: the layout is what any multi-round project gets,
# and a Godot line in every project's facts was H* showing through.
assert "Godot" not in text
def test_a_freshly_initialised_project_is_one_step_from_ready(tmp_path: Path) -> None:
"""Everything but the plan, which is the step that needs a model and a person."""
project = _project(tmp_path)
bootstrap.init(project)
reasons = roster_mod.preflight(project)
assert len(reasons) == 1
assert "raven playbook stint task add" in reasons[0]
# ------------------------------------------------------------------------- plan
def test_the_plan_prompt_names_the_three_files_it_works_between(tmp_path: Path) -> None:
project = _project(tmp_path)
bootstrap.init(project)
prompt = bootstrap.plan_prompt(project)
assert str(project / ".stint" / "SPEC.md") in prompt
assert str(project / ".stint" / "HUMAN_DECISIONS.md") in prompt
assert str(project / ".stint" / "backlog.json") in prompt
assert bootstrap.PLAN_SENTINEL in prompt
assert "depends_on" in prompt and "never a round number" in prompt
def test_gates_are_read_off_the_specification_only_to_check_coverage(tmp_path: Path) -> None:
assert bootstrap.gates_in(PRD) == ("7.1", "7.2", "7.3")
def test_review_reports_the_gates_no_task_covers(tmp_path: Path) -> None:
"""A gate nobody planned for is a requirement nobody planned for."""
project = _project(tmp_path)
bootstrap.init(project)
(project / ".stint" / "backlog.json").write_text(
json.dumps(
{
"meta": {},
"tasks": [
{"id": 1, "title": "boot", "gates": ["7.1"], "state": "open"},
{"id": 2, "title": "log", "gates": ["7.2"], "state": "open"},
],
}
),
encoding="utf-8",
)
review = bootstrap.review(project)
assert review.tasks == 2
assert review.gates == ("7.1", "7.2")
assert review.uncovered == ("7.3",)
assert "not covered: 7.3" in review.render()
def test_the_documents_a_project_carried_are_found_at_any_depth_under_docs(tmp_path: Path) -> None:
"""A half-built project keeps what it learned in the role folders it grew up with."""
project = _project(tmp_path, docs={"PRD-arena.md": PRD, "DECISIONS.md": "- settled: one boss\n"})
(project / "docs" / "verifier").mkdir()
(project / "docs" / "verifier" / "Verifier.md").write_text("# Verifier\n", encoding="utf-8")
(project / "CLAUDE.md").write_text("# instructions\n", encoding="utf-8")
(project / "AGENTS.md").symlink_to("CLAUDE.md")
(project / "reports").mkdir()
(project / "reports" / "verify_01.md").write_text("# not a document\n", encoding="utf-8")
bootstrap.init(project)
found = bootstrap.project_documents(project, spec=project / ".stint" / "SPEC.md")
assert found == ["CLAUDE.md", "docs/DECISIONS.md", "docs/verifier/Verifier.md"], (
"the spec, README, .stint/ and reports/ are not documents to absorb"
)
def test_the_plan_prompt_asks_for_the_carry_over_and_the_ledger(tmp_path: Path) -> None:
project = _project(tmp_path, docs={"PRD-arena.md": PRD, "DECISIONS.md": "- one boss\n"})
bootstrap.init(project)
prompt = bootstrap.plan_prompt(project)
assert "- `docs/DECISIONS.md`" in prompt
assert str(project / ".stint" / "SOURCES.md") in prompt
assert str(project / ".stint" / "PLAYBOOK.md") in prompt and "## Settled" in prompt
assert "do not edit or delete them" in prompt
assert "the reading map" in prompt, "reference material is mapped, not copied"
assert "is **not**\n copied" in prompt
assert "a reference image" in prompt and "open the image itself" in prompt, (
"a picture gets a row that sends the Builder and Verifier to look at it"
)
assert "a reference image" in prompt and "open the image itself" in prompt, (
"a picture gets a row that sends the Builder and Verifier to look at it"
)
def test_a_greenfield_project_is_told_there_is_nothing_to_carry(tmp_path: Path) -> None:
project = _project(tmp_path)
(project / "README.md").unlink()
bootstrap.init(project)
assert bootstrap.NO_DOCUMENTS in bootstrap.plan_prompt(project)
def test_review_names_the_documents_the_ledger_left_out(tmp_path: Path) -> None:
project = _project(tmp_path, docs={"PRD-arena.md": PRD, "DECISIONS.md": "- one boss\n", "LESSONS.md": "- x\n"})
bootstrap.init(project)
(project / ".stint" / "backlog.json").write_text(
json.dumps(
{"meta": {}, "tasks": [{"id": 1, "title": "boot", "gates": ["7.1", "7.2", "7.3"], "state": "open"}]}
),
encoding="utf-8",
)
assert bootstrap.review(project).unabsorbed == ("docs/DECISIONS.md", "docs/LESSONS.md")
(project / ".stint" / "SOURCES.md").write_text(
"| document | went to |\n| docs/DECISIONS.md | HUMAN_DECISIONS.md Settled |\n", encoding="utf-8"
)
plan = bootstrap.review(project)
assert plan.unabsorbed == ("docs/LESSONS.md",)
assert "not carried into .stint/" in plan.render() and "LESSONS.md" in plan.render()
def test_the_plan_is_told_what_the_tree_already_holds(tmp_path: Path) -> None:
"""The documents say what was meant; the tree says what is."""
project = _project(tmp_path)
(project / "project").mkdir()
(project / "project" / "project.godot").write_text("[application]\n", encoding="utf-8")
(project / "reports").mkdir()
(project / "reports" / "verify_02.md").write_text("# verifier 02\n", encoding="utf-8")
bootstrap.init(project)
prompt = bootstrap.plan_prompt(project)
assert "## Then, look at the tree" in prompt
assert "`project/`" in prompt and "`verify_02.md`" in prompt
assert "checks detected: `import`" in prompt
assert "look rather than assume" in prompt, "the layout is the repository's own, so the prompt guides a search"
assert "Work that already is\n goes in as `in_review`" in prompt
assert "Nothing is done yet" not in prompt
def test_a_greenfield_tree_is_named_as_such(tmp_path: Path) -> None:
project = _project(tmp_path)
bootstrap.init(project)
prompt = bootstrap.plan_prompt(project)
assert "none found -- this may be a greenfield project" in prompt and "Markdown under `reports/`: none" in prompt
def test_the_planner_is_asked_for_a_coherent_piece_not_a_count(tmp_path: Path) -> None:
project = _project(tmp_path)
bootstrap.init(project)
guard = (project / ".stint" / "planner.md").read_text(encoding="utf-8")
assert "one coherent piece of work" in guard and "the number is not the point" in guard
assert "At most three priorities" not in guard