Raven-1/tests/test_playbook_executor.py
Dizhan Xue 1b9cbf85e5
feat(*): let a dag decision node choose which of its branches run (#1001)
## Summary

A sub-agent DAG node can now decide which of its downstream branches
run. Before this, a background run could not turn on a result: every
node ran, and choosing a path meant the host winding the run down and
starting another.

How it works:

- A node that declares `routes` (route name to condition) is a decision
node; `select` is `"one"` (the default) or `"many"`. Raven's per-node
judge, the existing verdict call, picks the route or routes the node's
answer satisfies. The node's own text never selects a route. A route
that cannot be determined fails the node, with no fallback route.
- A downstream node carries `when`, a guard over `<node>.<route>` and
`<node>.done` atoms joined with `not`, `and`, `or` and parentheses,
evaluated with three-valued logic: a route of a decision node that was
never executed is unknown. Every decision node in a node's `depends_on`
must appear in its `when`; such graphs are refused at submission
otherwise, along with guards that can never hold and a route named
`done` in any letter case. Validation keeps negation parity and the
surrounding boolean expression: a double-negated `done` and a
disjunction with a satisfiable arm are accepted, without expanding
disjunctions into combinations.
- A node that does not run is `not_executed`, with a reason:
`when_false`, `when_unknown` or `upstream_not_executed`. A node whose
dependencies all went unexecuted is not executed either; a failed or
skipped dependency still skips its dependents first.
- A merge node reads a missing upstream as a fixed sentence, `<This task
is not executed and the output is missing, please go on without this
output.>`, written outside every untrusted-content fence, plus a note
listing the inputs that may be missing. A file reference to the files of
a node that may not execute is refused at submission however its path is
spelled: `@nodes/`, an absolute path into the node root, or a workdir
path that lands there, symlinks followed. Windows case, separator and
drive-relative aliases are recognized, and the advice keeps the declared
node ID.
- Node judgement is always on: the `subagentDag.verdictEnabled` switch
is retired. Config migration 14 removes it with a notice, and drops it
again on every later load should an older build sharing the config write
it back. `raven playbook run` now judges its nodes too, on the same
`subagentDag` settings and a pinned `verdictModel`'s own provider; with
nobody there to answer, a node judged not to have accomplished its task
fails on the judge's reason. A host with no judge at all refuses graphs
that declare routes.

Where it shows:

- Run events, the manifest, `dag.get`, `tasks.list` and
`dag.run_completed` carry `reason`, `selected_routes` and
`route_evidence` per node, and `routes`, `select` and `when` from the
graph. The instance registry's DAG-node rows (and `InstanceRow`) carry
the outcome too, so a client that reloads before the run finalizes shows
what a branch decided. `TaskCounts.not_executed` is a new counter.
- TUI: a not-executed node shows a hollow marker and its reason in
words; a decision node shows its selection on the node's second line,
and the guard and route evidence in the detail and in `/dag <node>`. The
Agents Overlay and the live agents strip leave not-executed nodes out,
as the backend's agent lists do.
- Web: the desk's tasks tab names not-executed steps and shows the same
fields.
- Playbooks pass the three fields through, namespace guard atoms with
the run prefix, and keep them through compilation.
- The `run_subagent_dag` schema, the DAG orchestration skill and the
docs site (English and Chinese) describe routes, guards and the merge
rule; CHANGELOG has the entries.

A graph with no `routes` and no `when` behaves exactly as before: the
same judge requests, events, records, summary and finish text.
`route_evidence` is a quote the judge picks from the sub-agent's output;
it is cut at 500 characters, keeps a lone surrogate from a malformed
escape as the escape's own text, and never enters model-facing text
(finish text, `dag_status`, exception reports). When a judge answer does
not report a usable route, the judge is asked again at most twice, each
time with one correction. The shared instance registry drops its cache
when a write fails on the data rather than the disk, so a row the file
cannot hold fails its own write and no other.

Two Windows CI steps run the alias and finish-text regressions; the
latter exposes the installed Git `cat` executable required by the
existing fixture.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Linux, Python 3.12.13, on this revision rebased onto the current main
(`$CHECKS` is a temporary evidence directory):

```sh
uv run --frozen --no-sync pytest -q -n 4
# 23 failed, 33229 passed, 162 skipped, 1 xfailed, 3 errors
uv run --frozen --no-sync pytest $(git diff --name-only origin/main...HEAD -- tests/) -q -n 4 --idle-ceiling-strict --cov=raven --cov-branch --cov-report=json:$CHECKS/coverage.json
# the 20 test files this branch changes: 2622 passed, 1 skipped (a POSIX permission case that does not apply to root)
uv run --frozen --no-sync python scripts/coverage_gate.py --coverage-json "$CHECKS/coverage.json" diff --base-ref origin/main --head HEAD --threshold 90
# Diff coverage: 99.48% (764/768 executable changed lines), passed the 90.00% threshold
```

The full run's 23 failures and 3 errors are 25 tests. 24 of them fail
the same way on the current main in this environment: the venv lacks
`nltk` and `datrie` (16), a proxy is set in the environment (5), this
box has its own `npm` in `/usr/bin` (1 install-script case), LibreOffice
writes no `.xls` here (1), and the run is root (1). The 25th,
`test_a_cli_run_left_on_the_machine_when_this_computer_stops_is_picked_up_by_the_next_start`,
failed with its teardown in the full run and in one of two runs of its
file, and passed in five runs on its own: it reads the remote-run record
the moment its pid file appears, a race in code this branch does not
change (#840 makes that test wait for both). The four uncovered changed
lines are the gateway's `_verdict_provider` body, whose helper is tested
on its own; a guard atom over a node that did not complete, which the
runner never evaluates because it reads a guard only once every
dependency completed or was not executed; and the cut of an over-long
terminal output.

Ruff check and format, `ty check` over the changed modules, the import
contracts, `deptry raven`, commitlint, and
`scripts/check_commit_messages.py`, `check_large_files.py` and
`check_source_language.py` over this branch's commits pass. ui-web:
`vitest run src/features/tasks/BoardCard.test.tsx scripts/gates` passes
50 files and 244 tests, and `tsc --noEmit` is clean. In Chromium a
not-executed title whose first word is wider than the room beside its
status now breaks that word instead of starting under a blank line, and
every other tasks-board card renders pixel-identical, English and
Chinese, light and dark. Firefox still starts such a title under a blank
line and cuts it without an ellipsis. CI's `TUI checks` covers ui-tui.

Mutation checks: removing any one of these makes its own test fail: the
every-load strip of the retired switch, the quote's escape before the
cut, the registry's cache drop, the node-root check in the tool, the
runner and the replan path, the physical path through a symlink, the
CLI's judge, its settings and its no-answer failure, and the pinned
judge provider. A model reading only the DAG skill's catch-all paragraph
answers that without a catch-all route the judge may pick the nearest
route or fail the node; an earlier wording led it to answer that the
graph always stops.

A live run on `openrouter/z-ai/glm-5.3-flash`, two graphs with synthetic
messages, selected the expected route at each decision node, left the
unselected branch `not_executed` with `when_false`, and put the fixed
sentence outside the fences. Earlier revisions of this change passed the
native Windows selections (aliases, finish text, guards; Windows build
26300, Python 3.13.12). The final revision has not been rerun on native
Windows or macOS.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

Security: sub-agent text stays fenced or out of model-facing text on
every new path. The routes block in the judge request sits after the
last untrusted fence and is labelled as the graph author's; the
missing-output sentence is written outside every fence; `route_evidence`
is bounded and reaches records, events, the registry and the UIs only.
In the TUI, route evidence is drawn through the same path as a node's
error text; neither strips terminal control sequences, which this PR
leaves as it is.

Behaviour changes: a config that set `subagentDag.verdictEnabled: false`
now gets judged nodes; migration 14 removes the key and says so once.
`raven playbook run` judges every node, one more model call per node,
and a node judged not to have accomplished its task fails there instead
of completing. New wire fields are optional except
`TaskCounts.not_executed`, which is always present (0 when nothing
branched). `InstanceRow` gains three optional fields, and a decision
node's registry row now also stores its selection and up to 500
characters of route evidence.

Conditional-artifact validation matches node IDs using the session
namespace's case-insensitive identity on every host, including macOS.
This conservatively rejects case-only spellings even on case-sensitive
filesystems. Separator and drive normalization remain native, and
unconditional file resolution keeps its existing behaviour.

Rollback: revert the squash commit. A config already stamped 14 loads
unchanged on the previous build, since the stamp only gates migrations;
a user who had set `verdictEnabled: false` can set it again there, and a
later build drops it again on load.

## Related Issues

Related, found while building this and not changed here: #998, #999,
#1000. Follow-ups from its review, not changed here: #1008.
2026-10-19 15:17:04 +08:00

1542 lines
60 KiB
Python

"""Execution side: what loading a playbook produces, and what it refuses.
"Funnel" is gone from this file along with the thing it named. The runtime is a
library plus one loader now, so what is covered is the loader's contract: params
and blanks reported together and before anything runs, ``fills`` able to complete
a playbook but never to edit one, the gap loop bounded, and the two modes going to
the party that can act on them.
"""
import json
from pathlib import Path
import pytest
from raven.agent.subagent.dag_graph import parse_dag_spec, validate_and_order
from raven.config.schema import MCPServerConfig
from raven.contracts.llm_provider import ErrorClassification, LLMResponse
from raven.playbook import (
MAX_GAP_ROUNDS,
NodeSpec,
ParamSpec,
PlaybookExecutor,
PlaybookRuntime,
PlaybookSpec,
PlaybookStore,
RouterSizes,
Triggers,
)
from raven.playbook.agent_profiles import PlaybookAgentProfile
from raven.playbook.executor import _namespace_run
from raven.playbook.prompt import SYSTEM_PROMPT, build_compose_prompt, build_revise_prompt
COMPOSE_PROFILES = {
"research-raven": PlaybookAgentProfile(
description="research",
stateful=True,
reads_local_files=True,
injectable_skills=True,
injectable_mcps=False,
),
"content-raven": PlaybookAgentProfile(
description="writing",
stateful=True,
reads_local_files=True,
injectable_skills=True,
injectable_mcps=False,
),
}
class FakeDagTool:
def __init__(self):
self.calls = []
self.context = None
def set_context(self, channel, chat_id, session_key=None):
self.context = (channel, chat_id, session_key)
async def execute(
self, nodes, task_summary="", background=True, confirm=False, mcp_servers=None, mcp_scope=None, **_
):
# The hand-off is a callable the graph tool re-reads per dispatch; the
# fake keeps it so a test can read it again after a credential lands.
render = mcp_servers if callable(mcp_servers) else (lambda: mcp_servers or {})
self.calls.append(
{
"nodes": nodes,
"task_summary": task_summary,
"background": background,
"confirm": confirm,
"mcp_servers": dict(render()),
"render": render,
"mcp_scope": mcp_scope,
}
)
return "DAG run wf-1 started in the background (%d nodes)." % len(nodes)
def _dag_spec(**over):
base = dict(
name="weekly-feedback",
description="weekly user-feedback analysis",
task_summary="pull this week's feedback and write the report",
mode="dag",
triggers=Triggers(keywords=["user feedback"]),
params={
"week_of": ParamSpec(type="string", required=True, description="which week should be analyzed?"),
"audience": ParamSpec(default="PM", description="who reads the report?"),
},
nodes=[
NodeSpec(
id="pull",
subagent="data-raven",
node_summary="pull the feedback for the week",
prompt_template="pull the feedback for ${params.week_of}",
skills=["sql-queries"],
),
NodeSpec(
id="report",
subagent="content-raven",
node_summary="write the weekly report from the pulled feedback",
prompt_template="write the weekly report for ${params.audience} from {{ pull.output_path }}",
depends_on=["pull"],
mcps=["slack"],
),
],
)
base.update(over)
return PlaybookSpec(**base)
def _branching_nodes() -> list[NodeSpec]:
"""A decision, the branch one of its routes opens, and a step that waits on both."""
return [
NodeSpec(
id="triage",
subagent="data-raven",
node_summary="classify the issue",
prompt_template="classify the issue",
routes={"bug": "the issue is a defect", "other": "anything else"},
),
NodeSpec(
id="fix",
subagent="content-raven",
node_summary="fix the defect",
prompt_template="fix the defect",
depends_on=["triage"],
when="triage.bug",
),
NodeSpec(
id="reply",
subagent="content-raven",
node_summary="reply to the reporter",
prompt_template="reply using {{ fix.output }}",
depends_on=["triage", "fix"],
when="triage.done",
),
]
# ---------------------------------------------------------------- executor
async def test_a_missing_param_is_a_gap_and_dispatches_nothing():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
plan = await ex.execute(_dag_spec(), params={})
# Its own outcome, not "questions": the caller can fix a gap by calling
# again, which is a different instruction from "this cannot proceed".
assert plan.kind == "gaps"
assert "which week should be analyzed?" in plan.reply
assert tool.calls == []
async def test_a_secret_never_reaches_a_dispatched_work_order():
"""Through the executor, because that is where it would leave.
A prompt template is dispatched to a sub-agent and written into the run's
transcript, so this is the one substitution that must not happen. Asserted on
what the graph tool was actually handed rather than on the helper alone -- a
helper that redacts correctly proves nothing if the dispatch path calls a
different one.
"""
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
params={
"week_of": ParamSpec(type="string", required=True, description="which week?"),
"PG_PASSWORD": ParamSpec(type="secret", description="the database password"),
},
nodes=[
NodeSpec(
id="pull",
subagent="data-raven",
node_summary="pull it",
prompt_template="connect with ${params.PG_PASSWORD} and pull ${params.week_of}",
)
],
)
plan = await ex.execute(spec, params={"week_of": "w32", "PG_PASSWORD": "s3cr3t-value"}, confirmed=True)
assert plan.kind != "gaps", plan.reply
assert tool.calls, "the run still happened -- withholding degrades it, it does not stop it"
dispatched = json.dumps(tool.calls[0]["nodes"])
assert "s3cr3t-value" not in dispatched, "the credential reached a sub-agent's work order"
assert "week_of" not in dispatched or "w32" in dispatched, "the ordinary param still filled"
async def test_the_gap_reply_names_the_argument_to_pass():
"""The reader is the caller, and it used to be the user.
The old wording ended "include a trigger word in your reply, e.g. ..." because
the answer had to re-enter through a stateless keyword funnel to be seen at
all. The caller holds the conversation now and simply calls again, so what it
needs is the argument path -- a caller that has to guess which argument a
complaint refers to will guess wrong and spend another round.
"""
ex = PlaybookExecutor(dag_tool=FakeDagTool())
plan = await ex.execute(_dag_spec(), params={})
assert "params.week_of" in plan.reply
assert "load_playbook again" in plan.reply
assert "trigger word" not in plan.reply
async def test_a_blank_node_field_is_a_gap_and_fills_closes_it():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[
NodeSpec(id="draft", subagent="content-raven", node_summary="draft the weekly report", prompt_template=""),
]
)
plan = await ex.execute(spec, params={"week_of": "w"})
assert plan.kind == "gaps"
assert "fills['draft']['promptTemplate']" in plan.reply.replace('"', "'")
assert tool.calls == []
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"draft": {"promptTemplate": "write it up"}})
assert plan.kind == "dag"
assert tool.calls[0]["nodes"][0]["prompt_template"] == "write it up"
async def test_a_blank_node_summary_is_a_gap_and_fills_closes_it():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[
NodeSpec(id="draft", subagent="content-raven", node_summary="", prompt_template="draft the report"),
]
)
plan = await ex.execute(spec, params={"week_of": "w"})
assert plan.kind == "gaps"
assert "fills['draft']['nodeSummary']" in plan.reply.replace('"', "'")
assert tool.calls == []
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"draft": {"nodeSummary": "draft the weekly report"}})
assert plan.kind == "dag"
assert tool.calls[0]["nodes"][0]["node_summary"] == "draft the weekly report"
async def test_fills_cannot_touch_a_field_the_playbook_already_wrote():
"""The check that makes "the playbook's values are fixed" true.
Without it ``fills`` is a general-purpose field editor: the caller could
rewrite any prompt, repoint a node at another agent, or drop its skills, and
the file in git would stop describing what ran.
"""
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
plan = await ex.execute(
_dag_spec(),
params={"week_of": "w"},
fills={"pull": {"promptTemplate": "do something else entirely"}},
)
assert plan.kind == "questions"
assert "already specifies" in plan.reply
assert tool.calls == [] # refused before dispatch, not repaired around
async def test_fills_naming_an_unknown_node_is_refused_by_name():
ex = PlaybookExecutor(dag_tool=FakeDagTool())
plan = await ex.execute(_dag_spec(), params={"week_of": "w"}, fills={"ghost": {"promptTemplate": "x"}})
assert plan.kind == "questions"
assert "ghost" in plan.reply
assert "pull" in plan.reply # the ids it could have meant
@pytest.mark.parametrize("instance", ["", " ", " worker", "worker ", 7])
async def test_fills_cannot_bypass_instance_field_validation(instance):
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[NodeSpec(id="pull", subagent="data-raven", node_summary="pull feedback", prompt_template="go")]
)
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"pull": {"instance": instance}})
assert plan.kind == "questions"
assert "instance" in plan.reply
assert tool.calls == []
async def test_a_valid_filled_instance_reaches_dispatch():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[NodeSpec(id="pull", subagent="data-raven", node_summary="pull feedback", prompt_template="go")]
)
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"pull": {"instance": "worker"}})
assert plan.kind == "dag"
assert tool.calls[0]["nodes"][0]["instance"].endswith("-worker")
async def test_default_project_path_is_filled_directly_without_node_inputs():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
params={"project_path": ParamSpec(default=".", description="which project directory?")},
nodes=[
NodeSpec(
id="detect",
subagent="data-raven",
node_summary="detect project changes",
prompt_template="inspect ${params.project_path}",
)
],
)
plan = await ex.execute(spec, params={})
assert plan.kind == "dag"
dispatched = tool.calls[0]["nodes"][0]
assert dispatched["prompt_template"] == "inspect ."
assert "inputs" not in dispatched
async def test_an_empty_skills_list_is_written_not_blank():
"""``skills: []`` means "no skills at all" -- a written instruction.
Treating it as an invitation would let a caller quietly widen what a step may
reach, which is the opposite of what the author asked for.
"""
ex = PlaybookExecutor(dag_tool=FakeDagTool())
spec = _dag_spec(
nodes=[
NodeSpec(
id="pull",
subagent="data-raven",
node_summary="pull the week's feedback with no extra skills",
prompt_template="go",
skills=[],
)
]
)
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"pull": {"skills": ["anything"]}})
assert plan.kind == "questions"
assert "already specifies" in plan.reply
async def test_executor_dag_hands_nodes_to_the_graph_tool_with_params_filled():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
plan = await ex.execute(_dag_spec(), params={"week_of": "2026-08-11"})
assert plan.kind == "dag"
# Node ids carry the run-unique namespace; the plain author ids survive as
# the last dash segment.
nodes = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
assert "2026-08-11" in nodes["pull"]["prompt_template"]
assert "PM" in nodes["report"]["prompt_template"] # default filled
# Runtime refs stay runtime refs, retargeted to the namespaced id.
assert "{{ %s.output_path }}" % nodes["pull"]["id"] in nodes["report"]["prompt_template"]
assert nodes["report"]["depends_on"] == [nodes["pull"]["id"]]
# The agent the author named reaches the graph tool as the node's own field:
# no backend is built here and no synthetic per-node agent name is invented.
assert nodes["pull"]["subagent"] == "data-raven"
assert nodes["report"]["subagent"] == "content-raven"
# Skills use the graph tool's native three-valued field: a list narrows the
# sub-agent's menu, while an omitted field keeps that agent's own menu.
assert nodes["pull"]["skills"] == ["sql-queries"]
assert "skills" not in nodes["report"]
assert "sql-queries" not in nodes["pull"]["prompt_template"]
assert "skills" not in nodes["report"]["prompt_template"].lower()
# mcps is the one this change makes travel: a real field, not prose, because
# unlike skills the receiving side now has something to do with it.
assert nodes["report"]["mcps"] == ["slack"]
assert "mcps" not in nodes["pull"]
async def test_executor_does_not_promote_multiline_text_to_a_node_reference():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
literal = "use {{ source.output\n}} as ordinary text"
spec = _dag_spec(
nodes=[
NodeSpec(
id="source",
subagent="data-raven",
node_summary="produce source material",
prompt_template="produce source material",
),
NodeSpec(
id="consume",
subagent="content-raven",
node_summary="keep multiline template text literal",
prompt_template=literal,
),
]
)
plan = await ex.execute(spec, params={"week_of": "w"})
assert plan.kind == "dag"
nodes = {node["id"].rsplit("-", 1)[-1]: node for node in tool.calls[0]["nodes"]}
assert nodes["consume"]["prompt_template"] == literal
async def test_executor_forwards_inputs_and_namespaces_node_input_references():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[
NodeSpec(
id="source",
subagent="data-raven",
node_summary="produce the source material",
prompt_template="produce source material",
),
NodeSpec(
id="consume",
subagent="content-raven",
node_summary="consume every declared input",
prompt_template=("source={{ inputs.source }} file={{ inputs.brief }} literal={{ inputs.audience }}"),
depends_on=["source"],
inputs={
"source": {"node": "source"},
"brief": {"file": "brief.md"},
"audience": "PM",
},
),
]
)
await ex.execute(spec, params={"week_of": "w"})
nodes = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
inputs = nodes["consume"]["inputs"]
assert inputs["source"] == {"node": nodes["source"]["id"]}
assert inputs["brief"] == {"file": "brief.md"}
assert inputs["audience"] == "PM"
assert spec.nodes[1].inputs["source"] == {"node": "source"}
async def test_executor_namespaces_the_instance_handle_per_run():
"""Two runs of one playbook must not pour their steps into one session."""
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[
NodeSpec(
id="a",
subagent="data-raven",
node_summary="run the first step of the shared session",
prompt_template="one",
instance="shared",
),
NodeSpec(
id="b",
subagent="data-raven",
node_summary="run the second step of the shared session",
prompt_template="two",
depends_on=["a"],
instance="shared",
),
]
)
await ex.execute(spec, params={"week_of": "w"})
await ex.execute(spec, params={"week_of": "w"})
first = {n["instance"] for n in tool.calls[0]["nodes"]}
second = {n["instance"] for n in tool.calls[1]["nodes"]}
# One handle within a run (that is what the author asked for), a different
# one between runs.
assert len(first) == 1 and len(second) == 1
assert first != second
async def test_a_playbook_node_keeps_its_routes_select_and_guard():
"""``_dispatch`` rebuilds each node from a hand-written field list, and a field
missing from it is dropped without a word -- which is how ``skills`` and ``mcps``
were once lost.
Asserted on what the graph tool was handed, because that is the only place a
dropped field shows: the playbook still loads, validates and runs, as a graph
that never branches.
"""
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
playbook = _dag_spec(
nodes=[
NodeSpec(
id="triage",
subagent="data-raven",
node_summary="classify the issue",
prompt_template="classify the issue",
routes={"bug": "the issue is a defect", "other": "anything else"},
),
NodeSpec(
id="scan",
subagent="data-raven",
node_summary="list what the change could affect",
prompt_template="list what the change could affect",
routes={"api": "the public api is affected", "cli": "the command line is affected"},
select="many",
),
NodeSpec(
id="fix",
subagent="content-raven",
node_summary="fix the defect",
prompt_template="fix the defect",
depends_on=["triage", "scan"],
when="triage.bug and not scan.cli",
),
NodeSpec(
id="note",
subagent="content-raven",
node_summary="write the closing note",
prompt_template="write the closing note",
),
]
)
plan = await ex.execute(playbook, params={"week_of": "w"})
assert plan.kind == "dag"
nodes = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
triage, scan, fix, note = (nodes[name] for name in ("triage", "scan", "fix", "note"))
assert triage["routes"] == {"bug": "the issue is a defect", "other": "anything else"}
assert "select" not in triage and "when" not in triage
assert scan["routes"] == {"api": "the public api is affected", "cli": "the command line is affected"}
assert scan["select"] == "many"
assert fix["when"] == f"{triage['id']}.bug and not {scan['id']}.cli"
assert "routes" not in fix and "select" not in fix
assert not {"routes", "select", "when"} & set(note)
assert playbook.nodes[2].when == "triage.bug and not scan.cli"
async def test_executor_asks_the_graph_tool_to_confirm_unless_the_caller_already_did():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
await ex.execute(_dag_spec(), params={"week_of": "w"})
assert tool.calls[0]["confirm"] is True
await ex.execute(_dag_spec(), params={"week_of": "w"}, confirmed=True)
assert tool.calls[1]["confirm"] is False
async def test_executor_passes_the_playbooks_own_task_summary_to_the_graph_tool():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec()
await ex.execute(spec, params={"week_of": "w"})
assert tool.calls[0]["task_summary"] == spec.task_summary
assert tool.calls[0]["task_summary"] != spec.description
class ComposeProvider:
"""Scripted graph-composition responses."""
def __init__(self, payloads):
self._payloads = list(payloads)
self.calls = []
async def chat_with_retry(self, messages, tools=None, model=None, tool_choice=None, **_):
self.calls.append(messages)
payload = self._payloads.pop(0)
if isinstance(payload, LLMResponse):
return payload
class _TC:
def __init__(self, args):
self.name = "emit_graph"
self.arguments = args
class _R:
def __init__(self, args):
self.has_tool_calls = args is not None
self.tool_calls = [_TC(args)] if args is not None else []
return _R(payload)
def _prompt_spec(**over):
base = dict(
name="due-diligence",
description="run due diligence on a target company",
task_summary="scan the target company and summarize what was found",
mode="prompt",
triggers=Triggers(keywords=["due diligence"]),
params={"target": ParamSpec(required=True, description="which company is the target?")},
prompts="compose a two-layer graph for ${params.target}: a breadth scan, then a summary.",
)
base.update(over)
return PlaybookSpec(**base)
GOOD_GRAPH = {
"nodes": [
{
"id": "scan",
"subagent": "research-raven",
"nodeSummary": "breadth-scan the target company",
"promptTemplate": "breadth-scan AcmeAI",
},
{
"id": "sum",
"subagent": "content-raven",
"nodeSummary": "summarize the scan findings",
"promptTemplate": "summarize {{ scan.output }}",
"dependsOn": ["scan"],
},
]
}
async def test_prompt_mode_returns_the_guidance_to_a_caller_that_can_compose():
"""In a conversation the caller is a model, so composing for it is worse.
A private composition call sees the guidance text and a cached roster and
nothing else; the caller has the whole conversation, and a graph it builds
wrong comes back as an ordinary tool error it can fix. Flexibility is the only
thing prompt mode offers over dag mode -- pinning the graph at load time would
make it a worse dag mode.
"""
tool = FakeDagTool()
provider = ComposeProvider([json.dumps(GOOD_GRAPH)])
ex = PlaybookExecutor(dag_tool=tool, provider=provider)
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
assert plan.kind == "guidance"
assert "AcmeAI" in plan.reply # params substituted into the guidance
assert "run_subagent_dag" in plan.reply # and it says where to take it
assert provider.calls == [] # no composition call spent
assert tool.calls == [] # and nothing dispatched behind the caller's back
async def test_composition_says_when_it_withheld_a_secret():
"""The path that changed a work order and said nothing.
The other two fills log it; this one dropped the names and dispatched a
normal-looking run, so an operator debugging a graph that could not do what
the file asked had no trace of why. All three go through one helper now, and
the message names which text to edit -- "a secret was withheld" is not
actionable without that.
"""
from loguru import logger as _logger
lines: list[str] = []
sink = _logger.add(lambda m: lines.append(str(m)), level="WARNING")
try:
provider = ComposeProvider([json.dumps(GOOD_GRAPH)])
ex = PlaybookExecutor(dag_tool=FakeDagTool(), provider=provider, compose_prompt_mode=True)
ex.set_agent_profiles(lambda: COMPOSE_PROFILES)
spec = _prompt_spec(
params={
"target": ParamSpec(required=True, description="which company?"),
"TOKEN": ParamSpec(type="secret", description="an api credential"),
},
prompts="scan ${params.target} using ${params.TOKEN}",
)
plan = await ex.execute(spec, params={"target": "AcmeAI", "TOKEN": "s3cr3t-value"})
finally:
_logger.remove(sink)
assert plan.kind == "dag", "withholding degrades the run, it does not stop it"
assert "s3cr3t-value" not in provider.calls[0][0]["content"], "the provider must not receive it"
text = "".join(lines)
assert "TOKEN" in text and "withheld" in text
assert "composition" in text, "the operator has to know which text to edit"
async def test_prompt_mode_composes_when_there_is_no_caller_to_hand_it_to():
"""The CLI's shape. Without it, ``raven playbook run`` could not run a
prompt-mode playbook at all -- there is no model in the room to compose."""
tool = FakeDagTool()
provider = ComposeProvider([json.dumps(GOOD_GRAPH)])
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
ex.set_agent_profiles(lambda: COMPOSE_PROFILES)
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
assert plan.kind == "dag"
assert len(tool.calls) == 1
assert "AcmeAI" in provider.calls[0][0]["content"]
async def test_prompt_mode_capability_errors_enter_the_repair_loop():
bad = json.loads(json.dumps(GOOD_GRAPH))
bad["nodes"][0].update(
{
"instance": "worker",
"skills": ["web-search"],
"mcps": ["browser"],
"promptTemplate": "inspect {{ ref_path:/tmp/report.txt }}",
}
)
profiles = dict(COMPOSE_PROFILES)
profiles["research-raven"] = PlaybookAgentProfile(
description="external research agent",
stateful=False,
reads_local_files=False,
injectable_skills=False,
injectable_mcps=False,
)
source_calls = 0
def source():
nonlocal source_calls
source_calls += 1
return profiles
tool = FakeDagTool()
provider = ComposeProvider([json.dumps(bad), json.dumps(GOOD_GRAPH)])
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
ex.set_agent_profiles(source)
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
assert plan.kind == "dag"
assert source_calls == 1
assert len(provider.calls) == 2
repair = provider.calls[1][-1]["content"]
assert "stateless agent" in repair
assert "cannot receive MCP injection" in repair
assert "passes local file paths" in repair
async def test_prompt_mode_repairs_injection_on_a_shared_instance_continuation():
bad = json.loads(json.dumps(GOOD_GRAPH))
bad["nodes"][0].update(instance="writer", skills=["research"])
bad["nodes"][1].update(subagent="research-raven", instance="writer", skills=["editing"])
corrected = json.loads(json.dumps(bad))
corrected["nodes"][1].pop("skills")
tool = FakeDagTool()
provider = ComposeProvider([json.dumps(bad), json.dumps(corrected)])
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
ex.set_agent_profiles(lambda: COMPOSE_PROFILES)
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
assert plan.kind == "dag"
assert len(provider.calls) == 2
repair = provider.calls[1][-1]["content"]
assert "continuing instance 'writer'" in repair
assert "opened by node 'scan'" in repair
async def test_prompt_mode_allows_a_continuation_to_clear_its_mcp_grant():
graph = json.loads(json.dumps(GOOD_GRAPH))
graph["nodes"][0].update(instance="writer", mcps=["browser"])
graph["nodes"][1].update(subagent="research-raven", instance="writer", mcps=[])
profiles = dict(COMPOSE_PROFILES)
profiles["research-raven"] = PlaybookAgentProfile(
description="research",
stateful=True,
reads_local_files=True,
injectable_skills=True,
injectable_mcps=True,
)
tool = FakeDagTool()
provider = ComposeProvider([json.dumps(graph)])
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
ex.set_agent_profiles(lambda: profiles)
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
assert plan.kind == "dag"
assert len(provider.calls) == 1
nodes = {node["id"].rsplit("-", 1)[-1]: node for node in tool.calls[0]["nodes"]}
assert nodes["scan"]["mcps"] == ["browser"]
assert nodes["sum"]["mcps"] == []
async def test_prompt_mode_bad_graph_gets_one_repair_then_degrades():
bad = {"nodes": [{"id": "a", "subagent": "research-raven", "promptTemplate": "use {{ ghost.output }}"}]}
provider = ComposeProvider([json.dumps(bad), json.dumps(bad)])
ex = PlaybookExecutor(dag_tool=FakeDagTool(), provider=provider, compose_prompt_mode=True)
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
assert plan.kind == "questions"
assert "Graph assembly" in plan.reply
assert len(provider.calls) == 2 # one repair round happened
async def test_prompt_mode_repairs_a_branch_error_and_dispatches_the_branch():
"""A composed graph goes through the same graph rules as a submitted one, so a
guard reading a node it does not depend on comes back to the composing model
with the rule's own words, and the corrected branch reaches the graph tool."""
decision = {
"id": "triage",
"subagent": "research-raven",
"nodeSummary": "classify the target company",
"promptTemplate": "classify AcmeAI",
"routes": {"risky": "the target looks risky", "fine": "the target looks fine"},
}
guarded = {
"id": "dig",
"subagent": "content-raven",
"nodeSummary": "dig into the risk",
"promptTemplate": "dig into the risk",
"when": "triage.risky",
}
bad = {"nodes": [decision, {**guarded, "dependsOn": []}]}
corrected = {"nodes": [decision, {**guarded, "dependsOn": ["triage"]}]}
tool = FakeDagTool()
provider = ComposeProvider([json.dumps(bad), json.dumps(corrected)])
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
ex.set_agent_profiles(lambda: COMPOSE_PROFILES)
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
assert plan.kind == "dag"
assert len(provider.calls) == 2
assert "guard names 'triage', which is not in its depends_on" in provider.calls[1][-1]["content"]
nodes = {node["id"].rsplit("-", 1)[-1]: node for node in tool.calls[0]["nodes"]}
assert nodes["triage"]["routes"] == decision["routes"]
assert nodes["dig"]["when"] == f"{nodes['triage']['id']}.risky"
# --- What the generation and composition prompts tell a model a playbook can express.
def test_the_prompts_allow_a_branch_and_still_refuse_a_loop():
system = " ".join(SYSTEM_PROMPT.split())
assert "conditional fallbacks cannot be expressed" not in system
assert "other than a branch" in system
assert "a loop cannot be expressed" in system
assert "A conditional branch can: give a step routes, and guard what follows with when." in system
compose = " ".join(build_compose_prompt("assemble it", COMPOSE_PROFILES, []).split())
assert "no run-time branches" not in compose
assert "design no run-time loops (a conditional branch is expressible)" in compose
def test_the_compose_prompt_gives_a_branch_its_fields_and_the_one_rule_about_them():
"""A prompt that says a branch is expressible has to hand over the fields to write one
with, and the one rule a branch breaks when it is forgotten: a node that depends on
a decision names it in its guard."""
compose = " ".join(build_compose_prompt("assemble it", COMPOSE_PROFILES, []).split())
listed = {word.strip() for word in compose.split("camelCase fields:", 1)[-1].split(")", 1)[0].split("/")}
assert {"routes", "select", "when"} <= listed
assert (
"a node that depends on a decision node (one with routes) must name it in its when, "
"as <id>.<route> or <id>.done"
) in compose
def _shown_playbook(prompt: str) -> dict:
block = prompt.split("# Current playbook\n", 1)[1].split("\n\n# User feedback", 1)[0]
return json.loads(block)
def test_a_playbook_without_a_branch_is_shown_to_the_revise_round_as_it_always_was():
"""The revise prompt dumps every node field, nulls included. Three new optional
fields written as ``null`` on every node would change what a playbook with no
branch looks like to the model revising it, so they are left out until a node
sets one -- and the older optional fields keep their nulls."""
playbook = _dag_spec()
prompt = build_revise_prompt(playbook, "make it faster", COMPOSE_PROFILES)
as_before = playbook.model_dump(by_alias=True, exclude={"nodes": {"__all__": {"routes", "select", "when"}}})
shown = json.dumps(as_before, ensure_ascii=False, indent=2)
assert f"# Current playbook\n{shown}\n\n# User feedback\nmake it faster" in prompt
assert '"instance": null' in prompt and '"mcps": null' in prompt and '"skills": null' in prompt
assert not any(word in prompt for word in ('"routes"', '"select"', '"when"'))
def test_a_branching_playbook_shows_the_revise_round_its_routes_and_guards():
nodes = _branching_nodes()
nodes[0] = nodes[0].model_copy(update={"select": "many"})
nodes.append(
NodeSpec(id="note", subagent="content-raven", node_summary="write the closing note", prompt_template="note")
)
prompt = build_revise_prompt(_dag_spec(nodes=nodes), "make it faster", COMPOSE_PROFILES)
shown = {node["id"]: node for node in _shown_playbook(prompt)["nodes"]}
assert shown["triage"]["routes"] == {"bug": "the issue is a defect", "other": "anything else"}
assert shown["triage"]["select"] == "many"
assert "when" not in shown["triage"]
assert shown["fix"]["when"] == "triage.bug"
assert "routes" not in shown["fix"] and "select" not in shown["fix"]
assert not {"routes", "select", "when"} & set(shown["note"])
async def test_prompt_mode_provider_error_does_not_enter_content_repair():
classification = ErrorClassification(
"upstream_transport_failure",
retryable=True,
should_fallback=True,
)
response = LLMResponse(
content="upstream did not process the request",
finish_reason="error",
error_classification=classification,
)
provider = ComposeProvider([response, json.dumps(GOOD_GRAPH)])
ex = PlaybookExecutor(dag_tool=FakeDagTool(), provider=provider, compose_prompt_mode=True)
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
assert plan.kind == "questions"
assert "upstream_transport_failure" in plan.reply
assert len(provider.calls) == 1
async def test_prompt_mode_missing_tool_does_not_enter_content_repair():
provider = ComposeProvider([None, json.dumps(GOOD_GRAPH)])
ex = PlaybookExecutor(dag_tool=FakeDagTool(), provider=provider, compose_prompt_mode=True)
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
assert plan.kind == "questions"
assert "required_tool_missing" in plan.reply
assert len(provider.calls) == 1
# --- The spec's own mcpServers, on the way to the graph tool. `raven playbook
# --- run` wires a pre-flight that has heard of them; a conversation wires
# --- nothing, so without this hand-off the same file resolved every server it
# --- ships to "not configured" through `load_playbook` and worked on the CLI.
def _mcp_spec(**over):
return _dag_spec(
params={
"week_of": ParamSpec(type="string", required=True, description="which week should be analyzed?"),
"PG_PASSWORD": ParamSpec(type="secret", description="Postgres password"),
},
mcp_servers={
"local-pg": MCPServerConfig(
command="pg-mcp",
args=["--db", "analytics"],
env={"PGPASSWORD": "{{ params.PG_PASSWORD }}"},
)
},
**over,
)
async def test_a_playbooks_own_servers_reach_the_graph_tool_with_their_params_filled():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
plan = await ex.execute(_mcp_spec(), params={"week_of": "w", "PG_PASSWORD": "hunter2"})
assert plan.kind == "dag"
handed = tool.calls[0]["mcp_servers"]
assert sorted(handed) == ["local-pg"]
assert handed["local-pg"].env == {"PGPASSWORD": "hunter2"}
async def test_the_spec_still_holds_the_reference_after_a_dispatch():
# The file is the distribution unit: filling in place would bake one run's
# secret into the library the next run reads.
tool = FakeDagTool()
spec = _mcp_spec()
await PlaybookExecutor(dag_tool=tool).execute(spec, params={"week_of": "w", "PG_PASSWORD": "hunter2"})
assert spec.mcp_servers["local-pg"].env == {"PGPASSWORD": "{{ params.PG_PASSWORD }}"}
async def test_a_playbook_that_ships_no_servers_hands_over_nothing():
tool = FakeDagTool()
await PlaybookExecutor(dag_tool=tool).execute(_dag_spec(), params={"week_of": "w"})
assert tool.calls[0]["mcp_servers"] == {}
async def test_a_prompt_mode_playbooks_servers_reach_the_composed_graph_too():
# The graph comes from a model rather than the file, but the servers still
# come from the file -- and the CLI is the entry point that composes one.
provider = ComposeProvider([json.dumps(GOOD_GRAPH)])
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
spec = _prompt_spec(
mcp_servers={"local-pg": MCPServerConfig(command="pg-mcp", args=["--db", "analytics"])},
)
plan = await ex.execute(spec, params={"target": "AcmeAI"})
assert plan.kind == "dag"
assert sorted(tool.calls[0]["mcp_servers"]) == ["local-pg"]
# ---------------------------------------------------------------- runtime
def _library(tmp_path: Path, specs) -> PlaybookStore:
store = PlaybookStore(tmp_path, builtin_root=tmp_path / "_no_builtin")
for spec in specs:
store.save(spec)
return store
def _runtime(tmp_path: Path, specs, *, dag_tool=None, disabled=(), router=None) -> PlaybookRuntime:
executor = PlaybookExecutor(dag_tool=dag_tool or FakeDagTool())
return PlaybookRuntime(
store=_library(tmp_path, specs),
executor=executor,
disabled=disabled,
router=router,
)
async def test_load_dispatches_a_complete_playbook_in_one_call(tmp_path: Path):
"""No gate call, no confirm round trip, no second tool call.
A playbook whose graph is fully specified is the common case, and the caller
should never see its node contents -- it decided from the description and the
engine did the rest.
"""
tool = FakeDagTool()
rt = _runtime(tmp_path, [_dag_spec()], dag_tool=tool)
plan = await rt.load("weekly-feedback", {"week_of": "2026-08-11"})
assert plan.kind == "dag"
assert len(tool.calls) == 1
async def test_an_unknown_name_is_none_so_the_tool_can_name_the_alternatives(tmp_path: Path):
rt = _runtime(tmp_path, [_dag_spec()])
assert await rt.load("no-such-thing", {}) is None
async def test_the_gap_loop_is_bounded(tmp_path: Path):
""" "Cannot fill it -> ask again -> still cannot" is a loop a caller can spend a
whole turn in, one tool call per pass, with no progress. Past the bound it is
told to stop rather than asked again."""
tool = FakeDagTool()
rt = _runtime(tmp_path, [_dag_spec()], dag_tool=tool)
rt.set_context(channel="web", chat_id="c", session_key="web:c")
for _ in range(MAX_GAP_ROUNDS):
plan = await rt.load("weekly-feedback", {})
assert plan.kind == "gaps"
final = await rt.load("weekly-feedback", {})
assert final.kind == "questions"
assert "do not call this again" in final.reply
assert tool.calls == []
async def test_filling_the_gap_restores_the_budget(tmp_path: Path):
"""The bound is per stuck attempt, not per playbook for the session: a second,
genuinely new run of the same playbook must not start already exhausted."""
rt = _runtime(tmp_path, [_dag_spec()])
rt.set_context(channel="web", chat_id="c", session_key="web:c")
assert (await rt.load("weekly-feedback", {})).kind == "gaps"
assert (await rt.load("weekly-feedback", {"week_of": "w"})).kind == "dag"
# Budget reset by the dispatch, so a later stuck call gets its full allowance.
for _ in range(MAX_GAP_ROUNDS):
assert (await rt.load("weekly-feedback", {})).kind == "gaps"
async def test_a_disabled_playbook_is_invisible_to_the_model_and_runnable_by_hand(tmp_path: Path):
"""Disabling can only mean "not listed" now.
There is no passive matcher left to mute, so the enforcement is that the model
never sees it -- and cannot name it, since the enum comes from the same list.
An explicit ``raven playbook run`` is the user's own hand and still resolves it.
"""
rt = _runtime(tmp_path, [_dag_spec()], disabled=["weekly-feedback"])
assert rt.names() == []
assert rt.listing() == []
assert rt.empty is True
assert await rt.load("weekly-feedback", {"week_of": "w"}) is None
assert (await rt.load("weekly-feedback", {"week_of": "w"}, allow_disabled=True)).kind == "dag"
async def test_the_listing_carries_what_a_caller_cannot_guess(tmp_path: Path):
"""Deleting the gate deleted the only reader of the param table.
Without it in the description the caller can only guess key names, and a
guessed key is dropped silently and comes back as the same question.
"""
rt = _runtime(tmp_path, [_dag_spec()])
detail = dict(rt.listing())["weekly-feedback"]
assert "week_of" in detail
assert "which week should be analyzed?" in detail
assert "required" in detail
assert "default='PM'" in detail
async def test_the_listing_names_the_fields_left_blank(tmp_path: Path):
rt = _runtime(
tmp_path,
[
_dag_spec(
nodes=[
NodeSpec(
id="draft", subagent="content-raven", node_summary="draft the weekly report", prompt_template=""
)
]
)
],
)
detail = dict(rt.listing())["weekly-feedback"]
assert "left for you to fill" in detail
assert "draft.prompt_template" in detail
async def test_a_library_bigger_than_the_router_window_is_narrowed_but_fully_nameable(tmp_path: Path):
"""The two costs are handled differently on purpose.
A description plus a param table is the expensive part, so it is what gets
narrowed. A name is a handful of tokens, and an enum missing one would turn a
retrieval miss into "the model cannot reach it" even when the user just said
its name out loud.
"""
specs = [_dag_spec(name=f"book-{i}", triggers=Triggers(keywords=[f"topic{i}"])) for i in range(6)]
rt = _runtime(tmp_path, specs, router=RouterSizes(top_k=2))
assert len(rt.listing("please do topic4")) == 2
assert dict(rt.listing("please do topic4")) # ranked, and topic4 is in it
assert "book-4" in dict(rt.listing("please do topic4"))
assert len(rt.names()) == 6
async def test_prompt_mode_reaches_the_caller_as_guidance(tmp_path: Path):
tool = FakeDagTool()
rt = _runtime(tmp_path, [_prompt_spec()], dag_tool=tool)
plan = await rt.load("due-diligence", {"target": "AcmeAI"})
assert plan.kind == "guidance"
assert tool.calls == []
async def test_context_is_recorded_for_the_dispatch_address(tmp_path: Path):
"""A tool call arrives without an address, so the last one the loop set is
what the dispatch announces into."""
tool = FakeDagTool()
rt = _runtime(tmp_path, [_dag_spec()], dag_tool=tool)
rt.set_context(channel="web", chat_id="room-7", session_key="web:room-7")
await rt.load("weekly-feedback", {"week_of": "w"})
assert tool.context == ("web", "room-7", "web:room-7")
# --- Run-unique node ids: the runner holds ids unique per session, authors
# write plain stable ones, and the executor's rewrite bridges the two.
class SessionUniqueDagTool(FakeDagTool):
"""Rejects a node id it has already seen, like the real runner's
per-session registry."""
def __init__(self):
super().__init__()
self.seen = set()
async def run_with_roles(self, nodes, *, roles, role_capabilities, background=True):
clashes = [n["id"] for n in nodes if n["id"] in self.seen]
if clashes:
return "Error: node ids already used in this session: %s" % ", ".join(clashes)
self.seen.update(n["id"] for n in nodes)
return await super().run_with_roles(nodes, roles=roles, role_capabilities=role_capabilities)
async def test_same_playbook_runs_twice_in_one_session():
tool = SessionUniqueDagTool()
executor = PlaybookExecutor(dag_tool=tool)
first = await executor.execute(_dag_spec(), {"week_of": "2026-08-10"})
second = await executor.execute(_dag_spec(), {"week_of": "2026-08-17"})
assert first.kind == "dag"
assert second.kind == "dag"
assert len(tool.calls) == 2
async def test_node_ids_and_references_are_rewritten_in_step():
tool = FakeDagTool()
executor = PlaybookExecutor(dag_tool=tool)
plan = await executor.execute(_dag_spec(), {"week_of": "2026-08-10"})
assert plan.kind == "dag"
by_plain = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
pull, report = by_plain["pull"], by_plain["report"]
assert pull["id"] != "pull" and pull["id"].startswith("weekly-feedback-")
# One rewrite table per dispatch: both nodes carry the same run tag.
assert pull["id"].rsplit("-", 1)[0] == report["id"].rsplit("-", 1)[0]
assert report["depends_on"] == [pull["id"]]
assert ("{{ %s.output_path }}" % pull["id"]) in report["prompt_template"]
assert "{{ pull.output_path }}" not in report["prompt_template"]
# Params were already compiled in; namespacing must not disturb them or the
# author's prompt. Skills travel separately as a native DAG node field.
assert pull["prompt_template"].startswith("pull the feedback for 2026-08-10")
async def test_rewrite_leaves_foreign_references_alone():
tool = FakeDagTool()
executor = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[
NodeSpec(
id="pull",
subagent="data-raven",
node_summary="read the summary file and an external reference",
prompt_template="read {{ ref:notes/summary.md }} then {{ elsewhere.output }}",
)
]
)
await executor.execute(spec, {"week_of": "2026-08-10"})
template = tool.calls[0]["nodes"][0]["prompt_template"]
assert "{{ ref:notes/summary.md }}" in template
# An id outside this graph is not ours to rewrite; the runner's own
# whitelist decides what happens to it.
assert "{{ elsewhere.output }}" in template
def test_a_namespaced_guard_names_the_namespaced_nodes():
"""Two runs of one playbook in one conversation: the second run's guard has to
follow the ids the namespace pass mints, or the graph is refused for naming a
node outside its depends_on -- the first run took the plain ids."""
nodes = [
NodeSpec(
id="r", subagent="x", node_summary="r", prompt_template="r", routes={"a": "a", "b": "b"}, depends_on=[]
),
NodeSpec(id="g", subagent="x", node_summary="g", prompt_template="g", depends_on=["r"], when="r.a"),
]
first = _namespace_run("pb", nodes)
second = _namespace_run("pb", nodes)
for run in (first, second):
router, guarded = run
assert guarded.depends_on == [router.id]
assert guarded.when == f"{router.id}.a"
assert router.routes == {"a": "a", "b": "b"}
assert first[0].id != second[0].id
assert first[0].id not in second[1].when
assert nodes[1].when == "r.a", "the playbook's own node is not rewritten in place"
@pytest.mark.parametrize(
("guard", "expected"),
[
("r.a", "{r}.a"),
("r.done", "{r}.done"),
("not (r.a or s.done)\tand\nr.b", "not ({r}.a or {s}.done)\tand\n{r}.b"),
# A route may be named like another node: only the part before the dot is an id.
("r.s and not s.r", "{r}.s and not {s}.r"),
# One id may be the front of another: each atom is read whole, not found by substring.
("rr.a or r.a", "{rr}.a or {r}.a"),
# An id this playbook does not have is not ours to rename; the graph tool's own checks name it.
("ghost.a", "ghost.a"),
(" r.a ", " {r}.a "),
(" ", " "),
("", ""),
(None, None),
],
)
def test_the_namespace_pass_moves_only_the_node_ids_inside_a_guard(guard, expected):
nodes = [
NodeSpec(id=name, subagent="x", node_summary=name, prompt_template=name, routes={"a": "a", "s": "s", "r": "r"})
for name in ("r", "s", "rr")
]
nodes.append(NodeSpec(id="g", subagent="x", node_summary="g", prompt_template="g", when=guard))
run = _namespace_run("pb", nodes)
minted = {original.id: renamed.id for original, renamed in zip(nodes, run, strict=True)}
assert run[-1].when == (expected.format(**minted) if expected is not None else None)
assert [node.routes for node in run[:3]] == [node.routes for node in nodes[:3]]
def test_the_namespace_pass_keeps_each_atom_of_a_repeated_route_to_its_own_node():
"""Both nodes declare a route `x`, so the route name says nothing about where an
atom ends: a rename splicing by the name alone would take the wrong span."""
nodes = [
NodeSpec(
id=name, subagent="x", node_summary=name, prompt_template=name, routes={"x": "x holds", "y": "y holds"}
)
for name in ("alpha", "beta")
]
nodes.append(NodeSpec(id="g", subagent="x", node_summary="g", prompt_template="g", when="alpha.x and beta.x"))
run = _namespace_run("pb", nodes)
minted = {original.id: renamed.id for original, renamed in zip(nodes, run, strict=True)}
assert run[-1].when == f"{minted['alpha']}.x and {minted['beta']}.x"
async def test_a_branching_playbook_survives_the_library_and_reaches_the_graph_tool(tmp_path: Path):
"""Saved, read back, validated and dispatched: the whole way a playbook travels,
with the guard following the node ids the dispatch mints."""
tool = FakeDagTool()
rt = _runtime(tmp_path, [_dag_spec(nodes=_branching_nodes())], dag_tool=tool)
plan = await rt.load("weekly-feedback", {"week_of": "w"})
assert plan.kind == "dag"
nodes = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
triage, fix, reply = nodes["triage"], nodes["fix"], nodes["reply"]
assert triage["routes"] == {"bug": "the issue is a defect", "other": "anything else"}
assert fix["when"] == f"{triage['id']}.bug"
assert reply["when"] == f"{triage['id']}.done"
assert reply["depends_on"] == [triage["id"], fix["id"]]
class ValidatingDagTool(FakeDagTool):
"""Holds each dispatched graph to the checks the graph tool applies on submission."""
async def execute(self, nodes, task_summary="", **kwargs):
validate_and_order(parse_dag_spec({"task_summary": task_summary, "nodes": nodes}))
return await super().execute(nodes, task_summary=task_summary, **kwargs)
async def test_two_runs_of_a_branching_playbook_each_pass_the_graph_tools_own_checks(tmp_path: Path):
"""The second run in one conversation cannot reuse the plain ids the first took,
so its guard has to name its own nodes: a guard still naming the plain ones would
read a node outside its depends_on, and the graph tool would refuse the graph."""
tool = ValidatingDagTool()
rt = _runtime(tmp_path, [_dag_spec(nodes=_branching_nodes())], dag_tool=tool)
first = await rt.load("weekly-feedback", {"week_of": "w"})
second = await rt.load("weekly-feedback", {"week_of": "w"})
assert first.kind == second.kind == "dag"
first_ids, second_ids = ({node["id"] for node in call["nodes"]} for call in tool.calls)
assert first_ids.isdisjoint(second_ids)
for call in tool.calls:
nodes = {node["id"].rsplit("-", 1)[-1]: node for node in call["nodes"]}
assert nodes["fix"]["when"] == f"{nodes['triage']['id']}.bug"
async def test_an_empty_skills_list_is_forwarded_without_changing_the_prompt():
"""``skills: []`` hides the skill menu; it is not the same as omission."""
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[
NodeSpec(
id="pull",
subagent="data-raven",
node_summary="pull the week's feedback with no extra skills",
prompt_template="go",
skills=[],
)
]
)
plan = await ex.execute(spec, params={"week_of": "w"})
assert plan.kind == "dag"
assert tool.calls[0]["nodes"][0]["prompt_template"] == "go"
assert tool.calls[0]["nodes"][0]["skills"] == []
async def test_mcps_are_passed_to_the_graph_tool_without_prompt_rewriting():
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[
NodeSpec(
id="pull",
subagent="data-raven",
node_summary="pull data that needs a github mcp",
prompt_template="go",
mcps=["github"],
)
]
)
plan = await ex.execute(spec, params={"week_of": "w"})
assert plan.kind == "dag"
node = tool.calls[0]["nodes"][0]
assert node["mcps"] == ["github"]
assert "github" not in node["prompt_template"]
async def test_the_dispatch_receipt_names_the_run() -> None:
"""A client restoring the card from history has no events, only this line.
It recovers the run from the result text, in the graph tool's own shape, so a
receipt naming only the playbook left a reopened conversation showing a
dispatch it could not connect to any run: no node states, no way back to the
graph. The model gets a handle on the run out of the same line.
"""
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
plan = await ex.execute(_dag_spec(), params={"week_of": "w"})
assert plan.kind == "dag"
assert plan.reply.startswith("DAG "), plan.reply
assert "weekly-feedback" in plan.reply and "2 steps" in plan.reply
async def test_both_param_spellings_fill_from_the_one_mechanism():
"""``{{ params.x }}`` is the spelling an author reaches for inside an MCP
server definition, where ``{{ ... }}`` is already the placeholder family.
It means the same thing everywhere, so it is one substitution and not two.
"""
tool = FakeDagTool()
ex = PlaybookExecutor(dag_tool=tool)
spec = _dag_spec(
nodes=[
NodeSpec(
id="pull",
subagent="data-raven",
node_summary="pull the feedback",
prompt_template="pull ${params.week_of} for {{ params.audience }}",
)
]
)
plan = await ex.execute(spec, params={"week_of": "2026-08-11"})
assert plan.kind == "dag"
assert tool.calls[0]["nodes"][0]["prompt_template"] == "pull 2026-08-11 for PM"
# ── Stored credentials (raven/playbook/credentials.py) ─────────────────────────
def _secret_pg_spec():
return _dag_spec(
params={"PG_PASSWORD": ParamSpec(type="secret", required=True, description="the db password")},
mcp_servers={"local-pg": MCPServerConfig(command="pg-mcp", env={"PGPASSWORD": "{{ params.PG_PASSWORD }}"})},
)
async def test_a_stored_secret_stands_in_for_one_the_caller_did_not_supply(tmp_path, monkeypatch):
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
from raven.playbook.credentials import set_secret_param
spec = _secret_pg_spec()
set_secret_param(spec.name, "PG_PASSWORD", "from-the-tab")
tool = FakeDagTool()
plan = await PlaybookExecutor(dag_tool=tool).execute(spec, {})
assert plan.kind == "dag"
handed = tool.calls[0]["mcp_servers"]
assert handed["local-pg"].env == {"PGPASSWORD": "from-the-tab"}
assert tool.calls[0]["mcp_scope"] == "playbooks/" + spec.name
async def test_an_unset_secret_is_a_reminder_not_a_gap(tmp_path, monkeypatch):
# A gap is closed by the caller collecting the value in the conversation,
# which is the one path a secret may not take. The run goes ahead, the server
# the secret fills is handed over without it, and the reply says where to set it.
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
spec = _secret_pg_spec()
tool = FakeDagTool()
plan = await PlaybookExecutor(dag_tool=tool).execute(spec, {})
assert plan.kind == "dag"
assert "was not run" not in plan.reply
assert "secret param 'PG_PASSWORD' is not set on this machine" in plan.reply
assert f"raven playbook secret set {spec.name} PG_PASSWORD" in plan.reply
assert tool.calls[0]["mcp_servers"]["local-pg"].env == {}
async def test_a_supplied_secret_outranks_the_stored_one(tmp_path, monkeypatch):
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
from raven.playbook.credentials import set_secret_param
spec = _secret_pg_spec()
set_secret_param(spec.name, "PG_PASSWORD", "stored")
tool = FakeDagTool()
await PlaybookExecutor(dag_tool=tool).execute(spec, {"PG_PASSWORD": "explicit"})
assert tool.calls[0]["mcp_servers"]["local-pg"].env == {"PGPASSWORD": "explicit"}
async def test_a_secret_stored_after_the_dispatch_reaches_the_next_read(tmp_path, monkeypatch):
# The hand-off is re-rendered on every read, which is what lets a node
# that is continued after the credential landed dial with it.
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
from raven.playbook.credentials import set_secret_param
spec = _secret_pg_spec()
set_secret_param(spec.name, "PG_PASSWORD", "v1")
tool = FakeDagTool()
await PlaybookExecutor(dag_tool=tool).execute(spec, {})
assert tool.calls[0]["mcp_servers"]["local-pg"].env == {"PGPASSWORD": "v1"}
set_secret_param(spec.name, "PG_PASSWORD", "v2")
assert tool.calls[0]["render"]()["local-pg"].env == {"PGPASSWORD": "v2"}
async def test_a_stored_value_for_a_non_secret_param_is_not_read_back(tmp_path, monkeypatch):
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
from raven.playbook.credentials import set_secret_param
spec = _dag_spec(params={"topic": ParamSpec(type="string", required=True, description="what to scan")})
set_secret_param(spec.name, "topic", "smuggled")
tool = FakeDagTool()
plan = await PlaybookExecutor(dag_tool=tool).execute(spec, {})
assert plan.kind == "gaps"
assert "params.topic" in plan.reply
assert tool.calls == []
async def test_a_playbook_without_carried_servers_hands_over_no_scope():
tool = FakeDagTool()
await PlaybookExecutor(dag_tool=tool).execute(_dag_spec(), params={"week_of": "w"})
assert tool.calls[0]["mcp_scope"] is None
async def test_an_entry_that_already_asked_can_say_so_through_load(tmp_path: Path):
"""``confirmed`` reaches the gate through the runtime, not only the executor.
The parameter exists for "an entry point that *does* ask", and every such
entry arrives through ``load`` -- the one door the model's tool and the
command line both use. Without the relay the flag is reachable only by a
caller that already holds the executor, which no entry point does.
"""
tool = FakeDagTool()
rt = _runtime(tmp_path, [_dag_spec(confirm=True)], dag_tool=tool)
await rt.load("weekly-feedback", {"week_of": "w32"}, confirmed=True)
assert tool.calls[-1]["confirm"] is False, "a caller that asked is not asked again"
await rt.load("weekly-feedback", {"week_of": "w32"}, confirmed=False)
assert tool.calls[-1]["confirm"] is True, "and one that did not still meets the gate"