## Summary A sub-agent DAG node can now decide which of its downstream branches run. Before this, a background run could not turn on a result: every node ran, and choosing a path meant the host winding the run down and starting another. How it works: - A node that declares `routes` (route name to condition) is a decision node; `select` is `"one"` (the default) or `"many"`. Raven's per-node judge, the existing verdict call, picks the route or routes the node's answer satisfies. The node's own text never selects a route. A route that cannot be determined fails the node, with no fallback route. - A downstream node carries `when`, a guard over `<node>.<route>` and `<node>.done` atoms joined with `not`, `and`, `or` and parentheses, evaluated with three-valued logic: a route of a decision node that was never executed is unknown. Every decision node in a node's `depends_on` must appear in its `when`; such graphs are refused at submission otherwise, along with guards that can never hold and a route named `done` in any letter case. Validation keeps negation parity and the surrounding boolean expression: a double-negated `done` and a disjunction with a satisfiable arm are accepted, without expanding disjunctions into combinations. - A node that does not run is `not_executed`, with a reason: `when_false`, `when_unknown` or `upstream_not_executed`. A node whose dependencies all went unexecuted is not executed either; a failed or skipped dependency still skips its dependents first. - A merge node reads a missing upstream as a fixed sentence, `<This task is not executed and the output is missing, please go on without this output.>`, written outside every untrusted-content fence, plus a note listing the inputs that may be missing. A file reference to the files of a node that may not execute is refused at submission however its path is spelled: `@nodes/`, an absolute path into the node root, or a workdir path that lands there, symlinks followed. Windows case, separator and drive-relative aliases are recognized, and the advice keeps the declared node ID. - Node judgement is always on: the `subagentDag.verdictEnabled` switch is retired. Config migration 14 removes it with a notice, and drops it again on every later load should an older build sharing the config write it back. `raven playbook run` now judges its nodes too, on the same `subagentDag` settings and a pinned `verdictModel`'s own provider; with nobody there to answer, a node judged not to have accomplished its task fails on the judge's reason. A host with no judge at all refuses graphs that declare routes. Where it shows: - Run events, the manifest, `dag.get`, `tasks.list` and `dag.run_completed` carry `reason`, `selected_routes` and `route_evidence` per node, and `routes`, `select` and `when` from the graph. The instance registry's DAG-node rows (and `InstanceRow`) carry the outcome too, so a client that reloads before the run finalizes shows what a branch decided. `TaskCounts.not_executed` is a new counter. - TUI: a not-executed node shows a hollow marker and its reason in words; a decision node shows its selection on the node's second line, and the guard and route evidence in the detail and in `/dag <node>`. The Agents Overlay and the live agents strip leave not-executed nodes out, as the backend's agent lists do. - Web: the desk's tasks tab names not-executed steps and shows the same fields. - Playbooks pass the three fields through, namespace guard atoms with the run prefix, and keep them through compilation. - The `run_subagent_dag` schema, the DAG orchestration skill and the docs site (English and Chinese) describe routes, guards and the merge rule; CHANGELOG has the entries. A graph with no `routes` and no `when` behaves exactly as before: the same judge requests, events, records, summary and finish text. `route_evidence` is a quote the judge picks from the sub-agent's output; it is cut at 500 characters, keeps a lone surrogate from a malformed escape as the escape's own text, and never enters model-facing text (finish text, `dag_status`, exception reports). When a judge answer does not report a usable route, the judge is asked again at most twice, each time with one correction. The shared instance registry drops its cache when a write fails on the data rather than the disk, so a row the file cannot hold fails its own write and no other. Two Windows CI steps run the alias and finish-text regressions; the latter exposes the installed Git `cat` executable required by the existing fixture. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Linux, Python 3.12.13, on this revision rebased onto the current main (`$CHECKS` is a temporary evidence directory): ```sh uv run --frozen --no-sync pytest -q -n 4 # 23 failed, 33229 passed, 162 skipped, 1 xfailed, 3 errors uv run --frozen --no-sync pytest $(git diff --name-only origin/main...HEAD -- tests/) -q -n 4 --idle-ceiling-strict --cov=raven --cov-branch --cov-report=json:$CHECKS/coverage.json # the 20 test files this branch changes: 2622 passed, 1 skipped (a POSIX permission case that does not apply to root) uv run --frozen --no-sync python scripts/coverage_gate.py --coverage-json "$CHECKS/coverage.json" diff --base-ref origin/main --head HEAD --threshold 90 # Diff coverage: 99.48% (764/768 executable changed lines), passed the 90.00% threshold ``` The full run's 23 failures and 3 errors are 25 tests. 24 of them fail the same way on the current main in this environment: the venv lacks `nltk` and `datrie` (16), a proxy is set in the environment (5), this box has its own `npm` in `/usr/bin` (1 install-script case), LibreOffice writes no `.xls` here (1), and the run is root (1). The 25th, `test_a_cli_run_left_on_the_machine_when_this_computer_stops_is_picked_up_by_the_next_start`, failed with its teardown in the full run and in one of two runs of its file, and passed in five runs on its own: it reads the remote-run record the moment its pid file appears, a race in code this branch does not change (#840 makes that test wait for both). The four uncovered changed lines are the gateway's `_verdict_provider` body, whose helper is tested on its own; a guard atom over a node that did not complete, which the runner never evaluates because it reads a guard only once every dependency completed or was not executed; and the cut of an over-long terminal output. Ruff check and format, `ty check` over the changed modules, the import contracts, `deptry raven`, commitlint, and `scripts/check_commit_messages.py`, `check_large_files.py` and `check_source_language.py` over this branch's commits pass. ui-web: `vitest run src/features/tasks/BoardCard.test.tsx scripts/gates` passes 50 files and 244 tests, and `tsc --noEmit` is clean. In Chromium a not-executed title whose first word is wider than the room beside its status now breaks that word instead of starting under a blank line, and every other tasks-board card renders pixel-identical, English and Chinese, light and dark. Firefox still starts such a title under a blank line and cuts it without an ellipsis. CI's `TUI checks` covers ui-tui. Mutation checks: removing any one of these makes its own test fail: the every-load strip of the retired switch, the quote's escape before the cut, the registry's cache drop, the node-root check in the tool, the runner and the replan path, the physical path through a symlink, the CLI's judge, its settings and its no-answer failure, and the pinned judge provider. A model reading only the DAG skill's catch-all paragraph answers that without a catch-all route the judge may pick the nearest route or fail the node; an earlier wording led it to answer that the graph always stops. A live run on `openrouter/z-ai/glm-5.3-flash`, two graphs with synthetic messages, selected the expected route at each decision node, left the unselected branch `not_executed` with `when_false`, and put the fixed sentence outside the fences. Earlier revisions of this change passed the native Windows selections (aliases, finish text, guards; Windows build 26300, Python 3.13.12). The final revision has not been rerun on native Windows or macOS. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes Security: sub-agent text stays fenced or out of model-facing text on every new path. The routes block in the judge request sits after the last untrusted fence and is labelled as the graph author's; the missing-output sentence is written outside every fence; `route_evidence` is bounded and reaches records, events, the registry and the UIs only. In the TUI, route evidence is drawn through the same path as a node's error text; neither strips terminal control sequences, which this PR leaves as it is. Behaviour changes: a config that set `subagentDag.verdictEnabled: false` now gets judged nodes; migration 14 removes the key and says so once. `raven playbook run` judges every node, one more model call per node, and a node judged not to have accomplished its task fails there instead of completing. New wire fields are optional except `TaskCounts.not_executed`, which is always present (0 when nothing branched). `InstanceRow` gains three optional fields, and a decision node's registry row now also stores its selection and up to 500 characters of route evidence. Conditional-artifact validation matches node IDs using the session namespace's case-insensitive identity on every host, including macOS. This conservatively rejects case-only spellings even on case-sensitive filesystems. Separator and drive normalization remain native, and unconditional file resolution keeps its existing behaviour. Rollback: revert the squash commit. A config already stamped 14 loads unchanged on the previous build, since the stamp only gates migrations; a user who had set `verdictEnabled: false` can set it again there, and a later build drops it again on load. ## Related Issues Related, found while building this and not changed here: #998, #999, #1000. Follow-ups from its review, not changed here: #1008.
1542 lines
60 KiB
Python
1542 lines
60 KiB
Python
"""Execution side: what loading a playbook produces, and what it refuses.
|
|
|
|
"Funnel" is gone from this file along with the thing it named. The runtime is a
|
|
library plus one loader now, so what is covered is the loader's contract: params
|
|
and blanks reported together and before anything runs, ``fills`` able to complete
|
|
a playbook but never to edit one, the gap loop bounded, and the two modes going to
|
|
the party that can act on them.
|
|
"""
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from raven.agent.subagent.dag_graph import parse_dag_spec, validate_and_order
|
|
from raven.config.schema import MCPServerConfig
|
|
from raven.contracts.llm_provider import ErrorClassification, LLMResponse
|
|
from raven.playbook import (
|
|
MAX_GAP_ROUNDS,
|
|
NodeSpec,
|
|
ParamSpec,
|
|
PlaybookExecutor,
|
|
PlaybookRuntime,
|
|
PlaybookSpec,
|
|
PlaybookStore,
|
|
RouterSizes,
|
|
Triggers,
|
|
)
|
|
from raven.playbook.agent_profiles import PlaybookAgentProfile
|
|
from raven.playbook.executor import _namespace_run
|
|
from raven.playbook.prompt import SYSTEM_PROMPT, build_compose_prompt, build_revise_prompt
|
|
|
|
COMPOSE_PROFILES = {
|
|
"research-raven": PlaybookAgentProfile(
|
|
description="research",
|
|
stateful=True,
|
|
reads_local_files=True,
|
|
injectable_skills=True,
|
|
injectable_mcps=False,
|
|
),
|
|
"content-raven": PlaybookAgentProfile(
|
|
description="writing",
|
|
stateful=True,
|
|
reads_local_files=True,
|
|
injectable_skills=True,
|
|
injectable_mcps=False,
|
|
),
|
|
}
|
|
|
|
|
|
class FakeDagTool:
|
|
def __init__(self):
|
|
self.calls = []
|
|
self.context = None
|
|
|
|
def set_context(self, channel, chat_id, session_key=None):
|
|
self.context = (channel, chat_id, session_key)
|
|
|
|
async def execute(
|
|
self, nodes, task_summary="", background=True, confirm=False, mcp_servers=None, mcp_scope=None, **_
|
|
):
|
|
# The hand-off is a callable the graph tool re-reads per dispatch; the
|
|
# fake keeps it so a test can read it again after a credential lands.
|
|
render = mcp_servers if callable(mcp_servers) else (lambda: mcp_servers or {})
|
|
self.calls.append(
|
|
{
|
|
"nodes": nodes,
|
|
"task_summary": task_summary,
|
|
"background": background,
|
|
"confirm": confirm,
|
|
"mcp_servers": dict(render()),
|
|
"render": render,
|
|
"mcp_scope": mcp_scope,
|
|
}
|
|
)
|
|
return "DAG run wf-1 started in the background (%d nodes)." % len(nodes)
|
|
|
|
|
|
def _dag_spec(**over):
|
|
base = dict(
|
|
name="weekly-feedback",
|
|
description="weekly user-feedback analysis",
|
|
task_summary="pull this week's feedback and write the report",
|
|
mode="dag",
|
|
triggers=Triggers(keywords=["user feedback"]),
|
|
params={
|
|
"week_of": ParamSpec(type="string", required=True, description="which week should be analyzed?"),
|
|
"audience": ParamSpec(default="PM", description="who reads the report?"),
|
|
},
|
|
nodes=[
|
|
NodeSpec(
|
|
id="pull",
|
|
subagent="data-raven",
|
|
node_summary="pull the feedback for the week",
|
|
prompt_template="pull the feedback for ${params.week_of}",
|
|
skills=["sql-queries"],
|
|
),
|
|
NodeSpec(
|
|
id="report",
|
|
subagent="content-raven",
|
|
node_summary="write the weekly report from the pulled feedback",
|
|
prompt_template="write the weekly report for ${params.audience} from {{ pull.output_path }}",
|
|
depends_on=["pull"],
|
|
mcps=["slack"],
|
|
),
|
|
],
|
|
)
|
|
base.update(over)
|
|
return PlaybookSpec(**base)
|
|
|
|
|
|
def _branching_nodes() -> list[NodeSpec]:
|
|
"""A decision, the branch one of its routes opens, and a step that waits on both."""
|
|
return [
|
|
NodeSpec(
|
|
id="triage",
|
|
subagent="data-raven",
|
|
node_summary="classify the issue",
|
|
prompt_template="classify the issue",
|
|
routes={"bug": "the issue is a defect", "other": "anything else"},
|
|
),
|
|
NodeSpec(
|
|
id="fix",
|
|
subagent="content-raven",
|
|
node_summary="fix the defect",
|
|
prompt_template="fix the defect",
|
|
depends_on=["triage"],
|
|
when="triage.bug",
|
|
),
|
|
NodeSpec(
|
|
id="reply",
|
|
subagent="content-raven",
|
|
node_summary="reply to the reporter",
|
|
prompt_template="reply using {{ fix.output }}",
|
|
depends_on=["triage", "fix"],
|
|
when="triage.done",
|
|
),
|
|
]
|
|
|
|
|
|
# ---------------------------------------------------------------- executor
|
|
|
|
|
|
async def test_a_missing_param_is_a_gap_and_dispatches_nothing():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
plan = await ex.execute(_dag_spec(), params={})
|
|
|
|
# Its own outcome, not "questions": the caller can fix a gap by calling
|
|
# again, which is a different instruction from "this cannot proceed".
|
|
assert plan.kind == "gaps"
|
|
assert "which week should be analyzed?" in plan.reply
|
|
assert tool.calls == []
|
|
|
|
|
|
async def test_a_secret_never_reaches_a_dispatched_work_order():
|
|
"""Through the executor, because that is where it would leave.
|
|
|
|
A prompt template is dispatched to a sub-agent and written into the run's
|
|
transcript, so this is the one substitution that must not happen. Asserted on
|
|
what the graph tool was actually handed rather than on the helper alone -- a
|
|
helper that redacts correctly proves nothing if the dispatch path calls a
|
|
different one.
|
|
"""
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
params={
|
|
"week_of": ParamSpec(type="string", required=True, description="which week?"),
|
|
"PG_PASSWORD": ParamSpec(type="secret", description="the database password"),
|
|
},
|
|
nodes=[
|
|
NodeSpec(
|
|
id="pull",
|
|
subagent="data-raven",
|
|
node_summary="pull it",
|
|
prompt_template="connect with ${params.PG_PASSWORD} and pull ${params.week_of}",
|
|
)
|
|
],
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w32", "PG_PASSWORD": "s3cr3t-value"}, confirmed=True)
|
|
|
|
assert plan.kind != "gaps", plan.reply
|
|
assert tool.calls, "the run still happened -- withholding degrades it, it does not stop it"
|
|
dispatched = json.dumps(tool.calls[0]["nodes"])
|
|
assert "s3cr3t-value" not in dispatched, "the credential reached a sub-agent's work order"
|
|
assert "week_of" not in dispatched or "w32" in dispatched, "the ordinary param still filled"
|
|
|
|
|
|
async def test_the_gap_reply_names_the_argument_to_pass():
|
|
"""The reader is the caller, and it used to be the user.
|
|
|
|
The old wording ended "include a trigger word in your reply, e.g. ..." because
|
|
the answer had to re-enter through a stateless keyword funnel to be seen at
|
|
all. The caller holds the conversation now and simply calls again, so what it
|
|
needs is the argument path -- a caller that has to guess which argument a
|
|
complaint refers to will guess wrong and spend another round.
|
|
"""
|
|
ex = PlaybookExecutor(dag_tool=FakeDagTool())
|
|
plan = await ex.execute(_dag_spec(), params={})
|
|
|
|
assert "params.week_of" in plan.reply
|
|
assert "load_playbook again" in plan.reply
|
|
assert "trigger word" not in plan.reply
|
|
|
|
|
|
async def test_a_blank_node_field_is_a_gap_and_fills_closes_it():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(id="draft", subagent="content-raven", node_summary="draft the weekly report", prompt_template=""),
|
|
]
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"})
|
|
assert plan.kind == "gaps"
|
|
assert "fills['draft']['promptTemplate']" in plan.reply.replace('"', "'")
|
|
assert tool.calls == []
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"draft": {"promptTemplate": "write it up"}})
|
|
assert plan.kind == "dag"
|
|
assert tool.calls[0]["nodes"][0]["prompt_template"] == "write it up"
|
|
|
|
|
|
async def test_a_blank_node_summary_is_a_gap_and_fills_closes_it():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(id="draft", subagent="content-raven", node_summary="", prompt_template="draft the report"),
|
|
]
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"})
|
|
assert plan.kind == "gaps"
|
|
assert "fills['draft']['nodeSummary']" in plan.reply.replace('"', "'")
|
|
assert tool.calls == []
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"draft": {"nodeSummary": "draft the weekly report"}})
|
|
assert plan.kind == "dag"
|
|
assert tool.calls[0]["nodes"][0]["node_summary"] == "draft the weekly report"
|
|
|
|
|
|
async def test_fills_cannot_touch_a_field_the_playbook_already_wrote():
|
|
"""The check that makes "the playbook's values are fixed" true.
|
|
|
|
Without it ``fills`` is a general-purpose field editor: the caller could
|
|
rewrite any prompt, repoint a node at another agent, or drop its skills, and
|
|
the file in git would stop describing what ran.
|
|
"""
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
|
|
plan = await ex.execute(
|
|
_dag_spec(),
|
|
params={"week_of": "w"},
|
|
fills={"pull": {"promptTemplate": "do something else entirely"}},
|
|
)
|
|
|
|
assert plan.kind == "questions"
|
|
assert "already specifies" in plan.reply
|
|
assert tool.calls == [] # refused before dispatch, not repaired around
|
|
|
|
|
|
async def test_fills_naming_an_unknown_node_is_refused_by_name():
|
|
ex = PlaybookExecutor(dag_tool=FakeDagTool())
|
|
plan = await ex.execute(_dag_spec(), params={"week_of": "w"}, fills={"ghost": {"promptTemplate": "x"}})
|
|
|
|
assert plan.kind == "questions"
|
|
assert "ghost" in plan.reply
|
|
assert "pull" in plan.reply # the ids it could have meant
|
|
|
|
|
|
@pytest.mark.parametrize("instance", ["", " ", " worker", "worker ", 7])
|
|
async def test_fills_cannot_bypass_instance_field_validation(instance):
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[NodeSpec(id="pull", subagent="data-raven", node_summary="pull feedback", prompt_template="go")]
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"pull": {"instance": instance}})
|
|
|
|
assert plan.kind == "questions"
|
|
assert "instance" in plan.reply
|
|
assert tool.calls == []
|
|
|
|
|
|
async def test_a_valid_filled_instance_reaches_dispatch():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[NodeSpec(id="pull", subagent="data-raven", node_summary="pull feedback", prompt_template="go")]
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"pull": {"instance": "worker"}})
|
|
|
|
assert plan.kind == "dag"
|
|
assert tool.calls[0]["nodes"][0]["instance"].endswith("-worker")
|
|
|
|
|
|
async def test_default_project_path_is_filled_directly_without_node_inputs():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
params={"project_path": ParamSpec(default=".", description="which project directory?")},
|
|
nodes=[
|
|
NodeSpec(
|
|
id="detect",
|
|
subagent="data-raven",
|
|
node_summary="detect project changes",
|
|
prompt_template="inspect ${params.project_path}",
|
|
)
|
|
],
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={})
|
|
|
|
assert plan.kind == "dag"
|
|
dispatched = tool.calls[0]["nodes"][0]
|
|
assert dispatched["prompt_template"] == "inspect ."
|
|
assert "inputs" not in dispatched
|
|
|
|
|
|
async def test_an_empty_skills_list_is_written_not_blank():
|
|
"""``skills: []`` means "no skills at all" -- a written instruction.
|
|
|
|
Treating it as an invitation would let a caller quietly widen what a step may
|
|
reach, which is the opposite of what the author asked for.
|
|
"""
|
|
ex = PlaybookExecutor(dag_tool=FakeDagTool())
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="pull",
|
|
subagent="data-raven",
|
|
node_summary="pull the week's feedback with no extra skills",
|
|
prompt_template="go",
|
|
skills=[],
|
|
)
|
|
]
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"}, fills={"pull": {"skills": ["anything"]}})
|
|
|
|
assert plan.kind == "questions"
|
|
assert "already specifies" in plan.reply
|
|
|
|
|
|
async def test_executor_dag_hands_nodes_to_the_graph_tool_with_params_filled():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
|
|
plan = await ex.execute(_dag_spec(), params={"week_of": "2026-08-11"})
|
|
|
|
assert plan.kind == "dag"
|
|
# Node ids carry the run-unique namespace; the plain author ids survive as
|
|
# the last dash segment.
|
|
nodes = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
|
|
assert "2026-08-11" in nodes["pull"]["prompt_template"]
|
|
assert "PM" in nodes["report"]["prompt_template"] # default filled
|
|
# Runtime refs stay runtime refs, retargeted to the namespaced id.
|
|
assert "{{ %s.output_path }}" % nodes["pull"]["id"] in nodes["report"]["prompt_template"]
|
|
assert nodes["report"]["depends_on"] == [nodes["pull"]["id"]]
|
|
# The agent the author named reaches the graph tool as the node's own field:
|
|
# no backend is built here and no synthetic per-node agent name is invented.
|
|
assert nodes["pull"]["subagent"] == "data-raven"
|
|
assert nodes["report"]["subagent"] == "content-raven"
|
|
# Skills use the graph tool's native three-valued field: a list narrows the
|
|
# sub-agent's menu, while an omitted field keeps that agent's own menu.
|
|
assert nodes["pull"]["skills"] == ["sql-queries"]
|
|
assert "skills" not in nodes["report"]
|
|
assert "sql-queries" not in nodes["pull"]["prompt_template"]
|
|
assert "skills" not in nodes["report"]["prompt_template"].lower()
|
|
# mcps is the one this change makes travel: a real field, not prose, because
|
|
# unlike skills the receiving side now has something to do with it.
|
|
assert nodes["report"]["mcps"] == ["slack"]
|
|
assert "mcps" not in nodes["pull"]
|
|
|
|
|
|
async def test_executor_does_not_promote_multiline_text_to_a_node_reference():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
literal = "use {{ source.output\n}} as ordinary text"
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="source",
|
|
subagent="data-raven",
|
|
node_summary="produce source material",
|
|
prompt_template="produce source material",
|
|
),
|
|
NodeSpec(
|
|
id="consume",
|
|
subagent="content-raven",
|
|
node_summary="keep multiline template text literal",
|
|
prompt_template=literal,
|
|
),
|
|
]
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"})
|
|
|
|
assert plan.kind == "dag"
|
|
nodes = {node["id"].rsplit("-", 1)[-1]: node for node in tool.calls[0]["nodes"]}
|
|
assert nodes["consume"]["prompt_template"] == literal
|
|
|
|
|
|
async def test_executor_forwards_inputs_and_namespaces_node_input_references():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="source",
|
|
subagent="data-raven",
|
|
node_summary="produce the source material",
|
|
prompt_template="produce source material",
|
|
),
|
|
NodeSpec(
|
|
id="consume",
|
|
subagent="content-raven",
|
|
node_summary="consume every declared input",
|
|
prompt_template=("source={{ inputs.source }} file={{ inputs.brief }} literal={{ inputs.audience }}"),
|
|
depends_on=["source"],
|
|
inputs={
|
|
"source": {"node": "source"},
|
|
"brief": {"file": "brief.md"},
|
|
"audience": "PM",
|
|
},
|
|
),
|
|
]
|
|
)
|
|
|
|
await ex.execute(spec, params={"week_of": "w"})
|
|
|
|
nodes = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
|
|
inputs = nodes["consume"]["inputs"]
|
|
assert inputs["source"] == {"node": nodes["source"]["id"]}
|
|
assert inputs["brief"] == {"file": "brief.md"}
|
|
assert inputs["audience"] == "PM"
|
|
assert spec.nodes[1].inputs["source"] == {"node": "source"}
|
|
|
|
|
|
async def test_executor_namespaces_the_instance_handle_per_run():
|
|
"""Two runs of one playbook must not pour their steps into one session."""
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="a",
|
|
subagent="data-raven",
|
|
node_summary="run the first step of the shared session",
|
|
prompt_template="one",
|
|
instance="shared",
|
|
),
|
|
NodeSpec(
|
|
id="b",
|
|
subagent="data-raven",
|
|
node_summary="run the second step of the shared session",
|
|
prompt_template="two",
|
|
depends_on=["a"],
|
|
instance="shared",
|
|
),
|
|
]
|
|
)
|
|
|
|
await ex.execute(spec, params={"week_of": "w"})
|
|
await ex.execute(spec, params={"week_of": "w"})
|
|
|
|
first = {n["instance"] for n in tool.calls[0]["nodes"]}
|
|
second = {n["instance"] for n in tool.calls[1]["nodes"]}
|
|
# One handle within a run (that is what the author asked for), a different
|
|
# one between runs.
|
|
assert len(first) == 1 and len(second) == 1
|
|
assert first != second
|
|
|
|
|
|
async def test_a_playbook_node_keeps_its_routes_select_and_guard():
|
|
"""``_dispatch`` rebuilds each node from a hand-written field list, and a field
|
|
missing from it is dropped without a word -- which is how ``skills`` and ``mcps``
|
|
were once lost.
|
|
|
|
Asserted on what the graph tool was handed, because that is the only place a
|
|
dropped field shows: the playbook still loads, validates and runs, as a graph
|
|
that never branches.
|
|
"""
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
playbook = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="triage",
|
|
subagent="data-raven",
|
|
node_summary="classify the issue",
|
|
prompt_template="classify the issue",
|
|
routes={"bug": "the issue is a defect", "other": "anything else"},
|
|
),
|
|
NodeSpec(
|
|
id="scan",
|
|
subagent="data-raven",
|
|
node_summary="list what the change could affect",
|
|
prompt_template="list what the change could affect",
|
|
routes={"api": "the public api is affected", "cli": "the command line is affected"},
|
|
select="many",
|
|
),
|
|
NodeSpec(
|
|
id="fix",
|
|
subagent="content-raven",
|
|
node_summary="fix the defect",
|
|
prompt_template="fix the defect",
|
|
depends_on=["triage", "scan"],
|
|
when="triage.bug and not scan.cli",
|
|
),
|
|
NodeSpec(
|
|
id="note",
|
|
subagent="content-raven",
|
|
node_summary="write the closing note",
|
|
prompt_template="write the closing note",
|
|
),
|
|
]
|
|
)
|
|
|
|
plan = await ex.execute(playbook, params={"week_of": "w"})
|
|
|
|
assert plan.kind == "dag"
|
|
nodes = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
|
|
triage, scan, fix, note = (nodes[name] for name in ("triage", "scan", "fix", "note"))
|
|
assert triage["routes"] == {"bug": "the issue is a defect", "other": "anything else"}
|
|
assert "select" not in triage and "when" not in triage
|
|
assert scan["routes"] == {"api": "the public api is affected", "cli": "the command line is affected"}
|
|
assert scan["select"] == "many"
|
|
assert fix["when"] == f"{triage['id']}.bug and not {scan['id']}.cli"
|
|
assert "routes" not in fix and "select" not in fix
|
|
assert not {"routes", "select", "when"} & set(note)
|
|
assert playbook.nodes[2].when == "triage.bug and not scan.cli"
|
|
|
|
|
|
async def test_executor_asks_the_graph_tool_to_confirm_unless_the_caller_already_did():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
|
|
await ex.execute(_dag_spec(), params={"week_of": "w"})
|
|
assert tool.calls[0]["confirm"] is True
|
|
|
|
await ex.execute(_dag_spec(), params={"week_of": "w"}, confirmed=True)
|
|
assert tool.calls[1]["confirm"] is False
|
|
|
|
|
|
async def test_executor_passes_the_playbooks_own_task_summary_to_the_graph_tool():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec()
|
|
|
|
await ex.execute(spec, params={"week_of": "w"})
|
|
|
|
assert tool.calls[0]["task_summary"] == spec.task_summary
|
|
assert tool.calls[0]["task_summary"] != spec.description
|
|
|
|
|
|
class ComposeProvider:
|
|
"""Scripted graph-composition responses."""
|
|
|
|
def __init__(self, payloads):
|
|
self._payloads = list(payloads)
|
|
self.calls = []
|
|
|
|
async def chat_with_retry(self, messages, tools=None, model=None, tool_choice=None, **_):
|
|
self.calls.append(messages)
|
|
|
|
payload = self._payloads.pop(0)
|
|
if isinstance(payload, LLMResponse):
|
|
return payload
|
|
|
|
class _TC:
|
|
def __init__(self, args):
|
|
self.name = "emit_graph"
|
|
self.arguments = args
|
|
|
|
class _R:
|
|
def __init__(self, args):
|
|
self.has_tool_calls = args is not None
|
|
self.tool_calls = [_TC(args)] if args is not None else []
|
|
|
|
return _R(payload)
|
|
|
|
|
|
def _prompt_spec(**over):
|
|
base = dict(
|
|
name="due-diligence",
|
|
description="run due diligence on a target company",
|
|
task_summary="scan the target company and summarize what was found",
|
|
mode="prompt",
|
|
triggers=Triggers(keywords=["due diligence"]),
|
|
params={"target": ParamSpec(required=True, description="which company is the target?")},
|
|
prompts="compose a two-layer graph for ${params.target}: a breadth scan, then a summary.",
|
|
)
|
|
base.update(over)
|
|
return PlaybookSpec(**base)
|
|
|
|
|
|
GOOD_GRAPH = {
|
|
"nodes": [
|
|
{
|
|
"id": "scan",
|
|
"subagent": "research-raven",
|
|
"nodeSummary": "breadth-scan the target company",
|
|
"promptTemplate": "breadth-scan AcmeAI",
|
|
},
|
|
{
|
|
"id": "sum",
|
|
"subagent": "content-raven",
|
|
"nodeSummary": "summarize the scan findings",
|
|
"promptTemplate": "summarize {{ scan.output }}",
|
|
"dependsOn": ["scan"],
|
|
},
|
|
]
|
|
}
|
|
|
|
|
|
async def test_prompt_mode_returns_the_guidance_to_a_caller_that_can_compose():
|
|
"""In a conversation the caller is a model, so composing for it is worse.
|
|
|
|
A private composition call sees the guidance text and a cached roster and
|
|
nothing else; the caller has the whole conversation, and a graph it builds
|
|
wrong comes back as an ordinary tool error it can fix. Flexibility is the only
|
|
thing prompt mode offers over dag mode -- pinning the graph at load time would
|
|
make it a worse dag mode.
|
|
"""
|
|
tool = FakeDagTool()
|
|
provider = ComposeProvider([json.dumps(GOOD_GRAPH)])
|
|
ex = PlaybookExecutor(dag_tool=tool, provider=provider)
|
|
|
|
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "guidance"
|
|
assert "AcmeAI" in plan.reply # params substituted into the guidance
|
|
assert "run_subagent_dag" in plan.reply # and it says where to take it
|
|
assert provider.calls == [] # no composition call spent
|
|
assert tool.calls == [] # and nothing dispatched behind the caller's back
|
|
|
|
|
|
async def test_composition_says_when_it_withheld_a_secret():
|
|
"""The path that changed a work order and said nothing.
|
|
|
|
The other two fills log it; this one dropped the names and dispatched a
|
|
normal-looking run, so an operator debugging a graph that could not do what
|
|
the file asked had no trace of why. All three go through one helper now, and
|
|
the message names which text to edit -- "a secret was withheld" is not
|
|
actionable without that.
|
|
"""
|
|
from loguru import logger as _logger
|
|
|
|
lines: list[str] = []
|
|
sink = _logger.add(lambda m: lines.append(str(m)), level="WARNING")
|
|
try:
|
|
provider = ComposeProvider([json.dumps(GOOD_GRAPH)])
|
|
ex = PlaybookExecutor(dag_tool=FakeDagTool(), provider=provider, compose_prompt_mode=True)
|
|
ex.set_agent_profiles(lambda: COMPOSE_PROFILES)
|
|
spec = _prompt_spec(
|
|
params={
|
|
"target": ParamSpec(required=True, description="which company?"),
|
|
"TOKEN": ParamSpec(type="secret", description="an api credential"),
|
|
},
|
|
prompts="scan ${params.target} using ${params.TOKEN}",
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"target": "AcmeAI", "TOKEN": "s3cr3t-value"})
|
|
finally:
|
|
_logger.remove(sink)
|
|
|
|
assert plan.kind == "dag", "withholding degrades the run, it does not stop it"
|
|
assert "s3cr3t-value" not in provider.calls[0][0]["content"], "the provider must not receive it"
|
|
text = "".join(lines)
|
|
assert "TOKEN" in text and "withheld" in text
|
|
assert "composition" in text, "the operator has to know which text to edit"
|
|
|
|
|
|
async def test_prompt_mode_composes_when_there_is_no_caller_to_hand_it_to():
|
|
"""The CLI's shape. Without it, ``raven playbook run`` could not run a
|
|
prompt-mode playbook at all -- there is no model in the room to compose."""
|
|
tool = FakeDagTool()
|
|
provider = ComposeProvider([json.dumps(GOOD_GRAPH)])
|
|
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
|
|
ex.set_agent_profiles(lambda: COMPOSE_PROFILES)
|
|
|
|
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert len(tool.calls) == 1
|
|
assert "AcmeAI" in provider.calls[0][0]["content"]
|
|
|
|
|
|
async def test_prompt_mode_capability_errors_enter_the_repair_loop():
|
|
bad = json.loads(json.dumps(GOOD_GRAPH))
|
|
bad["nodes"][0].update(
|
|
{
|
|
"instance": "worker",
|
|
"skills": ["web-search"],
|
|
"mcps": ["browser"],
|
|
"promptTemplate": "inspect {{ ref_path:/tmp/report.txt }}",
|
|
}
|
|
)
|
|
profiles = dict(COMPOSE_PROFILES)
|
|
profiles["research-raven"] = PlaybookAgentProfile(
|
|
description="external research agent",
|
|
stateful=False,
|
|
reads_local_files=False,
|
|
injectable_skills=False,
|
|
injectable_mcps=False,
|
|
)
|
|
source_calls = 0
|
|
|
|
def source():
|
|
nonlocal source_calls
|
|
source_calls += 1
|
|
return profiles
|
|
|
|
tool = FakeDagTool()
|
|
provider = ComposeProvider([json.dumps(bad), json.dumps(GOOD_GRAPH)])
|
|
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
|
|
ex.set_agent_profiles(source)
|
|
|
|
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert source_calls == 1
|
|
assert len(provider.calls) == 2
|
|
repair = provider.calls[1][-1]["content"]
|
|
assert "stateless agent" in repair
|
|
assert "cannot receive MCP injection" in repair
|
|
assert "passes local file paths" in repair
|
|
|
|
|
|
async def test_prompt_mode_repairs_injection_on_a_shared_instance_continuation():
|
|
bad = json.loads(json.dumps(GOOD_GRAPH))
|
|
bad["nodes"][0].update(instance="writer", skills=["research"])
|
|
bad["nodes"][1].update(subagent="research-raven", instance="writer", skills=["editing"])
|
|
|
|
corrected = json.loads(json.dumps(bad))
|
|
corrected["nodes"][1].pop("skills")
|
|
|
|
tool = FakeDagTool()
|
|
provider = ComposeProvider([json.dumps(bad), json.dumps(corrected)])
|
|
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
|
|
ex.set_agent_profiles(lambda: COMPOSE_PROFILES)
|
|
|
|
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert len(provider.calls) == 2
|
|
repair = provider.calls[1][-1]["content"]
|
|
assert "continuing instance 'writer'" in repair
|
|
assert "opened by node 'scan'" in repair
|
|
|
|
|
|
async def test_prompt_mode_allows_a_continuation_to_clear_its_mcp_grant():
|
|
graph = json.loads(json.dumps(GOOD_GRAPH))
|
|
graph["nodes"][0].update(instance="writer", mcps=["browser"])
|
|
graph["nodes"][1].update(subagent="research-raven", instance="writer", mcps=[])
|
|
profiles = dict(COMPOSE_PROFILES)
|
|
profiles["research-raven"] = PlaybookAgentProfile(
|
|
description="research",
|
|
stateful=True,
|
|
reads_local_files=True,
|
|
injectable_skills=True,
|
|
injectable_mcps=True,
|
|
)
|
|
tool = FakeDagTool()
|
|
provider = ComposeProvider([json.dumps(graph)])
|
|
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
|
|
ex.set_agent_profiles(lambda: profiles)
|
|
|
|
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert len(provider.calls) == 1
|
|
nodes = {node["id"].rsplit("-", 1)[-1]: node for node in tool.calls[0]["nodes"]}
|
|
assert nodes["scan"]["mcps"] == ["browser"]
|
|
assert nodes["sum"]["mcps"] == []
|
|
|
|
|
|
async def test_prompt_mode_bad_graph_gets_one_repair_then_degrades():
|
|
bad = {"nodes": [{"id": "a", "subagent": "research-raven", "promptTemplate": "use {{ ghost.output }}"}]}
|
|
provider = ComposeProvider([json.dumps(bad), json.dumps(bad)])
|
|
ex = PlaybookExecutor(dag_tool=FakeDagTool(), provider=provider, compose_prompt_mode=True)
|
|
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
|
|
assert plan.kind == "questions"
|
|
assert "Graph assembly" in plan.reply
|
|
assert len(provider.calls) == 2 # one repair round happened
|
|
|
|
|
|
async def test_prompt_mode_repairs_a_branch_error_and_dispatches_the_branch():
|
|
"""A composed graph goes through the same graph rules as a submitted one, so a
|
|
guard reading a node it does not depend on comes back to the composing model
|
|
with the rule's own words, and the corrected branch reaches the graph tool."""
|
|
decision = {
|
|
"id": "triage",
|
|
"subagent": "research-raven",
|
|
"nodeSummary": "classify the target company",
|
|
"promptTemplate": "classify AcmeAI",
|
|
"routes": {"risky": "the target looks risky", "fine": "the target looks fine"},
|
|
}
|
|
guarded = {
|
|
"id": "dig",
|
|
"subagent": "content-raven",
|
|
"nodeSummary": "dig into the risk",
|
|
"promptTemplate": "dig into the risk",
|
|
"when": "triage.risky",
|
|
}
|
|
bad = {"nodes": [decision, {**guarded, "dependsOn": []}]}
|
|
corrected = {"nodes": [decision, {**guarded, "dependsOn": ["triage"]}]}
|
|
tool = FakeDagTool()
|
|
provider = ComposeProvider([json.dumps(bad), json.dumps(corrected)])
|
|
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
|
|
ex.set_agent_profiles(lambda: COMPOSE_PROFILES)
|
|
|
|
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert len(provider.calls) == 2
|
|
assert "guard names 'triage', which is not in its depends_on" in provider.calls[1][-1]["content"]
|
|
nodes = {node["id"].rsplit("-", 1)[-1]: node for node in tool.calls[0]["nodes"]}
|
|
assert nodes["triage"]["routes"] == decision["routes"]
|
|
assert nodes["dig"]["when"] == f"{nodes['triage']['id']}.risky"
|
|
|
|
|
|
# --- What the generation and composition prompts tell a model a playbook can express.
|
|
|
|
|
|
def test_the_prompts_allow_a_branch_and_still_refuse_a_loop():
|
|
system = " ".join(SYSTEM_PROMPT.split())
|
|
assert "conditional fallbacks cannot be expressed" not in system
|
|
assert "other than a branch" in system
|
|
assert "a loop cannot be expressed" in system
|
|
assert "A conditional branch can: give a step routes, and guard what follows with when." in system
|
|
|
|
compose = " ".join(build_compose_prompt("assemble it", COMPOSE_PROFILES, []).split())
|
|
assert "no run-time branches" not in compose
|
|
assert "design no run-time loops (a conditional branch is expressible)" in compose
|
|
|
|
|
|
def test_the_compose_prompt_gives_a_branch_its_fields_and_the_one_rule_about_them():
|
|
"""A prompt that says a branch is expressible has to hand over the fields to write one
|
|
with, and the one rule a branch breaks when it is forgotten: a node that depends on
|
|
a decision names it in its guard."""
|
|
compose = " ".join(build_compose_prompt("assemble it", COMPOSE_PROFILES, []).split())
|
|
|
|
listed = {word.strip() for word in compose.split("camelCase fields:", 1)[-1].split(")", 1)[0].split("/")}
|
|
assert {"routes", "select", "when"} <= listed
|
|
assert (
|
|
"a node that depends on a decision node (one with routes) must name it in its when, "
|
|
"as <id>.<route> or <id>.done"
|
|
) in compose
|
|
|
|
|
|
def _shown_playbook(prompt: str) -> dict:
|
|
block = prompt.split("# Current playbook\n", 1)[1].split("\n\n# User feedback", 1)[0]
|
|
return json.loads(block)
|
|
|
|
|
|
def test_a_playbook_without_a_branch_is_shown_to_the_revise_round_as_it_always_was():
|
|
"""The revise prompt dumps every node field, nulls included. Three new optional
|
|
fields written as ``null`` on every node would change what a playbook with no
|
|
branch looks like to the model revising it, so they are left out until a node
|
|
sets one -- and the older optional fields keep their nulls."""
|
|
playbook = _dag_spec()
|
|
|
|
prompt = build_revise_prompt(playbook, "make it faster", COMPOSE_PROFILES)
|
|
|
|
as_before = playbook.model_dump(by_alias=True, exclude={"nodes": {"__all__": {"routes", "select", "when"}}})
|
|
shown = json.dumps(as_before, ensure_ascii=False, indent=2)
|
|
assert f"# Current playbook\n{shown}\n\n# User feedback\nmake it faster" in prompt
|
|
assert '"instance": null' in prompt and '"mcps": null' in prompt and '"skills": null' in prompt
|
|
assert not any(word in prompt for word in ('"routes"', '"select"', '"when"'))
|
|
|
|
|
|
def test_a_branching_playbook_shows_the_revise_round_its_routes_and_guards():
|
|
nodes = _branching_nodes()
|
|
nodes[0] = nodes[0].model_copy(update={"select": "many"})
|
|
nodes.append(
|
|
NodeSpec(id="note", subagent="content-raven", node_summary="write the closing note", prompt_template="note")
|
|
)
|
|
|
|
prompt = build_revise_prompt(_dag_spec(nodes=nodes), "make it faster", COMPOSE_PROFILES)
|
|
|
|
shown = {node["id"]: node for node in _shown_playbook(prompt)["nodes"]}
|
|
assert shown["triage"]["routes"] == {"bug": "the issue is a defect", "other": "anything else"}
|
|
assert shown["triage"]["select"] == "many"
|
|
assert "when" not in shown["triage"]
|
|
assert shown["fix"]["when"] == "triage.bug"
|
|
assert "routes" not in shown["fix"] and "select" not in shown["fix"]
|
|
assert not {"routes", "select", "when"} & set(shown["note"])
|
|
|
|
|
|
async def test_prompt_mode_provider_error_does_not_enter_content_repair():
|
|
classification = ErrorClassification(
|
|
"upstream_transport_failure",
|
|
retryable=True,
|
|
should_fallback=True,
|
|
)
|
|
response = LLMResponse(
|
|
content="upstream did not process the request",
|
|
finish_reason="error",
|
|
error_classification=classification,
|
|
)
|
|
provider = ComposeProvider([response, json.dumps(GOOD_GRAPH)])
|
|
ex = PlaybookExecutor(dag_tool=FakeDagTool(), provider=provider, compose_prompt_mode=True)
|
|
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "questions"
|
|
assert "upstream_transport_failure" in plan.reply
|
|
assert len(provider.calls) == 1
|
|
|
|
|
|
async def test_prompt_mode_missing_tool_does_not_enter_content_repair():
|
|
provider = ComposeProvider([None, json.dumps(GOOD_GRAPH)])
|
|
ex = PlaybookExecutor(dag_tool=FakeDagTool(), provider=provider, compose_prompt_mode=True)
|
|
plan = await ex.execute(_prompt_spec(), params={"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "questions"
|
|
assert "required_tool_missing" in plan.reply
|
|
assert len(provider.calls) == 1
|
|
|
|
|
|
# --- The spec's own mcpServers, on the way to the graph tool. `raven playbook
|
|
# --- run` wires a pre-flight that has heard of them; a conversation wires
|
|
# --- nothing, so without this hand-off the same file resolved every server it
|
|
# --- ships to "not configured" through `load_playbook` and worked on the CLI.
|
|
|
|
|
|
def _mcp_spec(**over):
|
|
return _dag_spec(
|
|
params={
|
|
"week_of": ParamSpec(type="string", required=True, description="which week should be analyzed?"),
|
|
"PG_PASSWORD": ParamSpec(type="secret", description="Postgres password"),
|
|
},
|
|
mcp_servers={
|
|
"local-pg": MCPServerConfig(
|
|
command="pg-mcp",
|
|
args=["--db", "analytics"],
|
|
env={"PGPASSWORD": "{{ params.PG_PASSWORD }}"},
|
|
)
|
|
},
|
|
**over,
|
|
)
|
|
|
|
|
|
async def test_a_playbooks_own_servers_reach_the_graph_tool_with_their_params_filled():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
|
|
plan = await ex.execute(_mcp_spec(), params={"week_of": "w", "PG_PASSWORD": "hunter2"})
|
|
|
|
assert plan.kind == "dag"
|
|
handed = tool.calls[0]["mcp_servers"]
|
|
assert sorted(handed) == ["local-pg"]
|
|
assert handed["local-pg"].env == {"PGPASSWORD": "hunter2"}
|
|
|
|
|
|
async def test_the_spec_still_holds_the_reference_after_a_dispatch():
|
|
# The file is the distribution unit: filling in place would bake one run's
|
|
# secret into the library the next run reads.
|
|
tool = FakeDagTool()
|
|
spec = _mcp_spec()
|
|
await PlaybookExecutor(dag_tool=tool).execute(spec, params={"week_of": "w", "PG_PASSWORD": "hunter2"})
|
|
|
|
assert spec.mcp_servers["local-pg"].env == {"PGPASSWORD": "{{ params.PG_PASSWORD }}"}
|
|
|
|
|
|
async def test_a_playbook_that_ships_no_servers_hands_over_nothing():
|
|
tool = FakeDagTool()
|
|
await PlaybookExecutor(dag_tool=tool).execute(_dag_spec(), params={"week_of": "w"})
|
|
|
|
assert tool.calls[0]["mcp_servers"] == {}
|
|
|
|
|
|
async def test_a_prompt_mode_playbooks_servers_reach_the_composed_graph_too():
|
|
# The graph comes from a model rather than the file, but the servers still
|
|
# come from the file -- and the CLI is the entry point that composes one.
|
|
provider = ComposeProvider([json.dumps(GOOD_GRAPH)])
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool, provider=provider, compose_prompt_mode=True)
|
|
spec = _prompt_spec(
|
|
mcp_servers={"local-pg": MCPServerConfig(command="pg-mcp", args=["--db", "analytics"])},
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert sorted(tool.calls[0]["mcp_servers"]) == ["local-pg"]
|
|
|
|
|
|
# ---------------------------------------------------------------- runtime
|
|
|
|
|
|
def _library(tmp_path: Path, specs) -> PlaybookStore:
|
|
store = PlaybookStore(tmp_path, builtin_root=tmp_path / "_no_builtin")
|
|
for spec in specs:
|
|
store.save(spec)
|
|
return store
|
|
|
|
|
|
def _runtime(tmp_path: Path, specs, *, dag_tool=None, disabled=(), router=None) -> PlaybookRuntime:
|
|
executor = PlaybookExecutor(dag_tool=dag_tool or FakeDagTool())
|
|
return PlaybookRuntime(
|
|
store=_library(tmp_path, specs),
|
|
executor=executor,
|
|
disabled=disabled,
|
|
router=router,
|
|
)
|
|
|
|
|
|
async def test_load_dispatches_a_complete_playbook_in_one_call(tmp_path: Path):
|
|
"""No gate call, no confirm round trip, no second tool call.
|
|
|
|
A playbook whose graph is fully specified is the common case, and the caller
|
|
should never see its node contents -- it decided from the description and the
|
|
engine did the rest.
|
|
"""
|
|
tool = FakeDagTool()
|
|
rt = _runtime(tmp_path, [_dag_spec()], dag_tool=tool)
|
|
|
|
plan = await rt.load("weekly-feedback", {"week_of": "2026-08-11"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert len(tool.calls) == 1
|
|
|
|
|
|
async def test_an_unknown_name_is_none_so_the_tool_can_name_the_alternatives(tmp_path: Path):
|
|
rt = _runtime(tmp_path, [_dag_spec()])
|
|
assert await rt.load("no-such-thing", {}) is None
|
|
|
|
|
|
async def test_the_gap_loop_is_bounded(tmp_path: Path):
|
|
""" "Cannot fill it -> ask again -> still cannot" is a loop a caller can spend a
|
|
whole turn in, one tool call per pass, with no progress. Past the bound it is
|
|
told to stop rather than asked again."""
|
|
tool = FakeDagTool()
|
|
rt = _runtime(tmp_path, [_dag_spec()], dag_tool=tool)
|
|
rt.set_context(channel="web", chat_id="c", session_key="web:c")
|
|
|
|
for _ in range(MAX_GAP_ROUNDS):
|
|
plan = await rt.load("weekly-feedback", {})
|
|
assert plan.kind == "gaps"
|
|
|
|
final = await rt.load("weekly-feedback", {})
|
|
assert final.kind == "questions"
|
|
assert "do not call this again" in final.reply
|
|
assert tool.calls == []
|
|
|
|
|
|
async def test_filling_the_gap_restores_the_budget(tmp_path: Path):
|
|
"""The bound is per stuck attempt, not per playbook for the session: a second,
|
|
genuinely new run of the same playbook must not start already exhausted."""
|
|
rt = _runtime(tmp_path, [_dag_spec()])
|
|
rt.set_context(channel="web", chat_id="c", session_key="web:c")
|
|
|
|
assert (await rt.load("weekly-feedback", {})).kind == "gaps"
|
|
assert (await rt.load("weekly-feedback", {"week_of": "w"})).kind == "dag"
|
|
# Budget reset by the dispatch, so a later stuck call gets its full allowance.
|
|
for _ in range(MAX_GAP_ROUNDS):
|
|
assert (await rt.load("weekly-feedback", {})).kind == "gaps"
|
|
|
|
|
|
async def test_a_disabled_playbook_is_invisible_to_the_model_and_runnable_by_hand(tmp_path: Path):
|
|
"""Disabling can only mean "not listed" now.
|
|
|
|
There is no passive matcher left to mute, so the enforcement is that the model
|
|
never sees it -- and cannot name it, since the enum comes from the same list.
|
|
An explicit ``raven playbook run`` is the user's own hand and still resolves it.
|
|
"""
|
|
rt = _runtime(tmp_path, [_dag_spec()], disabled=["weekly-feedback"])
|
|
|
|
assert rt.names() == []
|
|
assert rt.listing() == []
|
|
assert rt.empty is True
|
|
assert await rt.load("weekly-feedback", {"week_of": "w"}) is None
|
|
assert (await rt.load("weekly-feedback", {"week_of": "w"}, allow_disabled=True)).kind == "dag"
|
|
|
|
|
|
async def test_the_listing_carries_what_a_caller_cannot_guess(tmp_path: Path):
|
|
"""Deleting the gate deleted the only reader of the param table.
|
|
|
|
Without it in the description the caller can only guess key names, and a
|
|
guessed key is dropped silently and comes back as the same question.
|
|
"""
|
|
rt = _runtime(tmp_path, [_dag_spec()])
|
|
detail = dict(rt.listing())["weekly-feedback"]
|
|
|
|
assert "week_of" in detail
|
|
assert "which week should be analyzed?" in detail
|
|
assert "required" in detail
|
|
assert "default='PM'" in detail
|
|
|
|
|
|
async def test_the_listing_names_the_fields_left_blank(tmp_path: Path):
|
|
rt = _runtime(
|
|
tmp_path,
|
|
[
|
|
_dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="draft", subagent="content-raven", node_summary="draft the weekly report", prompt_template=""
|
|
)
|
|
]
|
|
)
|
|
],
|
|
)
|
|
detail = dict(rt.listing())["weekly-feedback"]
|
|
|
|
assert "left for you to fill" in detail
|
|
assert "draft.prompt_template" in detail
|
|
|
|
|
|
async def test_a_library_bigger_than_the_router_window_is_narrowed_but_fully_nameable(tmp_path: Path):
|
|
"""The two costs are handled differently on purpose.
|
|
|
|
A description plus a param table is the expensive part, so it is what gets
|
|
narrowed. A name is a handful of tokens, and an enum missing one would turn a
|
|
retrieval miss into "the model cannot reach it" even when the user just said
|
|
its name out loud.
|
|
"""
|
|
specs = [_dag_spec(name=f"book-{i}", triggers=Triggers(keywords=[f"topic{i}"])) for i in range(6)]
|
|
rt = _runtime(tmp_path, specs, router=RouterSizes(top_k=2))
|
|
|
|
assert len(rt.listing("please do topic4")) == 2
|
|
assert dict(rt.listing("please do topic4")) # ranked, and topic4 is in it
|
|
assert "book-4" in dict(rt.listing("please do topic4"))
|
|
assert len(rt.names()) == 6
|
|
|
|
|
|
async def test_prompt_mode_reaches_the_caller_as_guidance(tmp_path: Path):
|
|
tool = FakeDagTool()
|
|
rt = _runtime(tmp_path, [_prompt_spec()], dag_tool=tool)
|
|
|
|
plan = await rt.load("due-diligence", {"target": "AcmeAI"})
|
|
|
|
assert plan.kind == "guidance"
|
|
assert tool.calls == []
|
|
|
|
|
|
async def test_context_is_recorded_for_the_dispatch_address(tmp_path: Path):
|
|
"""A tool call arrives without an address, so the last one the loop set is
|
|
what the dispatch announces into."""
|
|
tool = FakeDagTool()
|
|
rt = _runtime(tmp_path, [_dag_spec()], dag_tool=tool)
|
|
|
|
rt.set_context(channel="web", chat_id="room-7", session_key="web:room-7")
|
|
await rt.load("weekly-feedback", {"week_of": "w"})
|
|
|
|
assert tool.context == ("web", "room-7", "web:room-7")
|
|
|
|
|
|
# --- Run-unique node ids: the runner holds ids unique per session, authors
|
|
# write plain stable ones, and the executor's rewrite bridges the two.
|
|
|
|
|
|
class SessionUniqueDagTool(FakeDagTool):
|
|
"""Rejects a node id it has already seen, like the real runner's
|
|
per-session registry."""
|
|
|
|
def __init__(self):
|
|
super().__init__()
|
|
self.seen = set()
|
|
|
|
async def run_with_roles(self, nodes, *, roles, role_capabilities, background=True):
|
|
clashes = [n["id"] for n in nodes if n["id"] in self.seen]
|
|
if clashes:
|
|
return "Error: node ids already used in this session: %s" % ", ".join(clashes)
|
|
self.seen.update(n["id"] for n in nodes)
|
|
return await super().run_with_roles(nodes, roles=roles, role_capabilities=role_capabilities)
|
|
|
|
|
|
async def test_same_playbook_runs_twice_in_one_session():
|
|
tool = SessionUniqueDagTool()
|
|
executor = PlaybookExecutor(dag_tool=tool)
|
|
|
|
first = await executor.execute(_dag_spec(), {"week_of": "2026-08-10"})
|
|
second = await executor.execute(_dag_spec(), {"week_of": "2026-08-17"})
|
|
|
|
assert first.kind == "dag"
|
|
assert second.kind == "dag"
|
|
assert len(tool.calls) == 2
|
|
|
|
|
|
async def test_node_ids_and_references_are_rewritten_in_step():
|
|
tool = FakeDagTool()
|
|
executor = PlaybookExecutor(dag_tool=tool)
|
|
|
|
plan = await executor.execute(_dag_spec(), {"week_of": "2026-08-10"})
|
|
|
|
assert plan.kind == "dag"
|
|
by_plain = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
|
|
pull, report = by_plain["pull"], by_plain["report"]
|
|
assert pull["id"] != "pull" and pull["id"].startswith("weekly-feedback-")
|
|
# One rewrite table per dispatch: both nodes carry the same run tag.
|
|
assert pull["id"].rsplit("-", 1)[0] == report["id"].rsplit("-", 1)[0]
|
|
assert report["depends_on"] == [pull["id"]]
|
|
assert ("{{ %s.output_path }}" % pull["id"]) in report["prompt_template"]
|
|
assert "{{ pull.output_path }}" not in report["prompt_template"]
|
|
# Params were already compiled in; namespacing must not disturb them or the
|
|
# author's prompt. Skills travel separately as a native DAG node field.
|
|
assert pull["prompt_template"].startswith("pull the feedback for 2026-08-10")
|
|
|
|
|
|
async def test_rewrite_leaves_foreign_references_alone():
|
|
tool = FakeDagTool()
|
|
executor = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="pull",
|
|
subagent="data-raven",
|
|
node_summary="read the summary file and an external reference",
|
|
prompt_template="read {{ ref:notes/summary.md }} then {{ elsewhere.output }}",
|
|
)
|
|
]
|
|
)
|
|
|
|
await executor.execute(spec, {"week_of": "2026-08-10"})
|
|
|
|
template = tool.calls[0]["nodes"][0]["prompt_template"]
|
|
assert "{{ ref:notes/summary.md }}" in template
|
|
# An id outside this graph is not ours to rewrite; the runner's own
|
|
# whitelist decides what happens to it.
|
|
assert "{{ elsewhere.output }}" in template
|
|
|
|
|
|
def test_a_namespaced_guard_names_the_namespaced_nodes():
|
|
"""Two runs of one playbook in one conversation: the second run's guard has to
|
|
follow the ids the namespace pass mints, or the graph is refused for naming a
|
|
node outside its depends_on -- the first run took the plain ids."""
|
|
nodes = [
|
|
NodeSpec(
|
|
id="r", subagent="x", node_summary="r", prompt_template="r", routes={"a": "a", "b": "b"}, depends_on=[]
|
|
),
|
|
NodeSpec(id="g", subagent="x", node_summary="g", prompt_template="g", depends_on=["r"], when="r.a"),
|
|
]
|
|
|
|
first = _namespace_run("pb", nodes)
|
|
second = _namespace_run("pb", nodes)
|
|
|
|
for run in (first, second):
|
|
router, guarded = run
|
|
assert guarded.depends_on == [router.id]
|
|
assert guarded.when == f"{router.id}.a"
|
|
assert router.routes == {"a": "a", "b": "b"}
|
|
assert first[0].id != second[0].id
|
|
assert first[0].id not in second[1].when
|
|
assert nodes[1].when == "r.a", "the playbook's own node is not rewritten in place"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("guard", "expected"),
|
|
[
|
|
("r.a", "{r}.a"),
|
|
("r.done", "{r}.done"),
|
|
("not (r.a or s.done)\tand\nr.b", "not ({r}.a or {s}.done)\tand\n{r}.b"),
|
|
# A route may be named like another node: only the part before the dot is an id.
|
|
("r.s and not s.r", "{r}.s and not {s}.r"),
|
|
# One id may be the front of another: each atom is read whole, not found by substring.
|
|
("rr.a or r.a", "{rr}.a or {r}.a"),
|
|
# An id this playbook does not have is not ours to rename; the graph tool's own checks name it.
|
|
("ghost.a", "ghost.a"),
|
|
(" r.a ", " {r}.a "),
|
|
(" ", " "),
|
|
("", ""),
|
|
(None, None),
|
|
],
|
|
)
|
|
def test_the_namespace_pass_moves_only_the_node_ids_inside_a_guard(guard, expected):
|
|
nodes = [
|
|
NodeSpec(id=name, subagent="x", node_summary=name, prompt_template=name, routes={"a": "a", "s": "s", "r": "r"})
|
|
for name in ("r", "s", "rr")
|
|
]
|
|
nodes.append(NodeSpec(id="g", subagent="x", node_summary="g", prompt_template="g", when=guard))
|
|
|
|
run = _namespace_run("pb", nodes)
|
|
|
|
minted = {original.id: renamed.id for original, renamed in zip(nodes, run, strict=True)}
|
|
assert run[-1].when == (expected.format(**minted) if expected is not None else None)
|
|
assert [node.routes for node in run[:3]] == [node.routes for node in nodes[:3]]
|
|
|
|
|
|
def test_the_namespace_pass_keeps_each_atom_of_a_repeated_route_to_its_own_node():
|
|
"""Both nodes declare a route `x`, so the route name says nothing about where an
|
|
atom ends: a rename splicing by the name alone would take the wrong span."""
|
|
nodes = [
|
|
NodeSpec(
|
|
id=name, subagent="x", node_summary=name, prompt_template=name, routes={"x": "x holds", "y": "y holds"}
|
|
)
|
|
for name in ("alpha", "beta")
|
|
]
|
|
nodes.append(NodeSpec(id="g", subagent="x", node_summary="g", prompt_template="g", when="alpha.x and beta.x"))
|
|
|
|
run = _namespace_run("pb", nodes)
|
|
|
|
minted = {original.id: renamed.id for original, renamed in zip(nodes, run, strict=True)}
|
|
assert run[-1].when == f"{minted['alpha']}.x and {minted['beta']}.x"
|
|
|
|
|
|
async def test_a_branching_playbook_survives_the_library_and_reaches_the_graph_tool(tmp_path: Path):
|
|
"""Saved, read back, validated and dispatched: the whole way a playbook travels,
|
|
with the guard following the node ids the dispatch mints."""
|
|
tool = FakeDagTool()
|
|
rt = _runtime(tmp_path, [_dag_spec(nodes=_branching_nodes())], dag_tool=tool)
|
|
|
|
plan = await rt.load("weekly-feedback", {"week_of": "w"})
|
|
|
|
assert plan.kind == "dag"
|
|
nodes = {n["id"].rsplit("-", 1)[-1]: n for n in tool.calls[0]["nodes"]}
|
|
triage, fix, reply = nodes["triage"], nodes["fix"], nodes["reply"]
|
|
assert triage["routes"] == {"bug": "the issue is a defect", "other": "anything else"}
|
|
assert fix["when"] == f"{triage['id']}.bug"
|
|
assert reply["when"] == f"{triage['id']}.done"
|
|
assert reply["depends_on"] == [triage["id"], fix["id"]]
|
|
|
|
|
|
class ValidatingDagTool(FakeDagTool):
|
|
"""Holds each dispatched graph to the checks the graph tool applies on submission."""
|
|
|
|
async def execute(self, nodes, task_summary="", **kwargs):
|
|
validate_and_order(parse_dag_spec({"task_summary": task_summary, "nodes": nodes}))
|
|
return await super().execute(nodes, task_summary=task_summary, **kwargs)
|
|
|
|
|
|
async def test_two_runs_of_a_branching_playbook_each_pass_the_graph_tools_own_checks(tmp_path: Path):
|
|
"""The second run in one conversation cannot reuse the plain ids the first took,
|
|
so its guard has to name its own nodes: a guard still naming the plain ones would
|
|
read a node outside its depends_on, and the graph tool would refuse the graph."""
|
|
tool = ValidatingDagTool()
|
|
rt = _runtime(tmp_path, [_dag_spec(nodes=_branching_nodes())], dag_tool=tool)
|
|
|
|
first = await rt.load("weekly-feedback", {"week_of": "w"})
|
|
second = await rt.load("weekly-feedback", {"week_of": "w"})
|
|
|
|
assert first.kind == second.kind == "dag"
|
|
first_ids, second_ids = ({node["id"] for node in call["nodes"]} for call in tool.calls)
|
|
assert first_ids.isdisjoint(second_ids)
|
|
for call in tool.calls:
|
|
nodes = {node["id"].rsplit("-", 1)[-1]: node for node in call["nodes"]}
|
|
assert nodes["fix"]["when"] == f"{nodes['triage']['id']}.bug"
|
|
|
|
|
|
async def test_an_empty_skills_list_is_forwarded_without_changing_the_prompt():
|
|
"""``skills: []`` hides the skill menu; it is not the same as omission."""
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="pull",
|
|
subagent="data-raven",
|
|
node_summary="pull the week's feedback with no extra skills",
|
|
prompt_template="go",
|
|
skills=[],
|
|
)
|
|
]
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert tool.calls[0]["nodes"][0]["prompt_template"] == "go"
|
|
assert tool.calls[0]["nodes"][0]["skills"] == []
|
|
|
|
|
|
async def test_mcps_are_passed_to_the_graph_tool_without_prompt_rewriting():
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="pull",
|
|
subagent="data-raven",
|
|
node_summary="pull data that needs a github mcp",
|
|
prompt_template="go",
|
|
mcps=["github"],
|
|
)
|
|
]
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "w"})
|
|
|
|
assert plan.kind == "dag"
|
|
node = tool.calls[0]["nodes"][0]
|
|
assert node["mcps"] == ["github"]
|
|
assert "github" not in node["prompt_template"]
|
|
|
|
|
|
async def test_the_dispatch_receipt_names_the_run() -> None:
|
|
"""A client restoring the card from history has no events, only this line.
|
|
|
|
It recovers the run from the result text, in the graph tool's own shape, so a
|
|
receipt naming only the playbook left a reopened conversation showing a
|
|
dispatch it could not connect to any run: no node states, no way back to the
|
|
graph. The model gets a handle on the run out of the same line.
|
|
"""
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
|
|
plan = await ex.execute(_dag_spec(), params={"week_of": "w"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert plan.reply.startswith("DAG "), plan.reply
|
|
assert "weekly-feedback" in plan.reply and "2 steps" in plan.reply
|
|
|
|
|
|
async def test_both_param_spellings_fill_from_the_one_mechanism():
|
|
"""``{{ params.x }}`` is the spelling an author reaches for inside an MCP
|
|
server definition, where ``{{ ... }}`` is already the placeholder family.
|
|
It means the same thing everywhere, so it is one substitution and not two.
|
|
"""
|
|
tool = FakeDagTool()
|
|
ex = PlaybookExecutor(dag_tool=tool)
|
|
spec = _dag_spec(
|
|
nodes=[
|
|
NodeSpec(
|
|
id="pull",
|
|
subagent="data-raven",
|
|
node_summary="pull the feedback",
|
|
prompt_template="pull ${params.week_of} for {{ params.audience }}",
|
|
)
|
|
]
|
|
)
|
|
|
|
plan = await ex.execute(spec, params={"week_of": "2026-08-11"})
|
|
|
|
assert plan.kind == "dag"
|
|
assert tool.calls[0]["nodes"][0]["prompt_template"] == "pull 2026-08-11 for PM"
|
|
|
|
|
|
# ── Stored credentials (raven/playbook/credentials.py) ─────────────────────────
|
|
|
|
|
|
def _secret_pg_spec():
|
|
return _dag_spec(
|
|
params={"PG_PASSWORD": ParamSpec(type="secret", required=True, description="the db password")},
|
|
mcp_servers={"local-pg": MCPServerConfig(command="pg-mcp", env={"PGPASSWORD": "{{ params.PG_PASSWORD }}"})},
|
|
)
|
|
|
|
|
|
async def test_a_stored_secret_stands_in_for_one_the_caller_did_not_supply(tmp_path, monkeypatch):
|
|
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
|
|
from raven.playbook.credentials import set_secret_param
|
|
|
|
spec = _secret_pg_spec()
|
|
set_secret_param(spec.name, "PG_PASSWORD", "from-the-tab")
|
|
tool = FakeDagTool()
|
|
plan = await PlaybookExecutor(dag_tool=tool).execute(spec, {})
|
|
|
|
assert plan.kind == "dag"
|
|
handed = tool.calls[0]["mcp_servers"]
|
|
assert handed["local-pg"].env == {"PGPASSWORD": "from-the-tab"}
|
|
assert tool.calls[0]["mcp_scope"] == "playbooks/" + spec.name
|
|
|
|
|
|
async def test_an_unset_secret_is_a_reminder_not_a_gap(tmp_path, monkeypatch):
|
|
# A gap is closed by the caller collecting the value in the conversation,
|
|
# which is the one path a secret may not take. The run goes ahead, the server
|
|
# the secret fills is handed over without it, and the reply says where to set it.
|
|
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
|
|
spec = _secret_pg_spec()
|
|
tool = FakeDagTool()
|
|
plan = await PlaybookExecutor(dag_tool=tool).execute(spec, {})
|
|
|
|
assert plan.kind == "dag"
|
|
assert "was not run" not in plan.reply
|
|
assert "secret param 'PG_PASSWORD' is not set on this machine" in plan.reply
|
|
assert f"raven playbook secret set {spec.name} PG_PASSWORD" in plan.reply
|
|
assert tool.calls[0]["mcp_servers"]["local-pg"].env == {}
|
|
|
|
|
|
async def test_a_supplied_secret_outranks_the_stored_one(tmp_path, monkeypatch):
|
|
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
|
|
from raven.playbook.credentials import set_secret_param
|
|
|
|
spec = _secret_pg_spec()
|
|
set_secret_param(spec.name, "PG_PASSWORD", "stored")
|
|
tool = FakeDagTool()
|
|
await PlaybookExecutor(dag_tool=tool).execute(spec, {"PG_PASSWORD": "explicit"})
|
|
|
|
assert tool.calls[0]["mcp_servers"]["local-pg"].env == {"PGPASSWORD": "explicit"}
|
|
|
|
|
|
async def test_a_secret_stored_after_the_dispatch_reaches_the_next_read(tmp_path, monkeypatch):
|
|
# The hand-off is re-rendered on every read, which is what lets a node
|
|
# that is continued after the credential landed dial with it.
|
|
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
|
|
from raven.playbook.credentials import set_secret_param
|
|
|
|
spec = _secret_pg_spec()
|
|
set_secret_param(spec.name, "PG_PASSWORD", "v1")
|
|
tool = FakeDagTool()
|
|
await PlaybookExecutor(dag_tool=tool).execute(spec, {})
|
|
assert tool.calls[0]["mcp_servers"]["local-pg"].env == {"PGPASSWORD": "v1"}
|
|
|
|
set_secret_param(spec.name, "PG_PASSWORD", "v2")
|
|
assert tool.calls[0]["render"]()["local-pg"].env == {"PGPASSWORD": "v2"}
|
|
|
|
|
|
async def test_a_stored_value_for_a_non_secret_param_is_not_read_back(tmp_path, monkeypatch):
|
|
monkeypatch.setenv("RAVEN_HOME", str(tmp_path))
|
|
from raven.playbook.credentials import set_secret_param
|
|
|
|
spec = _dag_spec(params={"topic": ParamSpec(type="string", required=True, description="what to scan")})
|
|
set_secret_param(spec.name, "topic", "smuggled")
|
|
tool = FakeDagTool()
|
|
plan = await PlaybookExecutor(dag_tool=tool).execute(spec, {})
|
|
|
|
assert plan.kind == "gaps"
|
|
assert "params.topic" in plan.reply
|
|
assert tool.calls == []
|
|
|
|
|
|
async def test_a_playbook_without_carried_servers_hands_over_no_scope():
|
|
tool = FakeDagTool()
|
|
await PlaybookExecutor(dag_tool=tool).execute(_dag_spec(), params={"week_of": "w"})
|
|
assert tool.calls[0]["mcp_scope"] is None
|
|
|
|
|
|
async def test_an_entry_that_already_asked_can_say_so_through_load(tmp_path: Path):
|
|
"""``confirmed`` reaches the gate through the runtime, not only the executor.
|
|
|
|
The parameter exists for "an entry point that *does* ask", and every such
|
|
entry arrives through ``load`` -- the one door the model's tool and the
|
|
command line both use. Without the relay the flag is reachable only by a
|
|
caller that already holds the executor, which no entry point does.
|
|
"""
|
|
tool = FakeDagTool()
|
|
rt = _runtime(tmp_path, [_dag_spec(confirm=True)], dag_tool=tool)
|
|
|
|
await rt.load("weekly-feedback", {"week_of": "w32"}, confirmed=True)
|
|
assert tool.calls[-1]["confirm"] is False, "a caller that asked is not asked again"
|
|
|
|
await rt.load("weekly-feedback", {"week_of": "w32"}, confirmed=False)
|
|
assert tool.calls[-1]["confirm"] is True, "and one that did not still meets the gate"
|