Raven-1/tests/test_a2a_executor.py
Dizhan Xue 7faa2060e2
feat(*): raven speaks Agent2Agent 1.0, inbound and outbound (#446)
## Summary

Raven speaks the Agent2Agent protocol, version 1.0, in both directions.
The host
agent is the peer: nothing here makes a sub-agent addressable from
outside, and a
raven launched as a sub-agent refuses to serve the face at all, because
it is
already reached over ACP by the host that started it.

Inbound is an aiohttp face carrying the two cards and a JSON-RPC
endpoint. It
mounts on the gateway when `a2a.server.enabled` is set, and `raven a2a
serve` runs
it standalone for a headless deployment, where running the command is
itself the
opt-in. Outbound is a single tool, `a2a_send`, that takes a card URL and
a message;
it enters the tool list only when `a2a.peers` is non-empty, so a host
with nowhere
to send a message does not carry the schema. The two switches are
deliberately
independent: calling out and being called are separate deployments.

Onboarding mints `a2a.server.token` and stops there. The face ships
closed:
finishing an unrelated install is not consent to open a second network
surface, so
`raven a2a enable` is what opens it, and it is the only writer of that
flag.
Enabling mints a missing token in the same command, because a face
switched on with
an empty token answers every caller with a 401 and reads as a broken
deployment
rather than the closed door it is. `raven a2a disable` closes it again
and keeps the
token, so reopening does not re-issue a credential callers already hold.
The wizard
never writes the flag at all, not even as false, so a repeat onboard
leaves a face
the operator opened by hand alone. A config nobody onboarded, and any
`A2aConfig()`
assembled in process, stays inert.

What that opens is narrower than a network face. The app carrying it
binds loopback
at both call sites, hardcoded rather than read from config, and every
request is
checked against a freshly minted 256-bit bearer token. The minting write
is also
what fixes `config.json` to owner-only, from the temporary file's first
byte rather
than with a chmod afterwards; a later enable or disable adds no secret
material, so
it leaves whatever mode the operator chose. Nothing else narrows that
file and it
already held provider API keys, so the obligation predates this work and
is merely
discharged here.

There are two cards because the protocol splits one description across
two
authentication states. The public card answers an unauthenticated GET,
and has to:
a caller reads it to learn which scheme to authenticate with, so
requiring the
credential first would be circular. It therefore names no installed
agent. The
extended card is a `GetExtendedAgentCard` RPC riding the same bearer
check as every
other method, and is the only one of the two that may say what this host
can
actually do.

On that extended card the sub-agent roster rides
`capabilities.extensions`, not
`skills`. A skill is the protocol's word for something an agent can be
asked to do,
and a peer cannot ask for one named sub-agent: it can only send a
message to this
host, which then decides what to dispatch. One skill per sub-agent would
advertise
call targets that do not exist. What a peer can genuinely request is the
orchestration, so that is a single skill, and the roster moves to the
protocol's
own extension point under a versioned URI. That is the only place a
conformant card
may carry a payload the spec does not define, because `AgentCard` is a
closed set
of fourteen protobuf fields: a custom top-level key is rejected by a
strict parser
and silently dropped by a lenient one. The entry declares `required`
false, so a
reader that has never heard of the URI ignores it and still talks to
this host.

Both cards derive `supportedInterfaces[].url` from the origin the caller
arrived
on. One face answers under every name that routes to it, and a URL fixed
when the
face is mounted is right for at most one of them. This matters more than
it looks:
a caller that finds an interface off the card's own origin must refuse
it, since a
card fetched from a trusted origin would otherwise redirect the
credential
elsewhere, and a relative URL reads as off-origin to that check.

## Type

- [x] Feature

## Verification

- `make test-python` at this head: 23729 passed, 112 skipped, 1 failed,
in 288 s.
  The one failure is

`tests/test_install_script.py::test_resolve_node_dir_answers_each_case_it_exists_for`,
which fails with the same assertion on `origin/main` by itself; this
branch changes
  no install or node file.
- `uv run --frozen --all-extras pytest
tests/integration/test_a2a_protocol_e2e.py -q`:
6 passed. That directory sits outside the default pytest scope, so it is
run by
  hand rather than reached by the line above.
- `make lint-imports`: 10 contracts kept, 0 broken. `make lint-types`:
all checks
  passed.
- `uv run ruff check raven/ tests/`: clean. `ruff format --check`:
clean.
- `make check-commits` over all 68 commits, `make
check-source-language`,
  `make check-large-files`: all exit 0.
- Live, over a real network boundary rather than in process: a raven in
a Docker
container and a raven on the host, each running this branch, called each
other in
both directions. Nineteen assertions per direction covering both cards,
the four
refusal paths, and a completed turn. The container publishes its face on
a
different port than it binds, so the card has to name the port the
caller used and
not the socket the server opened; both cards did, in both directions.
That run
predates the default-off change described above, which touches no file
under
`raven/a2a/` or `raven/a2a_client/`: both packages are byte-identical to
the head
  it ran against.
- Reverting each fix turns its own tests red, checked one fix at a time
rather than
all together, including the interface-URL fix mutated out inside the
running
  container.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

Behaviour changes for an operator who runs `raven onboard` after this
lands: a
token is minted into `a2a.server.token`, and `config.json` is narrowed
to
owner-only. No listening face is opened. Opening one takes `raven a2a
enable` and a
gateway restart, so the second network surface only ever exists because
somebody
typed for it.

Rolling an opened face back is `raven a2a disable` and a restart. The
token stays in
place and is never rotated, so callers already holding it keep working
when the face
is reopened, and a later onboard will not reopen it by itself. Nothing
else in the
install changes.

Costs and limits, stated rather than left to be discovered:

- A gateway that serves A2A imports `a2a-sdk`, measured at roughly 570
ms and 391
modules on top of the gateway's own stack, sqlalchemy and the
OpenTelemetry api
among them. It is paid once at mount, not per request, and an operator
who
switches the face off pays none of it. Moving it to the first request is
recorded
  as deferred in the design note rather than done here.
- `a2a_send` adds 157 tokens to the tool array of every turn, measured
off a
  captured provider call, and only on a host with a peer configured.
- The task store is in memory and unbounded, and inbound concurrency is
not capped.
A local face behind a bearer token is the deployment this is written
for; a
  host exposed more widely should bound both first.
- Pulling in `a2a-sdk` brings the OpenTelemetry api into the gateway's
import
graph. This is a trade accepted for the SDK's protocol types and
handler, not an
  adoption of OpenTelemetry by this project.

Checked during review and deliberately not fixed here: config validation
accepts an
agent row with an empty or whitespace name, which would reach the
extended card's
roster as an empty entry. It is not introduced by this work, the
previous shape
carried the same value, and closing it belongs with the agent schema
rather than
with the card.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-20 14:58:15 +08:00

279 lines
10 KiB
Python

"""The executor turns an A2A task into one raven turn, and leaks nothing when it raises."""
import asyncio
import pytest
from raven.a2a.executor import RavenAgentExecutor
class FakeQueue:
"""Stands in for EventQueue. `enqueue_event` is a coroutine on the real one."""
def __init__(self):
self.events = []
async def enqueue_event(self, event):
self.events.append(event)
class FakeContext:
"""Stands in for RequestContext, which exposes get_user_input(), not an attribute."""
def __init__(self, text="do the thing"):
self._text = text
self.task_id = "task-1"
self.context_id = "ctx-1"
def get_user_input(self, delimiter="\n"):
return self._text
async def test_a_completed_turn_enqueues_its_answer():
from a2a.types import Task, TaskState
async def run_turn(prompt, *, conversation_id, broker):
assert prompt == "do the thing"
return "the answer"
queue = FakeQueue()
await RavenAgentExecutor(run_turn).execute(FakeContext(), queue)
# The SDK's task consumer rejects a status update for a task it has never seen a
# Task object for, so the first event must be a bare submitted-state Task.
assert isinstance(queue.events[0], Task)
states = [e.status.state for e in queue.events]
assert states == [
TaskState.TASK_STATE_SUBMITTED,
TaskState.TASK_STATE_WORKING,
TaskState.TASK_STATE_COMPLETED,
]
assert queue.events[-1].status.message.parts[0].text == "the answer"
async def test_a_failed_turn_reports_the_failed_state():
from a2a.types import TaskState
async def run_turn(prompt, *, conversation_id, broker):
raise RuntimeError("boom")
queue = FakeQueue()
await RavenAgentExecutor(run_turn).execute(FakeContext(), queue)
assert queue.events[-1].status.state == TaskState.TASK_STATE_FAILED
async def test_a_raising_turn_does_not_put_the_traceback_on_the_wire():
async def run_turn(prompt, *, conversation_id, broker):
raise RuntimeError("/srv/secret/path.py exploded with API_KEY=abc123")
queue = FakeQueue()
await RavenAgentExecutor(run_turn).execute(FakeContext(), queue)
wire = " ".join(str(e) for e in queue.events)
assert "/srv/secret/path.py" not in wire
assert "abc123" not in wire
assert "Traceback" not in wire
async def _drain_until(condition, attempts=50):
"""Yield to the loop until `condition()` holds, or fail after `attempts` ticks.
The "question" status is enqueued from a task scheduled inside `on_park`
(see executor.py), not awaited directly by `execute`, so a test cannot
assume it has landed after a fixed number of bare `asyncio.sleep(0)` calls.
"""
for _ in range(attempts):
if condition():
return
await asyncio.sleep(0)
raise AssertionError("condition never became true")
async def test_the_conversation_id_reaches_run_turn_and_differs_from_the_task_id():
"""`execute` mints a raven conversation id per task and passes it to `run_turn`
as a keyword -- that id, not the A2A task id, is what `AskUserTool` actually
parks futures under (see `ask_user.py`'s `set_context`).
"""
seen: dict[str, str] = {}
async def run_turn(prompt, *, conversation_id, broker):
seen["conversation_id"] = conversation_id
return "ok"
await RavenAgentExecutor(run_turn).execute(FakeContext(), FakeQueue())
assert seen["conversation_id"], "run_turn must receive a non-empty conversation id"
assert seen["conversation_id"] != "task-1"
async def test_a_turn_that_asks_parks_then_completes_once_answered():
from a2a.types import TaskState
async def run_turn(prompt, *, conversation_id, broker):
# The broker and conversation id arrive as the same kwargs `execute`
# injects into the real turn path -- nothing here reaches into the
# executor's private bookkeeping to find them.
choice = await broker.await_question(conversation_id, prompt="which one?")
return f"chose: {choice}"
executor = RavenAgentExecutor(run_turn)
queue = FakeQueue()
turn = asyncio.create_task(executor.execute(FakeContext(), queue))
await _drain_until(lambda: len(queue.events) >= 3)
states = [e.status.state for e in queue.events]
assert states == [
TaskState.TASK_STATE_SUBMITTED,
TaskState.TASK_STATE_WORKING,
TaskState.TASK_STATE_INPUT_REQUIRED,
]
assert not turn.done()
# `answer()` is keyed by the task id -- the id a caller resuming a task
# actually has -- and translates it to the conversation id the broker
# parked the future under.
assert executor.answer("task-1", "the second one") is True
await turn
assert queue.events[-1].status.state == TaskState.TASK_STATE_COMPLETED
assert queue.events[-1].status.message.parts[0].text == "chose: the second one"
async def test_answer_on_an_unknown_task_id_reports_that_it_did_nothing():
async def run_turn(prompt, *, conversation_id, broker):
return "unused"
assert RavenAgentExecutor(run_turn).answer("no-such-task", "hi") is False
async def test_two_parked_tasks_are_tracked_and_answered_independently():
"""One broker serves both, keyed per conversation -- answering one
concurrently-parked task must not resolve or evict the other one.
"""
from a2a.types import TaskState
async def run_turn(prompt, *, conversation_id, broker):
choice = await broker.await_question(conversation_id, prompt="which one?")
return f"chose: {choice}"
executor = RavenAgentExecutor(run_turn)
ctx_a, ctx_b = FakeContext("first"), FakeContext("second")
ctx_a.task_id, ctx_b.task_id = "task-1", "task-2"
queue_a, queue_b = FakeQueue(), FakeQueue()
turn_a = asyncio.create_task(executor.execute(ctx_a, queue_a))
turn_b = asyncio.create_task(executor.execute(ctx_b, queue_b))
await _drain_until(lambda: len(queue_a.events) >= 3 and len(queue_b.events) >= 3)
assert set(executor._conversation_ids) == {"task-1", "task-2"}
assert executor.answer("task-1", "first answer") is True
await turn_a
assert queue_a.events[-1].status.state == TaskState.TASK_STATE_COMPLETED
assert queue_a.events[-1].status.message.parts[0].text == "chose: first answer"
# task-2 is untouched: still parked, still reachable, not swept away by
# task-1's completion popping its own entry out of the shared dict.
assert set(executor._conversation_ids) == {"task-2"}
assert not turn_b.done()
assert executor.answer("task-2", "second answer") is True
await turn_b
assert queue_b.events[-1].status.state == TaskState.TASK_STATE_COMPLETED
assert queue_b.events[-1].status.message.parts[0].text == "chose: second answer"
assert executor._conversation_ids == {}
assert executor._turns == {}
async def test_every_task_is_handed_the_same_broker():
"""One broker instance for the whole executor, not one per task.
`AskUserTool`'s broker is a per-process slot that the turn path installs
into, so a per-task broker is overwritten by the next task to start, and the
earlier task's question then parks in an object `answer` no longer looks in:
its resume does nothing and the turn falls through to the ask's default.
Measured before this invariant held -- injecting a single suspension between
the install and the park made two concurrent tasks both answer "(no answer)".
"""
seen = []
async def run_turn(prompt, *, conversation_id, broker):
seen.append(broker)
return "ok"
executor = RavenAgentExecutor(run_turn)
second = FakeContext()
second.task_id = "task-2"
await executor.execute(FakeContext(), FakeQueue())
await executor.execute(second, FakeQueue())
assert len(seen) == 2
assert seen[0] is seen[1], "a per-task broker loses the shared ask_user slot race"
async def test_a_second_message_on_a_running_task_is_refused():
"""One A2A task is one raven turn, and the SDK does not enforce that.
`DefaultRequestHandler` refuses only a TERMINAL task, so a second
`SendMessage` naming a task whose turn is still running would be handed
straight through and start a second turn for the same task id, with the two
turns' bookkeeping overwriting each other. The adapter refuses it instead.
"""
from a2a.utils.errors import InvalidRequestError
from raven.a2a.runtime import build_request_handler
from raven.config.schema import A2aConfig
started, release, calls = asyncio.Event(), asyncio.Event(), []
async def run_turn(prompt, *, conversation_id, broker):
calls.append(prompt)
started.set()
await release.wait()
return "done"
handler = build_request_handler(A2aConfig(), run_turn)
turn = asyncio.create_task(handler._handler.agent_executor.execute(FakeContext(), FakeQueue()))
await started.wait()
with pytest.raises(InvalidRequestError):
await handler.on_message_send(
{"message": {"role": "ROLE_USER", "parts": [{"text": "again"}], "messageId": "m2", "taskId": "task-1"}},
None,
)
release.set()
await turn
assert calls == ["do the thing"], "the refusal must not have started a second turn"
async def test_a_second_streaming_message_on_a_running_task_is_refused():
"""The streaming sibling needs the same guard, and its failure is quieter.
Without it the SDK queues the second `SendStreamingMessage` behind the
running task and drains it afterwards, so a full second raven turn starts
for one task id with no further action from the caller at all.
"""
from a2a.utils.errors import InvalidRequestError
from raven.a2a.runtime import build_request_handler
from raven.config.schema import A2aConfig
started, release, calls = asyncio.Event(), asyncio.Event(), []
async def run_turn(prompt, *, conversation_id, broker):
calls.append(prompt)
started.set()
await release.wait()
return "done"
handler = build_request_handler(A2aConfig(), run_turn)
turn = asyncio.create_task(handler._handler.agent_executor.execute(FakeContext(), FakeQueue()))
await started.wait()
stream = handler.on_message_send_stream(
{"message": {"role": "ROLE_USER", "parts": [{"text": "again"}], "messageId": "m2", "taskId": "task-1"}},
None,
)
with pytest.raises(InvalidRequestError):
await anext(stream)
release.set()
await turn
assert calls == ["do the thing"], "the refusal must not have started a second turn"