Raven-1/tests/test_skill_segment_builder.py
Handsome-wzw 7b45527b5b fix(tools): an array sent as its own JSON is still an array
## Summary

Reported as "deliver_files is broken". It is not: **the tool never ran.**

Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:

```
Error: Invalid parameters for tool 'deliver_files': files should be array
```

What the model sent was the array, spelled as a string:

```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```

`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.

## Type

- [x] Fix
- [ ] Feature

## The edge of the forgiving is the design

When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.

**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.

**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.

## Not just deliver_files

Six built-in tools declare an array or object parameter and were all exposed:

| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |

The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.

## Verification

```
make lint-python    ruff check + ruff format --check, 1850 files
make lint-imports   9 contracts kept, 0 broken
pytest tests/       19,699 passed, 7 failed (all pre-existing, see below)
```

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string

**End to end, through the real tool**, with the exact payload that failed five times in production:

```
files sent as:            str -> [{"path": ".../poster.png", "title": "..."}]
returned:                 Delivered 1 file(s): poster.png (72 B).
still "should be array":  False
registered in the store:  True
```

Twelve cases, five mutations, each killing at least one:

| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |

The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.

**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.

## Risk

- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert

## Related Issues

N/A

## Swept

`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.

- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.

Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).

## Deliberately not fixed

The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.

## Moved from !594

Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.

Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00

713 lines
26 KiB
Python

"""End-to-end tests for SkillsSegmentBuilder.
The builder owns the rewriter → router → pre-gate hydrate → gate →
post-gate hydrate → render pipeline. Tests use stub sources / clients
to exercise each stage without network or LLM calls.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from raven.agent.subagent.backends import AgentMeta
from raven.agent.subagent.builtin_agents import GENERIC_AGENT
from raven.context_engine.segments.skills import SkillsSegmentBuilder
from raven.contracts.assembled import TokenBudget
from raven.contracts.context import AssemblyContext
from raven.memory_engine.skill_forge import (
LLMGateFilter,
QueryRewriter,
SkillForgeRouter,
)
from raven.memory_engine.skill_forge.types import RouterHit
# ----------------------------------------------------------------------
# Stub doubles
# ----------------------------------------------------------------------
@dataclass
class _Resp:
content: str
finish_reason: str = "stop"
class _StubProvider:
def __init__(self, response: Any) -> None:
self._response = response
async def chat_with_retry(self, **_kwargs: Any) -> _Resp:
if isinstance(self._response, _Resp):
return self._response
return _Resp(content=str(self._response))
class _StubSource:
"""ForgeSkillSource that returns a hard-coded hit list."""
def __init__(self, name: str, hits: list[RouterHit], weight: float = 1.0) -> None:
self.name = name
self.weight = weight
self._hits = hits
async def search(
self,
query: str,
history: list[dict[str, Any]],
k: int,
) -> list[RouterHit]:
return list(self._hits[:k])
class _StubHubClient:
"""Records get / install calls and returns canned payloads."""
def __init__(self, payloads: dict[str, dict[str, Any]]) -> None:
self._payloads = payloads
self.get_calls: list[str] = []
self.install_calls: list[tuple[str, dict[str, Any] | None]] = []
async def get(self, skill_id: str) -> dict[str, Any]:
self.get_calls.append(skill_id)
return dict(self._payloads[skill_id])
async def install(
self,
skill_id: str,
*,
prefetched_meta: dict[str, Any] | None = None,
) -> dict[str, Any]:
self.install_calls.append((skill_id, prefetched_meta))
payload = dict(self._payloads[skill_id])
return {
"slug": payload.get("slug", skill_id),
"version": payload.get("version", "v0"),
"dir": payload.get("_dir", "/tmp/" + skill_id),
"scripts_dir": None,
"skill_md": payload.get("skill_md", ""),
}
def _hit(qid: str, name: str, body: str = "", **meta: Any) -> RouterHit:
source = qid.split("/", 1)[0]
meta.setdefault("source", source)
if source == "hub":
meta.setdefault("id", qid.split("/", 1)[1])
return RouterHit(
qualified_id=qid,
name=name,
content=body,
score=0.5,
meta=meta,
)
def _ctx(message: str) -> AssemblyContext:
return AssemblyContext(
session_key="s",
current_message=message,
media=None,
channel=None,
chat_id=None,
session_messages=[],
budget=TokenBudget(
context_length=200_000,
reserved_output=8000,
reserved_tools=4000,
reserved_system=4000,
available_history=184_000,
),
)
# ----------------------------------------------------------------------
# Baseline — no rewriter, no gate, no hub
# ----------------------------------------------------------------------
async def test_baseline_renders_local_hits() -> None:
src = _StubSource(
"local",
[
_hit("local/foo", "foo", body="body foo"),
_hit("local/bar", "bar", body="body bar"),
],
)
router = SkillForgeRouter([src])
builder = SkillsSegmentBuilder(router, skill_top_k=2)
seg = await builder.build(_ctx("anything"))
assert seg is not None
assert "# Skills" in seg.text
assert "foo" in seg.text and "bar" in seg.text
assert seg.meta["injected_skill_ids"] == ["local/foo", "local/bar"]
async def test_no_router_returns_empty_segment() -> None:
builder = SkillsSegmentBuilder(None)
seg = await builder.build(_ctx("q"))
assert seg.text == ""
assert seg.meta["injected_skill_ids"] == []
# ----------------------------------------------------------------------
# Rewriter stage
# ----------------------------------------------------------------------
async def test_rewriter_skip_short_circuits_segment() -> None:
src = _StubSource("local", [_hit("local/foo", "foo", body="b")])
router = SkillForgeRouter([src])
rewriter = QueryRewriter(_StubProvider(json.dumps({"need_retrieval": False})))
builder = SkillsSegmentBuilder(router, rewriter=rewriter)
seg = await builder.build(_ctx("hello there"))
assert seg.text == ""
assert seg.meta.get("rewriter_skipped") is True
assert seg.meta["injected_skill_ids"] == []
async def test_rewriter_rewrite_passes_through() -> None:
"""When rewriter returns a rewritten_query, the router should be
invoked with it (not the original)."""
received: list[str] = []
class _SpySource:
name = "local"
weight = 1.0
async def search(self, query, history, k): # noqa: D401
received.append(query)
return []
router = SkillForgeRouter([_SpySource()])
rewriter = QueryRewriter(
_StubProvider(
json.dumps(
{
"need_retrieval": True,
"rewritten_query": "pdf gen",
}
)
)
)
builder = SkillsSegmentBuilder(router, rewriter=rewriter)
await builder.build(_ctx("please generate me a pdf report"))
assert received == ["pdf gen"]
# ----------------------------------------------------------------------
# Pre-gate body hydrate (Hub)
# ----------------------------------------------------------------------
async def test_pre_gate_hydrate_fills_hub_body() -> None:
hub_hit = _hit("hub/abc", "Calendar", body="")
src = _StubSource("hub", [hub_hit])
hub_client = _StubHubClient(
{
"abc": {"name": "Calendar", "skill_md": "# Hub body content", "slug": "calendar", "version": "1.0"},
}
)
router = SkillForgeRouter([src])
builder = SkillsSegmentBuilder(router, skill_top_k=1, hub_client=hub_client)
seg = await builder.build(_ctx("schedule"))
# The hub body should appear post-hydrate.
assert "Hub body content" in seg.text
assert hub_client.get_calls == ["abc"]
async def test_pre_gate_skipped_when_no_hub_client() -> None:
"""Hub hit with empty content + no client → content stays empty,
render header still appears, body is just blank under the heading."""
src = _StubSource("hub", [_hit("hub/abc", "Calendar", body="")])
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), skill_top_k=1)
seg = await builder.build(_ctx("schedule"))
assert "Calendar" in seg.text
# No prefetch attempted, no body, no crash.
# ----------------------------------------------------------------------
# Gate stage
# ----------------------------------------------------------------------
async def test_gate_filters_pool_down_to_selected() -> None:
src = _StubSource(
"local",
[
_hit("local/keep", "keep", body="k"),
_hit("local/drop", "drop", body="d"),
],
)
gate = LLMGateFilter(
_StubProvider(json.dumps({"plan": "p", "skills": ["local/keep"]})),
max_select=2,
)
builder = SkillsSegmentBuilder(
SkillForgeRouter([src]),
gate=gate,
gate_pool_size=5,
)
seg = await builder.build(_ctx("task"))
assert seg.meta["injected_skill_ids"] == ["local/keep"]
assert "drop" not in seg.text
async def test_gate_empty_selection_yields_empty_segment() -> None:
src = _StubSource("local", [_hit("local/foo", "foo", body="x")])
gate = LLMGateFilter(
_StubProvider(json.dumps({"plan": "none fits", "skills": []})),
)
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), gate=gate)
seg = await builder.build(_ctx("task"))
assert seg.text == ""
assert seg.meta["injected_skill_ids"] == []
# ----------------------------------------------------------------------
# Post-gate refs hydrate
# ----------------------------------------------------------------------
async def test_post_gate_resolves_local_refs(tmp_path: Path) -> None:
skill_dir = tmp_path / "skill"
(skill_dir / "references").mkdir(parents=True)
(skill_dir / "references" / "x.md").write_text("ref body")
body = "Read {baseDir}/references/x.md."
src = _StubSource(
"local",
[
_hit(
"local/foo",
"foo",
body=body,
skill_dir=str(skill_dir),
)
],
)
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), skill_top_k=1)
seg = await builder.build(_ctx("anything"))
assert f"{skill_dir}/references/x.md" in seg.text
assert "{baseDir}" not in seg.text
async def test_post_gate_install_hub_passes_prefetched_meta(tmp_path: Path) -> None:
"""install() should receive prefetched_meta from the pre-gate get()
call — that's how we skip the redundant HTTP round-trip."""
hub_hit = _hit("hub/x1", "Hub Skill", body="")
src = _StubSource("hub", [hub_hit])
skill_dir = tmp_path / "x1"
skill_dir.mkdir()
hub_client = _StubHubClient(
{
"x1": {"name": "Hub Skill", "skill_md": "# Body", "slug": "x1", "version": "1.0", "_dir": str(skill_dir)},
}
)
builder = SkillsSegmentBuilder(
SkillForgeRouter([src]),
skill_top_k=1,
hub_client=hub_client,
)
await builder.build(_ctx("q"))
assert hub_client.get_calls == ["x1"]
assert len(hub_client.install_calls) == 1
sid, prefetched = hub_client.install_calls[0]
assert sid == "x1"
assert prefetched is not None
assert prefetched.get("skill_md") == "# Body"
# ----------------------------------------------------------------------
# Tool-names collection
# ----------------------------------------------------------------------
async def test_get_tool_names_extracts_from_openai_schema() -> None:
captured: list[list[str] | None] = []
class _CaptureGate(LLMGateFilter): # type: ignore[misc]
async def filter(self, task, candidates, available_tools=None, available_subagents=None): # type: ignore[override]
captured.append(available_tools)
return []
gate = _CaptureGate(_StubProvider(json.dumps({"plan": "", "skills": []})))
src = _StubSource("local", [_hit("local/a", "a")])
def tool_defs() -> list[dict]:
return [
{"type": "function", "function": {"name": "read_file"}},
{"type": "function", "function": {"name": "exec"}},
{"name": "flat_form"},
]
builder = SkillsSegmentBuilder(
SkillForgeRouter([src]),
gate=gate,
get_tool_definitions=tool_defs,
)
await builder.build(_ctx("task"))
assert captured == [["read_file", "exec", "flat_form"]]
# ----------------------------------------------------------------------
# Subagent-roster collection
# ----------------------------------------------------------------------
def _roster_capture() -> "tuple[list[str | None], type[LLMGateFilter]]":
captured: list[str | None] = []
class _CaptureGate(LLMGateFilter): # type: ignore[misc]
async def filter(self, task, candidates, available_tools=None, available_subagents=None): # type: ignore[override]
captured.append(available_subagents)
return []
return captured, _CaptureGate
async def _roster_seen_by_gate(list_subagents: Any) -> str | None:
captured, capture_gate = _roster_capture()
builder = SkillsSegmentBuilder(
SkillForgeRouter([_StubSource("local", [_hit("local/a", "a")])]),
gate=capture_gate(_StubProvider(json.dumps({"plan": "", "skills": []}))),
list_subagents=list_subagents,
)
await builder.build(_ctx("task"))
assert len(captured) == 1
return captured[0]
async def test_gate_sees_the_specialist_subagents() -> None:
roster = await _roster_seen_by_gate(
lambda: [
AgentMeta("Raven-Research", "reads live web pages and cites them", False, False, True),
AgentMeta("Raven-PPT", "turns a source document into a deck", False, True),
]
)
assert roster is not None
assert "Raven-Research" in roster
assert "reads live web pages and cites them" in roster
assert "Raven-PPT" in roster
async def test_generic_agent_stays_out_of_the_gate_roster() -> None:
"""It advertises no capability bias, so an overlap check that saw it would
read every skill as covered and empty the segment for good."""
roster = await _roster_seen_by_gate(
lambda: [
AgentMeta(GENERIC_AGENT, "the whole tool set and the whole skill catalogue", True, True, True),
AgentMeta("Raven-Research", "reads live web pages and cites them", False, False, True),
]
)
assert roster is not None
assert "the whole tool set and the whole skill catalogue" not in roster
assert "Raven-Research" in roster
async def test_no_roster_when_only_the_generic_agent_is_enabled() -> None:
roster = await _roster_seen_by_gate(lambda: [AgentMeta(GENERIC_AGENT, "no capability bias", True, True, True)])
assert roster is None
async def test_no_roster_without_subagent_awareness() -> None:
assert await _roster_seen_by_gate(None) is None
async def test_no_roster_when_the_lookup_raises() -> None:
"""A wiring gap must degrade to "do not gate on overlap" rather than to a
failed turn: the roster is a hint, and the segment is still valid without it."""
def _boom() -> list[AgentMeta]:
raise RuntimeError("no manager yet")
assert await _roster_seen_by_gate(_boom) is None
# ----------------------------------------------------------------------
# Safety policy — min_safety after detail hydrate, blocklist, install audit
# ----------------------------------------------------------------------
async def test_low_safety_hub_hit_dropped_after_hydrate() -> None:
"""A hub skill whose detail metadata carries a score_safety below the
bar must be dropped from the pool: never injected, never installed."""
src = _StubSource("hub", [_hit("hub/bad1", "tag-memory")])
hub = _StubHubClient({"bad1": {"skill_md": "evil body", "score_safety": 0.2}})
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), hub_client=hub)
seg = await builder.build(_ctx("remember this"))
assert "tag-memory" not in (seg.text or "")
assert "evil body" not in (seg.text or "")
assert hub.install_calls == []
assert seg.meta["injected_skill_ids"] == []
async def test_safe_hub_hit_still_injected() -> None:
src = _StubSource("hub", [_hit("hub/good1", "good-skill")])
hub = _StubHubClient({"good1": {"skill_md": "safe body", "score_safety": 0.93}})
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), hub_client=hub)
seg = await builder.build(_ctx("remember this"))
assert "good-skill" in seg.text
assert len(hub.install_calls) == 1
async def test_hub_hit_without_score_passes() -> None:
"""Deployments whose detail payload omits score_safety keep working."""
src = _StubSource("hub", [_hit("hub/nos1", "scoreless")])
hub = _StubHubClient({"nos1": {"skill_md": "body"}})
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), hub_client=hub)
seg = await builder.build(_ctx("remember this"))
assert "scoreless" in seg.text
async def test_blocklisted_hit_dropped_any_source() -> None:
"""Blocklist drops matching candidates from every source before the
gate, case-insensitively."""
local = _StubSource("local", [_hit("local/tm", "Tag-Memory", "local body")])
hub = _StubSource("hub", [_hit("hub/ok", "fine-skill")])
client = _StubHubClient({"ok": {"skill_md": "fine body", "score_safety": 0.9}})
builder = SkillsSegmentBuilder(
SkillForgeRouter([local, hub]),
hub_client=client,
blocklist=["tag-memory"],
)
seg = await builder.build(_ctx("remember this"))
assert "Tag-Memory" not in seg.text
assert "fine-skill" in seg.text
async def test_hub_hit_with_external_paths_dropped() -> None:
"""A hub skill whose body points at another product's home dotdir is
dropped at the same policy point as the safety bar."""
src = _StubSource("hub", [_hit("hub/tm1", "tag-memory")])
hub = _StubHubClient({"tm1": {"skill_md": "store at ~/.openclaw/tag-memory/db.sqlite", "score_safety": 0.9}})
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), hub_client=hub)
seg = await builder.build(_ctx("remember this"))
assert "tag-memory" not in (seg.text or "")
assert hub.install_calls == []
class _FlakyGetClient(_StubHubClient):
"""get() always fails transiently; install() would succeed if reached."""
async def get(self, skill_id: str) -> dict[str, Any]:
self.get_calls.append(skill_id)
raise RuntimeError("transient 503")
async def test_hub_hit_dropped_when_detail_fetch_fails() -> None:
"""A hub candidate whose detail fetch fails is unvetted and must be
dropped: leaving it in the pool lets the post-gate step call
install(prefetched_meta=None), whose internal re-fetch bypasses
SkillPolicy entirely."""
src = _StubSource("hub", [_hit("hub/flaky1", "flaky-skill")])
hub = _FlakyGetClient({"flaky1": {"skill_md": "body", "score_safety": 0.2, "slug": "flaky-skill"}})
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), hub_client=hub)
seg = await builder.build(_ctx("remember this"))
assert "flaky-skill" not in (seg.text or "")
assert hub.install_calls == []
async def test_hub_hit_without_vetted_meta_never_installs() -> None:
"""A hub hit that skipped the hydrate (content prefilled) carries no
vetted detail metadata; the install step must refuse rather than let
install() re-fetch unvetted."""
src = _StubSource("hub", [_hit("hub/pre1", "prefilled", "already has body")])
hub = _StubHubClient({"pre1": {"skill_md": "already has body", "score_safety": 0.9}})
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), hub_client=hub)
await builder.build(_ctx("remember this"))
assert hub.install_calls == []
async def test_hub_auto_install_writes_install_meta(tmp_path: Path) -> None:
skill_dir = tmp_path / "good-skill@v3"
skill_dir.mkdir()
src = _StubSource("hub", [_hit("hub/good1", "good-skill")])
hub = _StubHubClient(
{
"good1": {
"skill_md": "safe body",
"score_safety": 0.9,
"slug": "good-skill",
"version": "v3",
"_dir": str(skill_dir),
}
}
)
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), hub_client=hub)
await builder.build(_ctx("remember this"))
rec = json.loads((skill_dir / ".install-meta.json").read_text(encoding="utf-8"))
assert rec["slug"] == "good-skill"
assert rec["version"] == "v3"
assert rec["trigger"] == "auto_inject"
assert rec["installed_at"]
async def test_auto_install_off_keeps_body_skips_install(tmp_path: Path) -> None:
"""autoInstall=off skips only the bundle download: the body hydrated in
the pre-gate step still injects, and nothing is installed or audited."""
audit = tmp_path / "installs.jsonl"
src = _StubSource("hub", [_hit("hub/good1", "good-skill")])
hub = _StubHubClient({"good1": {"skill_md": "safe body", "score_safety": 0.9}})
builder = SkillsSegmentBuilder(
SkillForgeRouter([src]),
hub_client=hub,
auto_install="off",
install_audit_path=audit,
)
seg = await builder.build(_ctx("remember this"))
assert "safe body" in seg.text
assert hub.install_calls == []
assert not audit.exists()
async def test_hub_auto_install_appends_audit_record(tmp_path: Path) -> None:
audit = tmp_path / "installs.jsonl"
src = _StubSource("hub", [_hit("hub/good1", "good-skill")])
hub = _StubHubClient(
{"good1": {"skill_md": "safe body", "score_safety": 0.9, "slug": "good-skill", "version": "v3"}}
)
builder = SkillsSegmentBuilder(
SkillForgeRouter([src]),
hub_client=hub,
install_audit_path=audit,
)
await builder.build(_ctx("remember this"))
lines = audit.read_text(encoding="utf-8").strip().splitlines()
assert len(lines) == 1
rec = json.loads(lines[0])
assert rec["slug"] == "good-skill"
assert rec["trigger"] == "auto_inject"
assert rec["score_safety"] == 0.9
assert rec["ts"]
# ----------------------------------------------------------------------
# Delivery note -- the injected body's own ending is what it argues with
# ----------------------------------------------------------------------
def _defs(*names: str) -> Any:
return lambda: [{"type": "function", "function": {"name": n}} for n in names]
async def test_delivery_note_rides_an_injected_body_when_the_tool_is_offered() -> None:
"""The observed failure is a hub skill ending in ``print(path)`` winning over
the one sentence in deliver_files' description. The counter-line has to travel
with the body that carries the foreign convention."""
from raven.context_engine.segments import render
src = _StubSource("hub", [_hit("hub/ppt", "ppt", body="prs.save(f)\nprint(f)")])
builder = SkillsSegmentBuilder(
SkillForgeRouter([src]),
skill_top_k=1,
get_tool_definitions=_defs("read_file", "deliver_files"),
)
seg = await builder.build(_ctx("make me a deck"))
assert seg is not None
assert seg.text.startswith("# Skills")
assert seg.text.endswith(render.SKILL_DELIVERY_NOTE)
# Named as a conflict, not as a bare restatement: without the clause about
# where the other convention comes from, the model holds two rules and no
# reason to prefer this one.
assert "another product" in render.SKILL_DELIVERY_NOTE
async def test_no_delivery_note_where_the_tool_is_absent() -> None:
"""Pointing at a tool absent from the definitions cannot be followed."""
src = _StubSource("hub", [_hit("hub/ppt", "ppt", body="body")])
builder = SkillsSegmentBuilder(
SkillForgeRouter([src]),
skill_top_k=1,
get_tool_definitions=_defs("read_file", "message"),
)
seg = await builder.build(_ctx("make me a deck"))
assert seg is not None
assert "# Skills" in seg.text
assert "deliver_files" not in seg.text
async def test_no_delivery_note_without_tool_awareness() -> None:
"""``collect_tool_names`` returns None for a builder wired without tool
awareness. Content degrades to showing too much; this line degrades to
silence, because its whole content is "call this tool"."""
src = _StubSource("hub", [_hit("hub/ppt", "ppt", body="body")])
builder = SkillsSegmentBuilder(SkillForgeRouter([src]), skill_top_k=1)
seg = await builder.build(_ctx("make me a deck"))
assert seg is not None
assert "# Skills" in seg.text
assert "deliver_files" not in seg.text
async def test_no_delivery_note_when_nothing_was_injected() -> None:
"""An empty segment stays empty -- the note is an argument with a skill body,
so with no body there is nothing to argue with and nothing to pay for."""
builder = SkillsSegmentBuilder(
SkillForgeRouter([_StubSource("local", [])]),
get_tool_definitions=_defs("deliver_files"),
)
seg = await builder.build(_ctx("hi"))
assert seg is not None
assert seg.text == ""
async def test_the_note_follows_the_real_tool_on_every_channel(tmp_path: Path) -> None:
"""The join the stubs cannot make: the real tool in a real registry, driven
through the real _set_tool_context, read back through the real definitions.
Every surface receives the same delivery instruction."""
from raven.agent.loop.main import AgentLoop
from raven.agent.tools.deliver import DeliverFilesTool
from raven.agent.tools.deliverables import DeliverableStore
from raven.agent.tools.registry import ToolRegistry
from raven.context_engine.segments import render
from raven.contracts.tool import Tool
class _PlainTool(Tool):
"""A second tool proves channel filtering preserves the full schema."""
@property
def name(self) -> str:
return "read_file"
@property
def description(self) -> str:
return "read a file"
@property
def parameters(self) -> dict[str, Any]:
return {"type": "object", "properties": {}}
async def execute(self, **kwargs: Any) -> str:
return ""
workspace = tmp_path / "chanwork"
workspace.mkdir()
registry = ToolRegistry()
registry.register(_PlainTool())
registry.register(
DeliverFilesTool(
DeliverableStore(tmp_path / "deliverables.json"),
workspace=workspace,
allowed_dirs=(),
)
)
class _Loop:
tools = registry
_playbooks = None
builder = SkillsSegmentBuilder(
SkillForgeRouter([_StubSource("hub", [_hit("hub/ppt", "ppt", body="body")])]),
skill_top_k=1,
get_tool_definitions=registry.get_definitions,
)
AgentLoop._set_tool_context(_Loop(), "web", "default", None, session_key="web:s1")
assert "deliver_files" in (await builder.build(_ctx("deck"))).text
AgentLoop._set_tool_context(_Loop(), "telegram", "c1", None, session_key="telegram:c1")
assert render.collect_tool_names(registry.get_definitions) == ["read_file", "deliver_files"]
assert "deliver_files" in (await builder.build(_ctx("deck"))).text