## Summary `user_memory/episodic/episodes.md` had no writer on any surface people use. Its only one, the annotator, runs inside the token-pressure compaction loop, and all three calls into that loop are gated on `owns_compaction`, which the single shipped context engine hardcodes to `True`. The annotator survived as live code with one caller: `/new` typed as message text in a chat channel, which the web and TUI "new session" buttons never send. Six readers read the empty file: the prompt line, two attention producers and three predictor modules. This gives the log a writer that does not depend on compaction. After a user turn, once `memory.annotateEveryTurns` user turns have accumulated since the last annotation, the unannotated tail goes to the same annotator, in the background, under the consolidator's per-session lock so it cannot interleave with `/new`. Key decisions, each with the alternative it rejected: - **Its own cursor, not `last_consolidated`.** The `owns_compaction` gate exists to keep a second writer off the Curator's pointer, and `undo_last_turn` reads that pointer as a floor. The cursor is `session.metadata["annotated_through"]`. - **Patched, not saved.** A detached task that calls `sessions.save` with its own copy of the metadata drops keys the next turn wrote; `append_metadata_patch` exists for exactly this writer. - **Seeded at the length before the first seen turn.** The old loop fed the annotator chunks bounded by a token budget; this step has no chunking, so an unseeded long session would go to the provider in one call. Seeding also covers forks, which copy messages but not metadata. Seeding at the *current* length instead would skip the first turn of every new session. - **Truncation moves the cursor where it happens.** A lazy clamp at the next turn leaves the stale index on disk and swallows every turn written after an undo or a clear until the count climbs past it. `Session.clear` drops the cursor and `Session.undo_last_turn` pulls it back to the cut, beside the same methods' handling of `last_consolidated`. - **Only `Origin.USER` turns.** Two of the file's readers are Sentinel producers; writing Sentinel's own turns into it would feed the layer its own output. `memory.annotateEveryTurns` ships at 0, which turns the step off: the log has had no writer for the life of the project, so no install is waiting on one, and nobody is billed for a side provider call to fill a file they may never read. Set it to 5 (or any positive cadence) to turn it on. ### The second commit, and why it is here Driving this on a live gateway showed the annotator could not write at all on the default model. It sends `tool_choice="required"`, and DeepSeek's thinking mode answers `HTTP 400: Thinking mode does not support this tool_choice`. The Responses adapter returned that as the call's result, so the annotator logged "LLM did not call annotate_conversation" and gave up -- correct behaviour, over a feature that could never work. Nine call sites force a tool choice today: two in the memory consolidator, three in the proactive layer, four in the playbook compiler. Each already handles a model that declines under `auto`. So the adapter now does what its Anthropic sibling already did: a 400 that names `tool_choice` while the body forces one relaxes it to `auto` and retries once. Measured against the live vendor three ways -- `required` answered 400, `auto` and omitted both produced the tool call. The two commits are separable: the provider repair stands on its own and fixes eight other latent call sites; without it the first commit is inert on this model. ## Type - [x] Feature ## Verification Unit, on this branch: - `uv run --frozen --python 3.12 pytest tests/test_agent_loop_episodic_annotate.py tests/test_providers_forced_tool_choice.py tests/test_session_manager.py tests/test_segments.py tests/test_agent_loop_backend_dispatch.py tests/test_context_engine_factory.py tests/test_providers*.py -q` -> 344 passed - Broader sweep over the loop, session, provider, context-engine, memory and playbook suites -> 2919 passed, 103 skipped, 1 failed: `test_agent_loop_token_budget.py::test_a_ceiling_as_large_as_the_window_still_leaves_room_for_history` asserts `reserved_tools < 7_500` and measures 7557. Pre-existing: a clean `github/main` worktree at `829eb28e1` fails it with the same 7557. This change touches no tool schema. - `make lint-types` -> All checks passed - `uv run --frozen --all-extras lint-imports` -> 10 contracts kept, 0 broken - `ruff check` / `ruff format --check` on every touched file -> clean - `scripts/check_source_language.py github/main...HEAD` -> exit 0 - `scripts/check_large_files.py github/main` -> exit 0 - commitlint per sha, with the repo config -> both ok Every new guard was mutation-verified: deleted, the suite run, the matching test confirmed red on its own symptom, restored, md5-compared. Nine guards, nine reds -- the knob clause, the origin check, the session lock, the cursor-on-decline return, the seed, the undo pull-back, the clear drop, and the provider repair's two halves. Live gateway, isolated `RAVEN_HOME`, real provider, `memory.backend = null`: | What | Result | |---|---| | 12-message pre-existing session, one turn | cursor seeded at 12, `episodes.md` 0 bytes | | two turns past the cursor | no annotator call, cursor unmoved | | three substantive turns | `annotate done for 6 messages -> 3 episode(s)`; three lines carrying a token minted at run time | | those three lines through `parse_episode_line` | all three parse, tagged `#project-<token>` the way `project_rhythm` ranks by | | `annotateEveryTurns: 0` | file unchanged, no annotator call, cursor not even seeded | | `session.undo` of two turns | `removed: 4`, cursor pulled 26 -> 22 in the record the undo wrote | | kill by pid, restart | cursor still 26 | | trivia-only turns | `annotate done for 8 messages -> 0 episode(s)`, cursor advanced -- process-only episodes are dropped by design | | identity prompt, both branches | no-backend unchanged; with a backend the log is named read-only and the profile path stays out | - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk User-visible changes: - Nothing changes for an install that leaves `annotateEveryTurns` at its default of 0: the step short-circuits before the cursor is seeded. An install that sets a positive cadence runs one annotator LLM call per that many user turns, plus a profile refresh when a tag crosses the hot threshold; with a memory backend wired that means two extractors over the same conversation. The knob is read at boot, so a change needs a gateway reload. - With a backend wired the identity prompt gains one read-only line naming the episodic log. - A forced tool choice that a vendor refuses now costs one extra request instead of failing the call. Callers that relied on the error are none: all nine already handle a declining model. Rollback: back to `memory.annotateEveryTurns: 0` disables the first commit's behaviour without reverting it. The provider repair only changes a path that previously ended in an error response. Not covered: a crash between the file write and the metadata patch re-annotates one tail, duplicating lines. Harmless to grep and over-counts a tag by one batch at most; making it atomic means writing the cursor into the same file under the same lock. ## Related Issues #122 --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
718 lines
31 KiB
Python
718 lines
31 KiB
Python
"""ContextAssembler — factory wiring + SkillForgeRouter assembly.
|
|
|
|
The factory no longer dispatches on ``context.engine`` — it always
|
|
builds the single :class:`ContextAssembler` from a flat SegmentBuilder
|
|
list, assembling the ``SkillsSegmentBuilder``'s SkillForgeRouter from:
|
|
|
|
- Local (always),
|
|
- Mass (when ``mass.endpoint`` is set),
|
|
- the memory backend (when one is supplied).
|
|
|
|
With no backend the engine still constructs (recall lane yields [],
|
|
router runs Local-only). AgentLoop always delegates skill selection to
|
|
the engine (``_uses_default_engine`` is always True), and
|
|
``_collect_injected_skill_ids`` prefers the assembled-metadata stash.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
from raven.agent.context import ContextBuilder
|
|
from raven.agent.loop import AgentLoop
|
|
from raven.agent.loop.bundles import EngineWiring, SubagentWiring, ToolWiring, TurnPolicy
|
|
from raven.agent.subagent.builtin_agents import GENERIC_AGENT
|
|
from raven.config.raven import (
|
|
ContextConfig,
|
|
HubSourceConfig,
|
|
MemoryConfig,
|
|
SkillForgeConfig,
|
|
SkillForgeRouterConfig,
|
|
)
|
|
from raven.config.schema import SubagentsConfig
|
|
from raven.context_engine import ContextAssembler
|
|
from raven.context_engine.factory import _build_router, build_context_engine
|
|
from raven.context_engine.segments import (
|
|
IdentitySegmentBuilder,
|
|
MemorySegmentBuilder,
|
|
SkillsSegmentBuilder,
|
|
)
|
|
from raven.context_engine.segments.curator import CuratorSegmentBuilder
|
|
from raven.contracts.assembled import TokenBudget
|
|
from raven.contracts.context import AssemblyContext
|
|
from raven.memory_engine.skill_forge import (
|
|
BackendSkillSource,
|
|
HubSkillSource,
|
|
LocalSkillSource,
|
|
)
|
|
from raven.providers.binding import ModelBinding, use_binding
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Test doubles
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class _FakeBackend:
|
|
async def start(self):
|
|
pass
|
|
|
|
async def stop(self):
|
|
pass
|
|
|
|
async def feedback(self, signals):
|
|
pass
|
|
|
|
async def store(self, session_id, messages, *, metadata=None):
|
|
pass
|
|
|
|
async def recall(self, query, *, user_id=None, agent_id=None, top_k):
|
|
return []
|
|
|
|
|
|
class _StubProvider:
|
|
api_key = "test"
|
|
|
|
def get_default_model(self) -> str:
|
|
return "stub"
|
|
|
|
async def chat(self, *args, **kwargs):
|
|
raise NotImplementedError
|
|
|
|
async def chat_with_retry(self, *args, **kwargs):
|
|
raise NotImplementedError
|
|
|
|
|
|
def _stub_get_defs() -> list[dict]:
|
|
return []
|
|
|
|
|
|
class _StubPool:
|
|
"""Pairs any pin it is asked for, so a test can assert which one was asked."""
|
|
|
|
def bind_pin(self, model, provider_name=None):
|
|
from raven.providers.binding import ModelBinding
|
|
|
|
return ModelBinding(MagicMock(name=provider_name or "stub-provider"), model) if model else None
|
|
|
|
|
|
def _point_live_config(tmp_path: Path, monkeypatch, payload: dict) -> Path:
|
|
"""Point ``LiveConfig``'s default path at a file this test owns."""
|
|
cfg = tmp_path / "config.json"
|
|
cfg.write_text(json.dumps(payload), encoding="utf-8")
|
|
monkeypatch.setattr("raven.config.loader.get_config_path", lambda: cfg)
|
|
return cfg
|
|
|
|
|
|
def _build_engine(
|
|
tmp_path: Path,
|
|
*,
|
|
backend=None,
|
|
hub_endpoint: str | None = None,
|
|
memory_config: MemoryConfig | None = None,
|
|
model: str = "stub",
|
|
skill_forge_config: SkillForgeConfig | None = None,
|
|
rrf_k: int | None = None,
|
|
provider_pool=None,
|
|
) -> ContextAssembler:
|
|
builder = ContextBuilder(workspace=tmp_path)
|
|
engine = build_context_engine(
|
|
provider_pool=provider_pool,
|
|
workspace=tmp_path,
|
|
config=ContextConfig(),
|
|
builder=builder,
|
|
provider=_StubProvider(),
|
|
model=model,
|
|
context_window_tokens=8192,
|
|
get_tool_definitions=_stub_get_defs,
|
|
backend=backend,
|
|
memory_config=memory_config or MemoryConfig(),
|
|
skill_forge_router_config=SkillForgeRouterConfig(
|
|
hub=HubSourceConfig(endpoint=hub_endpoint),
|
|
**({} if rrf_k is None else {"rrf_k": rrf_k}),
|
|
),
|
|
# Most assertions in this file exercise the push pipeline's wiring;
|
|
# pull-mode wiring has its own tests below.
|
|
skill_forge_config=skill_forge_config or SkillForgeConfig(discovery="push"),
|
|
)
|
|
assert isinstance(engine, ContextAssembler)
|
|
return engine
|
|
|
|
|
|
def _router_sources(engine: ContextAssembler):
|
|
skills = next(b for b in engine._builders if isinstance(b, SkillsSegmentBuilder))
|
|
return [type(s) for s in skills._router._sources], skills._router._sources
|
|
|
|
|
|
def _memory_builder(engine: ContextAssembler) -> MemorySegmentBuilder:
|
|
return next(b for b in engine._builders if isinstance(b, MemorySegmentBuilder))
|
|
|
|
|
|
def _curator_builder(engine: ContextAssembler) -> CuratorSegmentBuilder:
|
|
return next(b for b in engine._builders if isinstance(b, CuratorSegmentBuilder))
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Factory — always builds the assembler
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestFactory:
|
|
def test_returns_assembler_with_backend(self, tmp_path: Path) -> None:
|
|
assert isinstance(_build_engine(tmp_path, backend=_FakeBackend()), ContextAssembler)
|
|
|
|
def test_returns_assembler_without_backend(self, tmp_path: Path) -> None:
|
|
engine = _build_engine(tmp_path, backend=None)
|
|
assert isinstance(engine, ContextAssembler)
|
|
assert _memory_builder(engine)._backend is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# SF6: set_context_window cascades down to the Curator's trimmer -- a
|
|
# /model switch must not leave it budgeting against the pre-switch window.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestSetContextWindow:
|
|
def test_engine_cascades_into_the_curator_trimmer(self, tmp_path: Path) -> None:
|
|
engine = _build_engine(tmp_path)
|
|
curator = _curator_builder(engine)
|
|
assert curator.context_window_tokens == 8192
|
|
assert curator.assembler.context_window_tokens == 8192
|
|
assert curator.assembler.trimmer.context_window_tokens == 8192
|
|
|
|
engine.set_context_window(4096)
|
|
|
|
assert curator.context_window_tokens == 4096
|
|
assert curator.assembler.context_window_tokens == 4096
|
|
assert curator.assembler.trimmer.context_window_tokens == 4096
|
|
|
|
def test_engine_cascade_does_not_raise_for_builders_without_the_hook(self, tmp_path: Path) -> None:
|
|
"""seg1-5 carry no budget and have no ``set_context_window`` -- the
|
|
cascade must skip them rather than assume every builder has it."""
|
|
engine = _build_engine(tmp_path)
|
|
non_curator = [b for b in engine._builders if not isinstance(b, CuratorSegmentBuilder)]
|
|
assert non_curator, "fixture should include builders other than the Curator"
|
|
|
|
engine.set_context_window(4096) # must not raise
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# SkillForgeRouter assembly — which sources are present
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestSkillForgeRouterAssembly:
|
|
def test_local_source_always_present(self, tmp_path: Path) -> None:
|
|
types, _ = _router_sources(_build_engine(tmp_path, backend=_FakeBackend()))
|
|
assert LocalSkillSource in types
|
|
|
|
def test_backend_source_present_when_backend(self, tmp_path: Path) -> None:
|
|
types, _ = _router_sources(_build_engine(tmp_path, backend=_FakeBackend()))
|
|
assert BackendSkillSource in types
|
|
|
|
def test_backend_source_absent_without_backend(self, tmp_path: Path) -> None:
|
|
types, _ = _router_sources(_build_engine(tmp_path, backend=None))
|
|
assert BackendSkillSource not in types
|
|
|
|
def test_hub_source_omitted_when_endpoint_unset(self, tmp_path: Path) -> None:
|
|
types, _ = _router_sources(_build_engine(tmp_path, backend=_FakeBackend(), hub_endpoint=None))
|
|
assert HubSkillSource not in types
|
|
|
|
def test_hub_source_present_when_endpoint_set(self, tmp_path: Path) -> None:
|
|
types, _ = _router_sources(_build_engine(tmp_path, backend=_FakeBackend(), hub_endpoint="http://hub.test"))
|
|
assert HubSkillSource in types
|
|
|
|
def test_rrf_k_forwarded_from_config(self, tmp_path: Path) -> None:
|
|
engine = _build_engine(tmp_path, backend=_FakeBackend(), rrf_k=25)
|
|
skills = next(b for b in engine._builders if isinstance(b, SkillsSegmentBuilder))
|
|
assert skills._router._rrf_k == 25
|
|
|
|
def test_rrf_k_defaults_to_config_default(self, tmp_path: Path) -> None:
|
|
engine = _build_engine(tmp_path, backend=_FakeBackend())
|
|
skills = next(b for b in engine._builders if isinstance(b, SkillsSegmentBuilder))
|
|
assert skills._router._rrf_k == SkillForgeRouterConfig().rrf_k
|
|
|
|
def test_track_ids_from_memory_config(self, tmp_path: Path) -> None:
|
|
engine = _build_engine(
|
|
tmp_path,
|
|
backend=_FakeBackend(),
|
|
memory_config=MemoryConfig(
|
|
user_id="alice",
|
|
agent_id="robo",
|
|
),
|
|
)
|
|
assert _memory_builder(engine)._user_id == "alice"
|
|
_, sources = _router_sources(engine)
|
|
everos = next(s for s in sources if isinstance(s, BackendSkillSource))
|
|
assert everos._agent_id == "robo"
|
|
|
|
def test_router_names_the_backend_source_after_memory_backend(self, tmp_path: Path) -> None:
|
|
router = _build_router(
|
|
builder=ContextBuilder(workspace=tmp_path),
|
|
backend=_FakeBackend(),
|
|
memory_config=MemoryConfig(backend="acme", agent_id="a"),
|
|
skill_forge_router_config=SkillForgeRouterConfig(weights={"local": 1.0, "acme": 0.7}),
|
|
)
|
|
names = {s.name: s.weight for s in router._sources}
|
|
assert names["acme"] == 0.7
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Rewriter / gate model wiring — both must follow the agent's main model
|
|
# unless the gate has its own dedicated override.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestRewriterGateModelWiring:
|
|
def test_the_rewriter_carries_no_model_of_its_own(self, tmp_path: Path) -> None:
|
|
"""It follows the conversation, so it must not be pinned to the model
|
|
that happened to build the engine -- a session on another model would
|
|
otherwise have its query rewritten by the first session's model."""
|
|
engine = _build_engine(
|
|
tmp_path,
|
|
model="main-model",
|
|
skill_forge_config=SkillForgeConfig(discovery="push", rewrite_enabled=True, llm_gate_enabled=False),
|
|
)
|
|
skills = next(b for b in engine._builders if isinstance(b, SkillsSegmentBuilder))
|
|
assert not hasattr(skills._rewriter, "_model")
|
|
|
|
def test_an_unset_gate_model_stays_unset(self, tmp_path: Path) -> None:
|
|
"""Same rule one subsystem over: unset means follow the turn, not
|
|
"inherit whatever build_context_engine was called with"."""
|
|
engine = _build_engine(
|
|
tmp_path,
|
|
model="main-model",
|
|
skill_forge_config=SkillForgeConfig(
|
|
discovery="push", rewrite_enabled=False, llm_gate_enabled=True, llm_gate_model=None
|
|
),
|
|
)
|
|
skills = next(b for b in engine._builders if isinstance(b, SkillsSegmentBuilder))
|
|
assert skills._gate._pin_resolver() is None
|
|
|
|
def test_gate_prefers_dedicated_llm_gate_model(self, tmp_path: Path, monkeypatch) -> None:
|
|
cfg = _point_live_config(tmp_path, monkeypatch, {"skillForge": {"llmGateModel": "gate-only-model"}})
|
|
engine = _build_engine(
|
|
tmp_path,
|
|
model="main-model",
|
|
skill_forge_config=SkillForgeConfig(
|
|
discovery="push",
|
|
rewrite_enabled=False,
|
|
llm_gate_enabled=True,
|
|
llm_gate_model="gate-only-model",
|
|
),
|
|
provider_pool=_StubPool(),
|
|
)
|
|
skills = next(b for b in engine._builders if isinstance(b, SkillsSegmentBuilder))
|
|
assert skills._gate._binding()[1] == "gate-only-model"
|
|
assert cfg.exists()
|
|
|
|
def test_the_gate_pin_follows_the_file_rather_than_the_build_argument(self, tmp_path: Path, monkeypatch) -> None:
|
|
"""A gate model repointed on the settings page reaches the next filter.
|
|
|
|
The factory argument is the config as it stood when the engine was
|
|
built; the pin is read from the file when the gate runs, so the two can
|
|
disagree and the file is what counts. Before this the pair was resolved
|
|
once here, and the settings surface answered every write to it with
|
|
"applies after the next gateway reload or restart".
|
|
"""
|
|
cfg = _point_live_config(tmp_path, monkeypatch, {"skillForge": {"llmGateModel": "gate-from-file"}})
|
|
engine = _build_engine(
|
|
tmp_path,
|
|
skill_forge_config=SkillForgeConfig(
|
|
discovery="push",
|
|
rewrite_enabled=False,
|
|
llm_gate_enabled=True,
|
|
llm_gate_model="gate-at-build",
|
|
),
|
|
provider_pool=_StubPool(),
|
|
)
|
|
gate = next(b for b in engine._builders if isinstance(b, SkillsSegmentBuilder))._gate
|
|
|
|
assert gate._binding()[1] == "gate-from-file"
|
|
cfg.write_text(json.dumps({"skillForge": {"llmGateModel": "gate-after-edit"}}), encoding="utf-8")
|
|
assert gate._binding()[1] == "gate-after-edit"
|
|
|
|
def test_the_curator_pin_follows_the_file_too(self, tmp_path: Path, monkeypatch) -> None:
|
|
"""Same seam, the other subsystem pin. Both blocks are written by the
|
|
same settings surface and both used to owe a reload."""
|
|
cfg = _point_live_config(tmp_path, monkeypatch, {"context": {"curatorModel": "curator-from-file"}})
|
|
engine = _build_engine(tmp_path, provider_pool=_StubPool())
|
|
curator = next(b for b in engine._builders if isinstance(b, CuratorSegmentBuilder))
|
|
|
|
assert curator.curator_model == "curator-from-file"
|
|
cfg.write_text(json.dumps({"context": {"curatorModel": "curator-after-edit"}}), encoding="utf-8")
|
|
assert curator.curator_model == "curator-after-edit"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# AgentLoop helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _make_loop(tmp_path: Path, *, backend=None, agents=None, skill_forge_config=None) -> AgentLoop:
|
|
return AgentLoop(
|
|
provider=_StubProvider(),
|
|
workspace=tmp_path,
|
|
model="stub",
|
|
policy=TurnPolicy(max_iterations=2),
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
|
engine=EngineWiring(
|
|
backend=backend,
|
|
context_config=ContextConfig(),
|
|
memory_config=MemoryConfig(),
|
|
skill_forge_router_config=SkillForgeRouterConfig(),
|
|
skill_forge_config=skill_forge_config,
|
|
),
|
|
subagents=SubagentWiring(agents=agents),
|
|
)
|
|
|
|
|
|
class TestSubagentRosterReachesTheGate:
|
|
"""The join the segment-builder stubs cannot make: a real agent table, read
|
|
through the real builder. The roster is collected lazily because the loop
|
|
builds its context engine before its ``SubagentManager`` exists, so a
|
|
snapshot taken at wiring time would be empty every run.
|
|
|
|
Push discovery, explicitly: pull is the default and builds no skills
|
|
segment at all, so there is no gate there to hand a roster to."""
|
|
|
|
@staticmethod
|
|
def _push(tmp_path: Path, agents) -> AgentLoop:
|
|
return _make_loop(tmp_path, agents=agents, skill_forge_config=SkillForgeConfig(discovery="push"))
|
|
|
|
def test_a_configured_specialist_reaches_the_gate(self, tmp_path: Path) -> None:
|
|
agents = SubagentsConfig(
|
|
agents=[
|
|
{
|
|
"name": "Scribe",
|
|
"kind": "cli",
|
|
"command": "echo hi",
|
|
"description": "turns a source document into a deck",
|
|
}
|
|
]
|
|
).agents
|
|
agent = self._push(tmp_path, agents)
|
|
skills = next(b for b in agent.context_engine._builders if isinstance(b, SkillsSegmentBuilder))
|
|
roster = skills._collect_subagent_roster()
|
|
assert roster is not None
|
|
assert "Scribe" in roster
|
|
assert "turns a source document into a deck" in roster
|
|
|
|
def test_the_generic_agent_is_not_offered_as_an_overlap(self, tmp_path: Path) -> None:
|
|
agents = SubagentsConfig(agents=[{"name": "Scribe", "kind": "cli", "command": "echo hi"}]).agents
|
|
agent = self._push(tmp_path, agents)
|
|
skills = next(b for b in agent.context_engine._builders if isinstance(b, SkillsSegmentBuilder))
|
|
roster = skills._collect_subagent_roster() or ""
|
|
assert "Scribe" in roster
|
|
assert "no capability bias" not in roster
|
|
|
|
|
|
class TestAgentLoopEngineDetection:
|
|
def test_uses_default_engine_always_true(self, tmp_path: Path) -> None:
|
|
assert _make_loop(tmp_path, backend=_FakeBackend())._uses_default_engine() is True
|
|
|
|
def test_uses_default_engine_true_without_backend(self, tmp_path: Path) -> None:
|
|
assert _make_loop(tmp_path, backend=None)._uses_default_engine() is True
|
|
|
|
|
|
class TestSelectSkillsGating:
|
|
async def test_skill_selection_short_circuits_to_none(self, tmp_path: Path) -> None:
|
|
agent = _make_loop(tmp_path, backend=_FakeBackend())
|
|
assert await agent._select_skills_for_turn("hi", []) is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Metadata-stash path for _collect_injected_skill_ids
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestInjectedIdsFromMetadata:
|
|
def test_returns_qualified_ids_when_stash_populated(self, tmp_path: Path) -> None:
|
|
agent = _make_loop(tmp_path, backend=_FakeBackend())
|
|
agent._last_injected_skill_ids = ["local/x", "everos/y"]
|
|
ids = agent._collect_injected_skill_ids(None)
|
|
assert "local/x" in ids
|
|
assert "everos/y" in ids
|
|
|
|
def test_falls_back_to_legacy_when_stash_none(self, tmp_path: Path) -> None:
|
|
agent = _make_loop(tmp_path, backend=None)
|
|
agent._last_injected_skill_ids = None
|
|
fake_meta = MagicMock(spec_set=["source", "id"])
|
|
fake_meta.source = "local"
|
|
fake_meta.id = "git-resolver"
|
|
ids = agent._collect_injected_skill_ids([fake_meta])
|
|
assert "local/git-resolver" in ids
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# SF6: the window every holder reads must be the bound model's, not a copy
|
|
# taken when the holder was built.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestTheWindowFollowsTheTurnsBinding:
|
|
"""The window is a fact about the model, so every holder has to read the
|
|
one the running turn is bound to. Held as a copy taken at construction, the
|
|
trimmer and the consolidator would size a 1M session against whatever the
|
|
session that built them happened to run on."""
|
|
|
|
def test_every_holder_reads_the_window_of_the_bound_model(self, tmp_path: Path, monkeypatch) -> None:
|
|
from raven.providers import rates
|
|
|
|
windows = {"stub": 8192, "other-model": 4096}
|
|
monkeypatch.setattr(rates, "resolve_context_window", lambda model, **kw: windows.get(model))
|
|
|
|
agent = _make_loop(tmp_path, backend=None)
|
|
curator = _curator_builder(agent.context_engine)
|
|
|
|
assert agent.context_window_tokens == 8192
|
|
assert curator.context_window_tokens == 8192
|
|
assert curator.assembler.trimmer.context_window_tokens == 8192
|
|
assert agent.memory_consolidator.context_window_tokens == 8192
|
|
|
|
with use_binding(ModelBinding(_StubProvider(), "other-model")):
|
|
assert agent.context_window_tokens == 4096
|
|
assert curator.context_window_tokens == 4096
|
|
assert curator.assembler.context_window_tokens == 4096
|
|
assert curator.assembler.trimmer.context_window_tokens == 4096
|
|
assert agent.memory_consolidator.context_window_tokens == 4096
|
|
|
|
# And back: leaving the turn leaves nothing behind on the holders.
|
|
assert agent.context_window_tokens == 8192
|
|
assert curator.assembler.trimmer.context_window_tokens == 8192
|
|
|
|
def test_a_pinned_window_answers_for_every_model(self, tmp_path: Path, monkeypatch) -> None:
|
|
"""An explicit ``context_window_tokens`` is a deliberate override -- the
|
|
model a session switched to does not get to discard it."""
|
|
from raven.providers import rates
|
|
|
|
monkeypatch.setattr(
|
|
rates,
|
|
"resolve_context_window",
|
|
lambda *a, **k: (_ for _ in ()).throw(AssertionError("must not be asked when pinned")),
|
|
)
|
|
|
|
agent = AgentLoop(
|
|
provider=_StubProvider(),
|
|
workspace=tmp_path,
|
|
model="stub",
|
|
policy=TurnPolicy(max_iterations=2),
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
|
engine=EngineWiring(
|
|
context_window_tokens=8192,
|
|
context_config=ContextConfig(),
|
|
memory_config=MemoryConfig(),
|
|
skill_forge_router_config=SkillForgeRouterConfig(),
|
|
),
|
|
)
|
|
curator = _curator_builder(agent.context_engine)
|
|
|
|
with use_binding(ModelBinding(_StubProvider(), "other-model", agent.default_binding.configured_window)):
|
|
assert agent.context_window_tokens == 8192
|
|
assert curator.context_window_tokens == 8192
|
|
assert curator.assembler.trimmer.context_window_tokens == 8192
|
|
assert agent.memory_consolidator.context_window_tokens == 8192
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Pull discovery (the default): no skills segment, scent + router exposed
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_pull_default_drops_the_skills_segment_and_wires_scent(tmp_path):
|
|
engine = _build_engine(tmp_path, skill_forge_config=SkillForgeConfig())
|
|
assert not any(isinstance(b, SkillsSegmentBuilder) for b in engine._builders)
|
|
assert engine._scent is not None
|
|
assert engine.skills_router is not None
|
|
|
|
|
|
def test_push_config_restores_the_legacy_pipeline(tmp_path):
|
|
engine = _build_engine(tmp_path, skill_forge_config=SkillForgeConfig(discovery="push"))
|
|
assert any(isinstance(b, SkillsSegmentBuilder) for b in engine._builders)
|
|
assert engine._scent is None
|
|
assert engine.skills_router is not None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Ownership reaching the identity prompt, and the paths it depends on
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestTheEngineTellsItsIdentityWhoOwnsMemory:
|
|
"""The turn's half of the flag, held where the engine decides it.
|
|
|
|
Handing ``IdentitySegmentBuilder`` the flag directly pins how it renders,
|
|
not who told it. This builds the engine the way the loop does and reads
|
|
the builder it chose, so deleting the argument at the factory's call site
|
|
reddens here -- the suite was otherwise green with the request path naming
|
|
both memory files again.
|
|
"""
|
|
|
|
@staticmethod
|
|
async def _identity_text(engine) -> str:
|
|
builder = next(b for b in engine._builders if isinstance(b, IdentitySegmentBuilder))
|
|
return (await builder.build(_assembly_ctx())).text
|
|
|
|
async def test_without_a_backend_the_turn_still_names_the_memory_files(self, tmp_path: Path) -> None:
|
|
text = await self._identity_text(_build_engine(tmp_path))
|
|
assert f"{tmp_path}/user_memory/profile/user.md" in text
|
|
|
|
async def test_with_a_backend_the_turn_names_no_memory_file(self, tmp_path: Path) -> None:
|
|
text = await self._identity_text(_build_engine(tmp_path, backend=_FakeBackend()))
|
|
assert "profile/user.md" not in text
|
|
assert "# Memory" in text
|
|
|
|
|
|
def _assembly_ctx() -> AssemblyContext:
|
|
return AssemblyContext(
|
|
session_key="s",
|
|
current_message="hi",
|
|
media=None,
|
|
channel=None,
|
|
chat_id=None,
|
|
session_messages=[],
|
|
budget=TokenBudget(100_000, 4_000, 2_000, 1_000, 93_000),
|
|
)
|
|
|
|
|
|
class TestOwnershipReachesTheIdentityPrompt:
|
|
"""The join no stub can make: a real config row, through the real registry,
|
|
into the assembled prompt -- and a real tool table deciding whether the
|
|
prohibition it carries is one the model can act on."""
|
|
|
|
OWNS = "owns decks. Do not build the deck yourself."
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _isolated_switches(self, tmp_path: Path, monkeypatch) -> None:
|
|
"""Point the live off-switch read at a file of our own.
|
|
|
|
Every case here goes through the real tool table, which reads
|
|
``tools.disabledTools`` from whatever config file is current -- so without
|
|
this they consult the developer's own ``~/.raven/config.json``, and a
|
|
locally disabled ``spawn`` makes them red on one machine and green in CI.
|
|
"""
|
|
self._switches = tmp_path / "config.json"
|
|
self._disable()
|
|
monkeypatch.setattr("raven.home._current_config_path", self._switches)
|
|
|
|
def _disable(self, *names: str) -> None:
|
|
self._switches.write_text(json.dumps({"tools": {"disabledTools": list(names)}}), encoding="utf-8")
|
|
|
|
@staticmethod
|
|
def _identity(agent: AgentLoop) -> IdentitySegmentBuilder:
|
|
return next(b for b in agent.context_engine._builders if isinstance(b, IdentitySegmentBuilder))
|
|
|
|
async def _text(self, agent: AgentLoop) -> str:
|
|
return (await self._identity(agent).build(_assembly_ctx())).text
|
|
|
|
def _scribe(self, kind: str = "cli") -> list:
|
|
row = {"name": "Scribe", "kind": kind, "owns": self.OWNS}
|
|
if kind == "cli":
|
|
row["command"] = "echo hi"
|
|
return SubagentsConfig(agents=[row]).agents
|
|
|
|
async def test_a_custom_builtin_specialist_gets_its_line(self, tmp_path: Path) -> None:
|
|
"""A built-in row is on the same dispatchable table as an external one, so
|
|
narrowing one into a specialist has to be declarable the same way."""
|
|
assert f"`Scribe` {self.OWNS}" in await self._text(_make_loop(tmp_path, agents=self._scribe("builtin")))
|
|
|
|
async def test_an_external_specialist_gets_its_line(self, tmp_path: Path) -> None:
|
|
"""Regression guard for the path that already worked, not a proof of the
|
|
fix: this one is green on either side of it."""
|
|
assert f"`Scribe` {self.OWNS}" in await self._text(_make_loop(tmp_path, agents=self._scribe()))
|
|
|
|
async def test_ownership_on_the_generic_row_claims_nothing(self, tmp_path: Path) -> None:
|
|
"""Paired with a real specialist so the absence is selective: asserting
|
|
only that the claim is missing would also pass on an install where the
|
|
whole section failed to render."""
|
|
agents = SubagentsConfig(
|
|
agents=[
|
|
{"name": GENERIC_AGENT, "kind": "builtin", "owns": "owns everything. Do not do anything yourself."},
|
|
{"name": "Scribe", "kind": "cli", "command": "echo hi", "owns": self.OWNS},
|
|
]
|
|
).agents
|
|
text = await self._text(_make_loop(tmp_path, agents=agents))
|
|
assert f"`Scribe` {self.OWNS}" in text
|
|
assert "owns everything" not in text
|
|
|
|
async def test_withholding_every_dispatch_path_retires_the_prohibition(self, tmp_path: Path) -> None:
|
|
"""Read from the file per turn, so a switch flipped on the settings page
|
|
takes the section away and putting it back brings it back -- without which
|
|
the prompt forbids work the turn has no way to hand off."""
|
|
agent = _make_loop(tmp_path, agents=self._scribe())
|
|
offered = {d["function"]["name"] for d in agent.tools.get_definitions()}
|
|
assert {"spawn", "run_subagent_dag"} <= offered
|
|
|
|
assert "## Delegation" in await self._text(agent)
|
|
|
|
self._disable("spawn", "run_subagent_dag")
|
|
assert "## Delegation" not in await self._text(agent)
|
|
|
|
self._disable()
|
|
assert "## Delegation" in await self._text(agent)
|
|
|
|
async def test_withholding_the_whole_tool_table_retires_it_too(self, tmp_path: Path) -> None:
|
|
"""Naming the two dispatch tools is not the only way to reach zero live
|
|
paths. Withholding every registered tool reaches it through a successful
|
|
but empty lookup, which the gate first shipped reporting as both paths
|
|
live -- so the prohibition survived on a turn offering no tools at all.
|
|
"""
|
|
agent = _make_loop(tmp_path, agents=self._scribe())
|
|
every = sorted({d["function"]["name"] for d in agent.tools.get_definitions()})
|
|
assert {"spawn", "run_subagent_dag"} <= set(every)
|
|
assert "## Delegation" in await self._text(agent)
|
|
|
|
self._disable(*every)
|
|
# Not empty any more: raven reserves the tool-search meta-pair from the
|
|
# off switch, so what an operator can withhold is everything else. Zero
|
|
# live delegation paths is what retires the section, and that is reached.
|
|
from raven.agent.tools.tool_search import META_TOOL_NAMES
|
|
|
|
assert {d["function"]["name"] for d in agent.tools.get_definitions()} == set(META_TOOL_NAMES)
|
|
assert "## Delegation" not in await self._text(agent)
|
|
|
|
async def test_withholding_one_path_keeps_the_other_named(self, tmp_path: Path) -> None:
|
|
self._disable("spawn")
|
|
text = await self._text(_make_loop(tmp_path, agents=self._scribe()))
|
|
assert f"`Scribe` {self.OWNS}" in text
|
|
assert "`run_subagent_dag`" in text
|
|
assert "`spawn`" not in text
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# context.dropSegments
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _engine_with(tmp_path: Path, config: ContextConfig) -> ContextAssembler:
|
|
return build_context_engine(
|
|
workspace=tmp_path,
|
|
config=config,
|
|
builder=ContextBuilder(workspace=tmp_path),
|
|
provider=_StubProvider(),
|
|
model="stub",
|
|
context_window_tokens=8192,
|
|
get_tool_definitions=_stub_get_defs,
|
|
memory_config=MemoryConfig(),
|
|
skill_forge_router_config=SkillForgeRouterConfig(hub=HubSourceConfig(endpoint=None)),
|
|
skill_forge_config=SkillForgeConfig(discovery="push"),
|
|
)
|
|
|
|
|
|
def test_a_product_drops_the_host_segments_it_names(tmp_path: Path) -> None:
|
|
full = [b.name for b in _engine_with(tmp_path, ContextConfig())._builders]
|
|
assert full[:2] == ["identity", "bootstrap"] and "memory" in full and "curator" in full
|
|
|
|
slim = [b.name for b in _engine_with(tmp_path, ContextConfig(drop_segments=["identity", "memory"]))._builders]
|
|
assert "identity" not in slim and "memory" not in slim
|
|
assert slim == [n for n in full if n not in ("identity", "memory")], "order and the rest untouched"
|
|
|
|
|
|
def test_an_unknown_segment_name_drops_nothing(tmp_path: Path) -> None:
|
|
full = [b.name for b in _engine_with(tmp_path, ContextConfig())._builders]
|
|
assert [b.name for b in _engine_with(tmp_path, ContextConfig(drop_segments=["nope"]))._builders] == full
|