## Summary
Eight keys on the settings page answered `Saved. Applies after the next
gateway
reload or restart.` They do not any more, and the reload-only set is
gone with
them -- there is nothing left for it to name.
They were all the same shape: the value was copied onto the objects the
loop was
built with, and nothing read the file again. The fix is not a
notification
mechanism. Each value is read from the file where it is used, so a write
from
the page, the CLI or a hand edit is covered by construction rather than
one
writer at a time.
**The two subsystem pins** -- `context.curatorModel` and
`skillForge.llmGateModel`, each with its provider -- are resolved per
call
instead of once in the context-engine factory. Both consumers already
funnelled
through one method (`CuratorSegmentBuilder._curator_binding`,
`LLMGateFilter._binding`) and already read the turn's binding when
unpinned;
only the pin itself was frozen. `permissions.judgeModel` and
`sessionTitle.model` already read their pins this way, which is why
those two
were never reload-only -- so the part that re-reads, binds through the
pool and
reports an unusable pair once is shared as
`providers.pool.live_pin_resolver`.
**`agents.defaults.maxToolIterations`** becomes a property over the live
file,
falling back to what the loop was built with. A turn reads it once at
its start,
so a change lands on the next turn rather than halfway through one.
**`agents.defaults.contextWindowTokens`** was the interesting one. It
had two
copies with different readers: one frozen on the loop that only the
usage report
read, and one on each `ModelBinding` that the budget read. A write
reached
neither until a restart -- and after a model switch the two disagreed,
because
the pool rebuilds its bindings when the number changes while the loop's
copy
never moves. Measured on a real gateway before this change: trimming ran
on the
new number while the page reported the previous model's catalogue size.
There is one source now, on the binding. The turn entry resolves it from
the
file once and holds it for the whole turn tree, the same contract the
provider/model pair already has (`AgentLoop._with_live_window`): the
budget is
read several times while a turn runs -- history trimming, the compaction
check,
the usage report -- and a number that moved between those reads would
leave one
turn disagreeing with itself. Outside a turn the default binding
answers, which
is the rule `context_window_tokens` already followed.
Two consequences a reviewer should weigh:
- **The file is now the source for these values, not the object passed
at
construction.** In every production path they are the same thing --
`EngineWiring` is filled from the same loader -- but a host that builds
a loop
with a synthesised `ContextConfig` or window and no config file behind
it now
gets the file's answer (or none). Clearing a key therefore takes effect
too,
which is what made `contextWindowTokens: null` on the settings page do
anything at all.
- **`set_provider` carries the current default's window forward** rather
than
re-reading, because that entry point exists for a caller holding a
provider of
its own, which need not be a process this config was loaded for.
Not in this change: `assembler.set_context_window` /
`ContextEngine.set_context_window`
have no caller in `raven/`, so the out-of-turn fallback windows held by
the
curator and the history trimmer have never been updated after
construction.
Pre-existing, untouched, and invisible to a turn (both read the active
binding).
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Real host, one `raven serve` process, one session that never switched
model, no
restart at any point:
```
turn 1 context_max=0
settings.set 50000 {"applied": true, "previous": null} <- no warning
turn 2 context_max=50000
settings.set null {"applied": true, "previous": 50000}
turn 3 context_max=0
```
The same host earlier, on the code before this change, answered
`context_max=0`
on turn 2 and carried `Saved. Applies after the next gateway reload or
restart.`
in the reply; a restart on the same RAVEN_HOME was what produced 50000.
The
disagreement described above was measured the same way: after
`config.set model`
in that process the report read 131072 while a pool-built binding
carried 50000.
`agents.defaults.maxToolIterations` was checked the same way on a real
host
before this branch existed: the gateway log goes from `Iteration 1/40`
to
`Iteration 1/3` across a `settings.set`, in one process.
The two pins are covered by tests rather than by a live host: the
curator's slow
path and the skill gate did not run on a short turn, so there was
nothing to
observe. The cases do a real factory build, rewrite the config file
underneath
it and assert the next call follows.
- `uv run pytest tests/ -q`: 23325 passed, 4 failed, 111 skipped. All
four
failures reproduce on a clean checkout of the base commit in the same
environment (`test_agent_loop_token_budget`, the two
`test_agents_*_launcher` tool-face cases,
`test_ppt_engine_image_search`);
the launcher pair looks environment-dependent (a `SERPER_API_KEY` in the
developer environment makes `web_search` available).
- After the rebase onto the current base: the same sweep over
settings/context/skill-forge/loop/pool/usage-sink/onboarding, 2781
passed,
1 failed (the pre-existing token-budget case).
- `npm run type-check` clean; `npx vitest run` 185 files, 2568 tests
passed.
- `uv run ruff check` and `ruff format --check`: clean.
- Eighteen new or changed cases, each mutation-verified on its own --
among
them: caching the resolver's first answer, warning on every call,
reading one
spelling of a block or a field, a turn rebuilding its window when the
number
has not changed, a running turn following a mid-turn edit,
`set_provider`
dropping a pinned window, and a settings write still claiming a restart
is
needed.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
## Risk
User-visible: these eight writes take effect where they are read -- the
next
call for a pin, the next turn for the cap and the window -- instead of
after a
restart, and the settings surface no longer says otherwise. A turn
already
running is never moved.
Contract: `settings.set` still returns `warning`; the only producer left
is an
embedding model change, which the page renders the same way.
Cost: the added reads are a byte comparison on one small file
(`LiveConfig`
re-parses only when the bytes change) plus a pool cache hit for a pin
already
built. Nothing on this path builds a provider that was not going to be
built.
Rollback: revert the commit. The two deleted holders
(`AgentLoop._configured_window`,
`_RELOAD_ONLY_KEYS`) come back with it, and no other branch reads
either.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
221 lines
8.6 KiB
Python
221 lines
8.6 KiB
Python
"""Tests for the LLM gate that filters router candidates."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from dataclasses import dataclass
|
|
from typing import Any
|
|
|
|
from raven.memory_engine.skill_forge.gate import LLMGateFilter
|
|
from raven.memory_engine.skill_forge.types import RouterHit
|
|
|
|
|
|
@dataclass
|
|
class _Resp:
|
|
content: str
|
|
finish_reason: str = "stop"
|
|
|
|
|
|
class _StubProvider:
|
|
def __init__(self, response: Any) -> None:
|
|
self._response = response
|
|
self.calls: list[dict[str, Any]] = []
|
|
|
|
async def chat_with_retry(self, **kwargs: Any) -> _Resp:
|
|
self.calls.append(kwargs)
|
|
if isinstance(self._response, BaseException):
|
|
raise self._response
|
|
if isinstance(self._response, _Resp):
|
|
return self._response
|
|
return _Resp(content=str(self._response))
|
|
|
|
|
|
def _hit(qid: str, name: str, body: str = "", desc: str = "") -> RouterHit:
|
|
return RouterHit(
|
|
qualified_id=qid,
|
|
name=name,
|
|
content=body,
|
|
score=0.5,
|
|
meta={"description": desc, "source": qid.split("/", 1)[0]},
|
|
)
|
|
|
|
|
|
# ----------------------------------------------------------------------
|
|
|
|
|
|
async def test_empty_candidates_returns_empty() -> None:
|
|
gate = LLMGateFilter(_StubProvider("{}"))
|
|
assert await gate.filter("any task", []) == []
|
|
|
|
|
|
async def test_selects_by_qualified_id() -> None:
|
|
provider = _StubProvider(
|
|
json.dumps(
|
|
{
|
|
"plan": "use pdf gen",
|
|
"skills": ["local/pdf-gen"],
|
|
}
|
|
)
|
|
)
|
|
gate = LLMGateFilter(provider, max_select=2)
|
|
hits = [
|
|
_hit("local/pdf-gen", "pdf-gen", body="generate pdf"),
|
|
_hit("local/weather", "weather", body="get weather"),
|
|
]
|
|
out = await gate.filter("make a pdf report", hits)
|
|
assert [h.qualified_id for h in out] == ["local/pdf-gen"]
|
|
|
|
|
|
async def test_empty_skills_list_is_valid_inject_nothing() -> None:
|
|
provider = _StubProvider(json.dumps({"plan": "nothing fits", "skills": []}))
|
|
out = await LLMGateFilter(provider).filter("task", [_hit("local/foo", "foo")])
|
|
assert out == []
|
|
|
|
|
|
async def test_respects_max_select_truncation() -> None:
|
|
provider = _StubProvider(
|
|
json.dumps(
|
|
{
|
|
"plan": "p",
|
|
"skills": ["local/a", "local/b", "local/c"],
|
|
}
|
|
)
|
|
)
|
|
gate = LLMGateFilter(provider, max_select=2)
|
|
hits = [_hit(f"local/{n}", n) for n in ("a", "b", "c")]
|
|
out = await gate.filter("task", hits)
|
|
assert [h.qualified_id for h in out] == ["local/a", "local/b"]
|
|
|
|
|
|
async def test_unknown_id_in_response_silently_dropped() -> None:
|
|
provider = _StubProvider(
|
|
json.dumps(
|
|
{
|
|
"plan": "p",
|
|
"skills": ["local/known", "ghost/missing"],
|
|
}
|
|
)
|
|
)
|
|
hits = [_hit("local/known", "known")]
|
|
out = await LLMGateFilter(provider).filter("task", hits)
|
|
assert [h.qualified_id for h in out] == ["local/known"]
|
|
|
|
|
|
async def test_provider_error_falls_back_to_top_n() -> None:
|
|
"""Infra failure ≠ deliberate empty. Falling back to legacy top-N
|
|
keeps the prompt populated; returning [] would silently kill skill
|
|
injection when the provider has a transient blip."""
|
|
provider = _StubProvider(RuntimeError("network blip"))
|
|
gate = LLMGateFilter(provider, legacy_top_k=2)
|
|
hits = [_hit(f"local/{n}", n) for n in ("a", "b", "c", "d")]
|
|
out = await gate.filter("task", hits)
|
|
assert [h.qualified_id for h in out] == ["local/a", "local/b"]
|
|
|
|
|
|
async def test_unparseable_response_falls_back_to_top_n() -> None:
|
|
provider = _StubProvider("garbage no json here")
|
|
gate = LLMGateFilter(provider, legacy_top_k=1)
|
|
hits = [_hit("local/a", "a"), _hit("local/b", "b")]
|
|
out = await gate.filter("task", hits)
|
|
assert [h.qualified_id for h in out] == ["local/a"]
|
|
|
|
|
|
async def test_think_block_stripped_before_parse() -> None:
|
|
"""Qwen3-style reasoning models emit <think>...</think> before JSON.
|
|
The gate must tolerate it without falling back."""
|
|
provider = _StubProvider('<think>let me think...</think>\n{"plan": "p", "skills": ["local/a"]}')
|
|
out = await LLMGateFilter(provider).filter("task", [_hit("local/a", "a")])
|
|
assert [h.qualified_id for h in out] == ["local/a"]
|
|
|
|
|
|
async def test_tools_block_present_when_tools_given() -> None:
|
|
provider = _StubProvider(json.dumps({"plan": "p", "skills": []}))
|
|
gate = LLMGateFilter(provider)
|
|
await gate.filter("task", [_hit("local/a", "a")], available_tools=["read_file", "exec"])
|
|
prompt = provider.calls[0]["messages"][0]["content"]
|
|
assert "# Agent Tools" in prompt
|
|
assert "read_file" in prompt
|
|
assert "exec" in prompt
|
|
|
|
|
|
async def test_tools_block_absent_when_tools_none() -> None:
|
|
provider = _StubProvider(json.dumps({"plan": "p", "skills": []}))
|
|
await LLMGateFilter(provider).filter("task", [_hit("local/a", "a")])
|
|
prompt = provider.calls[0]["messages"][0]["content"]
|
|
assert "# Agent Tools" not in prompt
|
|
|
|
|
|
async def test_subagent_block_present_when_roster_given() -> None:
|
|
provider = _StubProvider(json.dumps({"plan": "p", "skills": []}))
|
|
gate = LLMGateFilter(provider)
|
|
await gate.filter(
|
|
"task",
|
|
[_hit("local/a", "a")],
|
|
available_subagents="Raven-Research [stateless] (reads live web pages and cites them)",
|
|
)
|
|
prompt = provider.calls[0]["messages"][0]["content"]
|
|
assert "# Delegable Sub-Agents" in prompt
|
|
assert "Raven-Research" in prompt
|
|
assert "already covered by" in prompt
|
|
|
|
|
|
async def test_subagent_block_absent_when_no_roster() -> None:
|
|
provider = _StubProvider(json.dumps({"plan": "p", "skills": []}))
|
|
await LLMGateFilter(provider).filter("task", [_hit("local/a", "a")])
|
|
prompt = provider.calls[0]["messages"][0]["content"]
|
|
assert "# Delegable Sub-Agents" not in prompt
|
|
assert "already covered by" not in prompt
|
|
|
|
|
|
async def test_plan_step_offers_delegation_as_an_alternative_to_running_the_tools() -> None:
|
|
"""The overlap check in step 2 cannot undo a plan step 1 already committed
|
|
to. Asked only which tools to call, the gate plans the work inline and the
|
|
candidate is then genuinely relevant to that plan -- so delegation has to be
|
|
on the table while the plan is still forming. One sub-agent or several: a
|
|
task split across a graph is delegated just as much as a single spawn."""
|
|
provider = _StubProvider(json.dumps({"plan": "p", "skills": []}))
|
|
await LLMGateFilter(provider).filter(
|
|
"task",
|
|
[_hit("local/a", "a")],
|
|
available_subagents="Raven-Research [stateless] (reads live web pages and cites them)",
|
|
)
|
|
prompt = provider.calls[0]["messages"][0]["content"]
|
|
plan_step = prompt.split("1. **Plan**:")[1].split("2. **Filter**:")[0]
|
|
assert "one or several of the sub-agents above" in plan_step
|
|
# The roster sentence must not read as a one-agent-only offer either.
|
|
assert "split it across several of them as a graph" in prompt
|
|
|
|
|
|
async def test_plan_step_is_unchanged_when_no_roster_is_supplied() -> None:
|
|
"""With nothing to delegate to, "the sub-agents above" names nothing, so the
|
|
clause is withheld rather than left dangling."""
|
|
provider = _StubProvider(json.dumps({"plan": "p", "skills": []}))
|
|
await LLMGateFilter(provider).filter("task", [_hit("local/a", "a")])
|
|
prompt = provider.calls[0]["messages"][0]["content"]
|
|
assert "1. **Plan**: briefly think about what the task requires and which sequence" in prompt
|
|
assert "sub-agents above should carry it" not in prompt
|
|
|
|
|
|
async def test_an_unpaired_gate_model_is_never_sent() -> None:
|
|
"""``skill_forge.llm_gate_model`` without credentials of its own is a bare
|
|
id. Forwarding it would post it on whatever key this provider holds, which
|
|
is the mis-pairing the pool exists to prevent; the gate follows the turn
|
|
instead and names no model."""
|
|
provider = _StubProvider(json.dumps({"plan": "p", "skills": []}))
|
|
# What the resolver answers for a configured id the pool could not pair.
|
|
gate = LLMGateFilter(provider, pin_resolver=lambda: None)
|
|
await gate.filter("task", [_hit("local/a", "a")])
|
|
assert provider.calls[0]["model"] is None
|
|
|
|
|
|
async def test_a_paired_gate_pin_is_used_as_given() -> None:
|
|
"""With the pin paired by the pool, both halves are the gate's own."""
|
|
from raven.providers.binding import ModelBinding
|
|
|
|
provider = _StubProvider(json.dumps({"plan": "p", "skills": []}))
|
|
pinned = _StubProvider(json.dumps({"plan": "p", "skills": []}))
|
|
gate = LLMGateFilter(provider, pin_resolver=lambda: ModelBinding(pinned, "gpt-4o"))
|
|
await gate.filter("task", [_hit("local/a", "a")])
|
|
assert provider.calls == []
|
|
assert pinned.calls[0]["model"] == "gpt-4o"
|