## Summary
Two defects in one area, both measured on this machine.
**A heartbeat on an untouched `HEARTBEAT.md` cost one model call every
interval.** The shipped template promises the agent "will skip the
heartbeat" for a file of only headers and comments, and the service
skipped only a file with **no bytes at all**. Most workspaces never edit
it, so the decision call ran every 30 minutes forever -- 14 to 35 per
day here (`~/.raven/workspace/HEARTBEAT.md` is byte-identical to
`raven/templates/HEARTBEAT.md`).
What counts as a task is now read from the template itself
(`has_tasks`), so the template and the check cannot drift: blank lines,
HTML comments, empty list markers and the template's own lines are
scaffolding; anything else is a task, and one is still decided. A task
written as a heading, or a section the template does not have, still
reaches the model. The check errs towards asking, because a missed call
costs one request where a missed task costs the task. `trigger_now`
shares the predicate.
**Almost every model call outside the turn loop went unbilled.**
`UsageTracker`'s docstring says it records every LLM call; what it
actually heard was the turn loop (`turn_path.py`) and image generation
(`wiring.py`). Some forty other callers reach a provider directly -- the
heartbeat's decision, the sentinel planner and its predictors, memory
consolidation, session titles, the permission judge, the playbook
planner, the curator, the in-process sub-agent's own loop -- and none of
them reached it. Twenty-odd heartbeats in one day left no row in
`~/.raven/telemetry/usage-*.jsonl`.
The recording moves to the seam every caller shares:
`raven/providers/usage_record.py` wraps `chat`, `chat_with_retry` and
`chat_stream` on each provider class (`LLMProvider.__init_subclass__`
applies it to every adapter and wrapper), and `install_from_config`
hands the seam the registry that hears them by default, so the whole
registry is used rather than one tracker. The turn loop binds its own
registry over that default for each decision (`usage_record.bind`), so a
serving generation's turns are billed to that generation even while a
candidate generation is being built beside it (the second review
finding; see the fourth commit). Nothing is recorded until an entry
point assembles a runtime, which is what `raven doctor` and the unit
tests want.
Each call is billed **once**, in three senses:
1. **Nested providers and the retry ladder record at the outermost
entry.** `ResolvingProvider`, `PerModelProvider` and `LazyProvider`
delegate to an inner provider whose entry points are instrumented too,
and `chat_with_retry` calls `chat` for every attempt. The outermost
entry owns the call, so a retried or fallen-back call is one row under
the response the caller actually received -- the unit the turn loop has
always recorded.
2. **Every call an Action makes is billed, and the loop never adds a
copy.** One `decide()` may make several provider calls (best-of-n, a
critic pass -- `ActionRequest` allows it), so nothing inside it is
claimed: the seam records each call under the turn's session, which the
loop binds around `decide()`. The loop skips its own row when the seam
recorded any call under the turn's bind
(`usage_record.recorded_inbound`), where that row would copy a call
already recorded; when the seam recorded nothing (a provider no runtime
built, a test) the loop's row is still the only one. The count is held
in a box the bind creates, so calls an Action makes in tasks it spawns
(`gather`) count too. A critic that returns a response no single call
produced is billed for the calls actually made. (The first revision
claimed the whole decision instead, which lost every call but the
returned one -- the review finding; see the third commit.)
3. **A stream is one row, recorded at exhaustion**, from the delta that
carried the usage -- the terminal usage and the finish reason can arrive
on different deltas, so neither alone is the trigger. The wrapper
nearest the wire owns the stream; a wrapper that finds the first delta
already marked is passing an inner provider's stream through and records
nothing. The base `chat_stream`'s non-streaming fallback claims the
`chat` it rides on for the same reason.
A call that failed before reaching a model is not a row -- nothing was
spent, and a row of zeros would read as a cheap call. The heartbeat's
own decision is billed to a `heartbeat` session (`usage_context.bind`)
rather than to whichever conversation ran last. Image usage is
untouched: it is recorded from the tool's own httpx path, not through a
provider.
### Call sites surveyed, as asked
Only one path in the tree records usage itself, so nothing else double
counts: the turn loop (`recorded_by_caller`, above) and image generation
(`media_gen._record_usage`, `wiring._record_image_usage`, neither of
which goes through a provider). Every provider subclass in the tree is
covered by the seam -- checked by walking the package and testing each
class's `chat`/`chat_with_retry`/`chat_stream` for the wrapper or for an
instrumented base; the result was "none missing". `make_provider`
returns `LiteLLMProvider`, `ResolvingProvider` or a pluggable class, all
instrumented.
One deliberate non-change: `send_probe` (`raven/core/provider_stack.py`,
used by `raven doctor` and onboard) builds its own provider outside any
runtime, so it is not billed. The model list it would report into is not
the install's telemetry and, for a doctor check, does not belong there.
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Head `9a5839f24`, four commits (the heartbeat skip; the provider seam;
the fix for the review finding that a multi-call Action lost calls; the
fix for the review finding that a registry swap billed one call to two
registries), rebased onto `origin/main` at `8ee5ae0d8` (the latest
rebase replayed all four commits unchanged; an earlier one onto
`afdc04f89` conflicted only in `tests/test_token_wise_usage_tracker.py`,
where main's delegated-usage tests and this branch's seam tests were
added at the same spot; both are kept). The focused suite below reran at
this head: 92 passed.
At this head:
```
COLORTERM=truecolor make coverage
25918 passed, 109 skipped, 1 failed: tests/test_rpc_files.py::test_a_host_without_libreoffice_says_so, which fails on origin/main on this machine too (LibreOffice is installed here; CI has none)
COVERAGE_BASE_REF=origin/main make coverage-diff
Diff coverage: 97.47% (154/158 executable changed lines)
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q -p no:cacheprovider tests/test_token_wise_usage_tracker.py tests/test_agent_loop_usage_sink.py tests/test_event_wake.py
92 passed
ruff check / ruff format --check / make lint-imports lint-deps lint-types / git diff --check origin/main...HEAD
check_source_language.py origin/main...HEAD / make check-commits / check-large-files
all clean
```
The registry-swap finding, reproduced at the loop:
`test_a_serving_generation_is_billed_alone_after_a_candidate_installs`
installs a candidate registry process-wide and then runs a two-call turn
on a loop holding its own. With `raven/agent/loop/turn_path.py` restored
to the previous head it fails (the candidate's tracker gets the seam's
rows and the serving one the loop's copy); it passes at this head.
`test_a_registry_swap_does_not_bill_one_call_to_both_registries` pins
the same case at the seam.
`test_calls_made_in_tasks_spawned_under_a_bind_count_toward_it` fails
with the count held by value (`assert 0 == 3`) and passes with the box.
The numbers below are from the previous head `6a61cbae0` (on
`89a24c576`) and are kept for the record.
```
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q
25893 passed, 110 skipped in 327.36s, 0 failed
uv run --frozen --python 3.12 --extra dev ruff check raven/ tests/
All checks passed!
uv run --frozen --python 3.12 --extra dev ruff format --check raven/ tests/
1700 files already formatted
uv run --frozen --python 3.12 lint-imports
Contracts: 10 kept, 0 broken.
uv run --frozen --python 3.12 --all-extras deptry raven
Success! No dependency issues found.
make lint-types
All checks passed!
PYTHONPATH=. uv run ... scripts/check_source_language.py origin/main...HEAD ; make check-commits ; COMMIT_RANGE=origin/main...HEAD make check-large-files ; PR_TITLE=... make check-pr-title
exit 0, all four
```
The reproduction, on this machine's real files:
`~/.raven/workspace/HEARTBEAT.md` compared byte-for-byte equal to the
shipped template, 254 heartbeat ticks in `gateway.log`, and heartbeat
rows absent from `~/.raven/telemetry/usage-2026-09-2*.jsonl` (the files
carry `acp:`/`tui:`/`cli:` sessions only).
Revert-to-red, one production file at a time, with the commits already
landed so `git checkout origin/main -- <file>` is a real revert:
```
BASELINE (nothing reverted) 85 passed
REVERT raven/proactive_engine/.../service.py 3 failed, 82 passed
test_an_untouched_template_costs_no_llm_call
test_trigger_now_on_an_untouched_template_asks_nothing
test_the_heartbeat_decision_is_billed_to_the_heartbeat_session
REVERT raven/providers/base.py 7 failed, 78 passed
(the six seam tests, incl. the fallback-stream single-row case)
REVERT raven/core/token_wise_stack.py 1 failed, 84 passed
test_the_assembly_installs_its_registry_at_the_seam (now test_a_call_is_billed_to_the_registry_the_caller_bound)
REVERT raven/agent/loop/turn_path.py (revert-to-red measured on the previous head; the loop's part is now pinned by the review-fix matrix below)
RESTORED 85 passed
```
`raven/providers/usage_record.py` is new on this branch, so it has no
`origin/main` version to revert to; its guards are covered by mutation
instead, one mutant per guard, with the suite run by a self-restoring
script:
```
BASELINE 85 passed
MUTANT stream owner check removed 1 failed (test_a_stream_whose_usage_and_finish_arrive_apart_is_one_row)
MUTANT call-depth claim removed 4 failed (the four once-only tests)
MUTANT error-without-usage skip removed 1 failed (test_an_error_that_reached_no_model_is_not_a_row)
MUTANT heartbeat skip -> "file has bytes" 1 failed (test_an_untouched_template_costs_no_llm_call)
MUTANT heartbeat session bind removed 1 failed (test_the_heartbeat_decision_is_billed_to_the_heartbeat_session)
MUTANT fallback stream's claim on chat removed 1 failed (the fallback single-row case)
MUTANT loop's skip-when-the-seam-hears removed 2 failed (the two loop tests below: every call counted twice)
MUTANT assembly install removed 1 failed (test_the_assembly_installs_its_registry_at_the_seam)
CONTROL restored 85 passed
```
The review fix, mutated one guard at a time against
`tests/test_agent_loop_usage_sink.py`:
```
BASELINE 21 passed
MUTANT the whole-decision claim restored (as reviewed) 2 failed (test_every_call_a_multi_call_action_makes_is_billed, test_a_turn_is_billed_once_while_the_provider_seam_listens_too)
MUTANT loop's skip-when-the-seam-hears removed 2 failed (same two: every call counted twice)
MUTANT session bind around decide removed 2 failed (same two: rows land outside the turn's session)
CONTROL restored 21 passed
```
`test_every_call_a_multi_call_action_makes_is_billed` runs a real turn
with a best-of-two Action (`DefaultAction.decide` called twice, second
answer returned) and asserts two provider calls, two rows under the
turn's session, and a total of both costs. It fails on the reviewed
code, which is the finding reproduced.
Every mutant is caught by the test that pins its guard; the depth-claim
mutant trips four, which is right -- it is what keeps a nested call to
one row.
The guard tests carry their own controls: the template skip has a
positive twin that adds one task to the same template (as a bullet and
as a heading) and asserts the model is still asked; the loop's once-only
test asserts `provider.calls == 2` beside `tracker.total.calls == 2`, so
a fix that recorded neither would fail it.
New autouse fixture in `tests/conftest.py` clears the seam's sink around
each test: the install is process-wide, so without it a test that
assembles a runtime would point later tests' provider calls at a tracker
built under its temporary home.
`ui-tui`/`ui-web` are untouched. Integration tests were not run: nothing
here reaches a real resource.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
User-visible: an untouched `HEARTBEAT.md` no longer produces a decision
call, so a workspace that relied on the heartbeat to notice system
events with no file still does -- events are checked first and still
decide. A heartbeat that does have tasks is unchanged.
`~/.raven/telemetry/usage-*.jsonl` gains rows for calls that were
previously invisible: the heartbeat's decisions, the sentinel's,
consolidation, titles, the judge, the playbook planner, the curator, the
in-process sub-agent's loop. Daily totals will read higher than they
did, because they were previously undercounting -- not because anything
is now billed twice (the once-only guarantees and their tests are
above). Anything aggregating those files by session key will see new
session names (`heartbeat`, and whichever conversation was bound for the
rest).
Rollback: revert the squash commit. The seam is inert when nothing
installs it, so the second commit's risk is confined to providers that a
runtime built.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
757 lines
27 KiB
Python
757 lines
27 KiB
Python
"""Tests for the usage_sink populated by AgentLoop and surfaced to the TUI.
|
|
|
|
Pins the wire shape that ``turn.send`` relays as ``message.complete.payload.usage``:
|
|
per-turn token counts plus the live context-window gauge (used / max / percent)
|
|
and the provider-reported cost. Before this, only the token counts were populated, so the
|
|
TUI context bar stayed frozen at 0% and never showed cost.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import tempfile
|
|
import threading
|
|
import time
|
|
from dataclasses import replace
|
|
from pathlib import Path
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
import raven.agent.loop.turn_path as agent_loop_main
|
|
from raven.agent.harness.action import DefaultAction
|
|
from raven.agent.loop import AgentLoop
|
|
from raven.agent.loop.bundles import EngineWiring, ToolWiring, TurnPolicy
|
|
from raven.config.raven import ContextConfig
|
|
from raven.contracts.llm_provider import ToolCallRequest
|
|
from raven.contracts.token_strategy import UsageSnapshot
|
|
from raven.contracts.tool import Tool
|
|
from raven.providers import rates, usage_record
|
|
from raven.providers.base import LLMProvider, LLMResponse
|
|
from raven.providers.binding import ModelBinding, use_binding
|
|
from raven.spine.message import ChatType, Source
|
|
from raven.spine.turn import Origin, TurnRequest
|
|
from raven.token_wise import usage_context
|
|
from raven.token_wise.registry import StrategyRegistry
|
|
from raven.token_wise.usage_tracker import UsageTracker
|
|
|
|
# The real fetch, captured before conftest's autouse guard stubs it to {}.
|
|
_REAL_FETCH = rates._fetch_openrouter_models
|
|
|
|
|
|
class UsageProvider(LLMProvider):
|
|
"""Returns a fixed reply with a known usage snapshot. No tool calls."""
|
|
|
|
def __init__(self, model: str, prompt_tokens: int, completion_tokens: int, extra: dict | None = None):
|
|
super().__init__(api_key="test")
|
|
self._model = model
|
|
self._usage = {
|
|
"prompt_tokens": prompt_tokens,
|
|
"completion_tokens": completion_tokens,
|
|
"total_tokens": prompt_tokens + completion_tokens,
|
|
**(extra or {}),
|
|
}
|
|
|
|
async def chat(
|
|
self,
|
|
messages,
|
|
tools=None,
|
|
model=None,
|
|
max_tokens=4096,
|
|
temperature=0.7,
|
|
reasoning_effort=None,
|
|
tool_choice=None,
|
|
):
|
|
return LLMResponse(content="ok", finish_reason="stop", usage=self._usage)
|
|
|
|
def get_default_model(self) -> str:
|
|
return self._model
|
|
|
|
|
|
@pytest.fixture
|
|
def workspace():
|
|
with tempfile.TemporaryDirectory() as td:
|
|
yield Path(td)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _reset_openrouter_cache():
|
|
rates.reset_openrouter_cache()
|
|
yield
|
|
rates.reset_openrouter_cache()
|
|
|
|
|
|
def _make_agent(
|
|
workspace: Path,
|
|
provider: LLMProvider,
|
|
model: str,
|
|
window: int | None,
|
|
*,
|
|
strategies: StrategyRegistry | None = None,
|
|
context_config: ContextConfig | None = None,
|
|
) -> AgentLoop:
|
|
kwargs: dict = {}
|
|
if window is not None or strategies is not None or context_config is not None:
|
|
kwargs["engine"] = EngineWiring(
|
|
context_window_tokens=window,
|
|
strategies=strategies,
|
|
context_config=context_config,
|
|
)
|
|
return AgentLoop(
|
|
provider=provider,
|
|
workspace=workspace,
|
|
model=model,
|
|
**kwargs,
|
|
policy=TurnPolicy(max_iterations=2),
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_usage_sink_carries_context_gauge_and_cost(workspace):
|
|
"""A non-openrouter model fills used/percent against the configured window."""
|
|
provider = UsageProvider("stub", prompt_tokens=6000, completion_tokens=2000)
|
|
agent = _make_agent(workspace, provider, model="stub", window=40000)
|
|
sink: dict = {}
|
|
|
|
await agent._process_message(
|
|
TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
text="hi",
|
|
),
|
|
session_key="s1",
|
|
usage_sink=sink,
|
|
)
|
|
|
|
assert sink["context_max"] == 40000
|
|
assert sink["context_used"] == 8000
|
|
assert sink["context_percent"] == 20
|
|
assert sink["cost_usd"] is None
|
|
assert sink["cost_missing_calls"] == 1
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
("extra", "used", "percent"),
|
|
[
|
|
({"cache_read_input_tokens": 30000, "total_tokens": 38000, "prompt_tokens_include_cache": False}, 38000, 95),
|
|
({"cache_read_input_tokens": 30000}, 8000, 20),
|
|
],
|
|
ids=["cache_reported_apart_from_prompt_tokens", "cache_already_inside_prompt_tokens"],
|
|
)
|
|
async def test_the_gauge_measures_the_window_not_the_bill(workspace, extra, used, percent):
|
|
"""A warm cache bills a fraction of the prompt and occupies all of it.
|
|
|
|
The gauge divides by the context window, so it must count the cached prompt
|
|
the provider reports apart from ``prompt_tokens`` -- and must not count it
|
|
twice where ``prompt_tokens`` already contains it, which is why the
|
|
add-back reads the provider's own ``prompt_tokens_include_cache``. The
|
|
billed counts on the wire are untouched either way.
|
|
"""
|
|
provider = UsageProvider("stub", prompt_tokens=6000, completion_tokens=2000, extra=extra)
|
|
agent = _make_agent(workspace, provider, model="stub", window=40000)
|
|
sink: dict = {}
|
|
|
|
await agent._process_message(
|
|
TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
text="hi",
|
|
),
|
|
session_key="s1",
|
|
usage_sink=sink,
|
|
)
|
|
|
|
assert sink["context_used"] == used
|
|
assert sink["context_percent"] == percent
|
|
assert sink["prompt_tokens"] == 6000
|
|
assert sink["completion_tokens"] == 2000
|
|
|
|
|
|
def _patch_live_openrouter_window(monkeypatch, window: int) -> None:
|
|
"""Route the OpenRouter models fetch to report ``window`` for deepseek-v4-pro."""
|
|
models = [
|
|
{
|
|
"id": "deepseek/deepseek-v4-pro",
|
|
"context_length": window,
|
|
"pricing": {"prompt": "0.0000005", "completion": "0.0000015"},
|
|
}
|
|
]
|
|
|
|
def handler(_req):
|
|
return httpx.Response(200, content=json.dumps({"data": models}))
|
|
|
|
real_client = httpx.Client
|
|
|
|
def client_factory(*args, **kwargs):
|
|
kwargs.setdefault("transport", httpx.MockTransport(handler))
|
|
return real_client(*args, **kwargs)
|
|
|
|
monkeypatch.setattr(rates, "_fetch_openrouter_models", _REAL_FETCH)
|
|
monkeypatch.setattr(rates.httpx, "Client", client_factory)
|
|
monkeypatch.setattr(rates, "_OPENROUTER_CACHE_TIME", 0.0)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_usage_sink_context_max_from_live_openrouter(workspace, monkeypatch):
|
|
"""Without an explicit window, an OpenRouter model LiteLLM lags on gets its real window from /models."""
|
|
_patch_live_openrouter_window(monkeypatch, 163840)
|
|
|
|
provider = UsageProvider("openrouter/deepseek/deepseek-v4-pro", 1000, 500)
|
|
agent = _make_agent(
|
|
workspace,
|
|
provider,
|
|
model="openrouter/deepseek/deepseek-v4-pro",
|
|
window=None,
|
|
)
|
|
sink: dict = {}
|
|
|
|
await agent._process_message(
|
|
TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
text="hi",
|
|
),
|
|
session_key="s1",
|
|
usage_sink=sink,
|
|
)
|
|
|
|
assert sink["context_max"] == 163840
|
|
assert sink["context_used"] == 1500
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_usage_sink_context_max_stays_explicit_over_live_openrouter(workspace, monkeypatch):
|
|
"""An explicitly configured window wins even when the model's live window disagrees."""
|
|
_patch_live_openrouter_window(monkeypatch, 163840)
|
|
|
|
provider = UsageProvider("openrouter/deepseek/deepseek-v4-pro", 1000, 500)
|
|
agent = _make_agent(
|
|
workspace,
|
|
provider,
|
|
model="openrouter/deepseek/deepseek-v4-pro",
|
|
window=8192,
|
|
)
|
|
sink: dict = {}
|
|
|
|
await agent._process_message(
|
|
TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
text="hi",
|
|
),
|
|
session_key="s1",
|
|
usage_sink=sink,
|
|
)
|
|
|
|
assert sink["context_max"] == 8192
|
|
assert sink["context_used"] == 1500
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_usage_sink_context_max_is_resolved_off_the_event_loop_thread(workspace, monkeypatch):
|
|
"""SF10: this per-call tier defaults to ``allow_fetch=True``, so a cold
|
|
OpenRouter model with both caches expired can reach for a synchronous
|
|
~10s HTTP call. ``_run_agent_loop`` runs on the event loop, so that call
|
|
must run on a worker thread, not inline."""
|
|
seen: dict[str, threading.Thread] = {}
|
|
|
|
def fake_resolve(model: str) -> int:
|
|
seen["thread"] = threading.current_thread()
|
|
return 99_999
|
|
|
|
monkeypatch.setattr(agent_loop_main, "resolve_context_window", fake_resolve)
|
|
|
|
provider = UsageProvider("stub", 1000, 500)
|
|
agent = _make_agent(workspace, provider, model="stub", window=None)
|
|
sink: dict = {}
|
|
|
|
await agent._process_message(
|
|
TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
text="hi",
|
|
),
|
|
session_key="s1",
|
|
usage_sink=sink,
|
|
)
|
|
|
|
assert sink["context_max"] == 99_999
|
|
assert seen["thread"] is not threading.current_thread()
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# construction-time ladder: _context_window_explicit + refresh_context_window #
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
def test_no_configured_window_resolves_via_the_ladder_and_is_not_explicit(workspace):
|
|
"""An unresolvable model falls back to the documented default, not explicit."""
|
|
provider = UsageProvider("stub", 0, 0)
|
|
agent = _make_agent(workspace, provider, model="stub", window=None)
|
|
|
|
assert agent.default_binding.configured_window is None
|
|
assert agent.context_window_tokens == rates.DEFAULT_CONTEXT_WINDOW_TOKENS
|
|
|
|
|
|
def test_a_configured_window_is_explicit_at_construction(workspace):
|
|
provider = UsageProvider("stub", 0, 0)
|
|
agent = _make_agent(workspace, provider, model="stub", window=8192)
|
|
|
|
assert agent.default_binding.configured_window == 8192
|
|
assert agent.context_window_tokens == 8192
|
|
|
|
|
|
def test_a_pinned_window_outranks_the_model_a_session_switched_to(workspace, monkeypatch):
|
|
"""A pin is a deliberate override; the model a session runs on cannot discard it."""
|
|
_patch_live_openrouter_window(monkeypatch, 163840)
|
|
|
|
provider = UsageProvider("stub", 0, 0)
|
|
agent = _make_agent(workspace, provider, model="stub", window=8192)
|
|
|
|
switched = ModelBinding(provider, "openrouter/deepseek/deepseek-v4-pro", agent.default_binding.configured_window)
|
|
with use_binding(switched):
|
|
assert agent.context_window_tokens == 8192
|
|
|
|
|
|
def test_the_window_follows_the_model_the_turn_is_bound_to(workspace, monkeypatch):
|
|
"""Without an explicit pin, the window is whatever the turn's own model holds.
|
|
|
|
The switch runs inside the running event loop, so the ladder is walked with
|
|
``allow_fetch=False`` -- an in-process cache entry of any age answers rather
|
|
than a live fetch. Populated directly rather than through a mocked network
|
|
call, which ``allow_fetch=False`` never reaches.
|
|
"""
|
|
provider = UsageProvider("stub", 0, 0)
|
|
agent = _make_agent(workspace, provider, model="stub", window=None)
|
|
assert agent.context_window_tokens == rates.DEFAULT_CONTEXT_WINDOW_TOKENS
|
|
|
|
# conftest's autouse guard stubs the fetch to a zero-argument lambda; restore
|
|
# the real one so allow_fetch=False's in-process-cache branch actually runs.
|
|
monkeypatch.setattr(rates, "_fetch_openrouter_models", _REAL_FETCH)
|
|
rates._OPENROUTER_CACHE["deepseek/deepseek-v4-pro"] = {
|
|
"pricing": {"prompt": "0.0000005", "completion": "0.0000015"},
|
|
"context_length": 163840,
|
|
}
|
|
monkeypatch.setattr(rates, "_OPENROUTER_CACHE_TIME", time.time() - rates._OPENROUTER_CACHE_TTL - 1000)
|
|
switched = ModelBinding(provider, "openrouter/deepseek/deepseek-v4-pro")
|
|
with use_binding(switched):
|
|
assert agent.context_window_tokens == 163840
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# allow_fetch=False: construction and refresh never touch the network #
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
def _forbid_network_client(monkeypatch):
|
|
"""Restore the real fetch, then count real ``httpx.Client`` builds.
|
|
|
|
Not a raise: ``_fetch_openrouter_models`` wraps the fetch in
|
|
``except Exception`` to degrade on a network failure, so a raise from here
|
|
would be swallowed as "the network failed" and the test would pass for the
|
|
wrong reason. A counter the caller asserts is 0 actually distinguishes
|
|
"never touched the network" from "touched it and degraded".
|
|
"""
|
|
counter = {"calls": 0}
|
|
real_client = rates.httpx.Client
|
|
|
|
def _counting_client(*args, **kwargs):
|
|
counter["calls"] += 1
|
|
return real_client(*args, **kwargs)
|
|
|
|
monkeypatch.setattr(rates, "_fetch_openrouter_models", _REAL_FETCH)
|
|
monkeypatch.setattr(rates.httpx, "Client", _counting_client)
|
|
return counter
|
|
|
|
|
|
def test_construction_on_an_openrouter_model_never_touches_the_network(workspace, monkeypatch, tmp_path):
|
|
"""The regression this fixes: constructing on an unmapped OpenRouter model
|
|
used to fetch synchronously, blocking startup for up to 10s."""
|
|
from raven.providers import model_catalog_cache
|
|
|
|
monkeypatch.setattr(model_catalog_cache, "_CACHE_PATH", tmp_path / "model-catalog.json", raising=False)
|
|
counter = _forbid_network_client(monkeypatch)
|
|
|
|
provider = UsageProvider("openrouter/deepseek/deepseek-v4-pro", 0, 0)
|
|
agent = _make_agent(workspace, provider, model="openrouter/deepseek/deepseek-v4-pro", window=None)
|
|
|
|
assert agent.context_window_tokens == rates.DEFAULT_CONTEXT_WINDOW_TOKENS
|
|
assert counter["calls"] == 0
|
|
|
|
|
|
def test_reading_the_window_for_an_openrouter_model_never_touches_the_network(workspace, monkeypatch, tmp_path):
|
|
"""The window is read inside a running turn, so resolving it must not freeze
|
|
the event loop on a synchronous fetch."""
|
|
from raven.providers import model_catalog_cache
|
|
|
|
monkeypatch.setattr(model_catalog_cache, "_CACHE_PATH", tmp_path / "model-catalog.json", raising=False)
|
|
|
|
provider = UsageProvider("stub", 0, 0)
|
|
agent = _make_agent(workspace, provider, model="stub", window=None)
|
|
|
|
counter = _forbid_network_client(monkeypatch)
|
|
switched = ModelBinding(provider, "openrouter/deepseek/deepseek-v4-pro")
|
|
with use_binding(switched):
|
|
assert agent.context_window_tokens == rates.DEFAULT_CONTEXT_WINDOW_TOKENS
|
|
assert counter["calls"] == 0
|
|
|
|
|
|
class EffortProvider(UsageProvider):
|
|
"""Records the effort each call was sent at."""
|
|
|
|
def __init__(self, model: str):
|
|
super().__init__(model, 10, 5)
|
|
self.efforts: list = []
|
|
|
|
async def chat(
|
|
self,
|
|
messages,
|
|
tools=None,
|
|
model=None,
|
|
max_tokens=4096,
|
|
temperature=0.7,
|
|
reasoning_effort=None,
|
|
tool_choice=None,
|
|
): # noqa: E501
|
|
self.efforts.append(reasoning_effort)
|
|
return await super().chat(messages, tools, model, max_tokens, temperature, reasoning_effort, tool_choice)
|
|
|
|
|
|
def _point_config(tmp_path: Path, monkeypatch, payload: dict) -> None:
|
|
path = tmp_path / "config.json"
|
|
path.write_text(json.dumps(payload), encoding="utf-8")
|
|
monkeypatch.setattr("raven.config.loader.get_config_path", lambda: path)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_call_runs_at_the_configured_effort(workspace, monkeypatch):
|
|
"""`agents.defaults.reasoningEffort` reaches the wire.
|
|
|
|
The provider holds a configured default of its own, but it is frozen when
|
|
the provider is constructed -- deliberately, so a credentials refresh
|
|
cannot import a live `agents` section -- so an edit reached it only at a
|
|
restart. The loop sends it explicitly instead.
|
|
"""
|
|
_point_config(workspace, monkeypatch, {"agents": {"defaults": {"reasoningEffort": "high"}}})
|
|
provider = EffortProvider("stub")
|
|
agent = _make_agent(workspace, provider, model="stub", window=None)
|
|
|
|
await agent._process_message(
|
|
TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
text="hi",
|
|
),
|
|
session_key="s1",
|
|
)
|
|
|
|
assert provider.efforts == ["high"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_no_configured_effort_sends_none(workspace, monkeypatch):
|
|
"""Nothing passed, so the provider's own default stands. Sending an
|
|
explicit None would switch that default off instead."""
|
|
_point_config(workspace, monkeypatch, {})
|
|
provider = EffortProvider("stub")
|
|
agent = _make_agent(workspace, provider, model="stub", window=None)
|
|
|
|
await agent._process_message(
|
|
TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
text="hi",
|
|
),
|
|
session_key="s1",
|
|
)
|
|
|
|
assert provider.efforts == [None]
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# the cost is the turn's: every iteration, and every call it delegated #
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
def _usage(cost: float | None) -> dict:
|
|
usage: dict = {"prompt_tokens": 1000, "completion_tokens": 200, "total_tokens": 1200}
|
|
if cost is not None:
|
|
usage["cost_usd"] = cost
|
|
return usage
|
|
|
|
|
|
def _says(text: str, cost: float | None) -> LLMResponse:
|
|
return LLMResponse(content=text, finish_reason="stop", usage=_usage(cost))
|
|
|
|
|
|
def _asks_for(tool: str, cost: float | None, **arguments) -> LLMResponse:
|
|
return LLMResponse(
|
|
content="",
|
|
tool_calls=[ToolCallRequest(id="c1", name=tool, arguments=arguments)],
|
|
usage=_usage(cost),
|
|
)
|
|
|
|
|
|
class ScriptedProvider(LLMProvider):
|
|
"""One scripted reply per call; the last entry answers any call after it."""
|
|
|
|
def __init__(self, model: str, script: list[LLMResponse]):
|
|
super().__init__(api_key="test")
|
|
self._model = model
|
|
self._script = script
|
|
self.calls = 0
|
|
|
|
async def chat(
|
|
self,
|
|
messages,
|
|
tools=None,
|
|
model=None,
|
|
max_tokens=4096,
|
|
temperature=0.7,
|
|
reasoning_effort=None,
|
|
tool_choice=None,
|
|
):
|
|
reply = self._script[min(self.calls, len(self._script) - 1)]
|
|
self.calls += 1
|
|
return reply
|
|
|
|
def get_default_model(self) -> str:
|
|
return self._model
|
|
|
|
|
|
class DelegatingTool(Tool):
|
|
"""Spends the way a dispatched sub-agent does: in a session of its own,
|
|
naming the delegating turn's session as its root (``usage_owner``, which
|
|
is what the ACP prompt carries over and the sub-agent's turn binds)."""
|
|
|
|
def __init__(self, tracker: UsageTracker, root: str, cost: float | None):
|
|
self._tracker = tracker
|
|
self._root = root
|
|
self._cost = cost
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return "delegate"
|
|
|
|
@property
|
|
def description(self) -> str:
|
|
return "Hand the task to a sub-agent."
|
|
|
|
@property
|
|
def parameters(self) -> dict:
|
|
return {"type": "object", "properties": {}}
|
|
|
|
async def execute(self, **_kwargs) -> str:
|
|
with usage_context.bind("acp:child", {"root_session_key": self._root}):
|
|
await self._tracker.after_llm_call({}, UsageSnapshot(model="child/model", cost_usd=self._cost))
|
|
return "the sub-agent answered"
|
|
|
|
|
|
def _costing_agent(
|
|
workspace: Path,
|
|
script: list[LLMResponse],
|
|
*,
|
|
strategies: StrategyRegistry | None = None,
|
|
) -> AgentLoop:
|
|
"""An agent whose every billed call is the loop's own.
|
|
|
|
The Curator is dropped because its Slow Path is a bounded agent loop of its
|
|
own on the same provider, and a scripted provider cannot tell its calls from
|
|
the turn's. Its calls, like the watch-work judgement's, reach the usage
|
|
recorder only through the provider seam (``raven.providers.usage_record``),
|
|
which a test here installs only when the seam is what it pins.
|
|
"""
|
|
return _make_agent(
|
|
workspace,
|
|
ScriptedProvider("stub", script),
|
|
model="stub",
|
|
window=40000,
|
|
strategies=strategies,
|
|
context_config=ContextConfig(drop_segments=["curator"]),
|
|
)
|
|
|
|
|
|
async def _turn(agent: AgentLoop, sink: dict, session_key: str = "s1") -> None:
|
|
await agent._process_message(
|
|
TurnRequest(
|
|
origin=Origin.USER,
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
text="hi",
|
|
),
|
|
session_key=session_key,
|
|
usage_sink=sink,
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_turn_costs_every_iteration_it_ran(workspace):
|
|
"""Two model calls, two prices, one turn: the bar shows their sum.
|
|
|
|
The sink used to carry the final iteration's cost, so a turn that called a
|
|
tool and then answered reported only the answer.
|
|
"""
|
|
agent = _costing_agent(workspace, [_asks_for("list_dir", 0.002, path="."), _says("done", 0.003)])
|
|
sink: dict = {}
|
|
|
|
await _turn(agent, sink)
|
|
|
|
assert sink["cost_usd"] == pytest.approx(0.005), "the tool-calling iteration, plus the one that answered"
|
|
assert sink["cost_missing_calls"] == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_delegating_turn_costs_what_it_delegated(workspace):
|
|
"""The sub-agent's spend is the turn's spend.
|
|
|
|
A sub-agent bills in its own session, so nothing on the delegating turn's
|
|
own calls can see it; it names this turn's session as its root, and the
|
|
recorder every billed call passes bills it here as well.
|
|
"""
|
|
tracker = UsageTracker(persist=False)
|
|
agent = _costing_agent(
|
|
workspace,
|
|
[_asks_for("delegate", 0.002), _says("done", 0.003)],
|
|
strategies=StrategyRegistry([tracker]),
|
|
)
|
|
agent.tools.register(DelegatingTool(tracker, root="s1", cost=0.02))
|
|
sink: dict = {}
|
|
|
|
await _turn(agent, sink)
|
|
|
|
assert sink["cost_usd"] == pytest.approx(0.025), "0.002 + 0.003 of its own, plus the 0.02 it delegated"
|
|
assert sink["cost_missing_calls"] == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_turn_is_billed_once_while_the_provider_seam_listens_too(workspace):
|
|
"""Production installs the loop's own registry at the provider seam as well
|
|
as binding it for the turn, so the seam and the loop hear the same call. The
|
|
seam's row is the one kept, under the turn's session, and the loop adds
|
|
none."""
|
|
tracker = UsageTracker(persist=False)
|
|
registry = StrategyRegistry([tracker])
|
|
usage_record.install(registry.after_llm_call)
|
|
agent = _costing_agent(
|
|
workspace,
|
|
[_asks_for("list_dir", 0.002, path="."), _says("done", 0.003)],
|
|
strategies=registry,
|
|
)
|
|
sink: dict = {}
|
|
|
|
await _turn(agent, sink)
|
|
|
|
assert agent.provider.calls == 2
|
|
billed = tracker.snapshot("s1")
|
|
assert billed.calls == 2, "one row per model call, not one from the loop and one from the seam"
|
|
assert billed.cost_usd == pytest.approx(0.005)
|
|
assert tracker.total.calls == 2
|
|
assert sink["cost_usd"] == pytest.approx(0.005)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_serving_generation_is_billed_alone_after_a_candidate_installs(workspace):
|
|
"""A candidate generation's assembly installs its registry process-wide
|
|
before it is known to be usable, and the generation it would replace goes
|
|
on serving. That generation's turn is billed to its own registry only: the
|
|
seam used to record it into the candidate's, and the loop, no longer
|
|
matching the installed sink, recorded it again into its own."""
|
|
serving_tracker = UsageTracker(persist=False)
|
|
serving = StrategyRegistry([serving_tracker])
|
|
candidate_tracker = UsageTracker(persist=False)
|
|
usage_record.install(StrategyRegistry([candidate_tracker]).after_llm_call)
|
|
agent = _costing_agent(
|
|
workspace,
|
|
[_asks_for("list_dir", 0.002, path="."), _says("done", 0.003)],
|
|
strategies=serving,
|
|
)
|
|
sink: dict = {}
|
|
|
|
await _turn(agent, sink)
|
|
|
|
assert agent.provider.calls == 2
|
|
assert serving_tracker.total.calls == 2, "each call once, to the generation that made it"
|
|
assert candidate_tracker.total.calls == 0, "the candidate billed nothing it did not make"
|
|
|
|
|
|
class _BestOfTwoAction(DefaultAction):
|
|
"""An Action that asks twice and keeps the second answer -- the shape
|
|
``ActionRequest`` names (best-of-n, a critic pass): one decision, two
|
|
requests, one response handed back to the loop."""
|
|
|
|
async def decide(self, request):
|
|
await super().decide(request)
|
|
return await super().decide(request)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_every_call_a_multi_call_action_makes_is_billed(workspace):
|
|
"""The loop records the one response ``decide`` returns. A claim over the
|
|
whole decision would bill that one and lose the other request; the seam
|
|
records both, under the turn's session, and the loop adds nothing."""
|
|
tracker = UsageTracker(persist=False)
|
|
registry = StrategyRegistry([tracker])
|
|
usage_record.install(registry.after_llm_call)
|
|
agent = _costing_agent(workspace, [_says("draft", 0.002), _says("final", 0.003)], strategies=registry)
|
|
agent.harness = replace(agent.harness, action=_BestOfTwoAction())
|
|
sink: dict = {}
|
|
|
|
await _turn(agent, sink)
|
|
|
|
assert agent.provider.calls == 2
|
|
billed = tracker.snapshot("s1")
|
|
assert billed.calls == 2, "both requests of the one decision, each once"
|
|
assert billed.cost_usd == pytest.approx(0.005)
|
|
assert tracker.total.calls == 2
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_turn_that_ends_unpriced_still_reports_what_it_knows(workspace):
|
|
"""An unpriced call is unknown cost, not free, and it does not erase the
|
|
price of the calls beside it -- theirs is a sum, its own is a count."""
|
|
agent = _costing_agent(workspace, [_asks_for("list_dir", 0.002, path="."), _says("done", None)])
|
|
sink: dict = {}
|
|
|
|
await _turn(agent, sink)
|
|
|
|
assert sink["cost_usd"] == pytest.approx(0.002)
|
|
assert sink["cost_missing_calls"] == 1
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_delegation_that_reports_no_price_is_counted_not_dropped(workspace):
|
|
"""The sub-agent ran on a model with no published price: the turn says how
|
|
many calls it cannot price rather than billing them as zero."""
|
|
tracker = UsageTracker(persist=False)
|
|
agent = _costing_agent(
|
|
workspace,
|
|
[_asks_for("delegate", 0.002), _says("done", 0.003)],
|
|
strategies=StrategyRegistry([tracker]),
|
|
)
|
|
agent.tools.register(DelegatingTool(tracker, root="s1", cost=None))
|
|
sink: dict = {}
|
|
|
|
await _turn(agent, sink)
|
|
|
|
assert sink["cost_usd"] == pytest.approx(0.005)
|
|
assert sink["cost_missing_calls"] == 1
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_turn_is_not_billed_for_a_delegation_of_another_session(workspace):
|
|
"""The scope is found by root session key, so a sub-agent working for a
|
|
turn in another conversation bills that one and not this one."""
|
|
tracker = UsageTracker(persist=False)
|
|
agent = _costing_agent(
|
|
workspace,
|
|
[_asks_for("delegate", 0.002), _says("done", 0.003)],
|
|
strategies=StrategyRegistry([tracker]),
|
|
)
|
|
agent.tools.register(DelegatingTool(tracker, root="another-session", cost=0.02))
|
|
sink: dict = {}
|
|
|
|
await _turn(agent, sink)
|
|
|
|
assert sink["cost_usd"] == pytest.approx(0.005)
|