Raven-1/tests/test_provider_protocol.py
Zuyi Zhou 4728991883
fix(providers): retry a refused forced tool choice under auto (#870)
## Summary

On claude-sonnet-5.5 every call that forces a tool is refused with a 400
("tool_choice: type "tool" and "any" are not supported for this
model."). The research verify gate forces `review_verdict`, so on both
deep-research runs of 2026-09-30 its reviewer returned an error, the
gate failed open, and the reports shipped marked "shipped unreviewed".
The same gate returned real verdicts earlier (acp.log of 2026-09-18:
"draft rejected (4 unsupported claim(s))").

`rewrite_on_400` already repairs the other complaints a vendor can make
about a request and retries once. This adds one more: when the 400 names
`tool_choice` and the request forced a tool (`tool` or `any`), the
choice becomes `auto` and the request is sent again. The tool stays on
offer; a model that declines it answers in plain text, which the verify
gate already parses. Both call sites (`chat` and `chat_stream`) reach
the repair through their existing 400 handling.

Why here and not in the gate: the refusal is a property of the model and
the wire, and any caller that forces a tool meets it. A repair in the
transport needs no model list and no assumption about which models
think. The cost is one refused round trip per forced call on a model
that refuses; nothing is remembered between calls.

Scope: only the repair table in
`raven/providers/anthropic_messages_provider.py` and its tests. Nothing
under `agents/` changes, so the research flow label, the stamped prompts
and the twin-parity tests are untouched.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Base is origin/main b92afe133; this branch is two commits on top of it
(the fix, then a comment and docstring rewording). The suite and
coverage runs below were taken on the first commit; the second changes
one comment and one docstring, and the provider test file was rerun
after it.

```
uv run pytest tests/test_provider_protocol.py -q -k "forced_tool_choice"
  3 passed
uv run pytest tests/test_provider_protocol.py tests/test_provider_prompt_cache.py tests/test_provider_catalog.py tests/test_providers_first_byte_budget.py tests/test_providers_stream_idle_budget.py -q
  262 passed, 7 skipped
COLORTERM=truecolor uv run --frozen --python 3.12 --all-extras pytest -q
  3 failed, 27570 passed, 112 skipped
uv run pytest tests/test_provider_protocol.py -q      (after the rewording commit)
  106 passed
COLORTERM=truecolor make coverage && COVERAGE_BASE_REF=origin/main make coverage-diff
  make coverage exits 1 only because of the same 3 failures; coverage-diff: Diff coverage: 100.00% (4/4 executable changed lines), threshold 90
```

The 3 failures also fail on an untouched checkout of origin/main
(b92afe133), so they are not caused by this change; all three read the
machine rather than the code:
- `tests/test_rpc_files.py::test_a_host_without_libreoffice_says_so`
(LibreOffice is installed here, the test expects 503 and gets 500)
-
`tests/test_simulation_scenario.py::test_a_fresh_trial_refreshes_the_subagent_homes_before_it_runs`
(`user.md` and `USER.md` collide on a case-insensitive filesystem)
-
`tests/test_simulation_suite.py::test_a_stopped_run_is_marked_in_its_record_and_finished_with_its_evidence`
(child process returned -15)

Revert-to-red: restoring
`raven/providers/anthropic_messages_provider.py` from origin/main with
the new tests kept turns all 3 new tests red; restoring the fix turns
them green.

Mutation matrix, one mutant per guard, run against the whole of
`tests/test_provider_protocol.py`:

```
M1 drop the "tool_choice" in text guard       1 failed
M2 drop the forced-type guard                  1 failed
M3 rewrite to "none" instead of "auto"         2 failed
M4 return no repair name                       2 failed
CONTROL break the ceiling repair's name        1 failed   (the run reaches this file)
```

Lint and gates, all clean at this head: ruff check, ruff format --check,
`make lint-imports`, `make lint-deps`, `make lint-types`, `git diff
--check origin/main...HEAD`, `scripts/check_source_language.py
origin/main...HEAD`, `make check-commits`, `make check-pr-title`, `make
check-large-files`, and `git merge-tree --write-tree HEAD origin/main`
(clean).

Not run: a live request to OpenRouter, so the verify gate's handling of
the plain-text reply it will now get on this model has been read in
`research_flow/gates/verify.py` (it falls back to `_parse_verdict`) but
not exercised against the real model. The refusal text and the provider
class were taken from the recorded failing span of 2026-09-30 (audit
span `llm.call`, status ERROR) (`AnthropicMessagesProvider`, `thinking:
adaptive`, `output_config.effort=low`); the retry is exercised against a
mocked transport on both the streaming and the non-streaming path.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

A caller that forces a tool and cannot cope with a reply that has no
tool call would now see such a reply on a model that refuses forced
calls. Before this change the same call failed with an error, so no
working path is affected. The research verify gate is the known caller
and parses a text verdict. Rollback is a plain revert of this commit.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-sonnet-5-5) <noreply@anthropic.com>
2026-10-08 17:11:34 +08:00

1196 lines
56 KiB
Python

"""Tests for model wire protocol defaults and explicit overrides."""
import json
from dataclasses import replace
import httpx
import pytest
from raven.config.schema import Config
from raven.providers.anthropic_messages_provider import AnthropicMessagesProvider, convert_messages
from raven.providers.factory import make_provider
from raven.providers.litellm_provider import LiteLLMProvider
from raven.providers.openai_responses_provider import OpenAIResponsesProvider
from raven.providers.protocol import effective_protocol
from raven.providers.wire import wire_model
def _config(model: str, *, override: str | None = None) -> Config:
custom = {
"apiKey": "test-key",
"apiBase": "https://openrouter.ai/api/v1",
"models": [model],
}
if override is not None:
custom["modelProtocols"] = {model: override}
return Config.model_validate(
{
"agents": {"defaults": {"model": model, "provider": "custom"}},
"providers": {"custom": custom},
}
)
def _provider_config(provider: str, model: str, *, override: str | None = None) -> Config:
key = "sk-or-test" if provider == "openrouter" else "test-key"
section = {"apiKey": key, "models": [model]}
if override is not None:
section["modelProtocols"] = {model: override}
return Config.model_validate(
{
"agents": {"defaults": {"model": model, "provider": provider}},
"providers": {provider: section},
}
)
def test_model_protocol_defaults_and_overrides() -> None:
config = _config("openai/gpt-5.6-sol")
assert effective_protocol(config.providers.custom, "openai/gpt-5.6-sol") == "responses"
assert effective_protocol(config.providers.custom, "anthropic/claude-opus-5") == "anthropic"
assert effective_protocol(config.providers.custom, "google/gemini-3") == "chat"
assert (
effective_protocol(_config("openai/gpt-5.6-sol", override="chat").providers.custom, "openai/gpt-5.6-sol")
== "chat"
)
@pytest.mark.parametrize("model", ["gemini-2.5-flash", "unknown", "llama-4", "o3", None])
def test_unlisted_models_default_to_chat(model):
assert effective_protocol(None, model) == "chat"
assert effective_protocol({}, model) == "chat"
@pytest.mark.parametrize(
"model",
[
"qwen3.8-max",
"Qwen/Qwen3-Coder",
"seed-2.0",
"doubao-seed-2-0-pro",
"MiniMax-M3",
"deepseek-chat",
"deepseek/deepseek-v4-pro",
"openai/gpt-5",
"kimi-k3",
"openrouter/moonshotai/kimi-k2.5",
],
)
def test_response_series_defaults_preserve_manual_choices(model):
assert effective_protocol(None, model) == "responses"
for protocol in ("chat", "responses", "anthropic"):
assert effective_protocol({"protocol": protocol}, model) == protocol
assert effective_protocol({"model_protocols": {model: protocol}}, model) == protocol
def test_claude_default_and_explicit_protocol_priority():
assert effective_protocol(None, "anthropic/claude-opus-5") == "anthropic"
assert effective_protocol({"protocol": "chat"}, "claude-opus-5") == "chat"
assert effective_protocol({"protocol": "responses"}, "claude-opus-5") == "responses"
assert isinstance(make_provider(_config("anthropic/claude-opus-5")), AnthropicMessagesProvider)
def test_factory_selects_protocol_adapter() -> None:
assert isinstance(make_provider(_config("openai/gpt-5.6-sol")), OpenAIResponsesProvider)
assert isinstance(make_provider(_config("anthropic/claude-opus-5")), AnthropicMessagesProvider)
assert isinstance(make_provider(_config("openai/gpt-5.6-sol", override="chat")), LiteLLMProvider)
def test_openrouter_selects_native_protocol_adapters() -> None:
response = make_provider(_provider_config("openrouter", "openrouter/openai/gpt-5.6-sol"))
messages = make_provider(_provider_config("openrouter", "openrouter/anthropic/claude-opus-5"))
assert isinstance(response, OpenAIResponsesProvider)
assert isinstance(messages, AnthropicMessagesProvider)
assert response.wire_model_id("openrouter/openai/gpt-5.6-sol") == "openai/gpt-5.6-sol"
assert messages.wire_model_id("openrouter/anthropic/claude-opus-5") == "anthropic/claude-opus-5"
def test_native_clients_strip_their_storage_prefix() -> None:
assert wire_model("openai/gpt-5.6-sol", client_provider="openai") == "gpt-5.6-sol"
assert wire_model("anthropic/claude-opus-5", client_provider="anthropic") == "claude-opus-5"
def _anthropic_sse(*events: dict) -> bytes:
return "".join(f"event: {e['type']}\ndata: {json.dumps(e)}\n\n" for e in events).encode()
async def test_anthropic_stream_carries_thinking_blocks_and_tool_calls() -> None:
"""A thinking turn that ends in a tool call used to raise inside the stream
consumer: the terminal delta handed ``thinking_blocks`` to a ChatDelta that
had no such field, so every claude turn with extended thinking died with
``unexpected keyword argument`` and the page showed only the words said before
the tool call."""
from raven.providers.anthropic_messages_provider import consume_message_stream
body = _anthropic_sse(
{"type": "message_start", "message": {"usage": {"input_tokens": 12}}},
{
"type": "content_block_start",
"index": 0,
"content_block": {"type": "thinking", "thinking": "", "signature": ""},
},
{"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "fetch first"}},
{"type": "content_block_delta", "index": 0, "delta": {"type": "signature_delta", "signature": "sig-1"}},
{"type": "content_block_stop", "index": 0},
{"type": "content_block_start", "index": 1, "content_block": {"type": "text", "text": ""}},
{"type": "content_block_delta", "index": 1, "delta": {"type": "text_delta", "text": "Looking it up."}},
{"type": "content_block_stop", "index": 1},
{
"type": "content_block_start",
"index": 2,
"content_block": {"type": "tool_use", "id": "toolu_1", "name": "web_fetch", "input": {}},
},
{
"type": "content_block_delta",
"index": 2,
"delta": {"type": "input_json_delta", "partial_json": '{"url": "https://ex'},
},
{
"type": "content_block_delta",
"index": 2,
"delta": {"type": "input_json_delta", "partial_json": 'ample.com"}'},
},
{"type": "content_block_stop", "index": 2},
{"type": "message_delta", "delta": {"stop_reason": "tool_use"}, "usage": {"output_tokens": 30}},
{"type": "message_stop"},
)
deltas = [d async for d in consume_message_stream(httpx.Response(200, content=body), timeout=5.0)]
terminal = deltas[-1]
assert terminal.finish_reason == "tool_calls"
assert terminal.thinking_blocks == [{"type": "thinking", "thinking": "fetch first", "signature": "sig-1"}]
assert "".join(d.content or "" for d in deltas) == "Looking it up."
fragments = [d.tool_call_delta["tool_calls"][0] for d in deltas if d.tool_call_delta]
assert fragments[0]["function"]["name"] == "web_fetch"
assert "".join(f["function"]["arguments"] for f in fragments) == '{"url": "https://example.com"}'
async def test_stream_accumulator_keeps_the_thinking_blocks_for_the_next_turn() -> None:
"""The signed blocks must come back on the response the loop stores, or the
next Anthropic request replays a tool_use turn without its thinking and is
refused."""
from raven.contracts.llm_provider import ChatDelta
from raven.providers.streaming import stream_llm_call
blocks = [{"type": "thinking", "thinking": "plan", "signature": "sig"}]
class _Stub:
generation = None
async def chat_stream(self, **_kwargs):
yield ChatDelta(content=None, reasoning_content="plan")
yield ChatDelta(content="done", thinking_blocks=None)
yield ChatDelta(content=None, finish_reason="stop", usage={"output_tokens": 3}, thinking_blocks=blocks)
response = await stream_llm_call(_Stub(), messages=[{"role": "user", "content": "hi"}], tools=None, model="m")
assert response.content == "done"
assert response.thinking_blocks == blocks
assert response.reasoning_content == "plan"
def test_anthropic_parse_message_ignores_the_response_id() -> None:
"""The non-streaming parser handed ``response_id`` to LLMResponse, a field the
contract does not have, so every Visual Domain Selector call on a claude model
failed and the selector degraded to its full catalog."""
from raven.providers.anthropic_messages_provider import parse_message
response = parse_message(
{
"id": "msg_1",
"stop_reason": "end_turn",
"usage": {"input_tokens": 3, "output_tokens": 2},
"content": [{"type": "text", "text": "ok"}],
}
)
assert response.content == "ok" and response.finish_reason == "stop"
def test_anthropic_requests_end_on_the_user_side() -> None:
"""A hook note appended after the last exchange is a trailing assistant
message; claude models refuse that as prefill, so it crosses as user text."""
from raven.providers.anthropic_messages_provider import convert_messages
_, converted = convert_messages(
[
{"role": "user", "content": "draw the poster"},
{"role": "assistant", "content": "starting"},
{"role": "user", "content": "go on"},
{"role": "assistant", "content": "<task_state>items: 1, 2</task_state>"},
]
)
assert [m["role"] for m in converted] == ["user", "assistant", "user"]
assert converted[-1]["content"][-1]["text"] == "<task_state>items: 1, 2</task_state>"
def _anthropic_body(**overrides):
from raven.providers.anthropic_messages_provider import build_request_body
kwargs = dict(
model="claude-opus-5",
messages=[{"role": "user", "content": "write the page"}],
tools=None,
max_tokens=None,
temperature=None,
reasoning_effort="high",
tool_choice=None,
stream=True,
)
kwargs.update(overrides)
return build_request_body(**kwargs)
def test_anthropic_default_output_ceiling_leaves_room_for_the_file() -> None:
"""Every request went out with max_tokens 4096 unless the operator set one: a
poster's HTML was cut at the ceiling on every write."""
body = _anthropic_body(model="claude-haiku-4-5")
assert body["max_tokens"] == 64000
assert _anthropic_body(model="claude-haiku-4-5", max_tokens=4096)["max_tokens"] == 4096
assert _anthropic_body()["max_tokens"] == 64000
def test_anthropic_sends_the_effort_as_a_label_not_a_token_count() -> None:
"""Raven sizes no thinking budget. The effort goes out as the name the caller
gave it and whoever serves the model decides what it buys -- the table that used
to translate it here was half of a 16384 ceiling that no longer applies."""
labelled = _anthropic_body(model="claude-haiku-4-5", reasoning_effort="high")
assert labelled["reasoning"] == {"effort": "high"} and "thinking" not in labelled
assert _anthropic_body(model="z-ai/glm-5.3-flash", reasoning_effort="max")["reasoning"] == {"effort": "max"}
assert "reasoning" not in _anthropic_body(model="claude-haiku-4-5", reasoning_effort=None)
def test_anthropic_keeps_the_temperature_beside_a_labelled_effort() -> None:
"""Only Anthropic's own thinking request refuses a temperature. A pinned effort
used to send one of those on every model, so pinning a tier silently dropped the
repository's temperature and the tiers differed by more than the pin."""
labelled = _anthropic_body(model="z-ai/glm-5.3-flash", reasoning_effort="high", temperature=0.95)
assert labelled["reasoning"] == {"effort": "high"} and labelled["temperature"] == 0.95
adaptive = _anthropic_body(model="anthropic/claude-opus-5", reasoning_effort="high", temperature=0.95)
assert adaptive["thinking"] == {"type": "adaptive"} and "temperature" not in adaptive
def test_anthropic_thinks_adaptively_on_the_models_that_require_it() -> None:
"""Claude 4.7 and later take depth as ``output_config.effort`` beside
``thinking.type: "adaptive"``. The model id decides, prefixed or not."""
adaptive = _anthropic_body(model="anthropic/claude-opus-5", reasoning_effort="xhigh")
assert adaptive["thinking"] == {"type": "adaptive"} and adaptive["output_config"] == {"effort": "xhigh"}
assert "temperature" not in adaptive
assert _anthropic_body(model="claude-fable-5-1", reasoning_effort="minimal")["output_config"] == {"effort": "low"}
assert "thinking" not in _anthropic_body(model="claude-opus-5", reasoning_effort=None)
def test_anthropic_repairs_the_thinking_mode_a_400_complains_about() -> None:
"""An id the family regex cannot read is guessed wrong half the time; the
400 names the mode the model wants, and the body is rewritten for one retry."""
from raven.providers.anthropic_messages_provider import rewrite_on_400
body = _anthropic_body(model="claude-3-7-sonnet-latest", reasoning_effort="high")
body["thinking"], body["output_config"] = {"type": "enabled", "budget_tokens": 4096}, {"effort": "high"}
assert rewrite_on_400(body, 'thinking.type.enabled is not supported on this model; use thinking.type: "adaptive"')
assert body["thinking"] == {"type": "adaptive"} and body["output_config"] == {"effort": "high"}
level = _anthropic_body(model="claude-sonnet-5", reasoning_effort="xhigh")
assert rewrite_on_400(level, "output_config.effort: xhigh is not a valid effort level for this model")
assert "output_config" not in level and level["thinking"] == {"type": "adaptive"}
assert not rewrite_on_400(level, "something else entirely")
_VERDICT_TOOLS = [
{
"type": "function",
"function": {
"name": "review_verdict",
"description": "Record the verdict.",
"parameters": {"type": "object", "properties": {"pass": {"type": "boolean"}}, "required": ["pass"]},
},
}
]
_FORCED_VERDICT = {"type": "function", "function": {"name": "review_verdict"}}
_FORCED_TOOL_REFUSAL = 'tool_choice: type "tool" and "any" are not supported for this model.'
def test_anthropic_repairs_a_forced_tool_choice_a_400_refuses() -> None:
"""A model can refuse a call it is forced to make, and the 400 says so.
The tool stays on offer under ``auto``; only a forced choice is rewritten."""
from raven.providers.anthropic_messages_provider import rewrite_on_400
for choice, forced_type in ((_FORCED_VERDICT, "tool"), ("required", "any")):
body = _anthropic_body(
model="claude-sonnet-5-5", reasoning_effort="low", tools=_VERDICT_TOOLS, tool_choice=choice
)
assert body["tool_choice"]["type"] == forced_type
assert rewrite_on_400(body, _FORCED_TOOL_REFUSAL) == "tool_choice"
assert body["tool_choice"] == {"type": "auto"} and body["tools"][0]["name"] == "review_verdict"
already_auto = _anthropic_body(tools=_VERDICT_TOOLS, tool_choice="auto")
assert rewrite_on_400(already_auto, _FORCED_TOOL_REFUSAL) is None
no_tools = _anthropic_body()
assert "tool_choice" not in no_tools and rewrite_on_400(no_tools, _FORCED_TOOL_REFUSAL) is None
unrelated = _anthropic_body(tools=_VERDICT_TOOLS, tool_choice=_FORCED_VERDICT)
assert rewrite_on_400(unrelated, "something else entirely") is None
assert unrelated["tool_choice"]["type"] == "tool"
@pytest.mark.parametrize("streaming", [True, False])
async def test_a_forced_tool_choice_the_model_refuses_is_retried_under_auto(monkeypatch, streaming) -> None:
"""The research reviewer forces ``review_verdict``; on a model that refuses that,
every review failed open and the report shipped unreviewed. Both call sites
(``chat`` and ``chat_stream``) answer the 400 with one retry under ``auto``."""
from raven.providers import anthropic_messages_provider as mod
seen: list[dict] = []
def handler(request: httpx.Request) -> httpx.Response:
body = json.loads(request.content)
seen.append(body)
if body["tool_choice"]["type"] in {"tool", "any"}:
return httpx.Response(
400,
json={"type": "error", "error": {"type": "invalid_request_error", "message": _FORCED_TOOL_REFUSAL}},
)
if streaming:
return httpx.Response(200, content=_stream_ok("pass"))
return httpx.Response(
200,
json={
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "pass"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 3, "output_tokens": 1},
},
)
real_client = httpx.AsyncClient
monkeypatch.setattr(
mod.httpx, "AsyncClient", lambda *_a, **_kw: real_client(transport=httpx.MockTransport(handler))
)
monkeypatch.setattr(mod, "send_max_tokens", lambda gen, model, pinned=None, allow_fetch=True: pinned or 64000)
provider = AnthropicMessagesProvider(
api_key="k", api_base="https://api.anthropic.com/v1", default_model="claude-sonnet-5-5"
)
asked = dict(tools=_VERDICT_TOOLS, tool_choice=_FORCED_VERDICT, reasoning_effort="low")
messages = [{"role": "user", "content": "review this draft"}]
if streaming:
deltas = [d async for d in provider.chat_stream(messages, **asked)]
text, finish = "".join(d.content or "" for d in deltas), deltas[-1].finish_reason
else:
response = await provider.chat(messages, **asked)
text, finish = response.content, response.finish_reason
assert [b["tool_choice"]["type"] for b in seen] == ["tool", "auto"]
assert seen[1]["tools"] == seen[0]["tools"]
assert text == "pass" and finish == "stop"
assert provider._ceilings == {}
def test_anthropic_answers_a_refused_effort_label_with_litellms_number() -> None:
"""A vendor that takes no label has to be answered in tokens, and the number is
litellm's -- the only place in this transport where a budget is a number at all.
Half the ceiling at most, and no temperature beside it."""
from raven.providers.anthropic_messages_provider import rewrite_on_400
body = _anthropic_body(model="claude-haiku-4-5", reasoning_effort="max", temperature=0.95)
assert body["reasoning"] == {"effort": "max"} and body["temperature"] == 0.95
assert rewrite_on_400(body, "reasoning: Extra inputs are not permitted") == "thinking"
budget = body["thinking"]["budget_tokens"]
assert "reasoning" not in body and "temperature" not in body
assert 1024 <= budget <= body["max_tokens"] // 2
adaptive = _anthropic_body(model="anthropic/claude-opus-5", reasoning_effort="max")
assert rewrite_on_400(adaptive, "thinking.type: adaptive is not supported on this model")
assert adaptive["thinking"]["budget_tokens"] <= adaptive["max_tokens"] // 2
def test_anthropic_records_what_a_request_asks_for_without_the_key() -> None:
"""The trace kept the conversation and nothing about the request, so a truncated
turn could not be attributed to a ceiling, an effort or a backend pin. The record
carries the parameters and never the credentials."""
provider = AnthropicMessagesProvider(
api_key="sk-or-v1-secret",
api_base="https://openrouter.ai/api/v1",
default_model="z-ai/glm-5.3-flash",
model_overrides={"glm-5.3-flash": {"extra_body": {"provider": {"order": ["Z.AI"]}}}},
)
provider.generation = replace(provider.generation, reasoning_effort="high", temperature=0.95)
record = provider.request_generation(model="z-ai/glm-5.3-flash")
assert record["reasoning_effort"] == "high" and record["reasoning"] == {"effort": "high"}
assert record["temperature"] == 0.95 and isinstance(record["max_tokens"], int)
assert record["provider_fence"] == {"order": ["Z.AI"]}
blob = json.dumps(record)
assert "secret" not in blob and "openrouter.ai" not in blob and "Authorization" not in blob
provider.generation = replace(provider.generation, reasoning_effort=None)
assert provider.request_generation(model="z-ai/glm-5.3-flash")["reasoning_effort"] is None
def test_every_wrapper_forwards_the_question_of_what_a_rung_sends() -> None:
"""The empty-response retry asks the provider the loop holds, which in
production is a wrapper.
The base answer -- every label is its own request -- is the wrong answer for
the Anthropic wire, and answering it here would leave the retry re-sending an
unchanged request exactly where the collapse was measured. Pinned the way
``wire_model_id`` and ``supports_assistant_prefill`` are pinned: by naming
the wrappers, so a new one cannot inherit the default silently.
"""
from raven.providers.base import LLMProvider
from raven.providers.endpoint_rotor import EndpointRotorProvider
from raven.providers.lazy import LazyProvider
from raven.providers.per_model_provider import PerModelProvider
from raven.providers.resolving_provider import ResolvingProvider
for wrapper in (LazyProvider, PerModelProvider, ResolvingProvider, EndpointRotorProvider):
assert "reasoning_wire_keys" in wrapper.__dict__, (
f"{wrapper.__name__} would answer LLMProvider's label-only default, "
"which says two collapsed rungs are two different requests"
)
inner = AnthropicMessagesProvider(api_key="k", default_model="claude-opus-5")
lazy = LazyProvider.__new__(LazyProvider)
lazy._provider = inner
assert lazy.reasoning_wire_keys("claude-opus-5", "low") == inner.reasoning_wire_keys("claude-opus-5", "low")
assert lazy.reasoning_wire_keys("claude-opus-5", "minimal") == lazy.reasoning_wire_keys("claude-opus-5", "low")
assert LLMProvider.reasoning_wire_keys(inner, "claude-opus-5", "minimal") != LLMProvider.reasoning_wire_keys(
inner, "claude-opus-5", "low"
), "the base default really does read the two rungs as different requests"
def test_anthropic_replays_only_signed_thinking_blocks() -> None:
"""A stream cut at the output ceiling ends inside the thinking, so the block
carries no signature; sent back, the API refused it as a modified thinking
block and the next turn died."""
from raven.providers.anthropic_messages_provider import convert_messages
_, converted = convert_messages(
[
{"role": "user", "content": "go"},
{
"role": "assistant",
"content": "",
"thinking_blocks": [
{"type": "thinking", "thinking": "cut off here", "signature": ""},
{"type": "thinking", "thinking": "settled", "signature": "sig-2"},
{"type": "redacted_thinking", "data": ""},
],
"tool_calls": [{"id": "c1", "type": "function", "function": {"name": "read_file", "arguments": "{}"}}],
},
{"role": "tool", "tool_call_id": "c1", "content": "ok"},
]
)
replayed = [b for b in converted[1]["content"] if b["type"] in {"thinking", "redacted_thinking"}]
assert replayed == [{"type": "thinking", "thinking": "settled", "signature": "sig-2"}]
async def test_anthropic_retries_once_at_the_ceiling_the_model_names(monkeypatch) -> None:
"""The default ceiling suits the current models; an older one refuses it and
names its own limit, which is the one number the retry needs."""
from raven.providers import anthropic_messages_provider as mod
seen: list[dict] = []
def handler(request: httpx.Request) -> httpx.Response:
body = json.loads(request.content)
seen.append(body)
if body["max_tokens"] > 8192:
return httpx.Response(
400,
json={
"type": "error",
"error": {
"type": "invalid_request_error",
"message": f"max_tokens: {body['max_tokens']} > 8192, which is the maximum allowed number "
"of output tokens for claude-3-5-sonnet-20241022",
},
},
)
return httpx.Response(
200,
content=_anthropic_sse(
{"type": "message_start", "message": {"usage": {"input_tokens": 3}}},
{"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}},
{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "fits"}},
{"type": "content_block_stop", "index": 0},
{"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"output_tokens": 1}},
{"type": "message_stop"},
),
)
transport = httpx.MockTransport(handler)
real_client = httpx.AsyncClient
monkeypatch.setattr(mod.httpx, "AsyncClient", lambda *_a, **_kw: real_client(transport=transport))
# An owner that knows nothing about this model, so the request starts at the claude fallback.
monkeypatch.setattr(mod, "send_max_tokens", lambda gen, model, pinned=None, allow_fetch=True: pinned or 64000)
provider = AnthropicMessagesProvider(
api_key="k", api_base="https://api.anthropic.com/v1", default_model="claude-3-5-sonnet-20241022"
)
deltas = [d async for d in provider.chat_stream([{"role": "user", "content": "hi"}], reasoning_effort="high")]
again = [d async for d in provider.chat_stream([{"role": "user", "content": "hi again"}], reasoning_effort="high")]
# The refusal is earned once: the ceiling it named is remembered for the model.
assert [b["max_tokens"] for b in seen] == [64000, 8192, 8192]
assert seen[1]["reasoning"] == {"effort": "high"}
assert "".join(d.content or "" for d in deltas) == "fits" and deltas[-1].finish_reason == "stop"
assert again[-1].finish_reason == "stop"
def _stream_ok(text: str) -> bytes:
return _anthropic_sse(
{"type": "message_start", "message": {"usage": {"input_tokens": 3}}},
{"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}},
{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": text}},
{"type": "content_block_stop", "index": 0},
{"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"output_tokens": 1}},
{"type": "message_stop"},
)
def test_anthropic_asks_the_shared_owner_for_the_ceiling(monkeypatch) -> None:
"""The output ceiling has one owner, `send_max_tokens`, which the loop's
reservation also reads. A second definition inside this transport disagreed
with it on every model the catalogue knows: asking 64000 of a 32000 model,
and half of what a 128000 model offers."""
from raven.providers import anthropic_messages_provider as mod
asked: list[tuple[str, int | None]] = []
def owner(gen, model, pinned=None, allow_fetch=True):
asked.append((model, pinned))
return min(pinned, 32000) if pinned else 32000
monkeypatch.setattr(mod, "send_max_tokens", owner)
provider = AnthropicMessagesProvider(api_key="k", default_model="claude-opus-4-1")
common = dict(messages=[{"role": "user", "content": "hi"}], tools=None, model=None, tool_choice=None, stream=True)
body = provider._body(max_tokens=None, temperature=0.1, reasoning_effort="high", **common)
assert asked == [("claude-opus-4-1", None)] and body["max_tokens"] == 32000
pinned = provider._body(max_tokens=2048, temperature=0.1, reasoning_effort=None, **common)
assert pinned["max_tokens"] == 2048
async def test_a_400_that_named_no_ceiling_teaches_none(monkeypatch) -> None:
"""A curator-shaped call pins max_tokens=2048; the model answers that its
thinking mode is the other one. That repair leaves the pin in place, and
remembering the pin as the model's ceiling clamped every later request to
2048 -- a 31x reduction, silently. Only a ceiling complaint teaches."""
from raven.providers import anthropic_messages_provider as mod
seen: list[dict] = []
def handler(request: httpx.Request) -> httpx.Response:
body = json.loads(request.content)
seen.append(body)
if "reasoning" in body:
return httpx.Response(
400,
json={
"type": "error",
"error": {"type": "invalid_request_error", "message": "reasoning: Extra inputs are not permitted"},
},
)
return httpx.Response(200, content=_stream_ok("ok"))
transport = httpx.MockTransport(handler)
real_client = httpx.AsyncClient
monkeypatch.setattr(mod.httpx, "AsyncClient", lambda *_a, **_kw: real_client(transport=transport))
monkeypatch.setattr(mod, "send_max_tokens", lambda gen, model, pinned=None, allow_fetch=True: pinned or 64000)
provider = AnthropicMessagesProvider(
api_key="k", api_base="https://api.anthropic.com/v1", default_model="claude-sonnet-4-5"
)
short = [
d
async for d in provider.chat_stream(
[{"role": "user", "content": "curate"}], max_tokens=2048, reasoning_effort="high"
)
]
ordinary = [
d async for d in provider.chat_stream([{"role": "user", "content": "build the page"}], reasoning_effort="high")
]
assert short[-1].finish_reason == "stop" and ordinary[-1].finish_reason == "stop"
# The ordinary turn earns the same effort 400 once, then goes through -- at
# the owner's number, not the earlier pin.
assert [b["max_tokens"] for b in seen] == [2048, 2048, 64000, 64000]
assert seen[1]["thinking"]["type"] == "enabled" and "reasoning" not in seen[1]
assert provider._ceilings == {}
def test_anthropic_sends_no_temperature_to_a_model_that_thinks_by_default() -> None:
"""Claude 4.7 and later think adaptively unless told otherwise, and a
thinking request accepts no temperature but the default; the repository's
0.1 on the ordinary request (no reasoning effort configured) was refused."""
from raven.providers.anthropic_messages_provider import rewrite_on_400
quiet = _anthropic_body(model="anthropic/claude-opus-5", reasoning_effort=None, temperature=0.1)
assert "temperature" not in quiet and "thinking" not in quiet
older = _anthropic_body(model="claude-haiku-4-5", reasoning_effort=None, temperature=0.1)
assert older["temperature"] == 0.1
assert rewrite_on_400(older, "temperature may only be set to 1 when thinking is enabled") == "temperature"
assert "temperature" not in older
async def test_the_anthropic_transport_lets_the_cache_optimizer_place_breakpoints() -> None:
"""The base default says no provider caches; this transport must say yes
for a Claude model, or the strategy places no marks and every request of a
150k-token design session is billed uncached."""
from raven.token_wise.cache_optimizer import CacheOptimizer
provider = AnthropicMessagesProvider(
api_key="k",
api_base="https://openrouter.ai/api/v1",
default_model="openrouter/claude-opus-5",
provider_name="openrouter",
)
assert provider.supports_prompt_caching("openrouter/claude-opus-5") is True
assert provider.supports_prompt_caching("openrouter/gpt-5.6-sol") is False
optimizer = CacheOptimizer(supports_caching=provider.supports_prompt_caching)
messages = [{"role": "system", "content": "rules"}, {"role": "user", "content": "hi"}]
tools = [
{
"type": "function",
"function": {"name": "noop", "description": "n", "parameters": {"type": "object", "properties": {}}},
}
]
messages, tools, model = await optimizer.before_llm_call(messages, tools, "openrouter/claude-opus-5")
body = provider._body(
messages=messages,
tools=tools,
model=model,
max_tokens=64,
temperature=0.7,
reasoning_effort=None,
tool_choice=None,
stream=False,
)
assert json.dumps(body).count("cache_control") == 3
async def test_tail_breakpoints_survive_the_anthropic_conversion() -> None:
"""The optimizer marks the last block of the last two messages. On a turn
that only called a tool the marked text block is empty and dropped, and a
tool result is wrapped in a tool_result block; both used to lose the mark,
which left every iteration of a tool loop cached only up to the system
prompt (16.8k of a 150k prefix in production)."""
from raven.token_wise.cache_optimizer import CacheOptimizer
messages = [
{"role": "system", "content": "rules"},
{"role": "user", "content": "go"},
{
"role": "assistant",
"content": "",
"tool_calls": [{"id": "toolu_1", "type": "function", "function": {"name": "exec", "arguments": "{}"}}],
},
{"role": "tool", "tool_call_id": "toolu_1", "content": [{"type": "text", "text": "ok"}]},
]
marked, _, _ = await CacheOptimizer(supports_caching=lambda _m: True).before_llm_call(
messages, None, "openrouter/claude-opus-5"
)
system, wire = convert_messages(marked)
assert "cache_control" in system[-1]
assistant, tool_result = wire[-2], wire[-1]
assert assistant["content"][-1]["type"] == "tool_use" and "cache_control" in assistant["content"][-1]
assert tool_result["content"][-1]["type"] == "tool_result" and "cache_control" in tool_result["content"][-1]
assert not any("cache_control" in inner for inner in tool_result["content"][-1]["content"])
# The marked turn must render exactly like the same turn once the mark has
# moved on, or the cached prefix breaks at every iteration.
_, unmarked = convert_messages(messages)
assert [b for b in assistant["content"] if "cache_control" not in b] == unmarked[-2]["content"][:-1]
assert {k: v for k, v in assistant["content"][-1].items() if k != "cache_control"} == unmarked[-2]["content"][-1]
async def test_a_messages_stream_cut_after_its_first_delta_is_not_asked_again(monkeypatch):
"""The messages adapter catches a connection that died mid-answer and reports
it as a terminal error delta instead of raising -- the tree's only producer of
that shape, and the one the rule about rendered output used to miss. The call
comes back with the failure's own account and a verdict spent of its retry, so
the caller's ladder does not draw the same answer from the top again."""
from raven.providers.streaming import stream_llm_call
async def cut_after_two_deltas():
yield _anthropic_sse(
{"type": "message_start", "message": {"usage": {"input_tokens": 3}}},
{"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}},
{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "the first "}},
{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "half"}},
)
raise httpx.ReadError("connection reset by peer")
asked: list[httpx.Request] = []
def handler(request):
asked.append(request)
return httpx.Response(200, content=cut_after_two_deltas())
real_client = httpx.AsyncClient
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: real_client(transport=httpx.MockTransport(handler), **kw))
provider = AnthropicMessagesProvider(
api_key="test", api_base="https://api.anthropic.com", provider_name="claude", default_model="test"
)
seen: list[str] = []
async def on_delta(text: str) -> None:
seen.append(text)
response = await stream_llm_call(
provider,
messages=[{"role": "user", "content": "hi"}],
tools=None,
model="test",
on_token_delta=on_delta,
retry_delays=(0.0, 0.0),
)
assert seen == ["the first ", "half"]
assert len(asked) == 1, "the helper's own reconnect budget was not spent on it"
assert response.finish_reason == "error"
assert response.error_classification is not None
assert response.error_classification.retryable is False
assert (response.content or "").startswith("Error calling LLM (")
assert "the first half" not in (response.content or "")
@pytest.mark.parametrize("protocol", ["responses", "messages", "messages_stream"])
@pytest.mark.parametrize(
"name,endpoint",
[
("openrouter", "https://openrouter.ai/api/v1"),
("custom", "https://gateway.example/v1"),
],
)
@pytest.mark.parametrize(
"cost,expected_cost", [(0.84, 0.84), (0, 0), (None, None), (-1, None), (True, None), ("0.84", None)]
)
async def test_native_routes_preserve_reported_usage(monkeypatch, protocol, name, endpoint, cost, expected_cost):
from raven.agent.loop.main import AgentLoop
from raven.providers.anthropic_messages_provider import AnthropicMessagesProvider
from raven.providers.openai_responses_provider import OpenAIResponsesProvider
from raven.providers.streaming import stream_llm_call
def without_missing_cost(value):
if isinstance(value, dict):
return {key: without_missing_cost(item) for key, item in value.items() if key != "cost" or item is not None}
if isinstance(value, list):
return [without_missing_cost(item) for item in value]
return value
def handler(request):
if protocol == "responses":
events = [
{"type": "response.output_text.delta", "delta": "ok"},
{
"type": "response.completed",
"response": {
"status": "completed",
"usage": {
"input_tokens": 100,
"output_tokens": 5,
"total_tokens": 105,
"cost": cost,
"input_tokens_details": {"cached_tokens": 60, "cache_write_tokens": 10},
},
},
},
]
return httpx.Response(200, content=_anthropic_sse(*without_missing_cost(events)))
usage = {
"input_tokens": 30,
"output_tokens": 5,
"cache_read_input_tokens": 60,
"cache_creation_input_tokens": 10,
"cost": cost,
}
if protocol == "messages":
return httpx.Response(
200,
json={
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": without_missing_cost(usage),
},
)
events = [
{"type": "message_start", "message": {"usage": {**usage, "cost": None}}},
{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "ok"}},
{
"type": "message_delta",
"delta": {"stop_reason": "end_turn"},
"usage": {"output_tokens": 5, "cost": cost},
},
{"type": "message_stop"},
]
return httpx.Response(200, content=_anthropic_sse(*without_missing_cost(events)))
real_client = httpx.AsyncClient
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: real_client(transport=httpx.MockTransport(handler), **kw))
provider_class = OpenAIResponsesProvider if protocol == "responses" else AnthropicMessagesProvider
provider = provider_class(api_key="test", api_base=endpoint, provider_name=name, default_model="test")
if protocol == "messages_stream":
response = await stream_llm_call(
provider, messages=[{"role": "user", "content": "hi"}], tools=None, model="test"
)
else:
response = await provider.chat(messages=[{"role": "user", "content": "hi"}], model="test")
assert response.finish_reason == "stop", response.content
snap = AgentLoop._build_usage_snapshot(response, "test", "session")
assert snap.cost_usd == expected_cost
assert (snap.input_tokens, snap.output_tokens, snap.cache_read_tokens, snap.cache_write_tokens) == (30, 5, 60, 10)
async def test_responses_partial_usage_preserves_cost_and_cache():
from raven.providers.openai_codex_provider import _consume_sse
from raven.providers.usage import responses_usage
events = [
{
"type": "response.completed",
"response": {
"status": "completed",
"usage": {
"input_tokens": 100,
"output_tokens": 5,
"cost": 0.4,
"input_tokens_details": {"cached_tokens": 60},
},
},
},
{
"type": "response.done",
"response": {
"status": "completed",
"usage": {
"cost": None,
"total_tokens": 105,
"input_tokens_details": {"cached_tokens": None},
},
},
},
]
sink = {}
response = httpx.Response(200, content=_anthropic_sse(*events))
await _consume_sse(response, 5, usage_sink=sink)
usage = responses_usage(sink)
assert usage["cost_usd"] == 0.4
assert usage["cache_read_input_tokens"] == 60
assert usage["prompt_tokens"] == 100
assert usage["cache_creation_input_tokens"] is None
@pytest.mark.parametrize("model", ["claude-opus-5", "anthropic/claude-haiku-4-5"])
def test_anthropic_model_defaults_preserve_manual_overrides(model):
assert effective_protocol(None, model) == "anthropic"
assert effective_protocol(_config(model).providers.custom, model) == "anthropic"
for protocol in ("chat", "responses"):
config = _config(model, override=protocol)
assert effective_protocol(config.providers.custom, model) == protocol
@pytest.mark.parametrize("model", ["glm-5.3-flash", "z-ai/GLM-5", "openrouter/z-ai/glm-5.3-flash"])
def test_a_glm_model_is_not_inferred_onto_the_anthropic_wire(model):
"""It was, for two days, and the Messages transport has to turn an effort into a
token budget -- which the gateway serving these models ignores. Over chat the
effort goes out as an effort. An explicit protocol still wins, as always."""
assert effective_protocol(None, model) == "chat"
assert isinstance(make_provider(_config(model)), LiteLLMProvider)
config = _config(model, override="anthropic")
assert effective_protocol(config.providers.custom, model) == "anthropic"
def test_explicit_gemini_native_protocol_does_not_fall_back():
from raven.providers.auth import MissingCredentialsError
config = _provider_config("gemini", "gemini/gemini-2.5-pro", override="responses")
with pytest.raises(MissingCredentialsError, match="requires an explicit API base"):
make_provider(config)
def test_direct_glm_uses_vendor_anthropic_endpoint_when_asked_for_it():
"""The vendor's Anthropic-compatible address is still declared and still used --
but only when the operator names the protocol. Inferring it from the model name
is what put every glm model on a wire that cannot carry an effort."""
config = _provider_config("zai", "zai/glm-4.6", override="anthropic")
provider = make_provider(config)
assert isinstance(provider, AnthropicMessagesProvider)
assert provider.api_base == "https://api.z.ai/api/anthropic"
assert provider.api_protocol == effective_protocol(config.providers.get("zai"), "zai/glm-4.6")
assert effective_protocol(_provider_config("zai", "zai/glm-4.6").providers.get("zai"), "zai/glm-4.6") == "chat"
def test_explicit_native_protocol_requires_vendor_address():
from raven.providers.auth import MissingCredentialsError
config = _provider_config("zai", "zai/glm-4.6", override="responses")
with pytest.raises(MissingCredentialsError, match="requires an explicit API base"):
make_provider(config)
def test_direct_glm_with_explicit_endpoint_uses_that_endpoint():
config = _provider_config("zai", "zai/glm-4.6", override="anthropic")
config.providers.get("zai").api_base = "https://vendor.example/anthropic"
provider = make_provider(config)
assert isinstance(provider, AnthropicMessagesProvider)
assert provider.api_base == "https://vendor.example/anthropic"
@pytest.mark.parametrize(
"vendor,model,base",
[
("deepseek", "deepseek/deepseek-v4-pro", "https://api.deepseek.com"),
("volcengine", "volcengine/doubao-seed-2-0-pro", "https://ark.cn-beijing.volces.com/api/v3"),
("minimax", "minimax/MiniMax-M3", "https://api.minimax.io/v1"),
("minimax_cn_api", "minimax-cn-api/MiniMax-M3", "https://api.minimax.cn/v1"),
],
)
def test_response_series_use_declared_vendor_addresses(vendor, model, base):
provider = make_provider(_provider_config(vendor, model))
assert isinstance(provider, OpenAIResponsesProvider)
assert provider.api_base == base
@pytest.mark.parametrize(
"vendor,model",
[
("moonshot", "moonshot/kimi-k2-turbo-preview"),
("groq", "groq/qwen/qwen3-32b"),
("dashscope", "dashscope/qwen-plus"),
("nvidia_nim", "nvidia_nim/deepseek-ai/deepseek-v3.1"),
],
)
def test_an_inferred_native_protocol_steps_aside_where_the_provider_has_no_address_for_it(vendor, model):
"""The family name reads as responses, and these providers declare no
responses address; the base tree served them over chat and the inference
turned that into a refusal on 56 curated models of a key-only config. An
inference is not a choice, so it steps aside; the bare inference (no
provider named) still reads responses, which is what the tables show."""
config = _provider_config(vendor, model)
assert effective_protocol(None, model) == "responses"
assert effective_protocol(config.providers.get(vendor), model, vendor) == "chat"
provider = make_provider(config)
assert isinstance(provider, LiteLLMProvider) and provider.api_protocol == "chat"
def test_a_chosen_native_protocol_without_an_address_is_still_refused():
"""The user's own selection is final: refusing is the right answer to an
explicit responses on a provider with nowhere to send it."""
from raven.providers.auth import MissingCredentialsError
with pytest.raises(MissingCredentialsError, match="requires an explicit API base"):
make_provider(_provider_config("moonshot", "moonshot/kimi-k2-turbo-preview", override="responses"))
def test_a_gateway_declaring_some_protocols_does_not_serve_the_others_at_its_default_address():
"""volcengine names a responses address only. The gateway fallback handed
that address to an anthropic request as well, so a glm model on volcengine
posted /v1/messages to a responses endpoint, and the picker could not warn
because the fallback always answered. A gateway that declares nothing per
protocol (openrouter) still serves every protocol at its one address."""
from raven.providers.protocol import native_api_base
assert native_api_base("volcengine", "responses") == "https://ark.cn-beijing.volces.com/api/v3"
assert native_api_base("volcengine", "anthropic") is None
assert native_api_base("openrouter", "responses") == "https://openrouter.ai/api/v1"
assert native_api_base("openrouter", "anthropic") == "https://openrouter.ai/api/v1"
config = _provider_config("volcengine", "volcengine/glm-4-7-251222")
provider = make_provider(config)
assert isinstance(provider, LiteLLMProvider) and provider.api_protocol == "chat"
# The picker's half of the same-answer claim: it resolves through the same
# call, provider named, so its entry says chat too and carries no warning.
# Dropping the provider name at the picker alone would read anthropic here.
entry = _picker_entry("volcengine", config)
assert entry["protocols"]["volcengine/glm-4-7-251222"] == "chat" and entry["warning"] == ""
def _picker_entry(vendor: str, config: Config) -> dict:
"""The model picker's row for ``vendor``, built off the configured section."""
from raven.rpc.methods.model import _build_provider_entry
return _build_provider_entry(
vendor,
current_provider=None,
providers={vendor: {"name": vendor, "configured": True}},
section=config.providers.get(vendor),
)
def _pool_config(vendor: str, model: str, endpoints: list[dict]) -> Config:
return Config.model_validate(
{
"agents": {"defaults": {"model": model, "provider": vendor}},
"providers": {vendor: {"endpoints": endpoints}},
}
)
def test_a_mixed_endpoint_pool_stays_on_chat_until_every_member_has_an_address():
"""One addressed endpoint is not permission to build a native client for the
whole rotor: the key-only member has nowhere to send responses, and the
rotor failed eagerly while constructing it. The inference steps aside
unless every endpoint can be served."""
from raven.providers.endpoint_rotor import EndpointRotorProvider
model = "moonshot/kimi-k2-turbo-preview"
mixed = _pool_config(
"moonshot",
model,
[
{"label": "bare", "apiKey": "k1"},
{"label": "addressed", "apiKey": "k2", "apiBase": "https://kimi.example/v1"},
],
)
assert effective_protocol(mixed.providers.get("moonshot"), model, "moonshot") == "chat"
assert _picker_entry("moonshot", mixed)["protocols"][model] == "chat"
provider = make_provider(mixed)
assert isinstance(provider, EndpointRotorProvider)
addressed = _pool_config(
"moonshot",
model,
[
{"label": "a", "apiKey": "k1", "apiBase": "https://kimi-a.example/v1"},
{"label": "b", "apiKey": "k2", "apiBase": "https://kimi-b.example/v1"},
],
)
# The picker and the factory read the same endpoint set, so what one shows
# is what the other builds: responses here, from every member's address.
assert effective_protocol(addressed.providers.get("moonshot"), model, "moonshot") == "responses"
assert _picker_entry("moonshot", addressed)["protocols"][model] == "responses"
assert isinstance(make_provider(addressed), EndpointRotorProvider)
def test_display_resolution_logs_nothing_and_construction_logs_once():
"""The model picker resolves every catalogue row through effective_protocol;
an INFO line per fallback there was 86 records per all-provider row build.
The runtime decision -- a client actually built over chat -- is logged once."""
from loguru import logger
records: list[str] = []
sink = logger.add(lambda message: records.append(str(message)), level="INFO")
try:
for vendor, model in (("dashscope", "dashscope/qwen-plus"), ("moonshot", "moonshot/kimi-k2-turbo-preview")):
section = _provider_config(vendor, model).providers.get(vendor)
assert effective_protocol(section, model, vendor) == "chat"
assert records == []
make_provider(_provider_config("moonshot", "moonshot/kimi-k2-turbo-preview"))
assert sum("using chat" in r for r in records) == 1
# An operator's own chat choice is not a stepping-aside, and the provider
# it names is not missing an address: nothing to report.
make_provider(_provider_config("anthropic", "anthropic/claude-opus-5", override="chat"))
assert sum("using chat" in r for r in records) == 1
finally:
logger.remove(sink)
def test_gemini_defaults_to_litellm_chat_without_an_endpoint():
provider = make_provider(_provider_config("gemini", "gemini/gemini-2.5-flash"))
assert isinstance(provider, LiteLLMProvider)
assert provider.api_protocol == "chat"
@pytest.mark.parametrize("protocol", ["messages", "messages_stream", "responses", "chat"])
async def test_a_reported_reasoning_count_reaches_the_usage_record(monkeypatch, protocol):
"""Every protocol reports how much of the output was thinking, and every one
of them dropped it. Without the count a turn whose reasoning text never
arrived is indistinguishable from a turn that did not think, which is the
reading a 309-iteration deck run had to be diagnosed from.
The wire shapes here are captured responses: openrouter returns
``completion_tokens_details.reasoning_tokens`` on chat completions and
``output_tokens_details.thinking_tokens`` on the messages route.
"""
from raven.agent.loop.turn_path import TurnPathMixin
from raven.observability import semconv
from raven.providers.streaming import stream_llm_call
def handler(request):
if protocol == "responses":
return httpx.Response(
200,
content=_anthropic_sse(
{"type": "response.output_text.delta", "delta": "ok"},
{
"type": "response.completed",
"response": {
"status": "completed",
"usage": {
"input_tokens": 256,
"output_tokens": 417,
"output_tokens_details": {"reasoning_tokens": 235},
},
},
},
),
)
usage = {
"input_tokens": 256,
"output_tokens": 417,
"output_tokens_details": {"thinking_tokens": 235},
}
if protocol == "messages":
return httpx.Response(
200,
json={
"content": [{"type": "thinking", "thinking": "inspect first"}, {"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": usage,
},
)
return httpx.Response(
200,
content=_anthropic_sse(
{"type": "message_start", "message": {"usage": usage}},
{"type": "content_block_start", "index": 0, "content_block": {"type": "thinking", "thinking": ""}},
{
"type": "content_block_delta",
"index": 0,
"delta": {"type": "thinking_delta", "thinking": "inspect first"},
},
{"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": usage},
{"type": "message_stop"},
),
)
if protocol == "chat":
from types import SimpleNamespace
provider = LiteLLMProvider(api_key="test", api_base="https://openrouter.ai/api/v1", provider_name="custom")
response = provider._parse_response(
SimpleNamespace(
choices=[
SimpleNamespace(
message=SimpleNamespace(content="ok", tool_calls=None, reasoning_content="inspect first"),
finish_reason="stop",
)
],
usage=SimpleNamespace(
prompt_tokens=256,
completion_tokens=417,
total_tokens=673,
completion_tokens_details=SimpleNamespace(reasoning_tokens=235),
),
)
)
else:
real_client = httpx.AsyncClient
monkeypatch.setattr(
httpx, "AsyncClient", lambda **kw: real_client(transport=httpx.MockTransport(handler), **kw)
)
provider_class = OpenAIResponsesProvider if protocol == "responses" else AnthropicMessagesProvider
provider = provider_class(
api_key="test", api_base="https://openrouter.ai/api/v1", provider_name="custom", default_model="test"
)
messages = [{"role": "user", "content": "hi"}]
if protocol == "messages_stream":
response = await stream_llm_call(provider, messages=messages, tools=None, model="test")
else:
response = await provider.chat(messages=messages, model="test")
assert response.usage["reasoning_tokens"] == 235
assert semconv.llm_output_payload(response)["usage"]["reasoning_tokens"] == 235
assert semconv.llm_attrs(response, "custom", "test")["llm.usage.reasoning_tokens"] == 235
assert TurnPathMixin._build_usage_snapshot(response, "test", "sess").reasoning_tokens == 235