## Summary The canonical failure sentence `Error calling LLM (<category>[@<provider>]): <detail>` was built in eleven places and bounded in none, and every reader treated it as a sentence for a person: the gateway handed a channel the vendor's whole body (for an auth failure that is a masked key and the account URL it was rejected for; a rejected-request body can quote the prompt back and run past Telegram's split), the loop wrote the same body into the marker the model reads on its next turn and into `turn_ended.reason`, and cron (since #675) stores it as the job's `last_error`. Only the one-shot CLI had a 200-character cut and a per-category wording. One constructor now builds the sentence (`canonical_llm_error` in `raven/providers/base.py`) and cuts the detail to 200 characters after the verdict has been taken, so a needle deep in a rejected-request body still decides the category; the untouched body stays on the call record. A category table (`llm_error_summary`) gives every bucket the classifier can reach one sentence with no vendor text. `TurnFailed` gains an in-process `reported` flag, set by the Lane when the runner worded the failure itself, and the gateway's reply to a channel reads: a canonical failure -> the category sentence; a reported non-canonical failure -> its own words, bounded; a crash -> the canned sentence. The marker a broken turn leaves is worded for its two readers: the model-facing text names the category, `turn_ended.reason` keeps the provider's account for whoever diagnoses it. A crash's text is bounded by a kernel-local constant so `raven.spine` still imports nothing from `raven.providers`. Folded in: the cut-stream exit in `streaming.py` now spends the verdict's retryability when a thought had already reached a wired reasoning callback (the rule the exits beside it hold since #673), and the gateway pops a read-back conversation's stored reply on every exit so a failed cron run cannot hand its submitter the previous run's text. This reverses the choice #666 made to send a channel the provider's own sentence. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification ``` uv run pytest tests/test_error_classification.py tests/test_gateway_spine.py tests/test_agent_loop_broken_turn.py \ tests/test_cli_agent_commands.py tests/test_agent_loop_stream.py tests/test_agent_loop_llm_error_patience.py \ tests/test_spine_scheduler.py tests/test_spine_scheduler_lane.py tests/test_core_cron_stack.py tests/test_spine_events.py -q uv run pytest tests -q --ignore=tests/integration uv run ruff check . && uv run ruff format --check . && uv run lint-imports ``` Live check with an isolated agent home, `raven serve` and a playwright-driven page, against local fake endpoints. (1) OpenAI-compatible endpoint answering 401 with a masked key and the account URL: `raven agent -m hi` still prints the vendor reason and the `Fix:` line and exits 1; the page shows the red failed-turn row; the persisted `turn_ended.reason` is exactly 200 characters ending in `...` (this failure raises, so it takes the crash path and gets the kernel bound). (2) The same endpoint answering 500 with a 5000-character body: the persisted reason is again 200 characters. (3) Anthropic Messages endpoint that streams two text deltas and then sends an `error` event with a 5000-character message (an error response, so the canonical path): the page row reads `Error calling LLM (server@anthropic): overloaded_error: Overloaded: ...`, `turn_ended.reason` is the head plus 200 characters of detail (238 in total), and the model-facing marker text reads `(turn failed: The provider returned a server error (anthropic). The runtime log has the provider's own account.)`. (4) The same endpoint cutting the stream without a terminal event: the row reads the unchanged short canonical sentence. The channel reply is covered by tests/test_gateway_spine.py, which drives a real DeliveryHub through a fake channel. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed (CONTEXT.md entries updated in this change; no user-facing doc describes channel error wording) ## Risk - A channel reader gets the failure's category and endpoint (for example `The provider rejected the credentials (openrouter). The runtime log has the provider's own account.`) instead of the vendor's sentence; a single-user deployment loses detail it could only get from the chat, a group stops seeing key fragments and account URLs. - Every canonical failure sentence is cut to 200 characters of detail with a trailing `...`, on the page, in the TUI, in `turn_ended.reason`, in cron's `last_error` and in the CLI line; the full body stays on the call record and in the runtime log. A long `invalid_request` tail that the page used to show in full is now cut. - The marker text the model reads back after a failed turn names the category instead of quoting the vendor; the model's visible history changes. - With a wired reasoning callback (web, TUI) a stream cut after its thought is no longer retried by the loop's ladder unless `llmRetryAfterOutput` is on. - `TurnFailed` has a new in-process field; an out-of-tree consumer constructing it positionally with five arguments would now set it. - Rollback: the gateway commit alone restores the vendor sentence to channels; the other commits stand on their own. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
481 lines
20 KiB
Python
481 lines
20 KiB
Python
"""Structured error classification + jittered backoff (LLMProvider).
|
|
|
|
``classify_error`` is the seam that drives retry / fallback / compress
|
|
decisions. It works on a live exception (HTTP status + class name, walking the
|
|
__cause__ chain) or, degraded, on the swallowed error string.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from raven.providers.base import ErrorClassification, LLMProvider
|
|
|
|
# --- fakes mimicking provider exception shapes (no SDK import needed) -------- #
|
|
|
|
|
|
class _StatusError(Exception):
|
|
def __init__(self, msg: str, status_code: int):
|
|
super().__init__(msg)
|
|
self.status_code = status_code
|
|
|
|
|
|
class RateLimitError(Exception):
|
|
pass
|
|
|
|
|
|
class ContextWindowExceededError(Exception):
|
|
pass
|
|
|
|
|
|
def _c(exc=None, content=None) -> ErrorClassification:
|
|
return LLMProvider.classify_error(exc, content)
|
|
|
|
|
|
# --- by HTTP status code ---------------------------------------------------- #
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"status,category,retry,fb,comp",
|
|
[
|
|
(429, "rate_limit", True, True, False),
|
|
(503, "server", True, True, False),
|
|
(500, "server", True, True, False),
|
|
(401, "auth", False, False, False),
|
|
(403, "auth", False, False, False),
|
|
(402, "billing", False, True, False),
|
|
(404, "model_unavailable", False, True, False),
|
|
(400, "invalid_request", False, False, False),
|
|
],
|
|
)
|
|
def test_classify_by_status_code(status, category, retry, fb, comp):
|
|
c = _c(_StatusError("boom", status))
|
|
assert c.category == category
|
|
assert (c.retryable, c.should_fallback, c.should_compress) == (retry, fb, comp)
|
|
|
|
|
|
# --- by exception class name ------------------------------------------------ #
|
|
|
|
|
|
def test_classify_by_class_name_rate_limit():
|
|
c = _c(RateLimitError("slow down"))
|
|
assert c.category == "rate_limit" and c.retryable and c.should_fallback
|
|
|
|
|
|
def test_classify_connect_failure_as_network_despite_litellm_500():
|
|
# litellm wraps a connect failure in APIError with its default 500; the
|
|
# message, not the status, says what happened (S-C-01).
|
|
c = _c(_StatusError("OpenrouterException - Cannot connect to host 127.0.0.1:9", 500))
|
|
assert c.category == "network" and c.retryable and c.should_fallback
|
|
|
|
|
|
def test_classify_connection_refused_as_network():
|
|
c = _c(_StatusError("Connection refused by proxy", 500))
|
|
assert c.category == "network" and c.retryable and c.should_fallback
|
|
|
|
|
|
def test_classify_context_window_by_class_name_compresses_not_fallback():
|
|
# A bare 400 would look like invalid_request; the class name disambiguates.
|
|
c = _c(ContextWindowExceededError("400"))
|
|
assert c.category == "context_overflow"
|
|
assert c.should_compress is True
|
|
assert c.should_fallback is False
|
|
assert c.retryable is False
|
|
|
|
|
|
# --- walks the __cause__ chain for the status code -------------------------- #
|
|
|
|
|
|
def test_classify_follows_cause_chain():
|
|
inner = _StatusError("upstream 429", 429)
|
|
try:
|
|
try:
|
|
raise inner
|
|
except Exception as e:
|
|
raise RuntimeError("wrapped") from e
|
|
except Exception as outer:
|
|
c = _c(outer)
|
|
assert c.category == "rate_limit" and c.should_fallback
|
|
|
|
|
|
# --- degraded string path (provider already swallowed the exception) -------- #
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"text,category",
|
|
[
|
|
("429 rate limit hit", "rate_limit"),
|
|
("503 service unavailable", "server"),
|
|
("connection reset by peer", "network"),
|
|
("insufficient credit / billing", "billing"),
|
|
("model not found", "model_unavailable"),
|
|
# None of these carries any of the wordier model_unavailable markers,
|
|
# even though each one has "404" inside a larger number or id --
|
|
# matching it as a bare substring once burned a fallback model and
|
|
# cooled a healthy endpoint for an error no swap could fix. Azure's own
|
|
# rendered non-200 body no longer reaches this degraded path at all:
|
|
# see ``AzureOpenAIProvider.chat``, which classifies from the live
|
|
# status code before the response is turned into a string.
|
|
("Error: retry after 1404ms", "unknown"),
|
|
("upstream error id=req_a404bc7f", "unknown"),
|
|
("invalid JSON at char 4041", "unknown"),
|
|
("This model's maximum context length is 8192 tokens", "context_overflow"),
|
|
("401 unauthorized: invalid api key", "auth"),
|
|
("400 invalid request: bad schema", "invalid_request"),
|
|
("something totally unexpected", "unknown"),
|
|
],
|
|
)
|
|
def test_classify_by_string(text, category):
|
|
assert _c(content=text).category == category
|
|
|
|
|
|
def test_unknown_retries_once_and_swaps_nothing():
|
|
"""A cause nobody could name is retried and not routed elsewhere.
|
|
|
|
Which way "conservative" points was decided by two measured runs: one died at
|
|
iteration 72 after 82 minutes and 15.5M input tokens, one at iteration 104 after
|
|
62 minutes and 23.8M, each on a provider wording no branch recognised. The ladder
|
|
costs seven seconds and three calls against that, so the unnamed failure is tried
|
|
again -- but not sent to another model, because nothing here says one would do
|
|
better.
|
|
"""
|
|
c = _c(content="???")
|
|
assert c.retryable
|
|
assert not c.should_fallback and not c.should_compress
|
|
|
|
|
|
def test_a_response_that_is_not_json_is_transient():
|
|
"""The gateway served an error page where a completion was expected: the message
|
|
names a character offset because the client's own parser is what failed."""
|
|
c = _c(content="OpenrouterException - Unable to get json response - Expecting value: line 177 column 1")
|
|
|
|
assert c.category == "unparsable_response"
|
|
assert c.retryable and c.should_fallback
|
|
|
|
|
|
# --- jitter ----------------------------------------------------------------- #
|
|
|
|
|
|
def test_jitter_within_ten_percent():
|
|
for _ in range(50):
|
|
j = LLMProvider._jittered(4.0)
|
|
assert 3.6 <= j <= 4.4
|
|
|
|
|
|
def test_jitter_zero_stays_zero():
|
|
assert LLMProvider._jittered(0) == 0.0
|
|
|
|
|
|
# --- format_llm_error / parse_llm_error / _strip_json_error_body ------------- #
|
|
|
|
|
|
def test_a_first_byte_timeout_is_retryable_and_named() -> None:
|
|
"""The bound the loop's ladder reads. A ``TimeoutError`` subclass would land
|
|
on the network bucket regardless, with the same retryable/fallback verdict;
|
|
the named category is so the record can say which bound fired -- "never
|
|
started" is not the same event as a mid-answer stall."""
|
|
from raven.providers.first_byte import FirstByteTimeoutError
|
|
|
|
exc = FirstByteTimeoutError(phase="waiting for the first chunk", budget=120, waited=120.4)
|
|
verdict = LLMProvider.classify_error(exc)
|
|
assert verdict.category == "first_byte_timeout"
|
|
assert verdict.retryable is True
|
|
assert verdict.should_fallback is True
|
|
assert verdict.should_compress is False
|
|
|
|
|
|
@pytest.mark.parametrize("idle", [180.0, 429.0, 500.0])
|
|
def test_a_stream_idle_timeout_is_named_whatever_budget_its_message_carries(idle: float) -> None:
|
|
"""The message embeds the configured idle budget. Classified by substring it
|
|
would read as a rate limit at 429 and a server error at 500 -- the same
|
|
wrong-cause report the error exists to remove -- so it is named by type
|
|
before any substring runs, like the first-byte bound beside it."""
|
|
from raven.providers.first_byte import StreamIdleTimeoutError
|
|
|
|
verdict = LLMProvider.classify_error(StreamIdleTimeoutError(idle=idle))
|
|
assert verdict.category == "stream_idle_timeout"
|
|
assert verdict.retryable is True
|
|
assert verdict.should_fallback is True
|
|
|
|
|
|
def test_the_first_byte_error_content_names_the_bound_and_the_wait() -> None:
|
|
"""Half the 2026-09-10 defect was that nothing was recorded. The content the
|
|
provider hands back is what the loop logs and what the llm.output artefact
|
|
keeps, so the bound and the wait have to survive into it."""
|
|
from raven.providers.base import format_llm_error, parse_llm_error
|
|
from raven.providers.first_byte import FirstByteTimeoutError
|
|
|
|
exc = FirstByteTimeoutError(phase="opening the stream", budget=120, waited=120.4)
|
|
content = format_llm_error(exc, LLMProvider.classify_error(exc), provider="ppt")
|
|
assert "first_byte_timeout@ppt" in content
|
|
assert "120.4s" in content
|
|
assert "llmFirstByteTimeout=120s" in content
|
|
parsed = parse_llm_error(content)
|
|
assert parsed is not None and parsed[0] == "first_byte_timeout"
|
|
|
|
|
|
def test_format_llm_error_collapses_prefixes_and_json_body():
|
|
from raven.providers.base import format_llm_error, parse_llm_error
|
|
|
|
exc = _StatusError(
|
|
"litellm.AuthenticationError: AuthenticationError: OpenrouterException - "
|
|
'{"error":{"message":"User not found.","code":401}}',
|
|
status_code=401,
|
|
)
|
|
content = format_llm_error(exc, LLMProvider.classify_error(exc), provider="openrouter")
|
|
|
|
assert content == (
|
|
"Error calling LLM (auth@openrouter): AuthenticationError: OpenrouterException - User not found."
|
|
)
|
|
assert parse_llm_error(content) == (
|
|
"auth",
|
|
"openrouter",
|
|
"AuthenticationError: OpenrouterException - User not found.",
|
|
)
|
|
|
|
|
|
def test_parse_llm_error_rejects_ordinary_content():
|
|
from raven.providers.base import parse_llm_error
|
|
|
|
assert parse_llm_error("a normal reply that mentions Error calling LLM") is None
|
|
assert parse_llm_error(None) is None
|
|
|
|
|
|
def test_strip_json_error_body_keeps_trailing_text():
|
|
from raven.providers.base import _strip_json_error_body
|
|
|
|
assert _strip_json_error_body('X - {"error":{"message":"boom"}} (request id: abc)') == "X - boom (request id: abc)"
|
|
|
|
|
|
def test_strip_json_error_body_without_a_message_leaves_text_alone():
|
|
from raven.providers.base import _strip_json_error_body
|
|
|
|
text = 'Config invalid: {"foo": "bar"} retry with a valid key'
|
|
assert _strip_json_error_body(text) == text
|
|
|
|
|
|
def test_a_gateway_upstream_failure_is_retried_rather_than_fatal():
|
|
"""The wording OpenRouter uses when the host behind it failed.
|
|
|
|
It answers 200 with that body, so there is no status code and no exception
|
|
class to read, and the phrase matches none of the other server-bucket
|
|
substrings -- it fell through to `unknown`, which is neither retryable nor a
|
|
fallback. A measured run died on it at iteration 72 after 82 minutes, while
|
|
three earlier upstream failures in the same run recovered on their first
|
|
retry because those had arrived worded as "service unavailable".
|
|
"""
|
|
from raven.providers.base import LLMProvider
|
|
|
|
verdict = LLMProvider.classify_error(content="APIError: OpenrouterException - Provider returned error")
|
|
|
|
assert verdict.retryable and verdict.should_fallback
|
|
|
|
# And through the wrapper the loop actually logs, which is the form the run's
|
|
# own transcript carried.
|
|
wrapped = LLMProvider.classify_error(
|
|
content="Error calling LLM (unknown@ppt): APIError: OpenrouterException - Provider returned error"
|
|
)
|
|
assert wrapped.retryable and wrapped.should_fallback
|
|
|
|
|
|
def test_the_same_wording_over_a_permanent_refusal_stays_fatal():
|
|
"""The reason the branch is last and not in the server bucket.
|
|
|
|
"Provider returned error" is the *outer* wrapper OpenRouter puts on anything an
|
|
upstream host returns, so it also heads a permanent 400 whose real cause sits in
|
|
`metadata.raw`. Putting the phrase in the server bucket made this exact wire text
|
|
-- an endpoint refusing an image inside a role="tool" message -- read as a
|
|
transient failure worth four retries, which lost the one verdict that can be
|
|
acted on.
|
|
"""
|
|
from raven.providers.base import LLMProvider
|
|
|
|
refusal = (
|
|
"litellm.BadRequestError: OpenrouterException - "
|
|
'{"error":{"message":"Provider returned error","code":400,"metadata":{"raw":'
|
|
"\"Image URLs are only allowed for messages with role 'user', but this message "
|
|
"with role 'tool' contains an image URL.\"}}}"
|
|
)
|
|
|
|
verdict = LLMProvider.classify_error(content=refusal)
|
|
|
|
assert verdict.should_drop_tool_images is True
|
|
assert not verdict.retryable
|
|
|
|
|
|
def test_a_permanent_refusal_is_still_not_retried():
|
|
"""The bucket above is worded narrowly on purpose: a provider that refuses the
|
|
request rather than failing to serve it must stay fatal, or every bad key and
|
|
every rejected prompt buys four retries and a fallback."""
|
|
from raven.providers.base import LLMProvider
|
|
|
|
for text in ("invalid api key", "invalid_request: tools are not supported"):
|
|
verdict = LLMProvider.classify_error(content=text)
|
|
assert not verdict.retryable, text
|
|
|
|
|
|
# --- the failures that ended long autonomous runs ------------------------------ #
|
|
|
|
|
|
def test_a_non_json_body_is_a_transient_gateway_failure():
|
|
"""OpenRouter served something that was not JSON, and the client's parser is what
|
|
failed: the message names a character offset, not a cause. One 62-minute build
|
|
ended on it as `unknown`, which was fatal at the time."""
|
|
verdict = LLMProvider.classify_error(
|
|
Exception("OpenrouterException - Unable to get json response - Expecting value: line 177 column 1 (char 968)")
|
|
)
|
|
|
|
assert verdict.category == "unparsable_response"
|
|
assert verdict.retryable and verdict.should_fallback
|
|
|
|
|
|
def test_a_wording_nobody_has_named_is_still_retried():
|
|
"""The third unrecognised wording was always going to arrive; the default is what
|
|
changed. No fallback, because an unnamed cause is not evidence another model helps."""
|
|
verdict = LLMProvider.classify_error(Exception("OpenrouterException - something nobody has seen before"))
|
|
|
|
assert verdict.category == "unknown"
|
|
assert verdict.retryable and not verdict.should_fallback
|
|
|
|
|
|
def test_an_image_refused_for_its_size_is_stripped_not_retried():
|
|
"""Measured twice after a two-page render went to the model as pictures. Waiting
|
|
does not shrink the bytes and moving them does not either; the recovery is to
|
|
take the picture out and ask again."""
|
|
verdict = LLMProvider.classify_error(Exception("OpenrouterException - Downloaded image content cannot exceed 30MB"))
|
|
|
|
assert verdict.category == "image_too_large"
|
|
assert verdict.strip_images
|
|
assert not verdict.retryable and not verdict.should_drop_tool_images
|
|
|
|
|
|
def test_a_model_without_eyes_has_the_pictures_taken_out():
|
|
"""A text-only endpoint refusing a request that carried a render. Waiting does not
|
|
give the model eyes and another message does not either; the pictures come out."""
|
|
verdict = LLMProvider.classify_error(
|
|
Exception("OpenAIException - At most 0 image(s) may be provided in one prompt.")
|
|
)
|
|
|
|
assert verdict.category == "images_unsupported"
|
|
assert verdict.strip_images
|
|
assert not verdict.retryable and not verdict.should_drop_tool_images
|
|
|
|
|
|
# --- the bound and the category table --------------------------------------- #
|
|
|
|
|
|
def test_a_vendors_whole_body_is_bounded_but_still_reads_back():
|
|
"""A rejected-request body runs to kilobytes -- the prompt quoted back, an
|
|
HTML error page -- and the same sentence is read in a chat reply, a session
|
|
marker, a cron job record and a log line. It is cut once, where it is
|
|
built, and the cut sentence is still the machine-readable shape."""
|
|
from raven.providers.base import LLM_ERROR_DETAIL_MAX, format_llm_error, parse_llm_error
|
|
|
|
exc = _StatusError("the request was rejected: " + "x" * 5000, status_code=400)
|
|
content = format_llm_error(exc, LLMProvider.classify_error(exc), provider="openrouter")
|
|
|
|
parsed = parse_llm_error(content)
|
|
assert parsed is not None
|
|
category, provider, detail = parsed
|
|
assert (category, provider) == ("invalid_request", "openrouter")
|
|
assert len(detail) == LLM_ERROR_DETAIL_MAX
|
|
assert detail.endswith("...")
|
|
|
|
|
|
async def test_the_retry_ladder_classifies_a_deep_needle_before_the_sentence_is_bounded():
|
|
"""The place the hazard lives: ``chat_with_retry`` classifies a raised
|
|
exception and only then renders the bounded sentence. A needle four
|
|
kilobytes into a rejected-request body still decides the category the
|
|
response carries, and the detail that reaches the reader is cut."""
|
|
from raven.providers.base import LLM_ERROR_DETAIL_MAX, LLMResponse, parse_llm_error
|
|
|
|
body = "rejected: " + "x" * 4000 + " this model's maximum context length is 8192 tokens"
|
|
|
|
class _Refuses(LLMProvider):
|
|
_CHAT_RETRY_DELAYS = ()
|
|
|
|
def __init__(self) -> None:
|
|
super().__init__(api_key="test")
|
|
|
|
def get_default_model(self) -> str:
|
|
return "stub"
|
|
|
|
async def chat(self, messages, tools=None, model=None, max_tokens=4096, temperature=0.7, **_):
|
|
raise _StatusError(body, status_code=400)
|
|
|
|
async def chat_stream(self, *args, **kwargs): # pragma: no cover - the non-stream path is under test
|
|
raise NotImplementedError
|
|
|
|
response = await _Refuses().chat_with_retry(messages=[{"role": "user", "content": "hi"}])
|
|
|
|
assert isinstance(response, LLMResponse) and response.finish_reason == "error"
|
|
assert response.error_classification is not None
|
|
assert response.error_classification.category == "context_overflow"
|
|
parsed = parse_llm_error(response.content or "")
|
|
assert parsed is not None and parsed[0] == "context_overflow"
|
|
assert len(parsed[2]) <= LLM_ERROR_DETAIL_MAX + 3
|
|
assert "maximum context length" not in (response.content or "")
|
|
|
|
|
|
def test_a_needle_past_the_bound_still_decides_the_category():
|
|
"""The bound is applied after the verdict is taken, not before. A vendor
|
|
that buries "context length exceeded" behind four kilobytes of echoed
|
|
prompt must still be reported as an overflow -- classified by the string
|
|
the constructor was handed, whose category then travels in the head."""
|
|
from raven.providers.base import format_llm_error, parse_llm_error
|
|
|
|
body = "rejected: " + "x" * 4000 + " this model's maximum context length is 8192 tokens"
|
|
verdict = LLMProvider.classify_error(_StatusError(body, status_code=400))
|
|
assert verdict.category == "context_overflow"
|
|
|
|
content = format_llm_error(_StatusError(body, status_code=400), verdict, provider="openrouter")
|
|
parsed = parse_llm_error(content)
|
|
assert parsed is not None and parsed[0] == "context_overflow"
|
|
assert "maximum context length" not in content
|
|
|
|
|
|
def test_the_category_table_answers_for_every_category_the_classifier_can_reach():
|
|
"""The summary is what a chat reader is told, so a category with no clause
|
|
would reach them as the default and say nothing useful. Read off the
|
|
classifier's own source rather than a hand-kept list, plus the categories
|
|
providers attach directly, so a new bucket fails here instead of going
|
|
unsaid."""
|
|
import inspect
|
|
import re as _re
|
|
|
|
from raven.providers.base import _LLM_ERROR_SUMMARIES, llm_error_summary
|
|
|
|
source = inspect.getsource(LLMProvider._classify)
|
|
reached = set(_re.findall(r'ErrorClassification\(\s*(?:category=)?"([a-z_]+)"', source))
|
|
assert "unknown" in reached # the fallback arm; a sanity check on the scrape
|
|
reached |= {"tool_image_unsupported", "upstream_transport_failure"}
|
|
assert reached <= set(_LLM_ERROR_SUMMARIES)
|
|
|
|
assert llm_error_summary("auth", "openrouter") == (
|
|
"The provider rejected the credentials (openrouter). The runtime log has the provider's own account."
|
|
)
|
|
assert llm_error_summary("auth") == (
|
|
"The provider rejected the credentials. The runtime log has the provider's own account."
|
|
)
|
|
# A category nothing in the table names still gets a sentence.
|
|
assert llm_error_summary("a_bucket_invented_later") == (
|
|
"The model call failed. The runtime log has the provider's own account."
|
|
)
|
|
|
|
|
|
def test_the_canonical_sentence_has_exactly_one_constructor():
|
|
"""Every reader of this shape -- the CLI's diagnosis, the gateway's channel
|
|
reply, the marker the model reads -- trusts one invariant: the detail is
|
|
bounded. A second place that assembles the sentence by hand is how that
|
|
invariant comes to hold on some failures and not others."""
|
|
import pathlib
|
|
|
|
root = pathlib.Path(__file__).resolve().parent.parent / "raven"
|
|
home = root / "providers" / "base.py"
|
|
hand_built = [
|
|
path.relative_to(root).as_posix()
|
|
for path in root.rglob("*.py")
|
|
if path != home and "Error calling LLM (" in path.read_text(encoding="utf-8")
|
|
]
|
|
assert hand_built == []
|