## Summary A failed turn's Web row can expand to the provider's full diagnostic text, up to 8,000 characters, while the existing bounded reason stays unchanged. The optional `full_detail` field travels through the runner, lane, RPC event and session marker, so live display and replay expose the same account. Copy includes the expanded account. Quiet runtime notices use "Show details" instead of the error label. Failure details and call-record bodies remove known credential shapes before the final cap. An unterminated private-key block is removed through the end of the input, including when an earlier scan or call record cut away its closing armour. Known cut boundaries also remove incomplete JWTs, vendor tokens, URL passwords and labelled credential tails; each of those patterns is searched only from behind the run of its own value's characters that ends the input and the label in front of it, so the scan stays linear whatever the text holds and still reads a label gap of any width. The authorization and JWT patterns avoid repeated backtracking, and the turn path and outer runner reuse the computed detail. Attachment and lane reads use the base exception's instance storage, so a foreign read-only property or throwing getter cannot replace the scrubbed value or swallow the terminal failure event. The optional fields are mirrored in the RPC schema and both generated clients. The design document separates this delivery from the subsequent Turn Continuation feature; Retry still starts an ordinary turn. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification The automated suites and static/build checks below were re-run after synchronizing with main, using the existing local dependencies. The independent behavioral and timing probes were measured before that synchronization. The last three commits change only `raven/security/redact.py`, two test files and the design document; on that final tree the Python suite, the ui-web tests, the type, generation and lint checks, `ty`, the import contracts, the range gates and pre-commit were run again, and the ui-web build checks were not, since none of their inputs changed. - `uv run --no-sync pytest -q tests/test_agent_loop_failure_text.py tests/test_llm_call_record.py tests/test_agent_loop_broken_turn.py tests/test_agent_loop_llm_error_patience.py tests/test_agent_loop_run_emit.py tests/test_spine_scheduler_lane.py tests/test_spine_events.py tests/test_rpc_spine.py tests/test_rpc_schema_match.py tests/test_kernel_budget.py tests/test_acp_redact.py tests/test_security_untrusted_context.py -x --tb=short --show-capture=no`: 904 passed. - From `ui-web`: `npm test -- --no-file-parallelism src/features/transcript/TranscriptPage.test.tsx src/state/session/pipeline.events.test.ts scripts/gates/i18n-keys.test.mjs`: 225 passed. - From `ui-web`: `npm run type-check`, `npm run gen:check`, `node ../ui-tui/scripts/gen-i18n.mjs --check`, `node ../ui-tui/scripts/gen-rpc-types.mjs --check`, and `node node_modules/eslint/bin/eslint.js src/features/transcript/TranscriptPage.tsx src/features/transcript/TranscriptPage.test.tsx`: passed. - `uv run --no-sync lint-imports`: 10 contracts kept. `uv run --no-sync ty check raven evolver agents plugins-dist scripts`: passed. Ruff check and format check passed on the 20 Python files this PR changes. - From `ui-web`: `npm run build`, `uv run --no-sync python build.py`, `node scripts/check-page.mjs`, `node scripts/check-css.mjs`, and `node scripts/check-class-namespace.mjs`: passed, including both boot snapshots. - `UV_NO_SYNC=1 make check-large-files check-source-language COMMIT_RANGE=origin/main..HEAD`: passed. `PYTHONPATH=. uv run --no-sync python scripts/check_commit_messages.py origin/main..HEAD`, commitlint over the same range and `git diff --check origin/main...HEAD`: passed. - `uv run --no-sync pre-commit run --from-ref origin/main --to-ref HEAD`: all applicable hooks passed except the whole-TUI Prettier hook, which reports `newInstancePicker.tsx` and `subagentsHub.tsx`. Both files and the formatter config are unchanged from main; an isolated main-file control with the same locked Prettier version reproduces both failures. `node ui-tui/node_modules/prettier/bin/prettier.cjs --check ui-tui/src/i18n/messages.generated.ts`: passed. - Regression coverage includes truncated private keys, incomplete JWT and credential tails, terminal-event delivery with foreign exception attributes, scan reuse and CPU cost, the cut-tail scan's cost on label-shaped runs and long gaps, its agreement with a whole-text search, a label gap of any width, the JWT boundary and private-key END rules, and quiet-notice expansion. The five independent boundary probes passed before the base synchronization; the committed regression suite above passed again afterwards. - On the same 32,267-character synthetic crash input, the queued event-loop callback delay fell from 15.351 seconds to 0.00951 seconds. The regression checks use thread CPU time with a 0.5-second ceiling. - Cut-tail scan: a failed turn whose crash text holds 31,000 spaces before its last two words took 32.6 seconds while the region came from a regex search for the last whitespace, and 0.016 seconds now; 64,000 characters of `pwd=` ending in a comma took 10.8 seconds through the whole-text tail search and 0.003 seconds now, and a 300,000-character call-record body of either shape takes under 0.3 seconds. These and the figures below were measured with the machine's load average between 36 and 132. Across 3,728 timed texts, every label or prefix unit repeated to 64,000 characters behind a cut and to 300,000 past the scan cap, the slowest measured alone took about 0.2 seconds against about 0.1 seconds on the base revision, and a text past the cap takes the same time at twice the size. Searching the whole text again turns six timing cases red; a smaller name room, a skipped gap or scheme step, or any value class narrowed below its pattern's turns the property or gap test red, as does removing the JWT boundary or restoring the END-only private-key rule. On 12,008 generated texts with gaps up to 5,000 characters, the output equals the whole-text version's. - Chromium with synthetic local data: full text retained and HTML escaped, expanded tooltip removed, internal scrolling and Retry preserved, neutral notice label displayed, no page errors. No live vendor call or native Windows rerun is claimed for these fixes. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - [x] Security impact considered: the long text is scrubbed of recognized credential shapes before persistence and publication. It remains diagnostic text and may contain hostnames or paths. Model history and chat-channel replies keep their existing bounded content. A session export can include the long text. `raven.security.redact` is also the scrubber for every ACP surface (tool titles, result previews, permission prompts, replay, error messages) and for the outbound client's permission log, so two of its rules reach those surfaces too: a private-key header without END hides the rest of that text, and a JWT glued to a preceding word character, such as `/verify-eyJ...`, is left unredacted. Tests pin both. - [x] Backward compatibility considered: the new fields are optional, older markers fall back to their bounded reason, and the TUI can ignore the extra field. Incomplete private-key blocks are conservatively removed to the end, which can also omit trailing diagnostic text. Custom runners attach long details in the exception's instance dictionary; descriptors are ignored. - [x] Rollback path is clear for risky changes: revert the squash commit. Older code ignores the optional marker field. ## Related Issues N/A --------- Co-authored-by: Codex <noreply@openai.com>
345 lines
13 KiB
Python
345 lines
13 KiB
Python
"""The long account of a failed turn: unbounded, scrubbed, capped, and attached
|
|
only when it says more than the 200-character text already does."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import time
|
|
|
|
import pytest
|
|
|
|
from raven.agent.loop import failure_text
|
|
from raven.agent.loop.failure_text import (
|
|
attach_full_detail,
|
|
exception_failure_text,
|
|
full_failure_text,
|
|
response_failure_text,
|
|
)
|
|
from raven.contracts.llm_provider import CallRecord
|
|
from raven.providers.call_record import MAX_BODY_CHARS, REPLACEMENT, from_exception
|
|
from raven.spine.scheduler import describe_failure
|
|
from raven.spine.turn import AnswerlessTurnError
|
|
|
|
KEY = "sk-or-v1-" + "a" * 40
|
|
GOOGLE_KEY = "AIza" + "Ab1_-" * 7
|
|
GITHUB_TOKEN = "ghp_" + "a1B2" * 9
|
|
SLACK_TOKEN = "xoxb-" + "1234567890-abcdefghijKLMNOP"
|
|
|
|
|
|
class _Frozen(Exception):
|
|
def __setattr__(self, name, value):
|
|
raise AttributeError(name)
|
|
|
|
|
|
def test_the_long_account_keeps_what_the_short_one_cuts():
|
|
assert full_failure_text("upstream said: " + "x" * 600) == "upstream said: " + "x" * 600
|
|
|
|
|
|
def test_a_credential_in_the_account_is_scrubbed():
|
|
text = full_failure_text(f"401 from the gateway, Authorization: Bearer {KEY}")
|
|
assert KEY not in text
|
|
assert REPLACEMENT in text
|
|
|
|
|
|
def test_a_json_quoted_authorization_value_in_a_body_is_scrubbed():
|
|
"""A provider can echo the request's headers back as JSON. A token with no
|
|
vendor shape is caught only by the call record's body scrubber, which reads a
|
|
JSON-quoted ``Authorization`` value; ``redact`` alone passes it through."""
|
|
token = "gw_" + "q7" * 20
|
|
body = '{"error": {"message": "unauthorized", "headers": {"Authorization": "Bearer ' + token + '"}}}'
|
|
text = full_failure_text(f"401 from the gateway\n\n{body}")
|
|
assert token not in text
|
|
assert REPLACEMENT in text
|
|
|
|
|
|
def test_the_account_is_capped_at_the_call_record_body_ceiling():
|
|
text = full_failure_text("y" * (MAX_BODY_CHARS * 2))
|
|
assert len(text) == MAX_BODY_CHARS
|
|
assert text.endswith("...")
|
|
|
|
|
|
def test_an_account_exactly_at_the_ceiling_is_kept_whole():
|
|
text = "y" * MAX_BODY_CHARS
|
|
assert full_failure_text(text) == text
|
|
|
|
|
|
def test_an_exception_is_named_the_way_the_lane_names_it():
|
|
assert exception_failure_text(ValueError("boom")) == "ValueError: boom"
|
|
assert exception_failure_text(TimeoutError()) == "TimeoutError"
|
|
|
|
|
|
def test_an_answerless_turn_prefers_its_own_long_account():
|
|
exc = AnswerlessTurnError(
|
|
"Error calling LLM (rate_limit@openrouter): Provider returned error", full_detail="the body"
|
|
)
|
|
assert exception_failure_text(exc) == "the body"
|
|
|
|
|
|
def test_a_failed_call_reads_as_its_line_then_its_body():
|
|
record = CallRecord(http_status=429, body='{"error":{"metadata":{"raw":"rate-limited upstream"}}}')
|
|
text = response_failure_text("Error calling LLM (rate_limit@openrouter): Provider returned error", record)
|
|
assert text == (
|
|
"Error calling LLM (rate_limit@openrouter): Provider returned error\n\n"
|
|
'{"error":{"metadata":{"raw":"rate-limited upstream"}}}'
|
|
)
|
|
|
|
|
|
def test_a_body_the_line_already_quotes_is_not_repeated():
|
|
record = CallRecord(body="no first byte")
|
|
line = "Error calling LLM (first_byte_timeout): no first byte"
|
|
assert response_failure_text(line, record) == line
|
|
|
|
|
|
def test_a_call_with_nothing_to_say_has_no_account():
|
|
assert response_failure_text(None, None) is None
|
|
assert response_failure_text(" ", CallRecord()) is None
|
|
|
|
|
|
def test_attached_only_when_it_says_more_than_the_short_text():
|
|
short = ValueError("boom")
|
|
assert attach_full_detail(short) is None
|
|
assert short.full_detail is None
|
|
|
|
long = ValueError("z" * 500)
|
|
attached = attach_full_detail(long)
|
|
assert attached == "ValueError: " + "z" * 500
|
|
assert long.full_detail == attached
|
|
|
|
|
|
def test_a_second_attach_files_the_same_text():
|
|
exc = AnswerlessTurnError("Error calling LLM (server@x): 503", full_detail="503 body " + "b" * 400)
|
|
first = attach_full_detail(exc)
|
|
assert attach_full_detail(exc) == first
|
|
|
|
|
|
def test_a_foreign_full_detail_is_never_passed_on_unscrubbed():
|
|
exc = RuntimeError("boom " + "c" * 300)
|
|
exc.full_detail = f"leaked {KEY}"
|
|
attached = attach_full_detail(exc)
|
|
assert attached == "RuntimeError: boom " + "c" * 300
|
|
assert KEY not in (exc.full_detail or "")
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"exc",
|
|
[
|
|
pytest.param(ValueError("boom"), id="plain-message"),
|
|
pytest.param(RuntimeError("RuntimeError: already named"), id="colon-after-name"),
|
|
pytest.param(RuntimeError("RuntimeError already named"), id="space-after-name"),
|
|
pytest.param(RuntimeError("RuntimeError"), id="bare-name"),
|
|
pytest.param(TimeoutError(), id="no-message"),
|
|
pytest.param(ValueError(" padded "), id="padded-message"),
|
|
],
|
|
)
|
|
def test_a_short_exception_has_the_same_long_account_as_short_one(exc):
|
|
assert exception_failure_text(exc) == describe_failure(exc)
|
|
|
|
|
|
@pytest.mark.parametrize("detail", [" ", ["a"]], ids=["blank", "not-text"])
|
|
def test_a_blank_or_non_text_full_detail_is_no_account(detail):
|
|
exc = AnswerlessTurnError("msg", full_detail=detail)
|
|
assert exception_failure_text(exc) == "msg"
|
|
assert attach_full_detail(exc) is None
|
|
|
|
|
|
def test_an_exception_that_refuses_the_attribute_still_gets_its_long_account():
|
|
assert attach_full_detail(_Frozen("x" * 500)) == "_Frozen: " + "x" * 500
|
|
|
|
|
|
def test_a_failed_call_with_only_a_body_reads_as_that_body():
|
|
assert response_failure_text(None, CallRecord(body="the body")) == "the body"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("text", "secret"),
|
|
[
|
|
pytest.param(
|
|
f"GET https://maps.example.com/v1/geocode?key={GOOGLE_KEY}&address=x",
|
|
GOOGLE_KEY,
|
|
id="key-in-a-url",
|
|
),
|
|
pytest.param(
|
|
f"clone https://user:{GITHUB_TOKEN}@github.com/owner/repo.git failed",
|
|
GITHUB_TOKEN,
|
|
id="userinfo-in-a-url",
|
|
),
|
|
pytest.param("login failed: password=hunter2hunter2 for svc", "hunter2hunter2", id="password-assignment"),
|
|
pytest.param(f"slack rejected {SLACK_TOKEN}", SLACK_TOKEN, id="slack-bot-token"),
|
|
],
|
|
)
|
|
def test_a_credential_in_a_crash_message_is_scrubbed(text, secret):
|
|
out = full_failure_text(text)
|
|
assert secret not in out
|
|
assert REPLACEMENT in out
|
|
|
|
|
|
def test_the_scrubbers_never_read_more_than_their_window(monkeypatch):
|
|
"""Two of the scrubbers' patterns cost more than linear time, and the long
|
|
account is built on the event loop while a turn fails: a huge crash message
|
|
must not reach them whole."""
|
|
seen: list[int] = []
|
|
|
|
def _recording(text: str) -> str:
|
|
seen.append(len(text))
|
|
return text
|
|
|
|
monkeypatch.setattr(failure_text, "scrub_credentials", _recording)
|
|
text = full_failure_text("z" * (10 * MAX_BODY_CHARS))
|
|
assert seen == [4 * MAX_BODY_CHARS]
|
|
assert len(text) == MAX_BODY_CHARS
|
|
|
|
|
|
class _ClientError(Exception):
|
|
def __init__(self, message: str, body: object, status_code: int = 400) -> None:
|
|
super().__init__(message)
|
|
self.body = body
|
|
self.status_code = status_code
|
|
self.response = type("R", (), {"text": "", "headers": {}})()
|
|
|
|
|
|
def test_a_client_error_reads_as_its_message_then_its_body():
|
|
body = {"error": {"message": "stub refusal", "echo": f"Authorization: Bearer {KEY}", "tail": "TAIL-MARKER"}}
|
|
exc = _ClientError("litellm.BadRequestError: OpenAIException - stub refusal", body)
|
|
text = exception_failure_text(exc)
|
|
assert text.startswith("_ClientError: litellm.BadRequestError: OpenAIException - stub refusal\n\n")
|
|
assert "TAIL-MARKER" in text
|
|
assert KEY not in text
|
|
|
|
|
|
def test_a_client_error_is_handed_on_with_its_body():
|
|
exc = _ClientError("litellm.BadRequestError: OpenAIException - stub refusal", {"error": {"tail": "TAIL-MARKER"}})
|
|
attached = attach_full_detail(exc)
|
|
assert attached is not None and "TAIL-MARKER" in attached
|
|
|
|
|
|
def test_a_body_the_message_already_quotes_is_not_repeated():
|
|
body = "the whole body"
|
|
exc = _ClientError("APIError: the whole body", body)
|
|
assert exception_failure_text(exc).count("the whole body") == 1
|
|
|
|
|
|
def _unreadable(attribute: str) -> Exception:
|
|
"""An exception whose ``attribute`` raises something other than AttributeError when read."""
|
|
|
|
def _raise(self):
|
|
raise RuntimeError(f"{attribute} is unreadable")
|
|
|
|
members = {"status_code": 400, attribute: property(_raise)}
|
|
return type("_Unreadable", (Exception,), members)("z" * 500)
|
|
|
|
|
|
@pytest.mark.parametrize("attribute", ["response", "status_code", "body"])
|
|
def test_an_unreadable_attribute_costs_the_body_not_the_account(attribute):
|
|
assert attach_full_detail(_unreadable(attribute)) == "_Unreadable: " + "z" * 500
|
|
|
|
|
|
def test_a_scan_cut_private_key_stays_hidden_after_an_earlier_secret_shrinks():
|
|
secret = "synthetic-private-material-" + "a" * 1600
|
|
pem = "-----BEGIN " + f"PRIVATE KEY-----\n{secret}\n-----END PRIVATE KEY-----"
|
|
text = "Authorization: Bearer " + "b" * 31000 + "\n" + pem
|
|
|
|
full = full_failure_text(text)
|
|
|
|
assert "synthetic-private-material" not in full
|
|
assert len(full) <= MAX_BODY_CHARS
|
|
assert REPLACEMENT in full
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("tail", "visible", "kept"),
|
|
[
|
|
("eyJ" + "A" * 1400 + "." + "B" * 20 + "." + "C" * 20, "eyJ" + "A" * 30, 978),
|
|
("https://user:synthetic-password-" + "a" * 1400 + "@host", "synthetic-password", 978),
|
|
("ghp_" + "A" * 30, "ghp_" + "A" * 8, 12),
|
|
("password=abcdefghijk", "abcde", 14),
|
|
('"Authorization": "Bearer opaque-token-value"', "opaqu", 30),
|
|
],
|
|
ids=["jwt", "url-userinfo", "vendor-token", "named-secret", "quoted-header"],
|
|
)
|
|
def test_a_scan_cut_credential_tail_is_removed_before_earlier_redactions_move_it(tail, visible, kept):
|
|
prefix = "Authorization: Bearer "
|
|
text = prefix + "b" * (4 * MAX_BODY_CHARS - len(prefix) - kept - 1) + "\n" + tail
|
|
assert visible not in full_failure_text(text)
|
|
|
|
|
|
def test_a_previously_capped_record_does_not_expose_a_private_key_fragment():
|
|
body = "diagnostic " * 700 + "\n-----BEGIN " + "PRIVATE KEY-----\nsynthetic-private-material-" + "a" * 1600
|
|
full = response_failure_text("provider refused", CallRecord(body=body[:MAX_BODY_CHARS], body_truncated=True))
|
|
assert "synthetic-private-material" not in full
|
|
|
|
|
|
def test_a_previously_capped_record_removes_an_incomplete_jwt():
|
|
token = "eyJ" + "a" * 100
|
|
full = response_failure_text("provider refused", CallRecord(body=token, body_truncated=True))
|
|
assert token not in full
|
|
assert REPLACEMENT in full
|
|
|
|
|
|
def test_a_client_body_loses_its_private_key_before_the_record_is_capped():
|
|
body = "diagnostic " * 700 + "\n-----BEGIN " + "PRIVATE KEY-----\nsynthetic-private-material-" + "a" * 1600
|
|
record = from_exception(_ClientError("provider refused", body), status=400)
|
|
assert "synthetic-private-material" not in record.body
|
|
|
|
|
|
@pytest.mark.production_timing
|
|
@pytest.mark.parametrize(
|
|
"tail",
|
|
[
|
|
"Authorization:" + " " * 32000 + "!",
|
|
"eyJ" * 10000 + "!",
|
|
# A credential-shaped run that ends before the cut, with other text after
|
|
# it: the tail patterns retry at every position in the run, so each of
|
|
# those attempts walks it again.
|
|
"--token=" * 3900 + " " + "x" * 1500,
|
|
"pwd=" * 7800 + " " + "x" * 1500,
|
|
# The same run stopped by a comma rather than a space, and a long gap
|
|
# before the last two words: the window cut keeps both inside it.
|
|
"pwd=" * 7800 + "," + "x" * 2000,
|
|
" " * 16000 + "x y" + "z" * 17000,
|
|
],
|
|
ids=[
|
|
"authorization-whitespace",
|
|
"jwt-prefixes",
|
|
"cli-flag-run",
|
|
"named-secret-run",
|
|
"unspaced-named-secret-run",
|
|
"whitespace-gap",
|
|
],
|
|
)
|
|
def test_detail_processing_does_not_hold_the_event_loop_for_seconds(tail):
|
|
exc = RuntimeError("diagnostic " * 25 + tail)
|
|
start = time.thread_time()
|
|
attach_full_detail(exc)
|
|
attach_full_detail(exc)
|
|
assert time.thread_time() - start < 0.5
|
|
|
|
|
|
@pytest.mark.parametrize("value", [None, "x" * 500], ids=["no-detail", "long-detail"])
|
|
def test_the_same_failure_is_scrubbed_only_once(monkeypatch, value):
|
|
calls = []
|
|
original = failure_text.exception_failure_text
|
|
|
|
def record(exc):
|
|
calls.append(exc)
|
|
return original(exc)
|
|
|
|
monkeypatch.setattr(failure_text, "exception_failure_text", record)
|
|
exc = RuntimeError(value or "boom")
|
|
first = attach_full_detail(exc)
|
|
assert attach_full_detail(exc) == first
|
|
assert calls == [exc]
|
|
|
|
|
|
@pytest.mark.parametrize("getter_raises", [False, True], ids=["readonly", "unreadable"])
|
|
def test_a_foreign_property_cannot_replace_the_runners_safe_detail(getter_raises):
|
|
class ForeignError(Exception):
|
|
@property
|
|
def full_detail(self):
|
|
if getter_raises:
|
|
raise RuntimeError("cannot read full_detail")
|
|
return "x" * 9000 + " password=synthetic-password"
|
|
|
|
exc = ForeignError("diagnostic " * 25 + " password=synthetic-password")
|
|
expected = attach_full_detail(exc)
|
|
assert "synthetic-password" not in expected
|
|
assert failure_text.full_detail_of(exc) == expected
|