## Summary Absorbs the first of two remaining commits from the research fork before it stopped iterating. This one lands instruments only: nothing here changes what a shipped run does, and the research flow label does not move because no model-visible text changed. `raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended on the model's own prose. The test is structural, with a textual sub-label for the tail shape. Three decisions are worth review: - It lives under `raven/agent/loop/` rather than in the research plugin, because the loop is what would re-run a dead turn, and a host module cannot import a plugin without breaking `make lint-imports`. - The ask labeller is injected (`ask_kind`) rather than imported, because only the product that writes the harness asks knows how they are worded. With no labeller the failure mode is still recognised structurally and reported as `harness_ask_unknown`, so the boolean never depends on wording. - The fork marked its injected turns with a key of its own. This loop already had an equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather than adding a second marker for the same fact. The conditional rerun that reads the predicate arrived on this branch afterwards, in #418, along with the turn's wall clock. Both are supplied as data a hook writes onto the turn's metadata rather than as config the loop reads, so an agent whose hooks leave no budget runs exactly as it did before either existed: the rerun is unreachable for it and the clock is never consulted. The rerun starts from the original question rather than salvaging the failed attempt, because salvage was measured on 25 real duds to produce content 23 times and the right answer none of them, turning a detectable zero into a confident wrong answer. Three defects in that work were found by review and fixed on this branch, each with a reproduction that is red on the commit before it. The loop numbered episodes from an iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0] and the TUI, which keys episode rows and their fold state by that index, had the rerun's first step collide with the first attempt's. Each attempt took its own clock reading, so a first attempt ending just under the limit handed the rerun a fresh full budget and a ten-second turn ran to eighteen. And the research hook opens its per-turn ledger at iteration 1, which a rerun reaches again, so the second open replaced the path the close reads and left the first attempt's file unreachable on disk. Reading that back afterwards turned up three more, none of them raised in review. The turn's end record carried the attempt's iteration count beside the turn's elapsed time with nothing saying which was which, so it gained ``attempt``. The hook contract described that record as "status and iteration count" while it carried five fields, which hid ``stopped_by`` from anyone writing a terminal gate against the contract. And the rerun's progress line named research in a loop that serves every agent, and called itself the first attempt when the budget allows more than one. A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch results at three chokepoints. It is a second file rather than more columns on the ledger because a 250 KB row would leave no complete line in the 64 KB tail window `_resolve_session_seq` reads, which would silently reset the re-run split. Off unless the environment names a file, and a write failure is logged rather than raised. `DigestOutput` lets a digest return a write-only ledger annotation beside its text. The annotation is stripped from the tool result and merged into the fetch row, so a digest can never label its own output where the model can read the label. Nothing emits one yet; this is the pipe a later digest sidecar uses. `is_hard_tool_failure` was blind to the whole reader path. `web_fetch` never returns a bare string, it returns a JSON envelope, so a spent quota, a rejected URL and a failed validation all read as success and reset the streak `web_search` had accumulated. Transient markers are still tested first, so a 429 inside an envelope stays retryable. Forced finalize now books its own salvage spend. The phase fires on every run that produced no answer and burns up to `max_tokens` doing it, and it was counted nowhere. The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is the exemption zone this repo keeps Chinese forms in, and the predicate imports it from there rather than carrying a literal outside the zone. Two defects found in the first review round are fixed on this head, both reproduced before being changed. The verbatim sink borrowed the ledger's `_resolve_session_seq`, which recovers the re-run split by reading the last 64 KB of the file and parsing the last complete line. That works for the ledger because its rows are small; this sink stores the large ones, so the first record wider than the window leaves nothing parseable and the sequence resets to 1, making two runs indistinguishable. Runs are now told apart by a tag minted in memory once per process and never recovered from disk, which removes the read a large row defeats. The argument for keeping this file separate from the ledger was already in its docstring; it applies to the sequence too, and now says so. `is_hard_tool_failure` started counting JSON envelopes while `failure_class` still read them as undifferentiated text. The streak key is `(tool, failure_class)` and the break threshold is two, so a blocked URL followed by a reader's HTTP refusal fired the stop-repeating nudge at a model that had changed both its cause and its approach. Both predicates now share one envelope parse, and an envelope is classified by its own `error` string rather than by a textual ladder that cannot see into one. Coarseness is for model prose, which varies without meaning anything; a tool writes its error from a small fixed vocabulary, so equal strings are the same failure and different ones are different. The parts that vary with the page travel in the envelope's other keys, so they cannot split a streak. The last commit removes an `isinstance` guard in the envelope reader that the diff-coverage report flagged as the only uncovered changed line. JSON has one shape that opens with a brace, so the branch was unreachable; removed rather than covered. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run at this head (95c975b3), on `origin/main` with nothing to rebase onto. ``` uv run --frozen --python 3.12 --all-extras pytest -q tests/ 22132 passed, 43 skipped, 109 warnings in 388.67s ``` Integration and e2e are deselected by default (`-m "not integration and not e2e"`), so the run above does not include them. This change adds no integration surface. ``` make coverage make coverage-ratchet line current=88.66% baseline=87.30% delta=+1.36pp branch current=81.59% baseline=79.43% delta=+2.15pp Coverage ratchet passed with 0.05pp tolerance. make coverage-baseline-check Coverage baseline update is monotonic. COVERAGE_BASE_REF=origin/main make coverage-diff Diff coverage: 100.00% (126/126 executable changed lines) ``` Diff coverage reported 99.21% on the commit before last, with one uncovered changed line: the sentence the rerun says to the reader, which no test drove because none passed an `on_progress` into a retrying turn. The last commit pins it rather than waiving it. ``` make lint-python All checks passed! / 1946 files already formatted make lint-imports Contracts: 10 kept, 0 broken. make lint-deps Success! No dependency issues found. make check-source-language clean make check-large-files clean make check-commits clean ``` The kernel budget gate caught one thing worth naming: documenting the turn-end record in the hook contract put `raven/contracts` at 2808 lines against a ceiling of 2791. The per-field detail moved to the write site in `turn_path`, which is not budgeted, and the paper now names the fields and says where the rest is. `raven/contracts` is at 2788. Each of the three review fixes was proved against the commit it fixes rather than reasoned about: a detached worktree at that commit, the new tests copied in, red there and green here. They reproduce the reviewer's own numbers -- the episode list `[0, 1, 2, 0]`, the eighteen seconds, and the stranded ledger file. The strengthened comparison in `test_the_retry_starts_from_the_question_not_from_the_wreckage` is a tightened guarantee rather than a regression test and passes either way, which is why it is called out here. `agents/raven-research/.env` was moved aside for every suite run: it is gitignored and absent in CI, and two launcher tests that delete a key from the environment otherwise find one in the file. This machine has no search-provider key, so no research brief can actually be run here. The retrieval-path changes are verified by unit tests only, which is why the three chokepoints are pinned by tests that assert both the presence and the absence of a row rather than by an observed run. The same limit applies to the rerun: the clock and the dead-end trigger are driven by a deterministic clock in tests, never by a real turn. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed No user-facing surface changed, so there is nothing to document. The one operator-visible knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and is documented in the module that reads it. ## Risk The `is_hard_tool_failure` change is the only unconditional one, and it reaches every agent rather than just research: any tool returning a JSON envelope with a non-empty top-level `error` now counts toward the tool-failure streak that triggers the change-approach nudge. That is the intended correction, and the direction is conservative because transient markers are still tested first. Reverting it is a single branch. Everything else is inert by default. The body sink writes nothing unless the environment names a file. `DigestOutput` has no emitter in this change, and a digest returning a plain string produces the same envelope it always did, which `test_a_bare_string_digest_still_works` pins. The dead-end predicate now has a caller, and it reaches only an agent whose hooks write a budget onto the turn's metadata. No agent on this branch does: `agents/raven-research` ships `wallClockSeconds` and inherits the rerun from its plugin's class default, and every other agent leaves the key unwritten and is bounded exactly as before. The turn's overrun is one budget plus at most one iteration, because the clock is read between iterations and never mid-generation: cancelling a call in flight would discard a finished generation and leave no answer at all. Rolling the rerun back is the budget key; rolling the clock back is the same. Rollback is the whole commit; nothing here is depended on by anything already on main. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com> Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
151 lines
6.6 KiB
Python
151 lines
6.6 KiB
Python
"""The dead-end predicate: is a finished turn worth running again from scratch?
|
|
|
|
The predicate is the whole mechanism until a loop consumes it, so these tests pin
|
|
what it classifies rather than what anything does about it. Two of them exist for
|
|
pairings that would have silently emptied the trigger: an appendix wrapped around
|
|
the no-answer sentence, and a budget-exhausted turn that wrapped up with a real
|
|
answer.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from raven.agent.loop._shared import _HOOK_INJECTED_KEY
|
|
from raven.agent.loop.dead_end import (
|
|
NO_RESPONSE_FALLBACK,
|
|
REFUSAL_MARKERS,
|
|
dead_reasons,
|
|
stranded_of,
|
|
)
|
|
|
|
|
|
def _asst(text: str) -> dict:
|
|
return {"role": "assistant", "content": text}
|
|
|
|
|
|
def _injected(text: str) -> dict:
|
|
return {"role": "user", "content": text, _HOOK_INJECTED_KEY: True}
|
|
|
|
|
|
def test_an_appendix_wrapped_dud_is_still_a_dead_end():
|
|
"""The pairing that would have silently disabled the lever on the shipped config.
|
|
|
|
The loop fills an answerless turn with the no-answer sentence, and the research
|
|
appendix then wraps a full trail around it. Several hundred characters of real
|
|
content, and nothing in it is an answer.
|
|
"""
|
|
dressed = NO_RESPONSE_FALLBACK + "\n## How this was researched\n- 3 searches, 2 pages"
|
|
spoke = [{"role": "user", "content": "q"}, _asst("still looking")]
|
|
assert dead_reasons(messages=spoke, final_content=dressed) == ["no_response"]
|
|
assert len(dressed) > 100
|
|
|
|
|
|
def test_a_real_answer_is_not_a_dead_end():
|
|
msgs = [{"role": "user", "content": "q"}, _asst("## Answer\nAlice Smith.")]
|
|
assert dead_reasons(messages=msgs, final_content="## Answer\nAlice Smith.") == []
|
|
|
|
|
|
def test_a_budget_exhausted_turn_that_wrapped_up_is_not_dead():
|
|
"""The name collision that would make a retry double a clock it had just spent.
|
|
|
|
``"interrupted"`` is the turn outcome's status and it means the turn hit its
|
|
iteration cap or wall clock -- which runs the exhaustion wrap-up and returns a
|
|
real answer. Reading it as "the run did not end cleanly" costs a whole second
|
|
budget.
|
|
"""
|
|
wrapped = "Wrapped up: found X, Y unverified."
|
|
msgs = [{"role": "user", "content": "q"}, _asst(wrapped)]
|
|
assert dead_reasons(messages=msgs, final_content=wrapped, status="interrupted") == []
|
|
# ...but an exhausted turn that produced nothing is still dead, on the missing
|
|
# answer rather than on the budget.
|
|
silent = [{"role": "user", "content": "q"}, {"role": "tool", "content": "r"}]
|
|
assert dead_reasons(messages=silent, final_content=None, status="interrupted") == [
|
|
"stranded:tool_result",
|
|
"empty_answer",
|
|
]
|
|
|
|
|
|
def test_a_turn_that_blew_up_is_dead():
|
|
msgs = [{"role": "user", "content": "q"}, _asst("partial")]
|
|
assert dead_reasons(messages=msgs, final_content="partial", status="error") == ["status:error"]
|
|
|
|
|
|
def test_stranded_is_reported_first():
|
|
"""Order is the design: the structural test must not sit behind the proxies."""
|
|
msgs = [{"role": "user", "content": "q"}, {"role": "tool", "content": "results"}]
|
|
assert dead_reasons(messages=msgs, final_content=None)[0] == "stranded:tool_result"
|
|
assert stranded_of(msgs) == (True, "tool_result")
|
|
|
|
|
|
def test_a_declined_answer_is_dead_however_long_it_is():
|
|
"""The refusal opener is read from the module rather than spelled here: the
|
|
strings are Chinese and live in the i18n lexicon, which is the only place in
|
|
this repo that carries them."""
|
|
declined = REFUSAL_MARKERS[0] + ", " + "x" * 500
|
|
msgs = [{"role": "user", "content": "q"}, _asst(declined)]
|
|
assert dead_reasons(messages=msgs, final_content=declined) == ["refusal_string"]
|
|
|
|
|
|
def test_an_empty_message_list_is_not_stranded():
|
|
"""``stranded`` says the trajectory ended somewhere other than model prose. A
|
|
trajectory with nothing in it did not end anywhere, and reporting it as stranded
|
|
would put every construction error into the same bucket as a real failure."""
|
|
assert stranded_of([]) == (False, None)
|
|
|
|
|
|
def test_the_sub_label_names_the_tail_shape():
|
|
question_only = [{"role": "user", "content": "q"}]
|
|
tool_call = [_asst("")] + [{"role": "assistant", "content": "", "tool_calls": [{"id": "1"}]}]
|
|
empty_assistant = [{"role": "user", "content": "q"}, _asst(" ")]
|
|
assert stranded_of(question_only) == (True, "question")
|
|
assert stranded_of(tool_call) == (True, "tool_call")
|
|
assert stranded_of(empty_assistant) == (True, "empty_assistant")
|
|
|
|
|
|
def test_a_harness_ask_is_stranded_even_with_no_labeller():
|
|
"""The boolean is structural, the sub-label is textual, and only the second one
|
|
needs the product's wording. A caller that supplies no labeller still gets the
|
|
failure mode, under a name that says the label is missing rather than absent."""
|
|
msgs = [{"role": "user", "content": "q"}, _injected("[finalize] commit now.")]
|
|
assert stranded_of(msgs) == (True, "harness_ask_unknown")
|
|
|
|
|
|
def test_a_supplied_labeller_names_the_ask():
|
|
msgs = [{"role": "user", "content": "q"}, _injected("[finalize] commit now.")]
|
|
kinds = {"[finalize]": "finalize"}
|
|
|
|
def _ask_kind(content: object) -> str | None:
|
|
return next((v for k, v in kinds.items() if str(content).startswith(k)), None)
|
|
|
|
assert stranded_of(msgs, ask_kind=_ask_kind) == (True, "finalize")
|
|
assert dead_reasons(messages=msgs, final_content=None, ask_kind=_ask_kind)[0] == "stranded:finalize"
|
|
|
|
|
|
def test_an_injected_assistant_turn_is_not_the_models_own_prose():
|
|
"""Every injected entry carries the marker here, not only the user-role ones, so
|
|
a harness-written assistant turn cannot pass as the answer the model gave."""
|
|
msgs = [{"role": "user", "content": "q"}, {**_asst("the harness wrote this"), _HOOK_INJECTED_KEY: True}]
|
|
assert stranded_of(msgs)[0] is True
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"content,spoke",
|
|
[
|
|
([{"type": "text", "text": "Alice."}], True),
|
|
([{"type": "image", "url": "x"}], False),
|
|
("Alice.", True),
|
|
(None, False),
|
|
(42, True),
|
|
],
|
|
)
|
|
def test_what_counts_as_the_models_own_prose(content, spoke):
|
|
"""Channel-separated backends deliver a list of typed blocks, most deliver a
|
|
string, and a stub can deliver anything. A turn ended on prose when flattening
|
|
its last message leaves text; ``None`` flattens to nothing, and an unexpected
|
|
type is stringified rather than silently read as empty -- an instrument that
|
|
reports "no answer" for a shape it did not anticipate would invent dead turns.
|
|
"""
|
|
msgs = [{"role": "user", "content": "q"}, {"role": "assistant", "content": content}]
|
|
assert stranded_of(msgs) == ((False, None) if spoke else (True, "empty_assistant"))
|