Raven-1/tests/test_agent_loop_dead_end.py
Zuyi Zhou 210e2971c2
feat(*): dead-end predicates, verbatim tool bodies and envelope failures (#410)
## Summary

Absorbs the first of two remaining commits from the research fork before
it stopped
iterating. This one lands instruments only: nothing here changes what a
shipped run does,
and the research flow label does not move because no model-visible text
changed.

`raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended
on the model's own
prose. The test is structural, with a textual sub-label for the tail
shape. Three decisions
are worth review:

- It lives under `raven/agent/loop/` rather than in the research plugin,
because the loop
is what would re-run a dead turn, and a host module cannot import a
plugin without
  breaking `make lint-imports`.
- The ask labeller is injected (`ask_kind`) rather than imported,
because only the product
that writes the harness asks knows how they are worded. With no labeller
the failure mode
is still recognised structurally and reported as `harness_ask_unknown`,
so the boolean
  never depends on wording.
- The fork marked its injected turns with a key of its own. This loop
already had an
equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather
than adding a
  second marker for the same fact.

The conditional rerun that reads the predicate arrived on this branch
afterwards, in
#418, along with the turn's wall clock. Both are supplied as data a hook
writes onto the
turn's metadata rather than as config the loop reads, so an agent whose
hooks leave no
budget runs exactly as it did before either existed: the rerun is
unreachable for it and
the clock is never consulted. The rerun starts from the original
question rather than
salvaging the failed attempt, because salvage was measured on 25 real
duds to produce
content 23 times and the right answer none of them, turning a detectable
zero into a
confident wrong answer.

Three defects in that work were found by review and fixed on this
branch, each with a
reproduction that is red on the commit before it. The loop numbered
episodes from an
iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0]
and the TUI, which
keys episode rows and their fold state by that index, had the rerun's
first step collide
with the first attempt's. Each attempt took its own clock reading, so a
first attempt
ending just under the limit handed the rerun a fresh full budget and a
ten-second turn ran
to eighteen. And the research hook opens its per-turn ledger at
iteration 1, which a rerun
reaches again, so the second open replaced the path the close reads and
left the first
attempt's file unreachable on disk.

Reading that back afterwards turned up three more, none of them raised
in review. The
turn's end record carried the attempt's iteration count beside the
turn's elapsed time
with nothing saying which was which, so it gained ``attempt``. The hook
contract described
that record as "status and iteration count" while it carried five
fields, which hid
``stopped_by`` from anyone writing a terminal gate against the contract.
And the rerun's
progress line named research in a loop that serves every agent, and
called itself the
first attempt when the budget allows more than one.

A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch
results at three
chokepoints. It is a second file rather than more columns on the ledger
because a 250 KB
row would leave no complete line in the 64 KB tail window
`_resolve_session_seq` reads,
which would silently reset the re-run split. Off unless the environment
names a file, and a
write failure is logged rather than raised.

`DigestOutput` lets a digest return a write-only ledger annotation
beside its text. The
annotation is stripped from the tool result and merged into the fetch
row, so a digest can
never label its own output where the model can read the label. Nothing
emits one yet; this
is the pipe a later digest sidecar uses.

`is_hard_tool_failure` was blind to the whole reader path. `web_fetch`
never returns a bare
string, it returns a JSON envelope, so a spent quota, a rejected URL and
a failed
validation all read as success and reset the streak `web_search` had
accumulated.
Transient markers are still tested first, so a 429 inside an envelope
stays retryable.

Forced finalize now books its own salvage spend. The phase fires on
every run that produced
no answer and burns up to `max_tokens` doing it, and it was counted
nowhere.

The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is
the exemption zone
this repo keeps Chinese forms in, and the predicate imports it from
there rather than
carrying a literal outside the zone.

Two defects found in the first review round are fixed on this head, both
reproduced before
being changed.

The verbatim sink borrowed the ledger's `_resolve_session_seq`, which
recovers the re-run
split by reading the last 64 KB of the file and parsing the last
complete line. That works
for the ledger because its rows are small; this sink stores the large
ones, so the first
record wider than the window leaves nothing parseable and the sequence
resets to 1, making
two runs indistinguishable. Runs are now told apart by a tag minted in
memory once per
process and never recovered from disk, which removes the read a large
row defeats. The
argument for keeping this file separate from the ledger was already in
its docstring; it
applies to the sequence too, and now says so.

`is_hard_tool_failure` started counting JSON envelopes while
`failure_class` still read
them as undifferentiated text. The streak key is `(tool, failure_class)`
and the break
threshold is two, so a blocked URL followed by a reader's HTTP refusal
fired the
stop-repeating nudge at a model that had changed both its cause and its
approach. Both
predicates now share one envelope parse, and an envelope is classified
by its own `error`
string rather than by a textual ladder that cannot see into one.
Coarseness is for model
prose, which varies without meaning anything; a tool writes its error
from a small fixed
vocabulary, so equal strings are the same failure and different ones are
different. The
parts that vary with the page travel in the envelope's other keys, so
they cannot split a
streak.

The last commit removes an `isinstance` guard in the envelope reader
that the diff-coverage
report flagged as the only uncovered changed line. JSON has one shape
that opens with a
brace, so the branch was unreachable; removed rather than covered.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run at this head (95c975b3), on `origin/main` with nothing to rebase
onto.

```
uv run --frozen --python 3.12 --all-extras pytest -q tests/
  22132 passed, 43 skipped, 109 warnings in 388.67s
```

Integration and e2e are deselected by default (`-m "not integration and
not e2e"`), so the
run above does not include them. This change adds no integration
surface.

```
make coverage
make coverage-ratchet
  line   current=88.66% baseline=87.30% delta=+1.36pp
  branch current=81.59% baseline=79.43% delta=+2.15pp
  Coverage ratchet passed with 0.05pp tolerance.
make coverage-baseline-check
  Coverage baseline update is monotonic.
COVERAGE_BASE_REF=origin/main make coverage-diff
  Diff coverage: 100.00% (126/126 executable changed lines)
```

Diff coverage reported 99.21% on the commit before last, with one
uncovered changed line:
the sentence the rerun says to the reader, which no test drove because
none passed an
`on_progress` into a retrying turn. The last commit pins it rather than
waiving it.

```
make lint-python
  All checks passed! / 1946 files already formatted
make lint-imports
  Contracts: 10 kept, 0 broken.
make lint-deps
  Success! No dependency issues found.
make check-source-language
  clean
make check-large-files
  clean
make check-commits
  clean
```

The kernel budget gate caught one thing worth naming: documenting the
turn-end record in
the hook contract put `raven/contracts` at 2808 lines against a ceiling
of 2791. The
per-field detail moved to the write site in `turn_path`, which is not
budgeted, and the
paper now names the fields and says where the rest is. `raven/contracts`
is at 2788.

Each of the three review fixes was proved against the commit it fixes
rather than
reasoned about: a detached worktree at that commit, the new tests copied
in, red there and
green here. They reproduce the reviewer's own numbers -- the episode
list `[0, 1, 2, 0]`,
the eighteen seconds, and the stranded ledger file. The strengthened
comparison in
`test_the_retry_starts_from_the_question_not_from_the_wreckage` is a
tightened guarantee
rather than a regression test and passes either way, which is why it is
called out here.

`agents/raven-research/.env` was moved aside for every suite run: it is
gitignored and
absent in CI, and two launcher tests that delete a key from the
environment otherwise find
one in the file.

This machine has no search-provider key, so no research brief can
actually be run here.
The retrieval-path changes are verified by unit tests only, which is why
the three
chokepoints are pinned by tests that assert both the presence and the
absence of a row
rather than by an observed run. The same limit applies to the rerun: the
clock and the
dead-end trigger are driven by a deterministic clock in tests, never by
a real turn.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

No user-facing surface changed, so there is nothing to document. The one
operator-visible
knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and
is documented in
the module that reads it.

## Risk

The `is_hard_tool_failure` change is the only unconditional one, and it
reaches every agent
rather than just research: any tool returning a JSON envelope with a
non-empty top-level
`error` now counts toward the tool-failure streak that triggers the
change-approach nudge.
That is the intended correction, and the direction is conservative
because transient
markers are still tested first. Reverting it is a single branch.

Everything else is inert by default. The body sink writes nothing unless
the environment
names a file. `DigestOutput` has no emitter in this change, and a digest
returning a plain
string produces the same envelope it always did, which
`test_a_bare_string_digest_still_works` pins.

The dead-end predicate now has a caller, and it reaches only an agent
whose hooks write a
budget onto the turn's metadata. No agent on this branch does:
`agents/raven-research`
ships `wallClockSeconds` and inherits the rerun from its plugin's class
default, and every
other agent leaves the key unwritten and is bounded exactly as before.
The turn's overrun
is one budget plus at most one iteration, because the clock is read
between iterations and
never mid-generation: cancelling a call in flight would discard a
finished generation and
leave no answer at all. Rolling the rerun back is the budget key;
rolling the clock back is
the same.

Rollback is the whole commit; nothing here is depended on by anything
already on main.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 15:45:02 +08:00

151 lines
6.6 KiB
Python

"""The dead-end predicate: is a finished turn worth running again from scratch?
The predicate is the whole mechanism until a loop consumes it, so these tests pin
what it classifies rather than what anything does about it. Two of them exist for
pairings that would have silently emptied the trigger: an appendix wrapped around
the no-answer sentence, and a budget-exhausted turn that wrapped up with a real
answer.
"""
from __future__ import annotations
import pytest
from raven.agent.loop._shared import _HOOK_INJECTED_KEY
from raven.agent.loop.dead_end import (
NO_RESPONSE_FALLBACK,
REFUSAL_MARKERS,
dead_reasons,
stranded_of,
)
def _asst(text: str) -> dict:
return {"role": "assistant", "content": text}
def _injected(text: str) -> dict:
return {"role": "user", "content": text, _HOOK_INJECTED_KEY: True}
def test_an_appendix_wrapped_dud_is_still_a_dead_end():
"""The pairing that would have silently disabled the lever on the shipped config.
The loop fills an answerless turn with the no-answer sentence, and the research
appendix then wraps a full trail around it. Several hundred characters of real
content, and nothing in it is an answer.
"""
dressed = NO_RESPONSE_FALLBACK + "\n## How this was researched\n- 3 searches, 2 pages"
spoke = [{"role": "user", "content": "q"}, _asst("still looking")]
assert dead_reasons(messages=spoke, final_content=dressed) == ["no_response"]
assert len(dressed) > 100
def test_a_real_answer_is_not_a_dead_end():
msgs = [{"role": "user", "content": "q"}, _asst("## Answer\nAlice Smith.")]
assert dead_reasons(messages=msgs, final_content="## Answer\nAlice Smith.") == []
def test_a_budget_exhausted_turn_that_wrapped_up_is_not_dead():
"""The name collision that would make a retry double a clock it had just spent.
``"interrupted"`` is the turn outcome's status and it means the turn hit its
iteration cap or wall clock -- which runs the exhaustion wrap-up and returns a
real answer. Reading it as "the run did not end cleanly" costs a whole second
budget.
"""
wrapped = "Wrapped up: found X, Y unverified."
msgs = [{"role": "user", "content": "q"}, _asst(wrapped)]
assert dead_reasons(messages=msgs, final_content=wrapped, status="interrupted") == []
# ...but an exhausted turn that produced nothing is still dead, on the missing
# answer rather than on the budget.
silent = [{"role": "user", "content": "q"}, {"role": "tool", "content": "r"}]
assert dead_reasons(messages=silent, final_content=None, status="interrupted") == [
"stranded:tool_result",
"empty_answer",
]
def test_a_turn_that_blew_up_is_dead():
msgs = [{"role": "user", "content": "q"}, _asst("partial")]
assert dead_reasons(messages=msgs, final_content="partial", status="error") == ["status:error"]
def test_stranded_is_reported_first():
"""Order is the design: the structural test must not sit behind the proxies."""
msgs = [{"role": "user", "content": "q"}, {"role": "tool", "content": "results"}]
assert dead_reasons(messages=msgs, final_content=None)[0] == "stranded:tool_result"
assert stranded_of(msgs) == (True, "tool_result")
def test_a_declined_answer_is_dead_however_long_it_is():
"""The refusal opener is read from the module rather than spelled here: the
strings are Chinese and live in the i18n lexicon, which is the only place in
this repo that carries them."""
declined = REFUSAL_MARKERS[0] + ", " + "x" * 500
msgs = [{"role": "user", "content": "q"}, _asst(declined)]
assert dead_reasons(messages=msgs, final_content=declined) == ["refusal_string"]
def test_an_empty_message_list_is_not_stranded():
"""``stranded`` says the trajectory ended somewhere other than model prose. A
trajectory with nothing in it did not end anywhere, and reporting it as stranded
would put every construction error into the same bucket as a real failure."""
assert stranded_of([]) == (False, None)
def test_the_sub_label_names_the_tail_shape():
question_only = [{"role": "user", "content": "q"}]
tool_call = [_asst("")] + [{"role": "assistant", "content": "", "tool_calls": [{"id": "1"}]}]
empty_assistant = [{"role": "user", "content": "q"}, _asst(" ")]
assert stranded_of(question_only) == (True, "question")
assert stranded_of(tool_call) == (True, "tool_call")
assert stranded_of(empty_assistant) == (True, "empty_assistant")
def test_a_harness_ask_is_stranded_even_with_no_labeller():
"""The boolean is structural, the sub-label is textual, and only the second one
needs the product's wording. A caller that supplies no labeller still gets the
failure mode, under a name that says the label is missing rather than absent."""
msgs = [{"role": "user", "content": "q"}, _injected("[finalize] commit now.")]
assert stranded_of(msgs) == (True, "harness_ask_unknown")
def test_a_supplied_labeller_names_the_ask():
msgs = [{"role": "user", "content": "q"}, _injected("[finalize] commit now.")]
kinds = {"[finalize]": "finalize"}
def _ask_kind(content: object) -> str | None:
return next((v for k, v in kinds.items() if str(content).startswith(k)), None)
assert stranded_of(msgs, ask_kind=_ask_kind) == (True, "finalize")
assert dead_reasons(messages=msgs, final_content=None, ask_kind=_ask_kind)[0] == "stranded:finalize"
def test_an_injected_assistant_turn_is_not_the_models_own_prose():
"""Every injected entry carries the marker here, not only the user-role ones, so
a harness-written assistant turn cannot pass as the answer the model gave."""
msgs = [{"role": "user", "content": "q"}, {**_asst("the harness wrote this"), _HOOK_INJECTED_KEY: True}]
assert stranded_of(msgs)[0] is True
@pytest.mark.parametrize(
"content,spoke",
[
([{"type": "text", "text": "Alice."}], True),
([{"type": "image", "url": "x"}], False),
("Alice.", True),
(None, False),
(42, True),
],
)
def test_what_counts_as_the_models_own_prose(content, spoke):
"""Channel-separated backends deliver a list of typed blocks, most deliver a
string, and a stub can deliver anything. A turn ended on prose when flattening
its last message leaves text; ``None`` flattens to nothing, and an unexpected
type is stringified rather than silently read as empty -- an instrument that
reports "no answer" for a shape it did not anticipate would invent dead turns.
"""
msgs = [{"role": "user", "content": "q"}, {"role": "assistant", "content": content}]
assert stranded_of(msgs) == ((False, None) if spoke else (True, "empty_assistant"))