Raven-1/tests/test_agent_loop_tool_loop_break.py

936 lines
37 KiB
Python
Raw Permalink Normal View History

2026-06-28 18:17:27 +08:00
"""Tool-failure loop break: nudge the model off a tool it keeps failing on.
When the same tool fails deterministically N times running (transient errors
excluded), the loop appends a change-approach nudge to the tool result — once
per fresh streak, bounded per turn — so a weak model stops repeating a dead call.
"""
from __future__ import annotations
import tempfile
from pathlib import Path
import pytest
from raven.agent.loop import AgentLoop
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
from raven.agent.loop.bundles import ToolWiring, TurnPolicy
from raven.agent.loop.failure_streak import failure_class, is_hard_tool_failure
feat(docker): run raven in a docker container and make a first run configurable from the page ## Summary Two things arrive together because the second is what made the first usable. **Run raven from a container.** A Dockerfile builds the page, the Python environment and the engine into one image; `cd docker && docker compose up` serves it. nginx serves the built page and proxies the RPC socket to the engine over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a second container would have no address to dial. Compose publishes on 127.0.0.1, and the entrypoint prints a one-time sign-in link on boot. **A first run is no longer a dead end.** The gateway used to refuse to start until a provider was configured, while being the thing that serves the page where a provider gets configured. It now starts, says what is missing, and picks up a key saved in Settings on the next turn without a restart. The onboarding overlay no longer takes the page either: New chat, the model chip and Send send the reader to Settings > Models instead, and an empty model list is marked on the control that fills it. Also carried: the provider catalogue registry (capabilities, modalities and context windows per model), the reworked Settings > Models pane, and four fixes found in review of that work -- a sibling catalogue answering 200 with a non-catalogue body no longer crashes the whole fetch; adding a tag no longer erases the label and description already stored for that model; `authenticated` no longer demands a bearer token from a local provider reached by address; and registry rows are keyed by the spelling every lookup produces, which recovers 593 rows that no reader could reach. ## Type - [ ] Fix - [x] Feature - [x] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Commands run on the branch head, with their results: - `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped - `npx vitest run` in `ui-web` -- 1573 passed (101 files) - `npx vitest run` in `ui-tui` -- 2037 passed (142 files) - `uv run ruff check raven scripts tests` and `ruff format --check` -- clean - `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean - `npm run lint:rpc` in `ui-tui` -- generated.ts in sync - `make check-source-language`, `make check-large-files` -- clean Each of the four review fixes was checked by reverting it and watching a named test go red, then restoring it. The container image itself has not been built by CI or by me -- no Docker daemon was available on the machine this was written on. It has been built and run by the author of the branch. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk Two deliberate security trade-offs in the container deployment, both documented in `docker/README.md`: - nginx rewrites the `Origin` header, because the engine accepts only an Origin naming its own loopback port and no browser reaching a published port can send one. This drops a defence-in-depth layer. The session cookie is SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite ignores ports, so another page on localhost counts as same-site. - `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to every page load, so anyone who can reach the published port is signed in. It is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any deployment reachable from elsewhere. These are two settings that must agree, and nothing warns when they do not. Behaviour changes outside the container: the gateway now starts without a configured provider rather than exiting 1, and the first-run onboarding overlay no longer opens by default (`?onboard=1` still forces it). Rollback is by revert; nothing here migrates state or changes an on-disk format. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A
2026-09-11 03:27:52 +00:00
from raven.agent.loop.no_progress import (
NoProgressAction,
NoProgressGuard,
no_progress_key,
no_progress_stop,
)
build: whole-repo lint + pre-commit convention (config only) (#59) ## Change description > Whole-repo lint + pre-commit convention, CONFIG ONLY (no source reformat in this PR). Python (ruff) and TypeScript (eslint + prettier) share a 120-col margin via the committed pre-commit gate. This upgrades main's existing basic ruff setup and adds the TS lint stack (none existed on main). > > What this PR adds: > - ruff: line-length 120; security rules S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests + accepted CLI deferred-imports + product asserts + N818, each commented); ruff pinned <0.16. > - eslint/prettier: shared base (`eslint.base.mjs` + `prettier.config.mjs`); ui-tui extends it with React rules, bridge with a Node-only config; hermes-ink vendor and generated RPC types excluded. > - pre-commit: keeps main's hooks (conventional-commit, detect-private-key, etc.), adds a top-level exclude and 4 local eslint/prettier hooks; bumps the pre-commit dev dep. > - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel runner; lower-cased file-local cron-heartbeat constants (N806). (raven/config/raven.py's F821 was already fixed on main.) > > Sequencing: config only. `ruff check` is intentionally red on ~307 auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them (100% auto-resolvable, zero manual). Enforcement (pre-commit install / CI) turns on only after the reformat lands, so do not expect a green tree from this PR alone. > > CLI-owned files (the reformat and several per-file-ignores touch `raven/cli/*`) are under review by the CLI team. Judgment items (TS no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818 rename, C90, mypy) are deferred to v2. ## Type of change - [ ] Bug fix - [x] New feature - [ ] Document - [x] Others ## Related issues (if there is) > none ## Checklists ### Development - [ ] Lint rules pass locally - [ ] Application changes have been tested thoroughly - [ ] Automated tests covering modified code pass > config-only: `ruff check` is red on auto-fixables by design (the reformat branch clears them); eslint/prettier configs load in both ui-tui and bridge. ### Security - [x] Security impact of change has been considered - [ ] Code follows company security practices and guidelines > Only dependency change is the dev-only pre-commit tool plus the ruff pin; the detect-private-key hook from main is retained. ### Code review - [x] Pull request has a descriptive title and context useful to a reviewer - [ ] Pull request linked to JIRA ticket when applicable --------- Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai> Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
from raven.providers.base import LLMProvider, LLMResponse, ToolCallRequest
2026-06-28 18:17:27 +08:00
from raven.spine.message import ChatType, Source
from raven.spine.turn import Origin, TurnRequest
@pytest.fixture
def workspace():
with tempfile.TemporaryDirectory() as td:
yield Path(td)
# --------------------------------------------------------------------------- #
# unit: is_hard_tool_failure #
2026-06-28 18:17:27 +08:00
# --------------------------------------------------------------------------- #
build: whole-repo lint + pre-commit convention (config only) (#59) ## Change description > Whole-repo lint + pre-commit convention, CONFIG ONLY (no source reformat in this PR). Python (ruff) and TypeScript (eslint + prettier) share a 120-col margin via the committed pre-commit gate. This upgrades main's existing basic ruff setup and adds the TS lint stack (none existed on main). > > What this PR adds: > - ruff: line-length 120; security rules S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests + accepted CLI deferred-imports + product asserts + N818, each commented); ruff pinned <0.16. > - eslint/prettier: shared base (`eslint.base.mjs` + `prettier.config.mjs`); ui-tui extends it with React rules, bridge with a Node-only config; hermes-ink vendor and generated RPC types excluded. > - pre-commit: keeps main's hooks (conventional-commit, detect-private-key, etc.), adds a top-level exclude and 4 local eslint/prettier hooks; bumps the pre-commit dev dep. > - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel runner; lower-cased file-local cron-heartbeat constants (N806). (raven/config/raven.py's F821 was already fixed on main.) > > Sequencing: config only. `ruff check` is intentionally red on ~307 auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them (100% auto-resolvable, zero manual). Enforcement (pre-commit install / CI) turns on only after the reformat lands, so do not expect a green tree from this PR alone. > > CLI-owned files (the reformat and several per-file-ignores touch `raven/cli/*`) are under review by the CLI team. Judgment items (TS no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818 rename, C90, mypy) are deferred to v2. ## Type of change - [ ] Bug fix - [x] New feature - [ ] Document - [x] Others ## Related issues (if there is) > none ## Checklists ### Development - [ ] Lint rules pass locally - [ ] Application changes have been tested thoroughly - [ ] Automated tests covering modified code pass > config-only: `ruff check` is red on auto-fixables by design (the reformat branch clears them); eslint/prettier configs load in both ui-tui and bridge. ### Security - [x] Security impact of change has been considered - [ ] Code follows company security practices and guidelines > Only dependency change is the dev-only pre-commit tool plus the ruff pin; the detect-private-key hook from main is retained. ### Code review - [x] Pull request has a descriptive title and context useful to a reviewer - [ ] Pull request linked to JIRA ticket when applicable --------- Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai> Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
@pytest.mark.parametrize(
"result,expected",
[
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
("Error: Tool 'x' is not available. It may have been unloaded, or the name may be wrong.", True),
build: whole-repo lint + pre-commit convention (config only) (#59) ## Change description > Whole-repo lint + pre-commit convention, CONFIG ONLY (no source reformat in this PR). Python (ruff) and TypeScript (eslint + prettier) share a 120-col margin via the committed pre-commit gate. This upgrades main's existing basic ruff setup and adds the TS lint stack (none existed on main). > > What this PR adds: > - ruff: line-length 120; security rules S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests + accepted CLI deferred-imports + product asserts + N818, each commented); ruff pinned <0.16. > - eslint/prettier: shared base (`eslint.base.mjs` + `prettier.config.mjs`); ui-tui extends it with React rules, bridge with a Node-only config; hermes-ink vendor and generated RPC types excluded. > - pre-commit: keeps main's hooks (conventional-commit, detect-private-key, etc.), adds a top-level exclude and 4 local eslint/prettier hooks; bumps the pre-commit dev dep. > - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel runner; lower-cased file-local cron-heartbeat constants (N806). (raven/config/raven.py's F821 was already fixed on main.) > > Sequencing: config only. `ruff check` is intentionally red on ~307 auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them (100% auto-resolvable, zero manual). Enforcement (pre-commit install / CI) turns on only after the reformat lands, so do not expect a green tree from this PR alone. > > CLI-owned files (the reformat and several per-file-ignores touch `raven/cli/*`) are under review by the CLI team. Judgment items (TS no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818 rename, C90, mypy) are deferred to v2. ## Type of change - [ ] Bug fix - [x] New feature - [ ] Document - [x] Others ## Related issues (if there is) > none ## Checklists ### Development - [ ] Lint rules pass locally - [ ] Application changes have been tested thoroughly - [ ] Automated tests covering modified code pass > config-only: `ruff check` is red on auto-fixables by design (the reformat branch clears them); eslint/prettier configs load in both ui-tui and bridge. ### Security - [x] Security impact of change has been considered - [ ] Code follows company security practices and guidelines > Only dependency change is the dev-only pre-commit tool plus the ruff pin; the detect-private-key hook from main is retained. ### Code review - [x] Pull request has a descriptive title and context useful to a reviewer - [ ] Pull request linked to JIRA ticket when applicable --------- Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai> Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
("Error: file does not exist", True),
("No matches found.", False), # empty search = success, not a failure
("No files found", False), # find empty result
("route not found in cache, using local fallback", False), # success mentioning the phrase
("Exit code: 1\nboom", True),
("Exit code: 0\nok", False), # exit 0 = success
("ok, wrote 3 files", False),
("Error: 429 rate limit, retry later", False), # transient → not hard
("request timed out", False), # transient → not hard
feat(*): dead-end predicates, verbatim tool bodies and envelope failures (#410) ## Summary Absorbs the first of two remaining commits from the research fork before it stopped iterating. This one lands instruments only: nothing here changes what a shipped run does, and the research flow label does not move because no model-visible text changed. `raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended on the model's own prose. The test is structural, with a textual sub-label for the tail shape. Three decisions are worth review: - It lives under `raven/agent/loop/` rather than in the research plugin, because the loop is what would re-run a dead turn, and a host module cannot import a plugin without breaking `make lint-imports`. - The ask labeller is injected (`ask_kind`) rather than imported, because only the product that writes the harness asks knows how they are worded. With no labeller the failure mode is still recognised structurally and reported as `harness_ask_unknown`, so the boolean never depends on wording. - The fork marked its injected turns with a key of its own. This loop already had an equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather than adding a second marker for the same fact. The conditional rerun that reads the predicate arrived on this branch afterwards, in #418, along with the turn's wall clock. Both are supplied as data a hook writes onto the turn's metadata rather than as config the loop reads, so an agent whose hooks leave no budget runs exactly as it did before either existed: the rerun is unreachable for it and the clock is never consulted. The rerun starts from the original question rather than salvaging the failed attempt, because salvage was measured on 25 real duds to produce content 23 times and the right answer none of them, turning a detectable zero into a confident wrong answer. Three defects in that work were found by review and fixed on this branch, each with a reproduction that is red on the commit before it. The loop numbered episodes from an iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0] and the TUI, which keys episode rows and their fold state by that index, had the rerun's first step collide with the first attempt's. Each attempt took its own clock reading, so a first attempt ending just under the limit handed the rerun a fresh full budget and a ten-second turn ran to eighteen. And the research hook opens its per-turn ledger at iteration 1, which a rerun reaches again, so the second open replaced the path the close reads and left the first attempt's file unreachable on disk. Reading that back afterwards turned up three more, none of them raised in review. The turn's end record carried the attempt's iteration count beside the turn's elapsed time with nothing saying which was which, so it gained ``attempt``. The hook contract described that record as "status and iteration count" while it carried five fields, which hid ``stopped_by`` from anyone writing a terminal gate against the contract. And the rerun's progress line named research in a loop that serves every agent, and called itself the first attempt when the budget allows more than one. A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch results at three chokepoints. It is a second file rather than more columns on the ledger because a 250 KB row would leave no complete line in the 64 KB tail window `_resolve_session_seq` reads, which would silently reset the re-run split. Off unless the environment names a file, and a write failure is logged rather than raised. `DigestOutput` lets a digest return a write-only ledger annotation beside its text. The annotation is stripped from the tool result and merged into the fetch row, so a digest can never label its own output where the model can read the label. Nothing emits one yet; this is the pipe a later digest sidecar uses. `is_hard_tool_failure` was blind to the whole reader path. `web_fetch` never returns a bare string, it returns a JSON envelope, so a spent quota, a rejected URL and a failed validation all read as success and reset the streak `web_search` had accumulated. Transient markers are still tested first, so a 429 inside an envelope stays retryable. Forced finalize now books its own salvage spend. The phase fires on every run that produced no answer and burns up to `max_tokens` doing it, and it was counted nowhere. The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is the exemption zone this repo keeps Chinese forms in, and the predicate imports it from there rather than carrying a literal outside the zone. Two defects found in the first review round are fixed on this head, both reproduced before being changed. The verbatim sink borrowed the ledger's `_resolve_session_seq`, which recovers the re-run split by reading the last 64 KB of the file and parsing the last complete line. That works for the ledger because its rows are small; this sink stores the large ones, so the first record wider than the window leaves nothing parseable and the sequence resets to 1, making two runs indistinguishable. Runs are now told apart by a tag minted in memory once per process and never recovered from disk, which removes the read a large row defeats. The argument for keeping this file separate from the ledger was already in its docstring; it applies to the sequence too, and now says so. `is_hard_tool_failure` started counting JSON envelopes while `failure_class` still read them as undifferentiated text. The streak key is `(tool, failure_class)` and the break threshold is two, so a blocked URL followed by a reader's HTTP refusal fired the stop-repeating nudge at a model that had changed both its cause and its approach. Both predicates now share one envelope parse, and an envelope is classified by its own `error` string rather than by a textual ladder that cannot see into one. Coarseness is for model prose, which varies without meaning anything; a tool writes its error from a small fixed vocabulary, so equal strings are the same failure and different ones are different. The parts that vary with the page travel in the envelope's other keys, so they cannot split a streak. The last commit removes an `isinstance` guard in the envelope reader that the diff-coverage report flagged as the only uncovered changed line. JSON has one shape that opens with a brace, so the branch was unreachable; removed rather than covered. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run at this head (95c975b3), on `origin/main` with nothing to rebase onto. ``` uv run --frozen --python 3.12 --all-extras pytest -q tests/ 22132 passed, 43 skipped, 109 warnings in 388.67s ``` Integration and e2e are deselected by default (`-m "not integration and not e2e"`), so the run above does not include them. This change adds no integration surface. ``` make coverage make coverage-ratchet line current=88.66% baseline=87.30% delta=+1.36pp branch current=81.59% baseline=79.43% delta=+2.15pp Coverage ratchet passed with 0.05pp tolerance. make coverage-baseline-check Coverage baseline update is monotonic. COVERAGE_BASE_REF=origin/main make coverage-diff Diff coverage: 100.00% (126/126 executable changed lines) ``` Diff coverage reported 99.21% on the commit before last, with one uncovered changed line: the sentence the rerun says to the reader, which no test drove because none passed an `on_progress` into a retrying turn. The last commit pins it rather than waiving it. ``` make lint-python All checks passed! / 1946 files already formatted make lint-imports Contracts: 10 kept, 0 broken. make lint-deps Success! No dependency issues found. make check-source-language clean make check-large-files clean make check-commits clean ``` The kernel budget gate caught one thing worth naming: documenting the turn-end record in the hook contract put `raven/contracts` at 2808 lines against a ceiling of 2791. The per-field detail moved to the write site in `turn_path`, which is not budgeted, and the paper now names the fields and says where the rest is. `raven/contracts` is at 2788. Each of the three review fixes was proved against the commit it fixes rather than reasoned about: a detached worktree at that commit, the new tests copied in, red there and green here. They reproduce the reviewer's own numbers -- the episode list `[0, 1, 2, 0]`, the eighteen seconds, and the stranded ledger file. The strengthened comparison in `test_the_retry_starts_from_the_question_not_from_the_wreckage` is a tightened guarantee rather than a regression test and passes either way, which is why it is called out here. `agents/raven-research/.env` was moved aside for every suite run: it is gitignored and absent in CI, and two launcher tests that delete a key from the environment otherwise find one in the file. This machine has no search-provider key, so no research brief can actually be run here. The retrieval-path changes are verified by unit tests only, which is why the three chokepoints are pinned by tests that assert both the presence and the absence of a row rather than by an observed run. The same limit applies to the rerun: the clock and the dead-end trigger are driven by a deterministic clock in tests, never by a real turn. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed No user-facing surface changed, so there is nothing to document. The one operator-visible knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and is documented in the module that reads it. ## Risk The `is_hard_tool_failure` change is the only unconditional one, and it reaches every agent rather than just research: any tool returning a JSON envelope with a non-empty top-level `error` now counts toward the tool-failure streak that triggers the change-approach nudge. That is the intended correction, and the direction is conservative because transient markers are still tested first. Reverting it is a single branch. Everything else is inert by default. The body sink writes nothing unless the environment names a file. `DigestOutput` has no emitter in this change, and a digest returning a plain string produces the same envelope it always did, which `test_a_bare_string_digest_still_works` pins. The dead-end predicate now has a caller, and it reaches only an agent whose hooks write a budget onto the turn's metadata. No agent on this branch does: `agents/raven-research` ships `wallClockSeconds` and inherits the rerun from its plugin's class default, and every other agent leaves the key unwritten and is bounded exactly as before. The turn's overrun is one budget plus at most one iteration, because the clock is read between iterations and never mid-generation: cancelling a call in flight would discard a finished generation and leave no answer at all. Rolling the rerun back is the budget key; rolling the clock back is the same. Rollback is the whole commit; nothing here is depended on by anything already on main. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com> Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 15:45:02 +08:00
# A JSON envelope, which is the only thing web_fetch ever returns. Every
# deterministic failure it has arrives this way, and none of them are caught
# by the textual tests: the payload starts with "{" and spells the key
# '"error":' rather than "error:".
('{"error": "Jina API key not configured", "url": "https://a.example"}', True),
fix(agent): keep one cause one streak class in the media envelopes (#502) ## Summary `failure_class` keys the loop's tool-failure streak on an envelope's `error` string alone. That works only while the string comes from the small fixed vocabulary its producers are meant to keep, and it is a convention the module cannot enforce - the docstring added in #410 names exactly this failure mode: one cause counting as many. Every media tool broke it. `_format_http_error` interpolated up to 400 characters of vendor body (which carries a request id) into `error`; the unreadable-reference handlers interpolated the path; the four catch-alls put `str(e)` there, which spells the host. So a dead endpoint produced a class per call, `(tool, failure_class)` never repeated, the streak never reached `_LOOP_BREAK_THRESHOLD`, and the stop-repeating nudge was unreachable for `image_generate`, `text_to_speech` and `video_generate` alike. The URL gate in front of `web_fetch` had it too, in both copies of the tool: most reasons `validate_url_target` composes name the address they refused, so a model walking an internal range got one class per address. #410 fixed the handlers behind that gate and left the gate itself. `error` now carries only what a reader could enumerate. The variable part moves to `detail`, which keeps it in front of the model and inside `is_hard_tool_failure`'s transient-marker scan. Where the split gave the single call a further key, the batch reducer forwards that too: `hint`, which names the proxy a 403 can be routed through, and `url`, which names the reference a guarded fetch refused. A batched picture is told what the single call is told, which is the property the split has to preserve. Distinct causes stay distinct, which is the failure mode this trades against: two HTTP statuses, two exception types and the four vendor refusals each keep their own class, and the tests assert that direction too. Found by sweeping for the shape rather than by reading the code: every `"error":` whose value is not a literal is a candidate, and that grep is the cheapest guard against this recurring. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - `uv run pytest tests/ -q -p no:randomly` -> 23443 passed, 109 skipped - `make lint-python` -> ruff check and format clean - `make lint-types` -> all checks passed - `make lint-imports` -> 10 contracts kept, 0 broken - `make check-source-language`, `make check-large-files` -> clean - Revert-to-red, per fix, restoring the source from a backup afterwards: reverting `media_gen.py` alone turns the four new class tests red and leaves `test_two_different_statuses_stay_two_classes` green; reverting either copy of `web.py` turns only that copy's gate test red and leaves the two #410 streak tests green. For the batch fold, each key is pinned on its own: reverting it to `detail` alone turns both new batch tests red, restoring `hint` alone leaves the reference test red, restoring `url` alone leaves the 403 test red. - After rebase onto 45203a4a6: `uv run pytest tests/test_media_gen_tool.py tests/test_security_web_ssrf.py -q -p no:randomly` -> 100 passed; `make lint-python`, `make lint-types`, `make check-source-language`, `make check-large-files` -> clean. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk Tool output shape changes: a caller reading the reason out of `error` now finds it in `detail`. The readers in this repo are the model itself and the tests updated here; `tests/test_security_web_ssrf.py` is the one behavioural assertion that moved, and it still pins that a private address is refused for being private. Rollback is a plain revert of this commit - no state, no migration, no config. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A. Finishes the envelope work started in #410, which fixed the reader's handlers and left the gate in front of them and every media tool. --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-19 20:23:01 +08:00
('{"error": "URL validation failed", "detail": "blocked host", "url": "https://a.example"}', True),
feat(*): dead-end predicates, verbatim tool bodies and envelope failures (#410) ## Summary Absorbs the first of two remaining commits from the research fork before it stopped iterating. This one lands instruments only: nothing here changes what a shipped run does, and the research flow label does not move because no model-visible text changed. `raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended on the model's own prose. The test is structural, with a textual sub-label for the tail shape. Three decisions are worth review: - It lives under `raven/agent/loop/` rather than in the research plugin, because the loop is what would re-run a dead turn, and a host module cannot import a plugin without breaking `make lint-imports`. - The ask labeller is injected (`ask_kind`) rather than imported, because only the product that writes the harness asks knows how they are worded. With no labeller the failure mode is still recognised structurally and reported as `harness_ask_unknown`, so the boolean never depends on wording. - The fork marked its injected turns with a key of its own. This loop already had an equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather than adding a second marker for the same fact. The conditional rerun that reads the predicate arrived on this branch afterwards, in #418, along with the turn's wall clock. Both are supplied as data a hook writes onto the turn's metadata rather than as config the loop reads, so an agent whose hooks leave no budget runs exactly as it did before either existed: the rerun is unreachable for it and the clock is never consulted. The rerun starts from the original question rather than salvaging the failed attempt, because salvage was measured on 25 real duds to produce content 23 times and the right answer none of them, turning a detectable zero into a confident wrong answer. Three defects in that work were found by review and fixed on this branch, each with a reproduction that is red on the commit before it. The loop numbered episodes from an iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0] and the TUI, which keys episode rows and their fold state by that index, had the rerun's first step collide with the first attempt's. Each attempt took its own clock reading, so a first attempt ending just under the limit handed the rerun a fresh full budget and a ten-second turn ran to eighteen. And the research hook opens its per-turn ledger at iteration 1, which a rerun reaches again, so the second open replaced the path the close reads and left the first attempt's file unreachable on disk. Reading that back afterwards turned up three more, none of them raised in review. The turn's end record carried the attempt's iteration count beside the turn's elapsed time with nothing saying which was which, so it gained ``attempt``. The hook contract described that record as "status and iteration count" while it carried five fields, which hid ``stopped_by`` from anyone writing a terminal gate against the contract. And the rerun's progress line named research in a loop that serves every agent, and called itself the first attempt when the budget allows more than one. A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch results at three chokepoints. It is a second file rather than more columns on the ledger because a 250 KB row would leave no complete line in the 64 KB tail window `_resolve_session_seq` reads, which would silently reset the re-run split. Off unless the environment names a file, and a write failure is logged rather than raised. `DigestOutput` lets a digest return a write-only ledger annotation beside its text. The annotation is stripped from the tool result and merged into the fetch row, so a digest can never label its own output where the model can read the label. Nothing emits one yet; this is the pipe a later digest sidecar uses. `is_hard_tool_failure` was blind to the whole reader path. `web_fetch` never returns a bare string, it returns a JSON envelope, so a spent quota, a rejected URL and a failed validation all read as success and reset the streak `web_search` had accumulated. Transient markers are still tested first, so a 429 inside an envelope stays retryable. Forced finalize now books its own salvage spend. The phase fires on every run that produced no answer and burns up to `max_tokens` doing it, and it was counted nowhere. The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is the exemption zone this repo keeps Chinese forms in, and the predicate imports it from there rather than carrying a literal outside the zone. Two defects found in the first review round are fixed on this head, both reproduced before being changed. The verbatim sink borrowed the ledger's `_resolve_session_seq`, which recovers the re-run split by reading the last 64 KB of the file and parsing the last complete line. That works for the ledger because its rows are small; this sink stores the large ones, so the first record wider than the window leaves nothing parseable and the sequence resets to 1, making two runs indistinguishable. Runs are now told apart by a tag minted in memory once per process and never recovered from disk, which removes the read a large row defeats. The argument for keeping this file separate from the ledger was already in its docstring; it applies to the sequence too, and now says so. `is_hard_tool_failure` started counting JSON envelopes while `failure_class` still read them as undifferentiated text. The streak key is `(tool, failure_class)` and the break threshold is two, so a blocked URL followed by a reader's HTTP refusal fired the stop-repeating nudge at a model that had changed both its cause and its approach. Both predicates now share one envelope parse, and an envelope is classified by its own `error` string rather than by a textual ladder that cannot see into one. Coarseness is for model prose, which varies without meaning anything; a tool writes its error from a small fixed vocabulary, so equal strings are the same failure and different ones are different. The parts that vary with the page travel in the envelope's other keys, so they cannot split a streak. The last commit removes an `isinstance` guard in the envelope reader that the diff-coverage report flagged as the only uncovered changed line. JSON has one shape that opens with a brace, so the branch was unreachable; removed rather than covered. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run at this head (95c975b3), on `origin/main` with nothing to rebase onto. ``` uv run --frozen --python 3.12 --all-extras pytest -q tests/ 22132 passed, 43 skipped, 109 warnings in 388.67s ``` Integration and e2e are deselected by default (`-m "not integration and not e2e"`), so the run above does not include them. This change adds no integration surface. ``` make coverage make coverage-ratchet line current=88.66% baseline=87.30% delta=+1.36pp branch current=81.59% baseline=79.43% delta=+2.15pp Coverage ratchet passed with 0.05pp tolerance. make coverage-baseline-check Coverage baseline update is monotonic. COVERAGE_BASE_REF=origin/main make coverage-diff Diff coverage: 100.00% (126/126 executable changed lines) ``` Diff coverage reported 99.21% on the commit before last, with one uncovered changed line: the sentence the rerun says to the reader, which no test drove because none passed an `on_progress` into a retrying turn. The last commit pins it rather than waiving it. ``` make lint-python All checks passed! / 1946 files already formatted make lint-imports Contracts: 10 kept, 0 broken. make lint-deps Success! No dependency issues found. make check-source-language clean make check-large-files clean make check-commits clean ``` The kernel budget gate caught one thing worth naming: documenting the turn-end record in the hook contract put `raven/contracts` at 2808 lines against a ceiling of 2791. The per-field detail moved to the write site in `turn_path`, which is not budgeted, and the paper now names the fields and says where the rest is. `raven/contracts` is at 2788. Each of the three review fixes was proved against the commit it fixes rather than reasoned about: a detached worktree at that commit, the new tests copied in, red there and green here. They reproduce the reviewer's own numbers -- the episode list `[0, 1, 2, 0]`, the eighteen seconds, and the stranded ledger file. The strengthened comparison in `test_the_retry_starts_from_the_question_not_from_the_wreckage` is a tightened guarantee rather than a regression test and passes either way, which is why it is called out here. `agents/raven-research/.env` was moved aside for every suite run: it is gitignored and absent in CI, and two launcher tests that delete a key from the environment otherwise find one in the file. This machine has no search-provider key, so no research brief can actually be run here. The retrieval-path changes are verified by unit tests only, which is why the three chokepoints are pinned by tests that assert both the presence and the absence of a row rather than by an observed run. The same limit applies to the rerun: the clock and the dead-end trigger are driven by a deterministic clock in tests, never by a real turn. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed No user-facing surface changed, so there is nothing to document. The one operator-visible knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and is documented in the module that reads it. ## Risk The `is_hard_tool_failure` change is the only unconditional one, and it reaches every agent rather than just research: any tool returning a JSON envelope with a non-empty top-level `error` now counts toward the tool-failure streak that triggers the change-approach nudge. That is the intended correction, and the direction is conservative because transient markers are still tested first. Reverting it is a single branch. Everything else is inert by default. The body sink writes nothing unless the environment names a file. `DigestOutput` has no emitter in this change, and a digest returning a plain string produces the same envelope it always did, which `test_a_bare_string_digest_still_works` pins. The dead-end predicate now has a caller, and it reaches only an agent whose hooks write a budget onto the turn's metadata. No agent on this branch does: `agents/raven-research` ships `wallClockSeconds` and inherits the rerun from its plugin's class default, and every other agent leaves the key unwritten and is bounded exactly as before. The turn's overrun is one budget plus at most one iteration, because the clock is read between iterations and never mid-generation: cancelling a call in flight would discard a finished generation and leave no answer at all. Rolling the rerun back is the budget key; rolling the clock back is the same. Rollback is the whole commit; nothing here is depended on by anything already on main. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com> Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 15:45:02 +08:00
('{"url": "https://a.example", "text": "the page", "length": 9}', False),
# Transient markers are tested before the envelope, so a retryable failure
# stays retryable however it is wrapped.
('{"error": "Jina answered HTTP 429", "url": "https://a.example"}', False),
# An empty error field is not a failure: a tool that reports its outcome in
# this key and had none must not be read as having failed.
('{"error": "", "url": "https://a.example"}', False),
("{not json at all", False),
('{"error": "x"}'[:-1], False), # truncated payload: unparseable, so undecided
build: whole-repo lint + pre-commit convention (config only) (#59) ## Change description > Whole-repo lint + pre-commit convention, CONFIG ONLY (no source reformat in this PR). Python (ruff) and TypeScript (eslint + prettier) share a 120-col margin via the committed pre-commit gate. This upgrades main's existing basic ruff setup and adds the TS lint stack (none existed on main). > > What this PR adds: > - ruff: line-length 120; security rules S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests + accepted CLI deferred-imports + product asserts + N818, each commented); ruff pinned <0.16. > - eslint/prettier: shared base (`eslint.base.mjs` + `prettier.config.mjs`); ui-tui extends it with React rules, bridge with a Node-only config; hermes-ink vendor and generated RPC types excluded. > - pre-commit: keeps main's hooks (conventional-commit, detect-private-key, etc.), adds a top-level exclude and 4 local eslint/prettier hooks; bumps the pre-commit dev dep. > - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel runner; lower-cased file-local cron-heartbeat constants (N806). (raven/config/raven.py's F821 was already fixed on main.) > > Sequencing: config only. `ruff check` is intentionally red on ~307 auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them (100% auto-resolvable, zero manual). Enforcement (pre-commit install / CI) turns on only after the reformat lands, so do not expect a green tree from this PR alone. > > CLI-owned files (the reformat and several per-file-ignores touch `raven/cli/*`) are under review by the CLI team. Judgment items (TS no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818 rename, C90, mypy) are deferred to v2. ## Type of change - [ ] Bug fix - [x] New feature - [ ] Document - [x] Others ## Related issues (if there is) > none ## Checklists ### Development - [ ] Lint rules pass locally - [ ] Application changes have been tested thoroughly - [ ] Automated tests covering modified code pass > config-only: `ruff check` is red on auto-fixables by design (the reformat branch clears them); eslint/prettier configs load in both ui-tui and bridge. ### Security - [x] Security impact of change has been considered - [ ] Code follows company security practices and guidelines > Only dependency change is the dev-only pre-commit tool plus the ruff pin; the detect-private-key hook from main is retained. ### Code review - [x] Pull request has a descriptive title and context useful to a reviewer - [ ] Pull request linked to JIRA ticket when applicable --------- Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai> Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
],
)
2026-06-28 18:17:27 +08:00
def test_is_hard_tool_failure(result, expected):
assert is_hard_tool_failure(result) is expected
2026-06-28 18:17:27 +08:00
feat(*): dead-end predicates, verbatim tool bodies and envelope failures (#410) ## Summary Absorbs the first of two remaining commits from the research fork before it stopped iterating. This one lands instruments only: nothing here changes what a shipped run does, and the research flow label does not move because no model-visible text changed. `raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended on the model's own prose. The test is structural, with a textual sub-label for the tail shape. Three decisions are worth review: - It lives under `raven/agent/loop/` rather than in the research plugin, because the loop is what would re-run a dead turn, and a host module cannot import a plugin without breaking `make lint-imports`. - The ask labeller is injected (`ask_kind`) rather than imported, because only the product that writes the harness asks knows how they are worded. With no labeller the failure mode is still recognised structurally and reported as `harness_ask_unknown`, so the boolean never depends on wording. - The fork marked its injected turns with a key of its own. This loop already had an equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather than adding a second marker for the same fact. The conditional rerun that reads the predicate arrived on this branch afterwards, in #418, along with the turn's wall clock. Both are supplied as data a hook writes onto the turn's metadata rather than as config the loop reads, so an agent whose hooks leave no budget runs exactly as it did before either existed: the rerun is unreachable for it and the clock is never consulted. The rerun starts from the original question rather than salvaging the failed attempt, because salvage was measured on 25 real duds to produce content 23 times and the right answer none of them, turning a detectable zero into a confident wrong answer. Three defects in that work were found by review and fixed on this branch, each with a reproduction that is red on the commit before it. The loop numbered episodes from an iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0] and the TUI, which keys episode rows and their fold state by that index, had the rerun's first step collide with the first attempt's. Each attempt took its own clock reading, so a first attempt ending just under the limit handed the rerun a fresh full budget and a ten-second turn ran to eighteen. And the research hook opens its per-turn ledger at iteration 1, which a rerun reaches again, so the second open replaced the path the close reads and left the first attempt's file unreachable on disk. Reading that back afterwards turned up three more, none of them raised in review. The turn's end record carried the attempt's iteration count beside the turn's elapsed time with nothing saying which was which, so it gained ``attempt``. The hook contract described that record as "status and iteration count" while it carried five fields, which hid ``stopped_by`` from anyone writing a terminal gate against the contract. And the rerun's progress line named research in a loop that serves every agent, and called itself the first attempt when the budget allows more than one. A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch results at three chokepoints. It is a second file rather than more columns on the ledger because a 250 KB row would leave no complete line in the 64 KB tail window `_resolve_session_seq` reads, which would silently reset the re-run split. Off unless the environment names a file, and a write failure is logged rather than raised. `DigestOutput` lets a digest return a write-only ledger annotation beside its text. The annotation is stripped from the tool result and merged into the fetch row, so a digest can never label its own output where the model can read the label. Nothing emits one yet; this is the pipe a later digest sidecar uses. `is_hard_tool_failure` was blind to the whole reader path. `web_fetch` never returns a bare string, it returns a JSON envelope, so a spent quota, a rejected URL and a failed validation all read as success and reset the streak `web_search` had accumulated. Transient markers are still tested first, so a 429 inside an envelope stays retryable. Forced finalize now books its own salvage spend. The phase fires on every run that produced no answer and burns up to `max_tokens` doing it, and it was counted nowhere. The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is the exemption zone this repo keeps Chinese forms in, and the predicate imports it from there rather than carrying a literal outside the zone. Two defects found in the first review round are fixed on this head, both reproduced before being changed. The verbatim sink borrowed the ledger's `_resolve_session_seq`, which recovers the re-run split by reading the last 64 KB of the file and parsing the last complete line. That works for the ledger because its rows are small; this sink stores the large ones, so the first record wider than the window leaves nothing parseable and the sequence resets to 1, making two runs indistinguishable. Runs are now told apart by a tag minted in memory once per process and never recovered from disk, which removes the read a large row defeats. The argument for keeping this file separate from the ledger was already in its docstring; it applies to the sequence too, and now says so. `is_hard_tool_failure` started counting JSON envelopes while `failure_class` still read them as undifferentiated text. The streak key is `(tool, failure_class)` and the break threshold is two, so a blocked URL followed by a reader's HTTP refusal fired the stop-repeating nudge at a model that had changed both its cause and its approach. Both predicates now share one envelope parse, and an envelope is classified by its own `error` string rather than by a textual ladder that cannot see into one. Coarseness is for model prose, which varies without meaning anything; a tool writes its error from a small fixed vocabulary, so equal strings are the same failure and different ones are different. The parts that vary with the page travel in the envelope's other keys, so they cannot split a streak. The last commit removes an `isinstance` guard in the envelope reader that the diff-coverage report flagged as the only uncovered changed line. JSON has one shape that opens with a brace, so the branch was unreachable; removed rather than covered. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run at this head (95c975b3), on `origin/main` with nothing to rebase onto. ``` uv run --frozen --python 3.12 --all-extras pytest -q tests/ 22132 passed, 43 skipped, 109 warnings in 388.67s ``` Integration and e2e are deselected by default (`-m "not integration and not e2e"`), so the run above does not include them. This change adds no integration surface. ``` make coverage make coverage-ratchet line current=88.66% baseline=87.30% delta=+1.36pp branch current=81.59% baseline=79.43% delta=+2.15pp Coverage ratchet passed with 0.05pp tolerance. make coverage-baseline-check Coverage baseline update is monotonic. COVERAGE_BASE_REF=origin/main make coverage-diff Diff coverage: 100.00% (126/126 executable changed lines) ``` Diff coverage reported 99.21% on the commit before last, with one uncovered changed line: the sentence the rerun says to the reader, which no test drove because none passed an `on_progress` into a retrying turn. The last commit pins it rather than waiving it. ``` make lint-python All checks passed! / 1946 files already formatted make lint-imports Contracts: 10 kept, 0 broken. make lint-deps Success! No dependency issues found. make check-source-language clean make check-large-files clean make check-commits clean ``` The kernel budget gate caught one thing worth naming: documenting the turn-end record in the hook contract put `raven/contracts` at 2808 lines against a ceiling of 2791. The per-field detail moved to the write site in `turn_path`, which is not budgeted, and the paper now names the fields and says where the rest is. `raven/contracts` is at 2788. Each of the three review fixes was proved against the commit it fixes rather than reasoned about: a detached worktree at that commit, the new tests copied in, red there and green here. They reproduce the reviewer's own numbers -- the episode list `[0, 1, 2, 0]`, the eighteen seconds, and the stranded ledger file. The strengthened comparison in `test_the_retry_starts_from_the_question_not_from_the_wreckage` is a tightened guarantee rather than a regression test and passes either way, which is why it is called out here. `agents/raven-research/.env` was moved aside for every suite run: it is gitignored and absent in CI, and two launcher tests that delete a key from the environment otherwise find one in the file. This machine has no search-provider key, so no research brief can actually be run here. The retrieval-path changes are verified by unit tests only, which is why the three chokepoints are pinned by tests that assert both the presence and the absence of a row rather than by an observed run. The same limit applies to the rerun: the clock and the dead-end trigger are driven by a deterministic clock in tests, never by a real turn. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed No user-facing surface changed, so there is nothing to document. The one operator-visible knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and is documented in the module that reads it. ## Risk The `is_hard_tool_failure` change is the only unconditional one, and it reaches every agent rather than just research: any tool returning a JSON envelope with a non-empty top-level `error` now counts toward the tool-failure streak that triggers the change-approach nudge. That is the intended correction, and the direction is conservative because transient markers are still tested first. Reverting it is a single branch. Everything else is inert by default. The body sink writes nothing unless the environment names a file. `DigestOutput` has no emitter in this change, and a digest returning a plain string produces the same envelope it always did, which `test_a_bare_string_digest_still_works` pins. The dead-end predicate now has a caller, and it reaches only an agent whose hooks write a budget onto the turn's metadata. No agent on this branch does: `agents/raven-research` ships `wallClockSeconds` and inherits the rerun from its plugin's class default, and every other agent leaves the key unwritten and is bounded exactly as before. The turn's overrun is one budget plus at most one iteration, because the clock is read between iterations and never mid-generation: cancelling a call in flight would discard a finished generation and leave no answer at all. Rolling the rerun back is the budget key; rolling the clock back is the same. Rollback is the whole commit; nothing here is depended on by anything already on main. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com> Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 15:45:02 +08:00
def test_different_envelope_failures_are_different_classes():
"""The streak key is ``(tool, failure_class)``, and the break threshold is two. If
every envelope classified alike, a blocked URL followed by a reader's HTTP refusal
would fire the stop-repeating nudge at a model that had changed both its cause and
its approach -- the case this whole function exists to keep apart."""
fix(agent): keep one cause one streak class in the media envelopes (#502) ## Summary `failure_class` keys the loop's tool-failure streak on an envelope's `error` string alone. That works only while the string comes from the small fixed vocabulary its producers are meant to keep, and it is a convention the module cannot enforce - the docstring added in #410 names exactly this failure mode: one cause counting as many. Every media tool broke it. `_format_http_error` interpolated up to 400 characters of vendor body (which carries a request id) into `error`; the unreadable-reference handlers interpolated the path; the four catch-alls put `str(e)` there, which spells the host. So a dead endpoint produced a class per call, `(tool, failure_class)` never repeated, the streak never reached `_LOOP_BREAK_THRESHOLD`, and the stop-repeating nudge was unreachable for `image_generate`, `text_to_speech` and `video_generate` alike. The URL gate in front of `web_fetch` had it too, in both copies of the tool: most reasons `validate_url_target` composes name the address they refused, so a model walking an internal range got one class per address. #410 fixed the handlers behind that gate and left the gate itself. `error` now carries only what a reader could enumerate. The variable part moves to `detail`, which keeps it in front of the model and inside `is_hard_tool_failure`'s transient-marker scan. Where the split gave the single call a further key, the batch reducer forwards that too: `hint`, which names the proxy a 403 can be routed through, and `url`, which names the reference a guarded fetch refused. A batched picture is told what the single call is told, which is the property the split has to preserve. Distinct causes stay distinct, which is the failure mode this trades against: two HTTP statuses, two exception types and the four vendor refusals each keep their own class, and the tests assert that direction too. Found by sweeping for the shape rather than by reading the code: every `"error":` whose value is not a literal is a candidate, and that grep is the cheapest guard against this recurring. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - `uv run pytest tests/ -q -p no:randomly` -> 23443 passed, 109 skipped - `make lint-python` -> ruff check and format clean - `make lint-types` -> all checks passed - `make lint-imports` -> 10 contracts kept, 0 broken - `make check-source-language`, `make check-large-files` -> clean - Revert-to-red, per fix, restoring the source from a backup afterwards: reverting `media_gen.py` alone turns the four new class tests red and leaves `test_two_different_statuses_stay_two_classes` green; reverting either copy of `web.py` turns only that copy's gate test red and leaves the two #410 streak tests green. For the batch fold, each key is pinned on its own: reverting it to `detail` alone turns both new batch tests red, restoring `hint` alone leaves the reference test red, restoring `url` alone leaves the 403 test red. - After rebase onto 45203a4a6: `uv run pytest tests/test_media_gen_tool.py tests/test_security_web_ssrf.py -q -p no:randomly` -> 100 passed; `make lint-python`, `make lint-types`, `make check-source-language`, `make check-large-files` -> clean. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk Tool output shape changes: a caller reading the reason out of `error` now finds it in `detail`. The readers in this repo are the model itself and the tests updated here; `tests/test_security_web_ssrf.py` is the one behavioural assertion that moved, and it still pins that a private address is refused for being private. Rollback is a plain revert of this commit - no state, no migration, no config. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A. Finishes the envelope work started in #410, which fixed the reader's handlers and left the gate in front of them and every media tool. --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-19 20:23:01 +08:00
blocked = '{"error": "URL validation failed", "detail": "blocked host", "url": "https://a.example"}'
feat(*): dead-end predicates, verbatim tool bodies and envelope failures (#410) ## Summary Absorbs the first of two remaining commits from the research fork before it stopped iterating. This one lands instruments only: nothing here changes what a shipped run does, and the research flow label does not move because no model-visible text changed. `raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended on the model's own prose. The test is structural, with a textual sub-label for the tail shape. Three decisions are worth review: - It lives under `raven/agent/loop/` rather than in the research plugin, because the loop is what would re-run a dead turn, and a host module cannot import a plugin without breaking `make lint-imports`. - The ask labeller is injected (`ask_kind`) rather than imported, because only the product that writes the harness asks knows how they are worded. With no labeller the failure mode is still recognised structurally and reported as `harness_ask_unknown`, so the boolean never depends on wording. - The fork marked its injected turns with a key of its own. This loop already had an equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather than adding a second marker for the same fact. The conditional rerun that reads the predicate arrived on this branch afterwards, in #418, along with the turn's wall clock. Both are supplied as data a hook writes onto the turn's metadata rather than as config the loop reads, so an agent whose hooks leave no budget runs exactly as it did before either existed: the rerun is unreachable for it and the clock is never consulted. The rerun starts from the original question rather than salvaging the failed attempt, because salvage was measured on 25 real duds to produce content 23 times and the right answer none of them, turning a detectable zero into a confident wrong answer. Three defects in that work were found by review and fixed on this branch, each with a reproduction that is red on the commit before it. The loop numbered episodes from an iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0] and the TUI, which keys episode rows and their fold state by that index, had the rerun's first step collide with the first attempt's. Each attempt took its own clock reading, so a first attempt ending just under the limit handed the rerun a fresh full budget and a ten-second turn ran to eighteen. And the research hook opens its per-turn ledger at iteration 1, which a rerun reaches again, so the second open replaced the path the close reads and left the first attempt's file unreachable on disk. Reading that back afterwards turned up three more, none of them raised in review. The turn's end record carried the attempt's iteration count beside the turn's elapsed time with nothing saying which was which, so it gained ``attempt``. The hook contract described that record as "status and iteration count" while it carried five fields, which hid ``stopped_by`` from anyone writing a terminal gate against the contract. And the rerun's progress line named research in a loop that serves every agent, and called itself the first attempt when the budget allows more than one. A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch results at three chokepoints. It is a second file rather than more columns on the ledger because a 250 KB row would leave no complete line in the 64 KB tail window `_resolve_session_seq` reads, which would silently reset the re-run split. Off unless the environment names a file, and a write failure is logged rather than raised. `DigestOutput` lets a digest return a write-only ledger annotation beside its text. The annotation is stripped from the tool result and merged into the fetch row, so a digest can never label its own output where the model can read the label. Nothing emits one yet; this is the pipe a later digest sidecar uses. `is_hard_tool_failure` was blind to the whole reader path. `web_fetch` never returns a bare string, it returns a JSON envelope, so a spent quota, a rejected URL and a failed validation all read as success and reset the streak `web_search` had accumulated. Transient markers are still tested first, so a 429 inside an envelope stays retryable. Forced finalize now books its own salvage spend. The phase fires on every run that produced no answer and burns up to `max_tokens` doing it, and it was counted nowhere. The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is the exemption zone this repo keeps Chinese forms in, and the predicate imports it from there rather than carrying a literal outside the zone. Two defects found in the first review round are fixed on this head, both reproduced before being changed. The verbatim sink borrowed the ledger's `_resolve_session_seq`, which recovers the re-run split by reading the last 64 KB of the file and parsing the last complete line. That works for the ledger because its rows are small; this sink stores the large ones, so the first record wider than the window leaves nothing parseable and the sequence resets to 1, making two runs indistinguishable. Runs are now told apart by a tag minted in memory once per process and never recovered from disk, which removes the read a large row defeats. The argument for keeping this file separate from the ledger was already in its docstring; it applies to the sequence too, and now says so. `is_hard_tool_failure` started counting JSON envelopes while `failure_class` still read them as undifferentiated text. The streak key is `(tool, failure_class)` and the break threshold is two, so a blocked URL followed by a reader's HTTP refusal fired the stop-repeating nudge at a model that had changed both its cause and its approach. Both predicates now share one envelope parse, and an envelope is classified by its own `error` string rather than by a textual ladder that cannot see into one. Coarseness is for model prose, which varies without meaning anything; a tool writes its error from a small fixed vocabulary, so equal strings are the same failure and different ones are different. The parts that vary with the page travel in the envelope's other keys, so they cannot split a streak. The last commit removes an `isinstance` guard in the envelope reader that the diff-coverage report flagged as the only uncovered changed line. JSON has one shape that opens with a brace, so the branch was unreachable; removed rather than covered. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run at this head (95c975b3), on `origin/main` with nothing to rebase onto. ``` uv run --frozen --python 3.12 --all-extras pytest -q tests/ 22132 passed, 43 skipped, 109 warnings in 388.67s ``` Integration and e2e are deselected by default (`-m "not integration and not e2e"`), so the run above does not include them. This change adds no integration surface. ``` make coverage make coverage-ratchet line current=88.66% baseline=87.30% delta=+1.36pp branch current=81.59% baseline=79.43% delta=+2.15pp Coverage ratchet passed with 0.05pp tolerance. make coverage-baseline-check Coverage baseline update is monotonic. COVERAGE_BASE_REF=origin/main make coverage-diff Diff coverage: 100.00% (126/126 executable changed lines) ``` Diff coverage reported 99.21% on the commit before last, with one uncovered changed line: the sentence the rerun says to the reader, which no test drove because none passed an `on_progress` into a retrying turn. The last commit pins it rather than waiving it. ``` make lint-python All checks passed! / 1946 files already formatted make lint-imports Contracts: 10 kept, 0 broken. make lint-deps Success! No dependency issues found. make check-source-language clean make check-large-files clean make check-commits clean ``` The kernel budget gate caught one thing worth naming: documenting the turn-end record in the hook contract put `raven/contracts` at 2808 lines against a ceiling of 2791. The per-field detail moved to the write site in `turn_path`, which is not budgeted, and the paper now names the fields and says where the rest is. `raven/contracts` is at 2788. Each of the three review fixes was proved against the commit it fixes rather than reasoned about: a detached worktree at that commit, the new tests copied in, red there and green here. They reproduce the reviewer's own numbers -- the episode list `[0, 1, 2, 0]`, the eighteen seconds, and the stranded ledger file. The strengthened comparison in `test_the_retry_starts_from_the_question_not_from_the_wreckage` is a tightened guarantee rather than a regression test and passes either way, which is why it is called out here. `agents/raven-research/.env` was moved aside for every suite run: it is gitignored and absent in CI, and two launcher tests that delete a key from the environment otherwise find one in the file. This machine has no search-provider key, so no research brief can actually be run here. The retrieval-path changes are verified by unit tests only, which is why the three chokepoints are pinned by tests that assert both the presence and the absence of a row rather than by an observed run. The same limit applies to the rerun: the clock and the dead-end trigger are driven by a deterministic clock in tests, never by a real turn. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed No user-facing surface changed, so there is nothing to document. The one operator-visible knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and is documented in the module that reads it. ## Risk The `is_hard_tool_failure` change is the only unconditional one, and it reaches every agent rather than just research: any tool returning a JSON envelope with a non-empty top-level `error` now counts toward the tool-failure streak that triggers the change-approach nudge. That is the intended correction, and the direction is conservative because transient markers are still tested first. Reverting it is a single branch. Everything else is inert by default. The body sink writes nothing unless the environment names a file. `DigestOutput` has no emitter in this change, and a digest returning a plain string produces the same envelope it always did, which `test_a_bare_string_digest_still_works` pins. The dead-end predicate now has a caller, and it reaches only an agent whose hooks write a budget onto the turn's metadata. No agent on this branch does: `agents/raven-research` ships `wallClockSeconds` and inherits the rerun from its plugin's class default, and every other agent leaves the key unwritten and is bounded exactly as before. The turn's overrun is one budget plus at most one iteration, because the clock is read between iterations and never mid-generation: cancelling a call in flight would discard a finished generation and leave no answer at all. Rolling the rerun back is the budget key; rolling the clock back is the same. Rollback is the whole commit; nothing here is depended on by anything already on main. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com> Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 15:45:02 +08:00
refused = '{"error": "Jina answered HTTP 403", "url": "https://b.example"}'
keyless = '{"error": "Jina API key not configured. Set it in the config", "url": "https://c.example"}'
classes = {failure_class(blocked), failure_class(refused), failure_class(keyless)}
assert len(classes) == 3
# Not vacuous: all three must still COUNT, which is the other half of the fix.
assert all(is_hard_tool_failure(x) for x in (blocked, refused, keyless))
def test_the_same_envelope_failure_on_two_urls_is_one_class():
"""The other direction, and the one that keeps the nudge reachable: a model
walking a dead reader across different pages is repeating one dead approach. The
URL rides in the envelope's own key, so it cannot split the streak."""
first = '{"error": "Jina answered HTTP 403", "url": "https://a.example"}'
second = '{"error": "Jina answered HTTP 403", "url": "https://zzz.example/deep/path"}'
assert failure_class(first) == failure_class(second)
def test_an_envelope_class_is_read_off_the_error_not_the_payload():
"""A payload whose other keys differ is still the same failure, and a payload with
no error is not a failure at all."""
assert failure_class('{"error": "boom", "url": "https://a.example", "status": 500}') == failure_class(
'{"error": "boom", "url": "https://b.example"}'
)
assert failure_class('{"url": "https://a.example", "text": "the page"}') == "other"
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
@pytest.mark.parametrize(
"result,expected",
[
# The registry's own wording for a name that did not resolve. It says
# "is not available" rather than "not found" because it cannot tell a
# hallucinated name from a tool unloaded mid-turn -- but it is the same
# failure, and the streak must not file it under the catch-all.
("Error: Tool 'x' is not available. It may have been unloaded, or the name may be wrong.", "not_found"),
("Error: tool 'x' is not available. It may have been unloaded, or the name may be wrong.", "not_found"),
("Error: Invalid parameters for tool 'x': missing 'path'", "schema"),
("Error: Tool 'x' timed out after 300s.", "timeout"),
("Error: something else entirely", "other"),
],
)
def test_failure_class(result, expected):
assert failure_class(result) == expected
2026-06-28 18:17:27 +08:00
# --------------------------------------------------------------------------- #
# loop level: repeated same-tool failure -> bounded nudges #
# --------------------------------------------------------------------------- #
build: whole-repo lint + pre-commit convention (config only) (#59) ## Change description > Whole-repo lint + pre-commit convention, CONFIG ONLY (no source reformat in this PR). Python (ruff) and TypeScript (eslint + prettier) share a 120-col margin via the committed pre-commit gate. This upgrades main's existing basic ruff setup and adds the TS lint stack (none existed on main). > > What this PR adds: > - ruff: line-length 120; security rules S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests + accepted CLI deferred-imports + product asserts + N818, each commented); ruff pinned <0.16. > - eslint/prettier: shared base (`eslint.base.mjs` + `prettier.config.mjs`); ui-tui extends it with React rules, bridge with a Node-only config; hermes-ink vendor and generated RPC types excluded. > - pre-commit: keeps main's hooks (conventional-commit, detect-private-key, etc.), adds a top-level exclude and 4 local eslint/prettier hooks; bumps the pre-commit dev dep. > - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel runner; lower-cased file-local cron-heartbeat constants (N806). (raven/config/raven.py's F821 was already fixed on main.) > > Sequencing: config only. `ruff check` is intentionally red on ~307 auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them (100% auto-resolvable, zero manual). Enforcement (pre-commit install / CI) turns on only after the reformat lands, so do not expect a green tree from this PR alone. > > CLI-owned files (the reformat and several per-file-ignores touch `raven/cli/*`) are under review by the CLI team. Judgment items (TS no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818 rename, C90, mypy) are deferred to v2. ## Type of change - [ ] Bug fix - [x] New feature - [ ] Document - [x] Others ## Related issues (if there is) > none ## Checklists ### Development - [ ] Lint rules pass locally - [ ] Application changes have been tested thoroughly - [ ] Automated tests covering modified code pass > config-only: `ruff check` is red on auto-fixables by design (the reformat branch clears them); eslint/prettier configs load in both ui-tui and bridge. ### Security - [x] Security impact of change has been considered - [ ] Code follows company security practices and guidelines > Only dependency change is the dev-only pre-commit tool plus the ruff pin; the detect-private-key hook from main is retained. ### Code review - [x] Pull request has a descriptive title and context useful to a reviewer - [ ] Pull request linked to JIRA ticket when applicable --------- Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai> Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
2026-06-28 18:17:27 +08:00
class _AlwaysFailsSameToolProvider(LLMProvider):
"""Keeps calling one (nonexistent) tool that hard-fails every time."""
def __init__(self):
super().__init__(api_key="test")
self.loop_marker_counts: list[int] = []
build: whole-repo lint + pre-commit convention (config only) (#59) ## Change description > Whole-repo lint + pre-commit convention, CONFIG ONLY (no source reformat in this PR). Python (ruff) and TypeScript (eslint + prettier) share a 120-col margin via the committed pre-commit gate. This upgrades main's existing basic ruff setup and adds the TS lint stack (none existed on main). > > What this PR adds: > - ruff: line-length 120; security rules S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests + accepted CLI deferred-imports + product asserts + N818, each commented); ruff pinned <0.16. > - eslint/prettier: shared base (`eslint.base.mjs` + `prettier.config.mjs`); ui-tui extends it with React rules, bridge with a Node-only config; hermes-ink vendor and generated RPC types excluded. > - pre-commit: keeps main's hooks (conventional-commit, detect-private-key, etc.), adds a top-level exclude and 4 local eslint/prettier hooks; bumps the pre-commit dev dep. > - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel runner; lower-cased file-local cron-heartbeat constants (N806). (raven/config/raven.py's F821 was already fixed on main.) > > Sequencing: config only. `ruff check` is intentionally red on ~307 auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them (100% auto-resolvable, zero manual). Enforcement (pre-commit install / CI) turns on only after the reformat lands, so do not expect a green tree from this PR alone. > > CLI-owned files (the reformat and several per-file-ignores touch `raven/cli/*`) are under review by the CLI team. Judgment items (TS no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818 rename, C90, mypy) are deferred to v2. ## Type of change - [ ] Bug fix - [x] New feature - [ ] Document - [x] Others ## Related issues (if there is) > none ## Checklists ### Development - [ ] Lint rules pass locally - [ ] Application changes have been tested thoroughly - [ ] Automated tests covering modified code pass > config-only: `ruff check` is red on auto-fixables by design (the reformat branch clears them); eslint/prettier configs load in both ui-tui and bridge. ### Security - [x] Security impact of change has been considered - [ ] Code follows company security practices and guidelines > Only dependency change is the dev-only pre-commit tool plus the ruff pin; the detect-private-key hook from main is retained. ### Code review - [x] Pull request has a descriptive title and context useful to a reviewer - [ ] Pull request linked to JIRA ticket when applicable --------- Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai> Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
async def chat(
self,
messages,
tools=None,
model=None,
max_tokens=4096,
temperature=0.7,
reasoning_effort=None,
tool_choice=None,
):
self.loop_marker_counts.append(sum(1 for m in messages if "[loop]" in str(m.get("content", ""))))
2026-06-28 18:17:27 +08:00
if tools is None: # max-iter synthesis call
return LLMResponse(content="done", finish_reason="stop")
return LLMResponse(
content="",
tool_calls=[ToolCallRequest(id=f"c{len(self.loop_marker_counts)}", name="no_such_tool", arguments={})],
finish_reason="tool_calls",
)
def get_default_model(self) -> str:
return "stub"
@pytest.mark.asyncio
async def test_repeated_tool_failure_nudges_bounded(workspace):
provider = _AlwaysFailsSameToolProvider()
agent = AgentLoop(
build: whole-repo lint + pre-commit convention (config only) (#59) ## Change description > Whole-repo lint + pre-commit convention, CONFIG ONLY (no source reformat in this PR). Python (ruff) and TypeScript (eslint + prettier) share a 120-col margin via the committed pre-commit gate. This upgrades main's existing basic ruff setup and adds the TS lint stack (none existed on main). > > What this PR adds: > - ruff: line-length 120; security rules S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests + accepted CLI deferred-imports + product asserts + N818, each commented); ruff pinned <0.16. > - eslint/prettier: shared base (`eslint.base.mjs` + `prettier.config.mjs`); ui-tui extends it with React rules, bridge with a Node-only config; hermes-ink vendor and generated RPC types excluded. > - pre-commit: keeps main's hooks (conventional-commit, detect-private-key, etc.), adds a top-level exclude and 4 local eslint/prettier hooks; bumps the pre-commit dev dep. > - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel runner; lower-cased file-local cron-heartbeat constants (N806). (raven/config/raven.py's F821 was already fixed on main.) > > Sequencing: config only. `ruff check` is intentionally red on ~307 auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them (100% auto-resolvable, zero manual). Enforcement (pre-commit install / CI) turns on only after the reformat lands, so do not expect a green tree from this PR alone. > > CLI-owned files (the reformat and several per-file-ignores touch `raven/cli/*`) are under review by the CLI team. Judgment items (TS no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818 rename, C90, mypy) are deferred to v2. ## Type of change - [ ] Bug fix - [x] New feature - [ ] Document - [x] Others ## Related issues (if there is) > none ## Checklists ### Development - [ ] Lint rules pass locally - [ ] Application changes have been tested thoroughly - [ ] Automated tests covering modified code pass > config-only: `ruff check` is red on auto-fixables by design (the reformat branch clears them); eslint/prettier configs load in both ui-tui and bridge. ### Security - [x] Security impact of change has been considered - [ ] Code follows company security practices and guidelines > Only dependency change is the dev-only pre-commit tool plus the ruff pin; the detect-private-key hook from main is retained. ### Code review - [x] Pull request has a descriptive title and context useful to a reviewer - [ ] Pull request linked to JIRA ticket when applicable --------- Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai> Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
provider=provider,
workspace=workspace,
model="stub",
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
policy=TurnPolicy(max_iterations=6),
tools=ToolWiring(restrict_to_workspace=True),
2026-06-28 18:17:27 +08:00
)
await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="go",
),
session_key="s1",
)
# A nudge fired (>=1 [loop] marker seen) but never exceeded the per-turn cap.
assert max(provider.loop_marker_counts) == AgentLoop._LOOP_BREAK_MAX
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
class _NudgeTextProvider(_AlwaysFailsSameToolProvider):
"""Keeps the nudge text itself, not just a count of markers."""
def __init__(self):
super().__init__()
self.tool_results: list[str] = []
async def chat(self, messages, **kwargs):
self.tool_results += [
str(m.get("content", ""))
for m in messages
if m.get("role") == "tool" and "[loop]" in str(m.get("content", ""))
]
return await super().chat(messages, **kwargs)
def _find_skill_stub():
from raven.contracts.tool import Tool
class _FindSkill(Tool):
@property
def name(self) -> str:
return "find_skill"
@property
def description(self) -> str:
return "stub"
@property
def parameters(self) -> dict:
return {"type": "object", "properties": {}}
async def execute(self, **kwargs) -> str:
return "ran"
return _FindSkill()
async def _nudges_with_find_skill_switched(workspace, off: bool) -> list[str]:
"""Drive a real failing streak and hand back the nudges it produced.
Switched off through the config file rather than through the registry's
source directly, so the path under test is the one the settings page uses.
"""
import json
from pathlib import Path
from raven.config.loader import get_config_path
cfg = get_config_path()
cfg.parent.mkdir(parents=True, exist_ok=True)
cfg.write_text(json.dumps({"tools": {"disabledTools": ["find_skill"] if off else []}}), encoding="utf-8")
provider = _NudgeTextProvider()
agent = AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
policy=TurnPolicy(max_iterations=6),
tools=ToolWiring(restrict_to_workspace=True),
)
agent.tools.register(_find_skill_stub())
assert isinstance(cfg, Path)
await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="go",
),
session_key="s1",
)
return provider.tool_results
@pytest.mark.asyncio
async def test_the_nudge_does_not_name_a_switched_off_find_skill(workspace):
"""The nudge is model-visible text, so a registration check advertised a tool
the operator had switched off -- and `execute` then refuses it as absent.
Registration was a fair proxy for availability until the off switch stopped
unregistering; a withheld tool stays in the registry, which is what makes the
switch reversible.
"""
nudges = await _nudges_with_find_skill_switched(workspace, off=True)
assert nudges, "no nudge fired, so the assertion below would pass vacuously"
assert not any("find_skill" in n for n in nudges)
@pytest.mark.asyncio
async def test_and_does_name_it_when_it_is_on(workspace):
"""The pairing case, so the assertion above cannot pass by never mentioning
the tool at all."""
nudges = await _nudges_with_find_skill_switched(workspace, off=False)
assert nudges
assert any("find_skill" in n for n in nudges)
class _AlwaysTruncatedWriteProvider(LLMProvider):
"""Every turn is cut off inside the same `write_file` call."""
def __init__(self):
super().__init__(api_key="test")
self.nudges: list[str] = []
async def chat(self, messages, tools=None, model=None, **kwargs):
self.nudges.extend(str(m.get("content", "")) for m in messages if "[loop]" in str(m.get("content", "")))
if tools is None: # max-iter synthesis call
return LLMResponse(content="done", finish_reason="stop")
return LLMResponse(
content="",
tool_calls=[ToolCallRequest(id=f"c{len(self.nudges)}", name="write_file", arguments={"path": "snake.py"})],
finish_reason="length",
)
def get_default_model(self) -> str:
return "stub"
@pytest.mark.asyncio
async def test_a_truncation_streak_is_nudged_toward_a_smaller_payload(workspace):
"""The class reaches the nudge from the loop, not only from a direct call.
The streak key is already `(tool, class)`; only `[0]` was being read, so a
per-class text is inert until the call site passes the rest of what it has.
"""
provider = _AlwaysTruncatedWriteProvider()
agent = AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
policy=TurnPolicy(max_iterations=6),
tools=ToolWiring(restrict_to_workspace=True),
)
await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="write a long file",
),
session_key="s1",
)
assert provider.nudges, "two truncations running should have fired the loop break"
assert all("different tool" not in n for n in provider.nudges)
assert all("smaller" in n for n in provider.nudges)
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
# --------------------------------------------------------------------------- #
# no progress: a call that keeps succeeding and keeps answering the same #
# --------------------------------------------------------------------------- #
def test_no_progress_key_separates_two_spellings_of_one_read():
"""Why the count is per key rather than per turn: the run that motivated
this alternated a read with the same read behind a checksum, so the two
spellings key apart and each has to reach the threshold on its own. That
costs twice the calls and still fires -- a consecutive streak would have
stayed at one forever.
"""
plain = no_progress_key("exec", {"command": "cat t.json"}, "keys: [1]")
checksummed = no_progress_key("exec", {"command": "md5sum t.json && cat t.json"}, "keys: [1]")
assert plain != checksummed
assert plain == no_progress_key("exec", {"command": "cat t.json"}, "keys: [1]")
assert plain != no_progress_key("exec", {"command": "cat t.json"}, "keys: [2]"), "a changed answer is progress"
class _IdenticalSucceedingCallProvider(LLMProvider):
"""Calls one tool with one set of arguments, which succeeds the same way every time."""
def __init__(self):
super().__init__(api_key="test")
self.loop_marker_counts: list[int] = []
self.nudges: list[str] = []
async def chat(self, messages, tools=None, **kwargs):
self.loop_marker_counts.append(sum(1 for m in messages if "[loop]" in str(m.get("content", ""))))
self.nudges += [str(m.get("content", "")) for m in messages if "[loop]" in str(m.get("content", ""))]
if tools is None: # max-iter synthesis call
return LLMResponse(content="done", finish_reason="stop")
return LLMResponse(
content="",
tool_calls=[
ToolCallRequest(id=f"c{len(self.loop_marker_counts)}", name="list_dir", arguments={"path": "."})
],
finish_reason="tool_calls",
)
def get_default_model(self) -> str:
return "stub"
@pytest.mark.asyncio
async def test_an_unchanging_successful_call_is_nudged_and_bounded(workspace):
"""The case the failure streak structurally cannot see: every call returns
exit 0, and a success resets that counter to zero.
"""
provider = _IdenticalSucceedingCallProvider()
agent = AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
policy=TurnPolicy(max_iterations=AgentLoop._NO_PROGRESS_THRESHOLD + 4),
tools=ToolWiring(restrict_to_workspace=True),
)
await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="go",
),
session_key="s1",
)
assert max(provider.loop_marker_counts) >= 1, "an unchanging successful call was never nudged"
assert max(provider.loop_marker_counts) <= AgentLoop._NO_PROGRESS_MAX
assert any("same result for the same arguments" in n for n in provider.nudges)
class _ImageReadProvider(LLMProvider):
"""Reads one image over and over, repainting it first when asked to.
The repaint is the point: the file keeps its size, so `read_file` answers with
the same metadata line every time and different pixels every time.
"""
def __init__(self, target, repaint: bool):
super().__init__(api_key="test")
self._target = target
self._repaint = repaint
self.loop_marker_counts: list[int] = []
async def chat(self, messages, tools=None, **kwargs):
from PIL import Image
self.loop_marker_counts.append(sum(1 for m in messages if "[loop]" in str(m.get("content", ""))))
n = len(self.loop_marker_counts)
shade = (n * 23 % 256, 40, 90) if self._repaint else (10, 40, 90)
Image.new("RGB", (320, 240), shade).save(self._target)
if tools is None: # max-iter synthesis call
return LLMResponse(content="done", finish_reason="stop")
return LLMResponse(
content="",
tool_calls=[ToolCallRequest(id=f"c{n}", name="read_file", arguments={"path": "slide.png"})],
finish_reason="tool_calls",
)
def get_default_model(self) -> str:
return "stub"
async def _read_image_turn(workspace, *, repaint: bool, monkeypatch) -> _ImageReadProvider:
pytest.importorskip("PIL")
monkeypatch.setattr(AgentLoop, "_supports_vision", lambda self, m=None: True)
provider = _ImageReadProvider(workspace / "slide.png", repaint)
agent = AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
policy=TurnPolicy(max_iterations=AgentLoop._NO_PROGRESS_THRESHOLD + 4),
tools=ToolWiring(restrict_to_workspace=True),
)
await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="go",
),
session_key="s1",
)
return provider
@pytest.mark.asyncio
async def test_changing_pixels_behind_one_metadata_line_are_progress(workspace, monkeypatch):
"""`model_text` is not the whole result, so it cannot be the whole key.
An image read answers with path, dimensions and a token estimate -- stable
across a repaint at the same size -- while the pixels ride in the routed
blocks. Keyed on the text alone, a render-edit-inspect pass reads "the same
result" eight times and is told to stop looking, which is the workflow this
guard was written to help. The pair is what makes this discriminating: the
same loop over an unchanging image must still fire.
"""
repainted = await _read_image_turn(workspace, repaint=True, monkeypatch=monkeypatch)
assert max(repainted.loop_marker_counts) == 0, "a repainted image counted as no progress"
unchanged = await _read_image_turn(workspace, repaint=False, monkeypatch=monkeypatch)
assert max(unchanged.loop_marker_counts) >= 1, "an unchanging image was never nudged"
feat(docker): run raven in a docker container and make a first run configurable from the page ## Summary Two things arrive together because the second is what made the first usable. **Run raven from a container.** A Dockerfile builds the page, the Python environment and the engine into one image; `cd docker && docker compose up` serves it. nginx serves the built page and proxies the RPC socket to the engine over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a second container would have no address to dial. Compose publishes on 127.0.0.1, and the entrypoint prints a one-time sign-in link on boot. **A first run is no longer a dead end.** The gateway used to refuse to start until a provider was configured, while being the thing that serves the page where a provider gets configured. It now starts, says what is missing, and picks up a key saved in Settings on the next turn without a restart. The onboarding overlay no longer takes the page either: New chat, the model chip and Send send the reader to Settings > Models instead, and an empty model list is marked on the control that fills it. Also carried: the provider catalogue registry (capabilities, modalities and context windows per model), the reworked Settings > Models pane, and four fixes found in review of that work -- a sibling catalogue answering 200 with a non-catalogue body no longer crashes the whole fetch; adding a tag no longer erases the label and description already stored for that model; `authenticated` no longer demands a bearer token from a local provider reached by address; and registry rows are keyed by the spelling every lookup produces, which recovers 593 rows that no reader could reach. ## Type - [ ] Fix - [x] Feature - [x] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Commands run on the branch head, with their results: - `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped - `npx vitest run` in `ui-web` -- 1573 passed (101 files) - `npx vitest run` in `ui-tui` -- 2037 passed (142 files) - `uv run ruff check raven scripts tests` and `ruff format --check` -- clean - `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean - `npm run lint:rpc` in `ui-tui` -- generated.ts in sync - `make check-source-language`, `make check-large-files` -- clean Each of the four review fixes was checked by reverting it and watching a named test go red, then restoring it. The container image itself has not been built by CI or by me -- no Docker daemon was available on the machine this was written on. It has been built and run by the author of the branch. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk Two deliberate security trade-offs in the container deployment, both documented in `docker/README.md`: - nginx rewrites the `Origin` header, because the engine accepts only an Origin naming its own loopback port and no browser reaching a published port can send one. This drops a defence-in-depth layer. The session cookie is SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite ignores ports, so another page on localhost counts as same-site. - `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to every page load, so anyone who can reach the published port is signed in. It is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any deployment reachable from elsewhere. These are two settings that must agree, and nothing warns when they do not. Behaviour changes outside the container: the gateway now starts without a configured provider rather than exiting 1, and the first-run onboarding overlay no longer opens by default (`?onboard=1` still forces it). Rollback is by revert; nothing here migrates state or changes an on-disk format. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A
2026-09-11 03:27:52 +00:00
# --------------------------------------------------------------------------- #
# no progress: the ladder past the nudge #
# --------------------------------------------------------------------------- #
def _guard() -> NoProgressGuard:
return NoProgressGuard(
nudge_at=AgentLoop._NO_PROGRESS_THRESHOLD,
refuse_at=AgentLoop._NO_PROGRESS_REFUSE,
max_nudges=AgentLoop._NO_PROGRESS_MAX,
max_refusals=AgentLoop._NO_PROGRESS_REFUSALS_MAX,
)
def _walk(guard: NoProgressGuard, tool: str, args: dict, result: str, rounds: int) -> list[str]:
"""One call made `rounds` times, as the loop makes it: check, run, record."""
steps = []
for _ in range(rounds):
action, _text = guard.check(tool, args)
if action is not NoProgressAction.RUN:
steps.append(action.name.lower())
if action is NoProgressAction.END_TURN:
break
continue
guard.record(tool, args, result)
steps.append("run+nudge" if guard.take_nudge() else "run")
return steps
def test_the_ladder_nudges_then_refuses_then_ends_the_turn():
"""Why the nudge alone was not enough: it can only fire once for one answer,
and the model it has to reach may be a fixed point. The measured turn sent
288 byte-identical calls, 280 of them behind the only guard there was.
"""
steps = _walk(_guard(), "exec", {"command": "cat t.json"}, "Exit code: 0\nsame", 40)
assert steps == ["run"] * 7 + ["run+nudge"] + ["run"] * 4 + ["refuse"] * 3 + ["end_turn"]
assert steps.index("end_turn") == 15, "the turn has to end on the 16th identical call"
def test_a_refusal_says_what_happened_and_what_to_do_instead():
"""A bare refusal produces a different loop, so the way out has to be in the
error text -- that is the seat the model has been measured to read."""
guard = _guard()
for _ in range(AgentLoop._NO_PROGRESS_REFUSE):
guard.check("exec", {"command": "cat t.json"})
guard.record("exec", {"command": "cat t.json"}, "Exit code: 0\nsame")
action, text = guard.check("exec", {"command": "cat t.json"})
assert action is NoProgressAction.REFUSE
assert text.startswith("Error"), "the registry reads the leading word as the verdict"
assert "was not run" in text
assert "take the next real step" in text
assert "wait for it in one longer wait" in text, (
"for a poll, waiting explicitly *is* the call being refused, so the way out has to be "
"a wait the model can actually make instead"
)
assert "answered something new" in text
assert "then this turn ends" in text
def test_a_poll_whose_answer_changes_is_left_alone():
"""The shape the result has always been half the key for: identical
arguments, a moving answer. Forty rounds of it must not reach any step."""
guard = _guard()
for i in range(40):
action, _ = guard.check("read_file", {"path": "job.log"})
assert action is NoProgressAction.RUN, f"a moving answer was refused on round {i}"
guard.record("read_file", {"path": "job.log"}, f"lines read: {i}")
assert guard.take_nudge() is None
def test_a_wait_loop_keeps_the_identical_sleep_it_waits_with():
"""The case that made the enforcing step ask a harder question than the
nudge does. A model waiting on a background job sleeps with one identical
call and checks with another whose answer moves, so the sleep's own answer
never changes -- counted the way the nudge counts, it would be refused after
twelve rounds and the wait would die. The nudge still fires, deliberately:
injecting a line of advice into a workflow that is in fact progressing costs
a line of advice, and refusing its calls costs the workflow.
"""
guard = _guard()
nudged = False
for i in range(40):
action, _ = guard.check("exec", {"command": "sleep 5"})
assert action is NoProgressAction.RUN, f"the sleep a wait loop waits with was refused on round {i}"
guard.record("exec", {"command": "sleep 5"}, "Exit code: 0")
nudged = nudged or bool(guard.take_nudge())
guard.record("exec", {"command": "cat progress"}, f"{i}% done")
guard.take_nudge()
assert nudged, "the advisory step is meant to stay lenient, so it should still have fired"
def test_intervening_progress_lets_a_frozen_pair_run_again():
"""The freeze is evidence, not a verdict, and evidence goes stale.
Twelve identical answers establish what the call returns. Then real work
happens -- three writes, each a first sighting -- and what the call returns
is no longer what was established. Reading the freeze off a table instead of
re-asking left the model with three refusals and the end of the turn, and an
error telling it to change the call so that it asks a different question,
which for an exec means editing a command string in order to run the same
command.
"""
guard = _guard()
call = ("exec", {"command": "cat build.log"})
for _ in range(AgentLoop._NO_PROGRESS_REFUSE):
assert guard.check(*call)[0] is NoProgressAction.RUN
guard.record(*call, "Exit code: 0\nno such target")
assert guard.check(*call)[0] is NoProgressAction.REFUSE, "nothing else has answered anything yet"
for i in range(3):
guard.record("write_file", {"path": f"src/{i}.py"}, f"wrote src/{i}.py")
action, text = guard.check(*call)
assert action is NoProgressAction.RUN, "real work happened, so the established answer is stale"
assert text == ""
guard.record(*call, "Exit code: 0\nbuild ok")
assert guard.check(*call)[0] is NoProgressAction.RUN, "and the answer it gave was a new one"
def test_a_rescued_pair_that_goes_back_to_repeating_still_ends_the_turn():
"""The bound a rescue must not cost. One new answer per ``refuse_at``
repeats would otherwise rescue a pair forever -- 12 wasted calls in 13, and
no ladder at all -- so an establishment after the first spends from the same
budget the refusals spend.
"""
guard = _guard()
call = ("exec", {"command": "cat t.json"})
ran = novel = 0
ended = None
for attempt in range(600):
action, _ = guard.check(*call)
if action is NoProgressAction.END_TURN:
ended = attempt + 1
break
if action is NoProgressAction.REFUSE:
guard.record("exec", {"command": f"echo {novel}"}, str(novel))
novel += 1
continue
guard.record(*call, "Exit code: 0\nsame")
guard.take_nudge()
ran += 1
assert ended is not None, "a pair rescued by manufactured novelty has to stay bounded"
assert ran <= AgentLoop._NO_PROGRESS_REFUSE * (AgentLoop._NO_PROGRESS_REFUSALS_MAX + 1)
assert (ran, novel, ended) == (24, 2, 27)
def test_the_stop_says_a_rescued_call_was_run_again_rather_than_only_refused():
"""A model told it was refused three times when it was in fact run again
twice reads the wrong lesson out of the stop."""
guard = _guard()
call = ("exec", {"command": "cat t.json"})
text = ""
for attempt in range(600):
action, text = guard.check(*call)
if action is NoProgressAction.END_TURN:
break
if action is NoProgressAction.REFUSE:
guard.record("exec", {"command": f"echo {attempt}"}, str(attempt))
continue
guard.record(*call, "Exit code: 0\nsame")
assert "refused 2 times" in text
assert "run again 1 time after other work answered something new" in text
assert "run again 1 times" not in text, "one rescue is one time, not one times"
# A guard on nothing at all: at max_refusals 0 the turn ends on the freeze
# itself, and neither half of the budget was drawn on.
assert no_progress_stop("exec", 0, 0).endswith(
"The same call was already established, so it is stopped here rather than repeated to the iteration limit."
)
def test_a_wait_loop_whose_poll_answers_identically_is_stopped_and_says_so():
"""The shape the wait-loop guarantee does not cover, and had no test.
``_novelty`` counts first sightings of a (call, answer) key, so a poll
answering ``status: RUNNING`` byte-identically contributes none after the
first: neither key resets, both freeze, and the turn ends on a wait that was
working. Pinned rather than fixed, because there is nothing to count -- see
the test below for why -- and pinned with the numbers so the cost of the
honest answer stays visible: twelve polls, three chances, then the turn ends
with a wrap-up naming what it was waiting for.
"""
guard = _guard()
sleep = ("exec", {"command": "sleep 5"})
poll = ("exec", {"command": "cat status"})
polls = 0
ended = None
for round_no in range(1, 200):
for pair, answer in ((sleep, "Exit code: 0"), (poll, "status: RUNNING")):
action, _ = guard.check(*pair)
if action is NoProgressAction.END_TURN:
ended = round_no
break
if action is NoProgressAction.RUN:
guard.record(*pair, answer)
guard.take_nudge()
polls += pair is poll
if ended:
break
assert (polls, ended) == (12, 16)
def test_a_stable_wait_loop_and_a_stuck_read_pair_are_one_transcript():
"""Why the shape above is pinned instead of fixed.
A fixed-interval wait with a stable poll and the two spellings of one stuck
read hand this class the same thing: two pairs, each answering its own
constant, alternating, nothing else. Any rule that keeps the first running
keeps the second running too, and the second is the run this guard was
written for -- an hour spent reading one file 127 times. The distinguishing
fact, that one poll's answer depends on something outside the turn that will
change it, is not in the transcript at all.
"""
def verdicts(first, second):
guard = _guard()
out = []
for _ in range(20):
for tool, arguments, answer in (first, second):
action, _text = guard.check(tool, arguments)
out.append(action.name)
if action is NoProgressAction.END_TURN:
return out
if action is NoProgressAction.RUN:
guard.record(tool, arguments, answer)
guard.take_nudge()
return out
wait_loop = verdicts(
("exec", {"command": "sleep 5"}, "Exit code: 0"),
("exec", {"command": "cat status"}, "status: RUNNING"),
)
stuck_read = verdicts(
("exec", {"command": "cat t.json"}, "Exit code: 0\nsame"),
("exec", {"command": "md5sum t.json && cat t.json"}, "Exit code: 0\nsame md5"),
)
assert wait_loop == stuck_read, "no content rule can separate these, so neither can this guard"
assert wait_loop[-1] == "END_TURN"
def test_a_longer_wait_lets_the_poll_it_waits_with_run_again():
"""What the wait loop gets instead of a rule that cannot exist.
The refusal tells the model to wait for it in one longer wait. A wait of a
different length is a first sighting, which is exactly the evidence the
poll's freeze rests on being absent -- so the poll runs again, three times
over, and a backing-off wait outlives its own refusal.
"""
guard = _guard()
poll = ("exec", {"command": "cat status"})
seconds = 5
polls = 0
ended = None
for round_no in range(1, 200):
action, _ = guard.check("exec", {"command": f"sleep {seconds}"})
if action is NoProgressAction.END_TURN:
ended = round_no
break
if action is NoProgressAction.RUN:
guard.record("exec", {"command": f"sleep {seconds}"}, "Exit code: 0")
guard.take_nudge()
else:
seconds *= 2
continue
action, _ = guard.check(*poll)
if action is NoProgressAction.END_TURN:
ended = round_no
break
if action is NoProgressAction.RUN:
guard.record(*poll, "status: RUNNING")
guard.take_nudge()
polls += 1
else:
seconds *= 2
assert polls == 36, "three chances at twelve polls each, against twelve when the wait never changes"
assert ended == 40
def test_the_measured_identical_call_loop_is_stopped_where_it_always_was():
"""The 207-call turn this ladder exists for, run past the fix.
Nothing else answers anything new in it, so there is no evidence for the
freeze to go stale against and the pair never runs again: twelve calls,
three refusals, the turn ends on the sixteenth attempt -- the same numbers
as before the freeze became re-checkable.
"""
guard = _guard()
call = ("exec", {"command": "python train.py --dry-run"})
ran = refused = 0
ended = None
for attempt in range(207):
action, _ = guard.check(*call)
if action is NoProgressAction.END_TURN:
ended = attempt + 1
break
if action is NoProgressAction.REFUSE:
refused += 1
continue
guard.record(*call, "Exit code: 0\nok")
guard.take_nudge()
ran += 1
assert (ran, refused, ended) == (12, 3, 16)
def test_two_spellings_of_one_stuck_read_each_reach_the_ladder():
"""The alternating shape the per-turn count was written for still lands.
Each spelling keys apart and pays its own way there, and the first rounds
reset while the other spelling is still new, so it costs more calls -- but
it is bounded, which is the whole point.
"""
guard = _guard()
ended = 0
for _ in range(80):
for command in ("cat t.json", "md5sum t.json && cat t.json"):
action, _ = guard.check("exec", {"command": command})
if action is NoProgressAction.END_TURN:
ended += 1
break
if action is NoProgressAction.REFUSE:
continue
guard.record("exec", {"command": command}, "Exit code: 0\nsame")
guard.take_nudge()
if ended:
break
assert ended == 1
class _IdenticalCallPastTheLadderProvider(LLMProvider):
"""Makes one identical succeeding call for as long as it is allowed to."""
def __init__(self):
super().__init__(api_key="test")
self.turns = 0
self.synthesized = False
self.last_messages: list[dict] = []
async def chat(self, messages, tools=None, **kwargs):
self.last_messages = list(messages)
if tools is None: # the early-exit wrap-up
self.synthesized = True
return LLMResponse(content="here is what I got", finish_reason="stop")
self.turns += 1
return LLMResponse(
content="",
tool_calls=[ToolCallRequest(id=f"c{self.turns}", name="list_dir", arguments={"path": "."})],
finish_reason="tool_calls",
)
def get_default_model(self) -> str:
return "stub"
@pytest.mark.asyncio
async def test_a_frozen_call_stops_the_turn_without_reaching_the_iteration_cap(workspace):
"""End to end: the shape that reached iteration 317 of 600 having built
nothing now ends its own turn, the refusals never run the tool, and the user
gets the wrap-up rather than a canned apology.
"""
cap = AgentLoop._NO_PROGRESS_REFUSE + AgentLoop._NO_PROGRESS_REFUSALS_MAX + 8
provider = _IdenticalCallPastTheLadderProvider()
agent = AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
policy=TurnPolicy(max_iterations=cap),
tools=ToolWiring(restrict_to_workspace=True),
)
await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="go",
),
session_key="s1",
)
results = [str(m.get("content", "")) for m in provider.last_messages if m.get("role") == "tool"]
refused = [r for r in results if "was not run" in r]
assert len(refused) == AgentLoop._NO_PROGRESS_REFUSALS_MAX + 1, "three refusals then the one that ends the turn"
assert any("this turn is ending" in r for r in refused)
ran = [r for r in results if "was not run" not in r]
assert len(ran) == AgentLoop._NO_PROGRESS_REFUSE, "a refused call must not run the tool"
assert provider.turns < cap, "the turn should have ended on the ladder, not on the iteration cap"
assert provider.synthesized, "an early exit still owes the user an answer"
class _PollUntilWrittenProvider(LLMProvider):
"""Polls one path with one set of arguments while a job writes to it.
Identical arguments every round and a different answer every round, which is
the case the result has always been half the key for -- run through the loop
rather than the guard, so the wiring is what the assertion is about.
"""
def __init__(self, target, rounds: int):
super().__init__(api_key="test")
self._target = target
self._rounds = rounds
self.polls = 0
self.refusals: list[str] = []
async def chat(self, messages, tools=None, **kwargs):
self.refusals += [str(m.get("content", "")) for m in messages if "was not run" in str(m.get("content", ""))]
if tools is None:
return LLMResponse(content="done", finish_reason="stop")
if self.polls >= self._rounds:
return LLMResponse(content="the job finished", finish_reason="stop")
self.polls += 1
self._target.write_text("\n".join(f"line {i}" for i in range(self.polls)))
return LLMResponse(
content="",
tool_calls=[ToolCallRequest(id=f"c{self.polls}", name="read_file", arguments={"path": "job.log"})],
finish_reason="tool_calls",
)
def get_default_model(self) -> str:
return "stub"
@pytest.mark.asyncio
async def test_a_poll_runs_past_the_ladder_through_the_whole_loop(workspace):
"""More rounds than the ladder has steps, and not one of them is refused."""
rounds = AgentLoop._NO_PROGRESS_REFUSE + AgentLoop._NO_PROGRESS_REFUSALS_MAX + 5
(workspace / "job.log").write_text("")
provider = _PollUntilWrittenProvider(workspace / "job.log", rounds)
agent = AgentLoop(
provider=provider,
workspace=workspace,
model="stub",
policy=TurnPolicy(max_iterations=rounds + 5),
tools=ToolWiring(restrict_to_workspace=True),
)
await agent._process_message(
TurnRequest(
origin=Origin.USER,
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
text="wait for the job",
),
session_key="s1",
)
assert provider.polls == rounds, "the poll was cut short"
assert provider.refusals == []