2026-06-28 18:17:27 +08:00
|
|
|
"""Tool-failure loop break: nudge the model off a tool it keeps failing on.
|
|
|
|
|
|
|
|
|
|
When the same tool fails deterministically N times running (transient errors
|
|
|
|
|
excluded), the loop appends a change-approach nudge to the tool result — once
|
|
|
|
|
per fresh streak, bounded per turn — so a weak model stops repeating a dead call.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
|
|
|
|
|
import tempfile
|
|
|
|
|
from pathlib import Path
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
|
|
from raven.agent.loop import AgentLoop
|
fix(tools): an array sent as its own JSON is still an array
## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
|
|
|
from raven.agent.loop.bundles import ToolWiring, TurnPolicy
|
|
|
|
|
from raven.agent.loop.failure_streak import failure_class, is_hard_tool_failure
|
feat(docker): run raven in a docker container and make a first run configurable from the page
## Summary
Two things arrive together because the second is what made the first usable.
**Run raven from a container.** A Dockerfile builds the page, the Python
environment and the engine into one image; `cd docker && docker compose up`
serves it. nginx serves the built page and proxies the RPC socket to the engine
over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a
second container would have no address to dial. Compose publishes on 127.0.0.1,
and the entrypoint prints a one-time sign-in link on boot.
**A first run is no longer a dead end.** The gateway used to refuse to start
until a provider was configured, while being the thing that serves the page
where a provider gets configured. It now starts, says what is missing, and picks
up a key saved in Settings on the next turn without a restart. The onboarding
overlay no longer takes the page either: New chat, the model chip and Send send
the reader to Settings > Models instead, and an empty model list is marked on
the control that fills it.
Also carried: the provider catalogue registry (capabilities, modalities and
context windows per model), the reworked Settings > Models pane, and four fixes
found in review of that work -- a sibling catalogue answering 200 with a
non-catalogue body no longer crashes the whole fetch; adding a tag no longer
erases the label and description already stored for that model; `authenticated`
no longer demands a bearer token from a local provider reached by address; and
registry rows are keyed by the spelling every lookup produces, which recovers
593 rows that no reader could reach.
## Type
- [ ] Fix
- [x] Feature
- [x] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Commands run on the branch head, with their results:
- `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped
- `npx vitest run` in `ui-web` -- 1573 passed (101 files)
- `npx vitest run` in `ui-tui` -- 2037 passed (142 files)
- `uv run ruff check raven scripts tests` and `ruff format --check` -- clean
- `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean
- `npm run lint:rpc` in `ui-tui` -- generated.ts in sync
- `make check-source-language`, `make check-large-files` -- clean
Each of the four review fixes was checked by reverting it and watching a named
test go red, then restoring it.
The container image itself has not been built by CI or by me -- no Docker daemon
was available on the machine this was written on. It has been built and run by
the author of the branch.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
Two deliberate security trade-offs in the container deployment, both documented
in `docker/README.md`:
- nginx rewrites the `Origin` header, because the engine accepts only an Origin
naming its own loopback port and no browser reaching a published port can send
one. This drops a defence-in-depth layer. The session cookie is
SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite
ignores ports, so another page on localhost counts as same-site.
- `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to
every page load, so anyone who can reach the published port is signed in. It
is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any
deployment reachable from elsewhere. These are two settings that must agree,
and nothing warns when they do not.
Behaviour changes outside the container: the gateway now starts without a
configured provider rather than exiting 1, and the first-run onboarding overlay
no longer opens by default (`?onboard=1` still forces it).
Rollback is by revert; nothing here migrates state or changes an on-disk format.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
2026-09-11 03:27:52 +00:00
|
|
|
from raven.agent.loop.no_progress import (
|
|
|
|
|
NoProgressAction,
|
|
|
|
|
NoProgressGuard,
|
|
|
|
|
no_progress_key,
|
|
|
|
|
no_progress_stop,
|
|
|
|
|
)
|
build: whole-repo lint + pre-commit convention (config only) (#59)
## Change description
> Whole-repo lint + pre-commit convention, CONFIG ONLY (no source
reformat in this PR). Python (ruff) and TypeScript (eslint + prettier)
share a 120-col margin via the committed pre-commit gate. This upgrades
main's existing basic ruff setup and adds the TS lint stack (none
existed on main).
>
> What this PR adds:
> - ruff: line-length 120; security rules
S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests +
accepted CLI deferred-imports + product asserts + N818, each commented);
ruff pinned <0.16.
> - eslint/prettier: shared base (`eslint.base.mjs` +
`prettier.config.mjs`); ui-tui extends it with React rules, bridge with
a Node-only config; hermes-ink vendor and generated RPC types excluded.
> - pre-commit: keeps main's hooks (conventional-commit,
detect-private-key, etc.), adds a top-level exclude and 4 local
eslint/prettier hooks; bumps the pre-commit dev dep.
> - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel
runner; lower-cased file-local cron-heartbeat constants (N806).
(raven/config/raven.py's F821 was already fixed on main.)
>
> Sequencing: config only. `ruff check` is intentionally red on ~307
auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch
runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them
(100% auto-resolvable, zero manual). Enforcement (pre-commit install /
CI) turns on only after the reformat lands, so do not expect a green
tree from this PR alone.
>
> CLI-owned files (the reformat and several per-file-ignores touch
`raven/cli/*`) are under review by the CLI team. Judgment items (TS
no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818
rename, C90, mypy) are deferred to v2.
## Type of change
- [ ] Bug fix
- [x] New feature
- [ ] Document
- [x] Others
## Related issues (if there is)
> none
## Checklists
### Development
- [ ] Lint rules pass locally
- [ ] Application changes have been tested thoroughly
- [ ] Automated tests covering modified code pass
> config-only: `ruff check` is red on auto-fixables by design (the
reformat branch clears them); eslint/prettier configs load in both
ui-tui and bridge.
### Security
- [x] Security impact of change has been considered
- [ ] Code follows company security practices and guidelines
> Only dependency change is the dev-only pre-commit tool plus the ruff
pin; the detect-private-key hook from main is retained.
### Code review
- [x] Pull request has a descriptive title and context useful to a
reviewer
- [ ] Pull request linked to JIRA ticket when applicable
---------
Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai>
Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
|
|
|
from raven.providers.base import LLMProvider, LLMResponse, ToolCallRequest
|
2026-06-28 18:17:27 +08:00
|
|
|
from raven.spine.message import ChatType, Source
|
|
|
|
|
from raven.spine.turn import Origin, TurnRequest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture
|
|
|
|
|
def workspace():
|
|
|
|
|
with tempfile.TemporaryDirectory() as td:
|
|
|
|
|
yield Path(td)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
2026-08-18 17:30:49 +08:00
|
|
|
# unit: is_hard_tool_failure #
|
2026-06-28 18:17:27 +08:00
|
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
build: whole-repo lint + pre-commit convention (config only) (#59)
## Change description
> Whole-repo lint + pre-commit convention, CONFIG ONLY (no source
reformat in this PR). Python (ruff) and TypeScript (eslint + prettier)
share a 120-col margin via the committed pre-commit gate. This upgrades
main's existing basic ruff setup and adds the TS lint stack (none
existed on main).
>
> What this PR adds:
> - ruff: line-length 120; security rules
S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests +
accepted CLI deferred-imports + product asserts + N818, each commented);
ruff pinned <0.16.
> - eslint/prettier: shared base (`eslint.base.mjs` +
`prettier.config.mjs`); ui-tui extends it with React rules, bridge with
a Node-only config; hermes-ink vendor and generated RPC types excluded.
> - pre-commit: keeps main's hooks (conventional-commit,
detect-private-key, etc.), adds a top-level exclude and 4 local
eslint/prettier hooks; bumps the pre-commit dev dep.
> - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel
runner; lower-cased file-local cron-heartbeat constants (N806).
(raven/config/raven.py's F821 was already fixed on main.)
>
> Sequencing: config only. `ruff check` is intentionally red on ~307
auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch
runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them
(100% auto-resolvable, zero manual). Enforcement (pre-commit install /
CI) turns on only after the reformat lands, so do not expect a green
tree from this PR alone.
>
> CLI-owned files (the reformat and several per-file-ignores touch
`raven/cli/*`) are under review by the CLI team. Judgment items (TS
no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818
rename, C90, mypy) are deferred to v2.
## Type of change
- [ ] Bug fix
- [x] New feature
- [ ] Document
- [x] Others
## Related issues (if there is)
> none
## Checklists
### Development
- [ ] Lint rules pass locally
- [ ] Application changes have been tested thoroughly
- [ ] Automated tests covering modified code pass
> config-only: `ruff check` is red on auto-fixables by design (the
reformat branch clears them); eslint/prettier configs load in both
ui-tui and bridge.
### Security
- [x] Security impact of change has been considered
- [ ] Code follows company security practices and guidelines
> Only dependency change is the dev-only pre-commit tool plus the ruff
pin; the detect-private-key hook from main is retained.
### Code review
- [x] Pull request has a descriptive title and context useful to a
reviewer
- [ ] Pull request linked to JIRA ticket when applicable
---------
Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai>
Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"result,expected",
|
|
|
|
|
[
|
fix(tools): an array sent as its own JSON is still an array
## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
|
|
|
("Error: Tool 'x' is not available. It may have been unloaded, or the name may be wrong.", True),
|
build: whole-repo lint + pre-commit convention (config only) (#59)
## Change description
> Whole-repo lint + pre-commit convention, CONFIG ONLY (no source
reformat in this PR). Python (ruff) and TypeScript (eslint + prettier)
share a 120-col margin via the committed pre-commit gate. This upgrades
main's existing basic ruff setup and adds the TS lint stack (none
existed on main).
>
> What this PR adds:
> - ruff: line-length 120; security rules
S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests +
accepted CLI deferred-imports + product asserts + N818, each commented);
ruff pinned <0.16.
> - eslint/prettier: shared base (`eslint.base.mjs` +
`prettier.config.mjs`); ui-tui extends it with React rules, bridge with
a Node-only config; hermes-ink vendor and generated RPC types excluded.
> - pre-commit: keeps main's hooks (conventional-commit,
detect-private-key, etc.), adds a top-level exclude and 4 local
eslint/prettier hooks; bumps the pre-commit dev dep.
> - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel
runner; lower-cased file-local cron-heartbeat constants (N806).
(raven/config/raven.py's F821 was already fixed on main.)
>
> Sequencing: config only. `ruff check` is intentionally red on ~307
auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch
runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them
(100% auto-resolvable, zero manual). Enforcement (pre-commit install /
CI) turns on only after the reformat lands, so do not expect a green
tree from this PR alone.
>
> CLI-owned files (the reformat and several per-file-ignores touch
`raven/cli/*`) are under review by the CLI team. Judgment items (TS
no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818
rename, C90, mypy) are deferred to v2.
## Type of change
- [ ] Bug fix
- [x] New feature
- [ ] Document
- [x] Others
## Related issues (if there is)
> none
## Checklists
### Development
- [ ] Lint rules pass locally
- [ ] Application changes have been tested thoroughly
- [ ] Automated tests covering modified code pass
> config-only: `ruff check` is red on auto-fixables by design (the
reformat branch clears them); eslint/prettier configs load in both
ui-tui and bridge.
### Security
- [x] Security impact of change has been considered
- [ ] Code follows company security practices and guidelines
> Only dependency change is the dev-only pre-commit tool plus the ruff
pin; the detect-private-key hook from main is retained.
### Code review
- [x] Pull request has a descriptive title and context useful to a
reviewer
- [ ] Pull request linked to JIRA ticket when applicable
---------
Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai>
Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
|
|
|
("Error: file does not exist", True),
|
|
|
|
|
("No matches found.", False), # empty search = success, not a failure
|
|
|
|
|
("No files found", False), # find empty result
|
|
|
|
|
("route not found in cache, using local fallback", False), # success mentioning the phrase
|
|
|
|
|
("Exit code: 1\nboom", True),
|
|
|
|
|
("Exit code: 0\nok", False), # exit 0 = success
|
|
|
|
|
("ok, wrote 3 files", False),
|
|
|
|
|
("Error: 429 rate limit, retry later", False), # transient → not hard
|
|
|
|
|
("request timed out", False), # transient → not hard
|
feat(*): dead-end predicates, verbatim tool bodies and envelope failures (#410)
## Summary
Absorbs the first of two remaining commits from the research fork before
it stopped
iterating. This one lands instruments only: nothing here changes what a
shipped run does,
and the research flow label does not move because no model-visible text
changed.
`raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended
on the model's own
prose. The test is structural, with a textual sub-label for the tail
shape. Three decisions
are worth review:
- It lives under `raven/agent/loop/` rather than in the research plugin,
because the loop
is what would re-run a dead turn, and a host module cannot import a
plugin without
breaking `make lint-imports`.
- The ask labeller is injected (`ask_kind`) rather than imported,
because only the product
that writes the harness asks knows how they are worded. With no labeller
the failure mode
is still recognised structurally and reported as `harness_ask_unknown`,
so the boolean
never depends on wording.
- The fork marked its injected turns with a key of its own. This loop
already had an
equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather
than adding a
second marker for the same fact.
The conditional rerun that reads the predicate arrived on this branch
afterwards, in
#418, along with the turn's wall clock. Both are supplied as data a hook
writes onto the
turn's metadata rather than as config the loop reads, so an agent whose
hooks leave no
budget runs exactly as it did before either existed: the rerun is
unreachable for it and
the clock is never consulted. The rerun starts from the original
question rather than
salvaging the failed attempt, because salvage was measured on 25 real
duds to produce
content 23 times and the right answer none of them, turning a detectable
zero into a
confident wrong answer.
Three defects in that work were found by review and fixed on this
branch, each with a
reproduction that is red on the commit before it. The loop numbered
episodes from an
iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0]
and the TUI, which
keys episode rows and their fold state by that index, had the rerun's
first step collide
with the first attempt's. Each attempt took its own clock reading, so a
first attempt
ending just under the limit handed the rerun a fresh full budget and a
ten-second turn ran
to eighteen. And the research hook opens its per-turn ledger at
iteration 1, which a rerun
reaches again, so the second open replaced the path the close reads and
left the first
attempt's file unreachable on disk.
Reading that back afterwards turned up three more, none of them raised
in review. The
turn's end record carried the attempt's iteration count beside the
turn's elapsed time
with nothing saying which was which, so it gained ``attempt``. The hook
contract described
that record as "status and iteration count" while it carried five
fields, which hid
``stopped_by`` from anyone writing a terminal gate against the contract.
And the rerun's
progress line named research in a loop that serves every agent, and
called itself the
first attempt when the budget allows more than one.
A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch
results at three
chokepoints. It is a second file rather than more columns on the ledger
because a 250 KB
row would leave no complete line in the 64 KB tail window
`_resolve_session_seq` reads,
which would silently reset the re-run split. Off unless the environment
names a file, and a
write failure is logged rather than raised.
`DigestOutput` lets a digest return a write-only ledger annotation
beside its text. The
annotation is stripped from the tool result and merged into the fetch
row, so a digest can
never label its own output where the model can read the label. Nothing
emits one yet; this
is the pipe a later digest sidecar uses.
`is_hard_tool_failure` was blind to the whole reader path. `web_fetch`
never returns a bare
string, it returns a JSON envelope, so a spent quota, a rejected URL and
a failed
validation all read as success and reset the streak `web_search` had
accumulated.
Transient markers are still tested first, so a 429 inside an envelope
stays retryable.
Forced finalize now books its own salvage spend. The phase fires on
every run that produced
no answer and burns up to `max_tokens` doing it, and it was counted
nowhere.
The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is
the exemption zone
this repo keeps Chinese forms in, and the predicate imports it from
there rather than
carrying a literal outside the zone.
Two defects found in the first review round are fixed on this head, both
reproduced before
being changed.
The verbatim sink borrowed the ledger's `_resolve_session_seq`, which
recovers the re-run
split by reading the last 64 KB of the file and parsing the last
complete line. That works
for the ledger because its rows are small; this sink stores the large
ones, so the first
record wider than the window leaves nothing parseable and the sequence
resets to 1, making
two runs indistinguishable. Runs are now told apart by a tag minted in
memory once per
process and never recovered from disk, which removes the read a large
row defeats. The
argument for keeping this file separate from the ledger was already in
its docstring; it
applies to the sequence too, and now says so.
`is_hard_tool_failure` started counting JSON envelopes while
`failure_class` still read
them as undifferentiated text. The streak key is `(tool, failure_class)`
and the break
threshold is two, so a blocked URL followed by a reader's HTTP refusal
fired the
stop-repeating nudge at a model that had changed both its cause and its
approach. Both
predicates now share one envelope parse, and an envelope is classified
by its own `error`
string rather than by a textual ladder that cannot see into one.
Coarseness is for model
prose, which varies without meaning anything; a tool writes its error
from a small fixed
vocabulary, so equal strings are the same failure and different ones are
different. The
parts that vary with the page travel in the envelope's other keys, so
they cannot split a
streak.
The last commit removes an `isinstance` guard in the envelope reader
that the diff-coverage
report flagged as the only uncovered changed line. JSON has one shape
that opens with a
brace, so the branch was unreachable; removed rather than covered.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Run at this head (95c975b3), on `origin/main` with nothing to rebase
onto.
```
uv run --frozen --python 3.12 --all-extras pytest -q tests/
22132 passed, 43 skipped, 109 warnings in 388.67s
```
Integration and e2e are deselected by default (`-m "not integration and
not e2e"`), so the
run above does not include them. This change adds no integration
surface.
```
make coverage
make coverage-ratchet
line current=88.66% baseline=87.30% delta=+1.36pp
branch current=81.59% baseline=79.43% delta=+2.15pp
Coverage ratchet passed with 0.05pp tolerance.
make coverage-baseline-check
Coverage baseline update is monotonic.
COVERAGE_BASE_REF=origin/main make coverage-diff
Diff coverage: 100.00% (126/126 executable changed lines)
```
Diff coverage reported 99.21% on the commit before last, with one
uncovered changed line:
the sentence the rerun says to the reader, which no test drove because
none passed an
`on_progress` into a retrying turn. The last commit pins it rather than
waiving it.
```
make lint-python
All checks passed! / 1946 files already formatted
make lint-imports
Contracts: 10 kept, 0 broken.
make lint-deps
Success! No dependency issues found.
make check-source-language
clean
make check-large-files
clean
make check-commits
clean
```
The kernel budget gate caught one thing worth naming: documenting the
turn-end record in
the hook contract put `raven/contracts` at 2808 lines against a ceiling
of 2791. The
per-field detail moved to the write site in `turn_path`, which is not
budgeted, and the
paper now names the fields and says where the rest is. `raven/contracts`
is at 2788.
Each of the three review fixes was proved against the commit it fixes
rather than
reasoned about: a detached worktree at that commit, the new tests copied
in, red there and
green here. They reproduce the reviewer's own numbers -- the episode
list `[0, 1, 2, 0]`,
the eighteen seconds, and the stranded ledger file. The strengthened
comparison in
`test_the_retry_starts_from_the_question_not_from_the_wreckage` is a
tightened guarantee
rather than a regression test and passes either way, which is why it is
called out here.
`agents/raven-research/.env` was moved aside for every suite run: it is
gitignored and
absent in CI, and two launcher tests that delete a key from the
environment otherwise find
one in the file.
This machine has no search-provider key, so no research brief can
actually be run here.
The retrieval-path changes are verified by unit tests only, which is why
the three
chokepoints are pinned by tests that assert both the presence and the
absence of a row
rather than by an observed run. The same limit applies to the rerun: the
clock and the
dead-end trigger are driven by a deterministic clock in tests, never by
a real turn.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
No user-facing surface changed, so there is nothing to document. The one
operator-visible
knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and
is documented in
the module that reads it.
## Risk
The `is_hard_tool_failure` change is the only unconditional one, and it
reaches every agent
rather than just research: any tool returning a JSON envelope with a
non-empty top-level
`error` now counts toward the tool-failure streak that triggers the
change-approach nudge.
That is the intended correction, and the direction is conservative
because transient
markers are still tested first. Reverting it is a single branch.
Everything else is inert by default. The body sink writes nothing unless
the environment
names a file. `DigestOutput` has no emitter in this change, and a digest
returning a plain
string produces the same envelope it always did, which
`test_a_bare_string_digest_still_works` pins.
The dead-end predicate now has a caller, and it reaches only an agent
whose hooks write a
budget onto the turn's metadata. No agent on this branch does:
`agents/raven-research`
ships `wallClockSeconds` and inherits the rerun from its plugin's class
default, and every
other agent leaves the key unwritten and is bounded exactly as before.
The turn's overrun
is one budget plus at most one iteration, because the clock is read
between iterations and
never mid-generation: cancelling a call in flight would discard a
finished generation and
leave no answer at all. Rolling the rerun back is the budget key;
rolling the clock back is
the same.
Rollback is the whole commit; nothing here is depended on by anything
already on main.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 15:45:02 +08:00
|
|
|
# A JSON envelope, which is the only thing web_fetch ever returns. Every
|
|
|
|
|
# deterministic failure it has arrives this way, and none of them are caught
|
|
|
|
|
# by the textual tests: the payload starts with "{" and spells the key
|
|
|
|
|
# '"error":' rather than "error:".
|
|
|
|
|
('{"error": "Jina API key not configured", "url": "https://a.example"}', True),
|
fix(agent): keep one cause one streak class in the media envelopes (#502)
## Summary
`failure_class` keys the loop's tool-failure streak on an envelope's
`error`
string alone. That works only while the string comes from the small
fixed
vocabulary its producers are meant to keep, and it is a convention the
module
cannot enforce - the docstring added in #410 names exactly this failure
mode:
one cause counting as many.
Every media tool broke it. `_format_http_error` interpolated up to 400
characters of vendor body (which carries a request id) into `error`; the
unreadable-reference handlers interpolated the path; the four catch-alls
put
`str(e)` there, which spells the host. So a dead endpoint produced a
class per
call, `(tool, failure_class)` never repeated, the streak never reached
`_LOOP_BREAK_THRESHOLD`, and the stop-repeating nudge was unreachable
for
`image_generate`, `text_to_speech` and `video_generate` alike.
The URL gate in front of `web_fetch` had it too, in both copies of the
tool:
most reasons `validate_url_target` composes name the address they
refused, so
a model walking an internal range got one class per address. #410 fixed
the
handlers behind that gate and left the gate itself.
`error` now carries only what a reader could enumerate. The variable
part moves
to `detail`, which keeps it in front of the model and inside
`is_hard_tool_failure`'s transient-marker scan. Where the split gave the
single
call a further key, the batch reducer forwards that too: `hint`, which
names the
proxy a 403 can be routed through, and `url`, which names the reference
a
guarded fetch refused. A batched picture is told what the single call is
told,
which is the property the split has to preserve. Distinct causes stay
distinct,
which is the failure mode this trades against: two HTTP statuses, two
exception
types and the four vendor refusals each keep their own class, and the
tests
assert that direction too.
Found by sweeping for the shape rather than by reading the code: every
`"error":` whose value is not a literal is a candidate, and that grep is
the
cheapest guard against this recurring.
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
- `uv run pytest tests/ -q -p no:randomly` -> 23443 passed, 109 skipped
- `make lint-python` -> ruff check and format clean
- `make lint-types` -> all checks passed
- `make lint-imports` -> 10 contracts kept, 0 broken
- `make check-source-language`, `make check-large-files` -> clean
- Revert-to-red, per fix, restoring the source from a backup afterwards:
reverting `media_gen.py` alone turns the four new class tests red and
leaves
`test_two_different_statuses_stay_two_classes` green; reverting either
copy
of `web.py` turns only that copy's gate test red and leaves the two #410
streak tests green. For the batch fold, each key is pinned on its own:
reverting it to `detail` alone turns both new batch tests red, restoring
`hint` alone leaves the reference test red, restoring `url` alone leaves
the
403 test red.
- After rebase onto 45203a4a6: `uv run pytest
tests/test_media_gen_tool.py
tests/test_security_web_ssrf.py -q -p no:randomly` -> 100 passed;
`make lint-python`, `make lint-types`, `make check-source-language`,
`make check-large-files` -> clean.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
## Risk
Tool output shape changes: a caller reading the reason out of `error`
now
finds it in `detail`. The readers in this repo are the model itself and
the
tests updated here; `tests/test_security_web_ssrf.py` is the one
behavioural
assertion that moved, and it still pins that a private address is
refused for
being private. Rollback is a plain revert of this commit - no state, no
migration, no config.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A. Finishes the envelope work started in #410, which fixed the
reader's
handlers and left the gate in front of them and every media tool.
---------
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-19 20:23:01 +08:00
|
|
|
('{"error": "URL validation failed", "detail": "blocked host", "url": "https://a.example"}', True),
|
feat(*): dead-end predicates, verbatim tool bodies and envelope failures (#410)
## Summary
Absorbs the first of two remaining commits from the research fork before
it stopped
iterating. This one lands instruments only: nothing here changes what a
shipped run does,
and the research flow label does not move because no model-visible text
changed.
`raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended
on the model's own
prose. The test is structural, with a textual sub-label for the tail
shape. Three decisions
are worth review:
- It lives under `raven/agent/loop/` rather than in the research plugin,
because the loop
is what would re-run a dead turn, and a host module cannot import a
plugin without
breaking `make lint-imports`.
- The ask labeller is injected (`ask_kind`) rather than imported,
because only the product
that writes the harness asks knows how they are worded. With no labeller
the failure mode
is still recognised structurally and reported as `harness_ask_unknown`,
so the boolean
never depends on wording.
- The fork marked its injected turns with a key of its own. This loop
already had an
equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather
than adding a
second marker for the same fact.
The conditional rerun that reads the predicate arrived on this branch
afterwards, in
#418, along with the turn's wall clock. Both are supplied as data a hook
writes onto the
turn's metadata rather than as config the loop reads, so an agent whose
hooks leave no
budget runs exactly as it did before either existed: the rerun is
unreachable for it and
the clock is never consulted. The rerun starts from the original
question rather than
salvaging the failed attempt, because salvage was measured on 25 real
duds to produce
content 23 times and the right answer none of them, turning a detectable
zero into a
confident wrong answer.
Three defects in that work were found by review and fixed on this
branch, each with a
reproduction that is red on the commit before it. The loop numbered
episodes from an
iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0]
and the TUI, which
keys episode rows and their fold state by that index, had the rerun's
first step collide
with the first attempt's. Each attempt took its own clock reading, so a
first attempt
ending just under the limit handed the rerun a fresh full budget and a
ten-second turn ran
to eighteen. And the research hook opens its per-turn ledger at
iteration 1, which a rerun
reaches again, so the second open replaced the path the close reads and
left the first
attempt's file unreachable on disk.
Reading that back afterwards turned up three more, none of them raised
in review. The
turn's end record carried the attempt's iteration count beside the
turn's elapsed time
with nothing saying which was which, so it gained ``attempt``. The hook
contract described
that record as "status and iteration count" while it carried five
fields, which hid
``stopped_by`` from anyone writing a terminal gate against the contract.
And the rerun's
progress line named research in a loop that serves every agent, and
called itself the
first attempt when the budget allows more than one.
A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch
results at three
chokepoints. It is a second file rather than more columns on the ledger
because a 250 KB
row would leave no complete line in the 64 KB tail window
`_resolve_session_seq` reads,
which would silently reset the re-run split. Off unless the environment
names a file, and a
write failure is logged rather than raised.
`DigestOutput` lets a digest return a write-only ledger annotation
beside its text. The
annotation is stripped from the tool result and merged into the fetch
row, so a digest can
never label its own output where the model can read the label. Nothing
emits one yet; this
is the pipe a later digest sidecar uses.
`is_hard_tool_failure` was blind to the whole reader path. `web_fetch`
never returns a bare
string, it returns a JSON envelope, so a spent quota, a rejected URL and
a failed
validation all read as success and reset the streak `web_search` had
accumulated.
Transient markers are still tested first, so a 429 inside an envelope
stays retryable.
Forced finalize now books its own salvage spend. The phase fires on
every run that produced
no answer and burns up to `max_tokens` doing it, and it was counted
nowhere.
The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is
the exemption zone
this repo keeps Chinese forms in, and the predicate imports it from
there rather than
carrying a literal outside the zone.
Two defects found in the first review round are fixed on this head, both
reproduced before
being changed.
The verbatim sink borrowed the ledger's `_resolve_session_seq`, which
recovers the re-run
split by reading the last 64 KB of the file and parsing the last
complete line. That works
for the ledger because its rows are small; this sink stores the large
ones, so the first
record wider than the window leaves nothing parseable and the sequence
resets to 1, making
two runs indistinguishable. Runs are now told apart by a tag minted in
memory once per
process and never recovered from disk, which removes the read a large
row defeats. The
argument for keeping this file separate from the ledger was already in
its docstring; it
applies to the sequence too, and now says so.
`is_hard_tool_failure` started counting JSON envelopes while
`failure_class` still read
them as undifferentiated text. The streak key is `(tool, failure_class)`
and the break
threshold is two, so a blocked URL followed by a reader's HTTP refusal
fired the
stop-repeating nudge at a model that had changed both its cause and its
approach. Both
predicates now share one envelope parse, and an envelope is classified
by its own `error`
string rather than by a textual ladder that cannot see into one.
Coarseness is for model
prose, which varies without meaning anything; a tool writes its error
from a small fixed
vocabulary, so equal strings are the same failure and different ones are
different. The
parts that vary with the page travel in the envelope's other keys, so
they cannot split a
streak.
The last commit removes an `isinstance` guard in the envelope reader
that the diff-coverage
report flagged as the only uncovered changed line. JSON has one shape
that opens with a
brace, so the branch was unreachable; removed rather than covered.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Run at this head (95c975b3), on `origin/main` with nothing to rebase
onto.
```
uv run --frozen --python 3.12 --all-extras pytest -q tests/
22132 passed, 43 skipped, 109 warnings in 388.67s
```
Integration and e2e are deselected by default (`-m "not integration and
not e2e"`), so the
run above does not include them. This change adds no integration
surface.
```
make coverage
make coverage-ratchet
line current=88.66% baseline=87.30% delta=+1.36pp
branch current=81.59% baseline=79.43% delta=+2.15pp
Coverage ratchet passed with 0.05pp tolerance.
make coverage-baseline-check
Coverage baseline update is monotonic.
COVERAGE_BASE_REF=origin/main make coverage-diff
Diff coverage: 100.00% (126/126 executable changed lines)
```
Diff coverage reported 99.21% on the commit before last, with one
uncovered changed line:
the sentence the rerun says to the reader, which no test drove because
none passed an
`on_progress` into a retrying turn. The last commit pins it rather than
waiving it.
```
make lint-python
All checks passed! / 1946 files already formatted
make lint-imports
Contracts: 10 kept, 0 broken.
make lint-deps
Success! No dependency issues found.
make check-source-language
clean
make check-large-files
clean
make check-commits
clean
```
The kernel budget gate caught one thing worth naming: documenting the
turn-end record in
the hook contract put `raven/contracts` at 2808 lines against a ceiling
of 2791. The
per-field detail moved to the write site in `turn_path`, which is not
budgeted, and the
paper now names the fields and says where the rest is. `raven/contracts`
is at 2788.
Each of the three review fixes was proved against the commit it fixes
rather than
reasoned about: a detached worktree at that commit, the new tests copied
in, red there and
green here. They reproduce the reviewer's own numbers -- the episode
list `[0, 1, 2, 0]`,
the eighteen seconds, and the stranded ledger file. The strengthened
comparison in
`test_the_retry_starts_from_the_question_not_from_the_wreckage` is a
tightened guarantee
rather than a regression test and passes either way, which is why it is
called out here.
`agents/raven-research/.env` was moved aside for every suite run: it is
gitignored and
absent in CI, and two launcher tests that delete a key from the
environment otherwise find
one in the file.
This machine has no search-provider key, so no research brief can
actually be run here.
The retrieval-path changes are verified by unit tests only, which is why
the three
chokepoints are pinned by tests that assert both the presence and the
absence of a row
rather than by an observed run. The same limit applies to the rerun: the
clock and the
dead-end trigger are driven by a deterministic clock in tests, never by
a real turn.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
No user-facing surface changed, so there is nothing to document. The one
operator-visible
knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and
is documented in
the module that reads it.
## Risk
The `is_hard_tool_failure` change is the only unconditional one, and it
reaches every agent
rather than just research: any tool returning a JSON envelope with a
non-empty top-level
`error` now counts toward the tool-failure streak that triggers the
change-approach nudge.
That is the intended correction, and the direction is conservative
because transient
markers are still tested first. Reverting it is a single branch.
Everything else is inert by default. The body sink writes nothing unless
the environment
names a file. `DigestOutput` has no emitter in this change, and a digest
returning a plain
string produces the same envelope it always did, which
`test_a_bare_string_digest_still_works` pins.
The dead-end predicate now has a caller, and it reaches only an agent
whose hooks write a
budget onto the turn's metadata. No agent on this branch does:
`agents/raven-research`
ships `wallClockSeconds` and inherits the rerun from its plugin's class
default, and every
other agent leaves the key unwritten and is bounded exactly as before.
The turn's overrun
is one budget plus at most one iteration, because the clock is read
between iterations and
never mid-generation: cancelling a call in flight would discard a
finished generation and
leave no answer at all. Rolling the rerun back is the budget key;
rolling the clock back is
the same.
Rollback is the whole commit; nothing here is depended on by anything
already on main.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 15:45:02 +08:00
|
|
|
('{"url": "https://a.example", "text": "the page", "length": 9}', False),
|
|
|
|
|
# Transient markers are tested before the envelope, so a retryable failure
|
|
|
|
|
# stays retryable however it is wrapped.
|
|
|
|
|
('{"error": "Jina answered HTTP 429", "url": "https://a.example"}', False),
|
|
|
|
|
# An empty error field is not a failure: a tool that reports its outcome in
|
|
|
|
|
# this key and had none must not be read as having failed.
|
|
|
|
|
('{"error": "", "url": "https://a.example"}', False),
|
|
|
|
|
("{not json at all", False),
|
|
|
|
|
('{"error": "x"}'[:-1], False), # truncated payload: unparseable, so undecided
|
build: whole-repo lint + pre-commit convention (config only) (#59)
## Change description
> Whole-repo lint + pre-commit convention, CONFIG ONLY (no source
reformat in this PR). Python (ruff) and TypeScript (eslint + prettier)
share a 120-col margin via the committed pre-commit gate. This upgrades
main's existing basic ruff setup and adds the TS lint stack (none
existed on main).
>
> What this PR adds:
> - ruff: line-length 120; security rules
S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests +
accepted CLI deferred-imports + product asserts + N818, each commented);
ruff pinned <0.16.
> - eslint/prettier: shared base (`eslint.base.mjs` +
`prettier.config.mjs`); ui-tui extends it with React rules, bridge with
a Node-only config; hermes-ink vendor and generated RPC types excluded.
> - pre-commit: keeps main's hooks (conventional-commit,
detect-private-key, etc.), adds a top-level exclude and 4 local
eslint/prettier hooks; bumps the pre-commit dev dep.
> - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel
runner; lower-cased file-local cron-heartbeat constants (N806).
(raven/config/raven.py's F821 was already fixed on main.)
>
> Sequencing: config only. `ruff check` is intentionally red on ~307
auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch
runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them
(100% auto-resolvable, zero manual). Enforcement (pre-commit install /
CI) turns on only after the reformat lands, so do not expect a green
tree from this PR alone.
>
> CLI-owned files (the reformat and several per-file-ignores touch
`raven/cli/*`) are under review by the CLI team. Judgment items (TS
no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818
rename, C90, mypy) are deferred to v2.
## Type of change
- [ ] Bug fix
- [x] New feature
- [ ] Document
- [x] Others
## Related issues (if there is)
> none
## Checklists
### Development
- [ ] Lint rules pass locally
- [ ] Application changes have been tested thoroughly
- [ ] Automated tests covering modified code pass
> config-only: `ruff check` is red on auto-fixables by design (the
reformat branch clears them); eslint/prettier configs load in both
ui-tui and bridge.
### Security
- [x] Security impact of change has been considered
- [ ] Code follows company security practices and guidelines
> Only dependency change is the dev-only pre-commit tool plus the ruff
pin; the detect-private-key hook from main is retained.
### Code review
- [x] Pull request has a descriptive title and context useful to a
reviewer
- [ ] Pull request linked to JIRA ticket when applicable
---------
Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai>
Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
|
|
|
],
|
|
|
|
|
)
|
2026-06-28 18:17:27 +08:00
|
|
|
def test_is_hard_tool_failure(result, expected):
|
2026-08-18 17:30:49 +08:00
|
|
|
assert is_hard_tool_failure(result) is expected
|
2026-06-28 18:17:27 +08:00
|
|
|
|
|
|
|
|
|
feat(*): dead-end predicates, verbatim tool bodies and envelope failures (#410)
## Summary
Absorbs the first of two remaining commits from the research fork before
it stopped
iterating. This one lands instruments only: nothing here changes what a
shipped run does,
and the research flow label does not move because no model-visible text
changed.
`raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended
on the model's own
prose. The test is structural, with a textual sub-label for the tail
shape. Three decisions
are worth review:
- It lives under `raven/agent/loop/` rather than in the research plugin,
because the loop
is what would re-run a dead turn, and a host module cannot import a
plugin without
breaking `make lint-imports`.
- The ask labeller is injected (`ask_kind`) rather than imported,
because only the product
that writes the harness asks knows how they are worded. With no labeller
the failure mode
is still recognised structurally and reported as `harness_ask_unknown`,
so the boolean
never depends on wording.
- The fork marked its injected turns with a key of its own. This loop
already had an
equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather
than adding a
second marker for the same fact.
The conditional rerun that reads the predicate arrived on this branch
afterwards, in
#418, along with the turn's wall clock. Both are supplied as data a hook
writes onto the
turn's metadata rather than as config the loop reads, so an agent whose
hooks leave no
budget runs exactly as it did before either existed: the rerun is
unreachable for it and
the clock is never consulted. The rerun starts from the original
question rather than
salvaging the failed attempt, because salvage was measured on 25 real
duds to produce
content 23 times and the right answer none of them, turning a detectable
zero into a
confident wrong answer.
Three defects in that work were found by review and fixed on this
branch, each with a
reproduction that is red on the commit before it. The loop numbered
episodes from an
iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0]
and the TUI, which
keys episode rows and their fold state by that index, had the rerun's
first step collide
with the first attempt's. Each attempt took its own clock reading, so a
first attempt
ending just under the limit handed the rerun a fresh full budget and a
ten-second turn ran
to eighteen. And the research hook opens its per-turn ledger at
iteration 1, which a rerun
reaches again, so the second open replaced the path the close reads and
left the first
attempt's file unreachable on disk.
Reading that back afterwards turned up three more, none of them raised
in review. The
turn's end record carried the attempt's iteration count beside the
turn's elapsed time
with nothing saying which was which, so it gained ``attempt``. The hook
contract described
that record as "status and iteration count" while it carried five
fields, which hid
``stopped_by`` from anyone writing a terminal gate against the contract.
And the rerun's
progress line named research in a loop that serves every agent, and
called itself the
first attempt when the budget allows more than one.
A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch
results at three
chokepoints. It is a second file rather than more columns on the ledger
because a 250 KB
row would leave no complete line in the 64 KB tail window
`_resolve_session_seq` reads,
which would silently reset the re-run split. Off unless the environment
names a file, and a
write failure is logged rather than raised.
`DigestOutput` lets a digest return a write-only ledger annotation
beside its text. The
annotation is stripped from the tool result and merged into the fetch
row, so a digest can
never label its own output where the model can read the label. Nothing
emits one yet; this
is the pipe a later digest sidecar uses.
`is_hard_tool_failure` was blind to the whole reader path. `web_fetch`
never returns a bare
string, it returns a JSON envelope, so a spent quota, a rejected URL and
a failed
validation all read as success and reset the streak `web_search` had
accumulated.
Transient markers are still tested first, so a 429 inside an envelope
stays retryable.
Forced finalize now books its own salvage spend. The phase fires on
every run that produced
no answer and burns up to `max_tokens` doing it, and it was counted
nowhere.
The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is
the exemption zone
this repo keeps Chinese forms in, and the predicate imports it from
there rather than
carrying a literal outside the zone.
Two defects found in the first review round are fixed on this head, both
reproduced before
being changed.
The verbatim sink borrowed the ledger's `_resolve_session_seq`, which
recovers the re-run
split by reading the last 64 KB of the file and parsing the last
complete line. That works
for the ledger because its rows are small; this sink stores the large
ones, so the first
record wider than the window leaves nothing parseable and the sequence
resets to 1, making
two runs indistinguishable. Runs are now told apart by a tag minted in
memory once per
process and never recovered from disk, which removes the read a large
row defeats. The
argument for keeping this file separate from the ledger was already in
its docstring; it
applies to the sequence too, and now says so.
`is_hard_tool_failure` started counting JSON envelopes while
`failure_class` still read
them as undifferentiated text. The streak key is `(tool, failure_class)`
and the break
threshold is two, so a blocked URL followed by a reader's HTTP refusal
fired the
stop-repeating nudge at a model that had changed both its cause and its
approach. Both
predicates now share one envelope parse, and an envelope is classified
by its own `error`
string rather than by a textual ladder that cannot see into one.
Coarseness is for model
prose, which varies without meaning anything; a tool writes its error
from a small fixed
vocabulary, so equal strings are the same failure and different ones are
different. The
parts that vary with the page travel in the envelope's other keys, so
they cannot split a
streak.
The last commit removes an `isinstance` guard in the envelope reader
that the diff-coverage
report flagged as the only uncovered changed line. JSON has one shape
that opens with a
brace, so the branch was unreachable; removed rather than covered.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Run at this head (95c975b3), on `origin/main` with nothing to rebase
onto.
```
uv run --frozen --python 3.12 --all-extras pytest -q tests/
22132 passed, 43 skipped, 109 warnings in 388.67s
```
Integration and e2e are deselected by default (`-m "not integration and
not e2e"`), so the
run above does not include them. This change adds no integration
surface.
```
make coverage
make coverage-ratchet
line current=88.66% baseline=87.30% delta=+1.36pp
branch current=81.59% baseline=79.43% delta=+2.15pp
Coverage ratchet passed with 0.05pp tolerance.
make coverage-baseline-check
Coverage baseline update is monotonic.
COVERAGE_BASE_REF=origin/main make coverage-diff
Diff coverage: 100.00% (126/126 executable changed lines)
```
Diff coverage reported 99.21% on the commit before last, with one
uncovered changed line:
the sentence the rerun says to the reader, which no test drove because
none passed an
`on_progress` into a retrying turn. The last commit pins it rather than
waiving it.
```
make lint-python
All checks passed! / 1946 files already formatted
make lint-imports
Contracts: 10 kept, 0 broken.
make lint-deps
Success! No dependency issues found.
make check-source-language
clean
make check-large-files
clean
make check-commits
clean
```
The kernel budget gate caught one thing worth naming: documenting the
turn-end record in
the hook contract put `raven/contracts` at 2808 lines against a ceiling
of 2791. The
per-field detail moved to the write site in `turn_path`, which is not
budgeted, and the
paper now names the fields and says where the rest is. `raven/contracts`
is at 2788.
Each of the three review fixes was proved against the commit it fixes
rather than
reasoned about: a detached worktree at that commit, the new tests copied
in, red there and
green here. They reproduce the reviewer's own numbers -- the episode
list `[0, 1, 2, 0]`,
the eighteen seconds, and the stranded ledger file. The strengthened
comparison in
`test_the_retry_starts_from_the_question_not_from_the_wreckage` is a
tightened guarantee
rather than a regression test and passes either way, which is why it is
called out here.
`agents/raven-research/.env` was moved aside for every suite run: it is
gitignored and
absent in CI, and two launcher tests that delete a key from the
environment otherwise find
one in the file.
This machine has no search-provider key, so no research brief can
actually be run here.
The retrieval-path changes are verified by unit tests only, which is why
the three
chokepoints are pinned by tests that assert both the presence and the
absence of a row
rather than by an observed run. The same limit applies to the rerun: the
clock and the
dead-end trigger are driven by a deterministic clock in tests, never by
a real turn.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
No user-facing surface changed, so there is nothing to document. The one
operator-visible
knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and
is documented in
the module that reads it.
## Risk
The `is_hard_tool_failure` change is the only unconditional one, and it
reaches every agent
rather than just research: any tool returning a JSON envelope with a
non-empty top-level
`error` now counts toward the tool-failure streak that triggers the
change-approach nudge.
That is the intended correction, and the direction is conservative
because transient
markers are still tested first. Reverting it is a single branch.
Everything else is inert by default. The body sink writes nothing unless
the environment
names a file. `DigestOutput` has no emitter in this change, and a digest
returning a plain
string produces the same envelope it always did, which
`test_a_bare_string_digest_still_works` pins.
The dead-end predicate now has a caller, and it reaches only an agent
whose hooks write a
budget onto the turn's metadata. No agent on this branch does:
`agents/raven-research`
ships `wallClockSeconds` and inherits the rerun from its plugin's class
default, and every
other agent leaves the key unwritten and is bounded exactly as before.
The turn's overrun
is one budget plus at most one iteration, because the clock is read
between iterations and
never mid-generation: cancelling a call in flight would discard a
finished generation and
leave no answer at all. Rolling the rerun back is the budget key;
rolling the clock back is
the same.
Rollback is the whole commit; nothing here is depended on by anything
already on main.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 15:45:02 +08:00
|
|
|
def test_different_envelope_failures_are_different_classes():
|
|
|
|
|
"""The streak key is ``(tool, failure_class)``, and the break threshold is two. If
|
|
|
|
|
every envelope classified alike, a blocked URL followed by a reader's HTTP refusal
|
|
|
|
|
would fire the stop-repeating nudge at a model that had changed both its cause and
|
|
|
|
|
its approach -- the case this whole function exists to keep apart."""
|
fix(agent): keep one cause one streak class in the media envelopes (#502)
## Summary
`failure_class` keys the loop's tool-failure streak on an envelope's
`error`
string alone. That works only while the string comes from the small
fixed
vocabulary its producers are meant to keep, and it is a convention the
module
cannot enforce - the docstring added in #410 names exactly this failure
mode:
one cause counting as many.
Every media tool broke it. `_format_http_error` interpolated up to 400
characters of vendor body (which carries a request id) into `error`; the
unreadable-reference handlers interpolated the path; the four catch-alls
put
`str(e)` there, which spells the host. So a dead endpoint produced a
class per
call, `(tool, failure_class)` never repeated, the streak never reached
`_LOOP_BREAK_THRESHOLD`, and the stop-repeating nudge was unreachable
for
`image_generate`, `text_to_speech` and `video_generate` alike.
The URL gate in front of `web_fetch` had it too, in both copies of the
tool:
most reasons `validate_url_target` composes name the address they
refused, so
a model walking an internal range got one class per address. #410 fixed
the
handlers behind that gate and left the gate itself.
`error` now carries only what a reader could enumerate. The variable
part moves
to `detail`, which keeps it in front of the model and inside
`is_hard_tool_failure`'s transient-marker scan. Where the split gave the
single
call a further key, the batch reducer forwards that too: `hint`, which
names the
proxy a 403 can be routed through, and `url`, which names the reference
a
guarded fetch refused. A batched picture is told what the single call is
told,
which is the property the split has to preserve. Distinct causes stay
distinct,
which is the failure mode this trades against: two HTTP statuses, two
exception
types and the four vendor refusals each keep their own class, and the
tests
assert that direction too.
Found by sweeping for the shape rather than by reading the code: every
`"error":` whose value is not a literal is a candidate, and that grep is
the
cheapest guard against this recurring.
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
- `uv run pytest tests/ -q -p no:randomly` -> 23443 passed, 109 skipped
- `make lint-python` -> ruff check and format clean
- `make lint-types` -> all checks passed
- `make lint-imports` -> 10 contracts kept, 0 broken
- `make check-source-language`, `make check-large-files` -> clean
- Revert-to-red, per fix, restoring the source from a backup afterwards:
reverting `media_gen.py` alone turns the four new class tests red and
leaves
`test_two_different_statuses_stay_two_classes` green; reverting either
copy
of `web.py` turns only that copy's gate test red and leaves the two #410
streak tests green. For the batch fold, each key is pinned on its own:
reverting it to `detail` alone turns both new batch tests red, restoring
`hint` alone leaves the reference test red, restoring `url` alone leaves
the
403 test red.
- After rebase onto 45203a4a6: `uv run pytest
tests/test_media_gen_tool.py
tests/test_security_web_ssrf.py -q -p no:randomly` -> 100 passed;
`make lint-python`, `make lint-types`, `make check-source-language`,
`make check-large-files` -> clean.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
## Risk
Tool output shape changes: a caller reading the reason out of `error`
now
finds it in `detail`. The readers in this repo are the model itself and
the
tests updated here; `tests/test_security_web_ssrf.py` is the one
behavioural
assertion that moved, and it still pins that a private address is
refused for
being private. Rollback is a plain revert of this commit - no state, no
migration, no config.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A. Finishes the envelope work started in #410, which fixed the
reader's
handlers and left the gate in front of them and every media tool.
---------
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-19 20:23:01 +08:00
|
|
|
blocked = '{"error": "URL validation failed", "detail": "blocked host", "url": "https://a.example"}'
|
feat(*): dead-end predicates, verbatim tool bodies and envelope failures (#410)
## Summary
Absorbs the first of two remaining commits from the research fork before
it stopped
iterating. This one lands instruments only: nothing here changes what a
shipped run does,
and the research flow label does not move because no model-visible text
changed.
`raven/agent/loop/dead_end.py` (new) asks whether a finished turn ended
on the model's own
prose. The test is structural, with a textual sub-label for the tail
shape. Three decisions
are worth review:
- It lives under `raven/agent/loop/` rather than in the research plugin,
because the loop
is what would re-run a dead turn, and a host module cannot import a
plugin without
breaking `make lint-imports`.
- The ask labeller is injected (`ask_kind`) rather than imported,
because only the product
that writes the harness asks knows how they are worded. With no labeller
the failure mode
is still recognised structurally and reported as `harness_ask_unknown`,
so the boolean
never depends on wording.
- The fork marked its injected turns with a key of its own. This loop
already had an
equivalent in `_HOOK_INJECTED_KEY`, so the predicate reads that rather
than adding a
second marker for the same fact.
The conditional rerun that reads the predicate arrived on this branch
afterwards, in
#418, along with the turn's wall clock. Both are supplied as data a hook
writes onto the
turn's metadata rather than as config the loop reads, so an agent whose
hooks leave no
budget runs exactly as it did before either existed: the rerun is
unreachable for it and
the clock is never consulted. The rerun starts from the original
question rather than
salvaging the failed attempt, because salvage was measured on 25 real
duds to produce
content 23 times and the right answer none of them, turning a detectable
zero into a
confident wrong answer.
Three defects in that work were found by review and fixed on this
branch, each with a
reproduction that is red on the commit before it. The loop numbered
episodes from an
iteration count that restarts, so a rerun emitted indices [0, 1, 2, 0]
and the TUI, which
keys episode rows and their fold state by that index, had the rerun's
first step collide
with the first attempt's. Each attempt took its own clock reading, so a
first attempt
ending just under the limit handed the rerun a fresh full budget and a
ten-second turn ran
to eighteen. And the research hook opens its per-turn ledger at
iteration 1, which a rerun
reaches again, so the second open replaced the path the close reads and
left the first
attempt's file unreachable on disk.
Reading that back afterwards turned up three more, none of them raised
in review. The
turn's end record carried the attempt's iteration count beside the
turn's elapsed time
with nothing saying which was which, so it gained ``attempt``. The hook
contract described
that record as "status and iteration count" while it carried five
fields, which hid
``stopped_by`` from anyone writing a terminal gate against the contract.
And the rerun's
progress line named research in a loop that serves every agent, and
called itself the
first attempt when the budget allows more than one.
A verbatim body sink (`RAVEN_VERBATIM_SINK`) records search and fetch
results at three
chokepoints. It is a second file rather than more columns on the ledger
because a 250 KB
row would leave no complete line in the 64 KB tail window
`_resolve_session_seq` reads,
which would silently reset the re-run split. Off unless the environment
names a file, and a
write failure is logged rather than raised.
`DigestOutput` lets a digest return a write-only ledger annotation
beside its text. The
annotation is stripped from the tool result and merged into the fetch
row, so a digest can
never label its own output where the model can read the label. Nothing
emits one yet; this
is the pipe a later digest sidecar uses.
`is_hard_tool_failure` was blind to the whole reader path. `web_fetch`
never returns a bare
string, it returns a JSON envelope, so a spent quota, a rejected URL and
a failed
validation all read as success and reset the streak `web_search` had
accumulated.
Transient markers are still tested first, so a 429 inside an envelope
stays retryable.
Forced finalize now books its own salvage spend. The phase fires on
every run that produced
no answer and burns up to `max_tokens` doing it, and it was counted
nowhere.
The Chinese refusal opener lives in `raven/i18n/zh_lexicon.py`, which is
the exemption zone
this repo keeps Chinese forms in, and the predicate imports it from
there rather than
carrying a literal outside the zone.
Two defects found in the first review round are fixed on this head, both
reproduced before
being changed.
The verbatim sink borrowed the ledger's `_resolve_session_seq`, which
recovers the re-run
split by reading the last 64 KB of the file and parsing the last
complete line. That works
for the ledger because its rows are small; this sink stores the large
ones, so the first
record wider than the window leaves nothing parseable and the sequence
resets to 1, making
two runs indistinguishable. Runs are now told apart by a tag minted in
memory once per
process and never recovered from disk, which removes the read a large
row defeats. The
argument for keeping this file separate from the ledger was already in
its docstring; it
applies to the sequence too, and now says so.
`is_hard_tool_failure` started counting JSON envelopes while
`failure_class` still read
them as undifferentiated text. The streak key is `(tool, failure_class)`
and the break
threshold is two, so a blocked URL followed by a reader's HTTP refusal
fired the
stop-repeating nudge at a model that had changed both its cause and its
approach. Both
predicates now share one envelope parse, and an envelope is classified
by its own `error`
string rather than by a textual ladder that cannot see into one.
Coarseness is for model
prose, which varies without meaning anything; a tool writes its error
from a small fixed
vocabulary, so equal strings are the same failure and different ones are
different. The
parts that vary with the page travel in the envelope's other keys, so
they cannot split a
streak.
The last commit removes an `isinstance` guard in the envelope reader
that the diff-coverage
report flagged as the only uncovered changed line. JSON has one shape
that opens with a
brace, so the branch was unreachable; removed rather than covered.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Run at this head (95c975b3), on `origin/main` with nothing to rebase
onto.
```
uv run --frozen --python 3.12 --all-extras pytest -q tests/
22132 passed, 43 skipped, 109 warnings in 388.67s
```
Integration and e2e are deselected by default (`-m "not integration and
not e2e"`), so the
run above does not include them. This change adds no integration
surface.
```
make coverage
make coverage-ratchet
line current=88.66% baseline=87.30% delta=+1.36pp
branch current=81.59% baseline=79.43% delta=+2.15pp
Coverage ratchet passed with 0.05pp tolerance.
make coverage-baseline-check
Coverage baseline update is monotonic.
COVERAGE_BASE_REF=origin/main make coverage-diff
Diff coverage: 100.00% (126/126 executable changed lines)
```
Diff coverage reported 99.21% on the commit before last, with one
uncovered changed line:
the sentence the rerun says to the reader, which no test drove because
none passed an
`on_progress` into a retrying turn. The last commit pins it rather than
waiving it.
```
make lint-python
All checks passed! / 1946 files already formatted
make lint-imports
Contracts: 10 kept, 0 broken.
make lint-deps
Success! No dependency issues found.
make check-source-language
clean
make check-large-files
clean
make check-commits
clean
```
The kernel budget gate caught one thing worth naming: documenting the
turn-end record in
the hook contract put `raven/contracts` at 2808 lines against a ceiling
of 2791. The
per-field detail moved to the write site in `turn_path`, which is not
budgeted, and the
paper now names the fields and says where the rest is. `raven/contracts`
is at 2788.
Each of the three review fixes was proved against the commit it fixes
rather than
reasoned about: a detached worktree at that commit, the new tests copied
in, red there and
green here. They reproduce the reviewer's own numbers -- the episode
list `[0, 1, 2, 0]`,
the eighteen seconds, and the stranded ledger file. The strengthened
comparison in
`test_the_retry_starts_from_the_question_not_from_the_wreckage` is a
tightened guarantee
rather than a regression test and passes either way, which is why it is
called out here.
`agents/raven-research/.env` was moved aside for every suite run: it is
gitignored and
absent in CI, and two launcher tests that delete a key from the
environment otherwise find
one in the file.
This machine has no search-provider key, so no research brief can
actually be run here.
The retrieval-path changes are verified by unit tests only, which is why
the three
chokepoints are pinned by tests that assert both the presence and the
absence of a row
rather than by an observed run. The same limit applies to the rerun: the
clock and the
dead-end trigger are driven by a deterministic clock in tests, never by
a real turn.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
No user-facing surface changed, so there is nothing to document. The one
operator-visible
knob, `RAVEN_VERBATIM_SINK`, is an instrument that is off unless set and
is documented in
the module that reads it.
## Risk
The `is_hard_tool_failure` change is the only unconditional one, and it
reaches every agent
rather than just research: any tool returning a JSON envelope with a
non-empty top-level
`error` now counts toward the tool-failure streak that triggers the
change-approach nudge.
That is the intended correction, and the direction is conservative
because transient
markers are still tested first. Reverting it is a single branch.
Everything else is inert by default. The body sink writes nothing unless
the environment
names a file. `DigestOutput` has no emitter in this change, and a digest
returning a plain
string produces the same envelope it always did, which
`test_a_bare_string_digest_still_works` pins.
The dead-end predicate now has a caller, and it reaches only an agent
whose hooks write a
budget onto the turn's metadata. No agent on this branch does:
`agents/raven-research`
ships `wallClockSeconds` and inherits the rerun from its plugin's class
default, and every
other agent leaves the key unwritten and is bounded exactly as before.
The turn's overrun
is one budget plus at most one iteration, because the clock is read
between iterations and
never mid-generation: cancelling a call in flight would discard a
finished generation and
leave no answer at all. Rolling the rerun back is the budget key;
rolling the clock back is
the same.
Rollback is the whole commit; nothing here is depended on by anything
already on main.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-18 15:45:02 +08:00
|
|
|
refused = '{"error": "Jina answered HTTP 403", "url": "https://b.example"}'
|
|
|
|
|
keyless = '{"error": "Jina API key not configured. Set it in the config", "url": "https://c.example"}'
|
|
|
|
|
|
|
|
|
|
classes = {failure_class(blocked), failure_class(refused), failure_class(keyless)}
|
|
|
|
|
|
|
|
|
|
assert len(classes) == 3
|
|
|
|
|
# Not vacuous: all three must still COUNT, which is the other half of the fix.
|
|
|
|
|
assert all(is_hard_tool_failure(x) for x in (blocked, refused, keyless))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_the_same_envelope_failure_on_two_urls_is_one_class():
|
|
|
|
|
"""The other direction, and the one that keeps the nudge reachable: a model
|
|
|
|
|
walking a dead reader across different pages is repeating one dead approach. The
|
|
|
|
|
URL rides in the envelope's own key, so it cannot split the streak."""
|
|
|
|
|
first = '{"error": "Jina answered HTTP 403", "url": "https://a.example"}'
|
|
|
|
|
second = '{"error": "Jina answered HTTP 403", "url": "https://zzz.example/deep/path"}'
|
|
|
|
|
|
|
|
|
|
assert failure_class(first) == failure_class(second)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_an_envelope_class_is_read_off_the_error_not_the_payload():
|
|
|
|
|
"""A payload whose other keys differ is still the same failure, and a payload with
|
|
|
|
|
no error is not a failure at all."""
|
|
|
|
|
assert failure_class('{"error": "boom", "url": "https://a.example", "status": 500}') == failure_class(
|
|
|
|
|
'{"error": "boom", "url": "https://b.example"}'
|
|
|
|
|
)
|
|
|
|
|
assert failure_class('{"url": "https://a.example", "text": "the page"}') == "other"
|
|
|
|
|
|
|
|
|
|
|
fix(tools): an array sent as its own JSON is still an array
## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"result,expected",
|
|
|
|
|
[
|
|
|
|
|
# The registry's own wording for a name that did not resolve. It says
|
|
|
|
|
# "is not available" rather than "not found" because it cannot tell a
|
|
|
|
|
# hallucinated name from a tool unloaded mid-turn -- but it is the same
|
|
|
|
|
# failure, and the streak must not file it under the catch-all.
|
|
|
|
|
("Error: Tool 'x' is not available. It may have been unloaded, or the name may be wrong.", "not_found"),
|
|
|
|
|
("Error: tool 'x' is not available. It may have been unloaded, or the name may be wrong.", "not_found"),
|
|
|
|
|
("Error: Invalid parameters for tool 'x': missing 'path'", "schema"),
|
|
|
|
|
("Error: Tool 'x' timed out after 300s.", "timeout"),
|
|
|
|
|
("Error: something else entirely", "other"),
|
|
|
|
|
],
|
|
|
|
|
)
|
|
|
|
|
def test_failure_class(result, expected):
|
|
|
|
|
assert failure_class(result) == expected
|
|
|
|
|
|
|
|
|
|
|
2026-06-28 18:17:27 +08:00
|
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
# loop level: repeated same-tool failure -> bounded nudges #
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
build: whole-repo lint + pre-commit convention (config only) (#59)
## Change description
> Whole-repo lint + pre-commit convention, CONFIG ONLY (no source
reformat in this PR). Python (ruff) and TypeScript (eslint + prettier)
share a 120-col margin via the committed pre-commit gate. This upgrades
main's existing basic ruff setup and adds the TS lint stack (none
existed on main).
>
> What this PR adds:
> - ruff: line-length 120; security rules
S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests +
accepted CLI deferred-imports + product asserts + N818, each commented);
ruff pinned <0.16.
> - eslint/prettier: shared base (`eslint.base.mjs` +
`prettier.config.mjs`); ui-tui extends it with React rules, bridge with
a Node-only config; hermes-ink vendor and generated RPC types excluded.
> - pre-commit: keeps main's hooks (conventional-commit,
detect-private-key, etc.), adds a top-level exclude and 4 local
eslint/prettier hooks; bumps the pre-commit dev dep.
> - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel
runner; lower-cased file-local cron-heartbeat constants (N806).
(raven/config/raven.py's F821 was already fixed on main.)
>
> Sequencing: config only. `ruff check` is intentionally red on ~307
auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch
runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them
(100% auto-resolvable, zero manual). Enforcement (pre-commit install /
CI) turns on only after the reformat lands, so do not expect a green
tree from this PR alone.
>
> CLI-owned files (the reformat and several per-file-ignores touch
`raven/cli/*`) are under review by the CLI team. Judgment items (TS
no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818
rename, C90, mypy) are deferred to v2.
## Type of change
- [ ] Bug fix
- [x] New feature
- [ ] Document
- [x] Others
## Related issues (if there is)
> none
## Checklists
### Development
- [ ] Lint rules pass locally
- [ ] Application changes have been tested thoroughly
- [ ] Automated tests covering modified code pass
> config-only: `ruff check` is red on auto-fixables by design (the
reformat branch clears them); eslint/prettier configs load in both
ui-tui and bridge.
### Security
- [x] Security impact of change has been considered
- [ ] Code follows company security practices and guidelines
> Only dependency change is the dev-only pre-commit tool plus the ruff
pin; the detect-private-key hook from main is retained.
### Code review
- [x] Pull request has a descriptive title and context useful to a
reviewer
- [ ] Pull request linked to JIRA ticket when applicable
---------
Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai>
Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
|
|
|
|
2026-06-28 18:17:27 +08:00
|
|
|
class _AlwaysFailsSameToolProvider(LLMProvider):
|
|
|
|
|
"""Keeps calling one (nonexistent) tool that hard-fails every time."""
|
|
|
|
|
|
|
|
|
|
def __init__(self):
|
|
|
|
|
super().__init__(api_key="test")
|
|
|
|
|
self.loop_marker_counts: list[int] = []
|
|
|
|
|
|
build: whole-repo lint + pre-commit convention (config only) (#59)
## Change description
> Whole-repo lint + pre-commit convention, CONFIG ONLY (no source
reformat in this PR). Python (ruff) and TypeScript (eslint + prettier)
share a 120-col margin via the committed pre-commit gate. This upgrades
main's existing basic ruff setup and adds the TS lint stack (none
existed on main).
>
> What this PR adds:
> - ruff: line-length 120; security rules
S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests +
accepted CLI deferred-imports + product asserts + N818, each commented);
ruff pinned <0.16.
> - eslint/prettier: shared base (`eslint.base.mjs` +
`prettier.config.mjs`); ui-tui extends it with React rules, bridge with
a Node-only config; hermes-ink vendor and generated RPC types excluded.
> - pre-commit: keeps main's hooks (conventional-commit,
detect-private-key, etc.), adds a top-level exclude and 4 local
eslint/prettier hooks; bumps the pre-commit dev dep.
> - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel
runner; lower-cased file-local cron-heartbeat constants (N806).
(raven/config/raven.py's F821 was already fixed on main.)
>
> Sequencing: config only. `ruff check` is intentionally red on ~307
auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch
runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them
(100% auto-resolvable, zero manual). Enforcement (pre-commit install /
CI) turns on only after the reformat lands, so do not expect a green
tree from this PR alone.
>
> CLI-owned files (the reformat and several per-file-ignores touch
`raven/cli/*`) are under review by the CLI team. Judgment items (TS
no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818
rename, C90, mypy) are deferred to v2.
## Type of change
- [ ] Bug fix
- [x] New feature
- [ ] Document
- [x] Others
## Related issues (if there is)
> none
## Checklists
### Development
- [ ] Lint rules pass locally
- [ ] Application changes have been tested thoroughly
- [ ] Automated tests covering modified code pass
> config-only: `ruff check` is red on auto-fixables by design (the
reformat branch clears them); eslint/prettier configs load in both
ui-tui and bridge.
### Security
- [x] Security impact of change has been considered
- [ ] Code follows company security practices and guidelines
> Only dependency change is the dev-only pre-commit tool plus the ruff
pin; the detect-private-key hook from main is retained.
### Code review
- [x] Pull request has a descriptive title and context useful to a
reviewer
- [ ] Pull request linked to JIRA ticket when applicable
---------
Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai>
Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
|
|
|
async def chat(
|
|
|
|
|
self,
|
|
|
|
|
messages,
|
|
|
|
|
tools=None,
|
|
|
|
|
model=None,
|
|
|
|
|
max_tokens=4096,
|
|
|
|
|
temperature=0.7,
|
|
|
|
|
reasoning_effort=None,
|
|
|
|
|
tool_choice=None,
|
|
|
|
|
):
|
|
|
|
|
self.loop_marker_counts.append(sum(1 for m in messages if "[loop]" in str(m.get("content", ""))))
|
2026-06-28 18:17:27 +08:00
|
|
|
if tools is None: # max-iter synthesis call
|
|
|
|
|
return LLMResponse(content="done", finish_reason="stop")
|
|
|
|
|
return LLMResponse(
|
|
|
|
|
content="",
|
|
|
|
|
tool_calls=[ToolCallRequest(id=f"c{len(self.loop_marker_counts)}", name="no_such_tool", arguments={})],
|
|
|
|
|
finish_reason="tool_calls",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
def get_default_model(self) -> str:
|
|
|
|
|
return "stub"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_repeated_tool_failure_nudges_bounded(workspace):
|
|
|
|
|
provider = _AlwaysFailsSameToolProvider()
|
|
|
|
|
agent = AgentLoop(
|
build: whole-repo lint + pre-commit convention (config only) (#59)
## Change description
> Whole-repo lint + pre-commit convention, CONFIG ONLY (no source
reformat in this PR). Python (ruff) and TypeScript (eslint + prettier)
share a 120-col margin via the committed pre-commit gate. This upgrades
main's existing basic ruff setup and adds the TS lint stack (none
existed on main).
>
> What this PR adds:
> - ruff: line-length 120; security rules
S101/S105/S106/S107/S506/S605/S607; policy per-file-ignores (tests +
accepted CLI deferred-imports + product asserts + N818, each commented);
ruff pinned <0.16.
> - eslint/prettier: shared base (`eslint.base.mjs` +
`prettier.config.mjs`); ui-tui extends it with React rules, bridge with
a Node-only config; hermes-ink vendor and generated RPC types excluded.
> - pre-commit: keeps main's hooks (conventional-commit,
detect-private-key, etc.), adds a top-level exclude and 4 local
eslint/prettier hooks; bumps the pre-commit dev dep.
> - 2 lint-surfaced fixes: PlannerContext import (F821) in the sentinel
runner; lower-cased file-local cron-heartbeat constants (N806).
(raven/config/raven.py's F821 was already fixed on main.)
>
> Sequencing: config only. `ruff check` is intentionally red on ~307
auto-fixables (I001/F401/F541/E701/E702); a SEPARATE reformat branch
runs `ruff --fix` + `ruff format` + eslint/prettier --fix to clear them
(100% auto-resolvable, zero manual). Enforcement (pre-commit install /
CI) turns on only after the reformat lands, so do not expect a green
tree from this PR alone.
>
> CLI-owned files (the reformat and several per-file-ignores touch
`raven/cli/*`) are under review by the CLI team. Judgment items (TS
no-explicit-any, no-unused-expressions, S101 convert-vs-keep, N818
rename, C90, mypy) are deferred to v2.
## Type of change
- [ ] Bug fix
- [x] New feature
- [ ] Document
- [x] Others
## Related issues (if there is)
> none
## Checklists
### Development
- [ ] Lint rules pass locally
- [ ] Application changes have been tested thoroughly
- [ ] Automated tests covering modified code pass
> config-only: `ruff check` is red on auto-fixables by design (the
reformat branch clears them); eslint/prettier configs load in both
ui-tui and bridge.
### Security
- [x] Security impact of change has been considered
- [ ] Code follows company security practices and guidelines
> Only dependency change is the dev-only pre-commit tool plus the ruff
pin; the detect-private-key hook from main is retained.
### Code review
- [x] Pull request has a descriptive title and context useful to a
reviewer
- [ ] Pull request linked to JIRA ticket when applicable
---------
Co-authored-by: Sheng Zhao <sheng.zhao@evermind.ai>
Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2026-07-01 12:09:42 +08:00
|
|
|
provider=provider,
|
|
|
|
|
workspace=workspace,
|
|
|
|
|
model="stub",
|
fix(tools): an array sent as its own JSON is still an array
## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
|
|
|
policy=TurnPolicy(max_iterations=6),
|
|
|
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
2026-06-28 18:17:27 +08:00
|
|
|
)
|
|
|
|
|
|
|
|
|
|
await agent._process_message(
|
|
|
|
|
TurnRequest(
|
|
|
|
|
origin=Origin.USER,
|
|
|
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
|
|
|
text="go",
|
|
|
|
|
),
|
|
|
|
|
session_key="s1",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
# A nudge fired (>=1 [loop] marker seen) but never exceeded the per-turn cap.
|
|
|
|
|
assert max(provider.loop_marker_counts) == AgentLoop._LOOP_BREAK_MAX
|
2026-08-18 17:30:49 +08:00
|
|
|
|
|
|
|
|
|
fix(tools): an array sent as its own JSON is still an array
## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
|
|
|
class _NudgeTextProvider(_AlwaysFailsSameToolProvider):
|
|
|
|
|
"""Keeps the nudge text itself, not just a count of markers."""
|
|
|
|
|
|
|
|
|
|
def __init__(self):
|
|
|
|
|
super().__init__()
|
|
|
|
|
self.tool_results: list[str] = []
|
|
|
|
|
|
|
|
|
|
async def chat(self, messages, **kwargs):
|
|
|
|
|
self.tool_results += [
|
|
|
|
|
str(m.get("content", ""))
|
|
|
|
|
for m in messages
|
|
|
|
|
if m.get("role") == "tool" and "[loop]" in str(m.get("content", ""))
|
|
|
|
|
]
|
|
|
|
|
return await super().chat(messages, **kwargs)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _find_skill_stub():
|
|
|
|
|
from raven.contracts.tool import Tool
|
|
|
|
|
|
|
|
|
|
class _FindSkill(Tool):
|
|
|
|
|
@property
|
|
|
|
|
def name(self) -> str:
|
|
|
|
|
return "find_skill"
|
|
|
|
|
|
|
|
|
|
@property
|
|
|
|
|
def description(self) -> str:
|
|
|
|
|
return "stub"
|
|
|
|
|
|
|
|
|
|
@property
|
|
|
|
|
def parameters(self) -> dict:
|
|
|
|
|
return {"type": "object", "properties": {}}
|
|
|
|
|
|
|
|
|
|
async def execute(self, **kwargs) -> str:
|
|
|
|
|
return "ran"
|
|
|
|
|
|
|
|
|
|
return _FindSkill()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
async def _nudges_with_find_skill_switched(workspace, off: bool) -> list[str]:
|
|
|
|
|
"""Drive a real failing streak and hand back the nudges it produced.
|
|
|
|
|
|
|
|
|
|
Switched off through the config file rather than through the registry's
|
|
|
|
|
source directly, so the path under test is the one the settings page uses.
|
|
|
|
|
"""
|
|
|
|
|
import json
|
|
|
|
|
from pathlib import Path
|
|
|
|
|
|
|
|
|
|
from raven.config.loader import get_config_path
|
|
|
|
|
|
|
|
|
|
cfg = get_config_path()
|
|
|
|
|
cfg.parent.mkdir(parents=True, exist_ok=True)
|
|
|
|
|
cfg.write_text(json.dumps({"tools": {"disabledTools": ["find_skill"] if off else []}}), encoding="utf-8")
|
|
|
|
|
|
|
|
|
|
provider = _NudgeTextProvider()
|
|
|
|
|
agent = AgentLoop(
|
|
|
|
|
provider=provider,
|
|
|
|
|
workspace=workspace,
|
|
|
|
|
model="stub",
|
|
|
|
|
policy=TurnPolicy(max_iterations=6),
|
|
|
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
|
|
|
|
)
|
|
|
|
|
agent.tools.register(_find_skill_stub())
|
|
|
|
|
assert isinstance(cfg, Path)
|
|
|
|
|
|
|
|
|
|
await agent._process_message(
|
|
|
|
|
TurnRequest(
|
|
|
|
|
origin=Origin.USER,
|
|
|
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
|
|
|
text="go",
|
|
|
|
|
),
|
|
|
|
|
session_key="s1",
|
|
|
|
|
)
|
|
|
|
|
return provider.tool_results
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_the_nudge_does_not_name_a_switched_off_find_skill(workspace):
|
|
|
|
|
"""The nudge is model-visible text, so a registration check advertised a tool
|
|
|
|
|
the operator had switched off -- and `execute` then refuses it as absent.
|
|
|
|
|
|
|
|
|
|
Registration was a fair proxy for availability until the off switch stopped
|
|
|
|
|
unregistering; a withheld tool stays in the registry, which is what makes the
|
|
|
|
|
switch reversible.
|
|
|
|
|
"""
|
|
|
|
|
nudges = await _nudges_with_find_skill_switched(workspace, off=True)
|
|
|
|
|
|
|
|
|
|
assert nudges, "no nudge fired, so the assertion below would pass vacuously"
|
|
|
|
|
assert not any("find_skill" in n for n in nudges)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_and_does_name_it_when_it_is_on(workspace):
|
|
|
|
|
"""The pairing case, so the assertion above cannot pass by never mentioning
|
|
|
|
|
the tool at all."""
|
|
|
|
|
nudges = await _nudges_with_find_skill_switched(workspace, off=False)
|
|
|
|
|
|
|
|
|
|
assert nudges
|
|
|
|
|
assert any("find_skill" in n for n in nudges)
|
|
|
|
|
|
|
|
|
|
|
2026-08-18 17:30:49 +08:00
|
|
|
class _AlwaysTruncatedWriteProvider(LLMProvider):
|
|
|
|
|
"""Every turn is cut off inside the same `write_file` call."""
|
|
|
|
|
|
|
|
|
|
def __init__(self):
|
|
|
|
|
super().__init__(api_key="test")
|
|
|
|
|
self.nudges: list[str] = []
|
|
|
|
|
|
|
|
|
|
async def chat(self, messages, tools=None, model=None, **kwargs):
|
|
|
|
|
self.nudges.extend(str(m.get("content", "")) for m in messages if "[loop]" in str(m.get("content", "")))
|
|
|
|
|
if tools is None: # max-iter synthesis call
|
|
|
|
|
return LLMResponse(content="done", finish_reason="stop")
|
|
|
|
|
return LLMResponse(
|
|
|
|
|
content="",
|
|
|
|
|
tool_calls=[ToolCallRequest(id=f"c{len(self.nudges)}", name="write_file", arguments={"path": "snake.py"})],
|
|
|
|
|
finish_reason="length",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
def get_default_model(self) -> str:
|
|
|
|
|
return "stub"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_a_truncation_streak_is_nudged_toward_a_smaller_payload(workspace):
|
|
|
|
|
"""The class reaches the nudge from the loop, not only from a direct call.
|
|
|
|
|
|
|
|
|
|
The streak key is already `(tool, class)`; only `[0]` was being read, so a
|
|
|
|
|
per-class text is inert until the call site passes the rest of what it has.
|
|
|
|
|
"""
|
|
|
|
|
provider = _AlwaysTruncatedWriteProvider()
|
|
|
|
|
agent = AgentLoop(
|
|
|
|
|
provider=provider,
|
|
|
|
|
workspace=workspace,
|
|
|
|
|
model="stub",
|
fix(tools): an array sent as its own JSON is still an array
## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
|
|
|
policy=TurnPolicy(max_iterations=6),
|
|
|
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
2026-08-18 17:30:49 +08:00
|
|
|
)
|
|
|
|
|
|
|
|
|
|
await agent._process_message(
|
|
|
|
|
TurnRequest(
|
|
|
|
|
origin=Origin.USER,
|
|
|
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
|
|
|
text="write a long file",
|
|
|
|
|
),
|
|
|
|
|
session_key="s1",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
assert provider.nudges, "two truncations running should have fired the loop break"
|
|
|
|
|
assert all("different tool" not in n for n in provider.nudges)
|
|
|
|
|
assert all("smaller" in n for n in provider.nudges)
|
fix(tools): an array sent as its own JSON is still an array
## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
# no progress: a call that keeps succeeding and keeps answering the same #
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_no_progress_key_separates_two_spellings_of_one_read():
|
|
|
|
|
"""Why the count is per key rather than per turn: the run that motivated
|
|
|
|
|
this alternated a read with the same read behind a checksum, so the two
|
|
|
|
|
spellings key apart and each has to reach the threshold on its own. That
|
|
|
|
|
costs twice the calls and still fires -- a consecutive streak would have
|
|
|
|
|
stayed at one forever.
|
|
|
|
|
"""
|
|
|
|
|
plain = no_progress_key("exec", {"command": "cat t.json"}, "keys: [1]")
|
|
|
|
|
checksummed = no_progress_key("exec", {"command": "md5sum t.json && cat t.json"}, "keys: [1]")
|
|
|
|
|
assert plain != checksummed
|
|
|
|
|
assert plain == no_progress_key("exec", {"command": "cat t.json"}, "keys: [1]")
|
|
|
|
|
assert plain != no_progress_key("exec", {"command": "cat t.json"}, "keys: [2]"), "a changed answer is progress"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class _IdenticalSucceedingCallProvider(LLMProvider):
|
|
|
|
|
"""Calls one tool with one set of arguments, which succeeds the same way every time."""
|
|
|
|
|
|
|
|
|
|
def __init__(self):
|
|
|
|
|
super().__init__(api_key="test")
|
|
|
|
|
self.loop_marker_counts: list[int] = []
|
|
|
|
|
self.nudges: list[str] = []
|
|
|
|
|
|
|
|
|
|
async def chat(self, messages, tools=None, **kwargs):
|
|
|
|
|
self.loop_marker_counts.append(sum(1 for m in messages if "[loop]" in str(m.get("content", ""))))
|
|
|
|
|
self.nudges += [str(m.get("content", "")) for m in messages if "[loop]" in str(m.get("content", ""))]
|
|
|
|
|
if tools is None: # max-iter synthesis call
|
|
|
|
|
return LLMResponse(content="done", finish_reason="stop")
|
|
|
|
|
return LLMResponse(
|
|
|
|
|
content="",
|
|
|
|
|
tool_calls=[
|
|
|
|
|
ToolCallRequest(id=f"c{len(self.loop_marker_counts)}", name="list_dir", arguments={"path": "."})
|
|
|
|
|
],
|
|
|
|
|
finish_reason="tool_calls",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
def get_default_model(self) -> str:
|
|
|
|
|
return "stub"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_an_unchanging_successful_call_is_nudged_and_bounded(workspace):
|
|
|
|
|
"""The case the failure streak structurally cannot see: every call returns
|
|
|
|
|
exit 0, and a success resets that counter to zero.
|
|
|
|
|
"""
|
|
|
|
|
provider = _IdenticalSucceedingCallProvider()
|
|
|
|
|
agent = AgentLoop(
|
|
|
|
|
provider=provider,
|
|
|
|
|
workspace=workspace,
|
|
|
|
|
model="stub",
|
|
|
|
|
policy=TurnPolicy(max_iterations=AgentLoop._NO_PROGRESS_THRESHOLD + 4),
|
|
|
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
await agent._process_message(
|
|
|
|
|
TurnRequest(
|
|
|
|
|
origin=Origin.USER,
|
|
|
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
|
|
|
text="go",
|
|
|
|
|
),
|
|
|
|
|
session_key="s1",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
assert max(provider.loop_marker_counts) >= 1, "an unchanging successful call was never nudged"
|
|
|
|
|
assert max(provider.loop_marker_counts) <= AgentLoop._NO_PROGRESS_MAX
|
|
|
|
|
assert any("same result for the same arguments" in n for n in provider.nudges)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class _ImageReadProvider(LLMProvider):
|
|
|
|
|
"""Reads one image over and over, repainting it first when asked to.
|
|
|
|
|
|
|
|
|
|
The repaint is the point: the file keeps its size, so `read_file` answers with
|
|
|
|
|
the same metadata line every time and different pixels every time.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
def __init__(self, target, repaint: bool):
|
|
|
|
|
super().__init__(api_key="test")
|
|
|
|
|
self._target = target
|
|
|
|
|
self._repaint = repaint
|
|
|
|
|
self.loop_marker_counts: list[int] = []
|
|
|
|
|
|
|
|
|
|
async def chat(self, messages, tools=None, **kwargs):
|
|
|
|
|
from PIL import Image
|
|
|
|
|
|
|
|
|
|
self.loop_marker_counts.append(sum(1 for m in messages if "[loop]" in str(m.get("content", ""))))
|
|
|
|
|
n = len(self.loop_marker_counts)
|
|
|
|
|
shade = (n * 23 % 256, 40, 90) if self._repaint else (10, 40, 90)
|
|
|
|
|
Image.new("RGB", (320, 240), shade).save(self._target)
|
|
|
|
|
if tools is None: # max-iter synthesis call
|
|
|
|
|
return LLMResponse(content="done", finish_reason="stop")
|
|
|
|
|
return LLMResponse(
|
|
|
|
|
content="",
|
|
|
|
|
tool_calls=[ToolCallRequest(id=f"c{n}", name="read_file", arguments={"path": "slide.png"})],
|
|
|
|
|
finish_reason="tool_calls",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
def get_default_model(self) -> str:
|
|
|
|
|
return "stub"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
async def _read_image_turn(workspace, *, repaint: bool, monkeypatch) -> _ImageReadProvider:
|
|
|
|
|
pytest.importorskip("PIL")
|
|
|
|
|
monkeypatch.setattr(AgentLoop, "_supports_vision", lambda self, m=None: True)
|
|
|
|
|
provider = _ImageReadProvider(workspace / "slide.png", repaint)
|
|
|
|
|
agent = AgentLoop(
|
|
|
|
|
provider=provider,
|
|
|
|
|
workspace=workspace,
|
|
|
|
|
model="stub",
|
|
|
|
|
policy=TurnPolicy(max_iterations=AgentLoop._NO_PROGRESS_THRESHOLD + 4),
|
|
|
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
|
|
|
|
)
|
|
|
|
|
await agent._process_message(
|
|
|
|
|
TurnRequest(
|
|
|
|
|
origin=Origin.USER,
|
|
|
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
|
|
|
text="go",
|
|
|
|
|
),
|
|
|
|
|
session_key="s1",
|
|
|
|
|
)
|
|
|
|
|
return provider
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_changing_pixels_behind_one_metadata_line_are_progress(workspace, monkeypatch):
|
|
|
|
|
"""`model_text` is not the whole result, so it cannot be the whole key.
|
|
|
|
|
|
|
|
|
|
An image read answers with path, dimensions and a token estimate -- stable
|
|
|
|
|
across a repaint at the same size -- while the pixels ride in the routed
|
|
|
|
|
blocks. Keyed on the text alone, a render-edit-inspect pass reads "the same
|
|
|
|
|
result" eight times and is told to stop looking, which is the workflow this
|
|
|
|
|
guard was written to help. The pair is what makes this discriminating: the
|
|
|
|
|
same loop over an unchanging image must still fire.
|
|
|
|
|
"""
|
|
|
|
|
repainted = await _read_image_turn(workspace, repaint=True, monkeypatch=monkeypatch)
|
|
|
|
|
assert max(repainted.loop_marker_counts) == 0, "a repainted image counted as no progress"
|
|
|
|
|
|
|
|
|
|
unchanged = await _read_image_turn(workspace, repaint=False, monkeypatch=monkeypatch)
|
|
|
|
|
assert max(unchanged.loop_marker_counts) >= 1, "an unchanging image was never nudged"
|
feat(docker): run raven in a docker container and make a first run configurable from the page
## Summary
Two things arrive together because the second is what made the first usable.
**Run raven from a container.** A Dockerfile builds the page, the Python
environment and the engine into one image; `cd docker && docker compose up`
serves it. nginx serves the built page and proxies the RPC socket to the engine
over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a
second container would have no address to dial. Compose publishes on 127.0.0.1,
and the entrypoint prints a one-time sign-in link on boot.
**A first run is no longer a dead end.** The gateway used to refuse to start
until a provider was configured, while being the thing that serves the page
where a provider gets configured. It now starts, says what is missing, and picks
up a key saved in Settings on the next turn without a restart. The onboarding
overlay no longer takes the page either: New chat, the model chip and Send send
the reader to Settings > Models instead, and an empty model list is marked on
the control that fills it.
Also carried: the provider catalogue registry (capabilities, modalities and
context windows per model), the reworked Settings > Models pane, and four fixes
found in review of that work -- a sibling catalogue answering 200 with a
non-catalogue body no longer crashes the whole fetch; adding a tag no longer
erases the label and description already stored for that model; `authenticated`
no longer demands a bearer token from a local provider reached by address; and
registry rows are keyed by the spelling every lookup produces, which recovers
593 rows that no reader could reach.
## Type
- [ ] Fix
- [x] Feature
- [x] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Commands run on the branch head, with their results:
- `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped
- `npx vitest run` in `ui-web` -- 1573 passed (101 files)
- `npx vitest run` in `ui-tui` -- 2037 passed (142 files)
- `uv run ruff check raven scripts tests` and `ruff format --check` -- clean
- `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean
- `npm run lint:rpc` in `ui-tui` -- generated.ts in sync
- `make check-source-language`, `make check-large-files` -- clean
Each of the four review fixes was checked by reverting it and watching a named
test go red, then restoring it.
The container image itself has not been built by CI or by me -- no Docker daemon
was available on the machine this was written on. It has been built and run by
the author of the branch.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
Two deliberate security trade-offs in the container deployment, both documented
in `docker/README.md`:
- nginx rewrites the `Origin` header, because the engine accepts only an Origin
naming its own loopback port and no browser reaching a published port can send
one. This drops a defence-in-depth layer. The session cookie is
SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite
ignores ports, so another page on localhost counts as same-site.
- `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to
every page load, so anyone who can reach the published port is signed in. It
is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any
deployment reachable from elsewhere. These are two settings that must agree,
and nothing warns when they do not.
Behaviour changes outside the container: the gateway now starts without a
configured provider rather than exiting 1, and the first-run onboarding overlay
no longer opens by default (`?onboard=1` still forces it).
Rollback is by revert; nothing here migrates state or changes an on-disk format.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
2026-09-11 03:27:52 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
# no progress: the ladder past the nudge #
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _guard() -> NoProgressGuard:
|
|
|
|
|
return NoProgressGuard(
|
|
|
|
|
nudge_at=AgentLoop._NO_PROGRESS_THRESHOLD,
|
|
|
|
|
refuse_at=AgentLoop._NO_PROGRESS_REFUSE,
|
|
|
|
|
max_nudges=AgentLoop._NO_PROGRESS_MAX,
|
|
|
|
|
max_refusals=AgentLoop._NO_PROGRESS_REFUSALS_MAX,
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _walk(guard: NoProgressGuard, tool: str, args: dict, result: str, rounds: int) -> list[str]:
|
|
|
|
|
"""One call made `rounds` times, as the loop makes it: check, run, record."""
|
|
|
|
|
steps = []
|
|
|
|
|
for _ in range(rounds):
|
|
|
|
|
action, _text = guard.check(tool, args)
|
|
|
|
|
if action is not NoProgressAction.RUN:
|
|
|
|
|
steps.append(action.name.lower())
|
|
|
|
|
if action is NoProgressAction.END_TURN:
|
|
|
|
|
break
|
|
|
|
|
continue
|
|
|
|
|
guard.record(tool, args, result)
|
|
|
|
|
steps.append("run+nudge" if guard.take_nudge() else "run")
|
|
|
|
|
return steps
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_the_ladder_nudges_then_refuses_then_ends_the_turn():
|
|
|
|
|
"""Why the nudge alone was not enough: it can only fire once for one answer,
|
|
|
|
|
and the model it has to reach may be a fixed point. The measured turn sent
|
|
|
|
|
288 byte-identical calls, 280 of them behind the only guard there was.
|
|
|
|
|
"""
|
|
|
|
|
steps = _walk(_guard(), "exec", {"command": "cat t.json"}, "Exit code: 0\nsame", 40)
|
|
|
|
|
|
|
|
|
|
assert steps == ["run"] * 7 + ["run+nudge"] + ["run"] * 4 + ["refuse"] * 3 + ["end_turn"]
|
|
|
|
|
assert steps.index("end_turn") == 15, "the turn has to end on the 16th identical call"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_a_refusal_says_what_happened_and_what_to_do_instead():
|
|
|
|
|
"""A bare refusal produces a different loop, so the way out has to be in the
|
|
|
|
|
error text -- that is the seat the model has been measured to read."""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
for _ in range(AgentLoop._NO_PROGRESS_REFUSE):
|
|
|
|
|
guard.check("exec", {"command": "cat t.json"})
|
|
|
|
|
guard.record("exec", {"command": "cat t.json"}, "Exit code: 0\nsame")
|
|
|
|
|
action, text = guard.check("exec", {"command": "cat t.json"})
|
|
|
|
|
|
|
|
|
|
assert action is NoProgressAction.REFUSE
|
|
|
|
|
assert text.startswith("Error"), "the registry reads the leading word as the verdict"
|
|
|
|
|
assert "was not run" in text
|
|
|
|
|
assert "take the next real step" in text
|
|
|
|
|
assert "wait for it in one longer wait" in text, (
|
|
|
|
|
"for a poll, waiting explicitly *is* the call being refused, so the way out has to be "
|
|
|
|
|
"a wait the model can actually make instead"
|
|
|
|
|
)
|
|
|
|
|
assert "answered something new" in text
|
|
|
|
|
assert "then this turn ends" in text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_a_poll_whose_answer_changes_is_left_alone():
|
|
|
|
|
"""The shape the result has always been half the key for: identical
|
|
|
|
|
arguments, a moving answer. Forty rounds of it must not reach any step."""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
for i in range(40):
|
|
|
|
|
action, _ = guard.check("read_file", {"path": "job.log"})
|
|
|
|
|
assert action is NoProgressAction.RUN, f"a moving answer was refused on round {i}"
|
|
|
|
|
guard.record("read_file", {"path": "job.log"}, f"lines read: {i}")
|
|
|
|
|
assert guard.take_nudge() is None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_a_wait_loop_keeps_the_identical_sleep_it_waits_with():
|
|
|
|
|
"""The case that made the enforcing step ask a harder question than the
|
|
|
|
|
nudge does. A model waiting on a background job sleeps with one identical
|
|
|
|
|
call and checks with another whose answer moves, so the sleep's own answer
|
|
|
|
|
never changes -- counted the way the nudge counts, it would be refused after
|
|
|
|
|
twelve rounds and the wait would die. The nudge still fires, deliberately:
|
|
|
|
|
injecting a line of advice into a workflow that is in fact progressing costs
|
|
|
|
|
a line of advice, and refusing its calls costs the workflow.
|
|
|
|
|
"""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
nudged = False
|
|
|
|
|
for i in range(40):
|
|
|
|
|
action, _ = guard.check("exec", {"command": "sleep 5"})
|
|
|
|
|
assert action is NoProgressAction.RUN, f"the sleep a wait loop waits with was refused on round {i}"
|
|
|
|
|
guard.record("exec", {"command": "sleep 5"}, "Exit code: 0")
|
|
|
|
|
nudged = nudged or bool(guard.take_nudge())
|
|
|
|
|
guard.record("exec", {"command": "cat progress"}, f"{i}% done")
|
|
|
|
|
guard.take_nudge()
|
|
|
|
|
|
|
|
|
|
assert nudged, "the advisory step is meant to stay lenient, so it should still have fired"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_intervening_progress_lets_a_frozen_pair_run_again():
|
|
|
|
|
"""The freeze is evidence, not a verdict, and evidence goes stale.
|
|
|
|
|
|
|
|
|
|
Twelve identical answers establish what the call returns. Then real work
|
|
|
|
|
happens -- three writes, each a first sighting -- and what the call returns
|
|
|
|
|
is no longer what was established. Reading the freeze off a table instead of
|
|
|
|
|
re-asking left the model with three refusals and the end of the turn, and an
|
|
|
|
|
error telling it to change the call so that it asks a different question,
|
|
|
|
|
which for an exec means editing a command string in order to run the same
|
|
|
|
|
command.
|
|
|
|
|
"""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
call = ("exec", {"command": "cat build.log"})
|
|
|
|
|
for _ in range(AgentLoop._NO_PROGRESS_REFUSE):
|
|
|
|
|
assert guard.check(*call)[0] is NoProgressAction.RUN
|
|
|
|
|
guard.record(*call, "Exit code: 0\nno such target")
|
|
|
|
|
|
|
|
|
|
assert guard.check(*call)[0] is NoProgressAction.REFUSE, "nothing else has answered anything yet"
|
|
|
|
|
|
|
|
|
|
for i in range(3):
|
|
|
|
|
guard.record("write_file", {"path": f"src/{i}.py"}, f"wrote src/{i}.py")
|
|
|
|
|
|
|
|
|
|
action, text = guard.check(*call)
|
|
|
|
|
assert action is NoProgressAction.RUN, "real work happened, so the established answer is stale"
|
|
|
|
|
assert text == ""
|
|
|
|
|
guard.record(*call, "Exit code: 0\nbuild ok")
|
|
|
|
|
assert guard.check(*call)[0] is NoProgressAction.RUN, "and the answer it gave was a new one"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_a_rescued_pair_that_goes_back_to_repeating_still_ends_the_turn():
|
|
|
|
|
"""The bound a rescue must not cost. One new answer per ``refuse_at``
|
|
|
|
|
repeats would otherwise rescue a pair forever -- 12 wasted calls in 13, and
|
|
|
|
|
no ladder at all -- so an establishment after the first spends from the same
|
|
|
|
|
budget the refusals spend.
|
|
|
|
|
"""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
call = ("exec", {"command": "cat t.json"})
|
|
|
|
|
ran = novel = 0
|
|
|
|
|
ended = None
|
|
|
|
|
for attempt in range(600):
|
|
|
|
|
action, _ = guard.check(*call)
|
|
|
|
|
if action is NoProgressAction.END_TURN:
|
|
|
|
|
ended = attempt + 1
|
|
|
|
|
break
|
|
|
|
|
if action is NoProgressAction.REFUSE:
|
|
|
|
|
guard.record("exec", {"command": f"echo {novel}"}, str(novel))
|
|
|
|
|
novel += 1
|
|
|
|
|
continue
|
|
|
|
|
guard.record(*call, "Exit code: 0\nsame")
|
|
|
|
|
guard.take_nudge()
|
|
|
|
|
ran += 1
|
|
|
|
|
|
|
|
|
|
assert ended is not None, "a pair rescued by manufactured novelty has to stay bounded"
|
|
|
|
|
assert ran <= AgentLoop._NO_PROGRESS_REFUSE * (AgentLoop._NO_PROGRESS_REFUSALS_MAX + 1)
|
|
|
|
|
assert (ran, novel, ended) == (24, 2, 27)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_the_stop_says_a_rescued_call_was_run_again_rather_than_only_refused():
|
|
|
|
|
"""A model told it was refused three times when it was in fact run again
|
|
|
|
|
twice reads the wrong lesson out of the stop."""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
call = ("exec", {"command": "cat t.json"})
|
|
|
|
|
text = ""
|
|
|
|
|
for attempt in range(600):
|
|
|
|
|
action, text = guard.check(*call)
|
|
|
|
|
if action is NoProgressAction.END_TURN:
|
|
|
|
|
break
|
|
|
|
|
if action is NoProgressAction.REFUSE:
|
|
|
|
|
guard.record("exec", {"command": f"echo {attempt}"}, str(attempt))
|
|
|
|
|
continue
|
|
|
|
|
guard.record(*call, "Exit code: 0\nsame")
|
|
|
|
|
|
|
|
|
|
assert "refused 2 times" in text
|
|
|
|
|
assert "run again 1 time after other work answered something new" in text
|
|
|
|
|
assert "run again 1 times" not in text, "one rescue is one time, not one times"
|
|
|
|
|
# A guard on nothing at all: at max_refusals 0 the turn ends on the freeze
|
|
|
|
|
# itself, and neither half of the budget was drawn on.
|
|
|
|
|
assert no_progress_stop("exec", 0, 0).endswith(
|
|
|
|
|
"The same call was already established, so it is stopped here rather than repeated to the iteration limit."
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_a_wait_loop_whose_poll_answers_identically_is_stopped_and_says_so():
|
|
|
|
|
"""The shape the wait-loop guarantee does not cover, and had no test.
|
|
|
|
|
|
|
|
|
|
``_novelty`` counts first sightings of a (call, answer) key, so a poll
|
|
|
|
|
answering ``status: RUNNING`` byte-identically contributes none after the
|
|
|
|
|
first: neither key resets, both freeze, and the turn ends on a wait that was
|
|
|
|
|
working. Pinned rather than fixed, because there is nothing to count -- see
|
|
|
|
|
the test below for why -- and pinned with the numbers so the cost of the
|
|
|
|
|
honest answer stays visible: twelve polls, three chances, then the turn ends
|
|
|
|
|
with a wrap-up naming what it was waiting for.
|
|
|
|
|
"""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
sleep = ("exec", {"command": "sleep 5"})
|
|
|
|
|
poll = ("exec", {"command": "cat status"})
|
|
|
|
|
polls = 0
|
|
|
|
|
ended = None
|
|
|
|
|
for round_no in range(1, 200):
|
|
|
|
|
for pair, answer in ((sleep, "Exit code: 0"), (poll, "status: RUNNING")):
|
|
|
|
|
action, _ = guard.check(*pair)
|
|
|
|
|
if action is NoProgressAction.END_TURN:
|
|
|
|
|
ended = round_no
|
|
|
|
|
break
|
|
|
|
|
if action is NoProgressAction.RUN:
|
|
|
|
|
guard.record(*pair, answer)
|
|
|
|
|
guard.take_nudge()
|
|
|
|
|
polls += pair is poll
|
|
|
|
|
if ended:
|
|
|
|
|
break
|
|
|
|
|
|
|
|
|
|
assert (polls, ended) == (12, 16)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_a_stable_wait_loop_and_a_stuck_read_pair_are_one_transcript():
|
|
|
|
|
"""Why the shape above is pinned instead of fixed.
|
|
|
|
|
|
|
|
|
|
A fixed-interval wait with a stable poll and the two spellings of one stuck
|
|
|
|
|
read hand this class the same thing: two pairs, each answering its own
|
|
|
|
|
constant, alternating, nothing else. Any rule that keeps the first running
|
|
|
|
|
keeps the second running too, and the second is the run this guard was
|
|
|
|
|
written for -- an hour spent reading one file 127 times. The distinguishing
|
|
|
|
|
fact, that one poll's answer depends on something outside the turn that will
|
|
|
|
|
change it, is not in the transcript at all.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
def verdicts(first, second):
|
|
|
|
|
guard = _guard()
|
|
|
|
|
out = []
|
|
|
|
|
for _ in range(20):
|
|
|
|
|
for tool, arguments, answer in (first, second):
|
|
|
|
|
action, _text = guard.check(tool, arguments)
|
|
|
|
|
out.append(action.name)
|
|
|
|
|
if action is NoProgressAction.END_TURN:
|
|
|
|
|
return out
|
|
|
|
|
if action is NoProgressAction.RUN:
|
|
|
|
|
guard.record(tool, arguments, answer)
|
|
|
|
|
guard.take_nudge()
|
|
|
|
|
return out
|
|
|
|
|
|
|
|
|
|
wait_loop = verdicts(
|
|
|
|
|
("exec", {"command": "sleep 5"}, "Exit code: 0"),
|
|
|
|
|
("exec", {"command": "cat status"}, "status: RUNNING"),
|
|
|
|
|
)
|
|
|
|
|
stuck_read = verdicts(
|
|
|
|
|
("exec", {"command": "cat t.json"}, "Exit code: 0\nsame"),
|
|
|
|
|
("exec", {"command": "md5sum t.json && cat t.json"}, "Exit code: 0\nsame md5"),
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
assert wait_loop == stuck_read, "no content rule can separate these, so neither can this guard"
|
|
|
|
|
assert wait_loop[-1] == "END_TURN"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_a_longer_wait_lets_the_poll_it_waits_with_run_again():
|
|
|
|
|
"""What the wait loop gets instead of a rule that cannot exist.
|
|
|
|
|
|
|
|
|
|
The refusal tells the model to wait for it in one longer wait. A wait of a
|
|
|
|
|
different length is a first sighting, which is exactly the evidence the
|
|
|
|
|
poll's freeze rests on being absent -- so the poll runs again, three times
|
|
|
|
|
over, and a backing-off wait outlives its own refusal.
|
|
|
|
|
"""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
poll = ("exec", {"command": "cat status"})
|
|
|
|
|
seconds = 5
|
|
|
|
|
polls = 0
|
|
|
|
|
ended = None
|
|
|
|
|
for round_no in range(1, 200):
|
|
|
|
|
action, _ = guard.check("exec", {"command": f"sleep {seconds}"})
|
|
|
|
|
if action is NoProgressAction.END_TURN:
|
|
|
|
|
ended = round_no
|
|
|
|
|
break
|
|
|
|
|
if action is NoProgressAction.RUN:
|
|
|
|
|
guard.record("exec", {"command": f"sleep {seconds}"}, "Exit code: 0")
|
|
|
|
|
guard.take_nudge()
|
|
|
|
|
else:
|
|
|
|
|
seconds *= 2
|
|
|
|
|
continue
|
|
|
|
|
action, _ = guard.check(*poll)
|
|
|
|
|
if action is NoProgressAction.END_TURN:
|
|
|
|
|
ended = round_no
|
|
|
|
|
break
|
|
|
|
|
if action is NoProgressAction.RUN:
|
|
|
|
|
guard.record(*poll, "status: RUNNING")
|
|
|
|
|
guard.take_nudge()
|
|
|
|
|
polls += 1
|
|
|
|
|
else:
|
|
|
|
|
seconds *= 2
|
|
|
|
|
|
|
|
|
|
assert polls == 36, "three chances at twelve polls each, against twelve when the wait never changes"
|
|
|
|
|
assert ended == 40
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_the_measured_identical_call_loop_is_stopped_where_it_always_was():
|
|
|
|
|
"""The 207-call turn this ladder exists for, run past the fix.
|
|
|
|
|
|
|
|
|
|
Nothing else answers anything new in it, so there is no evidence for the
|
|
|
|
|
freeze to go stale against and the pair never runs again: twelve calls,
|
|
|
|
|
three refusals, the turn ends on the sixteenth attempt -- the same numbers
|
|
|
|
|
as before the freeze became re-checkable.
|
|
|
|
|
"""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
call = ("exec", {"command": "python train.py --dry-run"})
|
|
|
|
|
ran = refused = 0
|
|
|
|
|
ended = None
|
|
|
|
|
for attempt in range(207):
|
|
|
|
|
action, _ = guard.check(*call)
|
|
|
|
|
if action is NoProgressAction.END_TURN:
|
|
|
|
|
ended = attempt + 1
|
|
|
|
|
break
|
|
|
|
|
if action is NoProgressAction.REFUSE:
|
|
|
|
|
refused += 1
|
|
|
|
|
continue
|
|
|
|
|
guard.record(*call, "Exit code: 0\nok")
|
|
|
|
|
guard.take_nudge()
|
|
|
|
|
ran += 1
|
|
|
|
|
|
|
|
|
|
assert (ran, refused, ended) == (12, 3, 16)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_two_spellings_of_one_stuck_read_each_reach_the_ladder():
|
|
|
|
|
"""The alternating shape the per-turn count was written for still lands.
|
|
|
|
|
Each spelling keys apart and pays its own way there, and the first rounds
|
|
|
|
|
reset while the other spelling is still new, so it costs more calls -- but
|
|
|
|
|
it is bounded, which is the whole point.
|
|
|
|
|
"""
|
|
|
|
|
guard = _guard()
|
|
|
|
|
ended = 0
|
|
|
|
|
for _ in range(80):
|
|
|
|
|
for command in ("cat t.json", "md5sum t.json && cat t.json"):
|
|
|
|
|
action, _ = guard.check("exec", {"command": command})
|
|
|
|
|
if action is NoProgressAction.END_TURN:
|
|
|
|
|
ended += 1
|
|
|
|
|
break
|
|
|
|
|
if action is NoProgressAction.REFUSE:
|
|
|
|
|
continue
|
|
|
|
|
guard.record("exec", {"command": command}, "Exit code: 0\nsame")
|
|
|
|
|
guard.take_nudge()
|
|
|
|
|
if ended:
|
|
|
|
|
break
|
|
|
|
|
|
|
|
|
|
assert ended == 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class _IdenticalCallPastTheLadderProvider(LLMProvider):
|
|
|
|
|
"""Makes one identical succeeding call for as long as it is allowed to."""
|
|
|
|
|
|
|
|
|
|
def __init__(self):
|
|
|
|
|
super().__init__(api_key="test")
|
|
|
|
|
self.turns = 0
|
|
|
|
|
self.synthesized = False
|
|
|
|
|
self.last_messages: list[dict] = []
|
|
|
|
|
|
|
|
|
|
async def chat(self, messages, tools=None, **kwargs):
|
|
|
|
|
self.last_messages = list(messages)
|
|
|
|
|
if tools is None: # the early-exit wrap-up
|
|
|
|
|
self.synthesized = True
|
|
|
|
|
return LLMResponse(content="here is what I got", finish_reason="stop")
|
|
|
|
|
self.turns += 1
|
|
|
|
|
return LLMResponse(
|
|
|
|
|
content="",
|
|
|
|
|
tool_calls=[ToolCallRequest(id=f"c{self.turns}", name="list_dir", arguments={"path": "."})],
|
|
|
|
|
finish_reason="tool_calls",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
def get_default_model(self) -> str:
|
|
|
|
|
return "stub"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_a_frozen_call_stops_the_turn_without_reaching_the_iteration_cap(workspace):
|
|
|
|
|
"""End to end: the shape that reached iteration 317 of 600 having built
|
|
|
|
|
nothing now ends its own turn, the refusals never run the tool, and the user
|
|
|
|
|
gets the wrap-up rather than a canned apology.
|
|
|
|
|
"""
|
|
|
|
|
cap = AgentLoop._NO_PROGRESS_REFUSE + AgentLoop._NO_PROGRESS_REFUSALS_MAX + 8
|
|
|
|
|
provider = _IdenticalCallPastTheLadderProvider()
|
|
|
|
|
agent = AgentLoop(
|
|
|
|
|
provider=provider,
|
|
|
|
|
workspace=workspace,
|
|
|
|
|
model="stub",
|
|
|
|
|
policy=TurnPolicy(max_iterations=cap),
|
|
|
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
await agent._process_message(
|
|
|
|
|
TurnRequest(
|
|
|
|
|
origin=Origin.USER,
|
|
|
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
|
|
|
text="go",
|
|
|
|
|
),
|
|
|
|
|
session_key="s1",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
results = [str(m.get("content", "")) for m in provider.last_messages if m.get("role") == "tool"]
|
|
|
|
|
refused = [r for r in results if "was not run" in r]
|
|
|
|
|
assert len(refused) == AgentLoop._NO_PROGRESS_REFUSALS_MAX + 1, "three refusals then the one that ends the turn"
|
|
|
|
|
assert any("this turn is ending" in r for r in refused)
|
|
|
|
|
ran = [r for r in results if "was not run" not in r]
|
|
|
|
|
assert len(ran) == AgentLoop._NO_PROGRESS_REFUSE, "a refused call must not run the tool"
|
|
|
|
|
assert provider.turns < cap, "the turn should have ended on the ladder, not on the iteration cap"
|
|
|
|
|
assert provider.synthesized, "an early exit still owes the user an answer"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class _PollUntilWrittenProvider(LLMProvider):
|
|
|
|
|
"""Polls one path with one set of arguments while a job writes to it.
|
|
|
|
|
|
|
|
|
|
Identical arguments every round and a different answer every round, which is
|
|
|
|
|
the case the result has always been half the key for -- run through the loop
|
|
|
|
|
rather than the guard, so the wiring is what the assertion is about.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
def __init__(self, target, rounds: int):
|
|
|
|
|
super().__init__(api_key="test")
|
|
|
|
|
self._target = target
|
|
|
|
|
self._rounds = rounds
|
|
|
|
|
self.polls = 0
|
|
|
|
|
self.refusals: list[str] = []
|
|
|
|
|
|
|
|
|
|
async def chat(self, messages, tools=None, **kwargs):
|
|
|
|
|
self.refusals += [str(m.get("content", "")) for m in messages if "was not run" in str(m.get("content", ""))]
|
|
|
|
|
if tools is None:
|
|
|
|
|
return LLMResponse(content="done", finish_reason="stop")
|
|
|
|
|
if self.polls >= self._rounds:
|
|
|
|
|
return LLMResponse(content="the job finished", finish_reason="stop")
|
|
|
|
|
self.polls += 1
|
|
|
|
|
self._target.write_text("\n".join(f"line {i}" for i in range(self.polls)))
|
|
|
|
|
return LLMResponse(
|
|
|
|
|
content="",
|
|
|
|
|
tool_calls=[ToolCallRequest(id=f"c{self.polls}", name="read_file", arguments={"path": "job.log"})],
|
|
|
|
|
finish_reason="tool_calls",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
def get_default_model(self) -> str:
|
|
|
|
|
return "stub"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_a_poll_runs_past_the_ladder_through_the_whole_loop(workspace):
|
|
|
|
|
"""More rounds than the ladder has steps, and not one of them is refused."""
|
|
|
|
|
rounds = AgentLoop._NO_PROGRESS_REFUSE + AgentLoop._NO_PROGRESS_REFUSALS_MAX + 5
|
|
|
|
|
(workspace / "job.log").write_text("")
|
|
|
|
|
provider = _PollUntilWrittenProvider(workspace / "job.log", rounds)
|
|
|
|
|
agent = AgentLoop(
|
|
|
|
|
provider=provider,
|
|
|
|
|
workspace=workspace,
|
|
|
|
|
model="stub",
|
|
|
|
|
policy=TurnPolicy(max_iterations=rounds + 5),
|
|
|
|
|
tools=ToolWiring(restrict_to_workspace=True),
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
await agent._process_message(
|
|
|
|
|
TurnRequest(
|
|
|
|
|
origin=Origin.USER,
|
|
|
|
|
source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM),
|
|
|
|
|
text="wait for the job",
|
|
|
|
|
),
|
|
|
|
|
session_key="s1",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
assert provider.polls == rounds, "the poll was cut short"
|
|
|
|
|
assert provider.refusals == []
|