Raven-1/scripts/check_source_language.py

290 lines
9.4 KiB
Python
Raw Permalink Normal View History

fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
"""CI gate against non-English source additions outside the named exemption zones."""
from __future__ import annotations
import argparse
import ast
import os
import re
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
ZERO_SHA = "0" * 40
#: The zones where CJK is admissible, per AGENTS.md section 1.3: the two
#: engine wheels with their fixtures, the zh catalog and prompt pack, and the
#: zh-i18n fixture prefix. Markdown files govern themselves wherever they
#: live (owner rule) -- AGENTS.md itself carries zh counter-examples.
EXEMPT_PREFIXES = (
"plugins-dist/ppt-engine/",
"tests/test_ppt_engine_",
"plugins-dist/design-engine/",
"tests/test_design_engine_",
feat(docs): add a bilingual documentation site and its deploy pipeline (#507) ## Summary Stands up a bilingual MkDocs site under `docs-site/`, published to GitHub Pages by a workflow that builds on every pull request and deploys only from `main`. Fourteen topics ship in English and Chinese: the quick start, self-hosting, Docker, the WebUI, the sandbox manual, the runtime architecture, the command reference, the repository layout, the developer workflow, the proactivity design and reference pair, the self-evolution map and the tracing API. This pull request is additive. Both READMEs keep their prose and gain one link to the site in the header row; trimming them is deliberately a second pull request, so the site can be confirmed live before anything depends on it. Key decisions: - **`docs-site/`, not `docs/`.** `docs/README.md` defines `docs/` as design notes and dated records, and `docs/plans/` is an archive explicitly not held to today's layout. A user manual has the opposite lifecycle. - **Full bilingual parity**, via the `mkdocs-static-i18n` suffix structure (`page.md` beside `page.zh.md`). Parity decays immediately without a guard, so a test fails when a page has no twin. - **Anchors are pinned on the Chinese side.** A CJK heading slugifies to nothing, so Markdown numbers it `_1`, `_2`, and inserting one heading renumbers every anchor below it, silently breaking links readers already hold. Each Chinese heading therefore states the anchor its English twin gets for free: 133 heading anchors, identical across the two languages, none positional. - **A left rail, not a top bar.** The header carries the brand, the search box and the navigation tree down the left; the table of contents on the right is one continuous polyline clipped to the headings currently on screen. - **No template overrides.** Material customises through `theme.custom_dir` with Jinja partials, those partials are `.html`, and `scripts/check_large_files.py` holds `.html` in `BLOCKED_ASSET_EXTENSIONS`. The skin is therefore one stylesheet remapping Material's own custom properties, plus one script for the table-of-contents rail. The brand mark is a `data:` URI for the same reason. - **An owner-signed `docs-site/` exemption zone** in AGENTS.md section 1.3. The i18n config carries the Chinese navigation labels, and `mkdocs.yml` is not `*.md`, so the blanket suffix exemption does not reach it. A contract test holds the AGENTS.md list and `EXEMPT_PREFIXES` equal, so both moved together. - **The site never links out to a file it does not host.** Documents that stay in the repository are named as plain paths rather than linked, and a guard fails a page that links to an external webpage or a repository-relative path. Two defects found and closed here rather than after publication. The Chinese site rendered an English sidebar: navigation labels live in `mkdocs.yml`, not in any page, so comparing the two languages' Markdown finds nothing and only the rendered output shows it. And a control could be present, sized, animated and answer `querySelector` while being unclickable, because a popup Material sized for a top bar landed outside the viewport once the header became a rail. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification All commands run at the pushed head, rebased onto `origin/main`. ``` make check-large-files exit 0 make check-source-language exit 0 make docs-build exit 0, 0 errors docs-site/site/index.html and docs-site/site/zh/index.html both present mkdocs_static_i18n: Translated 17 navigation elements to zh uv run pytest tests/test_docs_site.py tests/test_source_language_check.py \ tests/test_readme_scope_canon.py tests/test_docker_runtime.py \ tests/test_living_docs.py 44 passed uv run pytest tests/integration/test_docs_site_controls_e2e.py 13 passed pre-commit run --files <changed files> all hooks pass git diff --check clean ``` Anchor parity was measured on the built HTML rather than assumed: 14 page pairs, 133 heading anchors compared at h2 through h6, every one identical between the two languages and none positional. The controls are hit-tested rather than queried. `document.elementFromPoint` at the exact point a reader clicks must return the control itself, which is the only check that catches an off-viewport popup; the language switcher, the search panel, the repository link and the table-of-contents rail each pass it. The brand lockup is measured, not eyeballed. A failing test was written first and reported the wordmark sitting 4.00px below the mark's centre line; it is 0.00px after the fix, and a 4x screenshot puts the residual optical offset at 0.25px. Every guard this branch adds was confirmed to fail first. The brand-class guard was mutated from both sides in turn: renaming `.em-eyebrow` in the stylesheet turned it red, renaming `class="em-standfirst"` in `index.md` turned it red, and it is green again with both restored. The anchor guard was shown to catch both failure modes, a missing anchor and a wrong one. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk No behaviour change to any shipped code path. Nothing imports `docs-site/`, the new Makefile targets are additive, and the only change outside `docs-site/` and its tests is one link added to each README's header row. Rollback is deleting the branch: the site does not exist until Pages is enabled with `Source: GitHub Actions`, and every other change is inert without it. Disclosed residue, checked and deliberately not changed: - The README header now links to `https://evermind-ai.github.io/Raven/`, which 404s until Pages is enabled. It is the same URL the Documentation section already carries. - The `deploy` job is skipped on pull requests by design, so a green check here does not prove deployment works. The first real evidence is the `deploy` job's own conclusion on the `main` run after merge. - GitHub Pages is not enabled on this repository. Measured: `GET /repos/.../pages` returns 404. Under `Deploy from a branch` the uploaded artifact is silently ignored while the workflow still reports success, so the source must be set to `GitHub Actions`. - `tests/integration/` is excluded by `norecursedirs`, so the browser suite is a local guard and never runs in CI. - The self-hosting page's Makefile target, docker path and port claims are unguarded. They were equally unguarded in the README they came from, and the owner ruled against adding a guard for them. - The self-evolution SOP and the benchmark contract stay in the repository and are named as plain paths from the pages that reference them. The SOP is a translation of an upstream project's internal document and carries redaction placeholders, which is a publication decision rather than a formatting one. - `docs/plans/` still shows the pre-restyle `mkdocs.yml`. CONTEXT.md defines that directory as an archive of the tree as it stood, with an explicit rule against updating an old plan to match a later change. - The Chinese pages drop the brand's italic serif accent word. Cormorant Garamond carries no CJK glyphs, so it would fall back to another face without any sign. - The code font is Material's default Roboto Mono. The brand names four typefaces and no monospace, so none was invented for it. ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>
2026-09-19 15:06:24 +08:00
"docs-site/",
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
"raven/i18n/",
"raven/templates/prompts/zh/",
"tests/test_i18n_",
chore: add git story film skill (#810) ## Summary Add a self-contained git-story-film skill for turning repository history into animated product stories, including the drawing engine, production helpers, and a complete Raven example. - Provide English instructions and a Chinese README while preserving the Chinese film variant and bilingual captions. - Fix bundled example script paths and bring the Python helpers into compliance with Ruff. - Keep the bundled storyboard as a text-only reference with all 24 shots and bilingual captions. No pre-rendered images are bundled; the studio draws storyboard frames from code. - Allow only the required example HTML and localized source data through the repository gates. Keep the 1 MiB limit; standalone images, generated exports, neighboring paths, and unsupported assets remain blocked. ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [x] Other ## Verification - `uv run pre-commit run --from-ref origin/main --to-ref HEAD`: all applicable hooks passed after rebasing onto the latest upstream main. - `make lint-python`: passed, including format checks for 2,147 files. - `make lint-types`: passed. - `make lint-imports lint-deps`: passed; all 10 import contracts held. - `uv run ruff format --check skills/git-story-film`: passed. - `uv run ruff check skills/git-story-film`: passed. - `uv run pytest tests/test_large_file_check.py tests/test_source_language_check.py -q -o addopts='' --idle-ceiling-strict`: 50 passed. - `make check-large-files check-source-language`: passed on the complete committed PR diff. Full CI, browser playback, and MP4 rendering have not been verified locally. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk The skill is an optional development tool and does not change Raven runtime behavior. Repository policy exceptions are limited to documented paths; other assets and oversized files remain blocked. Roll back by reverting this change, including the skill and its matching gate exceptions. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A Co-authored-by: Codex <noreply@openai.com>
2026-09-27 12:56:53 +08:00
"skills/git-story-film/examples/raven-story/zh/",
)
EXEMPT_FILES = frozenset(
{
"skills/git-story-film/examples/raven-story/storyboard/storyboard.html",
"skills/git-story-film/scripts/storyboard.py",
}
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
)
#: The fixture prefix holds only while the test exercises zh machinery: the
#: file must import one of these modules (AGENTS.md section 1.3), or the
#: exemption is revoked and the ordinary passes decide.
I18N_FIXTURE_PREFIX = "tests/test_i18n_"
ZH_MODULES = ("raven.i18n", "raven_ppt", "raven_design")
#: Han (unified + extension A + compatibility), kana, hangul, CJK punctuation
#: and full-width forms. Deliberately narrower than "any non-ASCII": an
#: em-dash in an English docstring is not a language violation.
CJK_RUN = re.compile(
"["
"\u3000-\u303f" # CJK symbols and punctuation
"\u3040-\u30ff" # hiragana and katakana
"\u3400-\u4dbf" # CJK unified ideographs extension A
"\u4e00-\u9fff" # CJK unified ideographs
"\uac00-\ud7af" # hangul syllables
"\uf900-\ufaff" # CJK compatibility ideographs
"\uff00-\uffef" # halfwidth and fullwidth forms
"]+"
)
@dataclass(frozen=True)
class SourceLanguageViolation:
path: str
line_number: int
line: str
@dataclass(frozen=True)
class AddedLine:
path: str
line_number: int
text: str
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Fail when a PR adds CJK source lines outside the exemption zones.")
parser.add_argument("revision_range", nargs="?", default=_default_range())
args = parser.parse_args(sys.argv[1:] if argv is None else argv)
if not args.revision_range:
print("No commit range detected; skipping source language check")
return 0
added, removed = diff_lines(args.revision_range)
violations = find_violations(
added,
removed_runs=cjk_runs(line.text for line in removed),
base_had_cjk=_base_reader(args.revision_range),
head_source=_head_reader(args.revision_range),
)
if not violations:
return 0
print("Non-English source additions are not allowed in PRs:", file=sys.stderr)
for violation in violations:
print(
f"- {violation.path}:{violation.line_number}: {violation.line.strip()}",
file=sys.stderr,
)
print(
"Move the text to an exemption zone named in AGENTS.md section 1.3, or write it in English.",
file=sys.stderr,
)
print(
"A zh test fixture is exempt only as tests/test_i18n_* AND importing raven.i18n or a zh plugin module.",
file=sys.stderr,
)
return 1
def find_violations(
added: list[AddedLine],
*,
removed_runs: set[str],
base_had_cjk,
head_source,
) -> list[SourceLanguageViolation]:
"""Apply the passes: zone, fixture import proof, relocation, carrier.
``base_had_cjk`` and ``head_source`` are each called at most once per
offending path, so the gate costs one ``git show`` per file that actually
needs the carrier rule or the fixture rule.
"""
violations: list[SourceLanguageViolation] = []
carrier_cache: dict[str, bool] = {}
fixture_cache: dict[str, bool] = {}
for line in added:
runs = CJK_RUN.findall(line.text)
if not runs:
continue
if is_exempt_path(line.path):
needs_import_proof = line.path.startswith(I18N_FIXTURE_PREFIX) and not line.path.endswith(".md")
if not needs_import_proof:
continue
if line.path not in fixture_cache:
fixture_cache[line.path] = imports_zh_machinery(head_source(line.path))
if fixture_cache[line.path]:
continue
if all(run in removed_runs for run in runs):
continue
if line.path not in carrier_cache:
carrier_cache[line.path] = base_had_cjk(line.path)
if carrier_cache[line.path]:
continue
violations.append(SourceLanguageViolation(path=line.path, line_number=line.line_number, line=line.text))
return violations
def is_exempt_path(path: str) -> bool:
chore: add git story film skill (#810) ## Summary Add a self-contained git-story-film skill for turning repository history into animated product stories, including the drawing engine, production helpers, and a complete Raven example. - Provide English instructions and a Chinese README while preserving the Chinese film variant and bilingual captions. - Fix bundled example script paths and bring the Python helpers into compliance with Ruff. - Keep the bundled storyboard as a text-only reference with all 24 shots and bilingual captions. No pre-rendered images are bundled; the studio draws storyboard frames from code. - Allow only the required example HTML and localized source data through the repository gates. Keep the 1 MiB limit; standalone images, generated exports, neighboring paths, and unsupported assets remain blocked. ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [x] Other ## Verification - `uv run pre-commit run --from-ref origin/main --to-ref HEAD`: all applicable hooks passed after rebasing onto the latest upstream main. - `make lint-python`: passed, including format checks for 2,147 files. - `make lint-types`: passed. - `make lint-imports lint-deps`: passed; all 10 import contracts held. - `uv run ruff format --check skills/git-story-film`: passed. - `uv run ruff check skills/git-story-film`: passed. - `uv run pytest tests/test_large_file_check.py tests/test_source_language_check.py -q -o addopts='' --idle-ceiling-strict`: 50 passed. - `make check-large-files check-source-language`: passed on the complete committed PR diff. Full CI, browser playback, and MP4 rendering have not been verified locally. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk The skill is an optional development tool and does not change Raven runtime behavior. Repository policy exceptions are limited to documented paths; other assets and oversized files remain blocked. Roll back by reverting this change, including the skill and its matching gate exceptions. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A Co-authored-by: Codex <noreply@openai.com>
2026-09-27 12:56:53 +08:00
return path in EXEMPT_FILES or path.startswith(EXEMPT_PREFIXES) or path.endswith(".md")
fix(tools): an array sent as its own JSON is still an array ## Summary Reported as "deliver_files is broken". It is not: **the tool never ran.** Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered: ``` Error: Invalid parameters for tool 'deliver_files': files should be array ``` What the model sent was the array, spelled as a string: ```json {"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."} ``` `cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify. ## Type - [x] Fix - [ ] Feature ## The edge of the forgiving is the design When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error. **Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through. **A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage. ## Not just deliver_files Six built-in tools declare an array or object parameter and were all exposed: | tool | parameter | |---|---| | `deliver_files` | `files` | | `ask_user` | `questions` | | `image_generate` | `images` | | `message` | `media` | | `create_playbook` | `skills` | | `video_generate` | `params` | The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch. ## Verification ``` make lint-python ruff check + ruff format --check, 1850 files make lint-imports 9 contracts kept, 0 broken pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below) ``` - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots -- no user-facing string **End to end, through the real tool**, with the exact payload that failed five times in production: ``` files sent as: str -> [{"path": ".../poster.png", "title": "..."}] returned: Delivered 1 file(s): poster.png (72 B). still "should be array": False registered in the store: True ``` Twelve cases, five mutations, each killing at least one: | mutation | red | |---|---| | drop the whole branch (the old behaviour) | 3 | | raise instead of falling through on a parse failure | 5 | | accept the parse whatever type it yielded | 1 | | do not re-cast the parsed value, so items are not cast | 1 | | widen the tuple to include `string` | **survived -- unreachable** | The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so. **Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own. ## Risk - [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before - [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed - [x] Rollback path is clear -- single-commit revert ## Related Issues N/A ## Swept `mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`. - **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects. - **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`. - **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case). - **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts. - **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none. - **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value. - **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed. Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text). ## Deliberately not fixed The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument. ## Moved from !594 Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name. Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
def imports_zh_machinery(source: str) -> bool:
"""True when the module imports ``raven.i18n`` or a zh plugin module.
The machine-checkable proxy AGENTS.md section 1.3 names for "the test
exercises zh functionality": a ``tests/test_i18n_*`` fixture keeps its
exemption only when its imports prove it drives the zh machinery.
"""
try:
tree = ast.parse(source)
except SyntaxError:
return False
for node in ast.walk(tree):
if isinstance(node, ast.Import):
modules = [alias.name for alias in node.names]
elif isinstance(node, ast.ImportFrom) and node.module and not node.level:
modules = [node.module, *(f"{node.module}.{alias.name}" for alias in node.names)]
else:
continue
for module in modules:
if any(module == zh or module.startswith(f"{zh}.") for zh in ZH_MODULES):
return True
return False
def cjk_runs(lines) -> set[str]:
runs: set[str] = set()
for line in lines:
runs.update(CJK_RUN.findall(line))
return runs
def diff_lines(revision_range: str) -> tuple[list[AddedLine], list[AddedLine]]:
output = subprocess.check_output(
["git", "diff", "--unified=0", "--no-color", revision_range],
text=True,
errors="replace",
)
added: list[AddedLine] = []
removed: list[AddedLine] = []
path: str | None = None
new_line = 0
for raw in output.splitlines():
if raw.startswith("+++ "):
target = raw[4:]
path = None if target == "/dev/null" else target.removeprefix("b/")
elif raw.startswith("@@ "):
match = re.search(r"\+(\d+)", raw)
new_line = int(match.group(1)) if match else 0
elif path is not None and raw.startswith("+") and not raw.startswith("+++"):
added.append(AddedLine(path=path, line_number=new_line, text=raw[1:]))
new_line += 1
elif raw.startswith("-") and not raw.startswith("---"):
removed.append(AddedLine(path=path or "", line_number=0, text=raw[1:]))
return added, removed
def _base_reader(revision_range: str):
base = _range_base(revision_range)
def base_had_cjk(path: str) -> bool:
if base is None:
return False
result = subprocess.run(
["git", "show", f"{base}:{path}"],
capture_output=True,
text=True,
errors="replace",
)
if result.returncode != 0:
return False
return bool(CJK_RUN.search(result.stdout))
return base_had_cjk
def _head_reader(revision_range: str):
head = _range_head(revision_range)
def head_source(path: str) -> str:
if head is None:
try:
return Path(path).read_text(encoding="utf-8", errors="replace")
except OSError:
return ""
result = subprocess.run(
["git", "show", f"{head}:{path}"],
capture_output=True,
text=True,
errors="replace",
)
return result.stdout if result.returncode == 0 else ""
return head_source
def _range_base(revision_range: str) -> str | None:
if "..." in revision_range:
left, _, right = revision_range.partition("...")
result = subprocess.run(
["git", "merge-base", left, right or "HEAD"],
capture_output=True,
text=True,
)
return result.stdout.strip() or None if result.returncode == 0 else None
if ".." in revision_range:
return revision_range.split("..", 1)[0] or None
return None
def _range_head(revision_range: str) -> str | None:
"""The side of the diff whose file contents the added lines belong to.
``None`` means a bare revision was given: ``git diff <rev>`` compares
against the working tree, so the fixture rule reads files from disk.
"""
if "..." in revision_range:
right = revision_range.partition("...")[2]
return right or "HEAD"
if ".." in revision_range:
right = revision_range.split("..", 1)[1]
return right or "HEAD"
return None
def _default_range() -> str | None:
pr_base = os.environ.get("GITHUB_PR_BASE_SHA", "").strip()
if pr_base:
return f"{pr_base}..HEAD"
before = os.environ.get("GITHUB_EVENT_BEFORE", "").strip()
if before and before != ZERO_SHA:
return f"{before}..HEAD"
return os.environ.get("RANGE", "").strip() or None
if __name__ == "__main__":
raise SystemExit(main())