86 Commits

Author SHA1 Message Date
zhanghui
d106a5c6ef
feat(plugins): add the everos-cloud-memory plugin and its host wiring (#883)
## Summary

Adds `everos-cloud-memory`, a separately distributed memory plugin for
EverOS Cloud, and wires the host so a second memory plugin is a
first-class choice.

The plugin (`plugins-dist/everos-cloud-memory`) is a thin REST client
over httpx: one API key (the config slice
`plugins.config.everos-cloud-memory.api_key`, or
`EVEROS_CLOUD_API_KEY`), owner ids from the host's `ServiceLocator`
(never from the slice), every add in `agent` mode, and a flush only when
a caller marks the conversation over (`metadata.flush` /
`metadata.is_final`: the sub-agent handoff and the importer's final
batch) and once at stop. It ships its own onboard screen (env or on-file
key used without a prompt; 401 / 403 / unreachable get their own
sentences with Re-enter or Skip), `delete` answers False (the cloud
deletes by scope, nothing calls it), and `health()` classifies seven
answers for `raven doctor` and the settings card.

Host changes, each guarded so `memory.backend` `everos` and `null`
behave exactly as before:

- `raven onboard` step 4 asks "Which memory backend?" when more than one
plugin contributes a screen (preselects the current backend, `Off`
clears it, cancel exits 1 and writes nothing); with one plugin the step
is byte-identical to today.
- The memory page reads whichever backend `memory.backend` names: the
cloud slice's key goes out as a Bearer header and the `/health` probe is
skipped for the cloud; a missing distribution is named in the page's
note.
- `settings.everos` reports `reason: "other_backend"` for any non-EverOS
backend, so the Model page hides the three EverOS-only roles (embedding
stays: knowledge bases use that pin); `settings.everosSet` refuses those
three sections for another backend; `settings.set` admits
`plugins.config.<id>.<field>` against the plugin's manifest through the
same admission door channels use; new `settings.everosCloud` feeds the
card (key source, endpoint, health).
- Provider-key and main-model saves no longer restart the local EverOS
when another backend is active.
- ui-web: an EverOS Cloud card (status chip, key row, endpoint) and the
`settings.everosCloud` data behind it. No page mounts the card: it drew
on Model settings until the owner ruled that a memory backend's endpoint
and credential answer neither question that page asks, and where it
belongs is the Memory section's to say once that section has a design.
Until then the key is entered through `raven onboard` or
`EVEROS_CLOUD_API_KEY`. The fixture gate pins the four optional fields
the offline library never sends.
- Gates: the plugin boundary test scans both memory plugins, coverage
and the release workflow build the second wheel, deptry's ignore list
names the plugin beside `raven_everos`, and a conftest fixture keeps the
second plugin out of entry-point discovery for the existing onboarding
tests (they describe a one-plugin install).

A second topic rides along, because it is what makes the hosted backend
reachable from an agent product at all. Every product under `agents/`
pinned its memory at the stock EverOS address and only `raven-design`
read the host's, so the other four addressed whatever answered on 18791
-- the wizard moves a managed server when the port is taken, and a
server the user runs themselves is wherever they put it.
`inherit_memory_backend` in `raven/config/product_render.py` now carries
`memory.backend`, the endpoint, and (only when the product holds none
itself) the credential from the host into every render, beside the host
denials and plugin opt-outs that already travel there; `raven-design`'s
private copy is deleted and the five product configs stop naming an
address. Identity never travels, which is what keeps five products apart
inside one store. `memory.backend: null` travels like any other value,
because that null is how an operator turns long-term memory off and a
product that kept writing would be the one surface ignoring them; the
product's own default stands only when the host carries no key at all.
Which store memories live in is not a product's choice either: the host
reads a sub-agent's memories back through its own backend, so a product
writing anywhere else left that record empty with nothing reporting it.
`owned` is forced off whenever the local backend is in force, because a
product infers ownership from a slice naming no root -- the shape all
five ship -- and so answered a host that was merely not serving by
starting a second server against a root of its own, stranding every
memory written so far.

Decisions worth a reviewer's eye: the fake cloud
(`tests/_everos_cloud_fake.py`) honours or refuses every field the
backend can send and serves both an in-process transport and an HTTP
face for the real-host acceptance; the live tests
(`tests/integration/test_everos_cloud_real_cloud.py`) skip without
`EVEROS_CLOUD_API_KEY`; run with a real key on 2026-10-09 all six pass
against api.evermind.ai (round trip, get by session, empty flush, 401
wording, probe with and without a key, and extraction without a flush
within two minutes), so the documentation and the service agree on
everything the plugin sends. Design, acceptance cases and plan are under
`docs/specs/2026-10-08-everos-cloud-memory-*.md` and `docs/plans/`.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

Rebased onto main `182dacac0` on 2026-10-13. What that rebase and the
four commits on top were verified with:

- Full suite, `uv run --frozen --python 3.12 --extra dev pytest -q` ->
**29748 passed, 8 failed** in 4m10s. All eight fail identically with
these four commits' sources reverted in-tree at the same base, so none
of them is this branch's: the two `test_the_products_tool_face_*` cases
in the code and oncall launchers (an extra `web_search` in the visible
tool set), `test_agent_loop_token_budget`,
`test_rpc_files::test_a_host_without_libreoffice_says_so`,
`test_ppt_engine_image_search`, `test_shell_background`, and the two
`test_simulation_*` cases.
- Real-host before and after for the inheritance, driving the launcher
the host spawns (`agents/raven-research/run.py`) against an isolated
`RAVEN_HOME` whose EverOS address is `19701`, not the stock port.
Before: the product rendered `base_url: http://localhost:18791` with no
`owned` key, ignored the host, and started an EverOS of its own -- an
orphaned `everos server start --root <its own state root>/everos`
holding 18791, which is the stranding this change exists to stop. After:
it rendered `{"base_url": "http://localhost:19701", "owned": false}` and
said so itself, "the EverOS you manage is not running at
http://localhost:19701 ... Raven does not start or stop it", with `ps`
and `lsof` confirming nothing was started. On the cloud the product
rendered `memory.backend: everos-cloud` with the host's key and left the
out-of-force local slice untouched, and `raven doctor` over that
rendered config answered `everos-cloud: ok https://api.evermind.ai`, so
the credential reached the service. Identity stayed `raven-research-ng`
throughout, and a fingerprint of the operator's own EverOS root was
identical before and after.
- The first CI round on this head found one that the local gates did
not: `tests/test_import_cycle_budget.py`. `inherit_memory_backend` read
the backend-to-distribution pairing from `raven.core.plugin_stack`, and
that edge from `raven.config` into `raven.core` -- which imports
`raven.config` back -- took mutual pairs from 13 to 14 and packages
inside a cycle from 18 to 22. `make lint-imports` keeps ten contracts
and none of them covers that pair, so it was green throughout. The
pairing now lives in `raven/config/raven.py` beside the two fields it
joins (`MemoryConfig.backend` names a contribution,
`PluginsConfig.config` is keyed by the distribution), which `raven.core`
may import and which imports nothing back; `plugin_stack` re-exports the
two names its own callers read from it. An intermediate commit put it in
`raven.contracts` instead, which broke the cycle but took the papers
ceiling from 3700 to 3719 lines -- `tests/test_kernel_budget.py` says a
new paper is a reviewed change to that ceiling, so it moved again rather
than raising it.
- The review round on this head raised one non-blocking finding, fixed
rather than carried forward: the inheritance read `memory.backend` with
`.get()`, which collapses an explicit `null` into an absent key, so a
host with long-term memory off left every packaged agent on its own
shipped backend and still writing. Reproduced by calling the function
directly before the change and after, now read on presence, and pinned
by a case whose mutation (restoring the `.get()`) turns it and nothing
else red.
- One ordering problem found while checking the above, pre-existing and
not fixed here: `tests/test_rpc_settings.py` run before
`tests/test_cli_onboard_commands.py` leaves
`raven.home._current_config_path` pointing at a dead tmp config, so the
onboard cases write somewhere else. With main's own copies of both files
and main's conftest in the tree it is 57 failures; with this branch's,
19. Which cases turn red changes run to run as xdist distributes them
differently.
- `cd ui-web && npx vitest run` -> 876 files, 3728 passed, 0 failed;
`npm run lint` -> 0 errors (4 warnings, all in files this change does
not touch); `npm run gen:check` -> in sync (228 methods); `npm run
type-check` -> clean; `cd ui-tui && npm run lint:i18n && npm run
lint:rpc` -> in sync
- `make lint-python` / `lint-imports` (10 contracts kept) / `lint-types`
/ `lint-deps` -> all pass; `uv lock --check` -> exit 0 with both sides'
new packages present; `scripts/check_source_language.py` and
`scripts/check_large_files.py` on `github/main..HEAD` -> exit 0;
commitlint over every commit -> ok
- Semantic-conflict audit of the rebase, beyond what the suite answers:
21 of the branch's 66 files are also in main's 499, and every one of
main's changes in them is additive and elsewhere in the file. The one
that could have broken silently is main's new `vision.model` /
`vision.provider` simple keys meeting this branch's new
`plugins.config.<id>.<field>` branch in `settings_set`; nothing in the
suite exercises `vision.*` through that method, so it was run directly
-> `{'applied': True}` and `{"model": "..."}` on disk. The subtraction
decisions were re-checked against the new base: no new caller of a
memory backend's `delete`, main's new onboard cases are about TLS and
not the memory step, `inherit_everos_address` exists nowhere, and
`DEFAULT_EVEROS_BASE_URL` is still defined.
- Mutation runs on the new code, restored afterwards: dropping `owned:
false`, letting the host's credential overwrite the product's own, not
writing the host's backend name, and carrying every memory slice instead
of the one in force each turn their own cases red; making the cloud card
render nothing turns 4 of its 5 new cases red and leaves the fifth
("draws nothing when not selected") green.

From the earlier round, on base `6d509a8c5`:

- `uv run --frozen --python 3.12 --extra dev pytest -p no:cacheprovider
-q tests/test_everos_cloud_fake.py tests/test_everos_cloud_backend.py
tests/test_everos_cloud_backend_contract.py
tests/test_everos_cloud_onboard.py tests/test_everos_cloud_discover.py
tests/test_cli_onboard_commands.py tests/test_rpc_memory.py
tests/test_rpc_settings.py tests/test_plugin_boundary.py
tests/test_rpc_schema_match.py tests/test_rpc_registration.py
tests/test_release_plugin_list.py
tests/integration/test_everos_cloud_real_cloud.py` -> 1001 passed, 6
skipped (the six live tests, no key)
- `cd ui-web && npx vitest run` -> 214 files, 3211 passed; `npm run
lint` -> 0 errors (4 warnings, all in files this change does not touch);
`npm run gen:check` -> in sync (207 methods); `cd ui-tui && npm run
lint:rpc && npm run lint:i18n` -> in sync
- `make lint` (ruff, ruff format, ui tsc, deptry) -> ok; `make
lint-types` -> All checks passed; `make build-ui` -> boot snapshots
match; `scripts/check_large_files.py` and
`scripts/check_source_language.py` on `6d509a8c5..HEAD` -> exit 0;
commitlint and `scripts/check_commit_messages.py` on each commit -> ok
- Mutation runs per change (restored afterwards, `git diff` clean): a
host import of the plugin turns the boundary test red; `import
raven_everos` in the plugin turns the import-isolation test red;
removing the chooser's cancel exit, the Bearer header, the restart
guard, the slice-admission guard, the Roles filter and the fake's two
fixes each turn their own test red
- Real-host acceptance (64 cases,
`docs/specs/2026-10-08-everos-cloud-memory-acceptance.md`) run against a
fake EverOS Cloud: the wizard in tmux (chooser, key typed / from env /
on file, 401 / 403 / unreachable wording, back, Off, Ctrl-C,
non-interactive, base-tree transcript byte-identical with one plugin),
`raven agent` turns (user and agent-track recall, agent-mode adds, 4 s
recall bound, a 300 KB message dropped with a warning, a sub-agent trace
stored under its own ids, flushed and read back, one exit flush per
unflushed session), `raven import run` (3 adds then one flush; a hung
flush retried 4x and the exit sweep defeated), `raven doctor` under
seven cloud conditions (seven sentences, exit 2 only for the five
faults) and with the distribution hidden, and the web page through
playwright-cli (memory page counts and the fake's episode, empty and
refused stores, the Model page hiding the three EverOS-only roles while
the embedding row stays, the cloud card rotating a key and showing the
401 wording, three refused settings.set frames over the gateway
WebSocket, the local everos / null / mem0 backends drawing the page as
before, and a provider-key save that restarts the local EverOS on the
base tree and does not on this branch). Evidence and the per-case
verdicts are kept outside the repo. The live service was reached with a
real key after the acceptance run: six tests, six passes, the service's
replies archived with the evidence.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

User-visible changes: the wizard's step 4 chooser (only when two memory
plugins are installed); Model settings hides the three EverOS-only role
rows when `memory.backend` is not `everos`/`null` and draws no cloud
card; the memory page sends a Bearer header for the cloud backend.

Two behaviour changes for agent products, both on the `agents/` lane
only. An agent product now follows the host's `memory.backend` and
endpoint instead of its own, which is a fix for the four products that
never followed the host and a change for `raven-design`, which followed
the endpoint but not the backend. And a product no longer starts an
EverOS server of its own: where it used to answer a host that was not
serving by starting one against an empty root of its own, it now reports
long-term memory as unavailable, which is what the host does. Rollback
for this part is the one commit; the product configs it edits are
shipped files, not user config. The plugin is not installed by default
(its wheel is built by the release workflow; nothing changes for an
install without it). Rollback: revert the squash commit; the config
schema gains no new key (the slice is a plugin slice), so an older raven
reads a config written by this one.

Credentials: the cloud key is written only by `set_plugin_config_fields`
into the plugin's slice; the settings page's key row goes through
`settings.set` with a manifest-declared field, and
`settings.everosCloud` reports `api_key_set` / `key_source`, never the
value.

## Related Issues

N/A

---------

Co-authored-by: gloryfromca <23442919+gloryfromca@users.noreply.github.com>
Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-10-14 11:08:11 +08:00
Kevin Hu
82c76a82a9
feat(knowledge): read every format, show what the index holds, and let an agent search it (#489)
<img width="2318" height="1448" alt="image"
src="https://github.com/user-attachments/assets/bca71d69-4972-420c-b14e-c8cf74a17de9"
/>

## Summary

Everything a knowledge base could not read, and the pieces it cut them
into
saying what they are.

**Reading a PDF.** A PDF records glyphs at coordinates; whether they are
a
heading, a table cell or a running footer is inferred from how they
look. There
are now two readers. The geometric one infers structure from type size
and
position and needs nothing but the file. The other asks deepdoc's
models, ported
into `raven/knowledge/parser/deepdoc`: a layout model that says what
each region
is, a detector and recognizer that read a page carrying no text at all,
and a
table-structure model that rebuilds a grid from a picture of one. 
 Detecting boxes around text the file already spells
out exactly costs an inference pass per page to learn less, and
recognising it
costs accuracy.

The layout graph answers with a shape its own postprocessing
raises on, and its classes are DocLayout-YOLO's order rather than the
list
Both were established by running the model, and the label order is
pinned by a test.

**Tables go in as tables.** `<table>` is the only shape that keeps what
a grid
states -- which column a cell stood under, which cells are merged, which
row is
the header -- and it is the shape a model reads a table in, which is
what these
chunks are for. PDF, Word and PowerPoint all state their merges exactly,
and
they now survive as `colspan` and `rowspan`. The flat form this replaces
had to
repeat a header standing over two columns across both of them, because
there was
nowhere to record a span; it had no callers left afterwards and is gone.
Two
deviations from upstream, both about what the string is for: every
cell's text
is escaped, where upstream interpolates it raw and a cell holding a `<`
produces
markup that is no longer a table, and the span attributes are written
only where
there is a span, where upstream emits `<td  >` for every ordinary cell.

**A picture of where each chunk came from.** A chunk is a few hundred
words with
a page number on it, which is enough to cite and not enough to answer
whether
the parser read the page correctly. So every PDF chunk that knows where
it sits
carries one image: the regions it covers, cropped out of the rendered
pages and
stacked. here they are already structured in the parser's element spans,
so
the markers would only put text nobody wrote into the thing being
embedded.
The panel draws a chunk's tables as tables, parsed and
rebuilt as elements rather than injected: a person can rewrite any piece
from
that same panel, so what comes back from the store is user input however
it got
there.

**The page that reads them.** Upstream retired the knowledge page while
this
branch was out; it is rebuilt against the new architecture, and the half
that
never existed is the one a reader needs -- what a document actually
became once
it was cut and embedded. Documents sit in one-level folders and arrive
by four
routes: a file, a folder, a URL, a written note. Adding and indexing are
two
calls on the contract and nothing drains that queue on its own, so a
page that
stopped after the first left every upload saying Queued for good.

Opening a file takes the whole page: the document on the left, the
pieces the
index holds of it on the right. The left half is the file drawn as its
own
pages rather than framed by the browser's PDF viewer, which reads where
to open
once and answers to nothing afterwards -- and which nothing can be drawn
on.
Pages are rendered once and kept, and the region a piece was cut from is
washed
over the page it came from, placed in fractions of the page so the
picture's
own scale never enters the arithmetic. Clicking a piece takes the file
there;
double-clicking opens it for rewriting.

The pieces can be searched, filtered by state, turned on and off,
deleted,
rewritten and added to, one at a time or in a batch. Every one of those
was
already on the contract with no reader for it. A base's settings are
back and
fuller: how many pieces a search returns, how documents are cut, and how
much
prose comes with a table or a figure. The embedding model is a picker
while the
base holds no pieces and a stated fact once it does -- moving it then is
the
collection made again at the new width and every document re-cut, which
is not
a thing a panel does by being saved.

**An agent can read them.** The retrieval engine was there and nothing
could
reach it: no agent had a tool that opened a library. A reader now points
a
conversation at bases from the composer, the way they pin a working
directory,
and the selection rides in the same place -- the session's metadata,
bound
around the turn.

The tool takes a query and never a base. The scope is the reader's
choice, so a
model cannot reach material nobody offered it, and it is withheld where
nothing
was picked rather than offered and refused. A hit says which document
and page
it came from, because a hit a reader cannot place is a hit they cannot
check --
and the passages are pressable in the transcript, walking into the base,
the
document, the folder it is filed under and then the piece. A citation a
reader
cannot follow is a claim they have to take on trust, and the whole point
of
searching their own material is that they can go and look.

**The word segmenter and the resources.** `raven.core.Tokenizer` is
Infinity's
`rag_tokenizer`, ported: a forward and backward maximum match over a
dictionary
trie, with stemming and lemmatisation on the latin path. XGBoost is not
carried
over. None of the data lives in git -- the dictionary, the NLTK corpora
and the
ONNX graphs run to about a hundred megabytes, so `make fetch-resources`
on an
install and an optional Docker layer behind `RAVEN_RESOURCES` both point
at one
script. A checkout that could not reach the network still works, and
every
feature that reads this data says so when asked.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run on macOS against Python 3.12, with the deepdoc weights fetched (the
model
tests skip without them):

```
uv run pytest                        27450 passed, 111 skipped, 3 failed
npm run test --prefix ui-web          3276 passed
npm run test --prefix ui-tui          2106 passed
npm run type-check --prefix ui-web    clean (tsc)
npm run lint --prefix ui-web          clean (4 pre-existing hook warnings)
make lint-python                      clean (ruff check + format)
make lint-types                       clean (ty)
make lint-imports                     10 contracts kept, 0 broken
make check-source-language            clean
make check-large-files                clean
make build-ui                         built, both boot snapshots match
node ui-web/scripts/check-css.mjs     clean
node ui-web/scripts/check-class-namespace.mjs  clean
```

The three Python failures are not this branch's and were each confirmed
on the
base alone: one asks a host without LibreOffice to refuse a render and
this host
has LibreOffice installed, and two simulation cases fail on an
unmodified
checkout of the base commit.

The RPC contract was regenerated for both consumers and both match it
(`npm run gen --prefix ui-web`, `npm run gen:rpc --prefix ui-tui`, then
`gen:check`). The page's boot snapshots and region goldens were
refreshed where
the composer bar gained a button, and each records that button and
nothing else.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

A PDF's tables, and a Word or PowerPoint file's, are now indexed as HTML
rather
than as header-prefixed lines. Chunk text changes shape for those
documents, so
a base indexed before this and searched after it holds both forms until
its
documents are reindexed; retrieval keeps working either way, because
both forms
are text. The same applies to how much prose a table or a figure carries
with
it: the setting now reaches as far as it says, where before it stopped
at the
nearest paragraph, so a chunk cut before this keeps the narrower context
until
its document is rebuilt. Rolling back is reverting the branch and
reindexing.

The deepdoc models are optional. An install that never ran `make
fetch-resources` reads PDFs geometrically, exactly as before, and says
so when
the models are asked for by name. Nothing here reaches the network at
request
time.

Chunk crops and rendered pages are written at index time and on first
view, both
under the state directory rather than in the repository, and both are
deleted
with their document. A page is immutable for the life of its document
id, so
the route serving them caches; the document route does not. All three
routes
take ids and no path, and the two image routes answer under the
strictest
sandbox in the tree.

What an agent may search is the reader's choice and nothing else. The
tool takes
no base argument, so a model cannot name one, and it is absent from the
schema
entirely for a conversation that named none -- a lane nobody pointed at
a base
is unchanged by this branch. The bases a session carries are stored
beside its
working directory and travel no further.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-10-12 19:05:16 +08:00
Dizhan Xue
3c88206b56
fix(*): resolve the latest release under windows powershell 5.1 (#863)
## Summary

Native Windows installs fail from the shell Windows ships with. Both
commands the README gives for Windows break under Windows PowerShell
5.1:

- `irm https://raven.evermind.ai/install.ps1 | iex` stops with `(308)
Permanent Redirect`. PowerShell 5.1 follows 301, 302 and 307 redirects
but not 308, which is what that URL answers with. This is the error #148
reported; the README already offered the direct URL for it.
- The direct URL downloads the script, which then stops at "Could not
resolve the latest Raven release wheel from GitHub".
`Resolve-RavenLatestVersion` requests the release page without following
the redirect and reads the Location header off the exception that
raises. Only PowerShell 7 attaches the response to that exception.
PowerShell 5.1 returns the unfollowed redirect as the response and
reports `MaximumRedirectExceeded` as an error with no response, so the
lookup always came back empty. This came in with #299 and has been on
main since.

Changes:

- `install.ps1`: the release-page request uses `-ErrorAction Ignore`
instead of `-ErrorAction Stop`, so PowerShell 5.1 reads Location from
the returned response. PowerShell 7 still raises regardless of
`-ErrorAction` and goes through the existing catch. The Location is
still matched against the same stable-tag pattern.
- CI: the installer job ran `install.ps1` under `pwsh` only, which is
how this shipped. It now also runs the piped install under Windows
PowerShell 5.1 (`shell: powershell`), into its own tool, bin and home
directories, sharing the uv cache.
- Docs: README, README.zh-CN, the docs-site quick start (en, zh) and the
release notes template now say that Windows PowerShell 5.1 stops on the
short URL with `(308) Permanent Redirect` and to switch to the direct
URL when that appears. The short URL's redirect itself is unchanged.
- Tests: tripwires for the lookup (`-ErrorAction Ignore`, catch kept)
and for the CI gate (both shells, and separate tool, bin and home
directories, since the version check runs the `raven.exe` in the bin
directory).

Second topic, Windows test hygiene: three tests failed when the suite
ran on a Chinese-locale Windows checkout. CI runs the unit tests on
Linux only, so it never saw them.

- `test_no_workflow_step_enters_the_removed_page_directory` read
workflows in the locale encoding (GBK there) and stopped on the UTF-8 in
`release.yml`. It now reads UTF-8.
- The font and LibreOffice harness tests run `install.sh` code under
`sh`, which on Windows is Git Bash: its curl sits outside `/usr/bin` and
paths mix separators, so two failed and others passed for the wrong
reason (the digest-mismatch test passed because curl was missing). The
ten harness tests now carry the POSIX-only skip the other three
sh-driven tests already had, shared as one `POSIX_SH_ONLY` marker. Linux
runs are unchanged.

#861 merged first, and this branch is rebased onto it. The one conflict
was in `tests/test_install_script.py`. The sh-driven tests #861 added,
its two Node harness tests and
`test_a_download_that_cannot_be_hashed_is_not_installed`, now carry
`@POSIX_SH_ONLY` like their neighbours, so the marker stays the one
spelling of that skip.

Third topic, test infrastructure: Python's locale-dependent text I/O
uses the system code page when no `encoding=` is passed. On a GBK-locale
Windows checkout, any test that reads or writes a file containing
non-ASCII bytes will fail or silently corrupt data. The class of bug was
found above; there are ~790 `read_text()` call sites without `encoding`
in `tests/`. Measured on Linux under a zh_CN.GBK locale, main without
UTF-8 mode is not one test short: 87 tests fail only there, across 25
files, and `tests/test_rpc_schema_match.py` (418 tests) does not
collect, mostly on `UnicodeDecodeError` from reading UTF-8 files.

- Makefile: `export PYTHONUTF8 := 1`, so every recipe runs in UTF-8
mode, `check-core-wheel` (part of `make ci`) included.
- `tests/conftest.py` raises at import time on any interpreter whose
default text encoding is not the UTF-8 codec, telling the caller to set
`PYTHONUTF8=1` or use `-X utf8`. It compares codecs rather than names,
so Windows with the system-wide UTF-8 option, which reports `cp65001`,
passes. `uv run pytest`, which AGENTS.md documents and which does not
read the Makefile, now stops here on a GBK checkout, as does an IDE
runner that does not set the variable. CI's `--noconftest` self-upgrade
job (its tests all name `encoding` explicitly) is unaffected.
`tests/test_conftest_utf8_guard.py` pins the codec comparison and drives
the refusal in an ASCII-locale child.
- CI sets `PYTHONUTF8` in no job. The jobs that load `tests/conftest.py`
run on ubuntu-latest, where the locale is UTF-8 already, and the `unit`
job gets the variable from the Makefile through `make coverage-shard`.
The Windows self-upgrade job runs its one file with `--noconftest`, and
that file names its encodings. The installer job runs no tests at all:
its steps are raven installing itself and answering `--version`, so the
variable there would only change how raven runs, and that gate exists to
run it the way users do.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run on Windows 11 (zh-CN) with Windows PowerShell 5.1.26100 and
PowerShell 7.6.6, before the rebase onto #861 and the second review
round. Every install below was isolated: `UV_TOOL_DIR`,
`UV_TOOL_BIN_DIR` and `RAVEN_HOME` in a temp directory,
`RAVEN_MINIMAL=1`, `RAVEN_NO_LAUNCH=1`.

- Before the fix, under 5.1: `irm https://raven.evermind.ai/install.ps1`
fails with `(308) Permanent Redirect`; `irm
https://raw.githubusercontent.com/EverMind-AI/Raven/refs/heads/main/install.ps1
| iex` stops at `Could not resolve the latest Raven release wheel from
GitHub`.
- `Invoke-WebRequest
https://github.com/EverMind-AI/Raven/releases/latest -MaximumRedirection
0 -UseBasicParsing -ErrorAction Stop`: 5.1 throws
`InvalidOperationException` (`MaximumRedirectExceeded`) with no
`Response`; 7.6.6 throws `HttpResponseException` whose
`Response.Headers.Location` is the tag URL. With `-ErrorAction Ignore`,
5.1 returns the 302 with Location set, and 7.6.6 still throws into the
catch.
- Redirects under 5.1, via httpbin `redirect-to`: 301, 302 and 307 are
followed; 308 is not.
- After the fix, `Get-Content install.ps1 -Raw | Invoke-Expression`
under 5.1 and under 7.6.6, with `uv tool update-shell` disabled in a
temp copy of the script to leave the user PATH alone: both resolved
v0.2.4, installed raven with everos-memory, design-engine and
ppt-engine, and `raven.exe --version` printed `Raven v0.2.4`.
- `uv run pytest tests/test_install_script.py
tests/test_release_plugin_list.py
tests/test_constraints_export_contract.py` plus the six installer tests
in `tests/test_cli_onboard_commands.py`, on Windows: 48 passed, 13
skipped (before: 55 passed, 3 skipped, 3 failed). The two new tripwires
fail against the previous `install.ps1` and `ci.yml`. With the UTF-8
gate added, a bare `uv run pytest --co -q tests/test_install_script.py`
on the GBK checkout raises RuntimeError naming `PYTHONUTF8=1`, while the
same invocations with the variable set pass (45 passed, 13 skipped).
- `uv run ruff check` and `uv run ruff format --check` on both test
files pass; `ty` does not cover `tests/`, the only Python this changes.
`PYTHONPATH=. uv run python scripts/check_source_language.py
origin/main` exits 0.

Run on Linux (Ubuntu 22.04, Python 3.12.13) at this head. A zh_CN.GBK
locale compiled into a private `LOCPATH` gives a non-UTF-8 interpreter
(`locale.getpreferredencoding(False)` is `GBK`, UTF-8 mode off).

- Full suite, `uv run --frozen --python 3.12 --all-extras pytest -q`: 7
failed, 27567 passed, 119 skipped. On main the same command gives 7
failed, 27556 passed, 119 skipped with the identical 7 failing IDs, this
machine's local baseline (five proxy tests, a root-only permission test
and an npm found on PATH), so 0 introduced; the 11 extra passes are this
PR's new tests.
- Main under GBK without UTF-8 mode: 86 failed, 27059 passed, 119
skipped, 11 errors. 87 IDs fail only there, and
`tests/test_rpc_schema_match.py` does not collect. One of the 87, a
non-ASCII file name, is specific to Linux, where GBK also becomes the
filesystem encoding.
- This head under GBK: a bare `uv run pytest --co -q
tests/test_install_script.py` stops at the conftest RuntimeError naming
`PYTHONUTF8=1`. `make test-python`, which exports it: 7 failed, 27567
passed, 119 skipped, the same 7 IDs as under UTF-8. `make
check-core-wheel`: 1 passed, where with the per-target exports it exited
2 at the guard.
- The harness tests marked `@POSIX_SH_ONLY` run on Linux in the full
suite above.
- Mutations: with the old name tuple back in the guard's predicate, only
the `cp65001` case of `tests/test_conftest_utf8_guard.py` fails; with
the guard deleted, its end-to-end refusal test fails. Giving the 5.1
step pwsh's `UV_TOOL_BIN_DIR`, or its `RAVEN_HOME`, now fails the CI
gate test naming the shared variable, where before it passed.
- `make lint-python` passes, and the commit-message, source-language and
large-file gates exit 0 over `origin/main..HEAD`.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

User-visible: the one-line Windows install works again from Windows
PowerShell 5.1 through the direct URL. The short URL still answers with
a 308 there, and the docs now name that error. `install.ps1` is served
from main, so the fix reaches users on merge, and reverting the squash
commit rolls it back the same way. The resolved Location is still
matched against the same stable-tag pattern before anything is
downloaded. The installer CI job gains one Windows install, which reuses
the uv cache.

Developer-visible: on a Windows checkout whose code page is not UTF-8, a
pytest run that loads `tests/conftest.py` without UTF-8 mode stops at
import with a message naming `PYTHONUTF8=1`; the `make` targets set it,
and Linux and macOS runs are unaffected. On Windows the sh-driven
harness tests report as skipped: the ten this PR marks and
`test_a_download_that_cannot_be_hashed_is_not_installed`, beside the
five that already skipped there.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

#148

---------

Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-10-08 17:30:59 +08:00
Cabeza
47398f45a1
fix: run ui-web page assembly on the project's pinned Python (#599)
## What

`make build-ui` assembled the served page with a bare `python3`, which
resolves to whatever interpreter is first on PATH. On a stock macOS
(no Command Line Tools ahead of `/usr/bin/python3` = 3.9.6) the
documented self-hosting recipe `make install-deps && make build-ui`
died after the vite build had already succeeded:

    File "ui-web/build.py", line 198, in main
(dist / "index.html").write_text(out, encoding="utf-8", newline="")
    TypeError: write_text() got an unexpected keyword argument 'newline'

`Path.write_text(newline=...)` exists from Python 3.10, while pyproject
`requires-python` is `>=3.12` and every other Makefile target already
runs its Python through uv. `build-ui` was the one caller that bypassed
the pinned interpreter.

## Change

One line in the Makefile: run the script through
`uv run --frozen --python $(PYTHON_VERSION) python ui-web/build.py`,
the same form `install.sh` already uses for this script (and already
documents the rationale for). No source, test, or dependency changes.

`ui-web/build.py` itself is untouched: its `newline=""` write needs
3.10+, but that half of the fix sits under the ui-web freeze, so this PR
takes the Makefile half only, as the issue suggests.

CI keeps its bare `python3` (the ubuntu runners provide a working one
through setup-python / the runner image); this PR changes the documented
local path only.

## Verification

Reproduced the failure with Python 3.9.13 (the stock-macOS analogue):

    $ python3.9 ui-web/build.py
    TypeError: write_text() got an unexpected keyword argument 'newline'
    (exit 1)

Then ran the changed target's commands in order, on Windows with
uv 0.12.17 and Python 3.12.7:

    npm ci --prefix ui-web
    npm run --prefix ui-web build
    uv run --frozen --python 3.12 python ui-web/build.py    -> exit 0
    -> built dist/index.html (1,496,450 bytes)

Artifact checks:

sha256 ui-web/dist/index.html =
a1068871929266cfe746b5f321eb7faed2158785c8b19d4984783099b4c23181
      (byte-identical to the pre-change output on 3.12)
node ui-web/scripts/check-page.mjs -> OK (3 scripts parse)
    node ui-web/scripts/count-shared-globals.mjs   -> OK
    node ui-web/scripts/check-css.mjs              -> OK
    node ui-web/scripts/check-class-namespace.mjs  -> OK
    dist/index.html carries 0 CR bytes (LF-only artifact)

Repo gates on this commit:

    scripts/check_source_language.py HEAD~1..HEAD  -> pass
    scripts/check_commit_messages.py HEAD~1..HEAD  -> pass

Based on main at 87f392866e95b6828023cd0305f308831f4105f0.

Closes #526

AI disclosure: this PR was drafted with AI assistance (ZCode agent,
model step-5-preview); the reproduction, commands, and results above
were produced by it. The diff is one line; please review before merging.

Co-authored-by: insuffer <237025162+insuffer@users.noreply.github.com>
Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
2026-09-23 23:11:32 +08:00
KT
1bed0cdb9e refactor(ui-web): the conventions a domain follows, with the gates that hold them (#481)
Stacked on #475 (`refactor/ui_web_architecture`): the conventions that
keep the rebuilt `ui-web/` from drifting, written down in
`ui-web/CONTRIBUTING.md` and enforced by gates. Twenty-two commits, each
landed with every gate green. Three of them change what a DOM diff sees
and are named below; the rest are DOM-invisible.

- **One store shape.** `state/store.ts` exports `makeStore<T>()` with
`get / set / subscribe / _resetForTests`; 33 hand-rolled stores
migrated, `getState` / `put` / `commit` / `wipe` retired; the
`store-shape` gate pins the eight modules that keep a different shape,
each with its reason.
- **One source seam.** `ds<K extends keyof Sources>(domain: K)` is
key-typed, so a renamed or missing domain is a compile error; every
`gateway()` call lives in a domain's `source.ts` (`rpc-names` enforces
placement); `app/install.ts` is the only module that assigns
`sources.*`; skills has its own `source.ts` and the shared `ext.list`
read is `features/installed/source.ts`.
- **No island bag.** `features/registry.ts`'s `islands` object is gone;
page-wide code imports a domain's store directly, and the three
callbacks `state/page.ts` needed are registered from `app/install.ts`.
That one edge was 18% of the import graph's closure weight and held the
tree's only module-evaluation-order hazard. The `import-direction` gate
now pins the layering (`lib <- components <- chrome <- app`, `lib <- rpc
<- state <- chrome, features, app`, `features -/-> features` except
`source.ts`/`types.ts`) with today's upward and cross-domain edges as a
down-or-gone list. Two of the four pinned import cycles are dissolved
the way the gate prescribes (a state module declares a slot, the domain
fills it): `state/page.ts` no longer reads `state/caps.ts`,
`state/navfly.ts` no longer imports the rail store; 17 files sit in a
cycle now, in two components, down from 21 in four.
- **Language is resolved at boot** (localStorage, then config, then
`<html lang>`), so `t(key)` is the only way to take a string; every
island root subscribes to the language store and the hand-kept repaint
list shrinks to the two imperative steps. `T()` and `lang.text()` are
retired, and so are the 61 inert `data-i18n*` attributes the old
document walk used to read.
- **Each domain declares itself once.** `features/<domain>/manifest.ts`
(aggregated in `features/manifests.ts`) and `state/pages.ts` drive the
page sections, the Escape order, the body order, the rail buttons, the
mounts and the region goldens; `domain-shape` and `domain-registration`
gates hold the two tables together.
- **Homes and names.** The desk is its own domain (`features/desk/`);
`resume` lives with the session runtime; the composer's three sheets
live in the composer; the session naming protocol is
`state/session/naming.ts`; `features/xa/` is `features/extAgents/` with
matching ids and exports; wiring modules are all `wire.ts`; page roots
are all `<Domain>App`; `popover`, `pane`, `escapeOrder`, `installEntry`
/ `installSkill`, `Frame` replace the competing words; `useInTask` (not
a hook) is `startTaskWith`; the `file-names` gate pins the case rules.
- **Hooks and types.** `react-hooks/rules-of-hooks` is an error (the
seven warnings were a misnamed verb and one hook after an early return,
now `PlainCallRow`); `tsconfig` adds `noUnusedParameters` and the eight
sites it named are gone; `npm run lint` is down to seven warnings, all
`exhaustive-deps` effects that deliberately run once plus one guarded
expression.
- **Gates and tooling.** The stylesheet hash pin is retired (the eleven
declaration-level CSS gates remain), the six CSS gates share one parser,
`check-css` and `check-class-namespace` run under `npm test`; the
class-namespace gate covers every domain with a per-domain prefix, reads
class literals inside `className={...}` expressions as well as
attributes, and pins today's debt as three down-or-gone lists (86
shared, 342 unprefixed, 91 unprefixed in expressions); fixture
responders return `Wire<ResultOf<M>>` instead of `as`-asserted literals
(which exposed five fixture answers that did not match the contract),
and the turn script fixture is a typed union with no cast left; the
offline library's coverage of every called method is gated with a named
exemption list; `boot-order` is three gates by concern; eslint with
import sorting and unused checks (the tui's config) runs as `npm run
lint` after a one-time 206-file sweep, and `make lint-ui` and the CI ui
job run it.
- **Docs and comments.** `ui-web/CONTRIBUTING.md` (layout, domain
skeleton, store shape, rendering, source and fixtures, naming, CSS,
tests and gates, tooling, the adding-a-domain checklist, and a table of
every pinned exception), new `ui-web/CONTEXT.md` entries, a README whose
layout table matches the tree, and `tests/test_living_docs.py` scanning
all three. Every code comment that explained a decision by what the
deleted script layers used to do now names the live owner or is gone.

What a reader can see change:

- The offline `?stub=1` page and the first live frame before the config
answers now render in one resolved language instead of Chinese markup
with English dynamic text; two tooltip strings in the boot golden moved
with it, and the golden now records tooltip attributes by name rather
than by sentence.
- Domain ids follow their directory: `#extAgentsPage`,
`#connectionsPage`, `#memoryPage`, `#playbooksPage` and their bodies and
rail buttons. `page.css` referenced none of them; the knowledge page
keeps `#kbPage` because the stylesheet pins it.
- The 61 `data-i18n`, `-ph`, `-title`, `-aria` and `-tip` attributes are
gone from the chrome markup. No selector, dataset read, stylesheet rule
or gate named one; the 14 goldens that carried them change by exactly
those attributes and nothing else (each new golden diffs clean against
the old one with the attribute text stripped).

Also restored: the cron page's run-history refetch on a language change,
which the earlier `redrawAll` deletion had dropped.

Outside `ui-web/`, three files: the `Makefile`'s `lint-ui` target and
the CI ui job gain `npm run lint`, and `tests/test_living_docs.py` scans
the three `ui-web/` documents.

Not in this PR, each recorded in `CONTRIBUTING.md`'s exceptions table:
i18n namespaces still use their old abbreviations because the catalogue
(`i18n/messages.json`) is shared with the TUI and lives outside
`ui-web/`; the two remaining import cycles (the six-file
composer/transcript/workspace knot and the eleven-file session knot);
the class names still defined in `page.css` rather than per domain; the
three places `rpc-schema/openrpc.json` disagrees with its own
descriptions, which is the backend's to settle.

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [x] Refactor
- [ ] Other

From the repo root, on the branch head:

- `cd ui-web && npm run type-check && npm run lint` -> clean, 7 warnings
(from 14 on the base's first lint run), 0 errors
- `cd ui-web && HOME=$(mktemp -d) npx vitest run` -> 162 files, 2534
tests passed, 0 skipped (from 152 / 2494 on the base; one test whose
subject was deleted went with it)
- `make build-ui` -> `boot-snapshot: OK (284 nodes match golden)` and
`boot-snapshot: OK (285 nodes match golden)`
- `make lint-ui` -> `gen:check`, `npm run lint`, `type-check` all clean
- `cd ui-web && node scripts/check-page.mjs && node
scripts/check-css.mjs && node scripts/check-class-namespace.mjs && npm
run gen:check` -> all OK
- `uv run --frozen pytest tests/test_ui_agent_marks.py
tests/test_ui_provider_marks.py tests/test_ui_css_token_gate.py
tests/test_living_docs.py -q` -> 55 passed
- `make check-source-language` over `origin/main..HEAD` -> exit 0
- `uv run --frozen pre-commit run --from-ref
refactor/ui_web_architecture --to-ref HEAD` -> all passed
- `git diff --stat refactor/ui_web_architecture..HEAD -- . ':!ui-web'`
-> exactly `Makefile`, `.github/workflows/ci.yml`,
`tests/test_living_docs.py`
- Every new gate was mutation-checked (one-line breakage red, restored
green); per-step happy-dom dumps of the body order, listener sequence,
boot walk and language flips are byte-identical across the DOM-invisible
steps and differ only as described above in the other three.
- Not done: a real-browser pass.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

- Security: none; no protocol or auth path changes.
- Backward compatibility: build entry points, script paths CI calls, URL
flags and DOM ids the stylesheet references are unchanged; renamed ids
and the dropped `data-i18n*` attributes appear only in goldens.
- Rollback: revert the squash merge into `refactor/ui_web_architecture`.

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-22 01:05:18 +08:00
Dizhan Xue
f10ac9f116
fix(docs): give each language its own search index and settle the site shell (#516)
## Summary

The documentation site built one search index for both languages, so a
reader on
the English pages was offered Chinese ones: results they cannot read,
behind
links that leave the language they chose. A build hook now splits the
merged
index in two and points the Chinese pages at their own copy, which the
theme
also resolves every result link against.

Where that split runs is the whole fix. The merge lands in a plugin's
own
`on_post_build`, so the split has to be the last thing that event does.
Running
it on `on_shutdown` looks equivalent and is not: `mkdocs serve` never
shuts
down, so the preview went on serving the merged index however often it
rebuilt.
The new guard makes the same assertion against `mkdocs build` and
against
`mkdocs serve`, because passing one says nothing about the other.

The rest settles the shell against the reference layout:

- the rail's foot is its own box the navigation cannot show through, and
the tab
  icon is the EverMind mark;
- the footer keeps to the content column instead of covering the fixed
columns
  once the reader reaches the page foot;
- the contents column reaches the screen foot with its scrollbar hidden
behind a
  fade, and follows the reader by centring the first lit entry;
- search opens as a centred dialog over the page. The form itself
travels into
that dialog, so what stays in the rail is drawn from the field's own
metrics
and its shape does not change on a click. Neither state animates: the
theme
transitions the two shapes into each other, and leaving search played
that out
  in the corner of the page after the reader had moved on.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- `uv run pytest tests/integration/test_docs_site_search_e2e.py` -> 2
passed.
Without this change the preview case fails: the Chinese index is HTTP
404 and
  the English index carries 146 Chinese pages.
- `uv run pytest tests/integration/test_docs_site_controls_e2e.py` -> 25
passed
- `uv run pytest tests/test_docs_site.py
tests/test_source_language_check.py` ->
  31 passed
- `make check-commits`, `make check-source-language`, `make
check-large-files` ->
  exit 0; `ruff check` on the added files -> no findings
- `make docs-build` -> the root index holds 149 entries and no Chinese
page; the
  Chinese index holds 146, none of them addressed from the site root

Driven in a real browser at 1560x820 in both languages: a word that
appears in
both trees returns 10 results and 0 from the other language, and the
corner of
the page is pixel-identical to its never-opened state at every frame
after
leaving search. Every page of the built site was checked at the page
foot (28
pages, 0 faults) for a footer covering a column, for a column whose last
entry
cannot be reached, and for the rail's foot icons being clickable.

No page content changed, so nothing else needed updating with it.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

User-visible: search no longer returns pages from the other language,
and the
search control, the footer and the two fixed columns change shape. The
field
left in the rail while the dialog is open is a drawn replica, and every
rule
behind it sits above the theme's desktop breakpoint, so narrow viewports
keep
the theme's own full-screen search untouched.

Rollback: revert the commit. The hook is one file, and dropping it from
the
`hooks:` list in `docs-site/mkdocs.yml` restores the previous merged
index on
its own.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>
2026-09-19 21:22:24 +08:00
Dizhan Xue
3dd06f6ea0
feat(docs): add a bilingual documentation site and its deploy pipeline (#507)
## Summary

Stands up a bilingual MkDocs site under `docs-site/`, published to
GitHub Pages
by a workflow that builds on every pull request and deploys only from
`main`.
Fourteen topics ship in English and Chinese: the quick start,
self-hosting,
Docker, the WebUI, the sandbox manual, the runtime architecture, the
command
reference, the repository layout, the developer workflow, the
proactivity design
and reference pair, the self-evolution map and the tracing API.

This pull request is additive. Both READMEs keep their prose and gain
one link
to the site in the header row; trimming them is deliberately a second
pull
request, so the site can be confirmed live before anything depends on
it.

Key decisions:

- **`docs-site/`, not `docs/`.** `docs/README.md` defines `docs/` as
design
notes and dated records, and `docs/plans/` is an archive explicitly not
held
  to today's layout. A user manual has the opposite lifecycle.
- **Full bilingual parity**, via the `mkdocs-static-i18n` suffix
structure
(`page.md` beside `page.zh.md`). Parity decays immediately without a
guard, so
  a test fails when a page has no twin.
- **Anchors are pinned on the Chinese side.** A CJK heading slugifies to
  nothing, so Markdown numbers it `_1`, `_2`, and inserting one heading
renumbers every anchor below it, silently breaking links readers already
hold.
Each Chinese heading therefore states the anchor its English twin gets
for
  free: 133 heading anchors, identical across the two languages, none
  positional.
- **A left rail, not a top bar.** The header carries the brand, the
search box
and the navigation tree down the left; the table of contents on the
right is
  one continuous polyline clipped to the headings currently on screen.
- **No template overrides.** Material customises through
`theme.custom_dir` with
Jinja partials, those partials are `.html`, and
`scripts/check_large_files.py`
  holds `.html` in `BLOCKED_ASSET_EXTENSIONS`. The skin is therefore one
stylesheet remapping Material's own custom properties, plus one script
for the
table-of-contents rail. The brand mark is a `data:` URI for the same
reason.
- **An owner-signed `docs-site/` exemption zone** in AGENTS.md section
1.3. The
i18n config carries the Chinese navigation labels, and `mkdocs.yml` is
not
`*.md`, so the blanket suffix exemption does not reach it. A contract
test
holds the AGENTS.md list and `EXEMPT_PREFIXES` equal, so both moved
together.
- **The site never links out to a file it does not host.** Documents
that stay
in the repository are named as plain paths rather than linked, and a
guard
fails a page that links to an external webpage or a repository-relative
path.

Two defects found and closed here rather than after publication. The
Chinese
site rendered an English sidebar: navigation labels live in
`mkdocs.yml`, not in
any page, so comparing the two languages' Markdown finds nothing and
only the
rendered output shows it. And a control could be present, sized,
animated and
answer `querySelector` while being unclickable, because a popup Material
sized
for a top bar landed outside the viewport once the header became a rail.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

All commands run at the pushed head, rebased onto `origin/main`.

```
make check-large-files                                         exit 0
make check-source-language                                     exit 0
make docs-build                                                exit 0, 0 errors
  docs-site/site/index.html and docs-site/site/zh/index.html   both present
  mkdocs_static_i18n: Translated 17 navigation elements to zh
uv run pytest tests/test_docs_site.py tests/test_source_language_check.py \
  tests/test_readme_scope_canon.py tests/test_docker_runtime.py \
  tests/test_living_docs.py                                    44 passed
uv run pytest tests/integration/test_docs_site_controls_e2e.py 13 passed
pre-commit run --files <changed files>                         all hooks pass
git diff --check                                               clean
```

Anchor parity was measured on the built HTML rather than assumed: 14
page pairs,
133 heading anchors compared at h2 through h6, every one identical
between the
two languages and none positional.

The controls are hit-tested rather than queried.
`document.elementFromPoint` at
the exact point a reader clicks must return the control itself, which is
the
only check that catches an off-viewport popup; the language switcher,
the search
panel, the repository link and the table-of-contents rail each pass it.

The brand lockup is measured, not eyeballed. A failing test was written
first
and reported the wordmark sitting 4.00px below the mark's centre line;
it is
0.00px after the fix, and a 4x screenshot puts the residual optical
offset at
0.25px.

Every guard this branch adds was confirmed to fail first. The
brand-class guard
was mutated from both sides in turn: renaming `.em-eyebrow` in the
stylesheet
turned it red, renaming `class="em-standfirst"` in `index.md` turned it
red, and
it is green again with both restored. The anchor guard was shown to
catch both
failure modes, a missing anchor and a wrong one.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

No behaviour change to any shipped code path. Nothing imports
`docs-site/`, the
new Makefile targets are additive, and the only change outside
`docs-site/` and
its tests is one link added to each README's header row.

Rollback is deleting the branch: the site does not exist until Pages is
enabled
with `Source: GitHub Actions`, and every other change is inert without
it.

Disclosed residue, checked and deliberately not changed:

- The README header now links to `https://evermind-ai.github.io/Raven/`,
which
404s until Pages is enabled. It is the same URL the Documentation
section
  already carries.
- The `deploy` job is skipped on pull requests by design, so a green
check here
does not prove deployment works. The first real evidence is the `deploy`
job's
  own conclusion on the `main` run after merge.
- GitHub Pages is not enabled on this repository. Measured: `GET
/repos/.../pages`
returns 404. Under `Deploy from a branch` the uploaded artifact is
silently
ignored while the workflow still reports success, so the source must be
set to
  `GitHub Actions`.
- `tests/integration/` is excluded by `norecursedirs`, so the browser
suite is a
  local guard and never runs in CI.
- The self-hosting page's Makefile target, docker path and port claims
are
unguarded. They were equally unguarded in the README they came from, and
the
  owner ruled against adding a guard for them.
- The self-evolution SOP and the benchmark contract stay in the
repository and
are named as plain paths from the pages that reference them. The SOP is
a
translation of an upstream project's internal document and carries
redaction
placeholders, which is a publication decision rather than a formatting
one.
- `docs/plans/` still shows the pre-restyle `mkdocs.yml`. CONTEXT.md
defines that
directory as an archive of the tree as it stood, with an explicit rule
against
  updating an old plan to match a later change.
- The Chinese pages drop the brand's italic serif accent word. Cormorant
Garamond
carries no CJK glyphs, so it would fall back to another face without any
sign.
- The code font is Material's default Roboto Mono. The brand names four
  typefaces and no monospace, so none was invented for it.

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>
2026-09-19 07:06:24 +00:00
zhanghui
a018351f95
chore(*): gate call correctness with a pinned type checker (#416)
## Summary

Nothing in this repo could see a signature change reach its callers.
Ruff reads one file at a time, the import contracts read edges rather
than calls, and the suite only covers lines it executes. The consequence
was concrete: a required `sandbox_home` argument added to
`BoxliteExecutor` reached `plugins-dist/design-engine`'s render path
with CI green, and a reviewer -- not a gate -- found it.

`ty` answers that question. This adds it as a pinned dev dependency, a
`make lint-types` target and a CI step, with **110 of its 133 rules
live**: nine switched off with their measured backlog recorded beside
each, fourteen shipped disabled upstream and untouched.

Getting there meant clearing 81 diagnostics. This PR folds what were
three stacked PRs (#422 and #423, now closed); each rule's story is a
commit of its own.

### 1. The gate, at 108 rules (`chore(*): gate call correctness...`)

21 diagnostics to clear, five of them real:

| Where | What |
|---|---|
| `evolver/orchestrator/production.py` | `design_of` annotated as taking
two parameters while its docstring, its only implementation and its one
call site all use three |
| `raven/cli/provider_commands.py` | `dict[str, callable]` -- the
builtin function, not a type, so the annotation asserted nothing |
| `raven/config/schema.py` | `Config` is a `BaseSettings` but declared
`env_prefix` / `env_nested_delimiter` through pydantic's `ConfigDict`,
which does not carry those keys |
| `raven/spine/delivery.py` | `StreamDelta.conversation_id` is optional
and was used as a `dict[str, str]` key; a delta without one stored a
`None` entry the close path then read as "no open stream", losing that
stream's done marker |
| `research_flow/support/harness_text.py` | a predicate table typed its
matchers as `object`, so nothing checked they are callable |

The rest were unproven invariants and annotations wider than the code:
three late-bound callables asserted where a comment used to carry the
claim, one closure binding the callable it guarded instead of re-reading
it, four `hasattr` probes that now check what the call actually needs,
and a contextmanager yielding the value it just set.

### 2. `unresolved-import` (`fix(*): resolve the imports the type gate
could not`)

21 diagnostics; three were stale module paths nothing had run into:

- `HarnessNode.current_core_version()` read `raven.__core_version__`, a
module that exists neither here nor in the wheel
`scripts/build_core_wheel.py` produces. The read sat inside `try/except
ImportError`, so the except branch was the only branch and **every node
`store.py` saved recorded `core_version="unknown"`**. It reads
`raven.__version__` now and returns `0.1.13`.
- `raven/agent/loop/wiring.py` imported `SkillForgeRouterConfig` from
`raven.config.schema`; it lives in `raven.config.raven`. Under
`TYPE_CHECKING`, so the annotation simply meant nothing.
- three scripts imported `raven.context_engine.base` and
`raven.memory_engine.base`, neither of which exists. **Two failed at
module import** -- running them was not possible.

The other eighteen were the environment: the three agent products keep
their plugin package beside them, so their roots are declared in
`[tool.ty.environment]`; `lxml` ships a compiled `etree` with no type
information, so `lxml-stubs` is a dev dependency (real types rather than
five silenced lines); `uno` and `com.sun.star.*` are suppressed per
line, since that worker runs inside LibreOffice's own interpreter.

### 3. `invalid-return-type` (`fix(*): make the declared return types
true`)

39 diagnostics, and **one stale annotation held eight of them**:
`run_turn` and the `_run_turn` behind it were declared `-> LoopOutcome`
while all three of their returns build a `TurnOutcome` -- a different
dataclass with no field in common. Nothing broke, because the spine
worker reads what it actually gets and `LoopOutcome` never leaves
`_run_agent_loop`. The annotation was the only thing wrong.

The other 31: nine annotations narrower than the code (five `return
tuple(<genexpr>)` typed as fixed-arity tuples, an inner `one()` still
declaring the two-tuple it returned before a third element was added,
`image_block`'s TypedDict), five declarations refusing a value the
function returns (four tool `execute` overrides saying `-> str` while
returning `ToolResult`), seven invariants that lived in a `# type:
ignore` comment, and ten honest widenings.

### 4. What the gates then caught in this PR's own work

- The qq factory guard passed locally only because an earlier test had
left an event loop behind. Reproduced the CI state
(`asyncio.set_event_loop(None)`), then made the test async so the loop
is its own.
- `invalid-return-type`, while still disabled, hid a regression
introduced here: the qq factory returning the class instead of an
instance. Found by diffing the rule histogram, which is why that rule is
on now.
- The diff-coverage gate named four lines in `raven/routing/cache.py` --
the asserts this work had just added, sitting inside a `try` whose
`except Exception` would have swallowed the `AssertionError` and served
the stale cache. `_do_refresh` returns its data now; the asserts, their
noqa and the swallowing all go with it.

`child_scope`, `QQChannel.start()` and `BenchmarkCache` had no tests at
all and have them now. Every one was mutation-checked.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [x] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- `make lint-types` -> All checks passed; 81 diagnostics cleared across
the three rule sets
- Gate proven live three times, once per rule: a missing argument, an
import of a module that does not exist, and a function returning a str
under `-> int` each turn it red, and removing the probe turns it green
- `uv run ruff check` + `ruff format --check` over all six trees ->
clean
- `uv run lint-imports` -> 10 kept, 0 broken; `deptry raven` (all
extras) -> no issues; `uv lock --check` -> resolved
- `make check-source-language` -> exit 0; `commitlint` -> exit 0
- `make coverage` locally: **22050 passed, 45 skipped, 4 failed**; diff
coverage 92.59%
- All four failures are **pre-existing on `main`**, checked by a real
checkout in the same environment where they fail identically: two
`test_the_products_tool_face_*` launcher tests,
`test_a_ceiling_as_large_as_the_window_still_leaves_room_for_history`
(`reserved_tools = 6044 > 6_000`), and
`test_a_missing_key_is_said_once_rather_than_per_query`
(environment-dependent: it asserts `SERPER_API_KEY` is absent)
- Rule counts measured with both `--extra dev` and `--all-extras` --
identical, so this gate's verdict does not depend on which optional
packages are installed

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

Behaviour changes are confined to the fixes listed above, and each
replaces an error path with a correct one: nodes are stamped with the
real raven version instead of `"unknown"`, two scripts that could not
start now start, the stream bookkeeping no longer stores a `None` key,
and a cache refresh returns what it fetched. Everything else is a
declaration moved to match what the function already returned.

`tests/` is out of the gate's scope (113 diagnostics under the same
rules). Tests are executed, so a broken call there is already loud; this
gate is for code CI does not run. Bringing them in is sized, separate
work.

Rollback: revert the whole PR to drop the gate and its fixes together,
or delete the `[tool.ty]` block, the `lint-types` target and its CI step
to keep the fixes and drop only the gate.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-16 07:33:14 -07:00
KT
a7f0502be0
ci(*): make the idle ceiling a gate and take the last waits out (#428)
## Summary

Follow-up to #413, from a measured look at what was left: a CI shard now
spends 72 s on fixed cost (22 s environment, 48 s collection and worker
start) against 82 s of tests, the four shards are within 15 s of each
other, and 70 unmarked tests were still over the 2 s wall-clock ceiling
on CI against 42 on a laptop because CPU-bound work runs three to four
times slower there. This PR makes the ceiling a gate that is portable
across machines, takes the remaining waits out, precompiles bytecode so
the workers stop doing it four times, and fixes two product waits the
same investigation surfaced.

- `test(*)`: the ceiling judges idle time instead of wall clock, where
idle is the wall clock minus every CPU second spent on the test's
behalf, the reaped children's included (`os.times`). A test waiting on a
production sleep or a process that answers nothing waits the same
seconds on every machine; a test that computes for 2 s here and 7 s on
CI, or drives a browser or eight spawned workers, is not the problem.
The numbers ride on the teardown report so the xdist controller can
judge them; options are `--idle-ceiling` / `--idle-ceiling-strict`. The
fixture that shortens production ladders takes in the ACP cancel settle
(5 s) and the gateway delivery grace (2 s); the CLI agent tests stub the
one-shot command's unconditional 2 s hand-off grace; the rpc subagent
tests no longer spawn a login shell or probe a real endpoint over the
network; three literal timeouts shrink where the stub never answers
anyway. What is left over the ceiling carries a marker: seven real
LibreOffice renders and two tests that wait on a process answering
nothing carry `slow`, two whose assertion is the wait itself carry
`production_timing`.
- `ci(*)`: every shard runs with `--idle-ceiling-strict` (on the make
line, not in `addopts`: the Windows job runs `--noconftest`). Bytecode
precompilation was tried and dropped, the numbers are below; the
workflow is otherwise unchanged from #413.
- `test(agent)`: a heartbeat test asked for five beats of the fifteen
its intervals should produce in a 0.6 s window, and a loaded runner
delivered three. It reads the two gaps instead, which the load cannot
invert: the wait before the first beat is one interval, the wait after
it is the other. Older than the gate that surfaced it; nothing here
touches the registry.
- `perf(ppt-engine)`: `MeasuredWidth.width` remembers measured widths
(bounded, oldest key out); a 15-slide template's QA gates made 1848
freetype calls over 162 distinct keys. Importing `text_metrics` first
raised on a circular import through `template.capacity`; capacity now
imports the module and resolves the name at call time.
- `perf(*)`: `raven gateway` and `raven web` start the litellm import on
a daemon thread at boot, so saving a key from the settings page no
longer waits three to five seconds on it.
- `fix(providers)`: one thread at a time initialises litellm. It raises
three logger levels for the duration of the import and puts the captured
values back after, which holds only while one thread is doing it; a
second caller arriving inside that window captures WARNING as the level
to restore, and litellm's DEBUG and INFO stop reaching the file sink
until a restart. The window was narrow while every import sat on a
startup path and is ordinary now that a server warms it in the
background, so the two callers share a module lock. The strict-mode
warm-up also runs before the fixtures that redirect the home, and
publishing the OAuth token directories created them, so it warms under a
temporary home.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [x] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Whole suite, `uv run --frozen --all-extras pytest -q` (macOS, 4
workers), before and after the waits came out and with the ceiling
strict:

| run | result | idle list |
|---|---|---|
| main after #413 | 22052 passed, 43 skipped in 219.41 s | 39 over 2 s
wall clock |
| this branch, `--idle-ceiling` warning | 22057 passed, 43 skipped in
210.20 s | 14 over 2 s idle, all real render or subprocess, now marked |
| this branch, `--idle-ceiling-strict` | 22058 passed, 43 skipped in
204.89 s, exit 0 | none |

The ceiling is three seconds, placed from the suite's own distribution
rather than guessed. Unmarked tests over one second of idle, measured
across the whole suite:

| idle | tests | what they are |
|---|---|---|
| 1.00-1.19 s | 19 | a test that sets a one second timeout and lets it
expire |
| 1.2-1.3 s | 3 | a subprocess handshake |
| 1.5-1.6 s | 3 | an ACP turn interrupted after 0.8 s, and two like it |
| above 1.6 s | 0 | the renders above this line all carry a marker |

A runner stretches that 1.6 s to about 2.2 s, which is how a two second
line failed two shards over tests waiting exactly as long as they meant
to. Three seconds sits in the gap the population leaves, and stays under
every ladder this guards against: the shortest constant the audit found
was a three second grace, the rest run 5, 15, 30 and 60 seconds.

What the metric separates, measured on this suite: a test that sleeps 5
s reports `4.96s idle of 5.02s`; a test that drives a headless browser
for 13 s reports under 0.8 s; six `raven doctor` subprocesses report 1.2
s where wall clock said 4.2 s; eight spawned processes contending for a
lock report under 0.8 s where wall clock said 3.6 s.

Files touched by the product changes and the new markers (`uv run
--frozen --all-extras pytest -q tests/test_litellm_setup.py
tests/test_cli_gateway_commands.py tests/test_cli_serve_commands.py
tests/test_ppt_engine_assets_fonts.py
tests/test_ppt_engine_template_capacity.py
tests/test_cli_doctor_commands.py tests/test_ppt_engine_end_to_end.py
tests/test_ppt_engine_render_service.py tests/test_rpc_model.py`): `434
passed, 2 skipped in 33.39s`, none over 1 s idle. `python -c "import
raven_ppt.services.assets.text_metrics"` in a fresh interpreter:
ImportError before, imports after.

Test-side speedups, measured per test: the ten CLI agent tests 2.0-4.4 s
to 0.05-0.36 s; the three rpc subagent tests 1.4-2.6 s to 0.01-0.02 s;
the rpc save-key test 2.2-2.8 s to 0.03 s; the two ACP probe tests 2.1 s
to 0.2 s.

Bytecode precompilation, seconds from the start of the pytest step to
the first test, per shard, over three runs of this workflow:

| what was compiled | shard 1 | shard 2 | shard 3 | shard 4 | mean |
install step |
|---|---|---|---|---|---|---|
| nothing (#413) | 52.7 | 46.5 | 45.9 | 46.0 | 47.8 | 3-4 s |
| site-packages and checkout | 39.8 | 39.3 | 35.6 | 45.5 | 40.1 | 12-17
s |
| checkout only | 55.5 | 50.1 | 44.4 | 53.8 | 51.0 | 3-4 s |

Compiling site-packages is the half that moved the number and it cost
more in the install step than it returned. Compiling the checkout alone
is inside the noise of doing nothing: the four workers share one `.pyc`
cache, so the first to import a module writes it for the rest, and most
of what a laptop measures as cold compilation is pytest rewriting test
files through a cache `compileall` cannot fill. Both are gone; the
comment in the workflow records it.

The reworked heartbeat assertion was checked against the bug it guards
by making the registry keep its first interval for every beat: one beat
arrives in the window and the assertion fails on it.

The diff gate is reproduced locally against `origin/main` after a full
`make coverage`: `Diff coverage: 100.00% (12/12 executable changed
lines)`. The two warm-up calls in the serving boot coroutines carry a
no-cover pragma naming the test that pins them, because those coroutines
bind a port and serve until they are killed.

The lock is checked against the race it closes: with it taken away, a
test that holds the first caller inside the import and starts a second
fails on `[30, 30, 30] == [20, 20, 20]`. A strict session under a clean
HOME now leaves no `.raven` behind.

`make lint-python` clean.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed (none)

## Risk

- The ceiling is now a gate on CI. A new test that waits more than 2 s
without `slow` or `production_timing` turns its shard red; the summary
names the test and the marker to use. A local run stays warning-only.
- A subprocess that does real work is charged to the test that reaped
it, so only a real wait trips the gate. A strict session imports litellm
up front, because its 1.7 s of file opens would otherwise land on
whichever test in the shard imports it first.
- The litellm warm-up runs on a daemon thread at server boot only; unit
tests never reach either boot path (their source is pinned instead), and
a failed import is logged, not raised.
- The width memo returns the same floats; the cap evicts the oldest key.
The capacity import change resolves the same name one step later.
- Rollback: revert the PR; every change is additive or a marker.

- [x] Security impact considered (no security surface touched)
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-16 14:51:50 +08:00
KT
a5f84b7164
ci(*): run unit tests as four shards and speed up the suite (#413)
## Summary

The unit job takes twelve minutes on one runner. The first cut of this
PR took the production waits out of the tests, which made the suite a
quarter faster on a laptop and moved CI by 14 s (pytest 720.95 s on
main, 706.57 s here): the runner is three to four times slower than a
laptop on CPU-bound tests, and two of the twelve minutes are the four
xdist workers importing every test module. So the PR now attacks the
wall clock from every side that CI's own numbers pointed at.

- `ci(*)`: the unit job runs as four shards. pytest gains `--shard K/N`
in `tests/conftest.py`, decided in `pytest_ignore_collect` so a shard
never imports the other shards' files; test files are dealt round-robin
over their sorted list so the slow families that sort together (the ppt
engine's) are spread out. Each shard writes coverage data and no report;
a `coverage` job downloads the four data files, combines them, writes
`coverage.json` and `coverage.xml`, and runs the gates that used to
follow the tests.
- `perf(ppt-engine)`: `menu()` caches its reading of a template, capped
at eight entries. One bind reads the bound template and then every other
bundled template for the borrow offer; with ten templates bundled, every
bind evicted what the next one needed and re-measured all of them with
freetype. The cap is 32 and a hit moves its entry to the back. On this
PR's first CI run the template-tool file alone was 257 s of the
over-ceiling list, its tests 10 to 26 s each.
- `test(*)`: an autouse fixture collapses the provider chat retry
ladder, the loop's LLM-error ladder and the memory store backoff to a
millisecond per step, and the steer hook grace to one poll. It keeps
each tuple's length because both ladders count attempts with `len()`;
the steps are a millisecond rather than zero because the store pipeline
waits with `wait_for(stopping.wait(), timeout=delay)`, and a zero
timeout cancels that wait before it runs. A test that proves a timing
property opts out with the new `production_timing` marker. Ten tests
spent 257 s of the suite's worker time in those waits; none asserted on
the time.
- `test(*)`: a wall-clock ceiling. Setup plus call over 2 s on a test
marked neither `slow` nor `production_timing` is listed at the end of
the run; `--wall-clock-ceiling-strict` fails an otherwise green session
on it. Warning only here; CI flips to strict once the remaining tail is
marked.
- `test(*)`: every test created two numbered temp directories, and
pytest picks each number by scanning the base temp dir; the fake home is
now a sibling of `tmp_path`. `MPLCONFIGDIR` points at a git-ignored dir
in the checkout, because the fake home changed per test and matplotlib
rebuilt its font list from the system fonts on every run.
- `refactor(agent)`: the outer LLM-error retry ladder default `(15, 30,
60)` lived in three places; it is one constant in
`raven/config/schema.py` now, read by both `RecoveryLimits` and
`_ladder`. Behaviour unchanged.
- `test(updates)`: the first sharded run turned up an order dependency.
`beta_channel` binds `ReleaseInfo` from `raven.updates.upgrade` at
import, the update-notice tests stand a fake in for that module through
`sys.modules`, and `_refresh` imports `beta_channel` lazily; a worker
that imported `beta_channel` for the first time inside that test got an
`ImportError` swallowed as a failed refresh. The test module now imports
`beta_channel` at the top.
- `ci(*)`: the PR diff gate named every `.py` under `plugins-dist` as
production code while coverage measures only `raven` and `raven_everos`,
so a change to the ppt or design engine could never appear in the report
and the gate refused every such PR on sight; the first sharded run hit
it on `menu.py`. The pathspec now names `plugins-dist/everos-memory`,
and a test holds every production pathspec to a package some `--cov`
flag in the Makefile measures.
- Two slow tests were not waits: the sandbox second-cancel test
cancelled one tick too early (two pending cancels coalesce into one
`CancelledError`) and now waits, under a one second timeout, for the
kill to land; the two thousand-session dispatch tests wrote a span per
refused enqueue through a store that opens its file per span, so tracing
is off for them. Follow-up on the product side: `raven/tracing/store.py`
should reuse its handle.

The suite has 30 upper-bound assertions of the form `assert elapsed <
X`; shortened delays can only make those looser.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [x] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Whole suite, `uv run --frozen --all-extras pytest -q` (macOS, 4
workers), 22051 passed / 43 skipped each time:

| tree | wall clock | unmarked tests over 2 s |
|---|---|---|
| main before this branch | 372.82 s | (no ceiling yet) |
| after the waits were taken out | 279.65 s | 63 |
| this branch | 224.25 s | 39 |

The shards partition the suite exactly (`uv run --frozen --all-extras
pytest --collect-only -q -n 0 --shard K/4`, K = 1..4):

| whole | shard 1 | shard 2 | shard 3 | shard 4 | union | ids in two
shards |
|---|---|---|---|---|---|---|
| 22095 | 5058 | 5555 | 5955 | 5527 | 22095 | 0 |

196 / 195 / 194 / 194 test files per shard. A shard collects in 7 s
where the whole suite takes 23 s on the same machine. `--ignore` still
works under `--shard`: with four `--ignore` flags on shard 1, the
ignored file that shard owns is gone from its collection (5034 ids
against 5058).

Combining loses nothing. Two small runs with `--cov=raven
--cov=raven_everos --cov-branch --cov-report=` and `COVERAGE_FILE` set,
then `make coverage-combine COVERAGE_DATA_DIR=<dir>`, against one `make
coverage` of the whole suite on main:

| report | files | statements | branches |
|---|---|---|---|
| one run of the whole suite | 704 | 82634 | 25548 |
| two shards combined | 704 | 82636 | 25548 |

Same file set; the two extra statements are the new constant and its
import. A source package's never-imported files reach the combined
report through the shard that imported it, as they reached the single
run.

The menu cache, the three files that bind templates (`uv run --frozen
--all-extras pytest -q tests/test_ppt_engine_template_tool.py
tests/test_ppt_engine_outline_checks.py
tests/test_ppt_engine_default_templates.py`): 101 s of worker time in
the last profile, now `196 passed in 11.84s` with no test over 2.9 s.
The files that call `menu()` directly: `219 passed in 8.06s`.

The font cache, the formula render test at `-n 0`: 8.91 s cold, 0.69 s
on the next run. Seven home-sensitive files: `484 passed in 5.30s`.

Ceiling guard, both modes, with the ceiling forced low so it has
something to list:

```
uv run --frozen --all-extras pytest -q -n 0 --wall-clock-ceiling=0.01 tests/test_import_cycle_budget.py tests/test_updates_install_guard.py
uv run --frozen --all-extras pytest -q -n 2 --wall-clock-ceiling=0.01 --wall-clock-ceiling-strict tests/test_updates_install_guard.py
```

First: 28 passed, five listed, exit 0. Second: 24 passed, four listed,
exit 1 (the controller flips the status; workers do not).

`make lint-python` clean, `make lint-imports` passes. The CI run on this
PR is the measurement that matters: four `unit` shards and one `coverage
gates` job.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed (none: CI
plumbing, a cache cap, test setup)

## Risk

- The coverage gates now run in a job after the four shards. A red shard
skips the gates, a deliberate trade: combining three of four data files
would report a coverage drop that is not one, and the PR is red either
way. The gate scripts and their thresholds are unchanged.
- The diff gate no longer sees `plugins-dist/ppt-engine` or
`plugins-dist/design-engine` at all, as it could never judge them;
measuring them (`--cov=raven_ppt --cov=raven_design`, a new ratchet
baseline) is the follow-up that would bring them under the gate.
- `--shard` is opt-in; a plain `pytest` run collects everything as
before. Shard data artifacts are kept one day, the combined reports 30
days as before.
- No product behaviour change: the ladder default is the same tuple read
from one place; the menu cap only changes how long the cache keeps
entries.
- Test-only risk: a test that implicitly relied on a production-length
retry delay would see different timing. The full suite passed with the
fixture on, and a single test can opt out with `production_timing`.
- Rollback: revert the PR. The unit job goes back to one runner with the
gates inline; both markers are additive.

- [x] Security impact considered (no security surface touched)
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-15 16:00:49 +08:00
gloryfromca
d0af1b89cc chore(*): merge the trunk into the sync branch
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-11 18:01:21 +08:00
gloryfromca
46029007ce chore(*): merge upstream main into the gitlab trunk
Eleven commits from the GitHub trunk since 377a1da14, fifteen files
conflicting. How each was resolved:

- raven/tracing/context.py, trace.py, spans.py and tests/test_tracing_api.py:
  upstream retired the runtime attempt id. attempt.id is now a reserved span
  attribute and attempts.json owns the grouping, so begin_attempt,
  end_attempt, current_attempt and every attempt_id parameter go with it.
  Nothing on this side called them outside tests. The fork's surface field
  and its use() helper share those hunks and stay.
- raven/trajectory/store.py: upstream's module docstring documents
  attempts.json and is a superset of ours.
- raven/cli/onboard_everos.py and tests/test_cli_onboard_commands.py:
  upstream only refreshed three model ids, which this side already carries.
  Kept our English-only role table (AGENTS.md 1.3).
- tests/test_everos_server.py: took upstream's inotify-gate tests and pointed
  them at raven_everos.server, where this side moved the module.
- ui-tui/src/app/useMainApp.ts: copy-on-select was never wired up here;
  upstream wires it.
- ui-tui/src/components/appLayout.tsx: took upstream's copy notice line only.
  Its FloatingOverlays hunk would render that component twice, because this
  side had already moved it into its own anchor box.
- README.md and README.zh-CN.md: upstream's wording under this side's emoji
  headings. The zh benchmark table had to be rebuilt: its separator row
  auto-merged with the prerequisites table's and dropped the data rows.
- tests/test_provider_rates.py and tests/test_trajectory_bundle.py: both
  sides appended independent tests at the end of the file; kept both.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-11 17:52:06 +08:00
0xKT
a15e49eac5 fix(*): stop the page hiding a refused delete and a refused upload
## Summary

Two bugs in the served page, both of the same shape: a refusal the surface did
not report as one. They are one MR because they were found in one pass over the
page's write paths and each is a few lines; the commits are separate and can be
read independently.

**1. Bulk delete dropped rows the server had kept.** `DS.sessions.deleteAll`
awaited `session.delete` and counted every call that did not throw as a
removal. A refusal does not throw: the method answers
`{deleted: null, still_on_disk: true}` when the filesystem refused to remove
the file, and that is a resolved promise. The rail dropped the row while the
transcript stayed on disk, so the session came back on the next reload -- the
one thing the note above that function says it must never do. The bulk path now
reads the answer with the predicate the single delete already uses. The
single-delete half was fixed in 3d4075c8, which added `still_on_disk`; the bulk
path was not part of that change.

**2. Attachments over about 3 MB killed the socket.** `fs.upload` accepts
25 MB. The socket carrying it accepted 4 MiB, because
`web.WebSocketResponse(heartbeat=30)` left `max_msg_size` at aiohttp's default.
An upload rides as base64 inside one JSON-RPC frame and base64 costs four bytes
for every three, so the real ceiling was about 2.99 MB of file -- enforced by
closing the connection, before `fs.upload` was ever called. The reader saw a
reconnect rather than a reason, and every other in-flight call on that socket
died with it. The advertised 25 MB limit was unreachable, and so was the
message that states it. The knowledge base feeds the same method, and manuals
and reports clear 3 MB far more often than pasted screenshots do.

The upload limit moves to `raven/rpc/files.py` beside `MAX_VIEW_BYTES`, and
`frame_ceiling_for_upload` derives the transport ceiling from it, so raising one
can no longer leave the other behind. The page also refuses an oversized file
itself: past the ceiling the socket still closes, so the server's own refusal
cannot reach a client that sent one.

**Checked and deliberately not changed.**

- The page carries its own copy of the 25 MB number. No limit rides the wire
  today, and publishing one through `system.hello` is a schema change with
  generated-client churn on both front ends, which this fix does not need. The
  Python constant is named as the source in the comment beside the copy.
- A client that is not the page -- a script, a future surface -- can still cross
  the frame ceiling and lose the socket, because the guard is in the page. The
  transport cannot answer a frame it refused to read; the fix for that is an
  upload route that does not ride the socket at all, which is a larger change
  than this bug warrants.
- A file one byte over the limit is refused by a message that rounds both
  numbers to `25.0 MB`. Showing exact byte counts reads worse for every other
  size, and the server refuses that file too.
- The composer still encodes before it refuses, unlike the knowledge path. Its
  caller in the React island holds the `File`, so moving the check there is an
  island change; the files it stages are screenshots, where the encode is cheap.
- `MAX_VIEW_BYTES` is unchanged. It bounds what the viewer reads back out and
  travels over HTTP, not over the socket.
- `<agent home>/uploads` has no TTL, no count cap and no reclaim path, and this
  change raises what a single upload can put there from about 3 MB to 25 MB.
  Deferred rather than solved here: a reclaim policy is a decision about that
  directory's whole lifetime, and the ceiling it would be sized against is the
  one this MR is what makes reachable. Named so that it reads as a deferral
  rather than as an oversight.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Gates, each run as its own target and its exit code read directly rather than
through a pipe:

- `make lint-python` -- 0. First run reported `raven/rpc/files.py` and
  `tests/test_rpc_transport.py` would be reformatted; `ruff format` applied and
  the rerun is clean.
- `make lint-imports` -- 0
- `make lint-ui` -- 0 (`gen:check` in sync, `type-check` clean)
- `make test-ui` -- 0. 103 files, 1671 tests, 0 failed.
- `make build-ui` -- 0. `dist/index.html` rebuilt.
- `make test-python` -- 0. 21341 passed, 55 skipped, 0 failed, in 389s. No
  failures to reconcile against a clean `origin/main`, because there were none.
- `make check-commits`, `make check-source-language`, `make check-large-files`
  -- all 0.

Mutation tests, each reverted afterwards with `git status` confirmed clean:

- Removing `max_msg_size` from `WebSocketResponse` fails
  `test_the_socket_carries_a_frame_larger_than_aiohttp_would_allow_by_default`
  and nothing else.
- Making `frame_ceiling_for_upload` return the limit without the base64
  expansion fails
  `test_the_frame_ceiling_can_carry_the_largest_upload_the_method_accepts` and
  nothing else.
- Restoring the bare `await` in `deleteAll` fails two of the four new bulk
  cases: `keeps the rows whose files survived their removal` and `keeps a row
  when the server does not carry the field`.

Verified against a running gateway, not only in tests. Uploading through
`/rpc` with the session token, before the change: 1 MB and 2.5 MB stored, 3 MB
and 3.2 MB and 5 MB lost the connection. After: all five stored, and 24 MB
stored. The page's own guard was then exercised out of the built
`dist/index.html` rather than the source: a file of exactly the limit passes,
one byte more is refused, and the refusal names the file and both sizes.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

An authenticated client can now make the gateway buffer a frame of about
33 MiB rather than 4 MiB. The socket is reachable only with the session cookie
or the token, and the method it feeds already accepted 25 MB, so this raises
what an authorised caller can spend, not who can spend it.

No contract changes: no method gains or loses a field, and no response shape
moves. A page built before this change works against a gateway carrying it, and
the reverse holds with the old ceiling. Rollback is a revert of either commit
independently.

## Related Issues

N/A
2026-09-11 10:37:15 +00:00
Handsome-wzw
e3c876339a fix(*): stop the page hiding a refused delete and a refused upload
## Summary

Two bugs in the served page, both of the same shape: a refusal the surface did
not report as one. They are one MR because they were found in one pass over the
page's write paths and each is a few lines; the commits are separate and can be
read independently.

**1. Bulk delete dropped rows the server had kept.** `DS.sessions.deleteAll`
awaited `session.delete` and counted every call that did not throw as a
removal. A refusal does not throw: the method answers
`{deleted: null, still_on_disk: true}` when the filesystem refused to remove
the file, and that is a resolved promise. The rail dropped the row while the
transcript stayed on disk, so the session came back on the next reload -- the
one thing the note above that function says it must never do. The bulk path now
reads the answer with the predicate the single delete already uses. The
single-delete half was fixed in 3d4075c8, which added `still_on_disk`; the bulk
path was not part of that change.

**2. Attachments over about 3 MB killed the socket.** `fs.upload` accepts
25 MB. The socket carrying it accepted 4 MiB, because
`web.WebSocketResponse(heartbeat=30)` left `max_msg_size` at aiohttp's default.
An upload rides as base64 inside one JSON-RPC frame and base64 costs four bytes
for every three, so the real ceiling was about 2.99 MB of file -- enforced by
closing the connection, before `fs.upload` was ever called. The reader saw a
reconnect rather than a reason, and every other in-flight call on that socket
died with it. The advertised 25 MB limit was unreachable, and so was the
message that states it. The knowledge base feeds the same method, and manuals
and reports clear 3 MB far more often than pasted screenshots do.

The upload limit moves to `raven/rpc/files.py` beside `MAX_VIEW_BYTES`, and
`frame_ceiling_for_upload` derives the transport ceiling from it, so raising one
can no longer leave the other behind. The page also refuses an oversized file
itself: past the ceiling the socket still closes, so the server's own refusal
cannot reach a client that sent one.

**Checked and deliberately not changed.**

- The page carries its own copy of the 25 MB number. No limit rides the wire
  today, and publishing one through `system.hello` is a schema change with
  generated-client churn on both front ends, which this fix does not need. The
  Python constant is named as the source in the comment beside the copy.
- A client that is not the page -- a script, a future surface -- can still cross
  the frame ceiling and lose the socket, because the guard is in the page. The
  transport cannot answer a frame it refused to read; the fix for that is an
  upload route that does not ride the socket at all, which is a larger change
  than this bug warrants.
- A file one byte over the limit is refused by a message that rounds both
  numbers to `25.0 MB`. Showing exact byte counts reads worse for every other
  size, and the server refuses that file too.
- The composer still encodes before it refuses, unlike the knowledge path. Its
  caller in the React island holds the `File`, so moving the check there is an
  island change; the files it stages are screenshots, where the encode is cheap.
- `MAX_VIEW_BYTES` is unchanged. It bounds what the viewer reads back out and
  travels over HTTP, not over the socket.
- `<agent home>/uploads` has no TTL, no count cap and no reclaim path, and this
  change raises what a single upload can put there from about 3 MB to 25 MB.
  Deferred rather than solved here: a reclaim policy is a decision about that
  directory's whole lifetime, and the ceiling it would be sized against is the
  one this MR is what makes reachable. Named so that it reads as a deferral
  rather than as an oversight.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Gates, each run as its own target and its exit code read directly rather than
through a pipe:

- `make lint-python` -- 0. First run reported `raven/rpc/files.py` and
  `tests/test_rpc_transport.py` would be reformatted; `ruff format` applied and
  the rerun is clean.
- `make lint-imports` -- 0
- `make lint-ui` -- 0 (`gen:check` in sync, `type-check` clean)
- `make test-ui` -- 0. 103 files, 1671 tests, 0 failed.
- `make build-ui` -- 0. `dist/index.html` rebuilt.
- `make test-python` -- 0. 21341 passed, 55 skipped, 0 failed, in 389s. No
  failures to reconcile against a clean `origin/main`, because there were none.
- `make check-commits`, `make check-source-language`, `make check-large-files`
  -- all 0.

Mutation tests, each reverted afterwards with `git status` confirmed clean:

- Removing `max_msg_size` from `WebSocketResponse` fails
  `test_the_socket_carries_a_frame_larger_than_aiohttp_would_allow_by_default`
  and nothing else.
- Making `frame_ceiling_for_upload` return the limit without the base64
  expansion fails
  `test_the_frame_ceiling_can_carry_the_largest_upload_the_method_accepts` and
  nothing else.
- Restoring the bare `await` in `deleteAll` fails two of the four new bulk
  cases: `keeps the rows whose files survived their removal` and `keeps a row
  when the server does not carry the field`.

Verified against a running gateway, not only in tests. Uploading through
`/rpc` with the session token, before the change: 1 MB and 2.5 MB stored, 3 MB
and 3.2 MB and 5 MB lost the connection. After: all five stored, and 24 MB
stored. The page's own guard was then exercised out of the built
`dist/index.html` rather than the source: a file of exactly the limit passes,
one byte more is refused, and the refusal names the file and both sizes.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

An authenticated client can now make the gateway buffer a frame of about
33 MiB rather than 4 MiB. The socket is reachable only with the session cookie
or the token, and the method it feeds already accepted 25 MB, so this raises
what an authorised caller can spend, not who can spend it.

No contract changes: no method gains or loses a field, and no response shape
moves. A page built before this change works against a gateway carrying it, and
the reverse holds with the old ceiling. Rollback is a revert of either commit
independently.

## Related Issues

N/A
2026-09-11 10:37:14 +00:00
0xKT
42e6fbdacb feat(agent): let image_generate take a list of prompts
## Summary

`image_generate` could only be handed one prompt, so a caller that needs several
pictures pays for them one call at a time. In a real deck run, eight backdrops
were asked for one after another and the nine calls took 38.6, 35.2, 43.3, 54.2,
81.6, 85.9, 95.4, 103.6 and 108.0 seconds: 645.8 seconds of pure serial waiting
for work that has no order between its pieces.

The deck engine had already answered exactly this for its own generator
(`ppt_generate_image` takes `prompts` and gathers them). This puts that shape on
the host tool every caller shares, rather than leaving it behind a plugin.

Key decisions:

- `prompts[]` carries a full spec per picture: its own `prompt`, and optional
  `filename`, `images`, `aspect_ratio` and `quality`. Whatever an item leaves
  out falls back to the call's top-level value, so a batch that varies only the
  prompt stays short. `model` and `output_dir` stay shared by the whole call --
  one call speaks to one deployment and writes into one directory.
- A top-level `prompt` given alongside `prompts` is generated as the first
  picture, so the two forms compose instead of conflicting.
- One prompt, in whichever form it arrives, is answered in exactly the shape it
  always was, key for key. Existing callers see no change.
- The batch holds at four in flight, the ceiling the deck engine settled on: a
  batch is waited on as a whole, so the cap only has to keep the slowest picture
  from arriving last, and a wider fan-out against one key buys 429s.
- Failures are per picture. The batch is gathered with `return_exceptions=True`,
  so a directory refused for one of eight does not throw the other seven away;
  the answer keeps every path under `paths` for the `message` tool's `media`
  field and says on each result what was made, or why it was not.
- An optional `filename` is sanitised to a bare stem before it names a file. The
  name comes from the model, so a directory inside it is dropped rather than
  honoured, and `output_dir` stays the only thing that decides where a picture
  lands.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

All run from the rebased branch tip, after `uv sync --frozen --all-extras`
(exit 0), with `set -o pipefail` and the exit code checked each time.

- `uv run --frozen pytest tests/test_media_gen_tool.py -q` -- 84 passed, exit 0.
  The file gained 211 lines of tests covering: the single-prompt reply being
  unchanged key for key, top-level values as batch defaults, per-item overrides,
  the top-level prompt landing first, a missing or blank prompt being refused,
  one picture failing while the rest come back, an exception inside one picture
  not taking the batch, concurrency capped at four, and `filename` refusing a
  path traversal.
- `uv run --frozen pytest` over the 15 test files that touch `image_generate` /
  `media_gen` / `ImageGenerateTool` (`test_media_gen_tool`,
  `test_ppt_engine_generate_image_tool`, `test_ppt_engine_plugin`,
  `test_ppt_engine_assembly`, `test_agent_loop_web_tools`, `test_tool_registry_timeout`,
  `test_tool_registry_turn_scope`, `test_tool_capabilities`, `test_tool_search`,
  `test_core_plugin_stack`, `test_plugin_contribution_ledger`, `test_rpc_console`,
  `test_cli_doctor_commands`, `test_provider_auth_method`,
  `test_agents_design_launcher`) -- 592 passed, exit 0.
- `make lint-python` -- exit 0. Ruff check: all checks passed. Ruff format:
  1921 files already formatted.
- `COMMIT_RANGE=origin/main..HEAD make check-source-language` -- exit 0.
- `COMMIT_RANGE=origin/main..HEAD make check-commits` -- exit 0.
- `COMMIT_RANGE=origin/main..HEAD make check-large-files` -- exit 0.

Not verified here: a live batch against the OpenRouter endpoint. The tests drive
the HTTP layer through a stub.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

User-visible behaviour changes:

- The tool schema no longer marks `prompt` as required, because either `prompt`
  or `prompts` will do. A call carrying neither is refused at runtime with
  `nothing to generate: give prompt, or prompts=[{"prompt": ...}, ...]` instead
  of being rejected by the schema, so the model reads a sentence rather than a
  validation error.
- `filename` is new at the top level and per item. A name that carries a
  directory is reduced to its stem; it is not honoured and not an error.
- The tool description now tells the caller to plan every picture and ask for
  them together, so an agent that used to emit N calls will tend to emit one.
  That is the point of the change, but it does shift how the tool gets called.
- Every reply for a single prompt keeps its old shape, so nothing that reads
  `paths` or `success` today needs to change.

Rollback: the change is one commit touching `raven/agent/tools/media_gen.py` and
`tests/test_media_gen_tool.py`. Reverting it restores the single-prompt tool
exactly; no state, config, schema or stored data is carried forward.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-11 10:08:38 +00:00
Tchen
1f9505e839 feat(agent): let image_generate take a list of prompts
## Summary

`image_generate` could only be handed one prompt, so a caller that needs several
pictures pays for them one call at a time. In a real deck run, eight backdrops
were asked for one after another and the nine calls took 38.6, 35.2, 43.3, 54.2,
81.6, 85.9, 95.4, 103.6 and 108.0 seconds: 645.8 seconds of pure serial waiting
for work that has no order between its pieces.

The deck engine had already answered exactly this for its own generator
(`ppt_generate_image` takes `prompts` and gathers them). This puts that shape on
the host tool every caller shares, rather than leaving it behind a plugin.

Key decisions:

- `prompts[]` carries a full spec per picture: its own `prompt`, and optional
  `filename`, `images`, `aspect_ratio` and `quality`. Whatever an item leaves
  out falls back to the call's top-level value, so a batch that varies only the
  prompt stays short. `model` and `output_dir` stay shared by the whole call --
  one call speaks to one deployment and writes into one directory.
- A top-level `prompt` given alongside `prompts` is generated as the first
  picture, so the two forms compose instead of conflicting.
- One prompt, in whichever form it arrives, is answered in exactly the shape it
  always was, key for key. Existing callers see no change.
- The batch holds at four in flight, the ceiling the deck engine settled on: a
  batch is waited on as a whole, so the cap only has to keep the slowest picture
  from arriving last, and a wider fan-out against one key buys 429s.
- Failures are per picture. The batch is gathered with `return_exceptions=True`,
  so a directory refused for one of eight does not throw the other seven away;
  the answer keeps every path under `paths` for the `message` tool's `media`
  field and says on each result what was made, or why it was not.
- An optional `filename` is sanitised to a bare stem before it names a file. The
  name comes from the model, so a directory inside it is dropped rather than
  honoured, and `output_dir` stays the only thing that decides where a picture
  lands.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

All run from the rebased branch tip, after `uv sync --frozen --all-extras`
(exit 0), with `set -o pipefail` and the exit code checked each time.

- `uv run --frozen pytest tests/test_media_gen_tool.py -q` -- 84 passed, exit 0.
  The file gained 211 lines of tests covering: the single-prompt reply being
  unchanged key for key, top-level values as batch defaults, per-item overrides,
  the top-level prompt landing first, a missing or blank prompt being refused,
  one picture failing while the rest come back, an exception inside one picture
  not taking the batch, concurrency capped at four, and `filename` refusing a
  path traversal.
- `uv run --frozen pytest` over the 15 test files that touch `image_generate` /
  `media_gen` / `ImageGenerateTool` (`test_media_gen_tool`,
  `test_ppt_engine_generate_image_tool`, `test_ppt_engine_plugin`,
  `test_ppt_engine_assembly`, `test_agent_loop_web_tools`, `test_tool_registry_timeout`,
  `test_tool_registry_turn_scope`, `test_tool_capabilities`, `test_tool_search`,
  `test_core_plugin_stack`, `test_plugin_contribution_ledger`, `test_rpc_console`,
  `test_cli_doctor_commands`, `test_provider_auth_method`,
  `test_agents_design_launcher`) -- 592 passed, exit 0.
- `make lint-python` -- exit 0. Ruff check: all checks passed. Ruff format:
  1921 files already formatted.
- `COMMIT_RANGE=origin/main..HEAD make check-source-language` -- exit 0.
- `COMMIT_RANGE=origin/main..HEAD make check-commits` -- exit 0.
- `COMMIT_RANGE=origin/main..HEAD make check-large-files` -- exit 0.

Not verified here: a live batch against the OpenRouter endpoint. The tests drive
the HTTP layer through a stub.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

User-visible behaviour changes:

- The tool schema no longer marks `prompt` as required, because either `prompt`
  or `prompts` will do. A call carrying neither is refused at runtime with
  `nothing to generate: give prompt, or prompts=[{"prompt": ...}, ...]` instead
  of being rejected by the schema, so the model reads a sentence rather than a
  validation error.
- `filename` is new at the top level and per item. A name that carries a
  directory is reduced to its stem; it is not honoured and not an error.
- The tool description now tells the caller to plan every picture and ask for
  them together, so an agent that used to emit N calls will tend to emit one.
  That is the point of the change, but it does shift how the tool gets called.
- Every reply for a single prompt keeps its old shape, so nothing that reads
  `paths` or `success` today needs to change.

Rollback: the change is one commit touching `raven/agent/tools/media_gen.py` and
`tests/test_media_gen_tool.py`. Reverting it restores the single-prompt tool
exactly; no state, config, schema or stored data is carried forward.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-11 10:08:37 +00:00
gloryfromca
ba82cee1d5 chore(*): a gate against importing what nobody declared
## Summary

`pyproject.toml` already carried three hand-written pins against a single
failure mode:

```
# transitive dep of litellm -- pinning it directly so the everos
# extractor keeps working if upstream ever drops it.

# transitive dep (via lancedb) -- pinning it directly so routing keeps
# working if upstream ever drops it.

# Reaches us anyway via everos[multimodal] -> everalgo-parser, but that is a
# three-hop transitive path -- pinning it directly so reading an image keeps
# working if any link in that chain drops it.
```

The mode is real and the team was already defending against it one package at a
time. This wires up `deptry`, which is that habit automated, and it found seven
more the manual pass had missed.

**What it catches.** A direct import of a package that only reaches the install
through someone else's dependency table. The day that package drops it, the
install stops carrying it and raven fails at import time, far from the cause.

**This is insurance, not iteration speed.** It adds a gate; it makes nothing
faster. The argument for it is that the failure it prevents is one this
repository has already decided is worth defending against by hand.

### What the 45 reports actually were

Four real, three correct-as-written:

| Package | Sites | Disposition |
|---|---|---|
| `aiohttp` | 15 (3 module-level) | **Declared.** Imported at module level in `rpc/control.py` and both rpc transports, yet it appeared only under `[tool.uv] constraint-dependencies` -- which constrains a version without declaring a dependency |
| `click` | 2 | **Declared.** Used directly, arrives through typer |
| `pyarrow` | 1 | **Declared.** Used directly, arrives through lancedb |
| `pydantic_core` | 1 | **No dependency added.** `pydantic.fields` re-exports the same `PydanticUndefined` object, verified `is`-identical, so the import moves instead |
| `raven_everos` | 24 | **Ignored, reason recorded.** A separate distribution discovered through the `raven.plugins` entry-point group; every import is inside a function for that reason, and declaring it would turn an optional plugin into a hard runtime requirement |
| `requests` | 1 | **Ignored, reason recorded.** Probed under `try: import ... except ImportError` in `channels/errors.py` so its connection errors join the transient set when the host happens to carry it |
| `cryptography` | 1 | **Ignored, reason recorded.** Fallback AES path in the weixin adapter under `with suppress(ImportError)` |

The `aiohttp` case is the one worth a reviewer's attention: a
`constraint-dependencies` entry looks like a declaration in a diff and is not
one. Three module-level imports were relying on something else to bring it.

### Only DEP003 is a gate

`DEP001` and `DEP002` are off. Both are dominated here by the gap between a
distribution name and its import name (`matrix-nio` -> `nio`, `qq-botpy` ->
`botpy`, `python-socketio` -> `socketio`), so they report the same fact twice as
noise. `DEP002` would additionally report the three deliberate pins quoted above
as unused dependencies, which is the opposite of their intent.

Wired as `make lint-deps`, run from `lint` and from CI beside the import
contracts. `lint-imports` was missing from `.PHONY`; both are added.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [x] Other

Other: a dependency gate, three dependency declarations, and one import moved.

## Verification

```
$ uv run --frozen --python 3.12 deptry raven
Success! No dependency issues found.

$ make lint-deps
Success! No dependency issues found.

$ uv run --frozen --python 3.12 --extra dev ruff check --no-cache \
    raven evolver agents plugins-dist tests scripts
All checks passed!

$ uv run --frozen --python 3.12 lint-imports
Contracts: 9 kept, 0 broken.

$ uv run --frozen --python 3.12 pytest tests/ -q -k "update_providers or provider_commands or knowledge or weixin or channels_errors or confirm_injection or cli_dispatch"
496 passed

$ make check-commits COMMIT_RANGE=origin/main..HEAD
exit 0
```

Checked in the failing direction, since a gate never seen red is not known to
work. The first probe was a mistake worth recording: importing `litellm` changed
nothing, because litellm is a declared dependency and the gate was right to stay
quiet. A genuinely undeclared transitive package does trip it:

```
$ echo 'import certifi' >> raven/utils/text.py   # arrives via httpx, declared nowhere
raven/utils/text.py:81:8: DEP003 'certifi' imported but it is a transitive dependency
Found 1 dependency issue.
```

The probe was reverted.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

No user-facing surface changes.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

Three packages move into `[project.dependencies]`. All three are already present
in every install today -- they arrive through typer, lancedb and the aiohttp
constraint -- so this records what is already true rather than adding weight.
The one code change is an import moved to a re-export of the identical object,
verified with `is`.

The risk the gate itself carries is a false negative on a package that is both
optional and genuinely needed; the three ignores are the place that could go
wrong, which is why each carries its reason in the config rather than a bare
name.

Rollback is reverting the one commit. Loosening instead is removing `lint-deps`
from the `lint` target, which leaves the declarations in place.

## Related Issues

N/A
2026-09-11 09:41:53 +00:00
Tchen
5b4cce3f43 feat(ppt-engine): deliver where the user named, one route onto a page, and the ten templates
## Summary

This branch folds four merge requests and one unpushed commit into one, so that `main`
takes one review, one approval and one merge instead of four, and so the eight-file
overlap between them is resolved once. It carries, in this order:

- **!602** `feat/ppt_deliver_to_named_path` -- `ppt_build` takes `deliver_to`; every
  publish writes the deck under `out/` and copies the same measured bytes to the path
  the user named, recording it with `role: "delivery"`. The PDF preview rides along only
  where its own `.pdf` name is free: a file already at that stem is the user's and is
  left alone, and the reply says the preview stayed under `out/`.
- **!628** `feat/ppt_copy_route_and_page_isolation` -- `adapt` is removed, so cloning a
  template page and writing into it with `replace_text` is the only route; a page whose
  block raises loses only itself and the pages after it still build; a draft counts
  against the tier's build cap and one is released at it.
- **!632** `feat/ship_templates_and_soffice_dep` -- the ten bundled deck templates are
  tracked in git with per-file sha256 pins, so a fresh clone can build a deck; the
  LibreOffice dependency is declared and resolved through one resolver.
- **!642** `fix/ppt_copy_route_and_clone_typography` -- a copy-route lookup that finds
  nothing names the near miss before the inventory; a text frame with no paragraph at
  all can be written into; a cloned page's title size is said; `row_type_drift` compares
  the same structural slot across a run's units.
- **D57's other-page clause, rewired.** D57 gave five refusals a "did you mean" hint,
  and one of them also answered "the string you named is on page N of this template".
  That clause needs the file a page was cloned out of, and the only refusal holding it
  was `adapt`'s, which !628 removed. It is restored by reading `PPT_TEMPLATE_SOURCE`,
  which the runner already sets in the author's own process and which this module
  already reads for `bundled()`; the two helpers behind the clause are unchanged.
- **D60** `read the ground under each word, not across the whole box` -- the `unreadable`
  contrast gate also measures the render's own word boxes, as extra regions through the
  same guards, so a two-line block crossed by a narrow bright ridge can no longer elect
  the dark ground and read as legible.

Decisions the union forced, none of which drops a behaviour:

- The publish record is built !602's way, with its delivery `role`, and written the way
  D49 requires -- read back off the file, atomic, and refused rather than silent when
  the record cannot be written.
- `clear_region`, `shapes_in`, `page_box` and the refusal of a bare four-number region
  survive `adapt`'s removal; so do `_heading_rows` and its helpers, which the row
  reading loads even though the call they were written for is gone.
- The author-facing guidance names !628's single route and keeps D59's rule that a
  region must declare which reading it is.
**The decision log carries a numbering collision, on purpose.** It is a set union with
no wording changed, and the numbers are left exactly as each branch wrote them, so a
reviewer meets this rather than discovers it:

- !602 and !628 each wrote a **D37** and a **D38**, about different decisions.
- !628's **D45** is not `main`'s D45; they are unrelated entries under one number.
- !628's **D42** is `main`'s **D45** word for word, so the merged log carries one
  decision under two numbers.

Renumbering touches every cross-reference in the log, and doing that inside a
seventy-commit fold immediately before a merge is how a silent break gets in. The
numbering is being fixed on a separate branch, which carries a script and a test that
fails when a citation points at nothing.

## Type

- [x] Feature
- [x] Fix
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run on an interpreter carrying `python-pptx`, with `PYTHONPATH` at this worktree and its
`plugins-dist/ppt-engine`, confirmed from a neutral working directory that `raven` and
`raven_ppt` resolve out of the worktree and not the host repo.

- `make test-python` (the whole suite, as CI runs it) -- 20826 passed, 49 skipped, and
  26 failures this machine produces on any branch: it has no `/usr/share/zoneinfo`, so
  every cron timezone case raises `ValueError: unknown timezone 'Asia/Shanghai'`. Checked
  out plain `origin/main` in a clean worktree and ran the same two files: the same 6
  failed and 20 errors, the ids identical character for character. Nothing on this branch
  touches cron or timezones. The stale `test_trunks_executor_would_have_dropped_that_output`
  that failed on the previous revision is retired on the new base and its replacement,
  `test_trunks_executor_preserves_output_on_timeout`, was run on its own and passes.
  The 6 failures and 20 errors are all cron timezone tests failing identically on plain
  `main` in a clean worktree, on a host with no `/usr/share/zoneinfo`.
- `make check-commits` -- exit 0.
- `make check-source-language` -- exit 0.
- `make check-large-files` -- exit 0.
- `make lint-imports` -- exit 0, 9 contracts kept, 0 broken.
- `ruff check` and `ruff format --check` over the 80 changed Python files -- exit 0.
- `pytest tests/test_ppt_engine_templates.py tests/test_utils_office.py
  tests/test_cli_doctor_commands.py` -- 97 passed; the ten templates are tracked and
  satisfy their pins, and the soffice resolver finds the local binary.
- The delivery contract, the record read-back, the page isolation and the near miss were
  each run as a measurement rather than asserted; the `unreadable` gate was run through
  the real LibreOffice render chain on a delivered 12-page deck and reports BLOCKING on
  page 12 under one word at 1.6:1.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

User-visible changes: `ppt_build` gains `deliver_to` and delivers outside `out/`;
`adapt` is gone from the author-facing template module, so a program calling it fails to
import -- `title=` and `subtitle=` have no replacement and the heading is named by the
words the template put there; a draft now counts against the tier's build cap; three
gates that nothing measured against being wrong no longer refuse; and the ten templates
are now bytes in the repository rather than a private registry fetch.

Two pieces here are about not destroying something the user has. The delivery
refuses to write its `.pdf` where a file the user already owns sits at the same
stem, rather than replacing it; and the large-file roster moved out of the manifest
inside the tree being checked and into the gate's own source, so a change can no
longer grant itself the exception the gate exists to withhold.

Rollback is per piece: each of the five sources is a contiguous run of commits on this
branch, in the order listed above, so any one of them can be reverted as a range without
touching the others.

- [x] Backward compatibility considered
- [x] Security impact considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-11 05:52:36 +00:00
KevinHuSh
4a4a3fe15b feat(docker): run raven in a docker container and make a first run configurable from the page
## Summary

Two things arrive together because the second is what made the first usable.

**Run raven from a container.** A Dockerfile builds the page, the Python
environment and the engine into one image; `cd docker && docker compose up`
serves it. nginx serves the built page and proxies the RPC socket to the engine
over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a
second container would have no address to dial. Compose publishes on 127.0.0.1,
and the entrypoint prints a one-time sign-in link on boot.

**A first run is no longer a dead end.** The gateway used to refuse to start
until a provider was configured, while being the thing that serves the page
where a provider gets configured. It now starts, says what is missing, and picks
up a key saved in Settings on the next turn without a restart. The onboarding
overlay no longer takes the page either: New chat, the model chip and Send send
the reader to Settings > Models instead, and an empty model list is marked on
the control that fills it.

Also carried: the provider catalogue registry (capabilities, modalities and
context windows per model), the reworked Settings > Models pane, and four fixes
found in review of that work -- a sibling catalogue answering 200 with a
non-catalogue body no longer crashes the whole fetch; adding a tag no longer
erases the label and description already stored for that model; `authenticated`
no longer demands a bearer token from a local provider reached by address; and
registry rows are keyed by the spelling every lookup produces, which recovers
593 rows that no reader could reach.

## Type

- [ ] Fix
- [x] Feature
- [x] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Commands run on the branch head, with their results:

- `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped
- `npx vitest run` in `ui-web` -- 1573 passed (101 files)
- `npx vitest run` in `ui-tui` -- 2037 passed (142 files)
- `uv run ruff check raven scripts tests` and `ruff format --check` -- clean
- `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean
- `npm run lint:rpc` in `ui-tui` -- generated.ts in sync
- `make check-source-language`, `make check-large-files` -- clean

Each of the four review fixes was checked by reverting it and watching a named
test go red, then restoring it.

The container image itself has not been built by CI or by me -- no Docker daemon
was available on the machine this was written on. It has been built and run by
the author of the branch.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

Two deliberate security trade-offs in the container deployment, both documented
in `docker/README.md`:

- nginx rewrites the `Origin` header, because the engine accepts only an Origin
  naming its own loopback port and no browser reaching a published port can send
  one. This drops a defence-in-depth layer. The session cookie is
  SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite
  ignores ports, so another page on localhost counts as same-site.
- `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to
  every page load, so anyone who can reach the published port is signed in. It
  is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any
  deployment reachable from elsewhere. These are two settings that must agree,
  and nothing warns when they do not.

Behaviour changes outside the container: the gateway now starts without a
configured provider rather than exiting 1, and the first-run onboarding overlay
no longer opens by default (`?onboard=1` still forces it).

Rollback is by revert; nothing here migrates state or changes an on-disk format.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-11 03:27:52 +00:00
Handsome-wzw
7b45527b5b fix(tools): an array sent as its own JSON is still an array
## Summary

Reported as "deliver_files is broken". It is not: **the tool never ran.**

Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:

```
Error: Invalid parameters for tool 'deliver_files': files should be array
```

What the model sent was the array, spelled as a string:

```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```

`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.

## Type

- [x] Fix
- [ ] Feature

## The edge of the forgiving is the design

When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.

**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.

**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.

## Not just deliver_files

Six built-in tools declare an array or object parameter and were all exposed:

| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |

The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.

## Verification

```
make lint-python    ruff check + ruff format --check, 1850 files
make lint-imports   9 contracts kept, 0 broken
pytest tests/       19,699 passed, 7 failed (all pre-existing, see below)
```

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string

**End to end, through the real tool**, with the exact payload that failed five times in production:

```
files sent as:            str -> [{"path": ".../poster.png", "title": "..."}]
returned:                 Delivered 1 file(s): poster.png (72 B).
still "should be array":  False
registered in the store:  True
```

Twelve cases, five mutations, each killing at least one:

| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |

The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.

**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.

## Risk

- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert

## Related Issues

N/A

## Swept

`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.

- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.

Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).

## Deliberately not fixed

The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.

## Moved from !594

Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.

Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00
forrestjgq
f01231fef6 fix(cli): overhaul trajectory browser interaction ux (#379)
## Summary

Overhaul the interactive `raven trajectory` browser end to end, fixing
every
issue from real-terminal testing:

- Navigation: every screen opens with a help area under its title (page
  purpose plus "[Key] function" hotkey hints, rendered in a new readable
help color). Esc walks one level up (action -> attempt -> session); on
the
top-level session screen Esc stays put and quitting is Ctrl+C only,
which
prints a single exit notice. Esc inside a running action cancels just
that
action through a dedicated exception at the action boundary, so the back
  sentinel can never reach a conversion or the data layer; a cancelled
  report keeps its pre-confirm bundle-and-pin side effects (the existing
  declined-report contract). Back/Exit menu entries are gone; answered
  prompts erase themselves and navigation is echoed as breadcrumb lines
  with markup-escaped untrusted text.
- Tables: the session list is TITLE | ATTEMPTS | LAST ACTIVITY and the
  attempt list is # | STARTED | TURNS | SPANS | VERDICT | PIN | MERGED |
PREVIEW with green check cells. Time cells keep the year. The layout is
  budgeted per screen build from the live terminal width (row indent, a
safety margin, and inter-column gaps are all part of the math); PREVIEW
  drops as a whole column when it no longer fits, and below the dynamic
  minimum width the tightest layout is kept while the renderer clips
  overlong lines at the terminal edge. Every dynamic cell passes a
  single-line sanitization gate before width math (verdict sidecars only
  guarantee a non-empty string), cells are measured and truncated by
terminal display width (CJK is 2 cells), and fixed column widths always
  fit their headers.
- Space preview: Space on an attempt row prints that attempt's per-turn
input/output previews, collected in the same snapshot scan and sorted by
a fully-stringified key so ordering never follows log source order (the
  table PREVIEW cell derives from the same sorted collection). Any key,
arrow keys included, returns with the cursor kept on the row; the waiter
  passes the terminal's CPR reply through to the renderer instead of
  treating it as a keypress, and Ctrl+C keeps the browser-wide cancel.
- Merge hotkey: with two or more attempts, m/M opens the multi-select
merge
  (advertised in the help line; the old "Merge attempts" menu entry is
gone). The checkbox screen carries its own help line for toggle,
confirm,
  and cancel. A single attempt neither binds nor advertises the key.
- Theme and feel: new success (green check) and help (readable hint
text)
colors in both palettes, all WCAG AA on their backgrounds; a lone ESC is
flushed after 50ms instead of prompt_toolkit's 0.5s default, removing
the
  laggy Esc feel.

The data layer is untouched: changes are confined to
raven/cli/trajectory_browse.py, raven/cli/_theme.py, and their tests.
The
id-taking subcommands (raven trajectory save/list/...) behave as before.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- `COLORTERM=truecolor uv run pytest tests/test_cli_theme.py
tests/test_cli_trajectory_browse.py
tests/test_cli_trajectory_commands.py -q` -> 220 passed
- `COLORTERM=truecolor uv run pytest tests/ --ignore=tests/integration
-q` -> 6917 passed; the 10 failed + 20 errors are the pre-existing
environment-dependent set, identical file-by-file on a clean main
checkout (cron, everos server, config loader, tracing viewer)
- `uv run ruff check` and `uv run ruff format --check` on the four
touched files -> clean
- PTY smoke on a CPR-answering pseudo terminal at 100 and 58 columns:
full flow (preview, merge cancel, action screen, staged Esc backout,
Ctrl+C quit) and the narrow-width degradation contract
- New coverage includes real pipe-input key-binding tests (Esc on all
four prompt kinds, Space/m/M hits, waiter key paths with CPR
pass-through), table budget/clipping boundaries at the computed minimum
width, and corrupt verdict/timestamp corpora

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- The interactive browser's look and key protocol change wholesale (that
is
  the point of this PR): Esc no longer quits from the top level (Ctrl+C
  does), Back/Exit entries are gone, and lists render as tables. The
machine-facing subcommands and the trajectory data layer are untouched.
- Rollback: revert the squash commit; the browser returns to the
previous
  interaction wholesale, no data migration involved.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: 江国庆 <guoqingjiang@deepglint.com>
Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
2026-09-05 23:58:20 +08:00
0xKT
28299e5f0e fix(*): relays on the page lane, live direct chats reported, continuation heads cut
## Summary

Four sub-agent orchestration defects found on 2026-09-08 while running the research
sub-agent from the TUI and the page; all share one root, the host loop not seeing what
its sub-agents are doing right now.

1. A sub-agent's result relay into a page session ran on the gateway spine while the
   page's own turns to that session ran on the RPC spine, so a relay and a user turn ran
   at once: the same tool calls were drawn twice and the result delivered twice, and
   `turn.send`'s one-turn-per-session guard saw neither. The RPC stack now exposes its
   turn scheduler (`RpcStack.turn_scheduler`, surfaced on `PageMount.submit`), and
   `raven.gateway.submit_router.route_submit` hands turns whose conversation carries
   the page prefix (`tui:`) to the page lane and everything else to the gateway lane.
   The sub-agent manager and the deep-research submit both take the routed callable;
   a gateway without a page keeps the gateway submit unchanged.

2. The watch-work judgement (one model call on a turn's first look) was paid on relay
   and sentinel turns, whose "last user message" is the runtime's re-injected report,
   and ran unbounded: one relay turn spent 1229 seconds in it on a reasoning runaway.
   The turn's origin now reaches the loop; a re-injected origin (SUBAGENT, SENTINEL)
   reads as no request, and the judgement is cut at 60 seconds and treated as no
   verdict. CRON and heartbeat turns are deliberately still judged: their text is the
   owner's own instruction.

3. The direct-chat handoff recorded a turn only when it landed, so an instance the user
   was mid-conversation with rendered to the main agent as "no turns yet"; the model
   read that as free and spawned onto it. The handoff now reads the manager's live
   direct turns (`live_direct_turns`) and renders an answering instance with the time
   it began answering (handle lock held) and a line saying it is the user's conversation. The spawn that lands
   behind such a turn no longer writes `running` before it holds the handle lock (it
   stays the `pending` it was announced as), and no longer takes over the instance's
   live view: `activity.watching_instance` registers only when the slot is free and
   pops only what it registered, so the direct chat's steps stay on screen and the
   spawn's appear once it runs.

4. When reasoning ran to the model's output ceiling with no content and the
   empty-response recovery fed it back as a prefill, the continuation opened on the
   rest of the cut thought and that fragment streamed and was stored as the head of the
   answer. `cut_reasoning_head` drops an opening fragment that starts in lowercase or
   whitespace and ends at the first sentence or line boundary within 200 characters;
   `ContinuationGate` holds streamed deltas until the same rule can decide, so what the
   reader sees matches what the session stores. Only a continuation after
   ceiling-cut reasoning (`finish_reason == "length"`) is treated this way.

Decisions worth knowing:

- The `tui:` prefix is spelled in `raven.gateway.submit_router` and as
  `RoutingQuestionBroker`'s default; the layer contract keeps `raven.gateway` from
  importing `raven.rpc`, so a test holds the two equal instead of one importing the
  other.
- The gateway wiring has no import seam, so its test pins the source of the mounted
  block, the same way the neighbouring cron-partition test does.
- The `subagent.status` payload shape is unchanged; only the `running` frame's timing
  moved (after the handle lock). The page's spawn card reads identity from `pending`
  and the record id from `running`, both still present.
- A CJK boundary class in the head-cut regex is spelled as escapes because the
  source-language gate reads the literal marks as non-English source.

Review round 1 (all findings non-blocking, all taken):

- The lane key is now one definition, `raven.spine.conversation_id`; the scheduler's
  method, the rpc spine and the submit router call it, and the router test pins the
  identity and compares against the scheduler's method instead of literals.
- The loop's wiring of the handoff to the manager's live turns has its own test (a bare
  `DirectChatHandoff()` went green before).
- `activity.live_instances_of` is gone: nothing called it.
- The watch-work judgement now reads any exception as no verdict, not only a timeout.
  The enclosing handler already kept the turn alive, but left the verdict unset, so a
  failing judgement was paid again on every path-touching tool call; the comment that
  claimed a raise was handled is corrected.
- A direct chat's start time is stamped once the handle lock is held, not when the slot
  is claimed, so a chat queued behind another holder is not reported as answering.
- The continuation cut deliberately leaves a continuation that opens directly on CJK
  whole (no case, so no mid-sentence signal); the comment and a test now say that the
  CJK stops end an admitted fragment rather than open one.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

```
uv run pytest tests -q -p no:cacheprovider -n 8 --ignore=tests/integration
  -> 7 failed, 19705 passed, 153 skipped. The same 7 fail on a detached origin/main
     worktree through the same interpreter (doctor memories path 1, NAT64 resolution 4,
     design-engine rendering 2): 0 introduced.
uv run pytest tests/test_agent_loop_empty_recovery.py tests/test_watch_work.py \
  tests/test_gateway_submit_router.py tests/test_subagent_activity.py \
  tests/test_subagent_handoff.py tests/test_subagent_direct_chat.py \
  tests/test_rpc_bootstrap.py tests/test_cli_gateway_page.py tests/test_cli_gateway_commands.py
  -> 279 passed (before round 1); round 1 adds 5 tests and removes 1, see the pipeline
uv run ruff check raven tests; uv run ruff format --check raven tests   -> clean
uv run lint-imports                                                       -> 9 kept, 0 broken
make check-source-language                                                -> exit 0
scripts/check_commit_messages.py origin/main..HEAD                        -> exit 0
```

Red-checked (fix reverted, test goes red, fix restored): activity.py alone reverted
-> `a second lane never unseats the first`; the spawn-path hunk of manager.py alone reverted
-> `['pending', 'running'] == ['pending']`; turn_path.py alone reverted -> the continuation
test sees the `Lagg in an token` head. Round 1: the handoff wiring test goes red on a bare
`DirectChatHandoff()`; the queued-direct-chat test goes red with the stamp back on the slot;
the raising-judgement test goes red with the narrow catch (two calls instead of one). The other new tests fail without their change by
construction (missing symbol or signature), which proves less and is noted as such.

Live check of the shipped page: the served `ui-web/dist` bundle booted under happy-dom in
stub and live mode against a gateway running this branch, desk mounted, no console
errors; `subagents.list` and `subagents.instances` answer as before.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed (CHANGELOG, CONTEXT.md
      Handoff Block)

## Risk

- A relay into a page session now waits behind the page's running turn on that session
  instead of running beside it. That is the intended serialisation; the cost is latency
  on a relay while a long page turn runs.
- A judgement that takes over 60 seconds is dropped and the turn runs unwatched for
  that turn only; before, the same turn hung. No configuration knob: the bound is a
  class attribute.
- The head cut only runs on the continuation after a ceiling-cut prefill. A
  continuation whose intended answer begins with a lowercase word and a full stop within
  200 characters would lose that first clause; the test file names the shapes kept.
- Rollback: revert the squash commit. No persisted format changes.

- [x] Security impact considered (no new surface; routing stays in-process)
- [x] Backward compatibility considered (payload shapes unchanged; a gateway without a
      page mount is unaffected)
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-09 06:25:28 +00:00
userName20260323
50ad9377bd fix(*): relays on the page lane, live direct chats reported, continuation heads cut
## Summary

Four sub-agent orchestration defects found on 2026-09-08 while running the research
sub-agent from the TUI and the page; all share one root, the host loop not seeing what
its sub-agents are doing right now.

1. A sub-agent's result relay into a page session ran on the gateway spine while the
   page's own turns to that session ran on the RPC spine, so a relay and a user turn ran
   at once: the same tool calls were drawn twice and the result delivered twice, and
   `turn.send`'s one-turn-per-session guard saw neither. The RPC stack now exposes its
   turn scheduler (`RpcStack.turn_scheduler`, surfaced on `PageMount.submit`), and
   `raven.gateway.submit_router.route_submit` hands turns whose conversation carries
   the page prefix (`tui:`) to the page lane and everything else to the gateway lane.
   The sub-agent manager and the deep-research submit both take the routed callable;
   a gateway without a page keeps the gateway submit unchanged.

2. The watch-work judgement (one model call on a turn's first look) was paid on relay
   and sentinel turns, whose "last user message" is the runtime's re-injected report,
   and ran unbounded: one relay turn spent 1229 seconds in it on a reasoning runaway.
   The turn's origin now reaches the loop; a re-injected origin (SUBAGENT, SENTINEL)
   reads as no request, and the judgement is cut at 60 seconds and treated as no
   verdict. CRON and heartbeat turns are deliberately still judged: their text is the
   owner's own instruction.

3. The direct-chat handoff recorded a turn only when it landed, so an instance the user
   was mid-conversation with rendered to the main agent as "no turns yet"; the model
   read that as free and spawned onto it. The handoff now reads the manager's live
   direct turns (`live_direct_turns`) and renders an answering instance with the time
   it began answering (handle lock held) and a line saying it is the user's conversation. The spawn that lands
   behind such a turn no longer writes `running` before it holds the handle lock (it
   stays the `pending` it was announced as), and no longer takes over the instance's
   live view: `activity.watching_instance` registers only when the slot is free and
   pops only what it registered, so the direct chat's steps stay on screen and the
   spawn's appear once it runs.

4. When reasoning ran to the model's output ceiling with no content and the
   empty-response recovery fed it back as a prefill, the continuation opened on the
   rest of the cut thought and that fragment streamed and was stored as the head of the
   answer. `cut_reasoning_head` drops an opening fragment that starts in lowercase or
   whitespace and ends at the first sentence or line boundary within 200 characters;
   `ContinuationGate` holds streamed deltas until the same rule can decide, so what the
   reader sees matches what the session stores. Only a continuation after
   ceiling-cut reasoning (`finish_reason == "length"`) is treated this way.

Decisions worth knowing:

- The `tui:` prefix is spelled in `raven.gateway.submit_router` and as
  `RoutingQuestionBroker`'s default; the layer contract keeps `raven.gateway` from
  importing `raven.rpc`, so a test holds the two equal instead of one importing the
  other.
- The gateway wiring has no import seam, so its test pins the source of the mounted
  block, the same way the neighbouring cron-partition test does.
- The `subagent.status` payload shape is unchanged; only the `running` frame's timing
  moved (after the handle lock). The page's spawn card reads identity from `pending`
  and the record id from `running`, both still present.
- A CJK boundary class in the head-cut regex is spelled as escapes because the
  source-language gate reads the literal marks as non-English source.

Review round 1 (all findings non-blocking, all taken):

- The lane key is now one definition, `raven.spine.conversation_id`; the scheduler's
  method, the rpc spine and the submit router call it, and the router test pins the
  identity and compares against the scheduler's method instead of literals.
- The loop's wiring of the handoff to the manager's live turns has its own test (a bare
  `DirectChatHandoff()` went green before).
- `activity.live_instances_of` is gone: nothing called it.
- The watch-work judgement now reads any exception as no verdict, not only a timeout.
  The enclosing handler already kept the turn alive, but left the verdict unset, so a
  failing judgement was paid again on every path-touching tool call; the comment that
  claimed a raise was handled is corrected.
- A direct chat's start time is stamped once the handle lock is held, not when the slot
  is claimed, so a chat queued behind another holder is not reported as answering.
- The continuation cut deliberately leaves a continuation that opens directly on CJK
  whole (no case, so no mid-sentence signal); the comment and a test now say that the
  CJK stops end an admitted fragment rather than open one.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

```
uv run pytest tests -q -p no:cacheprovider -n 8 --ignore=tests/integration
  -> 7 failed, 19705 passed, 153 skipped. The same 7 fail on a detached origin/main
     worktree through the same interpreter (doctor memories path 1, NAT64 resolution 4,
     design-engine rendering 2): 0 introduced.
uv run pytest tests/test_agent_loop_empty_recovery.py tests/test_watch_work.py \
  tests/test_gateway_submit_router.py tests/test_subagent_activity.py \
  tests/test_subagent_handoff.py tests/test_subagent_direct_chat.py \
  tests/test_rpc_bootstrap.py tests/test_cli_gateway_page.py tests/test_cli_gateway_commands.py
  -> 279 passed (before round 1); round 1 adds 5 tests and removes 1, see the pipeline
uv run ruff check raven tests; uv run ruff format --check raven tests   -> clean
uv run lint-imports                                                       -> 9 kept, 0 broken
make check-source-language                                                -> exit 0
scripts/check_commit_messages.py origin/main..HEAD                        -> exit 0
```

Red-checked (fix reverted, test goes red, fix restored): activity.py alone reverted
-> `a second lane never unseats the first`; the spawn-path hunk of manager.py alone reverted
-> `['pending', 'running'] == ['pending']`; turn_path.py alone reverted -> the continuation
test sees the `Lagg in an token` head. Round 1: the handoff wiring test goes red on a bare
`DirectChatHandoff()`; the queued-direct-chat test goes red with the stamp back on the slot;
the raising-judgement test goes red with the narrow catch (two calls instead of one). The other new tests fail without their change by
construction (missing symbol or signature), which proves less and is noted as such.

Live check of the shipped page: the served `ui-web/dist` bundle booted under happy-dom in
stub and live mode against a gateway running this branch, desk mounted, no console
errors; `subagents.list` and `subagents.instances` answer as before.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed (CHANGELOG, CONTEXT.md
      Handoff Block)

## Risk

- A relay into a page session now waits behind the page's running turn on that session
  instead of running beside it. That is the intended serialisation; the cost is latency
  on a relay while a long page turn runs.
- A judgement that takes over 60 seconds is dropped and the turn runs unwatched for
  that turn only; before, the same turn hung. No configuration knob: the bound is a
  class attribute.
- The head cut only runs on the continuation after a ceiling-cut prefill. A
  continuation whose intended answer begins with a lowercase word and a full stop within
  200 characters would lose that first clause; the test file names the shapes kept.
- Rollback: revert the squash commit. No persisted format changes.

- [x] Security impact considered (no new surface; routing stays in-process)
- [x] Backward compatibility considered (payload shapes unchanged; a gateway without a
      page mount is unaffected)
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-09 06:25:28 +00:00
daoxize1
5ee8a14f43 feat(rpc): attribute and filter usage by task
## Change description

Attach session and root-task identifiers to usage records. Propagate ownership through ACP child agents and asynchronous tool execution so recorded child usage can be attributed to its parent task.

Add a task selector to the usage settings page, including task titles and an all-tasks view. Prevent stale requests from replacing the selected task's results. Historical records without task identifiers remain visible only in the aggregate view.

Include usage recorded by child agents and the built-in image tool in the owning task. Extend the usage RPC schema and generated clients, and bump the contract version for the root-task field.

Validation:

- `uv run pytest tests/test_token_wise_usage_tracker.py tests/test_rpc_contract_shapes.py tests/test_subagent_acp_mcp_lifecycle.py tests/test_rpc_schema_match.py tests/test_kernel_budget.py tests/test_contracts_two_tier_ledger.py tests/test_media_gen_tool.py -x`: 498 passed.
- `npm --prefix ui-web test -- --run src/features/settings/SettingsPage.test.tsx`: 46 passed.
- `npm --prefix ui-web run type-check`: passed.
- `make lint-python`, `make check-source-language check-large-files`, and `git diff --check`: passed.
- Both repository commit-message checks passed.

## Type of change

- [ ] Bug fix
- [x] New feature
- [ ] Document
- [ ] Others

## Related issues (if there is)

N/A

## Checklists

### Development

- [x] Lint rules pass locally
- [x] Application changes have been tested thoroughly
- [x] Automated tests covering modified code pass

### Security

- [x] Security impact of change has been considered
- [x] Code follows company security practices and guidelines

### Code review

- [x] Merge request has a descriptive title and context useful to a reviewer. Screenshots or screencasts are attached as necessary
- [ ] Merge request linked to JIRA ticket when applicable

No related JIRA ticket was supplied.
2026-09-09 04:15:30 +00:00
daoxize1
135312987b feat(rpc): attribute and filter usage by task
## Change description

Attach session and root-task identifiers to usage records. Propagate ownership through ACP child agents and asynchronous tool execution so recorded child usage can be attributed to its parent task.

Add a task selector to the usage settings page, including task titles and an all-tasks view. Prevent stale requests from replacing the selected task's results. Historical records without task identifiers remain visible only in the aggregate view.

Include usage recorded by child agents and the built-in image tool in the owning task. Extend the usage RPC schema and generated clients, and bump the contract version for the root-task field.

Validation:

- `uv run pytest tests/test_token_wise_usage_tracker.py tests/test_rpc_contract_shapes.py tests/test_subagent_acp_mcp_lifecycle.py tests/test_rpc_schema_match.py tests/test_kernel_budget.py tests/test_contracts_two_tier_ledger.py tests/test_media_gen_tool.py -x`: 498 passed.
- `npm --prefix ui-web test -- --run src/features/settings/SettingsPage.test.tsx`: 46 passed.
- `npm --prefix ui-web run type-check`: passed.
- `make lint-python`, `make check-source-language check-large-files`, and `git diff --check`: passed.
- Both repository commit-message checks passed.

## Type of change

- [ ] Bug fix
- [x] New feature
- [ ] Document
- [ ] Others

## Related issues (if there is)

N/A

## Checklists

### Development

- [x] Lint rules pass locally
- [x] Application changes have been tested thoroughly
- [x] Automated tests covering modified code pass

### Security

- [x] Security impact of change has been considered
- [x] Code follows company security practices and guidelines

### Code review

- [x] Merge request has a descriptive title and context useful to a reviewer. Screenshots or screencasts are attached as necessary
- [ ] Merge request linked to JIRA ticket when applicable

No related JIRA ticket was supplied.
2026-09-09 04:15:30 +00:00
0xKT
231b270dd7 refactor(*): the vendored subagents tree retires and agents/ is the one home
## Summary

Retires the vendored fork tree (`subagents/`, ~7,500 files) and makes `agents/` the one home for the five agent products. The products were already thin launchers that render a config and exec the installed raven's own `raven acp`; this PR removes the frozen fork checkouts they replaced, migrates every consumer, and keeps the A/B verification power as byte snapshots.

Key decisions:

- **A side becomes a record.** The fork trees are deleted; the files the parity guards compare against live on as byte snapshots under `tests/fixtures/vendored_fork/` (added to ruff's exclude so a record can never be reformatted). The research flow's oracle is the fork's frozen measurement (`research_parity_probe.json`), a constant by construction.
- **Config floors v6 and v7.** Stored roster rows that pointed into the retired tree are migrated in place (floor six): rows whose folder maps to a shipped product are re-aimed at `agents/`, fork-era rows that cannot be re-aimed are left untouched and reported, unparseable rows get a fix-by-hand notice. Verdict-driven, path-root anchored, with a stamped sidecar so each pass runs once.
- **The research product reclaims its name (floor seven).** The `-NG` suffix existed only to coexist with the fork's own row, so it retires with the fork: registered `Raven-Research-NG` rows rename to `Raven-Research` on load; if a fork-era row still holds the name, the advisory repeats and the stamp holds at floor six, so the rename completes on a later load once the name frees (never two rows under one name). Display name only -- the machine id `raven-research-ng` (state root, ACP home, everos identity, the `RESEARCH_NG_*` variables) is a stable key and stays. The v6 name gate maps the research folder to None (never re-aim): the fork's research rows were cli-kind on a different flow, and after the rename a name match would select exactly the rows the gate exists to refuse.
- **Guards retire with their subject.** The vendored-invariant and vendored-subagents guard suites, their Makefile/pre-commit/CI hooks, and the fork-era install stage in `install.sh` are removed. The external-consumer tripwire now derives the promised surface from the five product installers and the charter's uninstall recipe (`add_third_party_subagent` and `remove_third_party_subagent` today) instead of restating a list that goes stale; the fork-era pin count (eight names) retired with its consumers, and `get_third_party_subagents` stays only as a courtesy to fork-era installer copies already carried into raven homes.
- **Wheel and sdist slim down.** The wheel force-includes `agents/` file-by-file from the git index (filled-in `.env` files are refused). The sdist shrinks 93.2 MB -> 2.76 MB (-97%).
- **Docs speak one era.** AGENTS.md, CONTEXT.md, README, CHANGELOG, the demo READMEs, and the RPC field descriptions (models.py -> openrpc.json -> both generated clients) describe the product tree in the present and the fork in the past.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [x] Refactor
- [ ] Other

## Verification

- Controlled A/B on a clean bench, same machine and env fingerprint, at the current pair: base 19,938 passed / 137 skipped / 0 failed vs branch 19,796 / 137 / 0, skip roster identical line by line. The -142 delta equals the collection delta and reconciles file by file: -159 across the twelve retired or trimmed test files, +9 floor-six migration tests, +8 floor-seven tests. The branch diff is fingerprint-stable across rebases (`git patch-id --stable` verified pairwise), so the reconciliation transfers to later bases. A second fingerprint (dev worktree env) shows 19,555 / 148 / 2 at the final tip; the 2 are `test_design_engine_rendering_outcome` renders hitting a local playwright-home quirk -- on the clean bench both sides skip them identically ("no system chromium"), and the branch does not touch that file.
- `uv run pytest tests/integration/test_research_subagent_smoke.py -m integration`: 2 passed (the smoke now drives the migrated product launcher end to end).
- `make lint-python` (ruff check + format), `make lint-imports` (tail line: `Contracts: 9 kept, 0 broken.` -- the ninth contract landed on main mid-branch), `make check-large-files`: all green.
- `npm run lint:rpc --prefix ui-tui` and `npm run gen:check --prefix ui-web`: generated clients in sync with `rpc-schema/openrpc.json`.
- `bash -n install.sh`: clean.
- Wheel built from the branch, installed into a clean venv: onboarding discovers the five products, and a filled-in `.env` placed in the tree fails the build (secrets refusal probed).
- Real-config upgrade rehearsal on a genuine floor-5 home config (owner present): the migration stamped the sidecar to 6, rewrote nothing (the five product rows were already registered against `agents/`; the one fork-era orphan row got its advisory notice and was left untouched), a second load was silent, and the upgraded config served a live probe plus one full one-shot agent turn. The floor-seven rehearsal on the same config renamed its one transition row (a one-line diff against the byte backup), stamped seven, was silent on the second load, and left the machine ids and the row's command untouched.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- User-visible: on first load of a pre-v6 config, roster rows pointing into the retired tree are re-aimed at the product tree, with a one-line `Migrated` notice per changed row and a cleanup hint for the stale on-disk copy. Rows the migration cannot decide are reported, never rewritten. The floor-seven rename tells once per changed row too; only a blocked rename repeats its advisory, deliberately, until the blocking row is gone. While the block holds, the roster shows the discovered `Raven-Research` row beside the stored `Raven-Research-NG` row -- same launcher, same state root, no state split -- and converges to one row once the name frees.
- Rollback: revert the squash commit. The fork trees return from git history, and a config the migration already re-aimed keeps working on both sides of the revert: the wheel carries `agents/` before and after (that packaging landed separately), so no config rollback is needed. Rows the migration could not decide were never rewritten, and the version sidecar is additive.
- Packaging: a wheel built from an sdist (what a combined `uv build` does) carries no agent products by design -- the sdist has no `.git` for the hook's tracked-files read; the hook warns loudly, and release.yml builds the wheel directly from the checkout and asserts the products are present. Release artifacts are unaffected; do not publish a combined-build wheel.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

BREAKING CHANGE: the `subagents/` source tree is gone. Installs never consumed it directly (the wheel carries `raven/agents`), but any local tooling that read `subagents/` paths must move to `agents/` or the snapshots under `tests/fixtures/vendored_fork/`.
2026-09-09 03:59:19 +00:00
0xKT
985b220b0e feat(agent): a failing dag node can hand back a new plan
## Summary

A DAG node that reports it could not accomplish its task suspends and asks the agent what
to do. Until now the answer could only be `continue` (send a correction and let the node
retry) or `abandon` (fail it and skip its dependents). Both keep the graph the agent
committed to at dispatch. When the plan itself is what turned out to be wrong, neither
helps: the agent's only recourse was to cancel the run and submit a fresh graph, which
throws away every node that had already succeeded.

This adds a third decision, `replan`. The agent hands `resolve_dag_node` a replacement node
list, the run being answered winds down, and a new run starts from that list, referencing
the wound-down run's completed nodes by id.

**Chained runs, not an in-place splice.** Node ids are claimed per conversation at
`store.init`, and every read-back path takes a run's structure from its `graph.json`.
Splicing would have to claim ids mid-run, rewrite that file into a cumulative graph, and
keep dropped nodes in it so they do not vanish from the record. Chaining instead lands on
the cross-run reference machinery that already exists for naming an earlier run's completed
node, and leaves both runs individually readable.

Decisions a later reader will want the reasons for:

- **A completed node is referenced, never re-declared.** Name it in `depends_on` and read it
  with `{{ <id>.output }}`. Its id belongs to the old run permanently, so re-declaring it is
  refused.
- **The adjudicated node is given up on.** No attempt budget crosses runs, because carrying
  one would require node identity to span ids that must differ.
- **In-flight nodes are cancelled, not drained.** A replan is a statement that the remaining
  plan is wrong; letting nodes finish work the new graph does not want spends dispatches on
  output nobody reads.
- **A replan ends the adjudication round for every node suspended alongside it**, answered or
  not. Several nodes can be waiting for the agent at once, and a sibling's answer can no longer
  change a graph that is being replaced. Waiting for one is not merely wasted: on a run bound to
  a live turn the wait carries no deadline, and the lane that would answer the sibling is the
  same one parked in `resolve_dag_node` waiting for this run to finalize, so both wait without a
  bound. Nodes left unanswered are recorded as superseded rather than as having timed out --
  they were never asked.
- **The successor passes the same preflight a submitted graph does** -- the confirmation gate
  and the hourly dispatch charge included. A replan therefore costs one unit of the same
  hourly quota, and if the original graph was confirmed by the user, so is its successor.
- **No replan cap.** The hourly dispatch quota is the only mechanical bound; see Risk.

Persistence keeps the two runs separate. The successor is saved as its own run with its own
directory and `graph.json`; the run it replaces gains a reserved `replan` key on its own
`graph.json` recording the decision, the successor's id, and whether it started. An
interrupted hand-off is recorded as started `false` with a reason, so a reader never sees
nodes pointing at a successor the file does not admit was attempted.

Both front ends learn about the link through a new `dag.run_replanned` wire event. It is
emitted the moment the desk accepts the decision, before the answered run is awaited. That
timing is forced by the two front ends disagreeing: one pins a run's state by id and keeps
it, so a link arriving at any time lands; the other resolves a card through a live index
keyed by run id and drops that entry when the run completes. An event emitted after the
successor starts arrives after that drop and resolves to nothing, while a unit test that
never plays those frames passes -- code that looks done and does nothing. The durable record
still waits for the outcome: a live hint may be optimistic, an audit may not.

Design and the decisions behind it are in `docs/specs/2026-09-04-dag-replan-design.md`,
including a "deliberately out of scope" section.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run on the rebased tree, at the head this description is written against.

- `make test-python` -- 18711 passed, 1 failed, 67 skipped, in 188.18s.
  The one failure is `test_cli_doctor_commands.py::test_doctor_answers_where_the_memories_are`,
  which this branch does not touch and does not cause; see Risk for how that was established.
- `make lint-python` -- clean (ruff check and ruff format --check).
- `make check-source-language` -- clean.
- `make check-large-files` -- clean.
- `npm run type-check --prefix ui-tui` -- clean.
- `npm test --prefix ui-tui -- dagRun.test` -- 24 passed.
- `npm test --prefix ui-web -- src/features/transcript` -- 130 passed.
- `npm run lint:i18n --prefix ui-tui` -- generated catalogue up to date.
- The four suites exercising the changed code, run together under xdist as CI runs them:
  438 passed.

Every new guard was sabotage-verified rather than assumed: the change it exists to catch was
reintroduced, the test was watched to go red with the assertion it was written for, and the
source was restored byte for byte. That includes the three defects found only by reviewing the
branch whole, after every individual task had passed its own review -- a replan landing while
the runner was parked was recorded as an abandonment, a foreground run whose turn had been
released announced its own wind-down as a raw dataclass, and a replan left the round waiting on
the siblings it had just made irrelevant.

The last of those has no deadline in the code under test to end it, so its regression drives the
real wait under `asyncio.timeout` and fails on the clock rather than hanging the suite: without
the guard, `TimeoutError` in 5.98s.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

**Behaviour change, opt-in.** `resolve_dag_node` gains a `replan` value and a `nodes`
parameter. The two existing decisions are untouched, and a caller that never sends `replan`
sees no change. Rolling back is reverting the branch: nothing migrates, and a `graph.json`
carrying a `replan` key stays readable to every reader of that file, all of which read named
keys rather than iterating.

**A replan chain is bounded only by the hourly dispatch quota.** That was deliberate -- a
count cap would have to guess how many corrections a task deserves. Each hop costs one unit
of the same quota a fresh graph costs, and each writes its own run directory, so a long
chain leaves one directory per hop with no TTL and no reclaim path. That matches how run
directories already accumulate; it is not made worse per run, only reachable more often.

**Known gaps, deliberately not closed here.** The bound foreground lane's silence is now an
explicit check rather than an eviction accident, but has no test -- reaching it needs a
released tray and a foreground turn to release it. The link record is written after the
successor is submitted or refused as invalid, so a cancellation raised inside the submission
still escapes both guards; the remaining window is documented where it happens.
`prepare_replan` refuses a graph that fails validation, but a backend IO error or a
non-object `graph.json` propagates rather than refusing cleanly -- no state is corrupted,
since that runs before the node is answered and the node stays waiting. The successor does
not draw under the tool row that spawned it in the web UI, for the reason given in the spec's
out-of-scope section; the link event still lands, so the replan itself is visible.

**The one failing test is not this branch's.** Established two ways rather than asserted:
every input it reads is byte-identical to the base
(`git diff --stat origin/main...HEAD -- raven/cli/ raven/config/ tests/test_cli_doctor_commands.py`
is empty), and the failure is console width plus path length, not code. Under xdist at the
default width it fails; serially at the default width it passes; under xdist with
`COLUMNS=200` it passes. xdist inserts a worker directory into the temp path and narrows the
captured console, and between them rich wraps the path across the substring the assertion
looks for.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-06 12:01:03 +00:00
LivXue
34cabbec7b feat(agent): a failing dag node can hand back a new plan
## Summary

A DAG node that reports it could not accomplish its task suspends and asks the agent what
to do. Until now the answer could only be `continue` (send a correction and let the node
retry) or `abandon` (fail it and skip its dependents). Both keep the graph the agent
committed to at dispatch. When the plan itself is what turned out to be wrong, neither
helps: the agent's only recourse was to cancel the run and submit a fresh graph, which
throws away every node that had already succeeded.

This adds a third decision, `replan`. The agent hands `resolve_dag_node` a replacement node
list, the run being answered winds down, and a new run starts from that list, referencing
the wound-down run's completed nodes by id.

**Chained runs, not an in-place splice.** Node ids are claimed per conversation at
`store.init`, and every read-back path takes a run's structure from its `graph.json`.
Splicing would have to claim ids mid-run, rewrite that file into a cumulative graph, and
keep dropped nodes in it so they do not vanish from the record. Chaining instead lands on
the cross-run reference machinery that already exists for naming an earlier run's completed
node, and leaves both runs individually readable.

Decisions a later reader will want the reasons for:

- **A completed node is referenced, never re-declared.** Name it in `depends_on` and read it
  with `{{ <id>.output }}`. Its id belongs to the old run permanently, so re-declaring it is
  refused.
- **The adjudicated node is given up on.** No attempt budget crosses runs, because carrying
  one would require node identity to span ids that must differ.
- **In-flight nodes are cancelled, not drained.** A replan is a statement that the remaining
  plan is wrong; letting nodes finish work the new graph does not want spends dispatches on
  output nobody reads.
- **A replan ends the adjudication round for every node suspended alongside it**, answered or
  not. Several nodes can be waiting for the agent at once, and a sibling's answer can no longer
  change a graph that is being replaced. Waiting for one is not merely wasted: on a run bound to
  a live turn the wait carries no deadline, and the lane that would answer the sibling is the
  same one parked in `resolve_dag_node` waiting for this run to finalize, so both wait without a
  bound. Nodes left unanswered are recorded as superseded rather than as having timed out --
  they were never asked.
- **The successor passes the same preflight a submitted graph does** -- the confirmation gate
  and the hourly dispatch charge included. A replan therefore costs one unit of the same
  hourly quota, and if the original graph was confirmed by the user, so is its successor.
- **No replan cap.** The hourly dispatch quota is the only mechanical bound; see Risk.

Persistence keeps the two runs separate. The successor is saved as its own run with its own
directory and `graph.json`; the run it replaces gains a reserved `replan` key on its own
`graph.json` recording the decision, the successor's id, and whether it started. An
interrupted hand-off is recorded as started `false` with a reason, so a reader never sees
nodes pointing at a successor the file does not admit was attempted.

Both front ends learn about the link through a new `dag.run_replanned` wire event. It is
emitted the moment the desk accepts the decision, before the answered run is awaited. That
timing is forced by the two front ends disagreeing: one pins a run's state by id and keeps
it, so a link arriving at any time lands; the other resolves a card through a live index
keyed by run id and drops that entry when the run completes. An event emitted after the
successor starts arrives after that drop and resolves to nothing, while a unit test that
never plays those frames passes -- code that looks done and does nothing. The durable record
still waits for the outcome: a live hint may be optimistic, an audit may not.

Design and the decisions behind it are in `docs/specs/2026-09-04-dag-replan-design.md`,
including a "deliberately out of scope" section.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run on the rebased tree, at the head this description is written against.

- `make test-python` -- 18711 passed, 1 failed, 67 skipped, in 188.18s.
  The one failure is `test_cli_doctor_commands.py::test_doctor_answers_where_the_memories_are`,
  which this branch does not touch and does not cause; see Risk for how that was established.
- `make lint-python` -- clean (ruff check and ruff format --check).
- `make check-source-language` -- clean.
- `make check-large-files` -- clean.
- `npm run type-check --prefix ui-tui` -- clean.
- `npm test --prefix ui-tui -- dagRun.test` -- 24 passed.
- `npm test --prefix ui-web -- src/features/transcript` -- 130 passed.
- `npm run lint:i18n --prefix ui-tui` -- generated catalogue up to date.
- The four suites exercising the changed code, run together under xdist as CI runs them:
  438 passed.

Every new guard was sabotage-verified rather than assumed: the change it exists to catch was
reintroduced, the test was watched to go red with the assertion it was written for, and the
source was restored byte for byte. That includes the three defects found only by reviewing the
branch whole, after every individual task had passed its own review -- a replan landing while
the runner was parked was recorded as an abandonment, a foreground run whose turn had been
released announced its own wind-down as a raw dataclass, and a replan left the round waiting on
the siblings it had just made irrelevant.

The last of those has no deadline in the code under test to end it, so its regression drives the
real wait under `asyncio.timeout` and fails on the clock rather than hanging the suite: without
the guard, `TimeoutError` in 5.98s.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

**Behaviour change, opt-in.** `resolve_dag_node` gains a `replan` value and a `nodes`
parameter. The two existing decisions are untouched, and a caller that never sends `replan`
sees no change. Rolling back is reverting the branch: nothing migrates, and a `graph.json`
carrying a `replan` key stays readable to every reader of that file, all of which read named
keys rather than iterating.

**A replan chain is bounded only by the hourly dispatch quota.** That was deliberate -- a
count cap would have to guess how many corrections a task deserves. Each hop costs one unit
of the same quota a fresh graph costs, and each writes its own run directory, so a long
chain leaves one directory per hop with no TTL and no reclaim path. That matches how run
directories already accumulate; it is not made worse per run, only reachable more often.

**Known gaps, deliberately not closed here.** The bound foreground lane's silence is now an
explicit check rather than an eviction accident, but has no test -- reaching it needs a
released tray and a foreground turn to release it. The link record is written after the
successor is submitted or refused as invalid, so a cancellation raised inside the submission
still escapes both guards; the remaining window is documented where it happens.
`prepare_replan` refuses a graph that fails validation, but a backend IO error or a
non-object `graph.json` propagates rather than refusing cleanly -- no state is corrupted,
since that runs before the node is answered and the node stays waiting. The successor does
not draw under the tool row that spawned it in the web UI, for the reason given in the spec's
out-of-scope section; the link event still lands, so the replan itself is visible.

**The one failing test is not this branch's.** Established two ways rather than asserted:
every input it reads is byte-identical to the base
(`git diff --stat origin/main...HEAD -- raven/cli/ raven/config/ tests/test_cli_doctor_commands.py`
is empty), and the failure is console width plus path length, not code. Under xdist at the
default width it fails; serially at the default width it passes; under xdist with
`COLUMNS=200` it passes. xdist inserts a worker directory into the temp path and narrows the
captured console, and between them rich wraps the path across the substring the assertion
looks for.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-06 12:01:03 +00:00
Tchen
9a87c73b27 feat(agents): port the deck engine's layout work and host its templates in the registry
## Summary

Brings the deck work done on the vendored fork onto this branch the way this branch is built: the shipping engine is the `ppt-engine` plugin (`plugins-dist/ppt-engine/raven_ppt`), the product wrapper lives under `agents/raven-ppt`, and the fork's changes are ported into the plugin file by file with the `raven.ppt` -> `raven_ppt` import rewrite. No plugin, hook or layer boundary changes.

What the engine gains: layout helpers the author kept re-implementing (`add_unit` / `remove_unit` grow and shrink a template's repeated units and reflow the row, column or grid; `clone_shape` and `swap_icon` re-draw a unit's mark; `fill` counts units in reading order and keeps a numbered tile's number); cross-template borrowing through a curated `REFERENCE_PAGES` set any deck may adapt with `adapt(prs, prototype(bundled('<template>'), N), ...)`, with the outline's `borrowed` field and the adherence gates aware of it; measurements that read a tinted card as a card (`is_filled`) and report a body nothing divides (`undivided_body`); `backdrop()` for a generated picture behind a page; and eight bundled templates instead of twelve (four dropped for looks).

The vendored fork under `subagents/` is the sealed A side of the parity comparison and is not touched: this tree is byte-identical to `main` there, and every product-side delta against the sealed fork -- the model, the 1800s `llmCallTimeout`, the retry ladder, `llmRetryAfterOutput`, the held-out trunk tools -- is ledgered in `tests/test_agents_ppt_launcher.py`.

The templates are no longer read out of the fork tree. They ride as the generic package `ppt-templates/2026.09.03` in this project's GitLab package registry: `templates.manifest.json` names that endpoint and pins each file by sha256, and `fetch_templates.py` sends `PRIVATE-TOKEN` from `GITLAB_TOKEN` or `JOB-TOKEN` from `CI_JOB_TOKEN` because the project is private. `make fetch-templates` wraps it and `agents/README.md` documents the step, so a fresh clone gets the bytes the wheel bundles. `uv.lock` resolves `matplotlib`, which the plugin declares and the lock had not.

Stacked on !480 (the host seams) -- review that first.

## Type

- [x] Feature

## Verification

`uv run pytest tests -q -k "ppt_engine or agents_ppt_launcher or vendored or hatch_build or contribution_ledger or runtime_binding"` -> 3603 passed, 9 skipped on the stack. `python3 scripts/check_vendored_subagents.py` passes without `--update`; `scripts/check_large_files.py main..HEAD` exits 0; `fetch_templates.py` against the registry verified all 8 pins; `uv sync --frozen --extra dev --dev` into a fresh venv imports `raven_ppt` and `matplotlib` with both plugin entry points present. `uv run ruff check` / `ruff format --check` clean.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

The template payload now needs one fetch after cloning (`make fetch-templates`, `GITLAB_TOKEN` set); without it the engine's catalogue is empty and a deck task has to bring its own template, which is the engine's own shipped degradation. Four bundled templates are gone, so a task naming one of them by stem no longer finds it. Rollback is reverting this commit; the sealed fork is unchanged either way.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-06 11:28:56 +00:00
0xKT
3a7cc17aeb fix(*): raven-research takes the host web vendor and key, and the ask_user gate closes
## Summary

Three strands of raven-research work, held together because they all touch the vendored fork and so share one `subagents/TREE_HASHES` line.

**Web search and fetch route through the vendor the deployment picked.** Both tools in the research-flow plugin were pinned to Serper and Jina, so a host on Tavily or Brave still needed a Serper key for the pair. They now carry the kernel's vendor tables (seven search backends, five fetch) with per-vendor request encoding and response normalisation. Serper and Jina wire traffic is byte-identical to before. A vendor that publishes no result offset declares it up front (`SearchProviderSpec.paginates`), so the saturation rule never escalates to a page rung the vendor cannot answer.

The launcher resolves the vendor first and mirrors the SELECTED vendor's key into the plugin slice, never a key by fixed path: a mismatched pair sends one vendor's credential to another's endpoint and every call comes back unauthorized. Every vendor gets an env slot at the same path the host holds it, inherited from the host config when unset, so a deployment configures a vendor once, centrally. The two legacy vendors are also read from the vendor slot, where a host configured after the vendor table landed writes them. `require_search` follows the selection rather than the default.

**The research modes take the host's effort vocabulary**: `deep` becomes `high`, `ultra` becomes `max`, and `medium` is the baseline. This is what lets the companion MR's host tier reach this agent by name.

**The ask_user gate closes after a hook rollback**, and host questions take the conversation lock like every other route, including the `direct_ask` grant. The digest follows its session's mode, a typo in a mode overlay refuses to launch rather than running the base value under the overlay's label, and the reviewer records which model judged.

Three defects found while building it, each with the test that pins it:

- **A credential reached disk.** The retry rows of the search and fetch ledger recorded `str(exc)` for a status error, and httpx puts the whole request URL in that text. SerpApi authenticates by query parameter, so a 429 or 503 wrote the key into the JSONL file. The redaction the search and fetch rows already had is now one function, `_error_text`, used by both writers, in the plugin and in the vendored copy. The regression test drives a retryable status and reads the ledger file off disk; the previous test used 401, which is not retryable and so never reached the leaking write. The key never reached the model or the answer: the appendix partitions on `search` / `fetch` / `verify` / `force_finalize` and reads nothing from a retry row.
- **A keyless fetch backend refused every call.** The kernel's `web_fetch` degrades a keyed backend with no resolvable key to Jina, out loud. The plugin's replacement did not, and it REPLACES the kernel's, so `fetch.provider = tavily` with no Tavily key made a research run the single place that config could not read a page, while a plain raven on it read the page fine. The launcher itself produces that state: it writes the provider unconditionally and the key only when one resolves. Fixed at the build site, where the kernel resolves it, and a test asserts both implementations answer identically for every vendor.

  Reported by chandler.zhang and completed in `ac7d364c`: the degraded backend was still handed the key of the vendor it replaced, which is absent by definition, so a deployment holding a Jina key read pages authenticated through raven's built-in fetch and anonymously through the plugin -- losing the configured quota on the very fallback the rule exists to match. The kernel looks the key up a second time under the EFFECTIVE provider; the slice carries one key per role and no vendor table, so the launcher now mirrors the default reader's key beside the selected vendor's and the build picks by destination, a lookup keyed on the provider the tool will actually call. It is mirrored whenever the selection is not already the default rather than only when the fallback will fire, because the rule that decides that lives in `effective_provider` and a launcher second-guessing it would be the copy that drifts.
- **A typo in a vendor name refused to launch for the wrong reason.** The plugin degrades an unknown vendor to its default, which is right for a slice nothing validates. Carried through the launcher, the same typo had `require_search` demand a key for a vendor that does not exist while the working Serper key sat in the config -- sending the operator to set a credential they already had. The launcher now refuses naming the name and the valid list.

`.env.example` and the plugin docstrings were still describing the fixed Serper and Jina pair; both now cover the provider switch and the per-vendor key slots. The plugin MANIFEST was missed in that pass and is corrected in `18ce29df`, also on chandler.zhang's report: it still called `search.apiKey` a Serper key and `fetch.apiKey` a Jina reader key, and said web_search declines "when no Serper key resolves". A direct plugin user reaching it without the launcher could set `search.provider = tavily` and supply a Serper credential, which this code then sends to Tavily's endpoint.

`subagents/TREE_HASHES` is recorded once, in the final commit, rather than per commit: the hash covers every tracked byte under the fork, so an intermediate value goes stale the moment the next edit or a rebase lands.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run on this branch at its current head:

- `uv run pytest` -- 5 failed, 18545 passed, 69 skipped. The five are this host's pre-existing baseline, identical on `origin/main`: `test_cli_doctor_commands.py::test_doctor_answers_where_the_memories_are` and four `test_security_network.py` NAT64 parameters. 0 introduced.
- `make lint-python` -- passed, 1787 files already formatted.
- `uv run lint-imports` -- 8 contracts kept, 0 broken.
- `make check-vendored-subagents` -- exit 0, hash recorded.
- `make check-vendored-invariants`, `make check-large-files`, `make lint-ui`, `make lint-tui`, `scripts/check_commit_messages.py origin/main..HEAD` -- all exit 0.
- The vendored fork's own suite, run with the fork on `PYTHONPATH` so `raven` resolves to its copy: 19 failed, 6223 passed, 4 errors. Reverting only this branch's change to the fork gives the same 19 failures and 4 errors, all in three files that never import the changed module, so 0 introduced. Those failures are an artifact of running the fork against the trunk's venv, not the fork's own.

Reverting each fix and confirming the matching test fails: the ledger redaction (the 429 case red, the 401 case correctly still green), the fetch degradation (4 keyed vendors red, Jina correctly unaffected), the vendor-name refusal (2 red), and the slice key following the selected vendor (2 red).

Three more for the review fix: the build passing the selected vendor's key to the reader that replaced it (1 red), the launcher not mirroring the fallback key (1 red), and the launcher mirroring it for the default vendor too (1 red, the case where a second copy of the same key is one more place for the pair to disagree).

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

Security: a credential stopped being written to disk. Any research-flow ledger file written before this change by a deployment on SerpApi may contain the key in a `search_retry` or `fetch_retry` row. Those files live under the plugin's state root, one per turn, and are unlinked at the end of the turn that wrote them; a process killed mid-turn leaves one behind. Treat a SerpApi key used on an affected build as exposed and rotate it, then check for leftover files.

Other user-visible changes:

- A research deployment that selected a keyed fetch vendor without its key now reads pages through Jina instead of failing every fetch, and does so on the host's configured Jina key rather than anonymously. Behaviour improves; the log says which vendor is serving.
- That fallback key now travels in the rendered plugin slice as `fetch.fallbackApiKey`, beside the selected vendor's key, wherever the rendered config already holds credentials. It is only ever handed to the default reader: the build looks the key up by destination, so a provider neither key belongs to gets none.
- An unknown vendor name in `search.provider` or `fetch.provider` now refuses to launch. A deployment running on a typo today would have been failing its search calls anyway.
- The research mode ids change: `deep` becomes `high` and `ultra` becomes `max`. A caller that names `deep` by id gets the agent's default instead.
- Ledger `*_retry` rows record `HTTPStatusError: HTTP <status>` where they recorded the full httpx message. The one offline reader, the fork's `bench/rollout.py`, buckets by status either way and already read the search and fetch rows in that shape.

Rollback: revert the commits on this branch. `TREE_HASHES` is recorded in its own final commit, so reverting the set restores the recorded hash with the tree.

## Related Issues

N/A

<!-- The repo guide says to leave this trailer to the commits, on the premise that
     the platform collects it into the squash commit. This GitLab does not, and the
     squash body is taken from this description, so it is repeated here to survive. -->

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-06 04:12:09 +00:00
userName20260323
fec055779d fix(*): raven-research takes the host web vendor and key, and the ask_user gate closes
## Summary

Three strands of raven-research work, held together because they all touch the vendored fork and so share one `subagents/TREE_HASHES` line.

**Web search and fetch route through the vendor the deployment picked.** Both tools in the research-flow plugin were pinned to Serper and Jina, so a host on Tavily or Brave still needed a Serper key for the pair. They now carry the kernel's vendor tables (seven search backends, five fetch) with per-vendor request encoding and response normalisation. Serper and Jina wire traffic is byte-identical to before. A vendor that publishes no result offset declares it up front (`SearchProviderSpec.paginates`), so the saturation rule never escalates to a page rung the vendor cannot answer.

The launcher resolves the vendor first and mirrors the SELECTED vendor's key into the plugin slice, never a key by fixed path: a mismatched pair sends one vendor's credential to another's endpoint and every call comes back unauthorized. Every vendor gets an env slot at the same path the host holds it, inherited from the host config when unset, so a deployment configures a vendor once, centrally. The two legacy vendors are also read from the vendor slot, where a host configured after the vendor table landed writes them. `require_search` follows the selection rather than the default.

**The research modes take the host's effort vocabulary**: `deep` becomes `high`, `ultra` becomes `max`, and `medium` is the baseline. This is what lets the companion MR's host tier reach this agent by name.

**The ask_user gate closes after a hook rollback**, and host questions take the conversation lock like every other route, including the `direct_ask` grant. The digest follows its session's mode, a typo in a mode overlay refuses to launch rather than running the base value under the overlay's label, and the reviewer records which model judged.

Three defects found while building it, each with the test that pins it:

- **A credential reached disk.** The retry rows of the search and fetch ledger recorded `str(exc)` for a status error, and httpx puts the whole request URL in that text. SerpApi authenticates by query parameter, so a 429 or 503 wrote the key into the JSONL file. The redaction the search and fetch rows already had is now one function, `_error_text`, used by both writers, in the plugin and in the vendored copy. The regression test drives a retryable status and reads the ledger file off disk; the previous test used 401, which is not retryable and so never reached the leaking write. The key never reached the model or the answer: the appendix partitions on `search` / `fetch` / `verify` / `force_finalize` and reads nothing from a retry row.
- **A keyless fetch backend refused every call.** The kernel's `web_fetch` degrades a keyed backend with no resolvable key to Jina, out loud. The plugin's replacement did not, and it REPLACES the kernel's, so `fetch.provider = tavily` with no Tavily key made a research run the single place that config could not read a page, while a plain raven on it read the page fine. The launcher itself produces that state: it writes the provider unconditionally and the key only when one resolves. Fixed at the build site, where the kernel resolves it, and a test asserts both implementations answer identically for every vendor.

  Reported by chandler.zhang and completed in `ac7d364c`: the degraded backend was still handed the key of the vendor it replaced, which is absent by definition, so a deployment holding a Jina key read pages authenticated through raven's built-in fetch and anonymously through the plugin -- losing the configured quota on the very fallback the rule exists to match. The kernel looks the key up a second time under the EFFECTIVE provider; the slice carries one key per role and no vendor table, so the launcher now mirrors the default reader's key beside the selected vendor's and the build picks by destination, a lookup keyed on the provider the tool will actually call. It is mirrored whenever the selection is not already the default rather than only when the fallback will fire, because the rule that decides that lives in `effective_provider` and a launcher second-guessing it would be the copy that drifts.
- **A typo in a vendor name refused to launch for the wrong reason.** The plugin degrades an unknown vendor to its default, which is right for a slice nothing validates. Carried through the launcher, the same typo had `require_search` demand a key for a vendor that does not exist while the working Serper key sat in the config -- sending the operator to set a credential they already had. The launcher now refuses naming the name and the valid list.

`.env.example` and the plugin docstrings were still describing the fixed Serper and Jina pair; both now cover the provider switch and the per-vendor key slots. The plugin MANIFEST was missed in that pass and is corrected in `18ce29df`, also on chandler.zhang's report: it still called `search.apiKey` a Serper key and `fetch.apiKey` a Jina reader key, and said web_search declines "when no Serper key resolves". A direct plugin user reaching it without the launcher could set `search.provider = tavily` and supply a Serper credential, which this code then sends to Tavily's endpoint.

`subagents/TREE_HASHES` is recorded once, in the final commit, rather than per commit: the hash covers every tracked byte under the fork, so an intermediate value goes stale the moment the next edit or a rebase lands.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run on this branch at its current head:

- `uv run pytest` -- 5 failed, 18545 passed, 69 skipped. The five are this host's pre-existing baseline, identical on `origin/main`: `test_cli_doctor_commands.py::test_doctor_answers_where_the_memories_are` and four `test_security_network.py` NAT64 parameters. 0 introduced.
- `make lint-python` -- passed, 1787 files already formatted.
- `uv run lint-imports` -- 8 contracts kept, 0 broken.
- `make check-vendored-subagents` -- exit 0, hash recorded.
- `make check-vendored-invariants`, `make check-large-files`, `make lint-ui`, `make lint-tui`, `scripts/check_commit_messages.py origin/main..HEAD` -- all exit 0.
- The vendored fork's own suite, run with the fork on `PYTHONPATH` so `raven` resolves to its copy: 19 failed, 6223 passed, 4 errors. Reverting only this branch's change to the fork gives the same 19 failures and 4 errors, all in three files that never import the changed module, so 0 introduced. Those failures are an artifact of running the fork against the trunk's venv, not the fork's own.

Reverting each fix and confirming the matching test fails: the ledger redaction (the 429 case red, the 401 case correctly still green), the fetch degradation (4 keyed vendors red, Jina correctly unaffected), the vendor-name refusal (2 red), and the slice key following the selected vendor (2 red).

Three more for the review fix: the build passing the selected vendor's key to the reader that replaced it (1 red), the launcher not mirroring the fallback key (1 red), and the launcher mirroring it for the default vendor too (1 red, the case where a second copy of the same key is one more place for the pair to disagree).

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

Security: a credential stopped being written to disk. Any research-flow ledger file written before this change by a deployment on SerpApi may contain the key in a `search_retry` or `fetch_retry` row. Those files live under the plugin's state root, one per turn, and are unlinked at the end of the turn that wrote them; a process killed mid-turn leaves one behind. Treat a SerpApi key used on an affected build as exposed and rotate it, then check for leftover files.

Other user-visible changes:

- A research deployment that selected a keyed fetch vendor without its key now reads pages through Jina instead of failing every fetch, and does so on the host's configured Jina key rather than anonymously. Behaviour improves; the log says which vendor is serving.
- That fallback key now travels in the rendered plugin slice as `fetch.fallbackApiKey`, beside the selected vendor's key, wherever the rendered config already holds credentials. It is only ever handed to the default reader: the build looks the key up by destination, so a provider neither key belongs to gets none.
- An unknown vendor name in `search.provider` or `fetch.provider` now refuses to launch. A deployment running on a typo today would have been failing its search calls anyway.
- The research mode ids change: `deep` becomes `high` and `ultra` becomes `max`. A caller that names `deep` by id gets the agent's default instead.
- Ledger `*_retry` rows record `HTTPStatusError: HTTP <status>` where they recorded the full httpx message. The one offline reader, the fork's `bench/rollout.py`, buckets by status either way and already read the search and fetch rows in that shape.

Rollback: revert the commits on this branch. `TREE_HASHES` is recorded in its own final commit, so reverting the set restores the recorded hash with the tree.

## Related Issues

N/A

<!-- The repo guide says to leave this trailer to the commits, on the premise that
     the platform collects it into the squash commit. This GitLab does not, and the
     squash body is taken from this description, so it is repeated here to survive. -->

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-06 04:12:09 +00:00
0xKT
2f862d8ca2 feat(build): RAVEN_WHEEL_SLIM switch for a headless cloud wheel
## What

Adds a `RAVEN_WHEEL_SLIM=1` build switch to `hatch_build.py`: when set, the wheel build skips the vendored sub-agent tree and the prebuilt TUI/web bundle inclusion. Default builds are byte-identical to before.

## Why

The release wheel is ~106 MiB compressed; ~94% is the `subagents/` snapshots (raven-ppt alone carries 74 MiB of `.pptx` templates and fonts) plus UI bundles. A cloud provisioner that only ever runs `raven acp` inside a sandbox uploads all of that per instance for assets nothing there opens. The slim wheel is ~3.0 MiB with the full python package intact, cutting sandbox provisioning upload from 15-23s to under a second.

## Behavior on slim installs

- Sub-agents: absent by design; onboarding step 5 reports no tree (headless installs never reach it). Their own distribution channel is future work.
- `raven tui` / `raven serve`: fall back to the source-checkout resolvers exactly as a dev build without prebuilt bundles does today; headless installs never call them.

## Verified

- `RAVEN_WHEEL_SLIM=1 uv build --wheel` -> 3.0 MiB wheel, 734 files, `raven/acp` complete, no `raven/subagents` or UI dist entries.
- The slim wheel provisioned a real E2B-compatible sandbox and served a full `raven acp` session end to end (delegated task with shell execution, ask_user elicitation round trip, multi-turn session continuity).
- Default build without the variable is unchanged.
2026-09-05 17:09:13 +00:00
hylarucoder
c703db6ffd feat(build): RAVEN_WHEEL_SLIM switch for a headless cloud wheel
## What

Adds a `RAVEN_WHEEL_SLIM=1` build switch to `hatch_build.py`: when set, the wheel build skips the vendored sub-agent tree and the prebuilt TUI/web bundle inclusion. Default builds are byte-identical to before.

## Why

The release wheel is ~106 MiB compressed; ~94% is the `subagents/` snapshots (raven-ppt alone carries 74 MiB of `.pptx` templates and fonts) plus UI bundles. A cloud provisioner that only ever runs `raven acp` inside a sandbox uploads all of that per instance for assets nothing there opens. The slim wheel is ~3.0 MiB with the full python package intact, cutting sandbox provisioning upload from 15-23s to under a second.

## Behavior on slim installs

- Sub-agents: absent by design; onboarding step 5 reports no tree (headless installs never reach it). Their own distribution channel is future work.
- `raven tui` / `raven serve`: fall back to the source-checkout resolvers exactly as a dev build without prebuilt bundles does today; headless installs never call them.

## Verified

- `RAVEN_WHEEL_SLIM=1 uv build --wheel` -> 3.0 MiB wheel, 734 files, `raven/acp` complete, no `raven/subagents` or UI dist entries.
- The slim wheel provisioned a real E2B-compatible sandbox and served a full `raven acp` session end to end (delegated task with shell execution, ask_user elicitation round trip, multi-turn session continuity).
- Default build without the variable is unchanged.
2026-09-05 17:09:13 +00:00
0xKT
009d0474e6 feat(*): subagent discovery switches rails from the forks to agents/
## Summary

Subagent discovery switches rails from the subagents/ fork trees to the
agents/ product tree. This makes the five migrated products (code, oncall,
ppt, research, design) the factory default: a fresh install discovers them
without running any per-product installer, and the fork trees stop being
scanned entirely (they remain on disk, untouched, until their separately
scheduled retirement).

Key decisions:

- Discovery resolves an agents root through the same three levels the fork
  scan used: raven home copy, beside the package, wheel-embedded. The copy
  to raven home is kept folders-only so a product's .env survives upgrades.
- Readiness has new semantics: the launcher's absolute-path command tokens
  must exist, and a product-declared engine distribution must be importable
  (importlib find_spec). A product whose engine wheel is missing is listed
  but disabled, naming the wheel to install. Credential checks leave the
  discovery phase: the launcher itself refuses loudly, which live handshake
  smoke verified for all five products.
- Config rows keep whole-row precedence over discovered rows; the merge
  mechanics are unchanged (verified byte-equivalent modulo renames by an
  independent adversarial review).
- The main wheel now ships the five product launchers under raven/agents
  and no longer embeds raven/subagents (118 files / 1.3 MB, verified by
  building and listing the wheel; zero pyproject changes, the inclusion
  lives in the hatch build hook).
- The design/ppt manifests gain one machine-readable field,
  `engine: {package, wheel}`, consumed by the readiness probe; launch
  behavior is untouched and older readers ignore the field (verified).
- A discovered row whose readiness is clean but whose runtime probe misses
  now renders as attention instead of a dead Install button (fix from the
  adversarial review).

An independent adversarial review (verdict: land after fixes) preceded this
MR; its one HIGH (net test-coverage loss on host_can_lend_a_key edge
branches) and one MED (the Install dead-end above) are fixed in the commit,
with companion tests. Fork-era behaviors it confirmed as bit-identical
inheritances (home-first shadowing, path-with-spaces refusal) are recorded,
not changed.

## Type

- [x] Feature

## Verification

- Full canonical suite on the branch: `uv run pytest tests -q` under
  `uv sync --all-extras` -> 18494 passed, 40 skipped, exit 0 (rehearsal
  tree); 18472 passed, 62 skipped on the drafting stack. Net def-level
  movement vs baseline accounted name-for-name in the wave report.
- Targeted families at landing: tests/test_agents_ppt_launcher.py,
  tests/test_cli_subagent_setup.py, tests/test_hatch_build.py,
  tests/test_rpc_subagents.py, tests/test_subagent_dag_machines.py,
  tests/test_subagent_vendored_agents.py -> 242 passed.
- `make lint-python` -> exit 0; `uv run lint-imports` -> 8 kept, 0 broken;
  `make check-vendored-subagents`, `check-vendored-invariants`,
  `check-large-files` -> exit 0. subagents/ is byte-untouched.
- Wheel audit: `uv build --wheel`, listed contents -> 5 launchers +
  5 manifests under raven/agents, 0 raven/subagents entries, no secrets.
- Fresh-venv install test: wheel installed into an isolated venv discovers
  and materializes all five rows; design/ppt correctly disabled without
  their engine wheels; research launcher runs to its credential verdict.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

User-visible change: fork rows stop appearing on fresh installs; the five
product rows appear instead. Machines with existing config rows are
unaffected (config precedence unchanged). The RPC wire shape is frozen
(vendored/building keys and subagents.build kept), so no client regeneration
is needed. Rollback: revert the squash commit; discovery returns to the
fork trees, which are still on disk in the repo and still embedded in any
wheel built before this change.

## Related Issues

N/A
2026-09-05 05:13:56 +00:00
0xKT
9a98c3107f feat(*): subagent discovery switches rails from the forks to agents/
## Summary

Subagent discovery switches rails from the subagents/ fork trees to the
agents/ product tree. This makes the five migrated products (code, oncall,
ppt, research, design) the factory default: a fresh install discovers them
without running any per-product installer, and the fork trees stop being
scanned entirely (they remain on disk, untouched, until their separately
scheduled retirement).

Key decisions:

- Discovery resolves an agents root through the same three levels the fork
  scan used: raven home copy, beside the package, wheel-embedded. The copy
  to raven home is kept folders-only so a product's .env survives upgrades.
- Readiness has new semantics: the launcher's absolute-path command tokens
  must exist, and a product-declared engine distribution must be importable
  (importlib find_spec). A product whose engine wheel is missing is listed
  but disabled, naming the wheel to install. Credential checks leave the
  discovery phase: the launcher itself refuses loudly, which live handshake
  smoke verified for all five products.
- Config rows keep whole-row precedence over discovered rows; the merge
  mechanics are unchanged (verified byte-equivalent modulo renames by an
  independent adversarial review).
- The main wheel now ships the five product launchers under raven/agents
  and no longer embeds raven/subagents (118 files / 1.3 MB, verified by
  building and listing the wheel; zero pyproject changes, the inclusion
  lives in the hatch build hook).
- The design/ppt manifests gain one machine-readable field,
  `engine: {package, wheel}`, consumed by the readiness probe; launch
  behavior is untouched and older readers ignore the field (verified).
- A discovered row whose readiness is clean but whose runtime probe misses
  now renders as attention instead of a dead Install button (fix from the
  adversarial review).

An independent adversarial review (verdict: land after fixes) preceded this
MR; its one HIGH (net test-coverage loss on host_can_lend_a_key edge
branches) and one MED (the Install dead-end above) are fixed in the commit,
with companion tests. Fork-era behaviors it confirmed as bit-identical
inheritances (home-first shadowing, path-with-spaces refusal) are recorded,
not changed.

## Type

- [x] Feature

## Verification

- Full canonical suite on the branch: `uv run pytest tests -q` under
  `uv sync --all-extras` -> 18494 passed, 40 skipped, exit 0 (rehearsal
  tree); 18472 passed, 62 skipped on the drafting stack. Net def-level
  movement vs baseline accounted name-for-name in the wave report.
- Targeted families at landing: tests/test_agents_ppt_launcher.py,
  tests/test_cli_subagent_setup.py, tests/test_hatch_build.py,
  tests/test_rpc_subagents.py, tests/test_subagent_dag_machines.py,
  tests/test_subagent_vendored_agents.py -> 242 passed.
- `make lint-python` -> exit 0; `uv run lint-imports` -> 8 kept, 0 broken;
  `make check-vendored-subagents`, `check-vendored-invariants`,
  `check-large-files` -> exit 0. subagents/ is byte-untouched.
- Wheel audit: `uv build --wheel`, listed contents -> 5 launchers +
  5 manifests under raven/agents, 0 raven/subagents entries, no secrets.
- Fresh-venv install test: wheel installed into an isolated venv discovers
  and materializes all five rows; design/ppt correctly disabled without
  their engine wheels; research launcher runs to its credential verdict.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

User-visible change: fork rows stop appearing on fresh installs; the five
product rows appear instead. Machines with existing config rows are
unaffected (config precedence unchanged). The RPC wire shape is frozen
(vendored/building keys and subagents.build kept), so no client regeneration
is needed. Rollback: revert the squash commit; discovery returns to the
fork trees, which are still on disk in the repo and still embedded in any
wheel built before this change.

## Related Issues

N/A
2026-09-05 05:13:56 +00:00
0xKT
8f6ef4c0ce ci(*): the governance batch brings the rulebook current and makes it enforced
## Summary

Governance hardening batch: the collaboration rulebook catches up with the
post-layering tree, the repository's comments and records are brought up to
its own standards, and the standards themselves become enforced gates.

Five commits, in series:

1. docs: the rulebook catches up -- AGENTS.md gains the source-language rule
   with the owner-signed CJK zones, a portable PR check replaces the
   GitHub-only wording, README's repo-layout table becomes the canonical
   scope list, CONTEXT-MAP gains its third route, and CONTEXT.md records the
   migration vocabulary.
2. docs: 133 surgical comment fixes across 83 files -- comments now say why,
   and only why; edit markers and restated-what comments are gone.
3. docs: stale orphans leave -- 5 dead documents deleted, 4 dated records
   stamped as historical, the zh README shrinks to a translated landing
   page, and the living-docs guard widens its glob.
4. ci: the honor system grows teeth -- scripts/check_source_language.py
   (diff-based, three passes), a commitlint scope enum computed from the
   tree, tests/test_readme_scope_canon.py, and the CI/make wiring for all
   three.
5. fix: a second pass brings the batch current with what landed on main
   after it was drafted -- the design engine's two owner-signed zones enter
   the gate and the rulebook, the scope enum learns the fifth product and
   the wheel distributions, CONTEXT.md records the third distributed member
   and the Visual Domain Selector, and one zh fixture added by a follow-on
   wave is respelled to English with its placeholder semantics preserved.

The batch is docs and CI only: no runtime behavior changes. The one source
edit outside docs/CI is the fixture respell in commit 5, verified by its own
test family.

## Type

- [x] CI / tooling

## Verification

- Full suite on the batch tree: `uv run --frozen --all-extras pytest tests -q`
  -> 18513 passed, 40 skipped, exit 0.
- `make lint-python` -> All checks passed; `uv run lint-imports` -> 0 broken.
- Guard families after landing commit 5: `uv run pytest
  tests/test_playbook_generator.py tests/test_commit_lint.py
  tests/test_living_docs.py tests/test_readme_scope_canon.py
  tests/test_source_language_check.py -q` -> 55 passed.
- Language-gate self-run over the batch's own range and over main's history
  since the batch base: exit 0 (one genuine pre-existing violation found and
  fixed in commit 5).
- Commitlint over all five batch subjects: 5/5 pass; `make check-large-files`
  -> exit 0.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

No runtime code paths change. The new CI gates can reject future commits by
design; if a gate misfires, reverting the squash commit removes the gates
and the rulebook text together. Existing CJK carriers in main's history are
exempt: the gate is diff-based and only judges new changes.

## Related Issues

N/A
2026-09-05 05:13:15 +00:00
0xKT
04b787f526 test(spine): every turn enters through submit, and the wheel smoke joins ci
Seventeen AST-scan guards watch this repo's seams and none watched the one the
scheduler's own tests name as the hole: a caller starting a turn without going
through Scheduler.submit. The guard pins the roster -- run_turn only from the
three spine adapters, _run_turn only from the scheduler and the loop's own two
files, tests excluded by design -- with the mutation audit built in (a
synthetic stray turns the checker red).

The raven-core wheel smoke lives in tests/integration, which norecursedirs
excludes from the default suite, so the artifact was honest only when someone
remembered to run it. It gets a make target (check-core-wheel), joins the
local make ci chain, and gets its own CI job; the integration seat stays
because building a wheel and a venv is not a cost the unit scope should pay.

Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
2026-08-31 08:42:56 +08:00
0xKT
dc8438993d refactor(plugins): everos leaves the wheel and installs like any other plugin
The memory backend shipped inside the raven package and was discovered
through the bundled directory -- the one source that no third party can
use. It is now its own distributable beside the host wheel,
plugins-dist/everos-memory, declaring itself in the raven.plugins
entry-point group: the same door a user's own plugin comes through. The
group was empty until now, so this move is what puts the first plugin on
the path every other plugin must take.

A workspace member reaches an environment only through a dependency, so
the dev group takes it and uv.sources resolves it to the member rather
than an index; without that the entry point would exist on disk and
nowhere in the environment, and everos would quietly vanish from the dev
tree. The default configuration is unchanged: memory.backend stays
"everos", and a missing plugin degrades loudly and keeps the agent
booting, which is what the assembly already did.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-08-31 04:46:47 +08:00
0xKT
0e95366f6c feat(*): the agents pilot ground opens with its charter and sixth contract
Ruled: product definitions built on the new architecture get a top-level
home (name provisional), A/B against the frozen vendored subagents/. This
is the ground only -- the package charter, a README stating what a product
directory carries and the acceptance bar (transport-face equivalence,
never code equality), the sixth import-linter contract "the runtime does
not import the agents pilots" (the evolver rule's shape), and lint
coverage. Outside the wheel by construction; pure addition, nothing in
raven/ moves. The first product lands separately.

Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
2026-08-31 00:38:32 +08:00
0xKT
fe123ba415 feat(spine): the raven-core wheel materializes from the kernel contract
The kernel contract promised a shippable core, but nothing shipped it:
"the kernel stands alone" was a proof with no artifact. make build-core
now runs scripts/build_core_wheel.py, which reads the roster from that
same import-linter contract (never a hand copy), stages the kernel with a
generated pyproject whose pins come from the main dependency list, and
builds a raven-core wheel. The integration smoke installs the wheel into
a clean venv and proves the artifact imports alone, closing the loop the
source-tree closure guard opened.

Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
2026-08-30 21:42:22 +08:00
0xKT
f98ed707d8 refactor(*): the evolver leaves the raven package for a repo-level tool package
raven.evolver was 13.6k lines with no importer in the runtime and its own
entry point: a tool that drives raven against a benchmark, not a part of what
raven ships. It becomes evolver/ beside raven/ in the checkout, still imports
raven as a library (config, providers, skill_forge catalog, atomic_io), and is
no longer in the wheel. The runtime may not import it back: a fifth
import-linter contract says so (the package is external to the linter's root,
so external packages are now included in the graph) and a test says the same
to pytest. Its tests, the appworld bench adapters, the specs and the README
follow the new address; the Makefile lints it with the rest.

Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
2026-08-30 04:36:04 +08:00
0xKT
d3c32c648f feat(*): machine-enforce the layer boundaries, and seat the last four packages
import-linter joins the dev toolchain with two contracts in
pyproject.toml, wired into make lint-imports and the CI lint job. The
forbidden contract says what the guards have said all along -- no inner
layer imports a surface -- now across seventeen source packages
including the newly seated security and auth; the independence contract
pins the twelve channel adapters as mutually ignorant cargo. Both hold
on first run. The debt allowlist carries three lazy edges, each named
and queued: the CronTool consumed by the loop, and the skill install
engine reached through its rpc wrapper. It may only shrink.

The four unseated packages take their places from the partition cards:
security is a cross-cutting mechanism and a member of the channels'
shared-services shelf (cargo may depend on it; two adapters already
do); auth is the same leaf rule; browser is a surface-side feature
library; templates is packaged data and takes no layer assignment. The
entrance guard enrolls the two code leaves, and the glossary records
all four.

Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
2026-08-29 10:45:33 +08:00
Handsome-wzw
c065cb0f3c refactor: rename ui to ui-web
## Summary

Rename `ui/` to `ui-web/`. Second half of the `ui-webui` retirement (!425): with three directories called some variant of "ui", the served page had the name that says least about it.

Mechanical, but it reaches CI, the release workflow and the wheel build, so it is its own MR rather than 155k lines of deletion with a rename hidden in it.

## The one decision worth reviewing

**The path inside the wheel stays `raven/ui/dist`. The source tree becomes `ui-web/`.**

Two names on purpose:

- `ui-web/` is what this repository calls the page's sources.
- `raven/ui/dist` is the **installed layout**. `_install_guard` reads it back out of the installed RECORD (`PAGE_RECORD_ENTRY`), the release workflow and `publish_beta.py` assert it inside the built wheel, and anything outside this repository that packages or copies an installed raven may name it.

Renaming a source directory is not a reason to move a path that leaves the repository, and "不破坏" was the constraint I was given. So `hatch_build.py` copies `ui-web/dist` → `raven/ui/dist`, with the reason written where the two sit next to each other, and the same note beside `_UI_DIR` / `_PACKAGED_UI_DIST` in `serve_commands.py`.

`test_hatch_build.py` now asserts the source and the destination as two *different* strings, with a comment saying it is deliberate — otherwise the next reader files the mismatch as drift and "fixes" it.

**If you would rather the wheel path move too, say so and I will do it in this MR** — it is `PAGE_RECORD_ENTRY`, two assertions in `release.yml`, two in `publish_beta.py`, `hatch_build.py`, `_PACKAGED_UI_DIST`, and the doctor/install-guard message text.

## What was deliberately not renamed

- **`docs/plans/` and `docs/specs/`.** They record what was true when written, and one (`2026-08-25-build-wrapup-precondition.md`) is a verification log of a run that actually happened — rewriting it would be editing a record. Same rule I applied in !425.
- **The vendored sub-agent trees under `subagents/`.** My first pass hit four of them; every match was `ui/composer` used as a *concept*, not a path, and those trees answer to `check_vendored_invariants`. Reverted.

## Verification

- `uv run pytest -q` — **12610 passed**, 50 skipped, **5 failed**
  - The 5 are pre-existing on `origin/main`: same node ids fail on an untouched `origin/main` worktree at the merge base (`e1011cb1`). Four `test_security_network.py` NAT64 cases, one `test_cli_doctor_commands.py::test_doctor_answers_where_the_memories_are`.
  - A 6th failed on my first run and was **mine**: `test_hatch_build.py::test_an_editable_install_gets_no_copy_of_the_web_assets`, which is exactly the test that should catch a rename touching the wheel path. Fixed as described above.
- `uv run ruff check raven tests scripts` — All checks passed
- `uv run ruff format --check raven tests scripts` — 1241 files already formatted
  - (Both halves. On !425 I ran only the first and CI caught the second.)
- `npm run build --prefix ui-web` + `python3 ui-web/build.py` + `check-page.mjs`, `check-css.mjs`, `count-shared-globals.mjs` — pass
- `npx vitest run` in `ui-web` — 89 files, 1222 passed
- `npm run gen:check --prefix ui-web` — in sync, 147 methods; `type-check` clean
- `scripts/check_vendored_invariants.py`, `scripts/check_vendored_subagents.py` — pass
- `git diff --check origin/main...HEAD` — clean
- `scripts/commit_lint` — no errors
- `git ls-files | grep -c node_modules` — 0

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [x] Refactor

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear

No wire change, no behaviour change. The installed layout is unchanged, which is the compatibility surface that matters. The blast radius is build and CI paths, all of which are exercised above. Rollback is a revert.

**Merge order:** independent of !425 except for one line in `AGENTS.md` that lists the application source trees; whichever lands second takes a one-line conflict there.
2026-08-28 12:51:24 +00:00
Handsome-wzw
9a87470fdf fix(rpc): turn.started is a turn event, and says so on all three sides
## Summary

Found while investigating the first row of Dizhan's `子智能体功能与显示问题` doc (*回注内容过程中未正常显示,刷新后才会正常显示*). This is not that bug's fix, but it is a real defect on the same wire, and it is the one that explains why "前后端接线没有约定好" was a fair thing to say.

**`turn.started` is not in the `TurnEvent` union — on either side.** It has a schema component, it has a Pydantic model, and neither is reachable from the union that `tests/test_rpc_schema_match.py` walks. The 290-test contract guard has never looked at this event.

So it drifted the first time anyone touched it. `05957452` (judge a dag node) taught `announce_dag_exception` to emit:

```python
mark = {"kind": "dag", "label": run_id, "status": "exception", "run_id": run_id, "node_id": node_id}
```

while `TurnStartedEvent.payload.delegated` still declared `status: enum ["ok","error"]` with `additionalProperties: false`, and `TurnStartedPayload` had **no `delegated` field at all** under `extra="forbid"`. Verified with `jsonschema` against the published contract before changing anything:

```
Additional properties are not allowed ('node_id' was unexpected)
```

That commit did carry `exception` into `TranscriptDelegated` and `SubagentDeliveredEvent` — two of the three places this status lives. The third is the one no test covers.

**It does not crash today**, and I want to be exact about that: `SubscriptionEmitter.emit` pushes a raw dict with no validation, and the page reads only the fields it knows. But the contract test's own docstring says what the cost is — *"both sides declare `additionalProperties: false` / `extra="forbid"`, so a consumer validating against the schema rejects the whole event, not just the unknown field"*. Losing this event loses the boundary that advances the workspace turn counter and draws the delegated row.

## What this does

- `TurnStartedEvent` joins the `TurnEvent` union, in `openrpc.json` and in `raven/rpc/models.py`, so the existing sweep covers it from now on.
- The schema's `delegated.status` gains `exception`; `node_id` is declared.
- `TurnStartedPayload` gains `delegated` as a typed model (`TurnStartedDelegated`), three-valued status included.
- Both generated TypeScript clients regenerated.
- A new test drives **the exact marker `manager.py` builds** through both validators, so the next status or field fails here instead of in a client nobody is watching.

## One thing the union surfaced

`ui-tui` never handled `turn.started`. Its exhaustiveness check said so the moment the variant existed. It draws the delegated row from `subagent.delivered`, which the page deliberately ignores in favour of this event — the two mark different moments: **submitted** versus **started**.

Left as it is, with an explicit `case 'turn.started'` saying why. Moving that surface onto the truer moment changes when the TUI draws the row, which is a product change, not a contract fix.

## Verification

- New test red before the fix, by construction: adding the variant to the union alone reds `test_schema_match_turn_event_discriminated_union` and `..._payload_fields` (schema/pydantic drift), which is how the gap was proved.
- Mutation checks, each reverting one thing:

| reverted | reds |
|---|---|
| `exception` out of the schema enum | 1 |
| `node_id` out of the schema | 1 |
| `delegated` off the Pydantic payload | 2 |

- `uv run pytest -q` — **12900 passed**, 50 skipped, 5 failed; the same five pre-existing on `origin/main` (four `test_security_network.py` NAT64 cases, one `test_doctor_answers_where_the_memories_are`), reproduced on an untouched merge-base worktree.
- `uv run ruff check` / `ruff format --check` on `raven tests scripts` — both clean (both halves of `make lint-python`).
- `ui`: `gen:check` 147 methods, `type-check` clean, `npx vitest run` 89 files / **1225 passed**.
- `ui-tui`: `type-check` clean, `lint:rpc` in sync, `lint:i18n` up to date, `npm test` 138 files / **1990 passed**.
  - Worth recording: with `ui-tui/node_modules` symlinked from another checkout the suite reports 82 failed / 59 failed — and reports **the same numbers on an untouched `origin/main`**. It is the symlink, not the branch. Ran `npm ci` in the worktree to get a trustworthy number.
- `scripts/commit_lint` — no errors.
- `git ls-files | grep -c node_modules` — 0.

## Type

- [x] Fix

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear

Widening a contract to match what the runtime already emits: every payload valid before is valid now. No emit site changes, no behaviour changes. The one code change outside the schema and the models is an explicit no-op branch in `ui-tui`. Rollback is a revert.
2026-08-28 12:51:10 +00:00
forrestjgq
8ecfae3e0c test(tracing): isolate span emission into per-test tmp dirs (#371)
## Summary

Unit tests were leaking tracing spans into the real `~/.raven/traces`
store. Tracing is on by default and `trace.span()` calls are embedded in
library code (agent loop, subagents, TUI RPC), so any test exercising
those paths emitted spans with fabricated session keys (`session-a`,
`weixin:c`, `tui:default`, ...). Those spans then surfaced as phantom
sessions in `raven trajectory`, while the TUI `/sessions` list (which
reads the real session store) stayed clean.

This PR adds an autouse fixture in `tests/conftest.py` that:

- redirects `RAVEN_TRACING_DIR` to a per-test tmp directory, and
- resets the lazy module-level `TraceStore` cache
(`raven.tracing.spans._store`) around each test, since it pins the
directory resolved at first emit.

Key decisions:

- Only the directory is redirected, not the enabled switch, so the suite
keeps exercising the real emission path with zero behavior change; leaks
land in tmp instead.
- This centralizes the setenv-plus-cache-reset pattern that 8
tracing/trajectory test files already perform manually. Those tests keep
working: their own `monkeypatch.setenv` runs after the fixture and wins,
and they already reset the cache themselves.
- Existing phantom sessions come from archived span logs written before
this fix; the fixture only prevents new leaks. One archive file mixes a
real session with test keys, so cleanup needs line-level filtering and
is left out of scope.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [x] Other

Test-suite hygiene (`test` commit type).

## Verification

- `uv run pytest tests/test_tracing_api.py
tests/test_trajectory_store.py tests/test_trajectory_bundle.py
tests/test_trajectory_replay.py tests/test_trajectory_redact.py
tests/test_cli_tracing_commands.py tests/test_cli_trajectory_commands.py
tests/test_cli_trajectory_browse.py -q`: 308 passed, 1 failed
(`test_viewer_health_true_for_our_viewer`, fails identically on
unmodified `main`, environment-specific)
- `uv run pytest tests/test_agent_loop_run_emit.py
tests/test_subagent_manager.py tests/test_tui_rpc_turn_send.py
tests/test_deep_research_tool.py tests/test_agent_loop_approval.py -q`:
109 passed; the real `~/.raven/traces` log did not grow during the run
- Positive check: running `tests/test_agent_loop_run_emit.py` with
`--basetemp` shows the previously leaked session keys (`cli:c`, `tg:c`,
`weixin:c`) now land in per-test tmp dirs
- `uv run pytest tests/ --ignore=tests/integration -q`: 6837 passed, 11
failed, 20 errors; the same failure set reproduces on unmodified `main`
(environment-specific: cron/tz, everos server, theme, config loader,
viewer health), zero regressions from this change
- `uv run ruff check tests/conftest.py` and `uv run ruff format --check
tests/conftest.py`: both pass

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

Docs box left blank: test-only change, no user-facing docs affected.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

Test-only change; no production code touched. One known interaction:
`RAVEN_TRACING_DIR` takes precedence over `RAVEN_HOME`, so a future test
that fakes `RAVEN_HOME` and expects traces under it must also set
`RAVEN_TRACING_DIR` itself. Rollback: revert the single commit.

## Related Issues

N/A

Co-authored-by: 江国庆 <guoqingjiang@deepglint.com>
Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
2026-08-28 19:21:16 +08:00
arelchan
7c33c4133a fix(rpc): let the viewer open the files a turn actually wrote
## Summary

A delivered artifact could not be opened from the page: the file viewer answered
403 and the transcript drew "Raven is confined to the workspace, so this file
cannot be opened here" over a delivery card it had just rendered.

The viewer's state-directory fence exempts the workspace subtree, on the stated
grounds that the default workspace lives inside raven's home and denying it
would refuse every file the agent itself wrote. Since agent home and the working
directory were split apart, a turn no longer writes there: it writes in
`~/.raven/tmp/<channel>`, a sibling of the workspace, inside the same state root
and outside the carve-out. So the fence denied exactly what the carve-out exists
to allow, one directory over, for every artifact produced since that split.

This exempts the working-directory root alongside the workspace, using the same
containment test in the same direction, so a working-directory root that happens
to sit above raven's home earns no exemption. `config.json`, `oauth/` and
`serve.json` are siblings of both and stay refused, which is what the fence is
for.

Worth noting for a reviewer: the bytes were already reachable. `/files/download`
serves a delivered file by token with no path fence, so this was never a
security boundary for deliveries -- only a viewer that refused to render what
the same page could already download.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- `uv run pytest tests/test_rpc_transport.py tests/test_rpc_files.py tests/test_rpc_knowledge.py tests/test_rpc_console.py -q` -> 123 passed
- `uv run ruff check` and `uv run ruff format --check` on both changed files -> clean
- Both new assertions were mutation-checked, each against its own mutation:
  - narrowing the carve-out back to the workspace alone reds
    `test_the_turn_working_directory_is_exempt_from_the_state_dir_fence` only (1 failed, 3 passed)
  - dropping the containment-direction guard reds
    `test_a_working_directory_root_that_contains_the_state_root_earns_no_exemption`
    and the pre-existing state-dir test (2 failed, 2 passed)
- Against the real reporting case: the two files that were answering 403
  (`~/.raven/tmp/tui/*.md`, `*.pptx`) now resolve, while `~/.raven/config.json`
  and `~/.raven/serve.json` still raise PermissionError.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

- Widens what the file viewer will render by exactly one subtree: the directory
  turns work in. Everything the fence was written to protect (provider keys in
  `config.json`, the OAuth token store, `serve.json`) sits beside that subtree,
  not inside it, and stays refused -- pinned by the tests above.
- `fs.list` / `fs.reveal` share the function but pass the session's working
  directory as the root, which their own containment check already bounds, so
  their behaviour is unchanged.
- Rollback is a one-line revert of the exemption tuple.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-08-28 04:57:29 +00:00
LivXue
4e35a5d627 fix(tui): show the copy-on-select confirmation as a transient notice (#366)
## Summary

PR #360 reports every copy-on-select write as a permanent transcript
line, so a
reading session that drags through the transcript piles up "copied N
characters" rows under the conversation. The confirmation splits by what
the
line is for: the unbounded terse repeats become a transient notice
rendered
above the input box -- one line that replaces itself on every new copy
and
clears after 3 seconds -- while the first copy of a session keeps its
path
caveat in the transcript. The caveat is the answer a user comes looking
for
after a paste comes up empty minutes later, and it cannot pile up
because it
is once per session by construction.

The reporter still owns the per-session caveat tally and now returns
`{ text, firstOfSession }` so the caller can route the caveat to the
transcript and the repeats to the notice. A new `copyNoticeStore` holds
the
single notice; `showCopyNotice` keeps one dismiss timer, clearing any
pending
one before arming the next, so an earlier deadline can never clear a
newer
notice. `/copy` is deliberately untouched: an explicit command's answer
still
belongs in the transcript, and it cannot stack on its own.

## Type

- [x] Fix

## Verification

Run from `ui-tui/` at branch head, in a fresh worktree of `github/main`.
Dependencies came from `npm ci` in `ui-tui/` and in the hermes package;
the
hermes package's own `node_modules` holds only esbuild, so the ink
render
tests run against one react copy (two copies break every hook call).

```
npx vitest run --no-file-parallelism
  Test Files  93 passed (93)
  Tests  1084 passed | 13 skipped (1097)

npm run type-check
  clean, exit 0

npm run lint
  22 problems (0 errors, 22 warnings)

npx prettier --check src/app/copyNoticeStore.ts src/__tests__/copyNotice.test.tsx
  src/__tests__/clipboard.test.ts src/app/useMainApp.ts
  src/components/appLayout.tsx src/lib/clipboard.ts
  All matched files use Prettier code style!

python3 -m scripts.check_commit_messages github/main..HEAD
  exit 0
```

The 22 lint warnings are the baseline, not new: 22 before the branch and
22
after, none in a file this branch touches. The suite is run with
`--no-file-parallelism` deliberately: the ink render tests fail under
default
worker parallelism at this suite size, and no CI job runs them.

Suites that exercise the changed code directly:
`src/__tests__/copyNotice.test.tsx`
(7 cases), `src/__tests__/clipboard.test.ts` (30 cases, the 3 reporter
cases
updated to the new return shape).

New tests were proved load-bearing rather than assumed:

- Removing the notice render line in `appLayout.tsx` makes the render
test
  fail (content and placement above the input box).
- Removing the pending-timer clear in `showCopyNotice` makes the
re-shown-notice test fail: the earlier 3s deadline then clears the newer
  notice.
- The dismissed-notice case now shows a notice first, and removing
  `dismissCopyNotice`'s atom clear turns it red (review fix).

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

No doc change. `ui-tui/CONTEXT.md` has no clipboard or selection entry,
and
"copy notice" continues the existing "notice" vocabulary of
`copyResultNotice`
/ `copyOnSelectNotice`, so nothing new is coined.

## Risk

Behaviour change: the per-drag copy confirmation is now transient and
above
the input box instead of a permanent transcript line, and the first copy
of a
session lands one transcript line, exactly as on `main`. Reverting the
branch
restores the #360 behaviour; no data or on-disk format is involved.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (deepseek-v4-pro) <noreply@anthropic.com>
2026-08-25 14:27:24 +08:00
ZuyiZhou
1303e77010 fix(agent): withhold web_search without a key in every vendored subagent
## Summary

Three of the four vendored subagent checkouts registered `web_search` unconditionally, in **both** registries each checkout has: the main loop and the nested sub-agent's tool set. On a machine with no Serper key that advertised a tool which could not answer -- the model reached for it, every call failed, and the error text (naming a config file and an env var) was relayed to whoever was on the other end of the channel. One TB run spent 17 of 17 `web_search` calls that way, which is why this trunk gated it in the first place.

This trunk and `raven-code` already carried the gate. `raven-oncall`, `raven-ppt` and `raven-research` did not, and nothing reported it.

**Baseline before this change** (line numbers on `origin/main` @ `73f95d9d`):

| checkout | main loop | nested sub-agent |
|---|---|---|
| trunk | `raven/agent/loop/main.py:1160-1171` gated | `raven/agent/subagent/backends/raven_loop.py:241-244` gated |
| raven-code | `:973` gated | `subagent/manager.py:203` gated |
| raven-oncall | `:706` **unconditional** | `subagent/manager.py:171` **unconditional** |
| raven-ppt | `:814` **unconditional** | `subagent/manager.py:202` **unconditional** |
| raven-research | `:1075` **unconditional** | `subagent/manager.py:190` **unconditional** |

### Why nothing caught it

`scripts/check_vendored_subagents.py` records one tree hash per fork, so it answers "has this tree been touched". A tree that was **always** behind has a stable hash, so it stays green forever. That guard exists for the opposite accident (the v0.1.13 sync, where git's rename detection landed an upstream delta inside three forks) and it is not the guard for this.

### The fix, and why it is two parts

**The gate itself** (6 source sites). `raven-oncall` and `raven-ppt` get `if web_search.api_key:`, matching this trunk exactly; the `else` branch logs one line, following `raven-code` rather than the trunk's main loop, because withdrawing a tool silently is the same class of problem this defect is about.

`raven-research` gates on the key **or** a configured corpus endpoint. Copying this trunk verbatim would have been wrong there: `Raven-X/raven/agent/tools/web.py:661` short-circuits to `_search_corpus` **before** the `if not self.api_key` branch, so a configured corpus needs no Serper key at all, and a key-only gate would withdraw `web_search` from every contained benchmark run.

**A guard that keeps it that way** (`scripts/check_vendored_invariants.py`, new). This is the durable half. Both `subagents/README.md` and `raven-research/run.py` state that a checkout is replaced wholesale on every upstream zip, so a patch to a vendored tree is short-lived by construction; a registry that fails CI when the fix is reverted is the only way anyone finds out.

Design points, each one a mistake this went through first:

- **Invariant-shaped, not diff-shaped.** It never compares two trees. The forks are third-party source with no shared history and thousands of legitimately divergent lines, so "is it in sync" has no answer -- and a fork is sometimes *ahead* of this trunk rather than behind. "Does it still hold this specific guarantee" does have an answer.
- **AST, not a text pattern.** The four forks format the call differently (this trunk two arguments on one line, `raven-ppt` four over five lines), and the thing under test is the extent of an `if` **body**, which no line-oriented pattern can see. A registration in the `else` of the guard counts as unguarded -- that is the unguarded case wearing a guard's shape.
- **It checks `register`, not construction.** The first version checked the construction site and failed this trunk and `raven-code`: both construct unconditionally and then `if web_search.api_key: register(web_search)`. Construction does no I/O; what the fix moved was the registration.
- **The trunk is asserted too.** An invariant only ever checked against the forks keeps passing while describing nothing, the moment the trunk rewrites the code it names.
- **A stale path fails rather than passes.** If a listed path no longer registers the tool at all, that is a failure, not a silent pass for lack of anything to look at.
- **An exemption must carry a reason.** "This fork's design makes the trunk's fix wrong here" and "nobody got round to it" are different things, and a bare skip cannot tell them apart.

Wired into `make lint` and pre-commit (`always_run: true` -- the file set it reads is the registry inside the script, which no `files:` pattern in the hook config can follow; it parses six files in well under a second).

**Test adaptation the fix requires**: `tests/test_agent_loop_tool_search.py:104` in each of the three forks asserts `loop.tools.has("web_search")` while its `_make_loop` passes no key, so it would go red. Fixed by adding `brave_api_key="test-key"` to `_make_loop`, which is what `raven-code` did when it took the gate (`raven-code/Raven-main/tests/test_agent_loop_tool_search.py:63-66`).

`subagents/TREE_HASHES` regenerated with `--update` (three hashes moved; `raven-code` unchanged). `subagents/README.md` gains a section on the two guards and their opposite directions.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Head `c29bfbaa`. Base `origin/main` @ `73f95d9d`. Every number below was re-measured at this head after the rebase described under "Rebase" further down.

```
uv run pytest -q
  -> 11989 passed, 33 skipped, 13 deselected in 660.83s (exit 0)

uv run pytest tests/test_vendored_invariants.py tests/test_vendored_subagents.py -q
  -> 18 passed

make lint-python
  -> ruff check: All checks passed; ruff format --check: 1197 files already formatted

python3 scripts/check_vendored_subagents.py   -> exit 0
python3 scripts/check_vendored_invariants.py  -> exit 0
make check-commits                            -> commitlint + check_commit_messages.py pass
make check-pr-title                           -> pass
git merge-tree --write-tree origin/main HEAD  -> clean (no conflict)

uv run pytest tests/integration/ -m integration -q
  -> 2 failed, 53 passed, 2 skipped, 120 deselected
```

**The two integration failures are pre-existing on the base, not from this change.** Verified twice, in a detached worktree at `origin/main`, once at `1d55b28a` and again five commits later at `73f95d9d`: both times `2 failed, 33 passed`, the same two --
`test_acp_stdio_smoke.py::test_a_whole_handshake_runs_in_one_process` and
`test_acp_adversarial_smoke.py::TestSessionLoad::test_a_planted_transcript_replays_before_the_load_returns`.
Measuring it twice across five commits of trunk movement rules out a short-lived breakage. This diff touches no file under `raven/` at all (see the file list under Coverage), so it cannot reach the ACP path.

The two skips in that run are `test_research_subagent_smoke.py` (`vendored Raven-X venv is not built`).

### How this change can be shown to fail

**Reverting the fix turns zero trunk tests red.** That is the number worth reading: all six defect sites live inside vendored trees, this trunk's 11702 tests cannot see them, and the forks' own suites do not run in this pipeline. The usual "revert the fix and watch a test go red" criterion does not work on this defect.

The real criterion is the new guard, exercised one site at a time. Each of the six sites was individually reverted with `git checkout HEAD~1 -- <file>`; `check_vendored_invariants.py` returned `exit=1` every time and named the fork, the file and the line:

```
raven-oncall   .../raven/agent/loop/main.py:706
raven-oncall   .../raven/agent/subagent/manager.py:171
raven-ppt      .../raven/agent/loop/main.py:814
raven-ppt      .../raven/agent/subagent/manager.py:202
raven-research .../raven/agent/loop/main.py:1075
raven-research .../raven/agent/subagent/manager.py:190
```

6 of 6 detected; the tree returned to `exit 0` after restoring.

`tests/test_vendored_invariants.py` covers the registry against the real trees (every fork holds every invariant; every listed path exists; every exemption carries a reason) and the reader against synthetic sources, because the shapes that matter -- a registration in the `else` of the guard, one behind a local name, an unrelated guard that must not satisfy it -- are not all present in the trees at any one time.

### Behaviour verified in a fork, not only read

None of the three forks ships a `.venv`, so `raven-oncall`'s was built with `uv sync` and the change was exercised there:

- `uv run pytest tests/test_agent_loop_tool_search.py -q` -> 5 passed
- two `AgentLoop` instances compared directly:
  - no key -> `web_search` registered: **False**, with `web_search not registered: no Serper API key configured` (`raven/agent/loop/main.py:715`)
  - key passed -> `web_search` registered: **True**
  - no key -> `web_fetch` registered: **True** (confirming `web_fetch` was not withdrawn along with it)

**Known gap:** `raven-ppt` and `raven-research` got `py_compile` plus the invariant check, but their own suites were not run -- `raven-ppt` needs `uv sync --extra ppt` for the whole rendering stack and `raven-research` is the largest checkout, and a third and fourth confirmation of the same three-line change did not justify it. The grep for registration assertions across all three forks is recorded below.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

### Please do not read the coverage gate as evidence here

```
COVERAGE_BASE_REF=origin/main make coverage-diff
  -> Diff coverage: 100.00% (0/0 executable changed lines) -- passed the 90.00% threshold
```

That 100% is an empty pass. Here is the gate's own input:

```
git diff --name-only origin/main...HEAD -- ':(glob)raven/**/*.py'
  -> (no output)
```

`[tool.coverage.run] source = ["raven"]`, and this diff changes nothing under `raven/`. Eleven of its fifteen files are inside vendored trees, which are not measured either; the rest are `scripts/`, `tests/`, `Makefile`, `.pre-commit-config.yaml` and a README:

```
.pre-commit-config.yaml
Makefile
scripts/check_vendored_invariants.py
subagents/README.md
subagents/TREE_HASHES
subagents/raven-oncall/Raven-Oncall/raven/agent/loop/main.py
subagents/raven-oncall/Raven-Oncall/raven/agent/subagent/manager.py
subagents/raven-oncall/Raven-Oncall/tests/test_agent_loop_tool_search.py
subagents/raven-ppt/Raven-PPT/raven/agent/loop/main.py
subagents/raven-ppt/Raven-PPT/raven/agent/subagent/manager.py
subagents/raven-ppt/Raven-PPT/tests/test_agent_loop_tool_search.py
subagents/raven-research/Raven-X/raven/agent/loop/main.py
subagents/raven-research/Raven-X/raven/agent/subagent/manager.py
subagents/raven-research/Raven-X/tests/test_agent_loop_tool_search.py
tests/test_vendored_invariants.py
```

The gate never saw the 6 changed registration sites, and it never will for a change shaped like this one. The evidence for this MR is the verification section above.

### Rebase

This branch was cut at `1d55b28a` and rebased to `73f95d9d`. The rebase conflicted, on `subagents/TREE_HASHES` alone, because `ffe9ba9b` landed a wholesale upgrade of the `raven-code` checkout on trunk (38 files, +6444, including a vendored `raven/acp/` and a new `--acp` mode in that folder's launcher) and moved that fork's recorded hash.

**The conflict was resolved by re-running `python3 scripts/check_vendored_subagents.py --update`, not by picking a side.** Picking is not safe here even when it looks obvious: the correct manifest is the hash of "the upgraded checkout, with this branch's changes on top", and that combination exists on neither side of a conflict. Choosing trunk's side reports this branch's fork edits as an accidental landing; choosing this branch's side reports the whole upgrade as one. Neither error surfaces at conflict-resolution time -- both wait for the next run of the guard. Re-running `--update` recomputes all four hashes from the trees actually on disk, which is the only side that is right. As it happens `raven-code` came out byte-identical to trunk's value (`d786a70f...`) and the other three kept this branch's, which is what a correct recomputation looks like here.

That upgrade is also the first real exercise of the guard this MR adds, so it was checked rather than assumed:

- `raven-code/Raven-main/raven/agent/loop/main.py:972-977` -- the gate survived the wholesale replacement, `if self.web_enabled:` and `if web_search.api_key:` both intact
- `raven-code/Raven-main/raven/agent/subagent/manager.py:202-205` -- likewise
- a repo-wide grep for `WebSearchTool` found no third registration site in the upgraded fork
- the new `--acp` mode does not build its own tool set (`grep -rn "tools.register" subagents/raven-code/Raven-main/raven/acp/` and the same over this trunk's `raven/acp/` are both empty; the only `"web_search"` literal in that layer is `acp/tool_kinds.py:43`, a name-to-kind mapping for display), so it reaches the same gate through `loop/main.py` and the two paths per fork remain the complete set

### TUI gates

Not run, and not applicable: this diff touches no TypeScript, no `rpc-schema/openrpc.json`, and no file under `ui/`, `ui-tui/` or `bridge/`.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

**Security.** The change only ever withdraws a tool; it grants nothing. It touches no credential handling. No key value is logged -- the `else` branch logs the absence, not the value.

**Compatibility.** A deployment that has a Serper key (in the fork's `.env`, inherited from the host raven's `tools.web.search.apiKey`, or exported as `SERPER_API_KEY`) sees no change: the tool resolves its key at call time from either source, so the gate asks the tool and not the config. A deployment with no key loses a tool that could not have answered. `raven-research` additionally keeps `web_search` whenever a corpus endpoint is configured, so contained benchmark runs are unaffected.

**Rollback.** Reverting the commit restores the previous behaviour and the previous `TREE_HASHES` in one step; the guard and the gate go together, so there is no half-reverted state where CI fails against a tree it no longer describes.

### Self-inflicted problems found while checking, both fixed before committing

1. The three forks' `tests/test_agent_loop_tool_search.py:104` would have gone red (see the test adaptation above). Found by grepping `has("web_search")` and `"web_search" in` across the three forks' `tests/`. Three further hits were examined and are **not** affected:
   - `raven-research/tests/test_agent_flow_dr.py:616,645` calls `tools.unregister("web_search")`; `tools/registry.py:43-45` is `self._tools.pop(name, None)`, which tolerates absence, and the test then registers a stand-in named `web_search` (`_BigResultTool`, `:560-565`).
   - `raven-research/tests/test_agent_loop_invariants.py:16` -- `DECLARED` is a literal passed into `turn_invariants`, not derived from a registry.
   - `raven-ppt/tests/ppt/test_brief.py:240` asserts the string appears in a finding's message, unrelated to registration.
2. The first version of the checker flagged this trunk and `raven-code` as violations (it checked construction, not registration). Acting on that report would have broken a correct implementation.

A third candidate -- "the checker only looks at one path per fork and misses another registration site" -- turned out to be **real**, and is the second registry now in the registry entry. A repo-wide grep for `WebSearchTool` found no fourth registration site: the remaining hits are the class definition in each `tools/web.py`, docstring references inside the forks, an `isinstance` check at `raven-research/.../loop/main.py:4126`, and this trunk's `raven/agent/tools/capabilities.py:172-174`, which queries `WebSearchTool.is_configured` rather than registering.

### Known blind spot of this approach

Drift is bidirectional. `raven-research` carries a crash-durable turn journal this trunk does not have at all (`Raven-X/raven/session/manager.py:154 partial_path()` and `Raven-X/raven/agent/loop/journal.py`; `TurnJournal` has zero hits in this trunk). An invariant registry cannot see that, by construction -- there is no invariant naming it -- which is exactly why it also cannot flatten it. The corollary is that this guard will not discover a fork that is ahead; that is a known limitation, not a defect. A three-way drift report (behind / ahead / both changed) was considered and rejected for the same reason a generic staleness warning was: it needs a prior judgement about which divergences are legitimate, and that judgement does not exist.

## Related Issues

N/A
2026-08-27 03:01:59 +00:00
arelchan
172b05a064 fix(ui): give the delivery card one shape, and unbreak four panels
## Summary

Six defects a review pass over the served page turned up, plus the registry read
underneath one of them.

**The delivery card was measured rather than stated.** A width-derived capacity
hook decided how many tiles fit and whether a description was drawn, so one card
was four different cards across six widths. The cap is a stated count now, the
description always shows, and a tile keeps one preview ratio with a max-height
instead of a growing box. Measured after: a single card is 142px at every width
with a description and 134px without, and a grid preview is a constant 198x124.
The preview itself lost its inset white sheet -- content fills the tile.

**A desk pane re-opened itself.** Clicking a row whose pane was already up
replaced the pane object, re-tiled the grid and re-revealed the workspace to
arrive at exactly the screen that was already there, and `revealWorkspace` ran a
full workspace draw even when the split was already open. Both are no-ops now;
workspace draws on a re-click went from one to zero, and none of it ever
repainted the pane's content -- that was measured before changing anything.

**A dag node header ran past its panel.** The node id is unbounded -- a playbook
names one `daily-ai-digest-36e275-scan-news` -- and with nothing able to shrink,
the flex row pushed the "open run" link off the right edge, where it read as the
card having been cut in half. Measured before: a 320px panel overflowed by 18px
and a 280px one by 58px. The name gives way now, and only the name.

**The file view's action row was `.acts`**, which the transcript already uses for
an answer's icon strip -- 27x27 square buttons. Two text buttons were squeezed
into those squares and their labels overlapped, which reads as mojibake, not as a
layout bug. Renamed to `.binacts`: the buttons measure 186x30 and 107x30 with no
overlap. Second class collision found in this stylesheet, after `.mini`.

**The floating desk covered the settings dialog.** It sits in its own band below
the takeover layer now, verified with `elementFromPoint` at the palette centre
returning the dialog's own panel.

**The instance registry was read once per process.** `_load` cached its records
for the life of the gateway, while sub-agents register from their own `cli`
processes into the same shared file -- so every instance created after startup
was invisible to the panel until a restart. The decisive test was appending a row
from outside a running server and getting zero back. It reloads on a change in
`(mtime_ns, size)`, stamped from the stat taken *before* the read so a writer
landing between the two is seen on the next call rather than stamped as
already-loaded and skipped forever; `_flush` restamps after its own write, so
writing does not cost a reparse. This also closes a silent data loss: `_flush`
writes the whole cached map back, so flushing from a stale cache dropped every
record another process had added in the meantime.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run at this head, on `da0f0124`:

- `uv run --frozen --all-extras pytest -q` -> `11214 passed, 33 skipped, 13 deselected, 2 warnings in 715.40s`, exit 0
- `npm test --prefix ui` -> `58 passed (58) / 686 passed (686)`
- `make lint-ui` -> `generated.ts matches the contract (132 methods)`, then `tsc --noEmit` with no output
- `uv run --frozen --extra dev ruff check` + `ruff format --check` on the two python files -> `All checks passed!`, `2 files already formatted`
- `PYTHONPATH=. python3 scripts/check_commit_messages.py origin/main..HEAD` -> no output, exit 0

The five registry tests were mutation-checked: disabling the reload reds three of
them, and removing the restamp in `_flush` reds the one that pins "our own write
is not a reason to reparse". The two desk-store tests were checked the same way.

`make lint-python` over the whole repo is not clean at this head, and was not
before this branch: the errors are in `raven/agent/tools/deliver.py`,
`raven/cli/tui_commands.py`, `tests/test_channels_outlet.py` and
`tests/test_mcp_client.py`, none of which appears in this diff.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

User-visible behaviour changes, all of them on the served page:

- a delivery card is one height per state instead of one per width, and always
  carries its description;
- clicking a desk row whose pane is open no longer redraws anything;
- a dag node's name truncates instead of pushing its run link out of view;
- the desk no longer paints over the settings dialog;
- an instance registered by another process appears without a restart.

Rollback: revert the two commits. Nothing is persisted that a rollback would
strand -- the registry change is read-side only, and the css and tsx changes hold
no state.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

Known gap, deliberately not fixed here: a spawn's answer is missing from three
instance logs on my own machine (the spawn's prompt is logged as a `user` row
with no `assistant` row beside it), so opening those instances shows a question
that was never answered. Reproduced on disk, not yet diagnosed, and unrelated to
anything in this diff.

## Related Issues

N/A
2026-08-25 13:04:45 +00:00
arelchan
35abacb489 fix(subagent): re-read the instance registry when another process wrote it
`_load` cached the records for the life of the process, so the gateway held
whichever rows existed when it started. Sub-agents register from their own
`cli` processes into the same shared file, which made every instance created
after startup invisible to the panel until a restart -- the decisive test was
appending a row from outside a running server and getting zero back.

Reloaded on a change in (mtime_ns, size), stamped from the stat taken before
the read so a writer landing between the two is seen on the next call rather
than stamped as already-loaded and skipped forever. `_flush` restamps after
its own write, so writing does not cost a reparse on the following call.

This also closes a silent data loss: `_flush` writes the whole cached map
back, so flushing from a stale cache dropped every record another process had
added in the meantime.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-08-25 20:41:38 +08:00