<img width="2318" height="1448" alt="image"
src="https://github.com/user-attachments/assets/bca71d69-4972-420c-b14e-c8cf74a17de9"
/>
## Summary
Everything a knowledge base could not read, and the pieces it cut them
into
saying what they are.
**Reading a PDF.** A PDF records glyphs at coordinates; whether they are
a
heading, a table cell or a running footer is inferred from how they
look. There
are now two readers. The geometric one infers structure from type size
and
position and needs nothing but the file. The other asks deepdoc's
models, ported
into `raven/knowledge/parser/deepdoc`: a layout model that says what
each region
is, a detector and recognizer that read a page carrying no text at all,
and a
table-structure model that rebuilds a grid from a picture of one.
Detecting boxes around text the file already spells
out exactly costs an inference pass per page to learn less, and
recognising it
costs accuracy.
The layout graph answers with a shape its own postprocessing
raises on, and its classes are DocLayout-YOLO's order rather than the
list
Both were established by running the model, and the label order is
pinned by a test.
**Tables go in as tables.** `<table>` is the only shape that keeps what
a grid
states -- which column a cell stood under, which cells are merged, which
row is
the header -- and it is the shape a model reads a table in, which is
what these
chunks are for. PDF, Word and PowerPoint all state their merges exactly,
and
they now survive as `colspan` and `rowspan`. The flat form this replaces
had to
repeat a header standing over two columns across both of them, because
there was
nowhere to record a span; it had no callers left afterwards and is gone.
Two
deviations from upstream, both about what the string is for: every
cell's text
is escaped, where upstream interpolates it raw and a cell holding a `<`
produces
markup that is no longer a table, and the span attributes are written
only where
there is a span, where upstream emits `<td >` for every ordinary cell.
**A picture of where each chunk came from.** A chunk is a few hundred
words with
a page number on it, which is enough to cite and not enough to answer
whether
the parser read the page correctly. So every PDF chunk that knows where
it sits
carries one image: the regions it covers, cropped out of the rendered
pages and
stacked. here they are already structured in the parser's element spans,
so
the markers would only put text nobody wrote into the thing being
embedded.
The panel draws a chunk's tables as tables, parsed and
rebuilt as elements rather than injected: a person can rewrite any piece
from
that same panel, so what comes back from the store is user input however
it got
there.
**The page that reads them.** Upstream retired the knowledge page while
this
branch was out; it is rebuilt against the new architecture, and the half
that
never existed is the one a reader needs -- what a document actually
became once
it was cut and embedded. Documents sit in one-level folders and arrive
by four
routes: a file, a folder, a URL, a written note. Adding and indexing are
two
calls on the contract and nothing drains that queue on its own, so a
page that
stopped after the first left every upload saying Queued for good.
Opening a file takes the whole page: the document on the left, the
pieces the
index holds of it on the right. The left half is the file drawn as its
own
pages rather than framed by the browser's PDF viewer, which reads where
to open
once and answers to nothing afterwards -- and which nothing can be drawn
on.
Pages are rendered once and kept, and the region a piece was cut from is
washed
over the page it came from, placed in fractions of the page so the
picture's
own scale never enters the arithmetic. Clicking a piece takes the file
there;
double-clicking opens it for rewriting.
The pieces can be searched, filtered by state, turned on and off,
deleted,
rewritten and added to, one at a time or in a batch. Every one of those
was
already on the contract with no reader for it. A base's settings are
back and
fuller: how many pieces a search returns, how documents are cut, and how
much
prose comes with a table or a figure. The embedding model is a picker
while the
base holds no pieces and a stated fact once it does -- moving it then is
the
collection made again at the new width and every document re-cut, which
is not
a thing a panel does by being saved.
**An agent can read them.** The retrieval engine was there and nothing
could
reach it: no agent had a tool that opened a library. A reader now points
a
conversation at bases from the composer, the way they pin a working
directory,
and the selection rides in the same place -- the session's metadata,
bound
around the turn.
The tool takes a query and never a base. The scope is the reader's
choice, so a
model cannot reach material nobody offered it, and it is withheld where
nothing
was picked rather than offered and refused. A hit says which document
and page
it came from, because a hit a reader cannot place is a hit they cannot
check --
and the passages are pressable in the transcript, walking into the base,
the
document, the folder it is filed under and then the piece. A citation a
reader
cannot follow is a claim they have to take on trust, and the whole point
of
searching their own material is that they can go and look.
**The word segmenter and the resources.** `raven.core.Tokenizer` is
Infinity's
`rag_tokenizer`, ported: a forward and backward maximum match over a
dictionary
trie, with stemming and lemmatisation on the latin path. XGBoost is not
carried
over. None of the data lives in git -- the dictionary, the NLTK corpora
and the
ONNX graphs run to about a hundred megabytes, so `make fetch-resources`
on an
install and an optional Docker layer behind `RAVEN_RESOURCES` both point
at one
script. A checkout that could not reach the network still works, and
every
feature that reads this data says so when asked.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Run on macOS against Python 3.12, with the deepdoc weights fetched (the
model
tests skip without them):
```
uv run pytest 27450 passed, 111 skipped, 3 failed
npm run test --prefix ui-web 3276 passed
npm run test --prefix ui-tui 2106 passed
npm run type-check --prefix ui-web clean (tsc)
npm run lint --prefix ui-web clean (4 pre-existing hook warnings)
make lint-python clean (ruff check + format)
make lint-types clean (ty)
make lint-imports 10 contracts kept, 0 broken
make check-source-language clean
make check-large-files clean
make build-ui built, both boot snapshots match
node ui-web/scripts/check-css.mjs clean
node ui-web/scripts/check-class-namespace.mjs clean
```
The three Python failures are not this branch's and were each confirmed
on the
base alone: one asks a host without LibreOffice to refuse a render and
this host
has LibreOffice installed, and two simulation cases fail on an
unmodified
checkout of the base commit.
The RPC contract was regenerated for both consumers and both match it
(`npm run gen --prefix ui-web`, `npm run gen:rpc --prefix ui-tui`, then
`gen:check`). The page's boot snapshots and region goldens were
refreshed where
the composer bar gained a button, and each records that button and
nothing else.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
A PDF's tables, and a Word or PowerPoint file's, are now indexed as HTML
rather
than as header-prefixed lines. Chunk text changes shape for those
documents, so
a base indexed before this and searched after it holds both forms until
its
documents are reindexed; retrieval keeps working either way, because
both forms
are text. The same applies to how much prose a table or a figure carries
with
it: the setting now reaches as far as it says, where before it stopped
at the
nearest paragraph, so a chunk cut before this keeps the narrower context
until
its document is rebuilt. Rolling back is reverting the branch and
reindexing.
The deepdoc models are optional. An install that never ran `make
fetch-resources` reads PDFs geometrically, exactly as before, and says
so when
the models are asked for by name. Nothing here reaches the network at
request
time.
Chunk crops and rendered pages are written at index time and on first
view, both
under the state directory rather than in the repository, and both are
deleted
with their document. A page is immutable for the life of its document
id, so
the route serving them caches; the document route does not. All three
routes
take ids and no path, and the two image routes answer under the
strictest
sandbox in the tree.
What an agent may search is the reader's choice and nothing else. The
tool takes
no base argument, so a model cannot name one, and it is absent from the
schema
entirely for a conversation that named none -- a lane nobody pointed at
a base
is unchanged by this branch. The bases a session carries are stored
beside its
working directory and travel no further.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
## Summary
Make `MemoryBackend` the only seam between the host and the memory
subsystem, so a second memory plugin can be installed without editing
host code.
The abstraction is the goal; no second backend is being added.
What the branch does:
- `health()` joins the Protocol. `raven doctor` and `raven import` read
a `BackendHealth(ready, checks)` the backend fills in with its own
vocabulary, instead of reaching into EverOS internals. Doctor prints the
checks verbatim and never interprets a label.
- A new `[[plugin.contributes.onboard]]` kind, with
`raven/contracts/onboard.py` as its paper. The host lends the wizard
shell as one `OnboardUI` and records the result; the screen returns a
`StepOutcome`. The plugin never writes `memory.backend` -- that key
stays the host's. EverOS's step-4 screen and its `everos.toml`
management move into the plugin.
- The skill source and the after-turn feedback prefix are named from
`memory.backend` rather than the literal `everos`.
- `enabled_by_default` is gone from the manifest: an installed plugin
the user has not disabled is active.
- `tests/test_plugin_boundary.py` pins the dependency direction in both
directions, with an equality assertion so the remaining exceptions
cannot grow.
That guard's allowlist started at five host files still speaking
EverOS's wire protocol. The follow-ups below close three of them.
Two things this branch itself introduced were found in manual acceptance
on an install without the plugin and fixed here: a turn printed which
backends were installed -- empty on exactly the install that needs the
sentence -- and `settings.everos` / `settings.everosSet` let
`ModuleNotFoundError` reach the dispatcher as an internal error carrying
a traceback. Both now carry the note doctor and the wizard already use,
naming the distribution to install.
Review on this PR found four blockers, all fixed and covered. A fifth
defect came out of manual acceptance rather than review, and four more
out of the rounds that followed it -- every one of those in the
embedding endpoint's own plumbing, each found by the round after the fix
that introduced it. They are listed under Risk; the blockers are:
1. `onboard` is an optional contribution, but the wizard equated "no
matching screen" with "not configured", so one `raven onboard
--skip-memory` wrote `memory.backend` back to null for a third-party
backend configured by hand. A backend with no screen now answers on the
strength of having been selected; only one whose own screen says it is
unconfigured is cleared, which is the case that rule exists for.
2. The record a sub-agent call leaves behind ran through a backend
nobody had started. The shipped adapter begins in an unknown state, so
the first flushing write only scheduled a readiness probe and answered
false -- the record said "unavailable" while the service was up the
whole time, and each per-record HTTP client was left open. One context
manager now owns start and stop around a record, in both the manager and
the DAG runner.
3. That record's write and the recall after it addressed different
owners: a real config spells its keys in camelCase and the block reaches
the backend as written, while the adapter read only the contract's
snake_case, so a sub-agent's memories were filed under the host
identity. The adapter reads both spellings.
4. Legacy embedding discovery had narrowed to the recorded plugin root,
so an install predating root recording lost its knowledge-base endpoint
with a valid file still on disk.
**The five follow-ups the seam left behind are in this PR too**, rather
than in three stacked ones, because they are one goal and the branch
protection dismisses an approval on every push:
- The embedding endpoint is raven's. A knowledge base indexes inside the
gateway process and never speaks to the memory service, yet it read its
endpoint out of everos.toml, so selecting another backend left it with
no vectors and no message naming the cause. Raven holds the block now
and hands it to the backend at start -- as a shared default, not an
override: a plugin that was given an endpoint of its own keeps it, and
only one that has none takes raven's. Discovery of an existing
everos.toml still checks all three places the plugin itself looks, read
with plain tomllib so it works where the plugin is not installed. No
data migrates: a base is keyed on its model and measured width, not on
the base url.
- `delete()` joins the Protocol. The browser used to delete a row out of
EverOS's index, in the host process, through the plugin's repository
layer -- while the markdown EverOS treats as the source of truth kept
the text, so the memory came back on the next rebuild, after the user
was told it was gone. The adapter now uses only operations EverOS
already performs, for the kinds it performs them for.
- Sub-agent memory goes through the contract. The read half is a new
optional `recall_session`; the write half needed no new surface, only
two metadata conventions written down. `agents.<name>.config.everos`
becomes `config.memory`, opaque to the host except the two keys the host
reads; the old spelling still loads.
- The memory and settings pages say why they are empty -- no plugin, or
a backend this page does not read -- instead of four zeros and a retry
button that cannot help.
- The `everos` package leaves the host's dependencies for the plugin's
own, at the same exact pin; the host reaches the tree through the
plugin. This is the package layer only: the installers already deliver
`everos-memory` as a sibling wheel from the same release, so an ordinary
install still has long-term memory.
The boundary guard's allowlist falls from five entries to two, both GUI
surfaces still shaped around EverOS's own store. Generalising their wire
is a frontend change and is deliberately not in scope.
This branch was reviewed on GitLab as merge request 639 over three
rounds, including a post-rebase range-diff check. It is being moved here
because GitHub is now the source of truth.
## Type
- [x] Refactor
## Verification
- `uv run pytest` on this branch, rebased onto `main`: 22146 passed, 4
failed, 45 skipped. Three of the four (`test_agents_oncall_launcher`,
`test_agent_loop_token_budget`, `test_agents_code_launcher`) were re-run
on a detached checkout of that same `main` commit in this environment
and fail there identically. The fourth (`test_ppt_engine_image_search`)
could not run in that checkout, whose environment lacks the deck
engine's extras; this branch's only change anywhere under
`plugins-dist/ppt-engine` is the removal of the `enabled_by_default`
manifest key, which that test does not reach. So this branch adds no
regression.
- `npx vitest run` in ui-web: 1680 passed across 103 files. `npm run
type-check` and `npm run gen:check` clean, so the generated client
matches the contract.
- Every fix in the follow-up commits was mutated away and its test
confirmed red, then restored and confirmed green. For the record path
that is three separate mutations: dropping `start()`, dropping `stop()`,
and narrowing the owner lookup back to one spelling.
- `uv run ruff check .` and `uv run ruff format --check` on the changed
files: clean.
- `uv lock --check`: consistent.
- `scripts/check_commit_messages.py`, `scripts/check_large_files.py`,
`scripts/check_source_language.py` over the branch range: clean.
- Manual acceptance against a real isolated install (temporary
RAVEN_HOME, isolated EverOS root and port), 34 cases: `raven plugins`,
`raven doctor` across four backend states, `raven onboard` interactive
and non-interactive including all three step-4 lanes, `raven import`
refusing and importing, an end-to-end write and recall, the web UI's
memory and settings pages driven through a real browser, a sub-agent
trace record, and an install with no memory plugin at all. 31 passed; 1
failed and was fixed; 1 partial; 1 carried over from the earlier round.
- The failure is the one worth reading. On a fresh managed install the
wizard reported embedding configured and the service it started the next
moment ran keyword-only: the spawned process carried no
`EVEROS_EMBEDDING__` values, and EverOS logged that the embedding key
was not configured. The rule deciding whether everos.toml already held
an operator's own endpoint asked whether its model name was a `"<...>"`
placeholder; the shipped template seeds every section with a real model
name and an empty key, so every fresh install read as a deliberate
choice and the host's endpoint was never bound. It now asks the
criterion the module already had -- model and key. The unit tests missed
it because each built its "operator's own" section with a key in it; the
regression copies the shipped template itself.
- The partial: the memory page's search returned relevant results
without error, but every memory in the test corpus was on one subject,
so narrowing could not be observed. The carried-over case is the
skill-source prefix, whose path this branch does not touch.
- Findings recorded and not fixed here, none introduced by this branch:
a progress spinner is still drawn after `import` reports it imported
nothing; an unsupported delete surfaces as a generic internal error; the
TUI log and `doctor` word "no memory plugin" differently; a turn that
did reach disk still prints that it was not written.
- The partial case: an induced agent skill is discovered through
`find_skill` with the backend's prefix, but `read_skill` / `use_skill`
on that id fail. Nothing writes the on-disk layer those tools look in,
on this branch or on `main`, and no test covers it. Pre-existing; filed
as a follow-up.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
User-visible changes:
1. User-track recall is now strictly bounded by `top_k`. A profile row
used to ride outside that bound.
2. `raven doctor` builds the plugin factory, so a managed EverOS setup
writes its env var and config templates during a doctor run. It used to
be read-only. The `--json` memory section is reshaped around
`health.checks`.
3. A third-party drop-in plugin without `enabled_by_default = true` used
to be skipped and now loads.
4. The loader no longer back-fills the EverOS config slice's `root` /
`owned`, nor strips a legacy `mode` key; a leftover `mode` now draws an
unknown-key warning.
5. The wizard's recap shows the contribution name `everos` rather than
`EverOS`.
6. `use_skill` with an unknown prefix reports "not found" after a disk
lookup instead of "unknown source".
From the follow-ups:
7. Config gains an optional top-level `embedding` block. Absent, the
knowledge base reads the EverOS file as before and says so once. An
operator who moves the values across changes nothing else; one who names
a *different* model there gets the existing `StaleBaseError` telling
them to rebuild that base. The block is a default for the memory plugin
rather than an override, so an install that had configured the plugin
separately keeps what it had. Note that a config carrying this block
does not load on an older Raven, which rejects unknown top-level keys.
8. `agents.<name>.everos` is written back as `agents.<name>.memory`. The
old spelling still loads; a per-agent `baseUrl` is ignored with a
warning, having never had a user.
9. Deleting a memory now changes what EverOS treats as the source of
truth, so it stays deleted. Profiles and cases report that they cannot
be deleted rather than appearing to succeed.
10. `pip install raven` no longer pulls the memory engine in.
`install.sh` and `install.ps1` are unaffected: both already resolve
`everos_memory-*.whl` from the release they install and pass it as
`--with everos-memory`, and the source path installs it editable, so
those installs keep long-term memory. An install that reaches neither --
a release predating the sibling wheel, or a bare `pip install` -- now
boots without memory and says which distribution to install rather than
failing on an import.
Found after the acceptance run, in review, all fixed here. They share
one cause worth stating once: the endpoint acquired a second home, and
each fix taught one more reader about it while leaving the next one
behind.
11. The endpoint the wizard saved never reached the EverOS process it
then started. The rule for "does everos.toml already hold an operator's
own endpoint" asked whether its model name was a `"<...>"` placeholder;
the shipped template seeds every section with a real model name and an
empty key, so every fresh managed install read as a deliberate choice
and the host's endpoint was never bound. The service came out of the
wizard running keyword-only while the wizard said embedding was
configured. The rule now asks the criterion the module already had:
model and key.
12. The exported `EVEROS_EMBEDDING__*` variables are a third home,
documented by EverOS and ranked above its file. They were being
overwritten. A complete triple is now an operator's choice; an
incomplete one is not an endpoint and is replaced whole rather than
topped up from the host.
13. Raven writes those same variables itself at startup, so a reader
that asked only whether all three were set called raven's own binding an
operator's export -- a blank settings card and a save refused by naming
variables nobody had set. The module now records which of them it set.
14. That record lived in a module-level set, and `raven gateway
--restart` re-launches through `os.execv`: the values survive, module
state does not. The record now travels in a variable beside them, and is
stripped from the environment handed to EverOS.
15. A settings save the environment would outrank is refused with the
variable names, rather than written and silently ignored.
Four of the five arrived one per review round, each in the fix for the
one before it. The tests that missed them all shared a shape: each built
its "an operator configured this" fixture with a key in it, or cleared
the environment before the case, so the states the product actually
produces -- a template with no key, a live process that has just bound
its own endpoint -- were never the input. The regressions now copy the
shipped template, and run the startup sequence and a second interpreter
rather than a prepared state.
Rollback is a revert of the branch: the seam is additive, and
`memory.backend` keeps its shipped default.
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: gloryfromca <23442919+gloryfromca@users.noreply.github.com>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
## Summary
<img width="2518" height="1798" alt="image"
src="https://github.com/user-attachments/assets/9bb2dbb6-fe29-42ff-8dba-2e035557b254"
/>
Turns the knowledge page into something a reader can actually run a
knowledge base from: put material into it four ways, look at what is in
it, test what it answers with, and configure it.
**Data sources.** The panel had one button that uploaded one file. It
now offers file, folder, note and URL behind one menu, plus a drop
target over the whole panel. A picked folder is walked by the browser; a
dropped one is walked here, because a drop hands over a directory entry
rather than files. Either way the files are filtered to what this build
can index, capped, and uploaded one at a time: each carries its bytes
base64 in a single websocket frame under a 25 MB ceiling, so N at once
is N of those in memory. One file failing no longer takes the rest of
the folder with it.
A note and a fetched page are stored as the markdown they were captured
as, in the same blob store as every upload, so nothing downstream learns
a second shape. Reading a page follows Cherry Studio's own capture: Jina
Reader, asked for JSON so the page title comes back as a field rather
than as a preamble to parse out of the text, and through
`validate_url_target` first, because the URL arrives from a browser and
without that check the gateway is a request forger.
**Recall Test.** Asks a base a question and shows what comes back: the
chunk, where in its document it sat, its score beside its rank, and what
the search cost. An explicit button rather than search-as-you-type,
because every keystroke used to spend a billed embedding call. Past
questions are kept in localStorage and nowhere else.
**Settings.** Each base carries its own Top K and chunking, with a
sentence on every field saying what moving it costs. The embedding model
is shown and not offered: the collection is sized to its vector width at
creation, so changing it is a rebuild of every vector rather than a
setting, which is what the stale-base check exists to detect. Letting a
reader change it is a separate piece of work.
**Two defects found while measuring the above.**
SiliconFlow refuses an entire request when one input is over the model's
token limit, naming neither which input nor what was wrong with it. A
document of eighteen chunks where seven were long indexed none of the
other eleven, and four documents in a real base were failing this way.
The cause is the chunker's token estimate -- utf-8 bytes over four, an
English-prose rule that reads a CJK character as three quarters of a
token when it is one, and a URL as a quarter of what it costs.
Binary-searching the longest accepted prefix of real documentation pages
against the endpoint put the true figure at 2.1 to 2.3 characters per
token. A vendor client now cuts each input to fit before sending; the
four documents embed whole.
Search reported one number for its cost. It now reports two, because
they answer different questions: embedding is a round trip to whatever
endpoint is configured and runs to hundreds of milliseconds, while the
store query is the index doing its job. A surface reporting one figure
as the search time wants the second, or it is describing the provider.
**Deliberately not wired yet.** File processing lists the processors it
will offer with every row disabled and saying why it is unavailable;
smart chunking, separator, chunk size and overlap are stored on the base
but not yet read by the chunker. Both are placeholders with their state
stated rather than controls that would be silently ignored.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Run on this branch rebased onto `main` at `4f5d7dac`:
- `uv run pytest -q` -- 22151 passed, 45 skipped
- `npx vitest run` in `ui-web` -- 1738 passed (103 files), of which 92
cover this page
- `npx vitest run` in `ui-tui` -- 2072 passed
- `npx tsc --noEmit` in `ui-web` and `ui-tui` -- clean
- `uv run ruff check raven/ tests/` and `ruff format --check` -- clean
- `node scripts/check-page.mjs`, `check-css.mjs`,
`check-class-namespace.mjs`, `count-shared-globals.mjs` -- all OK
- `make check-large-files` -- OK
- The SiliconFlow client was measured against the live endpoint, not
mocked: batch size is not the limit (64 short inputs pass), 500 CJK
characters pass and 512 do not, and all four previously failing
documents in a real base now embed whole.
- Every panel was checked in the browser in both light and dark themes.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
User-visible behaviour changes, all additive:
- A search that names no `top_k` now answers with the base's own setting
instead of a hardcoded 5. New bases default to 6.
- Documents carry an `origin` and bases carry six settings. Every field
is defaulted, so a registry written before them loads with the behaviour
it already had; an older gateway answering without them is read as those
same defaults. A field from a build *after* this one is ignored rather
than losing the row, which is the same protection in the other
direction.
- The SiliconFlow client truncates an over-long input rather than
failing the request. A cut chunk's vector speaks for its beginning; the
chunk is still stored and shown in full when retrieved. The proper fix
is chunking to the model's real limit, which this does not do.
- Four brand-logo SVGs are added under `ui-web/src/assets/providers/`
(16 KB total) and spared the dark-mode inversion that monochrome vendor
marks get.
- Previewing an office file retains a suffixed copy of its bytes in the
render cache, because the converter picks its input filter from the
extension and a blob is stored under a bare id. Deleting the document or
its base now removes that copy, and the cache sweep reaches it as a
backstop for one whose owner never got the chance.
Rollback is a `git revert` of the squash commit; nothing migrates on
disk.
A registry written by this branch stays readable by the previous build,
and that took work: its loader hands every stored key to a dataclass
constructor and drops the row when one is a keyword that constructor
does not take, so a new field written beside the record would make every
base and document vanish from a rolled-back binary, with the records
still on disk and nothing listing them. The settings and the document
origins are therefore written under two sibling keys of `bases` and
`documents`, which a loader that does not know them does not read; a
record that uses none of them is written exactly as before, down to its
keys. What a rollback loses is the settings, which that build has no use
for. What it keeps is every base and every document. A test pins the
stored shape against the field set the shipped loader accepts, rather
than against our own reader.
Security: the URL source takes an arbitrary address from a browser and
fetches it server-side, so it goes through the same
`validate_url_target` guard the web tools use, and there is a test
asserting a refused target never reaches the network.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: admin <admin@SH-HuKai.local>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
## Summary
Moves what a conversation defaults to onto a page of its own, under
Agent, and adds three model choices beside it. The providers page keeps
the accounts and their model lists, and is renamed Model providers to
say what it is now.
Five rows on the new page:
- Agent model -- the default binding, moved here with how it answers.
- Quick model -- the short errands beside a conversation, today the call
that names a new session.
- Translate model -- reserved. Nothing calls it yet, and the row says
so.
- Painting model -- the image tool's model and quality, moved out of the
tool row it was folded inside.
- Embedding model -- what knowledge bases are embedded and searched
with. New.
Each of the three new rows writes a model and the provider serving it,
because an id alone does not name a credential: `openrouter` +
`anthropic/claude-haiku-4-5` and `anthropic` + `claude-haiku-4-5` are
both valid and bill different accounts. That is the shape every
subsystem pin in the config already states.
A defect fixed on the way. `sessionTitle.model` had no provider half, so
a pinned naming model was sent on the conversation's provider and key --
exactly the mis-pairing the pin convention exists to prevent. It now
binds the pair through the provider pool and follows the conversation's
model when the pair cannot be built, reported once. A config that sets
only a model keeps working, since the pool derives the vendor from the
id.
The embedding pin is the first time raven decides where a knowledge base
embeds. Set, it wins; unset, the EverOS endpoint answers exactly as
before, so no existing install changes behaviour. Provider credentials
resolve through a new public helper in `update_providers`, which is
already the one place that knows what address a request goes out on,
rather than a second answer to that question.
Two decisions worth a reviewer's attention:
- Painting is not a pin. The image tool carries its own key and address
instead of borrowing a provider account, so giving it a provider half
would mean re-plumbing the media stack. The existing model/quality
picker moved as it is, and the row says how it is reached.
- Each slot's offer is filtered by what the slot is for, using the same
bucketing the backend does (`registry_data.kind_of`). A picker that
lists every model for every slot is how an embedding id lands in a
painting slot; for the embedding slot the cost of that is a vector store
that cannot be searched. Providers that are not connected are left out,
since picking from one fails on the next call rather than at the click.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
`make coverage`: full suite green.
`npm test --prefix ui-web` -> 103 files, 1681 tests. `npm test` in
ui-tui -> 143 files, 2072 tests. Type-checks, `lint:rpc`, `lint:i18n`
and ui-web's `gen:check` all clean, and `ruff check` / `ruff format
--check` pass.
New tests cover the embedding pin (a complete pair wins, half a pair
does not, an uncredentialed provider falls back, no pin behaves as
before), the quick pin (both halves reach the pool, a refused pin
follows the conversation), and the slot filtering. Each was confirmed to
fail with its fix removed rather than assumed to cover.
`check-page` fails on this branch. It is the two-versus-three script
block mismatch already failing on main, fixed in a separate PR;
repeating the fix here would collide when both land.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
## Risk
User-visible: the settings panel gains a Default models entry under
Agent, Models is renamed Model providers, and the agent model,
how-it-answers and painting controls are reached from the new page
rather than where they were.
Behaviour: an install that sets none of the new pins runs exactly as
before. The one behaviour change is the session-title fix above, which
drops a pin that cannot be paired instead of sending it on the
conversation's credential.
`settings.set` gains six writable keys. The whitelist is pinned
wholesale by a test so this had to be a deliberate edit: the pins name a
model and a provider the deployment has already credentialed, carry no
address or key of their own, and reach no endpoint the operator did not
already choose.
Rollback: revert the squash commit. No migration, and no config is
rewritten.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: admin <admin@SH-HuKai.local>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Co-authored-by: KT <74288668+0xKT@users.noreply.github.com>
## Summary
Reported as "deliver_files is broken". It is not: **the tool never ran.**
Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:
```
Error: Invalid parameters for tool 'deliver_files': files should be array
```
What the model sent was the array, spelled as a string:
```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```
`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.
## Type
- [x] Fix
- [ ] Feature
## The edge of the forgiving is the design
When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.
**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.
**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.
## Not just deliver_files
Six built-in tools declare an array or object parameter and were all exposed:
| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |
The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.
## Verification
```
make lint-python ruff check + ruff format --check, 1850 files
make lint-imports 9 contracts kept, 0 broken
pytest tests/ 19,699 passed, 7 failed (all pre-existing, see below)
```
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string
**End to end, through the real tool**, with the exact payload that failed five times in production:
```
files sent as: str -> [{"path": ".../poster.png", "title": "..."}]
returned: Delivered 1 file(s): poster.png (72 B).
still "should be array": False
registered in the store: True
```
Twelve cases, five mutations, each killing at least one:
| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |
The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.
**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.
## Risk
- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert
## Related Issues
N/A
## Swept
`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.
- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.
Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).
## Deliberately not fixed
The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.
## Moved from !594
Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.
Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
## Summary
Overhaul the interactive `raven trajectory` browser end to end, fixing
every
issue from real-terminal testing:
- Navigation: every screen opens with a help area under its title (page
purpose plus "[Key] function" hotkey hints, rendered in a new readable
help color). Esc walks one level up (action -> attempt -> session); on
the
top-level session screen Esc stays put and quitting is Ctrl+C only,
which
prints a single exit notice. Esc inside a running action cancels just
that
action through a dedicated exception at the action boundary, so the back
sentinel can never reach a conversion or the data layer; a cancelled
report keeps its pre-confirm bundle-and-pin side effects (the existing
declined-report contract). Back/Exit menu entries are gone; answered
prompts erase themselves and navigation is echoed as breadcrumb lines
with markup-escaped untrusted text.
- Tables: the session list is TITLE | ATTEMPTS | LAST ACTIVITY and the
attempt list is # | STARTED | TURNS | SPANS | VERDICT | PIN | MERGED |
PREVIEW with green check cells. Time cells keep the year. The layout is
budgeted per screen build from the live terminal width (row indent, a
safety margin, and inter-column gaps are all part of the math); PREVIEW
drops as a whole column when it no longer fits, and below the dynamic
minimum width the tightest layout is kept while the renderer clips
overlong lines at the terminal edge. Every dynamic cell passes a
single-line sanitization gate before width math (verdict sidecars only
guarantee a non-empty string), cells are measured and truncated by
terminal display width (CJK is 2 cells), and fixed column widths always
fit their headers.
- Space preview: Space on an attempt row prints that attempt's per-turn
input/output previews, collected in the same snapshot scan and sorted by
a fully-stringified key so ordering never follows log source order (the
table PREVIEW cell derives from the same sorted collection). Any key,
arrow keys included, returns with the cursor kept on the row; the waiter
passes the terminal's CPR reply through to the renderer instead of
treating it as a keypress, and Ctrl+C keeps the browser-wide cancel.
- Merge hotkey: with two or more attempts, m/M opens the multi-select
merge
(advertised in the help line; the old "Merge attempts" menu entry is
gone). The checkbox screen carries its own help line for toggle,
confirm,
and cancel. A single attempt neither binds nor advertises the key.
- Theme and feel: new success (green check) and help (readable hint
text)
colors in both palettes, all WCAG AA on their backgrounds; a lone ESC is
flushed after 50ms instead of prompt_toolkit's 0.5s default, removing
the
laggy Esc feel.
The data layer is untouched: changes are confined to
raven/cli/trajectory_browse.py, raven/cli/_theme.py, and their tests.
The
id-taking subcommands (raven trajectory save/list/...) behave as before.
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
- `COLORTERM=truecolor uv run pytest tests/test_cli_theme.py
tests/test_cli_trajectory_browse.py
tests/test_cli_trajectory_commands.py -q` -> 220 passed
- `COLORTERM=truecolor uv run pytest tests/ --ignore=tests/integration
-q` -> 6917 passed; the 10 failed + 20 errors are the pre-existing
environment-dependent set, identical file-by-file on a clean main
checkout (cron, everos server, config loader, tracing viewer)
- `uv run ruff check` and `uv run ruff format --check` on the four
touched files -> clean
- PTY smoke on a CPR-answering pseudo terminal at 100 and 58 columns:
full flow (preview, merge cancel, action screen, staged Esc backout,
Ctrl+C quit) and the narrow-width degradation contract
- New coverage includes real pipe-input key-binding tests (Esc on all
four prompt kinds, Space/m/M hits, waiter key paths with CPR
pass-through), table budget/clipping boundaries at the computed minimum
width, and corrupt verdict/timestamp corpora
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
- The interactive browser's look and key protocol change wholesale (that
is
the point of this PR): Esc no longer quits from the top level (Ctrl+C
does), Back/Exit entries are gone, and lists render as tables. The
machine-facing subcommands and the trajectory data layer are untouched.
- Rollback: revert the squash commit; the browser returns to the
previous
interaction wholesale, no data migration involved.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: 江国庆 <guoqingjiang@deepglint.com>
Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
## Summary
Found while investigating the first row of Dizhan's `子智能体功能与显示问题` doc (*回注内容过程中未正常显示,刷新后才会正常显示*). This is not that bug's fix, but it is a real defect on the same wire, and it is the one that explains why "前后端接线没有约定好" was a fair thing to say.
**`turn.started` is not in the `TurnEvent` union — on either side.** It has a schema component, it has a Pydantic model, and neither is reachable from the union that `tests/test_rpc_schema_match.py` walks. The 290-test contract guard has never looked at this event.
So it drifted the first time anyone touched it. `05957452` (judge a dag node) taught `announce_dag_exception` to emit:
```python
mark = {"kind": "dag", "label": run_id, "status": "exception", "run_id": run_id, "node_id": node_id}
```
while `TurnStartedEvent.payload.delegated` still declared `status: enum ["ok","error"]` with `additionalProperties: false`, and `TurnStartedPayload` had **no `delegated` field at all** under `extra="forbid"`. Verified with `jsonschema` against the published contract before changing anything:
```
Additional properties are not allowed ('node_id' was unexpected)
```
That commit did carry `exception` into `TranscriptDelegated` and `SubagentDeliveredEvent` — two of the three places this status lives. The third is the one no test covers.
**It does not crash today**, and I want to be exact about that: `SubscriptionEmitter.emit` pushes a raw dict with no validation, and the page reads only the fields it knows. But the contract test's own docstring says what the cost is — *"both sides declare `additionalProperties: false` / `extra="forbid"`, so a consumer validating against the schema rejects the whole event, not just the unknown field"*. Losing this event loses the boundary that advances the workspace turn counter and draws the delegated row.
## What this does
- `TurnStartedEvent` joins the `TurnEvent` union, in `openrpc.json` and in `raven/rpc/models.py`, so the existing sweep covers it from now on.
- The schema's `delegated.status` gains `exception`; `node_id` is declared.
- `TurnStartedPayload` gains `delegated` as a typed model (`TurnStartedDelegated`), three-valued status included.
- Both generated TypeScript clients regenerated.
- A new test drives **the exact marker `manager.py` builds** through both validators, so the next status or field fails here instead of in a client nobody is watching.
## One thing the union surfaced
`ui-tui` never handled `turn.started`. Its exhaustiveness check said so the moment the variant existed. It draws the delegated row from `subagent.delivered`, which the page deliberately ignores in favour of this event — the two mark different moments: **submitted** versus **started**.
Left as it is, with an explicit `case 'turn.started'` saying why. Moving that surface onto the truer moment changes when the TUI draws the row, which is a product change, not a contract fix.
## Verification
- New test red before the fix, by construction: adding the variant to the union alone reds `test_schema_match_turn_event_discriminated_union` and `..._payload_fields` (schema/pydantic drift), which is how the gap was proved.
- Mutation checks, each reverting one thing:
| reverted | reds |
|---|---|
| `exception` out of the schema enum | 1 |
| `node_id` out of the schema | 1 |
| `delegated` off the Pydantic payload | 2 |
- `uv run pytest -q` — **12900 passed**, 50 skipped, 5 failed; the same five pre-existing on `origin/main` (four `test_security_network.py` NAT64 cases, one `test_doctor_answers_where_the_memories_are`), reproduced on an untouched merge-base worktree.
- `uv run ruff check` / `ruff format --check` on `raven tests scripts` — both clean (both halves of `make lint-python`).
- `ui`: `gen:check` 147 methods, `type-check` clean, `npx vitest run` 89 files / **1225 passed**.
- `ui-tui`: `type-check` clean, `lint:rpc` in sync, `lint:i18n` up to date, `npm test` 138 files / **1990 passed**.
- Worth recording: with `ui-tui/node_modules` symlinked from another checkout the suite reports 82 failed / 59 failed — and reports **the same numbers on an untouched `origin/main`**. It is the symlink, not the branch. Ran `npm ci` in the worktree to get a trustworthy number.
- `scripts/commit_lint` — no errors.
- `git ls-files | grep -c node_modules` — 0.
## Type
- [x] Fix
## Risk
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear
Widening a contract to match what the runtime already emits: every payload valid before is valid now. No emit site changes, no behaviour changes. The one code change outside the schema and the models is an explicit no-op branch in `ui-tui`. Rollback is a revert.
## Summary
Unit tests were leaking tracing spans into the real `~/.raven/traces`
store. Tracing is on by default and `trace.span()` calls are embedded in
library code (agent loop, subagents, TUI RPC), so any test exercising
those paths emitted spans with fabricated session keys (`session-a`,
`weixin:c`, `tui:default`, ...). Those spans then surfaced as phantom
sessions in `raven trajectory`, while the TUI `/sessions` list (which
reads the real session store) stayed clean.
This PR adds an autouse fixture in `tests/conftest.py` that:
- redirects `RAVEN_TRACING_DIR` to a per-test tmp directory, and
- resets the lazy module-level `TraceStore` cache
(`raven.tracing.spans._store`) around each test, since it pins the
directory resolved at first emit.
Key decisions:
- Only the directory is redirected, not the enabled switch, so the suite
keeps exercising the real emission path with zero behavior change; leaks
land in tmp instead.
- This centralizes the setenv-plus-cache-reset pattern that 8
tracing/trajectory test files already perform manually. Those tests keep
working: their own `monkeypatch.setenv` runs after the fixture and wins,
and they already reset the cache themselves.
- Existing phantom sessions come from archived span logs written before
this fix; the fixture only prevents new leaks. One archive file mixes a
real session with test keys, so cleanup needs line-level filtering and
is left out of scope.
## Type
- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [x] Other
Test-suite hygiene (`test` commit type).
## Verification
- `uv run pytest tests/test_tracing_api.py
tests/test_trajectory_store.py tests/test_trajectory_bundle.py
tests/test_trajectory_replay.py tests/test_trajectory_redact.py
tests/test_cli_tracing_commands.py tests/test_cli_trajectory_commands.py
tests/test_cli_trajectory_browse.py -q`: 308 passed, 1 failed
(`test_viewer_health_true_for_our_viewer`, fails identically on
unmodified `main`, environment-specific)
- `uv run pytest tests/test_agent_loop_run_emit.py
tests/test_subagent_manager.py tests/test_tui_rpc_turn_send.py
tests/test_deep_research_tool.py tests/test_agent_loop_approval.py -q`:
109 passed; the real `~/.raven/traces` log did not grow during the run
- Positive check: running `tests/test_agent_loop_run_emit.py` with
`--basetemp` shows the previously leaked session keys (`cli:c`, `tg:c`,
`weixin:c`) now land in per-test tmp dirs
- `uv run pytest tests/ --ignore=tests/integration -q`: 6837 passed, 11
failed, 20 errors; the same failure set reproduces on unmodified `main`
(environment-specific: cron/tz, everos server, theme, config loader,
viewer health), zero regressions from this change
- `uv run ruff check tests/conftest.py` and `uv run ruff format --check
tests/conftest.py`: both pass
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
Docs box left blank: test-only change, no user-facing docs affected.
## Risk
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
Test-only change; no production code touched. One known interaction:
`RAVEN_TRACING_DIR` takes precedence over `RAVEN_HOME`, so a future test
that fakes `RAVEN_HOME` and expects traces under it must also set
`RAVEN_TRACING_DIR` itself. Rollback: revert the single commit.
## Related Issues
N/A
Co-authored-by: 江国庆 <guoqingjiang@deepglint.com>
Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
## Summary
A delivered artifact could not be opened from the page: the file viewer answered
403 and the transcript drew "Raven is confined to the workspace, so this file
cannot be opened here" over a delivery card it had just rendered.
The viewer's state-directory fence exempts the workspace subtree, on the stated
grounds that the default workspace lives inside raven's home and denying it
would refuse every file the agent itself wrote. Since agent home and the working
directory were split apart, a turn no longer writes there: it writes in
`~/.raven/tmp/<channel>`, a sibling of the workspace, inside the same state root
and outside the carve-out. So the fence denied exactly what the carve-out exists
to allow, one directory over, for every artifact produced since that split.
This exempts the working-directory root alongside the workspace, using the same
containment test in the same direction, so a working-directory root that happens
to sit above raven's home earns no exemption. `config.json`, `oauth/` and
`serve.json` are siblings of both and stay refused, which is what the fence is
for.
Worth noting for a reviewer: the bytes were already reachable. `/files/download`
serves a delivered file by token with no path fence, so this was never a
security boundary for deliveries -- only a viewer that refused to render what
the same page could already download.
## Type
- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
- `uv run pytest tests/test_rpc_transport.py tests/test_rpc_files.py tests/test_rpc_knowledge.py tests/test_rpc_console.py -q` -> 123 passed
- `uv run ruff check` and `uv run ruff format --check` on both changed files -> clean
- Both new assertions were mutation-checked, each against its own mutation:
- narrowing the carve-out back to the workspace alone reds
`test_the_turn_working_directory_is_exempt_from_the_state_dir_fence` only (1 failed, 3 passed)
- dropping the containment-direction guard reds
`test_a_working_directory_root_that_contains_the_state_root_earns_no_exemption`
and the pre-existing state-dir test (2 failed, 2 passed)
- Against the real reporting case: the two files that were answering 403
(`~/.raven/tmp/tui/*.md`, `*.pptx`) now resolve, while `~/.raven/config.json`
and `~/.raven/serve.json` still raise PermissionError.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
## Risk
- Widens what the file viewer will render by exactly one subtree: the directory
turns work in. Everything the fence was written to protect (provider keys in
`config.json`, the OAuth token store, `serve.json`) sits beside that subtree,
not inside it, and stays refused -- pinned by the tests above.
- `fs.list` / `fs.reveal` share the function but pass the session's working
directory as the root, which their own containment check already bounds, so
their behaviour is unchanged.
- Rollback is a one-line revert of the exemption tuple.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
## Summary
PR #360 reports every copy-on-select write as a permanent transcript
line, so a
reading session that drags through the transcript piles up "copied N
characters" rows under the conversation. The confirmation splits by what
the
line is for: the unbounded terse repeats become a transient notice
rendered
above the input box -- one line that replaces itself on every new copy
and
clears after 3 seconds -- while the first copy of a session keeps its
path
caveat in the transcript. The caveat is the answer a user comes looking
for
after a paste comes up empty minutes later, and it cannot pile up
because it
is once per session by construction.
The reporter still owns the per-session caveat tally and now returns
`{ text, firstOfSession }` so the caller can route the caveat to the
transcript and the repeats to the notice. A new `copyNoticeStore` holds
the
single notice; `showCopyNotice` keeps one dismiss timer, clearing any
pending
one before arming the next, so an earlier deadline can never clear a
newer
notice. `/copy` is deliberately untouched: an explicit command's answer
still
belongs in the transcript, and it cannot stack on its own.
## Type
- [x] Fix
## Verification
Run from `ui-tui/` at branch head, in a fresh worktree of `github/main`.
Dependencies came from `npm ci` in `ui-tui/` and in the hermes package;
the
hermes package's own `node_modules` holds only esbuild, so the ink
render
tests run against one react copy (two copies break every hook call).
```
npx vitest run --no-file-parallelism
Test Files 93 passed (93)
Tests 1084 passed | 13 skipped (1097)
npm run type-check
clean, exit 0
npm run lint
22 problems (0 errors, 22 warnings)
npx prettier --check src/app/copyNoticeStore.ts src/__tests__/copyNotice.test.tsx
src/__tests__/clipboard.test.ts src/app/useMainApp.ts
src/components/appLayout.tsx src/lib/clipboard.ts
All matched files use Prettier code style!
python3 -m scripts.check_commit_messages github/main..HEAD
exit 0
```
The 22 lint warnings are the baseline, not new: 22 before the branch and
22
after, none in a file this branch touches. The suite is run with
`--no-file-parallelism` deliberately: the ink render tests fail under
default
worker parallelism at this suite size, and no CI job runs them.
Suites that exercise the changed code directly:
`src/__tests__/copyNotice.test.tsx`
(7 cases), `src/__tests__/clipboard.test.ts` (30 cases, the 3 reporter
cases
updated to the new return shape).
New tests were proved load-bearing rather than assumed:
- Removing the notice render line in `appLayout.tsx` makes the render
test
fail (content and placement above the input box).
- Removing the pending-timer clear in `showCopyNotice` makes the
re-shown-notice test fail: the earlier 3s deadline then clears the newer
notice.
- The dismissed-notice case now shows a notice first, and removing
`dismissCopyNotice`'s atom clear turns it red (review fix).
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed
No doc change. `ui-tui/CONTEXT.md` has no clipboard or selection entry,
and
"copy notice" continues the existing "notice" vocabulary of
`copyResultNotice`
/ `copyOnSelectNotice`, so nothing new is coined.
## Risk
Behaviour change: the per-drag copy confirmation is now transient and
above
the input box instead of a permanent transcript line, and the first copy
of a
session lands one transcript line, exactly as on `main`. Reverting the
branch
restores the #360 behaviour; no data or on-disk format is involved.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: Claude (deepseek-v4-pro) <noreply@anthropic.com>
## Summary
A knowledge base is a named collection of documents the agent can search. This
adds the whole path: an embedded vector store, a registry for bases and
documents, parsers that cut on headings, an embedder that goes through the
endpoint the operator already configured, and a manager that makes the five
usable end to end. Nine RPC methods expose it, and the page in the rail --
between plugins and memory -- can make a base, add documents to it, watch them
index, search it, and empty it.
Two decisions worth naming:
- **The embedder measures its own width** rather than believing the config. A
configured number that disagrees with the endpoint does not fail; it produces
vectors that are silently the wrong shape, which is the kind of fault that
surfaces months later as bad search results.
- **Parsers cut on headings**, not at a byte budget, so a chunk is a passage
rather than whatever fell inside a window. A base whose chunks straddle
headings retrieves the tail of one section and the head of the next.
## Rebase note
This branch was 77 commits behind main. Two things did not survive the gap, and
both are worth a reviewer's attention because neither is visible as a conflict:
**`toast` left the shell bridge for its own module.** Five call sites in
`features/knowledge/store.ts` moved with it. Its two tests -- the ones that
assert a refusal says *why* instead of leaving a row that is not there -- were
capturing the message off a stub on the fake shell, which is no longer the
seam. They now read it off the page's `#toasts` host, which is what the module
actually does and a stronger claim than the stub made. Checked by mutation:
swallowing the reason in the store turns exactly those two red.
**The contract had been reformatted wholesale.** The branch's
`rpc-schema/openrpc.json` was 2465 insertions where the addition is nine
methods and three schemas -- every compact `"schema": { "$ref": ... }` line in
the file expanded. Taking that copy would also have silently dropped the two
schemas main added in the meantime. The file here is main's, with the knowledge
entries appended in its own style: **407 lines, all additions, no deletions**.
Both generated clients were regenerated from it rather than merged by hand,
which is the discipline the drift between `ui`'s generated client and the
contract exists to teach.
Net effect of doing it this way: 6166 insertions / 2 deletions, against
8212 / 607 before.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
```
pytest -q -> 11730 passed, 40 skipped
ui: npx vitest run -> 827 passed (70 files)
ui: npx tsc --noEmit -> clean
ui: npm run gen:check -> matches the contract (141 methods)
ruff check / format --check raven tests scripts -> clean, 1185 files
```
Not only unit tests: the page was opened against a real server on the preview
site after the rebase -- the rail entry, the create field, a base card with its
document count and embedding model, and no console errors. A green unit suite
says the store behaves; it does not say the page still mounts after 77 commits
of drift.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
- [x] Security impact considered. Documents are read into a local store and
embedded through the operator's own configured endpoint; no new outbound
destination is introduced. The embedder was argued onto the auth
allowlist deliberately, with a test that says why.
- [x] Backward compatibility considered. Nine new methods and three new
schemas; nothing existing changes shape, so an older client is unaffected
and a newer one degrades to "no bases" when the feature is unconfigured.
- [x] Rollback path is clear: one commit, and the store is a directory under
the raven home that can be removed.
## Related Issues
N/A