3 Commits

Author SHA1 Message Date
Dizhan Xue
24433cb7a6
fix(rpc): sign in a page reached through a port forward (#939)
## Summary

A page reached through a port forward could not sign in. The gateway
accepted a browser `Origin` only when it named the port the gateway
bound, and VS Code and `ssh -L` pick another local port whenever the
remote one is taken on the browser's machine, so every
`POST /auth/exchange` answered `403 bad origin` and every `/rpc`
upgrade was refused with it. The sign-in page reported an expired link
over a refusal it cannot see, and re-running `raven web` minted one
more link into the same refusal. Measured on 2026-10-13: 13 refusals
in six minutes, each from a page at `127.0.0.1:18793` against a
gateway on 18792, all within a minute of their nonce being minted. The
same check had already forced the container's nginx to rewrite `Origin`
for every request it proxies.

`_origin_ok` now also accepts an `Origin` that asks the request's own
origin (`Origin` equals `Host`) on a loopback name. That is what a
browser sends through a forward, since a forwarder relays `Host`
unchanged. It still refuses a page another local server shows, because
cookies and SameSite ignore the port and that origin names a port this
page does not ask; and it still refuses DNS rebinding, where a name
resolving to 127.0.0.1 asks its own origin and the two agree. The
bound-port allowlist stays for the container's nginx, which forwards
`Host` without a port.

Three smaller changes ride with it:

- The session cookie is named for the port the browser reached rather
  than the port bound, so two gateways that both bound 18792 no longer
  share one jar entry when one browser reaches them through different
  forwards. A `Host` without a port keeps the bound port, which is what
  the container's auto-login sets.
- `handle_auth_exchange` answers with the reason it refused, and the
  sign-in page shows that text rather than calling every failure an
  expired link.
- `_open` prints the sign-in link every time, and before handing it to
  a browser. `webbrowser.open` returns True whenever it ran something:
  VS Code's `browser.sh` exits 0 whatever happened, and a machine with
  no display hands the URL to a console browser that has no JavaScript
  and never redeems the nonce. The old fallback was unreachable, so the
  link was never printed at all.

The standalone serve test now uses a real aiohttp `Application` while
replacing only `AppRunner` and `TCPSite`. It selects temporary UI
locations for both an absent page and a built page, and checks that the
response revalidation hook is registered only for the built page. This
removes dependence on whether the checkout contains a UI build.

This PR also retains the reviewed Windows test guards and separate
`RAVEN_HOME` directories for real-upgrade fixtures.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Linux fixture follow-up (Python 3.12.13; the three-file suite and diff
coverage were rechecked after rebasing onto the current main):

- Before the fix, parameterizing `TestTheSignInLink` over an absent page
and a built page produced 4 passed / 1 failed. Only the built-page case
failed, with the same missing `on_response_prepare` as CI.
- `uv run --no-sync pytest -q -n 0
tests/test_cli_serve_commands.py::TestTheSignInLink`: 5 passed after the
fix.
- `uv run --no-sync pytest -q tests/test_rpc_transport.py
tests/test_cli_serve_commands.py tests/test_docker_runtime.py
--cov=raven.rpc.transports.ws --cov=raven.cli.serve_commands
--cov-report=json:<coverage-json>`: 234 passed. `<coverage-json>`
denotes the local coverage output file.
- `uv run --no-sync python scripts/coverage_gate.py --coverage-json
<coverage-json> diff --base-ref origin/main`: 100.00% (32/32 executable
changed lines), passing the 90% threshold. This measures diff coverage,
not the full-suite ratchet.
- A temporary test plugin removed the response hook without changing the
checkout. The built-page case failed its hook assertion and the
absent-page control passed (1 failed / 1 passed).
- `uv run --no-sync ruff check tests/test_cli_serve_commands.py
raven/cli/serve_commands.py`, `uv run --no-sync ruff format --check
tests/test_cli_serve_commands.py raven/cli/serve_commands.py`, and `uv
run --no-sync ty check raven/cli/serve_commands.py
tests/test_cli_serve_commands.py`: passed.
- `UV_NO_SYNC=1 make lint-python lint-imports lint-deps lint-types`:
passed after rebase (2268 Python files formatted; 10 import contracts
kept; no dependency issues; type checks passed). Both commit-message
validators and the repository source-language and large-file gates also
passed.
- The earlier description incorrectly called this an unrelated CI
failure. It was introduced by this PR's standalone-serve test, and the
CI build made that incomplete fixture fail. The fixture follow-up was
validated on Linux; the native Windows and full-suite measurements below
are earlier verification. CI must validate the pushed revision.

Native Windows follow-up (build 10.0.26300):

- Parallel real-upgrade cases now get separate `RAVEN_HOME` directories,
isolating both upgrade markers and downloaded wheels. A scratch barrier
forced both helpers to finish writing the same partial-wheel name before
rename: the original fixture produced 1 failed / 1 passed with
`Distribution not found` in the shared cache; the identical probe with
isolated homes produced 2 passed on Python 3.13.12. The ordinary run had
passed before the forced overlap, so this result identifies a race
rather than a consistently failing upgrade.
- Captured upgrade diagnostics tolerate localized Windows error bytes
and include stderr on a missing success message. Version and
companion-package assertions remain strict.
- Three reported unit-test constraints were reproduced. Symlink setup
now skips only Windows privilege error 1314; other errors propagate and
confinement still runs when a link can be created. Owner-only mode bits
and two SIGTERM-handler tests are explicitly POSIX-only. A native
subprocess probe confirmed that Windows Popen.terminate does not execute
the Python SIGTERM handler.
- `PYTHONUTF8=1`; `uv run --frozen --no-sync pytest
tests/test_rpc_transport.py tests/test_cli_serve_commands.py
tests/test_docker_runtime.py::test_container_bootstraps_the_browser_after_the_engine_is_ready
-n 0 -x --tb=short`: 195 passed, 30 skipped on Python 3.13.12.
Twenty-six skips were already POSIX-specific; four now identify the
reported mode/signal/symlink preconditions. The skipped checks are not
Windows equivalents.
- `PYTHONUTF8=1`, `UV_PYTHON=3.12`, and an isolated caller `RAVEN_HOME`;
`uv run --no-project --python 3.12 --with 'pytest>=8,<10' --with
pytest-asyncio --with pytest-xdist pytest --noconftest
tests/integration/test_cli_upgrade_real_uv.py -q --tb=short -n 4`: 3
passed, covering two real uv upgrade variants and the workflow contract.
- Whole-PR pre-commit, full Python-target Ruff check/format, all ten
import contracts, changed-production type checks, commitlint and
repository commit/asset/source-language gates passed.

Earlier author verification before this Windows follow-up:
- `uv run --frozen --python 3.12 --all-extras pytest -q
tests/test_rpc_transport.py tests/test_cli_serve_commands.py
tests/test_docker_runtime.py` -> `233 passed`.
- The same three files plus every `tests/test_rpc_*.py` and
`tests/test_cli_smoke.py` -> `3121 passed` on the first revision, before
the fixture follow-up.
- The earlier full suite, under `--cov=raven --cov-branch` -> `8 failed,
29791 passed, 158 skipped`, then `scripts/coverage_gate.py diff
--base-ref origin/main --threshold 90` -> `100.00% (32/32 executable
changed lines)`, passed. All eight failures are local baselines: five
proxy tests (`tests/test_config_update_providers.py`) that fail whenever
the shell carries this box's proxy;
`test_install_script.py::test_resolve_node_dir_answers_each_case_it_exists_for`
(node/npm resolution on this box);
`test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing`
(fails as root); and
`test_knowledge_parser_excel.py::test_a_legacy_workbook_is_converted_and_read`.
The last three were replayed in an unmodified baseline worktree and fail
there too (`3 failed`).
- The tests were written before the code and watched fail on it, each
for its own reason: the forwarded origin as `403: bad origin`, the
port-forward cookie case as `two sign-ins wrote one jar entry:
['raven_session_18792']`, the launcher cases on empty output. Against
`main`'s two source files the three test files now fail 18 tests.
- Eleven mutants, one construct each, all killed, with both files
restored and compared byte for byte: the forwarded rule off, Origin
equal to Host dropped, the loopback test dropped, the `localhost` branch
removed, either parse left unguarded, the cookie keyed to the bound
port, the two refusals given one text, the link not printed, the link
printed after the browser, and `raven serve --open` bypassing `_open`.
- `ruff check` and `ruff format --check` clean; `lint-imports` reports
`10 kept, 0 broken`.
- `scripts/check_source_language.py origin/main...HEAD` and
`scripts/check_large_files.py origin/main...HEAD` both exit 0.
- Real browser, end to end, against an isolated `raven serve` on its own
`RAVEN_HOME` reached through a TCP forwarder: the tab signed in and
landed on `/`, the cookie it kept was `raven_session_<forwarded port>`,
and an RPC call over a socket opened from that page was answered (server
log: `serve: ws client connected`, `system.hello`). A direct tab on the
bound port signed in too. With `fetch` stubbed, the page rendered each
of the four answers it can receive (refused address, used or expired
link, other status, no answer) with the matching text.

## Risk

- Security impact considered. The change widens one check: an `Origin`
that equals the request's `Host` on a loopback name. The cross-origin
local-page and DNS-rebinding cases remain refused, and both are covered
by tests. What it accepts that the old rule did not is a page whose
origin is exactly the address it asked for -- which only something
serving that address can show, and a file the agent produced arrives
with the CSP `sandbox` directive and so has an opaque origin.
- Backward compatibility: a tab signed in directly on the bound port
keeps its cookie name and stays signed in. A tab reaching the gateway
through a forward had no working session before, so there is nothing to
migrate. The container's auto-login cookie is unchanged, and a test
derives the entrypoint's name from `cookie_name` so the two cannot
drift.
- Rollback: revert the commit. Nothing is written outside the process
except the cookie already in the browser, and no config or format
changed.
- A `Host` that does not parse (`127.0.0.1:99999`, `127.0.0.1:abc`,
`[::1`) is answered 401 on `/rpc`, as on `main`, and an Origin equal to
such a Host is answered 403. Without the guard both would be a 500,
since the cookie's name is read before any credential is checked.
- Not fixed here: the page's own "this tab's sign-in expired" notice
(main page, i18n `gui.auth.stale`) still names the remedy correctly but
not the cause, so a tab that reaches the gateway by a non-loopback
address still reports a sign-in failure it cannot act on. Also
unmeasured: a real VS Code forward and a real container (both request
shapes were reproduced instead).

## Related Issues

N/A
2026-10-14 11:05:26 +08:00
Kevin Hu
6e7926a8e5
fix(knowledge): seven defects in the document reader, six of them docker-only (#938)
## Summary

Seven fixes to the knowledge reader, all found by running the Docker
image
against real documents. Six of the seven only reproduce there or only
reproduce in a browser, which is why they reached a release.

**The image could not draw the reader at all.** The document viewer
serves a
page of the file as a picture (`/knowledge/page`), the region a chunk
was cut
from (`/knowledge/crop`), and the file itself (`/knowledge/file`). All
three
are engine routes and none was in the nginx server block, so they fell
to
`location /` and were answered with index.html -- a 200 of text/html
where the
page put an `<img>`, which a browser draws as a broken image and reports
nowhere. Measured before: `200 text/html`. After, with a session:
`200 image/webp`, 126060 bytes, byte for byte what
`knowledge_pages.image_for`
writes. Source installs proxy nothing, so the reader has always worked
there.

**A document could be stranded at `indexing` for good.**
`index_document` is
awaited inside the RPC that asks for it, on the page's websocket, so a
reload
or a lost socket cancels it. `asyncio.CancelledError` is a
`BaseException`:
the five-type clause there and the `except Exception` in the RPC method
both
missed it, and the record was left reading `indexing` with nothing
indexing
it, no error beside it and nothing to retry it. It is set back to
`pending`
and re-raised now.

**Nothing drained that queue.** `_requeue_interrupted` puts back what a
restart interrupted and the above puts back what a cancel interrupted,
both
into a queue whose only other caller was a model switch on one base. The
row
read `Queued` and stayed there -- the same complaint
`_requeue_interrupted`'s own docstring makes about the state it rescues
documents from, one status further along. The gateway drains it at boot
now,
after a delay and one document at a time.

**Every .docx, .pptx and .xlsx failed to parse in the image.**
`mimetypes.guess_type` answers from the host, and the image carries none
of
the six files `mimetypes.knownfiles` names, so Python's built-in table
is all
it knows -- and that table predates OOXML. It knows `.doc` and `.ppt`
and
answers None for `.docx`, `.pptx`, `.xlsx` and `.rtf`. The manager falls
back
to `text/plain` for an unknown type, so a Word document went to the text
parser, which decoded the zip container as UTF-8 and died on the first
byte
that is not a character: "'utf-8' codec can't decode byte 0x91 in
position
15", against a file nothing is wrong with. The same table reversed is
what
`supported_extensions` builds the upload dialog's accept list from, so
those
formats were missing there too. The parsers state what they read now
instead
of asking the machine they run on.

**Indexing silenced every other control on the page.** `busy` is one
flag for
the whole store and both the upload and the rebuild held it across the
entire
parse-and-embed, tens of seconds on a long PDF. For all of that time
delete,
rename, star, pin, move, folder create and delete and every chunk edit
returned at their guard and did nothing -- no toast, nothing in the
console.
The confirm had already armed, so a reader answered "Delete, really" to
a
question that was then thrown away. Indexing has its own flag now; it
still
refuses to start a second one, which was the part that mattered.

That makes deleting a document while it indexes reachable, so the
manager
checks the record still exists before inserting. The delete already took
that
document's vectors, and inserting the ones that run computed would put
pieces
of it back with no row to own them, answering searches out of a deleted
file.

**A freshly uploaded file read `Queued` until the page was reloaded.**
The add
answers with the row as the server first wrote it, and the next line
starts
the call that indexes it. The row was left at what the add said for all
of it
while the server had already moved to `indexing` -- so reloading showed
the
truth and waiting did not. The rebuild path has never had this problem
because
it marks its own row, and the comment above that line is this bug
described in
advance: "a row that does not move until it is over reads as a press
that did
nothing". Both call one helper now.

**Picking a folder uploaded everything in it.** No browser applies
`accept` to
a directory, so a chosen folder arrived as every file under it --
.DS_Store,
lock files, images -- and each became an upload, a parse and a row
reading
Failed for a file nobody meant to add. The single-file picker set no
`accept`
at all. `knowledge.status` has answered with the readable extensions
since the
reader was written, and the comment above it asks for exactly this: "a
surface
that walks a folder has to filter by today's answer". Nothing on the
page had
ever called it. Both pickers carry that list now and what comes back is
filtered before anything is uploaded, because `accept` is advisory twice
over.
What is left out is reported rather than dropped quietly, and an install
whose
list could not be read filters nothing.

## Type

- [x] Fix

## Verification

- `uv run --frozen pytest tests/` -- 29694 passed, 121 skipped, 2
xfailed, 3
failed. The three are environmental and fail identically on this
branch's
base: macOS resolves soffice under /Applications so blanking PATH does
not
hide it, a case-insensitive filesystem, and process-descendant
enumeration.
- `npx vitest run` (ui-web) -- 238 files, 3724 passed.
- `npx tsc --noEmit` and `npx eslint src/features/knowledge/` -- clean.
- `uv run --frozen ruff check .` -- clean.
- `uv run --frozen --all-extras deptry raven` -- no dependency issues.
- `make check-source-language` and `make check-large-files` -- clean.
- `npm run --prefix ui-tui lint:i18n` -- clean after regenerating the
  catalogue for the one added message.
- In a running container: `/knowledge/page` answers 200 image/webp where
it
answered 200 text/html; a document deleted during a rebuild goes in
about
three seconds with the PDF still reading Indexing; opening a base sends
knowledge.status and both pickers are built carrying all 27 extensions.
- Every new test was checked against the unfixed code and fails there.

Not verified by hand: the upload picker is an `<input>` built detached
and
clicked, so it cannot be driven from a page script. The freshly-uploaded
status and the folder filtering rest on unit tests and on the element
the
picker builds, which was inspected live.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

User-visible behaviour changes, each of them a surface that did nothing
before:

- the knowledge reader's pictures load in Docker;
- a `.docx`, `.pptx` or `.xlsx` indexes in Docker instead of failing on
its
  zip header;
- an interrupted index is retried instead of sitting at `indexing`;
- the gateway indexes what is queued when it starts, which is embedding
work
at boot that did not happen before. It waits, runs one document at a
time,
  never raises, and does nothing when the queue is empty;
- structural controls work while a document indexes;
- a file picked from a folder that this install cannot read is left out
with a
  count rather than uploaded and failed.

Rollback is per commit: each of the seven stands alone and reverting any
one
restores the behaviour it changed. The nginx block, the mime
registration and
the cancel clause are additive and revert cleanly. The queue drain is
one call
at boot and removing that call disables it entirely.

Security: the mime registration adds only IANA mappings for formats the
parsers already accept, and routes no new input anywhere. The nginx
block
proxies three engine routes that enforce their own session check --
verified
by the 401 an unauthenticated request still gets.

Backward compatibility: no stored shape changes. The one record
transition
added, `indexing` to `pending` on cancel, is what a restart already did.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-10-14 10:43:57 +08:00
KevinHuSh
4a4a3fe15b feat(docker): run raven in a docker container and make a first run configurable from the page
## Summary

Two things arrive together because the second is what made the first usable.

**Run raven from a container.** A Dockerfile builds the page, the Python
environment and the engine into one image; `cd docker && docker compose up`
serves it. nginx serves the built page and proxies the RPC socket to the engine
over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a
second container would have no address to dial. Compose publishes on 127.0.0.1,
and the entrypoint prints a one-time sign-in link on boot.

**A first run is no longer a dead end.** The gateway used to refuse to start
until a provider was configured, while being the thing that serves the page
where a provider gets configured. It now starts, says what is missing, and picks
up a key saved in Settings on the next turn without a restart. The onboarding
overlay no longer takes the page either: New chat, the model chip and Send send
the reader to Settings > Models instead, and an empty model list is marked on
the control that fills it.

Also carried: the provider catalogue registry (capabilities, modalities and
context windows per model), the reworked Settings > Models pane, and four fixes
found in review of that work -- a sibling catalogue answering 200 with a
non-catalogue body no longer crashes the whole fetch; adding a tag no longer
erases the label and description already stored for that model; `authenticated`
no longer demands a bearer token from a local provider reached by address; and
registry rows are keyed by the spelling every lookup produces, which recovers
593 rows that no reader could reach.

## Type

- [ ] Fix
- [x] Feature
- [x] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Commands run on the branch head, with their results:

- `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped
- `npx vitest run` in `ui-web` -- 1573 passed (101 files)
- `npx vitest run` in `ui-tui` -- 2037 passed (142 files)
- `uv run ruff check raven scripts tests` and `ruff format --check` -- clean
- `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean
- `npm run lint:rpc` in `ui-tui` -- generated.ts in sync
- `make check-source-language`, `make check-large-files` -- clean

Each of the four review fixes was checked by reverting it and watching a named
test go red, then restoring it.

The container image itself has not been built by CI or by me -- no Docker daemon
was available on the machine this was written on. It has been built and run by
the author of the branch.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

Two deliberate security trade-offs in the container deployment, both documented
in `docker/README.md`:

- nginx rewrites the `Origin` header, because the engine accepts only an Origin
  naming its own loopback port and no browser reaching a published port can send
  one. This drops a defence-in-depth layer. The session cookie is
  SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite
  ignores ports, so another page on localhost counts as same-site.
- `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to
  every page load, so anyone who can reach the published port is signed in. It
  is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any
  deployment reachable from elsewhere. These are two settings that must agree,
  and nothing warns when they do not.

Behaviour changes outside the container: the gateway now starts without a
configured provider rather than exiting 1, and the first-run onboarding overlay
no longer opens by default (`?onboard=1` still forces it).

Rollback is by revert; nothing here migrates state or changes an on-disk format.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-11 03:27:52 +00:00