Raven-1/tests/test_knowledge_library.py
Kevin Hu 6e7926a8e5
fix(knowledge): seven defects in the document reader, six of them docker-only (#938)
## Summary

Seven fixes to the knowledge reader, all found by running the Docker
image
against real documents. Six of the seven only reproduce there or only
reproduce in a browser, which is why they reached a release.

**The image could not draw the reader at all.** The document viewer
serves a
page of the file as a picture (`/knowledge/page`), the region a chunk
was cut
from (`/knowledge/crop`), and the file itself (`/knowledge/file`). All
three
are engine routes and none was in the nginx server block, so they fell
to
`location /` and were answered with index.html -- a 200 of text/html
where the
page put an `<img>`, which a browser draws as a broken image and reports
nowhere. Measured before: `200 text/html`. After, with a session:
`200 image/webp`, 126060 bytes, byte for byte what
`knowledge_pages.image_for`
writes. Source installs proxy nothing, so the reader has always worked
there.

**A document could be stranded at `indexing` for good.**
`index_document` is
awaited inside the RPC that asks for it, on the page's websocket, so a
reload
or a lost socket cancels it. `asyncio.CancelledError` is a
`BaseException`:
the five-type clause there and the `except Exception` in the RPC method
both
missed it, and the record was left reading `indexing` with nothing
indexing
it, no error beside it and nothing to retry it. It is set back to
`pending`
and re-raised now.

**Nothing drained that queue.** `_requeue_interrupted` puts back what a
restart interrupted and the above puts back what a cancel interrupted,
both
into a queue whose only other caller was a model switch on one base. The
row
read `Queued` and stayed there -- the same complaint
`_requeue_interrupted`'s own docstring makes about the state it rescues
documents from, one status further along. The gateway drains it at boot
now,
after a delay and one document at a time.

**Every .docx, .pptx and .xlsx failed to parse in the image.**
`mimetypes.guess_type` answers from the host, and the image carries none
of
the six files `mimetypes.knownfiles` names, so Python's built-in table
is all
it knows -- and that table predates OOXML. It knows `.doc` and `.ppt`
and
answers None for `.docx`, `.pptx`, `.xlsx` and `.rtf`. The manager falls
back
to `text/plain` for an unknown type, so a Word document went to the text
parser, which decoded the zip container as UTF-8 and died on the first
byte
that is not a character: "'utf-8' codec can't decode byte 0x91 in
position
15", against a file nothing is wrong with. The same table reversed is
what
`supported_extensions` builds the upload dialog's accept list from, so
those
formats were missing there too. The parsers state what they read now
instead
of asking the machine they run on.

**Indexing silenced every other control on the page.** `busy` is one
flag for
the whole store and both the upload and the rebuild held it across the
entire
parse-and-embed, tens of seconds on a long PDF. For all of that time
delete,
rename, star, pin, move, folder create and delete and every chunk edit
returned at their guard and did nothing -- no toast, nothing in the
console.
The confirm had already armed, so a reader answered "Delete, really" to
a
question that was then thrown away. Indexing has its own flag now; it
still
refuses to start a second one, which was the part that mattered.

That makes deleting a document while it indexes reachable, so the
manager
checks the record still exists before inserting. The delete already took
that
document's vectors, and inserting the ones that run computed would put
pieces
of it back with no row to own them, answering searches out of a deleted
file.

**A freshly uploaded file read `Queued` until the page was reloaded.**
The add
answers with the row as the server first wrote it, and the next line
starts
the call that indexes it. The row was left at what the add said for all
of it
while the server had already moved to `indexing` -- so reloading showed
the
truth and waiting did not. The rebuild path has never had this problem
because
it marks its own row, and the comment above that line is this bug
described in
advance: "a row that does not move until it is over reads as a press
that did
nothing". Both call one helper now.

**Picking a folder uploaded everything in it.** No browser applies
`accept` to
a directory, so a chosen folder arrived as every file under it --
.DS_Store,
lock files, images -- and each became an upload, a parse and a row
reading
Failed for a file nobody meant to add. The single-file picker set no
`accept`
at all. `knowledge.status` has answered with the readable extensions
since the
reader was written, and the comment above it asks for exactly this: "a
surface
that walks a folder has to filter by today's answer". Nothing on the
page had
ever called it. Both pickers carry that list now and what comes back is
filtered before anything is uploaded, because `accept` is advisory twice
over.
What is left out is reported rather than dropped quietly, and an install
whose
list could not be read filters nothing.

## Type

- [x] Fix

## Verification

- `uv run --frozen pytest tests/` -- 29694 passed, 121 skipped, 2
xfailed, 3
failed. The three are environmental and fail identically on this
branch's
base: macOS resolves soffice under /Applications so blanking PATH does
not
hide it, a case-insensitive filesystem, and process-descendant
enumeration.
- `npx vitest run` (ui-web) -- 238 files, 3724 passed.
- `npx tsc --noEmit` and `npx eslint src/features/knowledge/` -- clean.
- `uv run --frozen ruff check .` -- clean.
- `uv run --frozen --all-extras deptry raven` -- no dependency issues.
- `make check-source-language` and `make check-large-files` -- clean.
- `npm run --prefix ui-tui lint:i18n` -- clean after regenerating the
  catalogue for the one added message.
- In a running container: `/knowledge/page` answers 200 image/webp where
it
answered 200 text/html; a document deleted during a rebuild goes in
about
three seconds with the PDF still reading Indexing; opening a base sends
knowledge.status and both pickers are built carrying all 27 extensions.
- Every new test was checked against the unfixed code and fails there.

Not verified by hand: the upload picker is an `<input>` built detached
and
clicked, so it cannot be driven from a page script. The freshly-uploaded
status and the folder filtering rest on unit tests and on the element
the
picker builds, which was inspected live.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

User-visible behaviour changes, each of them a surface that did nothing
before:

- the knowledge reader's pictures load in Docker;
- a `.docx`, `.pptx` or `.xlsx` indexes in Docker instead of failing on
its
  zip header;
- an interrupted index is retried instead of sitting at `indexing`;
- the gateway indexes what is queued when it starts, which is embedding
work
at boot that did not happen before. It waits, runs one document at a
time,
  never raises, and does nothing when the queue is empty;
- structural controls work while a document indexes;
- a file picked from a folder that this install cannot read is left out
with a
  count rather than uploaded and failed.

Rollback is per commit: each of the seven stands alone and reverting any
one
restores the behaviour it changed. The nginx block, the mime
registration and
the cancel clause are additive and revert cleanly. The queue drain is
one call
at boot and removing that call disables it entirely.

Security: the mime registration adds only IANA mappings for formats the
parsers already accept, and routes no new input anywhere. The nginx
block
proxies three engine routes that enforce their own session check --
verified
by the 401 an unauthenticated request still gets.

Backward compatibility: no stored shape changes. The one record
transition
added, `indexing` to `pending` on cancel, is what a restart already did.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-10-14 10:43:57 +08:00

68 lines
1.9 KiB
Python

"""The queue drainer: what finishes a document the gateway stopped indexing."""
from __future__ import annotations
import asyncio
import pytest
from raven.knowledge import _library
class _Manager:
def __init__(self, tried: int = 0, boom: Exception | None = None) -> None:
self.tried = tried
self.boom = boom
self.calls = 0
async def index_pending(self, base_id: str = "") -> int:
self.calls += 1
if self.boom:
raise self.boom
return self.tried
@pytest.fixture
def staged(monkeypatch: pytest.MonkeyPatch):
made = _Manager()
monkeypatch.setattr(_library, "knowledge_manager", lambda: made)
return made
async def test_the_queue_is_drained_once_the_gateway_has_settled(staged) -> None:
staged.tried = 2
task = _library.drain_pending_in_background(delay_s=0)
await task
assert staged.calls == 1, "the queue nothing else drains"
async def test_a_queue_that_cannot_be_drained_does_not_take_the_boot_with_it(staged) -> None:
"""A boot path answers None rather than failing: a gateway that does not
start is worse than a document that stays queued."""
staged.boom = RuntimeError("the embedding endpoint is down")
task = _library.drain_pending_in_background(delay_s=0)
await task
assert task.done() and task.exception() is None
async def test_it_waits_before_it_starts(staged) -> None:
"""The first page and the first turn are answered before this embeds."""
task = _library.drain_pending_in_background(delay_s=30)
await asyncio.sleep(0)
assert staged.calls == 0, "nothing yet"
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
def test_outside_a_loop_there_is_nothing_to_drain_into(staged) -> None:
"""Imported by a CLI process with no running loop, it answers None rather
than raising."""
assert _library.drain_pending_in_background(delay_s=0) is None
assert staged.calls == 0