## Summary Seven fixes to the knowledge reader, all found by running the Docker image against real documents. Six of the seven only reproduce there or only reproduce in a browser, which is why they reached a release. **The image could not draw the reader at all.** The document viewer serves a page of the file as a picture (`/knowledge/page`), the region a chunk was cut from (`/knowledge/crop`), and the file itself (`/knowledge/file`). All three are engine routes and none was in the nginx server block, so they fell to `location /` and were answered with index.html -- a 200 of text/html where the page put an `<img>`, which a browser draws as a broken image and reports nowhere. Measured before: `200 text/html`. After, with a session: `200 image/webp`, 126060 bytes, byte for byte what `knowledge_pages.image_for` writes. Source installs proxy nothing, so the reader has always worked there. **A document could be stranded at `indexing` for good.** `index_document` is awaited inside the RPC that asks for it, on the page's websocket, so a reload or a lost socket cancels it. `asyncio.CancelledError` is a `BaseException`: the five-type clause there and the `except Exception` in the RPC method both missed it, and the record was left reading `indexing` with nothing indexing it, no error beside it and nothing to retry it. It is set back to `pending` and re-raised now. **Nothing drained that queue.** `_requeue_interrupted` puts back what a restart interrupted and the above puts back what a cancel interrupted, both into a queue whose only other caller was a model switch on one base. The row read `Queued` and stayed there -- the same complaint `_requeue_interrupted`'s own docstring makes about the state it rescues documents from, one status further along. The gateway drains it at boot now, after a delay and one document at a time. **Every .docx, .pptx and .xlsx failed to parse in the image.** `mimetypes.guess_type` answers from the host, and the image carries none of the six files `mimetypes.knownfiles` names, so Python's built-in table is all it knows -- and that table predates OOXML. It knows `.doc` and `.ppt` and answers None for `.docx`, `.pptx`, `.xlsx` and `.rtf`. The manager falls back to `text/plain` for an unknown type, so a Word document went to the text parser, which decoded the zip container as UTF-8 and died on the first byte that is not a character: "'utf-8' codec can't decode byte 0x91 in position 15", against a file nothing is wrong with. The same table reversed is what `supported_extensions` builds the upload dialog's accept list from, so those formats were missing there too. The parsers state what they read now instead of asking the machine they run on. **Indexing silenced every other control on the page.** `busy` is one flag for the whole store and both the upload and the rebuild held it across the entire parse-and-embed, tens of seconds on a long PDF. For all of that time delete, rename, star, pin, move, folder create and delete and every chunk edit returned at their guard and did nothing -- no toast, nothing in the console. The confirm had already armed, so a reader answered "Delete, really" to a question that was then thrown away. Indexing has its own flag now; it still refuses to start a second one, which was the part that mattered. That makes deleting a document while it indexes reachable, so the manager checks the record still exists before inserting. The delete already took that document's vectors, and inserting the ones that run computed would put pieces of it back with no row to own them, answering searches out of a deleted file. **A freshly uploaded file read `Queued` until the page was reloaded.** The add answers with the row as the server first wrote it, and the next line starts the call that indexes it. The row was left at what the add said for all of it while the server had already moved to `indexing` -- so reloading showed the truth and waiting did not. The rebuild path has never had this problem because it marks its own row, and the comment above that line is this bug described in advance: "a row that does not move until it is over reads as a press that did nothing". Both call one helper now. **Picking a folder uploaded everything in it.** No browser applies `accept` to a directory, so a chosen folder arrived as every file under it -- .DS_Store, lock files, images -- and each became an upload, a parse and a row reading Failed for a file nobody meant to add. The single-file picker set no `accept` at all. `knowledge.status` has answered with the readable extensions since the reader was written, and the comment above it asks for exactly this: "a surface that walks a folder has to filter by today's answer". Nothing on the page had ever called it. Both pickers carry that list now and what comes back is filtered before anything is uploaded, because `accept` is advisory twice over. What is left out is reported rather than dropped quietly, and an install whose list could not be read filters nothing. ## Type - [x] Fix ## Verification - `uv run --frozen pytest tests/` -- 29694 passed, 121 skipped, 2 xfailed, 3 failed. The three are environmental and fail identically on this branch's base: macOS resolves soffice under /Applications so blanking PATH does not hide it, a case-insensitive filesystem, and process-descendant enumeration. - `npx vitest run` (ui-web) -- 238 files, 3724 passed. - `npx tsc --noEmit` and `npx eslint src/features/knowledge/` -- clean. - `uv run --frozen ruff check .` -- clean. - `uv run --frozen --all-extras deptry raven` -- no dependency issues. - `make check-source-language` and `make check-large-files` -- clean. - `npm run --prefix ui-tui lint:i18n` -- clean after regenerating the catalogue for the one added message. - In a running container: `/knowledge/page` answers 200 image/webp where it answered 200 text/html; a document deleted during a rebuild goes in about three seconds with the PDF still reading Indexing; opening a base sends knowledge.status and both pickers are built carrying all 27 extensions. - Every new test was checked against the unfixed code and fails there. Not verified by hand: the upload picker is an `<input>` built detached and clicked, so it cannot be driven from a page script. The freshly-uploaded status and the folder filtering rest on unit tests and on the element the picker builds, which was inspected live. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk User-visible behaviour changes, each of them a surface that did nothing before: - the knowledge reader's pictures load in Docker; - a `.docx`, `.pptx` or `.xlsx` indexes in Docker instead of failing on its zip header; - an interrupted index is retried instead of sitting at `indexing`; - the gateway indexes what is queued when it starts, which is embedding work at boot that did not happen before. It waits, runs one document at a time, never raises, and does nothing when the queue is empty; - structural controls work while a document indexes; - a file picked from a folder that this install cannot read is left out with a count rather than uploaded and failed. Rollback is per commit: each of the seven stands alone and reverting any one restores the behaviour it changed. The nginx block, the mime registration and the cancel clause are additive and revert cleanly. The queue drain is one call at boot and removing that call disables it entirely. Security: the mime registration adds only IANA mappings for formats the parsers already accept, and routes no new input anywhere. The nginx block proxies three engine routes that enforce their own session check -- verified by the 401 an unauthenticated request still gets. Backward compatibility: no stored shape changes. The one record transition added, `indexing` to `pending` on cancel, is what a restart already did. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
68 lines
1.9 KiB
Python
68 lines
1.9 KiB
Python
"""The queue drainer: what finishes a document the gateway stopped indexing."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
|
|
import pytest
|
|
|
|
from raven.knowledge import _library
|
|
|
|
|
|
class _Manager:
|
|
def __init__(self, tried: int = 0, boom: Exception | None = None) -> None:
|
|
self.tried = tried
|
|
self.boom = boom
|
|
self.calls = 0
|
|
|
|
async def index_pending(self, base_id: str = "") -> int:
|
|
self.calls += 1
|
|
if self.boom:
|
|
raise self.boom
|
|
return self.tried
|
|
|
|
|
|
@pytest.fixture
|
|
def staged(monkeypatch: pytest.MonkeyPatch):
|
|
made = _Manager()
|
|
monkeypatch.setattr(_library, "knowledge_manager", lambda: made)
|
|
return made
|
|
|
|
|
|
async def test_the_queue_is_drained_once_the_gateway_has_settled(staged) -> None:
|
|
staged.tried = 2
|
|
|
|
task = _library.drain_pending_in_background(delay_s=0)
|
|
await task
|
|
|
|
assert staged.calls == 1, "the queue nothing else drains"
|
|
|
|
|
|
async def test_a_queue_that_cannot_be_drained_does_not_take_the_boot_with_it(staged) -> None:
|
|
"""A boot path answers None rather than failing: a gateway that does not
|
|
start is worse than a document that stays queued."""
|
|
staged.boom = RuntimeError("the embedding endpoint is down")
|
|
|
|
task = _library.drain_pending_in_background(delay_s=0)
|
|
await task
|
|
|
|
assert task.done() and task.exception() is None
|
|
|
|
|
|
async def test_it_waits_before_it_starts(staged) -> None:
|
|
"""The first page and the first turn are answered before this embeds."""
|
|
task = _library.drain_pending_in_background(delay_s=30)
|
|
await asyncio.sleep(0)
|
|
|
|
assert staged.calls == 0, "nothing yet"
|
|
task.cancel()
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await task
|
|
|
|
|
|
def test_outside_a_loop_there_is_nothing_to_drain_into(staged) -> None:
|
|
"""Imported by a CLI process with no running loop, it answers None rather
|
|
than raising."""
|
|
assert _library.drain_pending_in_background(delay_s=0) is None
|
|
assert staged.calls == 0
|