## Summary Seven fixes to the knowledge reader, all found by running the Docker image against real documents. Six of the seven only reproduce there or only reproduce in a browser, which is why they reached a release. **The image could not draw the reader at all.** The document viewer serves a page of the file as a picture (`/knowledge/page`), the region a chunk was cut from (`/knowledge/crop`), and the file itself (`/knowledge/file`). All three are engine routes and none was in the nginx server block, so they fell to `location /` and were answered with index.html -- a 200 of text/html where the page put an `<img>`, which a browser draws as a broken image and reports nowhere. Measured before: `200 text/html`. After, with a session: `200 image/webp`, 126060 bytes, byte for byte what `knowledge_pages.image_for` writes. Source installs proxy nothing, so the reader has always worked there. **A document could be stranded at `indexing` for good.** `index_document` is awaited inside the RPC that asks for it, on the page's websocket, so a reload or a lost socket cancels it. `asyncio.CancelledError` is a `BaseException`: the five-type clause there and the `except Exception` in the RPC method both missed it, and the record was left reading `indexing` with nothing indexing it, no error beside it and nothing to retry it. It is set back to `pending` and re-raised now. **Nothing drained that queue.** `_requeue_interrupted` puts back what a restart interrupted and the above puts back what a cancel interrupted, both into a queue whose only other caller was a model switch on one base. The row read `Queued` and stayed there -- the same complaint `_requeue_interrupted`'s own docstring makes about the state it rescues documents from, one status further along. The gateway drains it at boot now, after a delay and one document at a time. **Every .docx, .pptx and .xlsx failed to parse in the image.** `mimetypes.guess_type` answers from the host, and the image carries none of the six files `mimetypes.knownfiles` names, so Python's built-in table is all it knows -- and that table predates OOXML. It knows `.doc` and `.ppt` and answers None for `.docx`, `.pptx`, `.xlsx` and `.rtf`. The manager falls back to `text/plain` for an unknown type, so a Word document went to the text parser, which decoded the zip container as UTF-8 and died on the first byte that is not a character: "'utf-8' codec can't decode byte 0x91 in position 15", against a file nothing is wrong with. The same table reversed is what `supported_extensions` builds the upload dialog's accept list from, so those formats were missing there too. The parsers state what they read now instead of asking the machine they run on. **Indexing silenced every other control on the page.** `busy` is one flag for the whole store and both the upload and the rebuild held it across the entire parse-and-embed, tens of seconds on a long PDF. For all of that time delete, rename, star, pin, move, folder create and delete and every chunk edit returned at their guard and did nothing -- no toast, nothing in the console. The confirm had already armed, so a reader answered "Delete, really" to a question that was then thrown away. Indexing has its own flag now; it still refuses to start a second one, which was the part that mattered. That makes deleting a document while it indexes reachable, so the manager checks the record still exists before inserting. The delete already took that document's vectors, and inserting the ones that run computed would put pieces of it back with no row to own them, answering searches out of a deleted file. **A freshly uploaded file read `Queued` until the page was reloaded.** The add answers with the row as the server first wrote it, and the next line starts the call that indexes it. The row was left at what the add said for all of it while the server had already moved to `indexing` -- so reloading showed the truth and waiting did not. The rebuild path has never had this problem because it marks its own row, and the comment above that line is this bug described in advance: "a row that does not move until it is over reads as a press that did nothing". Both call one helper now. **Picking a folder uploaded everything in it.** No browser applies `accept` to a directory, so a chosen folder arrived as every file under it -- .DS_Store, lock files, images -- and each became an upload, a parse and a row reading Failed for a file nobody meant to add. The single-file picker set no `accept` at all. `knowledge.status` has answered with the readable extensions since the reader was written, and the comment above it asks for exactly this: "a surface that walks a folder has to filter by today's answer". Nothing on the page had ever called it. Both pickers carry that list now and what comes back is filtered before anything is uploaded, because `accept` is advisory twice over. What is left out is reported rather than dropped quietly, and an install whose list could not be read filters nothing. ## Type - [x] Fix ## Verification - `uv run --frozen pytest tests/` -- 29694 passed, 121 skipped, 2 xfailed, 3 failed. The three are environmental and fail identically on this branch's base: macOS resolves soffice under /Applications so blanking PATH does not hide it, a case-insensitive filesystem, and process-descendant enumeration. - `npx vitest run` (ui-web) -- 238 files, 3724 passed. - `npx tsc --noEmit` and `npx eslint src/features/knowledge/` -- clean. - `uv run --frozen ruff check .` -- clean. - `uv run --frozen --all-extras deptry raven` -- no dependency issues. - `make check-source-language` and `make check-large-files` -- clean. - `npm run --prefix ui-tui lint:i18n` -- clean after regenerating the catalogue for the one added message. - In a running container: `/knowledge/page` answers 200 image/webp where it answered 200 text/html; a document deleted during a rebuild goes in about three seconds with the PDF still reading Indexing; opening a base sends knowledge.status and both pickers are built carrying all 27 extensions. - Every new test was checked against the unfixed code and fails there. Not verified by hand: the upload picker is an `<input>` built detached and clicked, so it cannot be driven from a page script. The freshly-uploaded status and the folder filtering rest on unit tests and on the element the picker builds, which was inspected live. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk User-visible behaviour changes, each of them a surface that did nothing before: - the knowledge reader's pictures load in Docker; - a `.docx`, `.pptx` or `.xlsx` indexes in Docker instead of failing on its zip header; - an interrupted index is retried instead of sitting at `indexing`; - the gateway indexes what is queued when it starts, which is embedding work at boot that did not happen before. It waits, runs one document at a time, never raises, and does nothing when the queue is empty; - structural controls work while a document indexes; - a file picked from a folder that this install cannot read is left out with a count rather than uploaded and failed. Rollback is per commit: each of the seven stands alone and reverting any one restores the behaviour it changed. The nginx block, the mime registration and the cancel clause are additive and revert cleanly. The queue drain is one call at boot and removing that call disables it entirely. Security: the mime registration adds only IANA mappings for formats the parsers already accept, and routes no new input anywhere. The nginx block proxies three engine routes that enforce their own session check -- verified by the 401 an unauthenticated request still gets. Backward compatibility: no stored shape changes. The one record transition added, `indexing` to `pending` on cancel, is what a restart already did. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
114 lines
4.4 KiB
Python
114 lines
4.4 KiB
Python
"""The media types the parsers are routed by, on a host that knows none of them.
|
|
|
|
`mimetypes` answers from the machine. A developer's carries a full Apache
|
|
table; a slim container image carries no table at all, and Python's built-in
|
|
one predates OOXML. These tests are written against the second, because the
|
|
first is what hid the bug.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import mimetypes
|
|
|
|
import pytest
|
|
|
|
from raven.knowledge.parser._media import MEDIA_TYPES, register
|
|
|
|
|
|
@pytest.fixture
|
|
def bare_host(monkeypatch: pytest.MonkeyPatch):
|
|
"""A process whose host supplies no mime table at all.
|
|
|
|
What the Raven image is: `mimetypes.knownfiles` names half a dozen paths
|
|
and a slim Debian carries none of them, so the built-in table is the whole
|
|
of what `guess_type` knows.
|
|
"""
|
|
monkeypatch.setattr(mimetypes, "knownfiles", [])
|
|
monkeypatch.setattr(mimetypes, "_db", None, raising=False)
|
|
mimetypes.init([])
|
|
yield
|
|
monkeypatch.setattr(mimetypes, "_db", None, raising=False)
|
|
mimetypes.init()
|
|
|
|
|
|
#: The four the built-in table has no answer for. `.doc` and `.ppt` are in it
|
|
#: and were never the problem, which is why only some uploads failed.
|
|
_OOXML = (".docx", ".xlsx", ".pptx")
|
|
|
|
|
|
class TestWithoutRegistering:
|
|
"""The failure, stated so the fix below is not tautological."""
|
|
|
|
def test_the_bare_host_cannot_name_an_ooxml_file(self, bare_host) -> None:
|
|
assert [mimetypes.guess_type(f"x{suffix}")[0] for suffix in _OOXML] == [None, None, None]
|
|
|
|
def test_but_it_can_name_their_predecessors(self, bare_host) -> None:
|
|
"""Which is why a .ppt indexed and a .pptx did not."""
|
|
assert mimetypes.guess_type("x.ppt")[0] == "application/vnd.ms-powerpoint"
|
|
assert mimetypes.guess_type("x.doc")[0] == "application/msword"
|
|
|
|
|
|
class TestRegistering:
|
|
def test_every_format_this_package_parses_resolves(self, bare_host) -> None:
|
|
register()
|
|
|
|
for suffix, media_type in MEDIA_TYPES:
|
|
assert mimetypes.guess_type(f"x{suffix}")[0] == media_type, suffix
|
|
|
|
def test_a_word_file_is_routed_to_the_word_parser(self, bare_host) -> None:
|
|
"""The whole point. Unregistered it guessed None, the manager fell back
|
|
to text/plain, and the text parser decoded a zip container as UTF-8 --
|
|
which the reader saw as "codec can't decode byte 0x91" against a file
|
|
nothing is wrong with."""
|
|
register()
|
|
|
|
from raven.knowledge.parser.docx_parser import DocxParser
|
|
|
|
guessed = mimetypes.guess_type("report.docx")[0]
|
|
assert guessed in DocxParser.supported_media_types
|
|
|
|
def test_the_file_picker_offers_them_too(self, bare_host) -> None:
|
|
"""`ParserBase.supported_extensions` reverses the same table, so the
|
|
formats missing from it were missing from the upload dialog's accept
|
|
list.
|
|
|
|
Asked of the base implementation rather than of a parser: every parser
|
|
that reads an OOXML format overrides the method with a literal, so
|
|
asserting against one of those is a tautology that a no-op `register`
|
|
survives. This asks the reverse lookup itself, which is the thing the
|
|
registration feeds.
|
|
"""
|
|
from raven.knowledge.parser import ParserBase
|
|
|
|
class _Reader(ParserBase):
|
|
supported_media_types = [
|
|
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
|
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
|
|
]
|
|
|
|
async def parse(self, file, filename): # pragma: no cover - never called
|
|
return []
|
|
|
|
assert _Reader.supported_extensions() == [], "nothing to reverse before registering"
|
|
|
|
register()
|
|
|
|
assert _Reader.supported_extensions() == [".docx", ".pptx"]
|
|
|
|
def test_it_can_be_called_twice(self, bare_host) -> None:
|
|
"""It runs at import, and an import can happen more than once in a
|
|
process that reloads a module."""
|
|
register()
|
|
register()
|
|
|
|
assert mimetypes.guess_type("x.docx")[0] == dict(MEDIA_TYPES)[".docx"]
|
|
|
|
def test_it_does_not_overrule_a_host_that_already_knows(self, bare_host) -> None:
|
|
"""Additive, not corrective: where the host has an answer it is the
|
|
same answer, so this supplies rather than replaces."""
|
|
mimetypes.add_type("application/pdf", ".pdf")
|
|
|
|
register()
|
|
|
|
assert mimetypes.guess_type("x.pdf")[0] == "application/pdf"
|