<img width="2318" height="1448" alt="image" src="https://github.com/user-attachments/assets/bca71d69-4972-420c-b14e-c8cf74a17de9" /> ## Summary Everything a knowledge base could not read, and the pieces it cut them into saying what they are. **Reading a PDF.** A PDF records glyphs at coordinates; whether they are a heading, a table cell or a running footer is inferred from how they look. There are now two readers. The geometric one infers structure from type size and position and needs nothing but the file. The other asks deepdoc's models, ported into `raven/knowledge/parser/deepdoc`: a layout model that says what each region is, a detector and recognizer that read a page carrying no text at all, and a table-structure model that rebuilds a grid from a picture of one. Detecting boxes around text the file already spells out exactly costs an inference pass per page to learn less, and recognising it costs accuracy. The layout graph answers with a shape its own postprocessing raises on, and its classes are DocLayout-YOLO's order rather than the list Both were established by running the model, and the label order is pinned by a test. **Tables go in as tables.** `<table>` is the only shape that keeps what a grid states -- which column a cell stood under, which cells are merged, which row is the header -- and it is the shape a model reads a table in, which is what these chunks are for. PDF, Word and PowerPoint all state their merges exactly, and they now survive as `colspan` and `rowspan`. The flat form this replaces had to repeat a header standing over two columns across both of them, because there was nowhere to record a span; it had no callers left afterwards and is gone. Two deviations from upstream, both about what the string is for: every cell's text is escaped, where upstream interpolates it raw and a cell holding a `<` produces markup that is no longer a table, and the span attributes are written only where there is a span, where upstream emits `<td >` for every ordinary cell. **A picture of where each chunk came from.** A chunk is a few hundred words with a page number on it, which is enough to cite and not enough to answer whether the parser read the page correctly. So every PDF chunk that knows where it sits carries one image: the regions it covers, cropped out of the rendered pages and stacked. here they are already structured in the parser's element spans, so the markers would only put text nobody wrote into the thing being embedded. The panel draws a chunk's tables as tables, parsed and rebuilt as elements rather than injected: a person can rewrite any piece from that same panel, so what comes back from the store is user input however it got there. **The page that reads them.** Upstream retired the knowledge page while this branch was out; it is rebuilt against the new architecture, and the half that never existed is the one a reader needs -- what a document actually became once it was cut and embedded. Documents sit in one-level folders and arrive by four routes: a file, a folder, a URL, a written note. Adding and indexing are two calls on the contract and nothing drains that queue on its own, so a page that stopped after the first left every upload saying Queued for good. Opening a file takes the whole page: the document on the left, the pieces the index holds of it on the right. The left half is the file drawn as its own pages rather than framed by the browser's PDF viewer, which reads where to open once and answers to nothing afterwards -- and which nothing can be drawn on. Pages are rendered once and kept, and the region a piece was cut from is washed over the page it came from, placed in fractions of the page so the picture's own scale never enters the arithmetic. Clicking a piece takes the file there; double-clicking opens it for rewriting. The pieces can be searched, filtered by state, turned on and off, deleted, rewritten and added to, one at a time or in a batch. Every one of those was already on the contract with no reader for it. A base's settings are back and fuller: how many pieces a search returns, how documents are cut, and how much prose comes with a table or a figure. The embedding model is a picker while the base holds no pieces and a stated fact once it does -- moving it then is the collection made again at the new width and every document re-cut, which is not a thing a panel does by being saved. **An agent can read them.** The retrieval engine was there and nothing could reach it: no agent had a tool that opened a library. A reader now points a conversation at bases from the composer, the way they pin a working directory, and the selection rides in the same place -- the session's metadata, bound around the turn. The tool takes a query and never a base. The scope is the reader's choice, so a model cannot reach material nobody offered it, and it is withheld where nothing was picked rather than offered and refused. A hit says which document and page it came from, because a hit a reader cannot place is a hit they cannot check -- and the passages are pressable in the transcript, walking into the base, the document, the folder it is filed under and then the piece. A citation a reader cannot follow is a claim they have to take on trust, and the whole point of searching their own material is that they can go and look. **The word segmenter and the resources.** `raven.core.Tokenizer` is Infinity's `rag_tokenizer`, ported: a forward and backward maximum match over a dictionary trie, with stemming and lemmatisation on the latin path. XGBoost is not carried over. None of the data lives in git -- the dictionary, the NLTK corpora and the ONNX graphs run to about a hundred megabytes, so `make fetch-resources` on an install and an optional Docker layer behind `RAVEN_RESOURCES` both point at one script. A checkout that could not reach the network still works, and every feature that reads this data says so when asked. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run on macOS against Python 3.12, with the deepdoc weights fetched (the model tests skip without them): ``` uv run pytest 27450 passed, 111 skipped, 3 failed npm run test --prefix ui-web 3276 passed npm run test --prefix ui-tui 2106 passed npm run type-check --prefix ui-web clean (tsc) npm run lint --prefix ui-web clean (4 pre-existing hook warnings) make lint-python clean (ruff check + format) make lint-types clean (ty) make lint-imports 10 contracts kept, 0 broken make check-source-language clean make check-large-files clean make build-ui built, both boot snapshots match node ui-web/scripts/check-css.mjs clean node ui-web/scripts/check-class-namespace.mjs clean ``` The three Python failures are not this branch's and were each confirmed on the base alone: one asks a host without LibreOffice to refuse a render and this host has LibreOffice installed, and two simulation cases fail on an unmodified checkout of the base commit. The RPC contract was regenerated for both consumers and both match it (`npm run gen --prefix ui-web`, `npm run gen:rpc --prefix ui-tui`, then `gen:check`). The page's boot snapshots and region goldens were refreshed where the composer bar gained a button, and each records that button and nothing else. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk A PDF's tables, and a Word or PowerPoint file's, are now indexed as HTML rather than as header-prefixed lines. Chunk text changes shape for those documents, so a base indexed before this and searched after it holds both forms until its documents are reindexed; retrieval keeps working either way, because both forms are text. The same applies to how much prose a table or a figure carries with it: the setting now reaches as far as it says, where before it stopped at the nearest paragraph, so a chunk cut before this keeps the narrower context until its document is rebuilt. Rolling back is reverting the branch and reindexing. The deepdoc models are optional. An install that never ran `make fetch-resources` reads PDFs geometrically, exactly as before, and says so when the models are asked for by name. Nothing here reaches the network at request time. Chunk crops and rendered pages are written at index time and on first view, both under the state directory rather than in the repository, and both are deleted with their document. A page is immutable for the life of its document id, so the route serving them caches; the document route does not. All three routes take ids and no path, and the two image routes answer under the strictest sandbox in the tree. What an agent may search is the reader's choice and nothing else. The tool takes no base argument, so a model cannot name one, and it is absent from the schema entirely for a conversation that named none -- a lane nobody pointed at a base is unchanged by this branch. The bases a session carries are stored beside its working directory and travel no further. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
1581 lines
68 KiB
Python
1581 lines
68 KiB
Python
"""End-to-end tests for a knowledge base: create, upload, index, search.
|
|
|
|
The vector store and the records are real; only the embedding endpoint is
|
|
stubbed, because it is the one piece that is a network call. The stub embeds
|
|
by counting words, which is enough for "the nearer text ranks first" without
|
|
pretending to be a model.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import base64
|
|
from dataclasses import replace
|
|
|
|
import pytest
|
|
|
|
from raven.knowledge._embedding import EmbeddingConfig, EmbeddingError
|
|
from raven.knowledge._manager import DuplicateBaseNameError, KnowledgeError, KnowledgeManager
|
|
|
|
DIM = 8
|
|
_VOCAB = ["alpha", "beta", "gamma", "delta", "epsilon", "zeta", "eta", "theta"]
|
|
|
|
|
|
class _StubClient:
|
|
"""Embeds a text as a bag-of-words count over a tiny vocabulary."""
|
|
|
|
def __init__(self, model: str = "stub-embed", dimensions: int = DIM) -> None:
|
|
self.model = model
|
|
self.dimensions = dimensions
|
|
self.calls: list[list[str]] = []
|
|
self.probes = 0
|
|
self.declared: int | None = None
|
|
|
|
@property
|
|
def declared_dimensions(self) -> int | None:
|
|
"""Unpinned by default, like a real config that names no width -- so
|
|
the manager has to measure, which is the path worth exercising."""
|
|
return self.declared
|
|
|
|
async def probe_dimensions(self) -> int:
|
|
self.probes += 1
|
|
return self.dimensions
|
|
|
|
async def embed(self, texts: list[str]) -> list[list[float]]:
|
|
self.calls.append(list(texts))
|
|
vectors = []
|
|
for text in texts:
|
|
lowered = text.lower()
|
|
counts = [float(lowered.count(word)) for word in _VOCAB]
|
|
# A zero vector has no direction, so cosine distance against it is
|
|
# undefined; a floor keeps every text somewhere on the sphere.
|
|
vectors.append(counts if any(counts) else [0.01] * DIM)
|
|
return vectors
|
|
|
|
|
|
class _NoStore:
|
|
"""Enough of a vector store to build a manager. Nothing here searches."""
|
|
|
|
async def delete_collection(self, collection: str) -> None:
|
|
return None
|
|
|
|
|
|
class _Endpoints:
|
|
"""The endpoints a manager can reach, other than the configured one.
|
|
|
|
A base built with a model the current pin no longer names has to be
|
|
embedded through whatever still serves that model, and the manager finds
|
|
it by building a client for the base's own model. This stands in for that
|
|
lookup: ``serve`` makes a model answerable, ``refuse`` makes it fail the
|
|
way an endpoint that does not host it would.
|
|
"""
|
|
|
|
def __init__(self) -> None:
|
|
self.clients: dict[str, _StubClient] = {}
|
|
self.refusals: dict[str, Exception] = {}
|
|
#: Every config the manager asked for a client with, so a test can say
|
|
#: which endpoint it decided on and not only which model.
|
|
self.built: list[EmbeddingConfig] = []
|
|
|
|
def serve(self, model: str, dimensions: int = DIM) -> "_StubClient":
|
|
client = _StubClient(model=model, dimensions=dimensions)
|
|
self.clients[model] = client
|
|
return client
|
|
|
|
def refuse(self, model: str, error: Exception) -> None:
|
|
self.refusals[model] = error
|
|
|
|
def build(self, config: EmbeddingConfig):
|
|
self.built.append(config)
|
|
if config.model in self.refusals:
|
|
raise self.refusals[config.model]
|
|
return self.clients.get(config.model) or self.serve(config.model)
|
|
|
|
|
|
@pytest.fixture
|
|
def endpoints(monkeypatch):
|
|
"""Route the manager's per-base client building at the stubs above."""
|
|
registry = _Endpoints()
|
|
monkeypatch.setattr("raven.knowledge._manager.embedding_client", registry.build)
|
|
monkeypatch.setattr(
|
|
"raven.knowledge._manager.embedding_config_for",
|
|
lambda provider, model, dimensions=None: (
|
|
EmbeddingConfig(
|
|
model=model, base_url="https://other/v1", api_key="k", provider=provider, dimensions=dimensions
|
|
)
|
|
if provider
|
|
else None
|
|
),
|
|
)
|
|
return registry
|
|
|
|
|
|
@pytest.fixture
|
|
def manager(tmp_path, monkeypatch):
|
|
client = _StubClient()
|
|
mgr = KnowledgeManager(
|
|
tmp_path / "knowledge",
|
|
embedding=EmbeddingConfig(model=client.model, base_url="https://x/v1", api_key="k", dimensions=DIM),
|
|
)
|
|
monkeypatch.setattr(mgr, "_client", lambda: client)
|
|
mgr.stub = client # type: ignore[attr-defined]
|
|
return mgr
|
|
|
|
|
|
#: The smallest PNG that decodes: one transparent pixel. What the parser is
|
|
#: routed by is the filename, and what the vision path needs is bytes Pillow
|
|
#: will open.
|
|
_PNG = base64.b64decode(
|
|
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
|
|
)
|
|
|
|
|
|
class _StubVision:
|
|
"""A vision model that answers without a network."""
|
|
|
|
max_figures = 64
|
|
|
|
def __init__(self, text: str) -> None:
|
|
self.text = text
|
|
self.calls: list[tuple[str, str]] = []
|
|
|
|
async def describe(self, image: bytes, *, mime: str = "", context_above: str = "", context_below: str = "") -> str:
|
|
self.calls.append((context_above, context_below))
|
|
return self.text
|
|
|
|
|
|
def _deck_of(slides: list[str]) -> bytes:
|
|
"""A minimal slide deck, one text box a slide."""
|
|
import io
|
|
import zipfile
|
|
|
|
namespaces = (
|
|
'xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main" '
|
|
'xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main" '
|
|
'xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships"'
|
|
)
|
|
buffer = io.BytesIO()
|
|
with zipfile.ZipFile(buffer, "w") as package:
|
|
entries = "".join(f'<p:sldId id="{256 + n}" r:id="rId{n + 2}"/>' for n in range(len(slides)))
|
|
package.writestr(
|
|
"ppt/presentation.xml",
|
|
f"<p:presentation {namespaces}><p:sldIdLst>{entries}</p:sldIdLst></p:presentation>",
|
|
)
|
|
rels = "".join(
|
|
f'<Relationship Id="rId{n + 2}" Target="slides/slide{n + 1}.xml" '
|
|
f'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide"/>'
|
|
for n in range(len(slides))
|
|
)
|
|
package.writestr(
|
|
"ppt/_rels/presentation.xml.rels",
|
|
'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
|
|
f"{rels}</Relationships>",
|
|
)
|
|
for n, text in enumerate(slides):
|
|
package.writestr(
|
|
f"ppt/slides/slide{n + 1}.xml",
|
|
f"<p:sld {namespaces}><p:cSld><p:spTree><p:sp><p:txBody><a:p><a:r>"
|
|
f"<a:t>{text}</a:t></a:r></a:p></p:txBody></p:sp></p:spTree></p:cSld></p:sld>",
|
|
)
|
|
return buffer.getvalue()
|
|
|
|
|
|
def _docx_with_picture() -> bytes:
|
|
"""A minimal Word package holding one paragraph and one embedded picture."""
|
|
import io
|
|
import zipfile
|
|
|
|
namespaces = (
|
|
'xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" '
|
|
'xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing" '
|
|
'xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main" '
|
|
'xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships"'
|
|
)
|
|
body = (
|
|
"<w:p><w:r><w:t>alpha alpha discussion of the first topic.</w:t></w:r></w:p>"
|
|
'<w:p><w:r><w:drawing><wp:inline><wp:docPr id="1" name="Picture 1"/>'
|
|
'<a:graphic><a:graphicData><a:blip r:embed="rId7"/></a:graphicData></a:graphic>'
|
|
"</wp:inline></w:drawing></w:r></w:p>"
|
|
)
|
|
buffer = io.BytesIO()
|
|
with zipfile.ZipFile(buffer, "w") as package:
|
|
package.writestr("word/document.xml", f"<w:document {namespaces}><w:body>{body}</w:body></w:document>")
|
|
package.writestr(
|
|
"word/_rels/document.xml.rels",
|
|
'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
|
|
'<Relationship Id="rId7" Target="media/image1.png" '
|
|
'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/image"/>'
|
|
"</Relationships>",
|
|
)
|
|
package.writestr("word/media/image1.png", _PNG)
|
|
return buffer.getvalue()
|
|
|
|
|
|
MARKDOWN = b"""# Handbook
|
|
|
|
## Alpha section
|
|
|
|
alpha alpha alpha discussion of the first topic.
|
|
|
|
## Beta section
|
|
|
|
beta beta beta notes on the second topic.
|
|
"""
|
|
|
|
|
|
async def _ready_base(manager, content: bytes = MARKDOWN, filename: str = "handbook.md"):
|
|
base = await manager.create_base(name="handbook")
|
|
# Small enough that the fixture document is more than one chunk, which is
|
|
# what the paging, deleting and disabling tests below are about. The base
|
|
# default would swallow it whole.
|
|
manager.configure_base(base.id, chunk_size=12)
|
|
doc = manager.add_document(base.id, filename=filename, content=content)
|
|
indexed = await manager.index_document(doc.id)
|
|
return base, indexed
|
|
|
|
|
|
# ── the whole path ────────────────────────────────────────────────
|
|
|
|
|
|
async def test_create_upload_index_search(manager) -> None:
|
|
base, doc = await _ready_base(manager)
|
|
|
|
assert doc.status == "ready"
|
|
assert doc.chunk_count == 2
|
|
|
|
hits = (await manager.search([base.id], "alpha", top_k=1)).hits
|
|
assert len(hits) == 1
|
|
assert "first topic" in hits[0].chunk.text
|
|
|
|
|
|
async def test_search_ranks_the_nearer_section_first(manager) -> None:
|
|
base, _ = await _ready_base(manager)
|
|
hits = (await manager.search([base.id], "beta", top_k=2)).hits
|
|
assert "second topic" in hits[0].chunk.text
|
|
|
|
|
|
async def test_headings_survive_into_the_indexed_chunks(manager) -> None:
|
|
"""The structured parser is what makes a hit citable. If the plain parser
|
|
had claimed markdown, every chunk would carry an empty heading path."""
|
|
base, _ = await _ready_base(manager)
|
|
hits = (await manager.search([base.id], "alpha", top_k=1)).hits
|
|
assert hits[0].chunk.metadata.get("heading_path") == ["Handbook", "Alpha section"]
|
|
|
|
|
|
async def test_a_plain_text_upload_still_indexes(manager) -> None:
|
|
"""TextParser has to stay behind the structured one, which claims only
|
|
markdown and HTML."""
|
|
base, doc = await _ready_base(manager, content=b"alpha notes", filename="notes.txt")
|
|
assert doc.status == "ready" and doc.chunk_count == 1
|
|
|
|
|
|
# ── indexing rules ────────────────────────────────────────────────
|
|
|
|
|
|
async def test_a_document_starts_queued_and_index_pending_drains_it(manager) -> None:
|
|
base = await manager.create_base(name="handbook")
|
|
manager.add_document(base.id, filename="a.md", content=MARKDOWN)
|
|
manager.add_document(base.id, filename="b.md", content=b"beta")
|
|
|
|
assert await manager.index_pending() == 2
|
|
assert {d.status for d in manager.list_documents(base.id)} == {"ready"}
|
|
assert await manager.index_pending() == 0
|
|
|
|
|
|
async def test_reindexing_replaces_instead_of_duplicating(manager) -> None:
|
|
"""Appending would leave the previous run's chunks in the collection and
|
|
every hit would come back twice."""
|
|
base, doc = await _ready_base(manager)
|
|
await manager.index_document(doc.id)
|
|
|
|
hits = (await manager.search([base.id], "alpha", top_k=10)).hits
|
|
assert len({h.chunk.text for h in hits}) == len(hits)
|
|
|
|
|
|
async def test_a_deck_indexes_one_chunk_a_slide(manager) -> None:
|
|
"""The end-to-end shape the slide parser exists for: a chunk and a page are
|
|
the same thing, so a hit can take the preview to the slide it came from."""
|
|
deck = _deck_of(["Opening remarks", "The second slide", "Closing"])
|
|
base = await manager.create_base(name="decks")
|
|
# Small enough that any other parser's sections would be cut several times
|
|
# over, which is what makes this a test of the guarantee rather than of the
|
|
# size of the fixture.
|
|
manager.configure_base(base.id, chunk_size=4)
|
|
document = manager.add_document(base.id, filename="deck.pptx", content=deck)
|
|
|
|
indexed = await manager.index_document(document.id)
|
|
|
|
assert indexed.status == "ready"
|
|
assert indexed.chunk_count == 3
|
|
chunks, _ = await manager.document_chunks(document.id)
|
|
assert [piece.chunk.metadata["page_number"] for piece in chunks] == [1, 2, 3]
|
|
assert [piece.chunk.text for piece in chunks] == ["Opening remarks", "The second slide", "Closing"]
|
|
|
|
|
|
async def test_an_unparseable_upload_fails_that_document_only(manager) -> None:
|
|
"""One bad upload must not stop the queue behind it, and the reason
|
|
belongs on the row the person who uploaded it is looking at."""
|
|
base = await manager.create_base(name="handbook")
|
|
bad = manager.add_document(base.id, filename="firmware.bin", content=b"\x00\x01\x02")
|
|
good = manager.add_document(base.id, filename="ok.md", content=MARKDOWN)
|
|
|
|
await manager.index_pending()
|
|
|
|
assert manager.get_document(bad.id).status == "failed"
|
|
assert "no parser" in manager.get_document(bad.id).error
|
|
assert manager.get_document(good.id).status == "ready"
|
|
|
|
|
|
async def test_an_image_with_no_vision_model_fails_with_the_reason(manager, monkeypatch) -> None:
|
|
"""A picture has no text of its own, so a build that cannot look at one has
|
|
nothing to index -- and the row has to say that rather than reporting a
|
|
document with no chunks as ready."""
|
|
monkeypatch.setattr("raven.knowledge._vision.load_vision_model", lambda: None)
|
|
base = await manager.create_base(name="handbook")
|
|
picture = manager.add_document(base.id, filename="chart.png", content=_PNG)
|
|
good = manager.add_document(base.id, filename="ok.md", content=MARKDOWN)
|
|
|
|
await manager.index_pending()
|
|
|
|
assert manager.get_document(picture.id).status == "failed"
|
|
assert "no vision model is configured" in manager.get_document(picture.id).error
|
|
assert manager.get_document(good.id).status == "ready"
|
|
|
|
|
|
async def test_an_image_is_indexed_as_one_chunk(manager, monkeypatch) -> None:
|
|
"""One picture, one chunk, however much the model had to say about it: the
|
|
section is marked as a figure, and the chunker never splits one."""
|
|
described = ("A bar chart. " * 400).strip()
|
|
monkeypatch.setattr(
|
|
"raven.knowledge._vision.load_vision_model",
|
|
lambda: _StubVision(described),
|
|
)
|
|
base = await manager.create_base(name="handbook")
|
|
picture = manager.add_document(base.id, filename="chart.png", content=_PNG)
|
|
|
|
indexed = await manager.index_document(picture.id)
|
|
|
|
assert indexed.status == "ready"
|
|
assert indexed.chunk_count == 1
|
|
chunks, _ = await manager.document_chunks(picture.id)
|
|
assert chunks[0].chunk.text == described
|
|
|
|
|
|
async def test_a_picture_nobody_could_read_warns_on_a_searchable_row(manager, monkeypatch) -> None:
|
|
"""The case this warning exists for: the document indexed, the row says
|
|
ready, and half of what the file shows is not in the index. Nothing else
|
|
would ever tell a reader -- a search about the missing part just answers
|
|
worse."""
|
|
monkeypatch.setattr("raven.knowledge._vision.load_vision_model", lambda: None)
|
|
base = await manager.create_base(name="handbook")
|
|
doc = manager.add_document(base.id, filename="report.docx", content=_docx_with_picture())
|
|
|
|
indexed = await manager.index_document(doc.id)
|
|
|
|
assert indexed.status == "ready", "still indexed, still searchable"
|
|
assert indexed.chunk_count >= 1
|
|
assert "no vision model is configured" in indexed.warning
|
|
assert indexed.error == "", "a warning is not a failure"
|
|
|
|
|
|
async def test_the_warning_goes_when_the_reindex_reads_the_picture(manager, monkeypatch) -> None:
|
|
"""Otherwise a row keeps reporting a gap that has since been filled."""
|
|
monkeypatch.setattr("raven.knowledge._vision.load_vision_model", lambda: None)
|
|
base = await manager.create_base(name="handbook")
|
|
doc = manager.add_document(base.id, filename="report.docx", content=_docx_with_picture())
|
|
assert (await manager.index_document(doc.id)).warning
|
|
|
|
monkeypatch.setattr("raven.knowledge._vision.load_vision_model", lambda: _StubVision("A bar chart."))
|
|
assert (await manager.index_document(doc.id)).warning == ""
|
|
|
|
|
|
async def test_a_document_that_parsed_whole_carries_no_warning(manager) -> None:
|
|
base, doc = await _ready_base(manager)
|
|
|
|
assert doc.warning == ""
|
|
|
|
|
|
async def test_indexing_a_document_whose_base_is_gone_fails_it(manager) -> None:
|
|
base = await manager.create_base(name="handbook")
|
|
doc = manager.add_document(base.id, filename="a.md", content=MARKDOWN)
|
|
await manager.delete_base(base.id)
|
|
|
|
assert await manager.index_document(doc.id) is None
|
|
|
|
|
|
# ── the staleness rule ────────────────────────────────────────────
|
|
|
|
|
|
async def test_a_base_is_searched_with_the_model_it_was_built_with(manager, endpoints) -> None:
|
|
"""The vectors answer to the model the base was built with, so that is the
|
|
model the query is embedded with -- moving the configured pin does not move
|
|
the space an existing collection lives in."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.stub.model = "a-different-model"
|
|
old_model = endpoints.serve("stub-embed")
|
|
|
|
hits = (await manager.search([base.id], "alpha", top_k=1)).hits
|
|
|
|
assert len(hits) == 1
|
|
assert old_model.calls == [["alpha"]], "the query went to the base's own model"
|
|
assert manager.stub.calls[-1] != ["alpha"], "and not to the one configured now"
|
|
|
|
|
|
async def test_a_base_whose_model_cannot_be_reached_answers_by_keyword(manager, endpoints) -> None:
|
|
"""The words still work. A base whose model has gone answers worse rather
|
|
than not at all, and says which of its answers came from words."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.stub.model = "a-different-model"
|
|
endpoints.refuse("stub-embed", EmbeddingError("model not served here"))
|
|
|
|
outcome = await manager.search([base.id], "alpha")
|
|
|
|
assert outcome.hits, "the text is still there to match against"
|
|
assert "alpha" in outcome.hits[0].chunk.text.lower()
|
|
assert base.id in outcome.by_keyword
|
|
assert "Point the base at a provider that serves it" in outcome.by_keyword[base.id]
|
|
|
|
|
|
async def test_a_width_change_is_also_stale(manager) -> None:
|
|
"""The base records both the model and the width, and the width leg has to
|
|
stand on its own: a model can be redeployed at a different width under the
|
|
same name. Clearing the memo is what a fresh process does.
|
|
|
|
Stale means the vectors cannot be queried, not that the base is gone: it
|
|
answers by keyword and says why, and the refusal that matters stays on the
|
|
indexing side, where writing into a mismatched collection would corrupt
|
|
it."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.stub.dimensions = DIM + 1
|
|
manager._widths.clear()
|
|
|
|
outcome = await manager.search([base.id], "alpha")
|
|
|
|
assert base.id in outcome.by_keyword
|
|
assert "rebuild the base" in outcome.by_keyword[base.id]
|
|
|
|
|
|
async def test_a_pin_that_contradicts_the_model_is_refused(manager) -> None:
|
|
"""Found by running against a real endpoint: the deployment's config pins
|
|
1024 for a model that returns 4096. Trusting the pin sized the collection
|
|
to 1024 and the first insert died on an Arrow cast error naming neither the
|
|
config nor the model."""
|
|
manager.stub.declared = DIM + 1
|
|
|
|
with pytest.raises(KnowledgeError, match="not what"):
|
|
await manager.create_base(name="handbook")
|
|
|
|
|
|
async def test_a_pin_that_agrees_is_accepted(manager) -> None:
|
|
manager.stub.declared = DIM
|
|
base = await manager.create_base(name="handbook")
|
|
assert base.dimensions == DIM
|
|
|
|
|
|
async def test_the_width_is_probed_once_per_model(manager) -> None:
|
|
base, _ = await _ready_base(manager)
|
|
await manager.search([base.id], "alpha")
|
|
await manager.search([base.id], "beta")
|
|
assert manager.stub.probes == 1
|
|
|
|
|
|
async def test_moving_the_endpoint_does_not_make_a_base_stale(manager) -> None:
|
|
"""Rotating a key or putting the same model behind a new gateway changes
|
|
neither the vectors nor what a query embeds to. Rebuilding for that throws
|
|
away a working index for nothing."""
|
|
base, _ = await _ready_base(manager)
|
|
manager._embedding = EmbeddingConfig(
|
|
model=manager.stub.model, base_url="https://moved/v1", api_key="rotated", dimensions=DIM
|
|
)
|
|
|
|
assert (await manager.search([base.id], "alpha", top_k=1)).hits
|
|
|
|
|
|
async def test_indexing_into_a_stale_base_fails_the_document_not_the_queue(manager) -> None:
|
|
base, _ = await _ready_base(manager)
|
|
manager.stub.dimensions = DIM + 1
|
|
manager._widths.clear()
|
|
doc = manager.add_document(base.id, filename="b.md", content=b"beta")
|
|
|
|
assert (await manager.index_document(doc.id)).status == "failed"
|
|
assert "rebuild" in manager.get_document(doc.id).error
|
|
|
|
|
|
async def test_a_new_document_is_embedded_with_the_base_own_model(manager, endpoints) -> None:
|
|
"""A document added to an older base has to land in the space that base's
|
|
collection already holds, or nothing it says is findable."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.stub.model = "a-different-model"
|
|
old_model = endpoints.serve("stub-embed")
|
|
doc = manager.add_document(base.id, filename="b.md", content=b"# beta\n\nbeta beta beta\n")
|
|
|
|
assert (await manager.index_document(doc.id)).status == "ready"
|
|
assert old_model.calls, "the chunks went to the base's own model"
|
|
assert (await manager.search([base.id], "beta", top_k=1)).hits
|
|
|
|
|
|
# -- one embedding per model, not one per search -------------------
|
|
|
|
|
|
async def test_each_base_is_embedded_with_its_own_model(manager, endpoints) -> None:
|
|
"""The point of the whole arrangement: two bases built with two models are
|
|
two different vector spaces, and one query vector cannot address both."""
|
|
first, _ = await _ready_base(manager)
|
|
# The pin moves, a second base is built on it, and it moves back -- so the
|
|
# two bases hold vectors from two models and neither is the odd one out.
|
|
manager.stub.model = "second-model"
|
|
second = await manager.create_base(name="second")
|
|
doc = manager.add_document(second.id, filename="s.md", content=b"# gamma\n\ngamma gamma\n")
|
|
assert (await manager.index_document(doc.id)).status == "ready"
|
|
manager.stub.model = "stub-embed"
|
|
other = endpoints.serve("second-model")
|
|
|
|
outcome = await manager.search([first.id, second.id], "alpha gamma")
|
|
|
|
assert manager.stub.calls[-1] == ["alpha gamma"], "the base on the current model used it"
|
|
assert other.calls == [["alpha gamma"]], "the base on the older model used that one"
|
|
assert outcome.by_keyword == {}
|
|
assert {hit.chunk.source for hit in outcome.hits} == {"handbook.md", "s.md"}
|
|
|
|
|
|
async def test_bases_sharing_a_model_share_one_embedding_call(manager) -> None:
|
|
"""The common case still costs one round trip: what is grouped is the
|
|
model, not the base."""
|
|
first, _ = await _ready_base(manager)
|
|
second = await manager.create_base(name="second")
|
|
doc = manager.add_document(second.id, filename="s.md", content=b"# gamma\n\ngamma\n")
|
|
await manager.index_document(doc.id)
|
|
before = len(manager.stub.calls)
|
|
|
|
await manager.search([first.id, second.id], "alpha")
|
|
|
|
assert len(manager.stub.calls) - before == 1
|
|
|
|
|
|
async def test_one_unreachable_base_does_not_take_the_others_down(manager, endpoints) -> None:
|
|
"""Asking a mixed set is ordinary. The base that fell back to words is
|
|
named, so a caller is not left thinking every hit was scored the same
|
|
way."""
|
|
good, _ = await _ready_base(manager)
|
|
manager.stub.model = "gone-model"
|
|
stranded = await manager.create_base(name="stranded")
|
|
doc = manager.add_document(stranded.id, filename="s.md", content=b"# gamma\n\ngamma\n")
|
|
await manager.index_document(doc.id)
|
|
manager.stub.model = "stub-embed"
|
|
endpoints.refuse("gone-model", EmbeddingError("no endpoint serves it"))
|
|
|
|
outcome = await manager.search([good.id, stranded.id], "alpha")
|
|
|
|
assert "handbook.md" in {hit.chunk.source for hit in outcome.hits}
|
|
assert list(outcome.by_keyword) == [stranded.id]
|
|
assert "no endpoint serves it" in outcome.by_keyword[stranded.id]
|
|
|
|
|
|
async def test_a_base_with_no_provider_falls_back_to_the_configured_endpoint(manager, endpoints) -> None:
|
|
"""The last thing left to try for a base written before providers were
|
|
recorded. Raven no longer reads the memory backend's own config for an
|
|
older endpoint: a knowledge base that needs the memory plugin installed to
|
|
answer is one that stops working when it is not.
|
|
"""
|
|
base, _ = await _ready_base(manager)
|
|
manager.stub.model = "a-different-model"
|
|
served = endpoints.serve("stub-embed")
|
|
|
|
hits = (await manager.search([base.id], "alpha", top_k=1)).hits
|
|
|
|
assert hits and served.calls == [["alpha"]], "asked for the base's own model"
|
|
assert endpoints.built[-1].model == "stub-embed"
|
|
|
|
|
|
async def test_a_base_can_be_pointed_at_the_provider_that_still_serves_it(manager, endpoints) -> None:
|
|
"""The way back for a base stranded by a change of pin, short of a rebuild:
|
|
the model and the width are what the collection was built to and cannot
|
|
move, but where that model is reached can."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.stub.model = "a-different-model"
|
|
endpoints.refuse("stub-embed", EmbeddingError("this endpoint does not serve it"))
|
|
served_elsewhere = _StubClient(model="stub-embed")
|
|
endpoints.clients["stub-embed"] = served_elsewhere
|
|
endpoints.refusals.clear()
|
|
|
|
assert manager.configure_base(base.id, embedding_provider="siliconflow").embedding_provider == "siliconflow"
|
|
hits = (await manager.search([base.id], "alpha", top_k=1)).hits
|
|
|
|
assert hits and served_elsewhere.calls == [["alpha"]]
|
|
|
|
|
|
async def test_a_base_can_be_built_on_a_picked_model(manager, endpoints) -> None:
|
|
"""The page offers every embedding model the install can reach, so the pair
|
|
it picked is what the base is built on -- not the configured pin, which is
|
|
only the default the picker starts on."""
|
|
picked = endpoints.serve("picked-model")
|
|
|
|
base = await manager.create_base(name="handbook", embedding_model="picked-model", embedding_provider="openai")
|
|
|
|
assert (base.embedding_model, base.embedding_provider) == ("picked-model", "openai")
|
|
assert picked.probes == 1, "the width was measured against the model that was picked"
|
|
assert manager.stub.probes == 0, "and the configured one was never asked"
|
|
|
|
|
|
async def test_switching_the_model_rebuilds_what_the_base_holds(manager, endpoints) -> None:
|
|
"""The vectors in the collection were made by the old model, and no query
|
|
embedded by the new one lands anywhere near them. So the documents go back
|
|
to the queue rather than staying ready against an index that is gone."""
|
|
base, doc = await _ready_base(manager)
|
|
endpoints.serve("second-model")
|
|
|
|
switched = await manager.switch_embedding(base.id, model="second-model", provider="openai")
|
|
|
|
assert (switched.embedding_model, switched.embedding_provider) == ("second-model", "openai")
|
|
assert [d.status for d in manager.list_documents(base.id)] == ["pending"]
|
|
assert manager.get_document(doc.id).chunk_count == 0
|
|
|
|
|
|
async def test_the_requeued_documents_index_on_the_new_model(manager, endpoints) -> None:
|
|
"""Which is the point of requeueing them: the blob is still on disk, so a
|
|
reindex is all it takes to have the base searchable again."""
|
|
base, _ = await _ready_base(manager)
|
|
second = endpoints.serve("second-model")
|
|
await manager.switch_embedding(base.id, model="second-model", provider="openai")
|
|
|
|
assert await manager.index_pending() == 1
|
|
|
|
hits = (await manager.search([base.id], "alpha", top_k=1)).hits
|
|
assert hits and "first topic" in hits[0].chunk.text
|
|
assert ["alpha"] in second.calls, "the query went to the model the base now holds"
|
|
|
|
|
|
async def test_the_same_model_through_another_account_is_not_a_rebuild(manager, endpoints) -> None:
|
|
"""Nothing about the vectors changed -- only the address the next call goes
|
|
out on -- so requeueing every document would be work for nothing."""
|
|
base, doc = await _ready_base(manager)
|
|
endpoints.serve("stub-embed")
|
|
|
|
switched = await manager.switch_embedding(base.id, model="stub-embed", provider="siliconflow")
|
|
|
|
assert switched.embedding_provider == "siliconflow"
|
|
assert manager.get_document(doc.id).status == "ready"
|
|
|
|
|
|
async def test_a_model_that_cannot_be_reached_leaves_the_base_alone(manager, endpoints) -> None:
|
|
"""The width is measured before anything is dropped, so a picked model that
|
|
refuses costs the reader a message rather than their index."""
|
|
base, doc = await _ready_base(manager)
|
|
endpoints.refuse("second-model", EmbeddingError("this endpoint does not serve it"))
|
|
|
|
with pytest.raises(EmbeddingError):
|
|
await manager.switch_embedding(base.id, model="second-model", provider="openai")
|
|
|
|
assert manager.get_base(base.id).embedding_model == "stub-embed"
|
|
assert manager.get_document(doc.id).status == "ready"
|
|
assert (await manager.search([base.id], "alpha", top_k=1)).hits
|
|
|
|
|
|
async def test_embedding_can_be_turned_off_and_on_again(manager, endpoints) -> None:
|
|
"""The one choice that used to be fixed at creation: a base made without a
|
|
model was rebuilt or nothing, and now it is the same call as any other
|
|
switch."""
|
|
base, doc = await _ready_base(manager)
|
|
|
|
off = await manager.switch_embedding(base.id, model="")
|
|
assert (off.embedding_model, off.dimensions, off.embedding_provider) == ("", 0, "")
|
|
assert not manager.embeds(off)
|
|
|
|
endpoints.serve("stub-embed")
|
|
on = await manager.switch_embedding(base.id, model="stub-embed", provider="openai")
|
|
assert manager.embeds(on)
|
|
assert await manager.index_pending() >= 1
|
|
assert manager.get_document(doc.id).status == "ready"
|
|
|
|
|
|
async def test_switching_to_what_the_base_already_has_changes_nothing(manager) -> None:
|
|
base, doc = await _ready_base(manager)
|
|
before = manager.get_base(base.id)
|
|
|
|
same = await manager.switch_embedding(base.id, model=before.embedding_model, provider=before.embedding_provider)
|
|
|
|
assert same == before
|
|
assert manager.get_document(doc.id).status == "ready"
|
|
|
|
|
|
async def test_a_base_keeps_its_own_provider_when_the_model_ids_collide(manager, endpoints) -> None:
|
|
"""Two providers reselling one model id is ordinary, and the id alone does
|
|
not name a credential -- which is the whole reason a base records the
|
|
provider beside it. Matching on the id, the base's queries went out on the
|
|
pinned account's key against vectors the other account had made, and fell
|
|
back to keywords whenever the pinned one was the one that was down."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.configure_base(base.id, embedding_provider="siliconflow")
|
|
# The pin serves the same model id, through somebody else.
|
|
manager.stub.provider = "dashscope"
|
|
|
|
client = manager._client_for(manager.get_base(base.id))
|
|
|
|
assert endpoints.built[-1].provider == "siliconflow", "the account the base records"
|
|
assert client is not manager.stub, "and not the pinned one that happens to share the id"
|
|
|
|
|
|
async def test_reach_asks_about_the_provider_the_base_records(manager, monkeypatch) -> None:
|
|
"""The other half of the same predicate. Answering from the model id alone
|
|
called a base reachable because the *pin* was usable, while the account it
|
|
actually queries through had no credential left."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.configure_base(base.id, embedding_provider="gone")
|
|
manager.stub.provider = "dashscope"
|
|
monkeypatch.setattr("raven.knowledge._manager.embedding_config_for", lambda *a, **k: None)
|
|
|
|
assert manager.embedding_reach(manager.get_base(base.id)) == "no_credential"
|
|
|
|
|
|
async def test_a_base_on_the_pin_itself_still_takes_it(manager) -> None:
|
|
"""The common case, and the one the check above must not cost: same model,
|
|
same provider, today's client unchanged."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.stub.provider = "siliconflow"
|
|
manager.configure_base(base.id, embedding_provider="siliconflow")
|
|
|
|
assert manager._client_for(manager.get_base(base.id)) is manager.stub
|
|
|
|
|
|
async def test_a_base_with_no_provider_recorded_still_follows_the_pin(manager) -> None:
|
|
"""An empty provider means "wherever this is configured", which is every
|
|
base written before the field existed -- the one case where the model id
|
|
alone decides."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.stub.provider = "dashscope"
|
|
|
|
assert manager.get_base(base.id).embedding_provider == ""
|
|
assert manager._client_for(manager.get_base(base.id)) is manager.stub
|
|
|
|
|
|
def _clear_the_pin(manager, monkeypatch) -> None:
|
|
"""Leave the deployment with no configured embedding endpoint at all.
|
|
|
|
The fixture hands the manager a stub through both `_embedding` and a
|
|
patched `_client`, so a test about the pin being absent has to undo both --
|
|
including the seam, or `_configured_client` answers with the stub it was
|
|
handed and the path under test is never reached.
|
|
"""
|
|
monkeypatch.setattr(manager, "_embedding", None)
|
|
monkeypatch.setattr("raven.knowledge._manager.load_embedding_config", lambda: None)
|
|
monkeypatch.setattr(manager, "_client", KnowledgeManager._client.__get__(manager))
|
|
|
|
|
|
async def test_a_base_keeps_its_own_provider_when_the_pin_is_cleared(manager, endpoints, monkeypatch) -> None:
|
|
"""The pin is the default for a *new* base, not a precondition for an old
|
|
one. Asking for it first and failing when there is none made every base
|
|
that records its own provider unsearchable the moment somebody cleared the
|
|
pin -- while `embedding_reach` went on calling those bases reachable."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.configure_base(base.id, embedding_provider="siliconflow")
|
|
served = endpoints.serve("stub-embed")
|
|
|
|
_clear_the_pin(manager, monkeypatch)
|
|
|
|
outcome = await manager.search([base.id], "alpha", top_k=1)
|
|
|
|
assert outcome.hits, "the base still answers by meaning"
|
|
assert served.calls == [["alpha"]], "through the provider it recorded"
|
|
assert outcome.by_keyword == {}, "and not by falling back to words"
|
|
|
|
|
|
async def test_reach_and_the_client_agree_about_a_base_with_no_pin(manager, endpoints, monkeypatch) -> None:
|
|
"""The two used to disagree, which is the part that made this hard to see:
|
|
the page said reachable and the indexer said no endpoint is configured."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.configure_base(base.id, embedding_provider="siliconflow")
|
|
endpoints.serve("stub-embed")
|
|
_clear_the_pin(manager, monkeypatch)
|
|
|
|
record = manager.get_base(base.id)
|
|
assert manager.embedding_reach(record) == "", "the page calls it reachable"
|
|
assert manager._client_for(record) is not None, "and so does the indexer"
|
|
|
|
|
|
async def test_a_base_with_no_provider_and_no_pin_still_says_so(manager, monkeypatch) -> None:
|
|
"""The one case that genuinely needs the configured endpoint keeps needing
|
|
it, and says which."""
|
|
base, _ = await _ready_base(manager)
|
|
_clear_the_pin(manager, monkeypatch)
|
|
|
|
with pytest.raises(KnowledgeError, match="no embedding endpoint is configured"):
|
|
manager._client_for(manager.get_base(base.id))
|
|
|
|
|
|
# ── chunk edits stay inside their document ────────────────────────
|
|
|
|
|
|
async def _two_documents(manager):
|
|
"""Two documents in one base holding the same sentence."""
|
|
base = await manager.create_base(name="handbook")
|
|
manager.configure_base(base.id, chunk_size=12)
|
|
one = manager.add_document(base.id, filename="one.md", content=b"alpha alpha shared sentence")
|
|
two = manager.add_document(base.id, filename="two.md", content=b"alpha alpha shared sentence")
|
|
await manager.index_pending()
|
|
return base, one, two
|
|
|
|
|
|
async def test_disabling_a_chunk_cannot_reach_another_document(manager) -> None:
|
|
"""A chunk id names its text, and two documents in one base can hold the
|
|
same sentence -- so an id from one of them must not switch the identical
|
|
row in the other."""
|
|
base, one, two = await _two_documents(manager)
|
|
held, _ = await manager.document_chunks(two.id)
|
|
borrowed = [piece.chunk_id for piece in held]
|
|
|
|
changed = await manager.set_chunks_enabled(one.id, borrowed, False)
|
|
|
|
assert changed == 0, "nothing in this document matched those ids"
|
|
after, _ = await manager.document_chunks(two.id)
|
|
assert all(piece.enabled for piece in after), "and the other document is untouched"
|
|
|
|
|
|
async def test_deleting_a_chunk_cannot_reach_another_document(manager) -> None:
|
|
"""The same rule, and here the count depends on it: an unscoped delete
|
|
removed the other document's rows and then rewrote this document's
|
|
count, leaving both wrong."""
|
|
base, one, two = await _two_documents(manager)
|
|
held, before = await manager.document_chunks(two.id)
|
|
borrowed = [piece.chunk_id for piece in held]
|
|
|
|
remaining = await manager.delete_chunks(one.id, borrowed)
|
|
|
|
_, still = await manager.document_chunks(two.id)
|
|
assert still == before, "the other document kept every piece"
|
|
assert remaining == manager.get_document(one.id).chunk_count
|
|
assert remaining > 0, "and this one kept its own"
|
|
|
|
|
|
# ── a hand edit leaves the numbering intact ───────────────────────
|
|
|
|
|
|
async def test_deleting_the_middle_piece_renumbers_the_rest(manager) -> None:
|
|
"""`chunk_index` runs 0..N-1 and every piece agrees on N -- the chunker's
|
|
own contract, which a hand edit has to leave standing. Deleting the middle
|
|
of three left 0 and 2 of 3, so a hit reported a position its document
|
|
disagreed with."""
|
|
base, doc = await _ready_base(manager)
|
|
held, total = await manager.document_chunks(doc.id)
|
|
assert total >= 2, "the fixture has to be more than one chunk for this to mean anything"
|
|
|
|
await manager.delete_chunks(doc.id, [held[0].chunk_id])
|
|
|
|
after, count = await manager.document_chunks(doc.id)
|
|
assert [piece.chunk.chunk_index for piece in after] == list(range(count))
|
|
assert {piece.chunk.total_chunks for piece in after} == {count}
|
|
|
|
|
|
async def test_an_appended_piece_renumbers_the_document(manager) -> None:
|
|
"""The mirror: the rows already there said "of N" while the new one said
|
|
"of N+1"."""
|
|
base, doc = await _ready_base(manager)
|
|
|
|
written = await manager.add_chunk(doc.id, "A note somebody typed.")
|
|
|
|
after, count = await manager.document_chunks(doc.id)
|
|
assert [piece.chunk.chunk_index for piece in after] == list(range(count))
|
|
assert {piece.chunk.total_chunks for piece in after} == {count}
|
|
assert written.chunk.total_chunks == count, "including the piece that was just written"
|
|
|
|
|
|
async def test_a_new_base_records_who_served_its_model(manager) -> None:
|
|
"""Recorded at creation because a model id does not name a credential: the
|
|
query has to go back out on that provider's address later."""
|
|
manager.stub.provider = "siliconflow"
|
|
|
|
base = await manager.create_base(name="recorded")
|
|
|
|
assert base.embedding_provider == "siliconflow"
|
|
|
|
|
|
async def test_editing_a_chunk_re_embeds_it(manager) -> None:
|
|
"""A piece whose text changed and whose vector did not would be found by
|
|
the old words and read as the new ones."""
|
|
base, doc = await _ready_base(manager)
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
target = held[0]
|
|
|
|
written = await manager.update_chunk(doc.id, target.chunk_id, "epsilon epsilon epsilon")
|
|
|
|
assert written.chunk.text == "epsilon epsilon epsilon"
|
|
assert written.chunk_id != target.chunk_id, "the id follows the content"
|
|
assert written.chunk.chunk_index == target.chunk.chunk_index, "and its place is kept"
|
|
assert written.manual is True
|
|
hits = (await manager.search([base.id], "epsilon")).hits
|
|
assert any("epsilon" in hit.chunk.text for hit in hits)
|
|
after, total = await manager.document_chunks(doc.id)
|
|
assert total == len(held), "an edit replaces, it does not add"
|
|
|
|
|
|
async def test_an_edited_chunk_keeps_the_state_it_was_in(manager) -> None:
|
|
_, doc = await _ready_base(manager)
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
target = held[0]
|
|
await manager.set_chunks_enabled(doc.id, [target.chunk_id], False)
|
|
|
|
written = await manager.update_chunk(doc.id, target.chunk_id, "rewritten while off")
|
|
|
|
assert written.enabled is False
|
|
after, _ = await manager.document_chunks(doc.id)
|
|
assert next(p for p in after if p.chunk_id == written.chunk_id).enabled is False
|
|
|
|
|
|
async def test_editing_a_chunk_to_the_same_text_is_not_a_write(manager) -> None:
|
|
"""No new vector, no new id: there is nothing for either to follow."""
|
|
_, doc = await _ready_base(manager)
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
target = held[0]
|
|
|
|
written = await manager.update_chunk(doc.id, target.chunk_id, target.chunk.text)
|
|
|
|
assert written.chunk_id == target.chunk_id
|
|
|
|
|
|
async def test_editing_a_chunk_that_is_not_there_is_refused(manager) -> None:
|
|
_, doc = await _ready_base(manager)
|
|
|
|
with pytest.raises(KnowledgeError):
|
|
await manager.update_chunk(doc.id, "no-such-chunk", "text")
|
|
|
|
|
|
async def test_a_legacy_word_document_has_a_parser(manager) -> None:
|
|
"""It had none, so every reindex of a base holding one failed on it -- a
|
|
file the panel can preview and the engine refused to read."""
|
|
from raven.knowledge._manager import _default_parsers
|
|
|
|
claimed = {media for parser in _default_parsers() for media in parser.supported_media_types}
|
|
|
|
assert "application/msword" in claimed
|
|
assert ".doc" in manager.supported_extensions()
|
|
|
|
|
|
async def test_a_table_carries_its_surroundings_without_being_asked(manager) -> None:
|
|
"""On by default, and it reaches the bases that predate the setting: none
|
|
of them stored a value, so they take what the record defaults to. A table
|
|
on its own embeds as a grid of values with nothing saying what they are
|
|
about."""
|
|
base = await manager.create_base(name="tables")
|
|
|
|
assert base.table_context_size == 64
|
|
assert base.image_context_size == 64
|
|
chunker = manager._chunker_for(base)
|
|
assert chunker.table_context_size == 64
|
|
|
|
|
|
# -- whether a base can embed at all -------------------------------
|
|
|
|
|
|
async def test_a_base_on_the_configured_model_can_embed(manager) -> None:
|
|
base = await manager.create_base(name="current")
|
|
|
|
assert manager.embedding_reach(base) == ""
|
|
|
|
|
|
async def test_a_base_with_no_provider_and_another_model_says_so(manager) -> None:
|
|
"""The case that actually happens: a base built before providers were
|
|
recorded, on a model the configured endpoint does not serve. Nothing it
|
|
holds can be indexed until that is fixed, and the reason belongs where the
|
|
reader is looking rather than in a line of the gateway log."""
|
|
base = await manager.create_base(name="stranded")
|
|
# The pin moves under it, which is what leaves a base naming a model the
|
|
# configured endpoint does not serve.
|
|
manager._embedding = replace(manager._embedding, model="a-different-model")
|
|
|
|
assert manager.embedding_reach(manager.get_base(base.id)) == "no_provider"
|
|
|
|
|
|
async def test_a_base_whose_provider_has_no_credential_says_which(manager, endpoints, monkeypatch) -> None:
|
|
"""A different repair from the one above, so a different answer."""
|
|
base = await manager.create_base(name="keyless")
|
|
manager._embedding = replace(manager._embedding, model="a-different-model")
|
|
manager.configure_base(base.id, embedding_provider="siliconflow")
|
|
monkeypatch.setattr("raven.knowledge._manager.embedding_config_for", lambda *a, **k: None)
|
|
|
|
assert manager.embedding_reach(manager.get_base(base.id)) == "no_credential"
|
|
|
|
|
|
async def test_a_base_with_no_model_is_not_unreachable(manager) -> None:
|
|
"""A base created without embedding is not broken; it is what it asked to
|
|
be, and warning about it would be warning about a choice."""
|
|
base = await manager.create_base(name="files only", embedding=False)
|
|
|
|
assert manager.embedding_reach(base) == ""
|
|
|
|
|
|
# -- the settings a base is chunked by -----------------------------
|
|
|
|
|
|
async def test_a_base_is_chunked_the_way_it_is_configured(manager) -> None:
|
|
"""The four chunking settings were saved, shown, and read by nothing: one
|
|
chunker was built at startup and used for every base."""
|
|
base = await manager.create_base(name="naive")
|
|
manager.configure_base(base.id, smart_chunking=False, separator="!", chunk_size=1)
|
|
doc = manager.add_document(base.id, filename="a.md", content=b"alpha!beta!gamma")
|
|
await manager.index_document(doc.id)
|
|
|
|
held, total = await manager.document_chunks(doc.id)
|
|
|
|
assert total == 3, "cut on the separator this base asked for"
|
|
assert [p.chunk.text for p in held] == ["alpha", "beta", "gamma"]
|
|
|
|
|
|
async def test_every_base_is_cut_the_naive_way_for_now(manager) -> None:
|
|
"""The structural chunker is being reworked, so the strategy setting is
|
|
kept and not consulted: a base asking for smart chunking is cut on its
|
|
delimiters like every other one."""
|
|
base = await manager.create_base(name="smart")
|
|
manager.configure_base(base.id, smart_chunking=True, separator="!", chunk_size=1)
|
|
doc = manager.add_document(base.id, filename="a.md", content=b"alpha!beta")
|
|
await manager.index_document(doc.id)
|
|
|
|
held, total = await manager.document_chunks(doc.id)
|
|
|
|
assert total == 2 and [p.chunk.text for p in held] == ["alpha", "beta"]
|
|
|
|
|
|
async def test_an_overlap_nobody_chose_is_not_applied(manager) -> None:
|
|
"""Every base carried 215 while nothing read the setting, so no document
|
|
was ever chunked with it and nobody picked it."""
|
|
base = await manager.create_base(name="legacy")
|
|
manager._records._bases[base.id] = replace(base, chunk_overlap=215)
|
|
manager._records._save()
|
|
manager._records._bases.clear()
|
|
manager._records._load()
|
|
|
|
assert manager.get_base(base.id).chunk_overlap == 0
|
|
|
|
|
|
# -- keywords, when the vectors cannot be reached ------------------
|
|
|
|
|
|
async def test_keyword_search_finds_chinese_text(manager, endpoints) -> None:
|
|
"""The reason the keyword index is tokenized by n-grams rather than by
|
|
words: a language that writes without spaces matches nothing under the
|
|
default tokenizer, and this deployment's documents are in one."""
|
|
base = await manager.create_base(name="zh")
|
|
body = "# \u6807\u9898\n\n\u5ef6\u8fdf\u5728\u7b2c\u4e8c\u5b63\u5ea6\u4e0a\u5347\uff0c\u541e\u5410\u91cf\u4e0b\u964d\u3002\n"
|
|
doc = manager.add_document(base.id, filename="zh.md", content=body.encode())
|
|
assert (await manager.index_document(doc.id)).status == "ready"
|
|
manager.stub.model = "gone"
|
|
endpoints.refuse("stub-embed", EmbeddingError("no endpoint"))
|
|
|
|
outcome = await manager.search([base.id], "\u5ef6\u8fdf")
|
|
|
|
assert outcome.hits, "a chinese query matched chinese text"
|
|
assert base.id in outcome.by_keyword
|
|
|
|
|
|
async def test_a_disabled_chunk_is_not_matched_by_keyword_either(manager, endpoints) -> None:
|
|
"""Off means out of retrieval, whichever way the retrieval is done."""
|
|
base, doc = await _ready_base(manager)
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
target = next(p for p in held if "first topic" in p.chunk.text)
|
|
await manager.set_chunks_enabled(doc.id, [target.chunk_id], False)
|
|
manager.stub.model = "gone"
|
|
endpoints.refuse("stub-embed", EmbeddingError("no endpoint"))
|
|
|
|
outcome = await manager.search([base.id], "alpha")
|
|
|
|
assert all("first topic" not in hit.chunk.text for hit in outcome.hits)
|
|
|
|
|
|
async def test_mixed_scoring_is_merged_by_rank_not_by_value(manager, endpoints) -> None:
|
|
"""A cosine similarity and a BM25 score are not comparable numbers. Sorting
|
|
the concatenation would let whichever scale runs hotter decide the order,
|
|
so the merge asks each list only for the part every scorer agrees on."""
|
|
first, _ = await _ready_base(manager)
|
|
# The pin moves, so the second base is built on a model that is still
|
|
# reachable while the first one's is not: one list of cosine scores, one of
|
|
# BM25 scores, in the same answer.
|
|
manager.stub.model = "second-model"
|
|
second = await manager.create_base(name="second")
|
|
doc = manager.add_document(second.id, filename="s.md", content=b"# alpha\n\nalpha alpha alpha\n")
|
|
await manager.index_document(doc.id)
|
|
endpoints.refuse("stub-embed", EmbeddingError("no endpoint"))
|
|
|
|
outcome = await manager.search([first.id, second.id], "alpha", top_k=5)
|
|
|
|
assert outcome.hits, "both kinds of answer are in the result"
|
|
assert first.id in outcome.by_keyword and second.id not in outcome.by_keyword
|
|
|
|
|
|
# -- reading a document's chunks back, and acting on them ----------
|
|
|
|
|
|
async def test_a_document_chunks_come_back_in_reading_order(manager) -> None:
|
|
"""Reading order is the chunker's numbering, and a scan of the store has no
|
|
order of its own to promise -- so the store sorts rather than the caller
|
|
hoping."""
|
|
_, doc = await _ready_base(manager)
|
|
|
|
held, total = await manager.document_chunks(doc.id)
|
|
|
|
assert [piece.chunk.chunk_index for piece in held] == list(range(total))
|
|
assert total == doc.chunk_count
|
|
assert "first topic" in held[0].chunk.text
|
|
assert all(piece.chunk_id for piece in held), "every piece is addressable"
|
|
assert all(piece.enabled and not piece.manual for piece in held)
|
|
|
|
|
|
async def test_chunks_of_a_document_that_indexed_nothing_are_empty(manager) -> None:
|
|
"""A base with no model, a document that failed, one still queued: all the
|
|
same answer, and none of them an error."""
|
|
base = await manager.create_base(name="files only", embedding=False)
|
|
doc = manager.add_document(base.id, filename="a.md", content=b"# alpha\n")
|
|
await manager.index_document(doc.id)
|
|
|
|
assert await manager.document_chunks(doc.id) == ([], 0)
|
|
assert await manager.document_chunks("no-such-document") == ([], 0)
|
|
|
|
|
|
async def test_chunks_of_one_document_do_not_include_another(manager) -> None:
|
|
base, _ = await _ready_base(manager)
|
|
second = manager.add_document(base.id, filename="other.md", content=b"# gamma\n\ngamma\n")
|
|
await manager.index_document(second.id)
|
|
|
|
held, _ = await manager.document_chunks(second.id)
|
|
|
|
assert held and all("gamma" in piece.chunk.text for piece in held)
|
|
|
|
|
|
async def test_a_page_is_a_window_on_the_document(manager) -> None:
|
|
"""Paged after the ordering, so page two follows page one through the
|
|
document rather than through whatever the scan returned."""
|
|
_, doc = await _ready_base(manager)
|
|
|
|
first, total = await manager.document_chunks(doc.id, offset=0, limit=1)
|
|
second, again = await manager.document_chunks(doc.id, offset=1, limit=1)
|
|
|
|
assert total == again == doc.chunk_count
|
|
assert [p.chunk.chunk_index for p in first] == [0]
|
|
assert [p.chunk.chunk_index for p in second] == [1]
|
|
|
|
|
|
async def test_an_id_is_the_same_piece_after_a_rebuild(manager) -> None:
|
|
"""Content-derived on purpose: a reader who turned a paragraph off last
|
|
week is still looking at the same paragraph after a reindex."""
|
|
_, doc = await _ready_base(manager)
|
|
before, _ = await manager.document_chunks(doc.id)
|
|
|
|
await manager.index_document(doc.id)
|
|
after, _ = await manager.document_chunks(doc.id)
|
|
|
|
assert [p.chunk_id for p in before] == [p.chunk_id for p in after]
|
|
|
|
|
|
async def test_a_repeated_passage_still_gets_two_ids(manager) -> None:
|
|
"""Hashing content alone would give both copies one name, and acting on one
|
|
would act on the other."""
|
|
base = await manager.create_base(name="repeats")
|
|
twice = b"# alpha\n\nsame words here\n\n# beta\n\nsame words here\n"
|
|
doc = manager.add_document(base.id, filename="twice.md", content=twice)
|
|
await manager.index_document(doc.id)
|
|
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
texts = [p.chunk.text for p in held]
|
|
|
|
assert len(texts) != len(set(texts)) or len(held) == len({p.chunk_id for p in held})
|
|
assert len({p.chunk_id for p in held}) == len(held), "no two pieces share a name"
|
|
|
|
|
|
async def test_a_disabled_chunk_is_not_retrieved(manager) -> None:
|
|
"""Off is not a ranking penalty. The filter is in the store's own search,
|
|
where every caller passes through it."""
|
|
base, doc = await _ready_base(manager)
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
target = next(p for p in held if "first topic" in p.chunk.text)
|
|
|
|
changed = await manager.set_chunks_enabled(doc.id, [target.chunk_id], False)
|
|
|
|
assert changed == 1
|
|
hits = (await manager.search([base.id], "alpha")).hits
|
|
assert all("first topic" not in hit.chunk.text for hit in hits)
|
|
back, _ = await manager.document_chunks(doc.id)
|
|
assert next(p for p in back if p.chunk_id == target.chunk_id).enabled is False
|
|
|
|
|
|
async def test_turning_a_chunk_back_on_restores_it(manager) -> None:
|
|
base, doc = await _ready_base(manager)
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
target = held[0]
|
|
await manager.set_chunks_enabled(doc.id, [target.chunk_id], False)
|
|
|
|
await manager.set_chunks_enabled(doc.id, [target.chunk_id], True)
|
|
|
|
hits = (await manager.search([base.id], "alpha")).hits
|
|
assert any(hit.chunk.text == target.chunk.text for hit in hits)
|
|
|
|
|
|
async def test_listing_can_ask_for_one_state(manager) -> None:
|
|
_, doc = await _ready_base(manager)
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
await manager.set_chunks_enabled(doc.id, [held[0].chunk_id], False)
|
|
|
|
off, off_total = await manager.document_chunks(doc.id, enabled=False)
|
|
on, on_total = await manager.document_chunks(doc.id, enabled=True)
|
|
|
|
assert off_total == 1 and [p.chunk_id for p in off] == [held[0].chunk_id]
|
|
assert on_total == len(held) - 1
|
|
|
|
|
|
async def test_deleting_a_chunk_leaves_the_count_agreeing_with_the_index(manager) -> None:
|
|
"""The row on the page reads that number to tell an indexed document from
|
|
an empty one."""
|
|
_, doc = await _ready_base(manager)
|
|
held, total = await manager.document_chunks(doc.id)
|
|
|
|
left = await manager.delete_chunks(doc.id, [held[0].chunk_id])
|
|
|
|
assert left == total - 1
|
|
assert manager.get_document(doc.id).chunk_count == total - 1
|
|
remaining, _ = await manager.document_chunks(doc.id)
|
|
assert held[0].chunk_id not in {p.chunk_id for p in remaining}
|
|
|
|
|
|
async def test_a_written_chunk_is_appended_and_findable(manager) -> None:
|
|
"""Appended rather than inserted: it was not cut from anywhere in the
|
|
document, and putting it between two pieces that were would claim a place
|
|
in the text it does not have."""
|
|
base, doc = await _ready_base(manager)
|
|
_, before = await manager.document_chunks(doc.id)
|
|
|
|
written = await manager.add_chunk(doc.id, " epsilon epsilon epsilon ")
|
|
|
|
assert written.manual is True and written.enabled is True
|
|
assert written.chunk.text == "epsilon epsilon epsilon"
|
|
held, total = await manager.document_chunks(doc.id)
|
|
assert total == before + 1
|
|
assert held[-1].chunk_id == written.chunk_id, "at the end of the reading order"
|
|
assert manager.get_document(doc.id).chunk_count == total
|
|
hits = (await manager.search([base.id], "epsilon")).hits
|
|
assert any("epsilon" in hit.chunk.text for hit in hits)
|
|
|
|
|
|
async def test_a_written_chunk_does_not_survive_a_reindex(manager) -> None:
|
|
"""A reindex is the document being cut again, and a piece nobody cut has
|
|
nothing to be attached to."""
|
|
_, doc = await _ready_base(manager)
|
|
written = await manager.add_chunk(doc.id, "written by hand")
|
|
|
|
await manager.index_document(doc.id)
|
|
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
assert written.chunk_id not in {p.chunk_id for p in held}
|
|
|
|
|
|
async def test_an_empty_written_chunk_is_refused(manager) -> None:
|
|
_, doc = await _ready_base(manager)
|
|
|
|
with pytest.raises(KnowledgeError):
|
|
await manager.add_chunk(doc.id, " \n ")
|
|
|
|
|
|
# ── deletion ──────────────────────────────────────────────────────
|
|
|
|
|
|
async def test_deleting_a_document_removes_its_vectors_and_its_bytes(manager) -> None:
|
|
base, doc = await _ready_base(manager)
|
|
|
|
assert await manager.delete_document(doc.id) is True
|
|
assert manager.read_document(doc.id) is None
|
|
assert (await manager.search([base.id], "alpha")).hits == []
|
|
|
|
|
|
async def test_deleting_a_base_takes_the_collection_with_it(manager) -> None:
|
|
"""A records-only delete leaves vectors under an id nothing lists any
|
|
more -- an index no one can name or reclaim."""
|
|
base, doc = await _ready_base(manager)
|
|
|
|
assert await manager.delete_base(base.id) is True
|
|
assert await manager._store.has_collection(base.id) is False
|
|
assert manager.read_document(doc.id) is None
|
|
assert manager.get_base(base.id) is None
|
|
|
|
|
|
async def test_deleting_a_missing_base_says_so(manager) -> None:
|
|
assert await manager.delete_base("nope") is False
|
|
|
|
|
|
# ── configuration ─────────────────────────────────────────────────
|
|
|
|
|
|
async def test_a_base_cannot_be_created_without_an_embedding_endpoint(tmp_path, monkeypatch) -> None:
|
|
monkeypatch.setattr("raven.knowledge._manager.load_embedding_config", lambda: None)
|
|
mgr = KnowledgeManager(tmp_path / "knowledge")
|
|
|
|
assert mgr.embedding_available() is False
|
|
with pytest.raises(KnowledgeError, match="no embedding endpoint"):
|
|
await mgr.create_base(name="handbook")
|
|
|
|
|
|
async def test_uploading_to_a_missing_base_is_refused(manager) -> None:
|
|
with pytest.raises(KnowledgeError, match="no knowledge base"):
|
|
manager.add_document("nope", filename="a.md", content=MARKDOWN)
|
|
|
|
|
|
async def test_the_collection_is_named_for_the_id_not_the_name(manager) -> None:
|
|
"""A rename is an edit; a collection that followed the display name would
|
|
strand every row indexed under the old one."""
|
|
base, _ = await _ready_base(manager)
|
|
manager.rename_base(base.id, name="renamed")
|
|
|
|
assert await manager._store.has_collection(base.id) is True
|
|
assert len((await manager.search([base.id], "alpha", top_k=1)).hits) == 1
|
|
|
|
|
|
async def test_search_merges_and_reranks_across_bases(manager) -> None:
|
|
first = await manager.create_base(name="first")
|
|
second = await manager.create_base(name="second")
|
|
manager.add_document(first.id, filename="a.txt", content=b"alpha alpha alpha")
|
|
manager.add_document(second.id, filename="b.txt", content=b"alpha")
|
|
await manager.index_pending()
|
|
|
|
hits = (await manager.search([first.id, second.id], "alpha alpha alpha", top_k=2)).hits
|
|
assert len(hits) == 2
|
|
assert hits[0].score >= hits[1].score
|
|
|
|
|
|
async def test_the_query_is_embedded_once_for_all_bases(manager) -> None:
|
|
first = await manager.create_base(name="first")
|
|
second = await manager.create_base(name="second")
|
|
manager.add_document(first.id, filename="a.txt", content=b"alpha")
|
|
manager.add_document(second.id, filename="b.txt", content=b"alpha")
|
|
await manager.index_pending()
|
|
|
|
before = len(manager.stub.calls)
|
|
await manager.search([first.id, second.id], "alpha")
|
|
assert len(manager.stub.calls) == before + 1
|
|
|
|
|
|
async def test_an_empty_query_searches_nothing(manager) -> None:
|
|
base, _ = await _ready_base(manager)
|
|
assert (await manager.search([base.id], " ")).hits == []
|
|
|
|
|
|
class TestABaseWithNoEmbeddingModel:
|
|
"""A base that keeps its documents and is never searched by vector.
|
|
|
|
The case is real -- somewhere to put files the agent reads whole, or that a
|
|
person opens from the page -- and it has to reach no endpoint at all. It is
|
|
also not revisable: a collection's width is fixed when it is made, so a base
|
|
created without one is rebuilt rather than switched.
|
|
"""
|
|
|
|
async def test_it_is_created_without_reaching_the_endpoint(self, manager) -> None:
|
|
base = await manager.create_base(name="files", embedding=False)
|
|
|
|
assert base.embedding_model == ""
|
|
assert base.dimensions == 0
|
|
# Nothing was embedded, so nothing was asked of the endpoint -- the
|
|
# width probe is the call this avoids.
|
|
assert manager.stub.calls == []
|
|
|
|
async def test_a_document_is_stored_and_not_indexed(self, manager) -> None:
|
|
"""Ready, because the file is in the base and can be opened. Failed
|
|
would send a reader looking for a fault; pending would promise an
|
|
indexer that is never coming."""
|
|
base = await manager.create_base(name="files", embedding=False)
|
|
doc = manager.add_document(base.id, filename="notes.md", content=b"# hi\n")
|
|
|
|
indexed = await manager.index_document(doc.id)
|
|
|
|
assert indexed.status == "ready"
|
|
assert indexed.chunk_count == 0
|
|
assert manager.stub.calls == []
|
|
# And the bytes are still there to open.
|
|
assert manager.read_document(doc.id) == b"# hi\n"
|
|
|
|
async def test_it_is_skipped_rather_than_refused_by_a_search(self, manager) -> None:
|
|
"""Asking a mixed set is ordinary, and a base with no vectors is not an
|
|
error in the others."""
|
|
plain = await manager.create_base(name="files", embedding=False)
|
|
vectored = await manager.create_base(name="handbook")
|
|
doc = manager.add_document(vectored.id, filename="handbook.md", content=b"onboarding is here\n")
|
|
await manager.index_document(doc.id)
|
|
|
|
hits = (await manager.search([plain.id, vectored.id], "onboarding")).hits
|
|
|
|
assert [h.chunk.source for h in hits] == ["handbook.md"]
|
|
|
|
async def test_searching_only_such_a_base_answers_nothing(self, manager) -> None:
|
|
plain = await manager.create_base(name="files", embedding=False)
|
|
|
|
assert (await manager.search([plain.id], "anything")).hits == []
|
|
|
|
async def test_the_question_has_one_answer(self, manager) -> None:
|
|
"""Three readers skip these bases; three spellings of "is the model
|
|
empty" is how one of them ends up not skipping."""
|
|
plain = await manager.create_base(name="files", embedding=False)
|
|
vectored = await manager.create_base(name="handbook")
|
|
|
|
assert manager.embeds(plain) is False
|
|
assert manager.embeds(vectored) is True
|
|
|
|
|
|
class TestTwoBasesCannotShareAName:
|
|
"""The rail shows a base's name and nothing else.
|
|
|
|
Two rows reading "t3" leave a reader picking between them and finding out
|
|
which was which by opening both -- and a delete then asks them to be sure
|
|
about which of two identical rows they meant.
|
|
"""
|
|
|
|
async def test_a_second_base_cannot_take_the_name(self, manager) -> None:
|
|
await manager.create_base(name="t3")
|
|
|
|
with pytest.raises(DuplicateBaseNameError):
|
|
await manager.create_base(name="t3")
|
|
|
|
assert [b.name for b in manager.list_bases()] == ["t3"]
|
|
|
|
@pytest.mark.parametrize("second", ["T3", "t3 ", " T3"])
|
|
async def test_case_and_spacing_do_not_make_it_a_different_name(self, manager, second: str) -> None:
|
|
"""Two bases called "t3" and "T3 " are the same problem as two called
|
|
"t3": the reader cannot tell those apart either."""
|
|
await manager.create_base(name="t3")
|
|
|
|
with pytest.raises(DuplicateBaseNameError):
|
|
await manager.create_base(name=second)
|
|
|
|
async def test_what_was_typed_is_what_is_stored(self, manager) -> None:
|
|
"""Compared casefolded, kept as written: the rule is about telling
|
|
bases apart, not about how a name may be spelled."""
|
|
base = await manager.create_base(name="Handbook")
|
|
|
|
assert base.name == "Handbook"
|
|
|
|
async def test_a_rename_cannot_take_a_name_either(self, manager) -> None:
|
|
"""Or renaming is simply the way around the rule."""
|
|
await manager.create_base(name="t1")
|
|
second = await manager.create_base(name="t2")
|
|
|
|
with pytest.raises(DuplicateBaseNameError):
|
|
manager.rename_base(second.id, name="t1")
|
|
|
|
assert manager.get_base(second.id).name == "t2"
|
|
|
|
async def test_a_base_may_keep_its_own_name(self, manager) -> None:
|
|
"""Without this, saving a rename that touched only the description
|
|
would fail against the base itself."""
|
|
base = await manager.create_base(name="t1")
|
|
|
|
renamed = manager.rename_base(base.id, name="t1", description="notes")
|
|
|
|
assert renamed.name == "t1"
|
|
assert renamed.description == "notes"
|
|
|
|
async def test_a_freed_name_can_be_taken_again(self, manager) -> None:
|
|
first = await manager.create_base(name="t3")
|
|
await manager.delete_base(first.id)
|
|
|
|
again = await manager.create_base(name="t3")
|
|
|
|
assert again.name == "t3"
|
|
|
|
|
|
# ── rewriting a document in place ─────────────────────────────────
|
|
|
|
|
|
async def test_rewriting_a_document_replaces_its_text_and_requeues_it(manager) -> None:
|
|
base, doc = await _ready_base(manager)
|
|
|
|
rewritten = await manager.replace_document(
|
|
doc.id,
|
|
filename="handbook v2.md",
|
|
content=b"# Handbook\n\n## Gamma section\n\ngamma gamma gamma on the third topic.\n",
|
|
)
|
|
|
|
assert (rewritten.source, rewritten.status, rewritten.chunk_count) == ("handbook v2.md", "pending", 0)
|
|
assert manager.read_document(doc.id) == (
|
|
b"# Handbook\n\n## Gamma section\n\ngamma gamma gamma on the third topic.\n"
|
|
)
|
|
indexed = await manager.index_document(doc.id)
|
|
assert indexed.status == "ready"
|
|
hits = (await manager.search([base.id], "gamma", top_k=5)).hits
|
|
assert "third topic" in hits[0].chunk.text
|
|
|
|
|
|
async def test_the_old_text_stops_being_searchable_the_moment_it_is_rewritten(manager) -> None:
|
|
"""Before the reindex, not after: chunks are what a search answers with,
|
|
and answering from a note the reader has already rewritten is worse than
|
|
answering with nothing. If the reindex then fails, they stay gone."""
|
|
base, doc = await _ready_base(manager)
|
|
|
|
await manager.replace_document(doc.id, filename="handbook.md", content=b"# Handbook\n\nquite different now.\n")
|
|
|
|
assert (await manager.search([base.id], "alpha")).hits == []
|
|
|
|
|
|
async def test_rewriting_a_document_that_is_gone_answers_nothing(manager) -> None:
|
|
assert await manager.replace_document("nope", filename="x.md", content=b"x") is None
|
|
|
|
|
|
async def test_a_document_remembers_which_kind_of_source_it_came_from(manager) -> None:
|
|
base = await manager.create_base(name="handbook")
|
|
|
|
note = manager.add_document(base.id, filename="plan.md", content=b"# Plan", origin="note")
|
|
page = manager.add_document(
|
|
base.id, filename="docs.md", content=b"# Docs", origin="url", origin_ref="https://example.com/docs"
|
|
)
|
|
uploaded = manager.add_document(base.id, filename="handbook.md", content=MARKDOWN)
|
|
|
|
assert (note.origin, note.origin_ref) == ("note", "")
|
|
assert (page.origin, page.origin_ref) == ("url", "https://example.com/docs")
|
|
# The default, so a registry written before the field existed still loads,
|
|
# and every document in one is what the default says it is.
|
|
assert (uploaded.origin, uploaded.origin_ref) == ("file", "")
|
|
|
|
|
|
async def test_the_manager_builds_the_client_the_endpoint_calls_for(tmp_path) -> None:
|
|
"""Which vendor is being spoken to is a fact about the base URL, so the
|
|
manager must not hardcode the plain client past it."""
|
|
from raven.knowledge._embedding import EmbeddingClient, SiliconFlowEmbeddingClient
|
|
|
|
def built(base_url: str, model: str):
|
|
mgr = KnowledgeManager(
|
|
tmp_path / base_url.replace("/", "_"),
|
|
store=_NoStore(),
|
|
embedding=EmbeddingConfig(model=model, base_url=base_url, api_key="k"),
|
|
)
|
|
return mgr._client()
|
|
|
|
assert isinstance(built("https://api.siliconflow.cn/v1", "BAAI/bge-large-zh-v1.5"), SiliconFlowEmbeddingClient)
|
|
assert type(built("https://api.openai.com/v1", "text-embedding-3-small")) is EmbeddingClient
|
|
|
|
|
|
# -- searching a document written before chunk ids existed ---------
|
|
|
|
|
|
async def test_searching_a_document_from_before_ids_answers_distinct_pieces(manager) -> None:
|
|
"""Rows written before chunk ids existed all carry the empty one.
|
|
|
|
Keyed by id, those rows collapsed onto a single entry and every one of
|
|
their hits was answered with whichever of them happened to be last -- so a
|
|
search of such a document came back as one piece repeated, and the branch
|
|
written for them could never be reached.
|
|
"""
|
|
base = await manager.create_base(name="notes")
|
|
# Small enough to cut: the point of the case is several rows sharing the
|
|
# empty id, and the default size takes a short file whole.
|
|
manager.configure_base(base.id, chunk_size=16, chunk_overlap=0, separator="\n\n")
|
|
body = b"\n\n".join(b"alpha beta gamma delta epsilon %d" % n for n in range(6))
|
|
doc = manager.add_document(base.id, filename="handbook.md", content=body)
|
|
await manager.index_document(doc.id)
|
|
|
|
held, _ = await manager.document_chunks(doc.id)
|
|
assert len(held) >= 2, "the document was cut into pieces"
|
|
|
|
# Put the base back the way a build before ids wrote it: every row with
|
|
# the empty id, which is what `list_chunks` then reads back.
|
|
store = manager._store
|
|
rows = await (await store._writable(base.id)).query().to_list()
|
|
for row in rows:
|
|
row["chunk_id"] = ""
|
|
merge = (await store._writable(base.id)).merge_insert(["chunk_index", "document_id"])
|
|
await merge.when_matched_update_all().execute(rows)
|
|
|
|
found = await manager.search_document(doc.id, "alpha", limit=5)
|
|
|
|
assert len(found) >= 2, "every hit came back"
|
|
indices = [piece.chunk.chunk_index for piece in found]
|
|
assert len(set(indices)) == len(indices), f"one piece repeated: {indices}"
|