Raven-1/tests/test_knowledge_sources.py
Kevin Hu 7beffb9d0f
feat(knowledge): four data sources for a knowledge base, recall test, and per-base settings (#426)
## Summary

<img width="2518" height="1798" alt="image"
src="https://github.com/user-attachments/assets/9bb2dbb6-fe29-42ff-8dba-2e035557b254"
/>


Turns the knowledge page into something a reader can actually run a
knowledge base from: put material into it four ways, look at what is in
it, test what it answers with, and configure it.

**Data sources.** The panel had one button that uploaded one file. It
now offers file, folder, note and URL behind one menu, plus a drop
target over the whole panel. A picked folder is walked by the browser; a
dropped one is walked here, because a drop hands over a directory entry
rather than files. Either way the files are filtered to what this build
can index, capped, and uploaded one at a time: each carries its bytes
base64 in a single websocket frame under a 25 MB ceiling, so N at once
is N of those in memory. One file failing no longer takes the rest of
the folder with it.

A note and a fetched page are stored as the markdown they were captured
as, in the same blob store as every upload, so nothing downstream learns
a second shape. Reading a page follows Cherry Studio's own capture: Jina
Reader, asked for JSON so the page title comes back as a field rather
than as a preamble to parse out of the text, and through
`validate_url_target` first, because the URL arrives from a browser and
without that check the gateway is a request forger.

**Recall Test.** Asks a base a question and shows what comes back: the
chunk, where in its document it sat, its score beside its rank, and what
the search cost. An explicit button rather than search-as-you-type,
because every keystroke used to spend a billed embedding call. Past
questions are kept in localStorage and nowhere else.

**Settings.** Each base carries its own Top K and chunking, with a
sentence on every field saying what moving it costs. The embedding model
is shown and not offered: the collection is sized to its vector width at
creation, so changing it is a rebuild of every vector rather than a
setting, which is what the stale-base check exists to detect. Letting a
reader change it is a separate piece of work.

**Two defects found while measuring the above.**

SiliconFlow refuses an entire request when one input is over the model's
token limit, naming neither which input nor what was wrong with it. A
document of eighteen chunks where seven were long indexed none of the
other eleven, and four documents in a real base were failing this way.
The cause is the chunker's token estimate -- utf-8 bytes over four, an
English-prose rule that reads a CJK character as three quarters of a
token when it is one, and a URL as a quarter of what it costs.
Binary-searching the longest accepted prefix of real documentation pages
against the endpoint put the true figure at 2.1 to 2.3 characters per
token. A vendor client now cuts each input to fit before sending; the
four documents embed whole.

Search reported one number for its cost. It now reports two, because
they answer different questions: embedding is a round trip to whatever
endpoint is configured and runs to hundreds of milliseconds, while the
store query is the index doing its job. A surface reporting one figure
as the search time wants the second, or it is describing the provider.

**Deliberately not wired yet.** File processing lists the processors it
will offer with every row disabled and saying why it is unavailable;
smart chunking, separator, chunk size and overlap are stored on the base
but not yet read by the chunker. Both are placeholders with their state
stated rather than controls that would be silently ignored.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run on this branch rebased onto `main` at `4f5d7dac`:

- `uv run pytest -q` -- 22151 passed, 45 skipped
- `npx vitest run` in `ui-web` -- 1738 passed (103 files), of which 92
cover this page
- `npx vitest run` in `ui-tui` -- 2072 passed
- `npx tsc --noEmit` in `ui-web` and `ui-tui` -- clean
- `uv run ruff check raven/ tests/` and `ruff format --check` -- clean
- `node scripts/check-page.mjs`, `check-css.mjs`,
`check-class-namespace.mjs`, `count-shared-globals.mjs` -- all OK
- `make check-large-files` -- OK
- The SiliconFlow client was measured against the live endpoint, not
mocked: batch size is not the limit (64 short inputs pass), 500 CJK
characters pass and 512 do not, and all four previously failing
documents in a real base now embed whole.
- Every panel was checked in the browser in both light and dark themes.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

User-visible behaviour changes, all additive:

- A search that names no `top_k` now answers with the base's own setting
instead of a hardcoded 5. New bases default to 6.
- Documents carry an `origin` and bases carry six settings. Every field
is defaulted, so a registry written before them loads with the behaviour
it already had; an older gateway answering without them is read as those
same defaults. A field from a build *after* this one is ignored rather
than losing the row, which is the same protection in the other
direction.
- The SiliconFlow client truncates an over-long input rather than
failing the request. A cut chunk's vector speaks for its beginning; the
chunk is still stored and shown in full when retrieved. The proper fix
is chunking to the model's real limit, which this does not do.
- Four brand-logo SVGs are added under `ui-web/src/assets/providers/`
(16 KB total) and spared the dark-mode inversion that monochrome vendor
marks get.
- Previewing an office file retains a suffixed copy of its bytes in the
render cache, because the converter picks its input filter from the
extension and a blob is stored under a bare id. Deleting the document or
its base now removes that copy, and the cache sweep reaches it as a
backstop for one whose owner never got the chance.

Rollback is a `git revert` of the squash commit; nothing migrates on
disk.

A registry written by this branch stays readable by the previous build,
and that took work: its loader hands every stored key to a dataclass
constructor and drops the row when one is a keyword that constructor
does not take, so a new field written beside the record would make every
base and document vanish from a rolled-back binary, with the records
still on disk and nothing listing them. The settings and the document
origins are therefore written under two sibling keys of `bases` and
`documents`, which a loader that does not know them does not read; a
record that uses none of them is written exactly as before, down to its
keys. What a rollback loses is the settings, which that build has no use
for. What it keeps is every base and every document. A test pins the
stored shape against the field set the shipped loader accepts, rather
than against our own reader.

Security: the URL source takes an arbitrary address from a browser and
fetches it server-side, so it goes through the same
`validate_url_target` guard the web tools use, and there is a test
asserting a refused target never reaches the network.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: admin <admin@SH-HuKai.local>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-16 03:02:44 -07:00

163 lines
6.3 KiB
Python

"""Data sources other than an uploaded file: what a note and a page are called,
and how a page is read."""
from __future__ import annotations
import httpx
import pytest
from raven.knowledge import _sources
def _reader(monkeypatch, handler) -> dict[str, list[httpx.Request]]:
"""Route the Jina Reader fetch through a MockTransport."""
seen: dict[str, list[httpx.Request]] = {"requests": []}
def recording(request: httpx.Request) -> httpx.Response:
seen["requests"].append(request)
return handler(request)
real = httpx.AsyncClient
def factory(*args, **kwargs):
kwargs.setdefault("transport", httpx.MockTransport(recording))
return real(*args, **kwargs)
monkeypatch.setattr(_sources.httpx, "AsyncClient", factory)
monkeypatch.setattr(_sources, "validate_url_target", lambda url: (True, ""))
return seen
def test_a_page_is_named_after_its_title_before_anything_else() -> None:
assert _sources.page_filename("# Heading\n\nbody", "https://ex.com/a/b", "Real Title") == "Real Title.md"
def test_a_titleless_page_falls_back_down_the_ladder() -> None:
"""Each rung is there because the one above it is empty often enough to
matter: a page served with no title, one with no headings, a URL that is
only a host."""
assert _sources.page_filename("# Heading\n\nbody", "https://ex.com/a/b") == "Heading.md"
assert _sources.page_filename("plain body", "https://ex.com/docs/guide") == "plain body.md"
assert _sources.page_filename("", "https://ex.com/docs/guide") == "ex.com-guide.md"
assert _sources.page_filename("", "https://ex.com") == "ex.com.md"
def test_a_title_cannot_walk_out_of_the_blob_store() -> None:
"""The title comes off the open web, and it is about to name a file."""
assert "/" not in _sources.page_filename("", "", "../../etc/passwd")
assert _sources.safe_stem("../../etc/passwd") == "etc passwd"
def test_a_long_title_is_cut_to_something_that_fits_a_table_cell() -> None:
name = _sources.page_filename("", "", "x" * 200)
assert len(name) == _sources.TITLE_MAX + len(".md")
def test_a_note_is_named_from_its_title_or_its_first_line() -> None:
assert _sources.note_filename("Release plan", "body") == "Release plan.md"
assert _sources.note_filename("", "Standup notes\n\nshipped it") == "Standup notes.md"
assert _sources.note_filename("", " ") == "note.md"
@pytest.mark.asyncio
async def test_reading_a_page_asks_the_reader_for_json_not_a_preamble(monkeypatch) -> None:
"""The title has to come back as a field. Asked for as text it arrives as a
header block above the body, which then has to be parsed back off, and ends
up in the chunks when it is not."""
seen = _reader(
monkeypatch,
lambda request: httpx.Response(200, json={"data": {"title": "Raven", "content": "# Raven\n\nhello"}}),
)
page = await _sources.fetch_page("https://example.com/docs")
assert (page.title, page.markdown) == ("Raven", "# Raven\n\nhello")
request = seen["requests"][0]
assert str(request.url) == "https://r.jina.ai/https://example.com/docs"
assert request.headers["Accept"] == "application/json"
# No key: Reader answers anonymously at a lower rate limit, and a
# deployment that has not configured one still gets this feature.
assert "Authorization" not in request.headers
@pytest.mark.asyncio
async def test_a_configured_key_is_sent_and_an_absent_one_is_not(monkeypatch) -> None:
seen = _reader(monkeypatch, lambda request: httpx.Response(200, json={"data": {"content": "body"}}))
await _sources.fetch_page("https://example.com", api_key="k-1")
assert seen["requests"][0].headers["Authorization"] == "Bearer k-1"
@pytest.mark.asyncio
async def test_a_refused_url_never_reaches_the_network(monkeypatch) -> None:
"""Without this the endpoint is a request forger: it takes a URL from a
browser and answers with the body of whatever the gateway can reach."""
monkeypatch.setattr(_sources, "validate_url_target", lambda url: (False, "blocked host"))
def _explode(request: httpx.Request) -> httpx.Response:
pytest.fail("must not fetch a refused target")
real = httpx.AsyncClient
monkeypatch.setattr(
_sources.httpx,
"AsyncClient",
lambda *a, **k: real(*a, **{**k, "transport": httpx.MockTransport(_explode)}),
)
with pytest.raises(_sources.SourceFetchError) as caught:
await _sources.fetch_page("http://169.254.169.254/latest/meta-data/")
assert "blocked host" in str(caught.value)
@pytest.mark.asyncio
async def test_an_http_error_names_the_status_rather_than_the_stack(monkeypatch) -> None:
_reader(monkeypatch, lambda request: httpx.Response(404))
with pytest.raises(_sources.SourceFetchError) as caught:
await _sources.fetch_page("https://example.com/gone")
assert "404" in str(caught.value)
@pytest.mark.asyncio
async def test_a_page_with_no_text_is_an_error_not_an_empty_document(monkeypatch) -> None:
"""An empty document indexes to nothing and sits in the list looking as if
it worked."""
_reader(monkeypatch, lambda request: httpx.Response(200, json={"data": {"title": "Nothing", "content": " "}}))
with pytest.raises(_sources.SourceFetchError) as caught:
await _sources.fetch_page("https://example.com/empty")
assert "no readable text" in str(caught.value)
@pytest.mark.asyncio
async def test_an_answer_that_is_not_a_page_is_reported_as_one(monkeypatch) -> None:
_reader(monkeypatch, lambda request: httpx.Response(200, content=b"<html>not json</html>"))
with pytest.raises(_sources.SourceFetchError):
await _sources.fetch_page("https://example.com/html")
def test_a_page_with_nothing_to_name_it_by_still_gets_a_name() -> None:
"""An untitled page at a bare address is rare and not an error; a document
with no filename at all would be."""
assert _sources.page_filename("", "", "") == "page.md"
assert _sources.page_filename("", "http://[", "") == "page.md"
@pytest.mark.asyncio
async def test_a_reader_that_cannot_be_reached_says_that(monkeypatch) -> None:
def _refuse(request: httpx.Request) -> httpx.Response:
raise httpx.ConnectError("connection refused", request=request)
_reader(monkeypatch, _refuse)
with pytest.raises(_sources.SourceFetchError) as caught:
await _sources.fetch_page("https://example.com")
assert "could not reach" in str(caught.value)