## Summary <img width="2518" height="1798" alt="image" src="https://github.com/user-attachments/assets/9bb2dbb6-fe29-42ff-8dba-2e035557b254" /> Turns the knowledge page into something a reader can actually run a knowledge base from: put material into it four ways, look at what is in it, test what it answers with, and configure it. **Data sources.** The panel had one button that uploaded one file. It now offers file, folder, note and URL behind one menu, plus a drop target over the whole panel. A picked folder is walked by the browser; a dropped one is walked here, because a drop hands over a directory entry rather than files. Either way the files are filtered to what this build can index, capped, and uploaded one at a time: each carries its bytes base64 in a single websocket frame under a 25 MB ceiling, so N at once is N of those in memory. One file failing no longer takes the rest of the folder with it. A note and a fetched page are stored as the markdown they were captured as, in the same blob store as every upload, so nothing downstream learns a second shape. Reading a page follows Cherry Studio's own capture: Jina Reader, asked for JSON so the page title comes back as a field rather than as a preamble to parse out of the text, and through `validate_url_target` first, because the URL arrives from a browser and without that check the gateway is a request forger. **Recall Test.** Asks a base a question and shows what comes back: the chunk, where in its document it sat, its score beside its rank, and what the search cost. An explicit button rather than search-as-you-type, because every keystroke used to spend a billed embedding call. Past questions are kept in localStorage and nowhere else. **Settings.** Each base carries its own Top K and chunking, with a sentence on every field saying what moving it costs. The embedding model is shown and not offered: the collection is sized to its vector width at creation, so changing it is a rebuild of every vector rather than a setting, which is what the stale-base check exists to detect. Letting a reader change it is a separate piece of work. **Two defects found while measuring the above.** SiliconFlow refuses an entire request when one input is over the model's token limit, naming neither which input nor what was wrong with it. A document of eighteen chunks where seven were long indexed none of the other eleven, and four documents in a real base were failing this way. The cause is the chunker's token estimate -- utf-8 bytes over four, an English-prose rule that reads a CJK character as three quarters of a token when it is one, and a URL as a quarter of what it costs. Binary-searching the longest accepted prefix of real documentation pages against the endpoint put the true figure at 2.1 to 2.3 characters per token. A vendor client now cuts each input to fit before sending; the four documents embed whole. Search reported one number for its cost. It now reports two, because they answer different questions: embedding is a round trip to whatever endpoint is configured and runs to hundreds of milliseconds, while the store query is the index doing its job. A surface reporting one figure as the search time wants the second, or it is describing the provider. **Deliberately not wired yet.** File processing lists the processors it will offer with every row disabled and saying why it is unavailable; smart chunking, separator, chunk size and overlap are stored on the base but not yet read by the chunker. Both are placeholders with their state stated rather than controls that would be silently ignored. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Run on this branch rebased onto `main` at `4f5d7dac`: - `uv run pytest -q` -- 22151 passed, 45 skipped - `npx vitest run` in `ui-web` -- 1738 passed (103 files), of which 92 cover this page - `npx vitest run` in `ui-tui` -- 2072 passed - `npx tsc --noEmit` in `ui-web` and `ui-tui` -- clean - `uv run ruff check raven/ tests/` and `ruff format --check` -- clean - `node scripts/check-page.mjs`, `check-css.mjs`, `check-class-namespace.mjs`, `count-shared-globals.mjs` -- all OK - `make check-large-files` -- OK - The SiliconFlow client was measured against the live endpoint, not mocked: batch size is not the limit (64 short inputs pass), 500 CJK characters pass and 512 do not, and all four previously failing documents in a real base now embed whole. - Every panel was checked in the browser in both light and dark themes. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk User-visible behaviour changes, all additive: - A search that names no `top_k` now answers with the base's own setting instead of a hardcoded 5. New bases default to 6. - Documents carry an `origin` and bases carry six settings. Every field is defaulted, so a registry written before them loads with the behaviour it already had; an older gateway answering without them is read as those same defaults. A field from a build *after* this one is ignored rather than losing the row, which is the same protection in the other direction. - The SiliconFlow client truncates an over-long input rather than failing the request. A cut chunk's vector speaks for its beginning; the chunk is still stored and shown in full when retrieved. The proper fix is chunking to the model's real limit, which this does not do. - Four brand-logo SVGs are added under `ui-web/src/assets/providers/` (16 KB total) and spared the dark-mode inversion that monochrome vendor marks get. - Previewing an office file retains a suffixed copy of its bytes in the render cache, because the converter picks its input filter from the extension and a blob is stored under a bare id. Deleting the document or its base now removes that copy, and the cache sweep reaches it as a backstop for one whose owner never got the chance. Rollback is a `git revert` of the squash commit; nothing migrates on disk. A registry written by this branch stays readable by the previous build, and that took work: its loader hands every stored key to a dataclass constructor and drops the row when one is a keyword that constructor does not take, so a new field written beside the record would make every base and document vanish from a rolled-back binary, with the records still on disk and nothing listing them. The settings and the document origins are therefore written under two sibling keys of `bases` and `documents`, which a loader that does not know them does not read; a record that uses none of them is written exactly as before, down to its keys. What a rollback loses is the settings, which that build has no use for. What it keeps is every base and every document. A test pins the stored shape against the field set the shipped loader accepts, rather than against our own reader. Security: the URL source takes an arbitrary address from a browser and fetches it server-side, so it goes through the same `validate_url_target` guard the web tools use, and there is a test asserting a refused target never reaches the network. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: admin <admin@SH-HuKai.local> Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
163 lines
6.3 KiB
Python
163 lines
6.3 KiB
Python
"""Data sources other than an uploaded file: what a note and a page are called,
|
|
and how a page is read."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
from raven.knowledge import _sources
|
|
|
|
|
|
def _reader(monkeypatch, handler) -> dict[str, list[httpx.Request]]:
|
|
"""Route the Jina Reader fetch through a MockTransport."""
|
|
seen: dict[str, list[httpx.Request]] = {"requests": []}
|
|
|
|
def recording(request: httpx.Request) -> httpx.Response:
|
|
seen["requests"].append(request)
|
|
return handler(request)
|
|
|
|
real = httpx.AsyncClient
|
|
|
|
def factory(*args, **kwargs):
|
|
kwargs.setdefault("transport", httpx.MockTransport(recording))
|
|
return real(*args, **kwargs)
|
|
|
|
monkeypatch.setattr(_sources.httpx, "AsyncClient", factory)
|
|
monkeypatch.setattr(_sources, "validate_url_target", lambda url: (True, ""))
|
|
return seen
|
|
|
|
|
|
def test_a_page_is_named_after_its_title_before_anything_else() -> None:
|
|
assert _sources.page_filename("# Heading\n\nbody", "https://ex.com/a/b", "Real Title") == "Real Title.md"
|
|
|
|
|
|
def test_a_titleless_page_falls_back_down_the_ladder() -> None:
|
|
"""Each rung is there because the one above it is empty often enough to
|
|
matter: a page served with no title, one with no headings, a URL that is
|
|
only a host."""
|
|
assert _sources.page_filename("# Heading\n\nbody", "https://ex.com/a/b") == "Heading.md"
|
|
assert _sources.page_filename("plain body", "https://ex.com/docs/guide") == "plain body.md"
|
|
assert _sources.page_filename("", "https://ex.com/docs/guide") == "ex.com-guide.md"
|
|
assert _sources.page_filename("", "https://ex.com") == "ex.com.md"
|
|
|
|
|
|
def test_a_title_cannot_walk_out_of_the_blob_store() -> None:
|
|
"""The title comes off the open web, and it is about to name a file."""
|
|
assert "/" not in _sources.page_filename("", "", "../../etc/passwd")
|
|
assert _sources.safe_stem("../../etc/passwd") == "etc passwd"
|
|
|
|
|
|
def test_a_long_title_is_cut_to_something_that_fits_a_table_cell() -> None:
|
|
name = _sources.page_filename("", "", "x" * 200)
|
|
|
|
assert len(name) == _sources.TITLE_MAX + len(".md")
|
|
|
|
|
|
def test_a_note_is_named_from_its_title_or_its_first_line() -> None:
|
|
assert _sources.note_filename("Release plan", "body") == "Release plan.md"
|
|
assert _sources.note_filename("", "Standup notes\n\nshipped it") == "Standup notes.md"
|
|
assert _sources.note_filename("", " ") == "note.md"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_reading_a_page_asks_the_reader_for_json_not_a_preamble(monkeypatch) -> None:
|
|
"""The title has to come back as a field. Asked for as text it arrives as a
|
|
header block above the body, which then has to be parsed back off, and ends
|
|
up in the chunks when it is not."""
|
|
seen = _reader(
|
|
monkeypatch,
|
|
lambda request: httpx.Response(200, json={"data": {"title": "Raven", "content": "# Raven\n\nhello"}}),
|
|
)
|
|
|
|
page = await _sources.fetch_page("https://example.com/docs")
|
|
|
|
assert (page.title, page.markdown) == ("Raven", "# Raven\n\nhello")
|
|
request = seen["requests"][0]
|
|
assert str(request.url) == "https://r.jina.ai/https://example.com/docs"
|
|
assert request.headers["Accept"] == "application/json"
|
|
# No key: Reader answers anonymously at a lower rate limit, and a
|
|
# deployment that has not configured one still gets this feature.
|
|
assert "Authorization" not in request.headers
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_configured_key_is_sent_and_an_absent_one_is_not(monkeypatch) -> None:
|
|
seen = _reader(monkeypatch, lambda request: httpx.Response(200, json={"data": {"content": "body"}}))
|
|
|
|
await _sources.fetch_page("https://example.com", api_key="k-1")
|
|
|
|
assert seen["requests"][0].headers["Authorization"] == "Bearer k-1"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_refused_url_never_reaches_the_network(monkeypatch) -> None:
|
|
"""Without this the endpoint is a request forger: it takes a URL from a
|
|
browser and answers with the body of whatever the gateway can reach."""
|
|
monkeypatch.setattr(_sources, "validate_url_target", lambda url: (False, "blocked host"))
|
|
|
|
def _explode(request: httpx.Request) -> httpx.Response:
|
|
pytest.fail("must not fetch a refused target")
|
|
|
|
real = httpx.AsyncClient
|
|
monkeypatch.setattr(
|
|
_sources.httpx,
|
|
"AsyncClient",
|
|
lambda *a, **k: real(*a, **{**k, "transport": httpx.MockTransport(_explode)}),
|
|
)
|
|
|
|
with pytest.raises(_sources.SourceFetchError) as caught:
|
|
await _sources.fetch_page("http://169.254.169.254/latest/meta-data/")
|
|
|
|
assert "blocked host" in str(caught.value)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_http_error_names_the_status_rather_than_the_stack(monkeypatch) -> None:
|
|
_reader(monkeypatch, lambda request: httpx.Response(404))
|
|
|
|
with pytest.raises(_sources.SourceFetchError) as caught:
|
|
await _sources.fetch_page("https://example.com/gone")
|
|
|
|
assert "404" in str(caught.value)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_page_with_no_text_is_an_error_not_an_empty_document(monkeypatch) -> None:
|
|
"""An empty document indexes to nothing and sits in the list looking as if
|
|
it worked."""
|
|
_reader(monkeypatch, lambda request: httpx.Response(200, json={"data": {"title": "Nothing", "content": " "}}))
|
|
|
|
with pytest.raises(_sources.SourceFetchError) as caught:
|
|
await _sources.fetch_page("https://example.com/empty")
|
|
|
|
assert "no readable text" in str(caught.value)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_answer_that_is_not_a_page_is_reported_as_one(monkeypatch) -> None:
|
|
_reader(monkeypatch, lambda request: httpx.Response(200, content=b"<html>not json</html>"))
|
|
|
|
with pytest.raises(_sources.SourceFetchError):
|
|
await _sources.fetch_page("https://example.com/html")
|
|
|
|
|
|
def test_a_page_with_nothing_to_name_it_by_still_gets_a_name() -> None:
|
|
"""An untitled page at a bare address is rare and not an error; a document
|
|
with no filename at all would be."""
|
|
assert _sources.page_filename("", "", "") == "page.md"
|
|
assert _sources.page_filename("", "http://[", "") == "page.md"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_reader_that_cannot_be_reached_says_that(monkeypatch) -> None:
|
|
def _refuse(request: httpx.Request) -> httpx.Response:
|
|
raise httpx.ConnectError("connection refused", request=request)
|
|
|
|
_reader(monkeypatch, _refuse)
|
|
|
|
with pytest.raises(_sources.SourceFetchError) as caught:
|
|
await _sources.fetch_page("https://example.com")
|
|
|
|
assert "could not reach" in str(caught.value)
|