Raven-1/tests/test_ppt_engine_fetch_tool.py
Tchen-data 08482231ed
fix: ppt lane inherits host vendor web keys and ppt_fetch names itself (#736)
## Summary

Two faults that left the template-based deck lane (Raven-PPT) without
search and without web pictures on a current raven install.

1. **Web keys were not inherited from a vendor-table host.**
`agents/raven-ppt/run.py` only merged the two pre-vendor leaves
(`tools.web.search.apiKey`, `tools.web.jinaApiKey`). A host configured
on a current raven keeps those keys under
`tools.web.providers.<vendor>.apiKey`, so the rendered product config
carried no web key: `web_search` was withheld and `ppt_image_search`
declined. In a live run the author called both, got "tool is not
available", and spent six minutes paging a wiki API for picture file
names instead. The render now inherits every vendor slot the host fills,
unless the product already holds a key for that vendor (its own config,
or `PPT_SERPER_API_KEY` / `PPT_JINA_API_KEY`, which land in the leaf
trunk reads after the vendor slot, so inheriting the host slot would
outrank them). The host's `search.provider` / `fetch.provider` follow
where the product names none, so a key and its vendor choice travel
together. The engine slice's `imageSearch.apiKey` is taken in trunk's
order: vendor slot, then leaf. `raven-design` already copies the host's
`tools.web` whole; `raven-research` already reads the vendor table.

2. **`ppt_fetch` was refused by Wikimedia and others.** Downloads used
httpx's default `python-httpx/<version>` user agent, which
upload/thumb.wikimedia.org, en.wikipedia.org and sz.gov.cn answer with
403. All five Commons pictures in that run failed and the author fell
back to `curl`, which lands pictures without their captions (the figure
catalogue then held 19 images with `caption: null`). Downloads now send
`raven-ppt/<version> (+https://github.com/EverMind-AI/Raven)`. Measured
with curl on macOS: the same URLs return 403 with the httpx agent and
200 with this one (also checked bbc.com, arxiv.org, github.com: 200).

Not in this PR: `agents/raven-code` and `agents/raven-oncall` read the
same two legacy leaves only and likely have the same gap on a
vendor-table host.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- `uv run pytest tests/test_agents_ppt_launcher.py
tests/test_ppt_engine_fetch_tool.py -q`: 93 passed.
- The 5 new tests with the source changes reverted: 4 failed (the
own-key-precedence test is a guard and passes either way); with the
changes: all pass.
- `uv run pytest tests/test_subagent_vendored_agents.py
tests/test_config_product_render.py
tests/test_ppt_engine_prompt_claims.py
tests/test_agents_design_launcher.py -q`: 188 passed, 1 skipped.
- `uv run ruff check` and `uv run ruff format --check` on the four
changed files: clean.
- curl on macOS against the Wikimedia, Wikipedia and sz.gov.cn URLs from
the failing run, with both user agents (results above).

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

A deck lane on a vendor-table host now gets the host's web keys and
vendor choice, so `web_search` and `ppt_image_search` appear where they
were missing; a product-level key still wins. The fetch user agent
changes for every download; a site that only accepted browser agents
would still refuse, as before. Rollback is reverting this commit; the
env keys (`PPT_SERPER_API_KEY`) keep working either way.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-09-23 23:07:22 +08:00

761 lines
31 KiB
Python

"""Bringing something from the web into a deck's materials.
The tests that matter are about the bytes. A URL ending in `.png` that returns
HTML, and a `Content-Type: image/png` on something that is not an image, are both
ordinary on the open web -- and writing either into the materials directory would
fail later somewhere that reads like a bug in the ingest.
"""
from __future__ import annotations
import io
import json
import zipfile
from pathlib import Path
import httpx
import pytest
from raven.security.network import validate_url_target as _real_validate
from raven_ppt.contracts import Project
from raven_ppt.tools.fetch import MAX_BYTES, PptFetchTool
pytest.importorskip("PIL")
def _png(width: int = 8, height: int = 8) -> bytes:
from PIL import Image
buffer = io.BytesIO()
Image.new("RGB", (width, height), "white").save(buffer, format="PNG")
return buffer.getvalue()
def _described_png(description: str, *, key: str = "Description") -> bytes:
"""A PNG carrying its own description in a text chunk, as an export tool writes it."""
from PIL import Image
from PIL.PngImagePlugin import PngInfo
chunks = PngInfo()
chunks.add_text(key, description)
buffer = io.BytesIO()
Image.new("RGB", (320, 180), "white").save(buffer, format="PNG", pnginfo=chunks)
return buffer.getvalue()
def _described_jpeg(description: str) -> bytes:
"""A JPEG carrying EXIF `ImageDescription`, NUL-padded the way cameras write it."""
from PIL import Image
image = Image.new("RGB", (320, 180), "white")
exif = image.getexif()
exif[0x010E] = f"{description}\x00\x00"
buffer = io.BytesIO()
image.save(buffer, format="JPEG", exif=exif.tobytes())
return buffer.getvalue()
def _pptx() -> bytes:
"""A real presentation, because a downloaded template is now prepared as it
lands: the two ways one arrives -- by URL and by path -- have to end in the
same state, and a zip that only looks like a deck no longer reaches it."""
from pptx import Presentation
buffer = io.BytesIO()
presentation = Presentation()
presentation.slides.add_slide(presentation.slide_layouts[6])
presentation.save(buffer)
return buffer.getvalue()
def _not_a_deck() -> bytes:
"""A zip shaped like a .pptx that python-pptx will not open."""
buffer = io.BytesIO()
with zipfile.ZipFile(buffer, "w") as archive:
archive.writestr("[Content_Types].xml", "<Types/>")
archive.writestr("ppt/presentation.xml", "<presentation/>")
return buffer.getvalue()
@pytest.fixture()
def fetch(monkeypatch, tmp_path: Path):
"""A tool whose HTTP layer is a MockTransport.
Patched at httpx rather than at the tool, so the code under test is the code
that ships -- including the streaming size guard, which a fake download method
would have skipped.
"""
def make(body: bytes, *, headers: dict | None = None, status: int = 200):
def handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(status, content=body, headers=headers or {})
transport = httpx.MockTransport(handler)
real = httpx.AsyncClient
def factory(*args, **kwargs):
kwargs.pop("proxy", None)
return real(*args, transport=transport, **kwargs)
monkeypatch.setattr(httpx, "AsyncClient", factory)
monkeypatch.setattr("raven.security.network.validate_url_target", lambda url: (True, ""))
return PptFetchTool(tmp_path)
return make
async def _run(tool: PptFetchTool, **kw) -> dict:
args = {"project": "tarvis", "url": "https://example.com/thing"}
return json.loads(await tool.execute(**{**args, **kw}))
@pytest.mark.asyncio
async def test_an_image_lands_in_the_decks_own_sources(fetch, tmp_path: Path) -> None:
"""One place, always. A fetch that could land elsewhere started a second pile, and
the ingest that followed replaced the index rather than adding to it."""
body = await _run(fetch(_png()), url="https://example.com/figure")
assert body["ok"] is True and body["kind"] == "image"
landed = Path(body["path"])
assert landed.parent == Project(workspace=tmp_path, slug="tarvis").sources_dir
assert landed.suffix == ".png" and landed.read_bytes()
@pytest.mark.asyncio
async def test_the_url_is_recorded_beside_what_it_fetched(fetch, tmp_path: Path) -> None:
"""The attribution `documents.source_urls` exists to provide and never had:
nothing wrote the manifest it reads, so every fetched figure reached a slide with
no source at all."""
from raven_ppt.services.ingest import fetched
await _run(fetch(_png()), url="https://example.com/figure")
held = fetched(Project(workspace=tmp_path, slug="tarvis"))
assert [source.url for source in held] == ["https://example.com/figure"]
@pytest.mark.asyncio
async def test_the_caption_the_caller_gives_reaches_the_figure_catalogue(fetch, tmp_path: Path) -> None:
"""The whole point of the parameter. A picture off a web page carries no caption in
its bytes -- the words are in the HTML beside it -- so before this every fetched
figure entered the catalogue with `caption: null` while a paper's figure arrived
with the line printed under it. One live deck captioned a marketing banner as an
architecture diagram with nothing to hold it to."""
from raven_ppt.services.ingest import ingest_materials, load_catalogue
tool = fetch(_png(320, 180))
tool.ingest = ingest_materials
caption = "Figure 2: Architectural overview of the Mem0 system"
body = await _run(tool, url="https://example.com/figure.png", caption=caption)
assert body["ok"] is True and body["caption"] == caption
deck = Project(workspace=tmp_path, slug="tarvis")
catalogue = load_catalogue(deck.ingest_dir / "figures.json", figures_dir=deck.ingest_dir / "figures")
(asset,) = catalogue.values()
assert asset.caption == caption
assert asset.source_url == "https://example.com/figure.png"
@pytest.mark.asyncio
async def test_the_caption_is_recorded_beside_the_url_it_came_with(fetch, tmp_path: Path) -> None:
from raven_ppt.services.ingest import fetched
await _run(fetch(_png()), url="https://example.com/figure", caption=" Fig. 3\n Retrieval latency ")
(held,) = fetched(Project(workspace=tmp_path, slug="tarvis"))
assert held.caption == "Fig. 3 Retrieval latency"
@pytest.mark.asyncio
async def test_a_picture_that_describes_itself_needs_no_caption_from_the_caller(fetch, tmp_path: Path) -> None:
"""What is reachable from the bytes is taken rather than asked for twice."""
from raven_ppt.services.ingest import fetched
await _run(fetch(_described_png("Figure 4: token cost per turn")))
(held,) = fetched(Project(workspace=tmp_path, slug="tarvis"))
assert held.caption == "Figure 4: token cost per turn"
@pytest.mark.asyncio
async def test_an_exif_image_description_is_read_and_its_padding_dropped(fetch, tmp_path: Path) -> None:
"""EXIF strings are NUL-padded, and a NUL in the catalogue makes the file a
search tool treats as binary."""
from raven_ppt.services.ingest import fetched
await _run(fetch(_described_jpeg("Mem0 memory lifecycle")))
(held,) = fetched(Project(workspace=tmp_path, slug="tarvis"))
assert held.caption == "Mem0 memory lifecycle"
@pytest.mark.asyncio
async def test_the_pages_words_beat_the_files_own_metadata(fetch, tmp_path: Path) -> None:
"""The caller's caption is what the page printed next to this picture; an
`ImageDescription` field is whatever the file was exported with."""
from raven_ppt.services.ingest import fetched
await _run(fetch(_described_png("stock photo 118372")), caption="Figure 1: the extraction phase")
(held,) = fetched(Project(workspace=tmp_path, slug="tarvis"))
assert held.caption == "Figure 1: the extraction phase"
@pytest.mark.asyncio
async def test_a_second_fetch_without_a_caption_does_not_blank_the_first(fetch, tmp_path: Path) -> None:
"""Same URL, twice: a retry, or the same picture wanted for a second page. That is
not new information about these bytes, so it must not take the words away."""
from raven_ppt.services.ingest import fetched
# One tool for both calls: the factory wraps whatever `httpx.AsyncClient` is when
# it runs, so calling it twice in one test nests the transport into itself.
tool = fetch(_png(320, 180))
await _run(tool, caption="Figure 5: end-to-end latency")
await _run(tool)
(held,) = fetched(Project(workspace=tmp_path, slug="tarvis"))
assert held.caption == "Figure 5: end-to-end latency"
@pytest.mark.asyncio
async def test_a_caption_on_a_document_is_refused_out_loud_rather_than_dropped(fetch, tmp_path: Path) -> None:
"""A caption belongs to one picture and a paper holds many, so it is not recorded
for a document -- and saying so is the difference between this and the silence the
parameter exists to end. The paper's figures carry the captions its pages print."""
from raven_ppt.services.ingest import fetched
body = await _run(fetch(b"%PDF-1.7\n trailer"), caption="Figure 2: architectural overview")
assert body["ok"] is True and "caption" not in body
assert "describes one picture" in body["note"]
assert [held.caption for held in fetched(Project(workspace=tmp_path, slug="tarvis"))] == [""]
@pytest.mark.asyncio
async def test_the_suffix_comes_from_the_bytes_not_the_url(fetch, tmp_path: Path) -> None:
"""A URL ending in .png that returns HTML is ordinary on the open web."""
body = await _run(fetch(b"<html><body>not an image</body></html>"), url="https://example.com/photo.png")
assert body["ok"] is True and body["kind"] == "document"
assert Path(body["path"]).suffix == ".html"
@pytest.mark.asyncio
async def test_a_lying_content_type_does_not_get_a_pass(fetch) -> None:
body = await _run(fetch(b"\x00\x01\x02 not an image", headers={"content-type": "image/png"}))
assert body["ok"] is False
assert "not a PDF, a PowerPoint file, an image, or text" in body["error"]
@pytest.mark.asyncio
async def test_a_pdf_is_recognised_by_its_header(fetch) -> None:
body = await _run(fetch(b"%PDF-1.7\n trailer"))
assert body["kind"] == "document" and Path(body["path"]).suffix == ".pdf"
@pytest.mark.asyncio
async def test_a_pptx_lands_in_the_template_slot_not_the_materials(fetch, tmp_path: Path) -> None:
"""A deck is not a source to quote; it is a house style to work inside."""
body = await _run(fetch(_pptx()), url="https://example.com/house-style.pptx")
assert body["ok"] is True and body["kind"] == "template"
project = Project(workspace=tmp_path, slug="tarvis")
assert Path(body["path"]).parent == project.root / "template"
assert "house style" in body["next_step"]
@pytest.mark.asyncio
async def test_a_downloaded_template_is_prepared_where_it_lands(fetch, tmp_path: Path) -> None:
"""The two ways a template arrives have to end in the same state. Downloading
one and leaving it in the slot was the earlier behaviour, and it meant a
template that came by URL was never prepared, so nothing downstream saw one."""
from raven_ppt.services.template import bound
body = await _run(fetch(_pptx()), url="https://example.com/house-style.pptx")
assert body["ok"] is True
assert bound(Project(workspace=tmp_path, slug="tarvis")) is not None
assert body["template"] == "house-style.pptx"
@pytest.mark.asyncio
async def test_a_pptx_that_will_not_open_is_refused_rather_than_stored(fetch, tmp_path: Path) -> None:
body = await _run(fetch(_not_a_deck()))
assert body["ok"] is False
assert "does not open as a presentation" in body["error"]
assert not list((Project(workspace=tmp_path, slug="tarvis").root / "template").glob("*.pptx"))
@pytest.mark.asyncio
async def test_a_zip_that_is_not_a_presentation_is_refused(fetch) -> None:
buffer = io.BytesIO()
with zipfile.ZipFile(buffer, "w") as archive:
archive.writestr("readme.txt", "hello")
body = await _run(fetch(buffer.getvalue()))
assert body["ok"] is False and "not a PowerPoint file" in body["error"]
@pytest.mark.asyncio
async def test_a_declared_size_over_the_limit_is_refused_before_the_body_arrives(fetch) -> None:
body = await _run(fetch(b"%PDF-1.7", headers={"content-length": str(MAX_BYTES + 1)}))
assert body["ok"] is False and "32MB" in body["error"]
@pytest.mark.asyncio
async def test_an_undeclared_oversized_body_is_refused_as_it_arrives(fetch) -> None:
"""A server may not declare a length, so the bytes are counted too."""
body = await _run(fetch(b"%PDF-" + b"x" * (MAX_BYTES + 1)))
assert body["ok"] is False and "32MB" in body["error"]
@pytest.mark.asyncio
async def test_a_failed_request_says_so_rather_than_writing_nothing_quietly(fetch) -> None:
body = await _run(fetch(b"nope", status=404))
assert body["ok"] is False and "download failed" in body["error"]
@pytest.mark.asyncio
async def test_the_download_names_itself_rather_than_sending_the_library_default(monkeypatch, tmp_path: Path) -> None:
"""Wikimedia's hosts answer httpx's own `python-httpx/<version>` with a 403 and
serve the same file to an agent that names itself and a contact. The handler
here is that policy, so the fetch has to arrive with a name of its own."""
agents: list[str] = []
def handler(request: httpx.Request) -> httpx.Response:
agent = request.headers.get("user-agent", "")
agents.append(agent)
if agent.startswith("python-httpx"):
return httpx.Response(403)
return httpx.Response(200, content=_png())
transport = httpx.MockTransport(handler)
real = httpx.AsyncClient
def factory(*args, **kwargs):
kwargs.pop("proxy", None)
return real(*args, transport=transport, **kwargs)
monkeypatch.setattr(httpx, "AsyncClient", factory)
monkeypatch.setattr("raven.security.network.validate_url_target", lambda url: (True, ""))
body = await _run(PptFetchTool(tmp_path), url="https://upload.wikimedia.org/wikipedia/commons/a/a4/x.jpg")
assert body["ok"] is True, body
(agent,) = agents
assert agent.startswith("raven-ppt/") and "https://github.com/EverMind-AI/Raven" in agent
@pytest.mark.asyncio
async def test_there_is_no_materials_path_to_point_anywhere(fetch) -> None:
"""The parameter is gone, not defaulted: it existed to let a fetch land somewhere
other than the deck's sources, and there was never a good reason to."""
from raven_ppt.tools.fetch import PptFetchTool
assert "materials_dir" not in PptFetchTool(Path("/tmp")).parameters["properties"]
@pytest.mark.asyncio
async def test_a_bad_project_name_is_refused_before_the_download(fetch) -> None:
body = await _run(fetch(_png()), project="../etc")
assert body["ok"] is False and "usable project name" in body["error"]
@pytest.mark.asyncio
async def test_a_hostile_filename_is_reduced_to_something_safe(fetch, tmp_path: Path) -> None:
body = await _run(fetch(_png()), filename="../../etc/passwd")
assert Path(body["path"]).parent == Project(workspace=tmp_path, slug="tarvis").sources_dir
assert Path(body["path"]).name == "passwd.png"
@pytest.mark.asyncio
async def test_a_body_of_control_bytes_is_binary_however_well_it_decodes(fetch) -> None:
"""UTF-8 accepts control characters, so decoding is not a test for text.
Without this, a truncated video saved as `.md` becomes the ingest's problem
to explain.
"""
body = await _run(fetch(bytes(range(1, 32)) * 40))
assert body["ok"] is False and "not a PDF" in body["error"]
@pytest.mark.asyncio
async def test_an_empty_body_is_not_a_document(fetch) -> None:
body = await _run(fetch(b" \n\n "))
assert body["ok"] is False
@pytest.mark.asyncio
async def test_ordinary_prose_with_punctuation_and_cjk_is_still_text(fetch) -> None:
body = await _run(fetch("# 标题\n\n正文 — with an em dash, 90% coverage.\n".encode()))
assert body["ok"] is True and Path(body["path"]).suffix == ".md"
SVG = (
b'<svg xmlns="http://www.w3.org/2000/svg" width="240" height="60" viewBox="0 0 240 60">'
b'<rect width="240" height="60" fill="#155FFD"/><text x="12" y="40" fill="#fff">mem0</text></svg>'
)
def _needs_cairo() -> None:
from raven_ppt.tools.fetch import _load_cairosvg
module, missing = _load_cairosvg()
if module is None:
pytest.skip(f"rasterising an SVG needs the native cairo library: {missing}")
def test_a_brand_mark_arrives_as_svg_and_is_kept_as_a_picture() -> None:
"""A logo saved as `.md` is a competitor analysis with no competitor's mark on it.
Rasterising needs cairo's native library, which `cairosvg` cannot supply itself, so
this asserts the conversion only where the conversion is possible. The skip goes
through the same loader the tool uses, so a Homebrew libcairo the plain import
cannot see no longer skips it. The degraded path is a case of its own below.
"""
_needs_cairo()
from raven_ppt.tools.fetch import _sniff
payload, suffix, kind = _sniff(SVG)
assert (suffix, kind) == (".png", "image")
assert payload.startswith(b"\x89PNG"), "python-pptx places rasters only"
def test_an_svg_that_will_not_draw_is_refused_not_kept_as_text() -> None:
_needs_cairo()
from raven_ppt.tools.fetch import _sniff
with pytest.raises(ValueError, match="SVG that would not draw"):
_sniff(b"<svg this is not really an svg at all")
def test_a_page_with_inline_svg_icons_is_a_page(monkeypatch) -> None:
"""A page is not a picture because an icon sits near its top, and the rasteriser
is not asked about it -- a missing cairo must not refuse ordinary HTML."""
from raven_ppt.tools import fetch
from raven_ppt.tools.fetch import _sniff
monkeypatch.setattr(fetch, "_load_cairosvg", lambda: pytest.fail("a page reached the rasteriser"))
page = b'<!doctype html><html><body><svg viewBox="0 0 1 1"><path d="M0 0"/></svg>hi</body></html>'
payload, suffix, kind = _sniff(page)
assert (payload, suffix, kind) == (page, ".html", "document")
def test_an_svg_behind_a_declaration_and_comment_is_still_an_svg(monkeypatch) -> None:
from raven_ppt.tools import fetch
from raven_ppt.tools.fetch import _sniff
monkeypatch.setattr(fetch, "_load_cairosvg", lambda: (None, "no cairo here"))
exported = (
b'<?xml version="1.0"?>\n<!-- Generator: Sketch -->\n<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN">\n' + SVG
)
with pytest.raises(ValueError, match="no cairo here"):
_sniff(exported)
def test_an_svg_behind_a_byte_order_mark_is_still_an_svg(monkeypatch) -> None:
"""Editors that save UTF-8 with a BOM put it before the root element; read as
text, it would be kept as path data in a document."""
from raven_ppt.tools import fetch
from raven_ppt.tools.fetch import _sniff
monkeypatch.setattr(fetch, "_load_cairosvg", lambda: (None, "no cairo here"))
with pytest.raises(ValueError, match="no cairo here"):
_sniff(b"\xef\xbb\xbf" + SVG)
def _without_libcairo(monkeypatch) -> None:
"""`import cairosvg` pulls in cairocffi, which raises OSError -- not ImportError --
when the native library is absent."""
import builtins
import sys
real_import = builtins.__import__
def no_libcairo(name, *args, **kwargs):
if name == "cairosvg":
raise OSError('no library called "cairo-2" was found\nno library called "cairo" was found')
return real_import(name, *args, **kwargs)
monkeypatch.delitem(sys.modules, "cairosvg", raising=False)
monkeypatch.setattr(builtins, "__import__", no_libcairo)
def test_a_machine_without_libcairo_refuses_the_svg_and_says_what_installs_it(monkeypatch) -> None:
"""The SVG used to be kept as a `.md` of path data, with no figure id and no log
line, so a deck lost its logos while every fetch reported success."""
from loguru import logger
from raven.utils import cairo
from raven_ppt.tools.fetch import _sniff
_without_libcairo(monkeypatch)
logged: list[str] = []
sink = logger.add(logged.append, level="WARNING", format="{message}")
try:
with pytest.raises(ValueError) as refused:
_sniff(SVG)
finally:
logger.remove(sink)
said = str(refused.value)
assert 'no library called "cairo-2" was found' in said
assert 'cairo" was found' not in said, "one line of cairocffi's list, not all of it"
assert cairo.install_hint() in said
assert any("cairo could not be loaded" in line for line in logged), logged
@pytest.mark.asyncio
async def test_a_refused_svg_leaves_nothing_in_the_decks_sources(fetch, monkeypatch, tmp_path: Path) -> None:
_without_libcairo(monkeypatch)
body = await _run(fetch(SVG), url="https://example.com/mem0-logo.svg")
assert body["ok"] is False and "SVG" in body["error"]
sources_dir = Project(workspace=tmp_path, slug="tarvis").sources_dir
assert not sources_dir.exists() or not any(sources_dir.iterdir())
@pytest.mark.asyncio
async def test_a_host_never_reached_names_the_setting_that_would_reach_it(fetch, monkeypatch, tmp_path: Path) -> None:
"""The refusal a run actually got was `the download failed: ` and nothing else.
`httpx.ConnectTimeout("")` renders empty, so the class was the only thing there
was to say and it was not said. And the cause is knowable: every client in this
tool is built `trust_env=False`, so a machine that reaches the internet through
a proxy goes direct here however its environment is set -- measured at 50s per
timeout while curl through that same proxy answered in 0.86s.
"""
from raven_ppt.tools.fetch import PptFetchTool
tool = PptFetchTool(tmp_path)
async def _timeout(url):
raise httpx.ConnectTimeout("")
monkeypatch.setattr(tool, "_download", _timeout)
monkeypatch.setattr("raven.security.network.validate_url_target", lambda url: (True, ""))
said = await _run(tool, url="https://example.com/a.png")
assert said["ok"] is False
assert "ConnectTimeout" in said["error"], said["error"]
assert "tools.web.proxy" in said["hint"]
assert "HTTPS_PROXY" in said["hint"]
# A configured proxy means the network is already being routed, so the hint
# would be wrong; and a status is the server talking, which needs no hint.
routed = PptFetchTool(tmp_path, proxy="http://127.0.0.1:7890")
monkeypatch.setattr(routed, "_download", _timeout)
assert "hint" not in await _run(routed, url="https://example.com/a.png")
async def _refused(url):
raise httpx.HTTPStatusError("403", request=None, response=None)
monkeypatch.setattr(tool, "_download", _refused)
assert "hint" not in await _run(tool, url="https://example.com/a.png")
@pytest.mark.asyncio
async def test_an_address_the_network_guard_refuses_is_not_downloaded(fetch, monkeypatch, tmp_path: Path) -> None:
"""`validate_url_target` answers, it does not throw.
The call read it by catching an exception, so every refused address went
through: loopback, the link-local range, and the cloud metadata endpoint that
hands out credentials to whoever asks from inside the host.
"""
tool = fetch(_png())
# After the factory, which installs the permissive stub the other cases need.
monkeypatch.setattr("raven.security.network.validate_url_target", _real_validate)
out = await _run(tool, url="http://169.254.169.254/latest/meta-data/iam/security-credentials/")
assert out.get("ok") is False, "the metadata endpoint must not be fetched"
assert "cannot be fetched" in json.dumps(out), out
landed = list((tmp_path / "tarvis").rglob("*")) if (tmp_path / "tarvis").exists() else []
assert not [p for p in landed if p.is_file()], "nothing may be written for a refused address"
@pytest.mark.asyncio
async def test_a_fetched_image_hands_back_the_id_the_catalogue_keys_it_under(fetch, tmp_path: Path) -> None:
"""Nothing else in the reply can be turned into one.
`ppt_figure_inspect` takes catalogue ids and nothing else, and the catalogue keys a
picture as `<stem>-<digest>`. A measured run fetched ten images, called inspect with
the filenames it had just passed to this tool, was told none of them was in the
catalogue, then guessed at the digests and got their length wrong -- two iterations
spent recovering a string the fetch already had on disk.
"""
from raven_ppt.services.ingest import ingest_materials, load_catalogue
tool = fetch(_png(320, 180))
tool.ingest = ingest_materials
body = await _run(tool, url="https://example.com/figure.png", filename="diagram.png")
deck = Project(workspace=tmp_path, slug="tarvis")
catalogue = load_catalogue(deck.ingest_dir / "figures.json", figures_dir=deck.ingest_dir / "figures")
assert body["figure_id"] in catalogue, "the id has to be one inspect will accept"
assert body["figure_id"].startswith("diagram-")
@pytest.mark.asyncio
async def test_the_id_a_fetch_hands_back_is_one_figure_inspect_accepts(fetch, tmp_path: Path) -> None:
"""End to end across the two tools, because the ids agreeing is the whole claim and
each tool builds its own view of the catalogue."""
from raven_ppt.services.ingest import ingest_materials
from raven_ppt.tools.inspect import PptFigureInspectTool
tool = fetch(_png(320, 180))
tool.ingest = ingest_materials
body = await _run(tool, url="https://example.com/figure.png", filename="diagram.png")
inspect = PptFigureInspectTool(tmp_path, _Views())
seen = await inspect.execute(project="tarvis", figures=[body["figure_id"]])
reply = json.loads(seen.model_text if hasattr(seen, "model_text") else seen)
assert reply["ok"] is True, reply
assert [f["figure_id"] for f in reply["figures"]] == [body["figure_id"]]
class _Views:
"""Just the surface `ppt_figure_inspect` asks of the render service."""
async def pages(self, *_args, **_kwargs):
return {}
def data_uri(self, png, budget=None):
return f"data:image/png;base64,{Path(png).stem}"
def test_a_webp_picture_is_kept_as_a_png_the_deck_can_place() -> None:
"""Image hosts serve WebP; python-pptx places none of them, and the ingest
refused the file as unreadable -- a run spent three fetches on one picture."""
import io
from PIL import Image
from raven_ppt.tools.fetch import _sniff
raw = io.BytesIO()
Image.new("RGB", (12, 8), (200, 40, 40)).save(raw, format="WEBP")
payload, suffix, kind = _sniff(raw.getvalue())
assert (suffix, kind) == (".png", "image")
assert payload.startswith(b"\x89PNG")
with Image.open(io.BytesIO(payload)) as png:
assert png.size == (12, 8)
def test_a_webp_keeps_the_description_it_carried_across_the_conversion() -> None:
"""A fetch without a caption falls back to what the file says about itself, and a
re-encode that dropped the EXIF block would silently cost that provenance."""
import io
from PIL import Image
from raven_ppt.tools.fetch import _embedded_caption, _sniff
image = Image.new("RGB", (12, 8), (40, 40, 200))
exif = image.getexif()
exif[0x010E] = "A red brand mark"
raw = io.BytesIO()
image.save(raw, format="WEBP", exif=exif.tobytes())
assert _embedded_caption(raw.getvalue()) == "A red brand mark"
payload, _suffix, _kind = _sniff(raw.getvalue())
assert _embedded_caption(payload) == "A red brand mark"
@pytest.mark.asyncio
async def test_several_urls_are_fetched_in_one_call(fetch, tmp_path: Path) -> None:
"""A live run spent nine model round trips on nine fetches, minutes each on that
model; one call with the list fetches them together and ingests once."""
tool = fetch(_png())
body = json.loads(
await tool.execute(project="tarvis", urls=["https://example.com/a.png", "https://example.com/b.png"])
)
assert body["ok"] and len(body["results"]) == 2
assert all(item["ok"] for item in body["results"])
assert {Path(item["path"]).name for item in body["results"]} == {"a.png", "b.png"}
assert "urls" not in body
@pytest.mark.asyncio
async def test_two_sources_whose_paths_end_the_same_do_not_overwrite_each_other(fetch, tmp_path: Path) -> None:
"""`one.example/image.png` and `two.example/image.png` both landed on
`sources/image.png`: two rows reported success and `receive` replaced the manifest
row and the bytes, so ingestion saw one source. The url's digest tells them apart.
"""
tool = fetch(_png())
body = json.loads(
await tool.execute(project="tarvis", urls=["https://one.example/image.png", "https://two.example/image.png"])
)
assert body["ok"] and len(body["results"]) == 2
names = [Path(item["path"]).name for item in body["results"]]
assert len(set(names)) == 2, names
assert all(name.startswith("image-") and name.endswith(".png") for name in names), names
from raven_ppt.contracts import Project
from raven_ppt.services.ingest import sources
held = {source.origin for source in sources.held(Project(workspace=tmp_path, slug="tarvis"))}
assert held == {"fetch:https://one.example/image.png", "fetch:https://two.example/image.png"}
@pytest.mark.asyncio
async def test_two_urls_whose_names_normalize_alike_are_told_apart_too(fetch, tmp_path: Path) -> None:
"""The collision is in the destination, not in the url. `_safe_name` folds every
character outside its allowlist into an underscore, so `chart+final.png` and
`chart@final.png` are two basenames and one file; grouping on the raw basename saw
no collision and let the second overwrite the first."""
tool = fetch(_png())
body = json.loads(
await tool.execute(
project="tarvis", urls=["https://one.example/chart+final.png", "https://two.example/chart@final.png"]
)
)
assert body["ok"] and len(body["results"]) == 2
paths = [Path(item["path"]) for item in body["results"]]
assert len({path.name for path in paths}) == 2, paths
assert all(path.is_file() for path in paths)
from raven_ppt.contracts import Project
from raven_ppt.services.ingest import sources
held = {source.origin for source in sources.held(Project(workspace=tmp_path, slug="tarvis"))}
assert held == {"fetch:https://one.example/chart+final.png", "fetch:https://two.example/chart@final.png"}
def test_a_mark_survives_the_name_it_is_added_to() -> None:
"""`_safe_name` strips what it reads as an extension from the name it is given, so
a mark appended after one -- `image.tar` becoming `image.tar-1a2b3c4d` -- came back
as `image`, and the two urls were on one destination again."""
from raven_ppt.tools.fetch import _batch_names, _safe_name
urls = ["https://one.example/image.tar.gz", "https://two.example/image.tar.bz2"]
names = _batch_names(urls)
assert len({_safe_name(url, names[url], ".png") for url in urls}) == 2
@pytest.mark.asyncio
async def test_a_batch_without_a_collision_keeps_the_names_the_urls_gave(fetch, tmp_path: Path) -> None:
tool = fetch(_png())
body = json.loads(
await tool.execute(project="tarvis", urls=["https://example.com/a.png", "https://example.com/b.png"])
)
assert {Path(item["path"]).name for item in body["results"]} == {"a.png", "b.png"}