Teknium 6fc7c7dbf2 fix(web): Firecrawl re-checks the website policy against the URL it was redirected to
_scrape_one read metadata.sourceURL as the post-redirect URL, but Firecrawl
reports the REQUESTED url there (sourceURL on REST, source_url in the v4 SDK's
model_dump used by the direct-key and Tool Gateway paths) and the final URL as
metadata.url. The SSRF / website-blocklist re-check therefore re-checked the
requested URL, and a public redirector returned a blocklisted page.

Read the final URL from metadata.url, and fill sourceURL from source_url so the
extract cache still files a redirected page under the requested URL.

Found on Hermes Cloud staging.
2026-10-28 13:54:02 -07:00
..