Miguel 6e75725b39
Add screenshot option to Extract (#2149)
# why
Some pages do not properly encapsulate the necessary information
required to extract the content requested by the user, and can benefit
from a hybrid approach with vision.

# what changed
- New `extract({ options: { screenshot: true } })` flag. Captures the
current viewport as PNG and sends it as an `image_url` part alongside
the a11y tree text in the extraction LLM call.
- Default is `false` — existing callers see no behavior change.
- Gated to AI SDK clients; throws `StagehandInvalidArgumentError`
otherwise (validated in both the handler and `inference.extract`, with
the handler check running before the screenshot is taken).
- Screenshot capture is bracketed by `ensureTimeRemaining()` so it
respects the extract timeout.
- Wired end-to-end: public `ExtractOptions` type,
`ExtractOptionsSchema`, OpenAPI `ExtractOptions`,
`StagehandAPIClient.extract` wire payload, `V3.extract` →
`ExtractHandler` → `inference.extract` →
`buildExtract{System,User}Prompt`.

# test plan
- `extract-screenshot.test.ts` (new) — multimodal message shape on happy
path; rejection for non-AI SDK clients.
- `timeout-handlers.test.ts` — handler captures viewport when enabled,
skips capture by default, rejects screenshot + non-AI SDK client without
touching `page.screenshot` or the snapshot pipeline.
- `api-client-serialization.test.ts` (renamed from
`api-client-observe-variables.test.ts`) — wire payload preserves
`options.screenshot`.


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds a screenshot option to `extract()` that sends the current viewport
screenshot with the accessibility tree to improve results on
visually-driven pages.

- New Features
- `extract({ options: { screenshot: boolean } })` captures a PNG of the
current viewport and includes it with the accessibility tree; defaults
to false; only supported with `aisdk` clients (throws
`StagehandInvalidArgumentError` otherwise).
- Prompts are consolidated and multimodal when enabled: system and user
prompts mention the screenshot and using it with the accessibility tree;
attaches an `image_url` data URI; logs when a screenshot is used.
- Public API, types, and OpenAPI updated (`ExtractOptions.screenshot`);
the API client forwards the flag; tests cover prompt shape, viewport
capture, default-off behavior, timeout guards, client-type validation,
and serialization.

<sup>Written for commit d0c75002884eb42fcebb132c0d333259e8d9efbf.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2149?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-05-20 13:59:33 -07:00
..
2026-05-11 14:17:14 -07:00
2026-05-13 20:31:36 -07:00

Stagehand API

The Stagehand is a powerful service that provides a RESTful interface for browser automation and session management using the Browserbase platform. It enables recording, playback, and manipulation of browser sessions with a focus on reliability and performance.

📋 Prerequisites

To run the Stagehand API locally, ensure you have the following installed:

  • Node.js
  • pnpm

🛠 Installation

  1. Clone the repository:
git clone https://github.com/browserbase/stagehand/
cd stagehand/packages/server-v3
  1. Install dependencies:
pnpm install
  1. Set up environment variables:
cp .env.example .env
  1. Configure your .env file with the environment variables required by src/lib/env.ts (BB environment, API base URLs, etc.).

  2. pnpm dev