1298 Commits

Author SHA1 Message Date
monadoid
ea143a674c Fix SDK API reference parameter labels 2026-05-25 18:15:45 +02:00
Miguel
2cd60a34b0
feat(verifier): add verifier evaluator shell and types (#2157) 2026-05-22 12:38:21 -07:00
Nick Sweeting
295ac6d1ce
Delete packages/server-v4 entirely for now (#2151) 2026-05-21 17:45:26 -07:00
Miguel
278890b116
fix[evals] remove braintrust api key requirement (#2145) (#2153)
# why
Evals should be able to run without the braintrust dependency

# what changed
- Added `hasBraintrustApiKey()` and made `tracedSpan()` a passthrough
when no key.
- `AISdkClientWrapped` falls back to the plain `ai` SDK when
`braintrust` wrapping isn’t available; introduced a `WrappedAI` type and
lazy load keyed on `BRAINTRUST_API_KEY`.

# test plan
- [x] Added tests for key-present/absent paths and updated `core-runner`
test to set a dummy key; fixes for test stability and build.


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Make Braintrust optional for evals. Evals run without
`BRAINTRUST_API_KEY` and skip sending logs; behavior is unchanged when a
key is set.

- **Bug Fixes**
- Added `hasBraintrustApiKey()` and made `tracedSpan()` a passthrough
when no key.
- Runner now passes `noSendLogs: true` to `Eval` and skips `flush()`
without a key; sends logs and flushes when present.
- `AISdkClientWrapped` falls back to the plain `ai` SDK when
`braintrust` wrapping isn’t available; introduced a `WrappedAI` type and
lazy load keyed on `BRAINTRUST_API_KEY`.
- Added tests for key-present/absent paths and updated `core-runner`
test to set a dummy key; fixes for test stability and build.

<sup>Written for commit 482e49bac6dce88ea110aa9a4f06a50346617a06.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2153?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Chromie <miguel@browserbase.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-05-21 15:39:40 -07:00
Miguel
43548374b9
Workflow: publish eval results (#2093) 2026-05-21 13:08:12 -07:00
Miguel
765861c04c
feat(verifier): add evaluator backend facade (#2129)
# Why

The verifier rewrite needs to coexist with the legacy v3 evaluator so
existing evals and callers do not silently change behavior while the new
pipeline is reviewed. This PR creates the compatibility boundary that
lets us select the evaluator backend explicitly.

# What Changed

- Extracted the current v3 evaluator behavior into `LegacyV3Evaluator`.
- Kept `V3Evaluator` as the public facade.
- Added `STAGEHAND_EVALUATOR_BACKEND=legacy|verifier` and constructor
backend options.
- Defaulted the backend to `legacy` to preserve current `ask()` and
`batchAsk()` semantics.
- Added public API tests for backend selection and invalid backend
handling.

# Tests

- `pnpm --filter @browserbasehq/stagehand run typecheck`
- `pnpm --filter @browserbasehq/stagehand run test:core --
packages/core/dist/esm/tests/unit/public-api/v3-core.test.js`
- `git diff --check`

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 14:56:53 -07:00
Miguel
6e75725b39
Add screenshot option to Extract (#2149)
# why
Some pages do not properly encapsulate the necessary information
required to extract the content requested by the user, and can benefit
from a hybrid approach with vision.

# what changed
- New `extract({ options: { screenshot: true } })` flag. Captures the
current viewport as PNG and sends it as an `image_url` part alongside
the a11y tree text in the extraction LLM call.
- Default is `false` — existing callers see no behavior change.
- Gated to AI SDK clients; throws `StagehandInvalidArgumentError`
otherwise (validated in both the handler and `inference.extract`, with
the handler check running before the screenshot is taken).
- Screenshot capture is bracketed by `ensureTimeRemaining()` so it
respects the extract timeout.
- Wired end-to-end: public `ExtractOptions` type,
`ExtractOptionsSchema`, OpenAPI `ExtractOptions`,
`StagehandAPIClient.extract` wire payload, `V3.extract` →
`ExtractHandler` → `inference.extract` →
`buildExtract{System,User}Prompt`.

# test plan
- `extract-screenshot.test.ts` (new) — multimodal message shape on happy
path; rejection for non-AI SDK clients.
- `timeout-handlers.test.ts` — handler captures viewport when enabled,
skips capture by default, rejects screenshot + non-AI SDK client without
touching `page.screenshot` or the snapshot pipeline.
- `api-client-serialization.test.ts` (renamed from
`api-client-observe-variables.test.ts`) — wire payload preserves
`options.screenshot`.


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds a screenshot option to `extract()` that sends the current viewport
screenshot with the accessibility tree to improve results on
visually-driven pages.

- New Features
- `extract({ options: { screenshot: boolean } })` captures a PNG of the
current viewport and includes it with the accessibility tree; defaults
to false; only supported with `aisdk` clients (throws
`StagehandInvalidArgumentError` otherwise).
- Prompts are consolidated and multimodal when enabled: system and user
prompts mention the screenshot and using it with the accessibility tree;
attaches an `image_url` data URI; logs when a screenshot is used.
- Public API, types, and OpenAPI updated (`ExtractOptions.screenshot`);
the API client forwards the flag; tests cover prompt shape, viewport
capture, default-off behavior, timeout guards, client-type validation,
and serialization.

<sup>Written for commit d0c75002884eb42fcebb132c0d333259e8d9efbf.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2149?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-05-20 13:59:33 -07:00
Sam F
3e95a8722a
[STG-1756] add Vertex auth params to Stagehand spec (#2118)
## Why
The generated SDK/OpenAPI surface needs to expose the Vertex
service-account auth fields that Stagehand can already pass through.

## What changed
- Add `vertex` model config fields to the canonical Stagehand Zod
schema: `project`, `location`, `baseURL`, and `googleAuthOptions`.
- Model `googleAuthOptions.credentials` as a strict
service-account-compatible object requiring `client_email` and
`private_key`.
- Omit file-path auth, external credential-source auth, and Vertex
Express Mode API-key auth from the AI SDK v5-compatible schema.
- Regenerate the v3 OpenAPI spec from the Zod source of truth.
- Add schema tests covering Vertex auth passthrough, minimal
credentials, and rejection of unsupported auth shapes.

---------

Co-authored-by: samfinton <samfinton@samLaptopm5.galaxus.box>
Co-authored-by: samfinton <samfinton@samLaptopm5.local>
stagehand-server-v3/v3.6.10
2026-05-20 21:15:28 +02:00
shriyatheunicorn
3a53ed4ea9
Pass local launch options when attaching over CDP (#2146)
## Summary
- Pass `localBrowserLaunchOptions` into `V3Context.create()` when LOCAL
mode attaches to an existing browser via `cdpUrl`.
- Add a regression test that verifies the CDP attach path forwards
viewport, device scale factor, and CDP headers through context creation.
- Add a patch changeset for the published Stagehand runtime behavior
change.

## Why
The launch path already passes local browser options into `V3Context`,
which lets `Page.create()` apply the configured viewport and
`deviceScaleFactor`. The CDP attach path only forwarded `env` and
`cdpHeaders`, so attached local sessions skipped that Page-level
viewport setup.

## E2E Test Matrix

| Command / flow | Observed output | Confidence / sufficiency |
| --- | --- | --- |
| `pnpm install --frozen-lockfile` | Installed all 7 workspace projects
from the lockfile; completed with `Done in 13.9s`. | Confirms the fresh
PR worktree can resolve dependencies exactly from the committed
lockfile. |
| `pnpm exec turbo run build:esm --filter=@browserbasehq/stagehand` |
Turbo ran `gen-version`, `build-dom-scripts`, and `build:esm`; output
ended with `Tasks: 3 successful, 3 total`. | Builds the exact local
package under review, including generated runtime artifacts needed by
the ESM output. |
| `pnpm --filter @browserbasehq/stagehand exec vitest run --config
vitest.esm.config.mjs
dist/esm/tests/unit/browserbase-session-accessors.test.js` | Vitest
output: `Test Files 1 passed (1)`, `Tests 6 passed (6)`. | Confirms the
regression test and nearby Browserbase/LOCAL accessor tests pass against
the built ESM output. |
| `node --input-type=module <inline LOCAL CDP attach viewport smoke>` |
Launched local Stagehand, reattached via `cdpUrl` with `viewport: {
width: 1440, height: 900 }`, and printed
`{"status":"pass","cdpAttachViewport":{"innerWidth":1440,"innerHeight":900,"devicePixelRatio":1,"bottomRightHit":"target"}}`.
| Exercises the real built Stagehand package against a real local Chrome
CDP session. Proves explicit CDP attach viewport settings control
browser CSS viewport and bottom-right coordinate hit-testing; does not
exercise kernel.sh specifically. |
| `pnpm exec changeset status --since=origin/main` | Changesets reported
patch bumps for `@browserbasehq/stagehand` and dependent workspace
packages. | Confirms the new changeset is detected as a
release-triggering patch for this runtime behavior fix. |

---------

Co-authored-by: Shrey Pandya <shrey@browserbase.com>
2026-05-19 14:23:13 -07:00
Sean McGuire
4ae6f8e094
[chore]: rm cli release workflow (#2143)
# why
- the cli has moved to a different repo
# what changed
- removed release-cli.yml & changelog.md
- cleaned up cli publish related metadata from package.json


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Remove the CLI release workflow and publish hooks now that
`@browserbasehq/browse-cli` lives in its own repo. This prevents
accidental publishes from this monorepo and simplifies releases.

- **Refactors**
- Deleted `.github/workflows/release-cli.yml` and removed CLI
publish/canary steps from `.github/workflows/release.yml`.
  - Removed `packages/cli/CHANGELOG.md`.
- Cleaned up publish-related fields in `packages/cli/package.json`
(publishConfig, files, prepublishOnly).
- Cleared `.changeset/config.json` ignore list to stop special-casing
the CLI.

<sup>Written for commit 880c857cf93643500165ed722e29e9ca69218aa9.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2143?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-05-19 12:45:47 -07:00
Sean McGuire
78bcde88e2
[feat]: forward ignoreDefaultArgs to chrome-launcher (#2127)
thanks @CalebBarnes for the contribution! (original PR: #2074)
# why

When running Stagehand locally, chrome-launcher adds its own set of
default flags (e.g. `--disable-extensions`). There's currently no way
for consumers to selectively remove these flags —
`localBrowserLaunchOptions.chromeFlags` can only *add* flags, not remove
chrome-launcher's built-in defaults.

This is needed when users want to run Chrome with extensions enabled, or
need to remove other chrome-launcher defaults for their use case.

# what changed

- Added `ignoreDefaultArgs` option to `LaunchLocalOptions` in `local.ts`
- Forwarded `localBrowserLaunchOptions.ignoreDefaultArgs` from `v3.ts`
through to `launchLocalChrome()`
- When `true`: drops all chrome-launcher defaults (only Stagehand's own
flags and user-supplied `chromeFlags` are used)
- When `string[]`: selectively removes only the listed flags from
chrome-launcher defaults, keeping the rest

This mirrors the behavior of Playwright's

[`ignoreDefaultArgs`](https://playwright.dev/docs/api/class-browsertype#browser-type-launch-option-ignore-default-args)
option.

# test plan

- Verified locally that setting `ignoreDefaultArgs:
["--disable-extensions"]` successfully removes the flag from the
launched Chrome instance (confirmed via `chrome://version/`)
- Verified that other chrome-launcher defaults are preserved when using
the array form
- Verified that `ignoreDefaultArgs: true` drops all chrome-launcher
defaults


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds `ignoreDefaultArgs` to local Chrome launch so you can drop all or
selected default flags from `chrome-launcher` with exact-match control
(including Stagehand defaults), while keeping your own `args`.

- **New Features**
- Added `ignoreDefaultArgs` to `LaunchLocalOptions` and moved flag
assembly into `launchLocalChrome()`, forwarding to `chrome-launcher` via
`ignoreDefaultFlags`.
- `true` drops all `chrome-launcher` defaults; `string[]` removes only
exact-matched defaults and re-adds the rest from
`Launcher.defaultFlags()`. Stagehand defaults and user `args` stay
unless explicitly listed.

<sup>Written for commit e04da99099ed354d775c1a67253358abdfd36fe3.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2127?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->



---------

---------

Co-authored-by: Caleb Barnes <caleb.d.barnes@gmail.com>
Co-authored-by: Caleb Barnes <CalebBarnes@users.noreply.github.com>
Co-authored-by: Mastra Code (anthropic/claude-opus-4-6) <noreply@mastra.ai>
2026-05-15 14:35:34 -07:00
Sean McGuire
ebbdcd33cb
[fix]: import ToolSet from ai public export (#2126)
thanks @asim48-ctrl for the contribution!

## Summary
- replace the remaining `ai/dist` deep import with the public `ai`
package export
- aligns `AgentProvider` with the other Stagehand v3 CUA files that
already import `ToolSet` from `ai`

Fixes #1531.

## Verification
- `corepack pnpm install --ignore-scripts --frozen-lockfile`
- `corepack pnpm --filter @browserbasehq/stagehand run gen-version`
- `corepack pnpm --filter @browserbasehq/stagehand run
build-dom-scripts`
- `corepack pnpm exec tsc -p packages/core/tsconfig.json --noEmit`
- `corepack pnpm exec prettier --check
packages/core/lib/v3/agent/AgentProvider.ts`
- `git diff --check`

<!-- This is an auto-generated description by cubic. --> ---
## Summary by cubic
Switch AgentProvider to import ToolSet from the public `ai` export
instead of `ai/dist` to prevent deep import breakage. Adds a patch
changeset and aligns with other Stagehand v3 CUA files; fixes #1531.

<sup>Written for commit 1c1767582d16ac2333f171ac87dc877dd6d6ba93.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->

Co-authored-by: asim48-ctrl <asim48@gmail.com>
2026-05-15 13:14:05 -07:00
Sean McGuire
7ed26a87b4
[chore]: rm langchain deps (#2123)
# why
- these were unneeded dependencies:
  - they were not used in library code at all
  - nor were they included in the public export surface
# what changed
- removed `lanchain/core` and `langchain/openai` from dependencies
- removed langchain examples
- updated lockfile

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Removed unused LangChain dependencies and examples from `packages/core`
to reduce install size and simplify the codebase. No API or runtime
behavior changes.

- **Dependencies**
- Removed `@langchain/core` and `@langchain/openai` from
`packages/core/package.json`.
  - Updated `pnpm-lock.yaml` to drop related transitive packages.

- **Refactors**
- Deleted LangChain examples:
`packages/core/examples/langchain-client.ts` and
`packages/core/examples/custom-client-langchain.ts`.

<sup>Written for commit e46cfa4d1f946adac98769253cc2c01378bf9cd1.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-14 16:16:13 -07:00
Sean McGuire
f9622459b7
[chore]: rm deepmerge from peer deps (#2124)
# why
- unused peer dep
# what changed
- removed `deepmerge` from peer deps


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Removed unused peer dependency `deepmerge` from `packages/core`. This
simplifies installs and removes unnecessary peer warnings.

<sup>Written for commit 50652b85227a06627d557c1c431b33c96470555c.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-14 16:10:43 -07:00
Sean McGuire
4d1c2fdb95
[chore]: omit test files from published package (#2122)
# why
- users don't need to be installing our tests
# what changed
- added explicit negations for `dist/esm/tests` and `dist/cjs/tests` in
`packages/core/package.json`
- this keeps our test suite running with the built code in `dist`, while
still avoiding publishing test files to npm
# test plan
- verified that test are not included in tarball with `npm pack
--dry-run`
- existing test suite

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Stop publishing test files with the package to reduce the npm tarball
and avoid installing unnecessary files. Updated
`packages/core/package.json` to exclude `dist/esm/tests` and
`dist/cjs/tests` via the `files` array; tests still run against built
code locally and were verified with `npm pack --dry-run`.

<sup>Written for commit ccd7212304bcec1b8a8e32d32c035eb4263cb81e.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-14 14:51:51 -07:00
Miguel
12703a6659
Fix structuredOutputMode for newer Anthropic models (#2120) 2026-05-14 14:12:18 -07:00
Miguel
448eb36dfe
Pin github action/workflow versions (#2121) 2026-05-14 14:11:50 -07:00
Sean McGuire
cb586a14e4
[fix]: include native selector state into snapshot (#2116)
thanks to @imorhun for this contribution! original PR can be found here
#2105

# why
Stagehand accessibility snapshot output was missing native selection
state markers (`selected` / `checked`) that are visible in Playwright
`ariaSnapshot({ mode: "ai" })`.
That mismatch made it impossible to extract the selected/checked item
for dropdowns and radio groups.

Example:
```html
<select aria-label="Select field">
  <option>Option A</option>
  <option selected>Option B</option>
  <option>Option C</option>
</select>
```

Before this fix, Stagehand-style snapshot output did not indicate the
selected option:

```
[1-4] select: Select field
  [1-5] option: Option A
  [1-6] option: Option B
  [1-7] option: Option C
```
Expected (and now aligned more closely with Playwright-style output):
selected state is explicit:

```
[1-4] select: Select field
  [1-5] option: Option A
  [1-6] option: Option B [selected]
  [1-7] option: Option C
```

# what changed
- added explicit boolean fields on `A11yNode`:
    - `selected?: boolean`
    - `checked?: boolean`
- updated AX mapping in `a11yTree` to:
    - read native AX `selected` / `checked` from AX properties
- normalize boolean-like AX values (`true/false`, `1/0`,
`"true"/"false"`)
    - store the result directly on `selected` / `checked`
- updated tree formatting in `treeFormatUtils` to render:
    - `[selected]` when `selected === true`
    - `[checked]` when `checked === true`

# test plan
- AX selected/checked extraction into boolean fields
- select/radio parent-child rendering with exactly one selected/checked
child
    - rendering both flags when a node carries both states

---------

Co-authored-by: Ihor <vreysu@gmail.com>
2026-05-14 13:10:23 -07:00
Sean McGuire
9806b2d13c
[chore]: bump mintlify version (#2115)
# why
- bumps to latest mintlify version with various transitive dependency
fixes
# what changed
- bumped mintlify to version `5.2.563`


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Bump `mintlify` in `packages/docs` to ^4.2.563 to pull in upstream fixes
and refresh transitive dependencies for the docs build.

- **Dependencies**
- Updated lockfile; notable bumps include `@mintlify/cli@4.0.1166`,
`@mintlify/common@1.0.895`, `@mintlify/previewing@4.0.1100`,
`@mintlify/validation@0.1.701`, `js-yaml@4.1.1`.
- Modernized tar stack (`tar@7`, `minizlib@3`, `chownr@3`, `yallist@5`),
aligning with Node 18+.

<sup>Written for commit 9bd8ff99768a8b4dce134163162535599892a4c6.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-14 10:21:42 -07:00
Sean McGuire
c55536f785
[chore]: bump more deps (#2114)
# why

this updates a handful of deps we were behind on & avoids breaking docs

# what changed

bumped the following deps:

- `@langchain/openai` to `^1.4.5`
- `@langchain/core` to `^1.1.46`
- `@anthropic-ai/claude-agent-sdk` to `^0.2.141`
- `@fastify/swagger-ui` to `^5.2.6` in `packages/server-v3` &
`packages/server-v4`
- `uuid` to `^11.1.1`
- `vite` to `^7.3.3` in the packages that use `vitest`
- workspace tooling like `@changesets/cli`, `c8`, `eslint`, and `tsup`

added the following `pnpm` overrides:

- `axios`
- `follow-redirects`
- `jws`
- `lodash`
- `minimatch`
- `uuid`

also fixed a `vitest` mock typing issue in
`packages/evals/tests/tui/commandTree.test.ts` that resulted from the
vite bump

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Update workspace dependencies and add `pnpm` overrides to lock
transitive versions, stabilizing builds and preventing docs from
breaking. Also fixes a Vitest typing issue introduced by the Vite
upgrade.

- **Dependencies**
- Bumped: `@langchain/openai` → `^1.4.5`, `@langchain/core` → `^1.1.46`,
`@anthropic-ai/claude-agent-sdk` → `^0.2.141`, `uuid` → `^11.1.1`,
`@fastify/swagger-ui` → `^5.2.6` (server v3/v4).
- Tooling: `vite` → `^7.3.3`, `vitest` → `^4.1.6`, `tsup` → `^8.5.1`,
plus `@changesets/cli`, `c8`, `eslint`.
- Docs: `mintlify` → `^4.2.546` and move OpenAPI path to `api.openapi`
in `packages/docs/docs.json`.
- Added `pnpm.overrides` for `axios`, `follow-redirects`, `jws`,
`lodash`, `minimatch`, `uuid`.

- **Bug Fixes**
- Fixed `vitest` mock typing in
`packages/evals/tests/tui/commandTree.test.ts` by typing handlers
(`CommandHandler`) and `printHelp` mocks.

<sup>Written for commit cd57bdf55f36ffe05bfa0de3e43aa2619d159546.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-13 20:31:36 -07:00
Miguel
342681fbaf
Updated readme for evals package (#2112)
# why

# what changed

# test plan

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Revamped the `packages/evals` README to explain the current Stagehand
Evals TUI/CLI: quickstart, commands, run targets, common flags
(harness/agent modes), preview/progress/results, task scaffolding, and
Braintrust tracing. Added screenshots plus a run demo GIF, clarified
`.env` loading, and removed outdated notes to match auto-discovered
tasks and the `evals` binary.

<sup>Written for commit 0f80088278a6d0c37285ee75a2f9f67973d61814.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-13 17:56:03 -07:00
Miguel
bda6e60f6b
[evals] Onboarding flow (#2103)
# why
Improving the evals onboarding experience

# what changed
Added a Welcome screen on first run after install on TUI and a net new
`evals doctor` subcommand to triage common issues with
installation/setup
<img width="732" height="814" alt="Screenshot 2026-05-11 at 12 08 41 PM"
src="https://github.com/user-attachments/assets/c7ae377b-3219-4b42-b8c6-e70182b76709"
/>


# test plan
- [x] 500+ LOC changes of new tests
- [x] Existing test suite passess


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds a first-run onboarding flow to the `evals` REPL and a new `evals
doctor` (alias `health`) command for environment and setup diagnostics.
Integrates with the command-tree dispatcher, adds a REPL-only quiet
mode, and persists a `_meta` first-run marker so the welcome shows once.

- **New Features**
- REPL onboarding: one-time welcome; suppress with `--quiet`/`-q`
(REPL-only) or `EVALS_NO_WELCOME=1`. Banner shows art only. When no
welcome, print a single inline warning only if no provider keys;
otherwise a compact tip.
- First-run state: writes `_meta.firstRunCompletedAt` (with version) to
`evals.config.json`; preserved across CLI rebuilds. REPL marks
pre-prompt unless `--quiet`. Argv marks only after real handlers run.
Help/meta paths and `doctor` (including nested help) do not mark.
- `evals doctor`/`health`: human output and `--json` (always exits 0);
verdicts `ok|warn|fail` with exit codes `0|0|1`. Hidden `--probe`
verifies an OpenAI key. Report includes runtime (Node, Stagehand, mode),
config (path/defaults/core), task discovery, and a key matrix
(OpenAI/Anthropic/Google/Browserbase/Braintrust) with source provenance
and Browserbase alias hints. Listed in help and callable from the REPL.

- **Migration**
- Run `evals doctor` to verify setup. Set
`OPENAI_API_KEY`/`ANTHROPIC_API_KEY`/`GOOGLE_GENERATIVE_AI_API_KEY` (or
`GEMINI_API_KEY`) and `BRAINTRUST_API_KEY`; for `env=browserbase`, set
`BROWSERBASE_API_KEY` and `BROWSERBASE_PROJECT_ID` (or `BB_*`).
- To skip onboarding on launch, use `--quiet` or set
`EVALS_NO_WELCOME=1`.

<sup>Written for commit ed61696358943a09540249b947cf287e45f3e698.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 16:16:10 -07:00
Miguel
8fc16d2e18
[fix] Anthropic CUA triple_click mapping (#2104) (#2107)
# Why

`AnthropicCUAClient.convertToolUseToAction()` handled actions like
`click`, `double_click`, and `left_click`, but did not explicitly
support `triple_click`.

Two issues:

1. The action type remained in snake_case (`triple_click`) instead of
being normalized to `tripleClick` — the format expected by
`v3CuaAgentHandler.executeAction()`.
2. Coordinates provided via Anthropic’s coordinate: [x, y] format were
not converted into the separate x / y fields required by the handler.

As a result, Anthropic triple-click actions were silently ignored by the
handler switch.

# What Changed

- AnthropicCUAClient.ts
- Added explicit handling for triple_click / tripleClick in
convertToolUseToAction()
  - Mirrors the existing double_click implementation
  - Normalizes action type to tripleClick
  - Extracts coordinates from both:
    - coordinate: [x, y]
    - direct x / y fields

- v3CuaAgentHandler.ts
  - Added "triple_click" as a switch-case alias in executeAction()

# Test Plan

Added `anthropic-cua-triple-click.test.ts` (Vitest) covering:

- triple_click with coordinate: [x, y]
- triple_click with direct x / y fields

All existing unit tests continue to pass.

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes Anthropic CUA `triple_click` so it runs as `tripleClick` with x/y
parsed from `coordinate: [x, y]` or explicit fields. Also adds a CI job
to run evals unit tests.

- **Bug Fixes**
- Normalize `triple_click`/`tripleClick` to `tripleClick` in
`AnthropicCUAClient.convertToolUseToAction()`.
  - Parse `x`/`y` from `coordinate: [x, y]` or direct fields.
  - Add `"triple_click"` alias in `v3CuaAgentHandler.executeAction()`.

- **Dependencies**
  - Changeset to publish a patch for `@browserbasehq/stagehand`.

<sup>Written for commit a230bfcd987b1ec229956f338ac8caa4b58c0a2f.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Chromie <miguel@browserbase.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-05-12 12:26:20 -07:00
Sean McGuire
ae32f73f9b
[docs]: add ignoreSelectors to docs for extract() (#2088)
# what changed
- adds new param to reference section
- adds example to "targeted extract" section


<img width="723" height="974" alt="Screenshot 2026-05-06 at 10 36 25 AM"
src="https://github.com/user-attachments/assets/35d676c6-48f0-46c7-8b76-cf319fa7af5e"
/>


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Documented the new `ignoreSelectors` option for `extract()` to exclude
elements (and their descendants) from the snapshot. Added a
targeted-extract example and clarified how it interacts with `selector`,
plus minor wording updates for supported page types.

<sup>Written for commit f039e4503888986e418a3fb4688e11af6551698b.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-12 11:34:17 -07:00
Sean McGuire
82bf5d4a7d
[docs]: add docs for ignoreSelectors (#2097)
# why
- to add docs for new `ignoreSelectors` param in `observe()`
# what changed
- adds new param to reference section 
- adds an example snippet in 'the basics' section

<img width="761" height="792" alt="Screenshot 2026-05-07 at 11 41 26 AM"
src="https://github.com/user-attachments/assets/8e6c636a-a9ed-4c56-ad28-9030c532d995"
/>

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Added docs for the new `ignoreSelectors` option in `observe()` to let
users exclude page regions from observation. Updated the reference with
type and behavior, plus a basics example using XPath and a note on how
it interacts with `selector`.

<sup>Written for commit 35f9577986ff7ec134fbe8c3927f605d39128eb6.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-12 11:33:38 -07:00
Miguel
406db90295
Evals TUI tree traversal (#2100)
# why

# what changed

# test plan

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Unifies the Evals TUI/CLI behind a shared command tree with path-style
traversal and contextual help. Supports Linear STG-1880 and fixes root
fallback parsing by stripping a leading `evals` sigil to improve
onboarding and command discovery.

- **New Features**
  - Single `buildCommandTree` powers both REPL and argv dispatch.
- `>` works as a path separator in REPL and argv; REPL preserves quoted
`>`, argv re-splits any arg containing `>`.
- Context navigation and prompt: `..` goes up; bare `evals` jumps to
root; invoking a node with children auto-enters that context in REPL;
prompt shows the current path; banner mentions `..`.
- Help routing: `--help` after a leaf calls that node’s help; `help`
shows help for the current context.
- Root fallback: unknown tokens at root call `run` and now strip a
leading `evals` before forwarding (e.g., `evals act`); unknown at depth
errors for clarity.
- REPL quality of life: Esc pops context when idle; Esc still aborts
in-flight runs (cooperative/double-press aggressive).

- **Refactors**
- `cli.ts` and `repl.ts` now resolve and execute via `dispatch` from the
shared command tree; added `tokenizeArgv` for argv `>` splitting and
`renderPrompt` for context-aware prompts.
- Moved line tokenization to `tui/tokenize.ts`; added
`tui/commandTree.ts` with resolution, prompt rendering, and argv
tokenization.
- Kept command handlers lazy-loaded; added focused tests for the tree,
`>` chaining, argv sigil handling, and CLI edge cases.

<sup>Written for commit 2f5c929685407e78762a0938b58f989e831aab6d.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Chromie <miguel@browserbase.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-05-11 16:43:59 -07:00
Sean McGuire
ffb9b200ce
[chore]: bump mcp sdk & fastify (#2106) 2026-05-11 14:44:08 -07:00
github-actions[bot]
9b07e99420
Version Packages (#2067)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and the packages will
be published to npm automatically. If you're not ready to do a release
yet, that's fine, whenever you add more changesets to main, this PR will
be updated.


# Releases
## @browserbasehq/stagehand@3.4.0

### Minor Changes

- [#2084](https://github.com/browserbase/stagehand/pull/2084)
[`0641d44`](0641d44a84)
Thanks [@seanmcguire12](https://github.com/seanmcguire12)! - add
ignoreSelectors param to extract()

- [#2096](https://github.com/browserbase/stagehand/pull/2096)
[`a11603d`](a11603d09d)
Thanks [@seanmcguire12](https://github.com/seanmcguire12)! - add
ignoreSelectors to observe()

### Patch Changes

- [#2080](https://github.com/browserbase/stagehand/pull/2080)
[`21c78b3`](21c78b3a50)
Thanks [@miguelg719](https://github.com/miguelg719)! - Add variables
support to v3 agentExecute API schema and remove experimental
requirement

- [#2077](https://github.com/browserbase/stagehand/pull/2077)
[`f437f73`](f437f738d2)
Thanks [@monadoid](https://github.com/monadoid)! - Fix frame registry
handling for OOPIF pages

- [#2098](https://github.com/browserbase/stagehand/pull/2098)
[`a783b99`](a783b99fb9)
Thanks [@seanmcguire12](https://github.com/seanmcguire12)! - bump
transitive deps to patched versions

- [#2089](https://github.com/browserbase/stagehand/pull/2089)
[`8d2f354`](8d2f354142)
Thanks [@shrey150](https://github.com/shrey150)! - Strengthen observe
prompts so LLMs return complete encoded element IDs.

- [#2047](https://github.com/browserbase/stagehand/pull/2047)
[`a87c1fc`](a87c1fc435)
Thanks [@tkattkat](https://github.com/tkattkat)! - Set default agent
mode to hybrid with auto routing to dom for non compatible models

- [#2101](https://github.com/browserbase/stagehand/pull/2101)
[`26e6c96`](26e6c960ca)
Thanks [@seanmcguire12](https://github.com/seanmcguire12)! - move
playwright-core, puppeteer-core, patchright-core from optional
dependencies to peer dependencies

- [#2068](https://github.com/browserbase/stagehand/pull/2068)
[`1d176c4`](1d176c466e)
Thanks [@filip-michalsky](https://github.com/filip-michalsky)! - Remove
the default temperature setting from v3 agent AI SDK calls so reasoning
models that do not support temperature run without provider warnings.

- [#2040](https://github.com/browserbase/stagehand/pull/2040)
[`1fa9613`](1fa96130ab)
Thanks [@monadoid](https://github.com/monadoid)! - Prefer
`STAGEHAND_API_URL` for Stagehand API overrides while retaining
`STAGEHAND_BASE_URL` as a deprecated fallback.

- [#2065](https://github.com/browserbase/stagehand/pull/2065)
[`9ff70dd`](9ff70dd26c)
Thanks [@miguelg719](https://github.com/miguelg719)! - Add support for
CUA models: openai/gpt-5.4-mini, openai/gpt-5.5, and
anthropic/claude-haiku-4-5

- [#2039](https://github.com/browserbase/stagehand/pull/2039)
[`7640381`](76403819b8)
Thanks [@monadoid](https://github.com/monadoid)! - Deprecate Browserbase
project ID configuration.

## @browserbasehq/stagehand-evals@2.0.1

### Patch Changes

- Updated dependencies
\[[`21c78b3`](21c78b3a50),
[`0641d44`](0641d44a84),
[`f437f73`](f437f738d2),
[`a783b99`](a783b99fb9),
[`8d2f354`](8d2f354142),
[`a11603d`](a11603d09d),
[`a87c1fc`](a87c1fc435),
[`26e6c96`](26e6c960ca),
[`1d176c4`](1d176c466e),
[`1fa9613`](1fa96130ab),
[`9ff70dd`](9ff70dd26c),
[`7640381`](76403819b8)]:
    -   @browserbasehq/stagehand@3.4.0

## @browserbasehq/stagehand-server-v3@3.6.4

### Patch Changes

- [#2098](https://github.com/browserbase/stagehand/pull/2098)
[`a783b99`](a783b99fb9)
Thanks [@seanmcguire12](https://github.com/seanmcguire12)! - bump
transitive deps to patched versions

- [#2091](https://github.com/browserbase/stagehand/pull/2091)
[`de132bc`](de132bc6f2)
Thanks [@pirate](https://github.com/pirate)! - Default the v3 server
listener to localhost, respect explicit HOST values, and warn when
HOST=0.0.0.0 exposes all interfaces.

- [#2040](https://github.com/browserbase/stagehand/pull/2040)
[`1fa9613`](1fa96130ab)
Thanks [@monadoid](https://github.com/monadoid)! - Prefer
`STAGEHAND_API_URL` for Stagehand API overrides while retaining
`STAGEHAND_BASE_URL` as a deprecated fallback.

- [#2094](https://github.com/browserbase/stagehand/pull/2094)
[`4a808c4`](4a808c4378)
Thanks [@monadoid](https://github.com/monadoid)! - Make the model API
key optional in the v3 OpenAPI security requirements.

- [#2039](https://github.com/browserbase/stagehand/pull/2039)
[`7640381`](76403819b8)
Thanks [@monadoid](https://github.com/monadoid)! - Deprecate Browserbase
project ID configuration.

- Updated dependencies
\[[`21c78b3`](21c78b3a50),
[`0641d44`](0641d44a84),
[`f437f73`](f437f738d2),
[`a783b99`](a783b99fb9),
[`8d2f354`](8d2f354142),
[`a11603d`](a11603d09d),
[`a87c1fc`](a87c1fc435),
[`26e6c96`](26e6c960ca),
[`1d176c4`](1d176c466e),
[`1fa9613`](1fa96130ab),
[`9ff70dd`](9ff70dd26c),
[`7640381`](76403819b8)]:
    -   @browserbasehq/stagehand@3.4.0

## @browserbasehq/stagehand-server-v4@3.6.4

### Patch Changes

- [#2098](https://github.com/browserbase/stagehand/pull/2098)
[`a783b99`](a783b99fb9)
Thanks [@seanmcguire12](https://github.com/seanmcguire12)! - bump
transitive deps to patched versions

- Updated dependencies
\[[`21c78b3`](21c78b3a50),
[`0641d44`](0641d44a84),
[`f437f73`](f437f738d2),
[`a783b99`](a783b99fb9),
[`8d2f354`](8d2f354142),
[`a11603d`](a11603d09d),
[`a87c1fc`](a87c1fc435),
[`26e6c96`](26e6c960ca),
[`1d176c4`](1d176c466e),
[`1fa9613`](1fa96130ab),
[`9ff70dd`](9ff70dd26c),
[`7640381`](76403819b8)]:
    -   @browserbasehq/stagehand@3.4.0

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@browserbasehq/stagehand@3.4.0
2026-05-11 14:17:14 -07:00
Sean McGuire
6c78ecd23c
[chore]: rm evals changeset (#2108)
# why
- changsets are not properly set up in evals workspace, & this package
is not yet published on npm
# what changed
- removed patch changeset for evals pkg


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Removed the patch changeset for `@browserbasehq/stagehand-evals` to
prevent an accidental release. The evals workspace isn’t configured for
Changesets and the package isn’t published to npm.

<sup>Written for commit 86a520040ad50a933ea18d2defe0a2f8c17c9ba9.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
stagehand-server-v3/v3.6.9
2026-05-11 14:13:01 -07:00
Sean McGuire
26e6c960ca
[chore]: move integration libs into peer deps (#2101)
# why
- pnpm installs optional deps by default, which meant installing
`stagehand` would also pull in `playwright-core`, `patchright-core`, &
`puppeteer-core`
- these packages should be supplied by the consumer **_only if_** they
want to use `stagehand` with an existing browser automation framework.
they are otherwise not required
- moving them into peer dependencies prevents unnecessary installs, &
reflects a more accurate dependency graph

# what changed
- moved `playwright-core`, `patchright-core`, & `puppeteer-core` in
`packages/core` from optional deps to optional peer deps


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Moved `playwright-core`, `patchright-core`, and `puppeteer-core` to
optional peer dependencies in `@browserbasehq/stagehand` to stop
automatic installs and let consumers opt in only if needed. Also aligned
`playwright` version ranges across packages.

- **Dependencies**
- In `packages/core`: moved `playwright-core`, `patchright-core`,
`puppeteer-core` from optional deps to optional peer deps (all marked
optional).
- Updated internal dev deps to keep tests working: `playwright`,
`playwright-core`, `puppeteer-core`, `patchright-core`.
- Aligned `playwright` ranges: `packages/evals` uses `>=1.55.1 <1.57.0`;
`packages/server-v3` and `packages/server-v4` use `^1.55.1`.

- **Migration**
  - If you use Stagehand with Playwright, add `playwright-core@^1.55.1`.
  - If you use Puppeteer, add `puppeteer-core@^24.43.0`.
  - If you use Patchright, add `patchright-core@^1.55.2`.

<sup>Written for commit dc726fc12933305106cb7936eb6a18310cfb7c53.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-11 10:00:45 -07:00
Sean McGuire
8e5a4606e8
[docs]: rm lockfile from docs package (#2099)
# why
- lockfile was unused, leftover from monorepo reorg 
# what changed
- removed `packages/docs/pnpm-lock.yaml`


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Removed the unused docs lockfile left over from the monorepo reorg.
Deleting `packages/docs/pnpm-lock.yaml` reduces noise and avoids
confusion about dependency state.

<sup>Written for commit d8a5c2b1da751e3edf2c2271382bb7984278e364.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-08 16:22:24 -07:00
Miguel
84bfc8426f
Evals man help (#2092)
# Why

Help discovery was inconsistent across the evals CLI.

- `--help` / `-h` worked everywhere.
- The bare word `help` only worked at the top level and on three
dispatcher commands.
- `config core` had no dedicated help page at all — `evals config core
--help` was intercepted upstream and printed the parent `config` help
instead.

# What Changed

- Added `printConfigCoreHelp` documenting:
  - `evals config core`
  - `path`
  - `set`
  - `reset`
  - `setup`

- Standardized `help` as a positional alias for `--help` at every level:
  - Top-level dispatchers:
    - `run`
    - `list`
    - `new`
    - `config`
    - `experiments`
  - Nested dispatchers:
    - `config core`
  - Leaf verbs:
    - `set`
    - `reset`
    - `path`
    - `setup`
    - `list`
    - `show`
    - `open`
    - `compare`

- Removed the early `wantsHelp → printConfigHelp()` interception in:
  - `cli.ts`
  - `repl.ts`

This allows `config core --help` to correctly reach the core-specific
help page.

- Cleaned up:
  - unused imports
  - dead experiments help branch

- Help interception now only occurs when `help` appears as the first
argument to a verb, so legitimate values such as `evals config set
provider help` continue to work properly.

# Tests

- [x] Added 35 parameterized `it.each` cases in `tests/cli.test.ts`
covering:
  - `help`
  - `--help`
  - `-h`
across every command and subcommand level.

- [x] Full evals unit suite:
  - `251 / 251` passing

- [x] Verified built CLI prints correct help for:
  - `config core help`
  - `experiments compare help`
  - `run help`
2026-05-08 16:00:56 -07:00
Sean McGuire
a783b99fb9
[chore]: bump various transitive deps across monorepo (#2098)
# why
- to bump a small set of direct and transitive dependencies to pick up
patched upstream releases
# what changed
- bumped `playwright` to 1.55.1 across workspace all workspace packages
- bumped `@playwright/test` in `packages/core` to 1.55.1
- bumped `playwright-core` in `packages/core` to 1.55.1
- bumped `puppeteer-core` in `packages/core` to 24.43.0
- bumped `jsdom` in `packages/core` to 29.1.1.
- bumped `@browserbasehq/sdk` where needed to pick up a patched
transitive dependency tree
# test plan
- existing test suite + pnpm audit

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Bumps key direct and transitive dependencies across the monorepo to pick
up patched upstream releases and align browser tooling. Ships patch
releases for Stagehand packages.

- **Dependencies**
- `packages/core`: `playwright` → `^1.59.1`, `playwright-core` →
`^1.59.1`, `@playwright/test` → `^1.59.1`, `puppeteer-core` →
`^24.43.0`, `jsdom` → `^29.1.1`, `openai` → `^4.104.0`,
`@langchain/openai` → `^0.4.9`.
- `packages/server-v3` and `packages/server-v4`: `playwright` →
`1.55.1`.
- `packages/evals`: `openai` → `^4.104.0`, `playwright` → `^1.55.1`,
`braintrust` → `^0.4.10`, added `@types/node` and `types: ["node"]` in
`tsconfig`.
  - Updated `pnpm-lock.yaml`.
- Patch releases: `@browserbasehq/stagehand`,
`@browserbasehq/stagehand-evals`, `@browserbasehq/stagehand-server-v3`,
`@browserbasehq/stagehand-server-v4`.

- **Migration**
  - Reinstall deps: `pnpm install`.
  - Reinstall Playwright browsers if needed: `npx playwright install`.

<sup>Written for commit db96e496ead3c081d74484784763aa86b6d096a9.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
stagehand-server-v3/v3.6.8
2026-05-08 13:37:14 -07:00
Nick Sweeting
de132bc6f2
Default SEA binary server-v3 to only bind to localhost (#2091)
## Summary
- default the v3 server listener to localhost unless HOST is explicitly
set
- warn when HOST=0.0.0.0 opts into all-interface binding
- make the v4 SEA build choose a Node binary that contains the SEA fuse

## Validation
- pnpm --filter @browserbasehq/stagehand-server-v3 run build:esm-tests
- pnpm --filter @browserbasehq/stagehand-server-v3 run typecheck
- pnpm --filter @browserbasehq/stagehand-server-v4 run typecheck
- pnpm --filter @browserbasehq/stagehand-server-v3 run test:unit --
packages/server-v3/dist/tests/unit/listenHost.test.js
- pnpm --filter @browserbasehq/stagehand-server-v4 run build:sea:esm --
--binary-name=stagehand-server-v4-localhost-binding-test

Leaving the full suite to CI.

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Default `@browserbasehq/stagehand-server-v3` to listen on localhost
unless `HOST` is set, and warn on `HOST=0.0.0.0` to prevent accidental
exposure. Also make the `@browserbasehq/stagehand-server-v4` SEA build
use a Node binary that includes the SEA fuse for reliable builds.

- New Features
- Default listen host to `localhost` when `HOST` is unset or blank;
respect explicit values.
  - Warn when `HOST=0.0.0.0` to make all-interface binding explicit.

- Migration
- If you need external access, set `HOST=0.0.0.0` (or a specific
interface).

<sup>Written for commit 3cecdce8251f43aa3853ca56f8924eb0dda9dff7.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
stagehand-server-v3/v3.6.7
2026-05-07 14:06:34 -07:00
Sean McGuire
a11603d09d
[feat]: add ignoreSelectors to observe() (#2096)
# why
- to let users exclude parts of the page from the snapshot that
`observe()` sends to the model, the same way they can with `extract()`

# what changed
- this PR is just wiring the added param through observe & updating
types. it uses the same underlying functionality that was added in #2084
when adding `ignoreSelectors` to `extract()`
- `packages/core/lib/v3/types/public/methods.ts`,
`packages/core/lib/v3/types/private/handlers.ts`, &
`packages/core/lib/v3/types/public/api.ts`:
- added the `ignoreSelectors` option to the public api, internal handler
params, & request schema
- `packages/core/lib/v3/v3.ts` &
`packages/core/lib/v3/handlers/observeHandler.ts`:
- passed `ignoreSelectors` through the observe path into
`captureHybridSnapshot()`

# test plan
- updated `packages/core/tests/unit/public-api/public-types.test.ts` to
cover the new `ignoreSelectors` param
- updated `packages/core/tests/unit/api-variables-schema.test.ts` &
`packages/core/tests/unit/api-client-observe-variables.test.ts` to cover
request validation and serialization for `observe({ ignoreSelectors })`
- updated `packages/core/tests/unit/timeout-handlers.test.ts` to cover
`ObserveHandler` passing `ignoreSelectors` into
`captureHybridSnapshot()`

### notes
- the test file placement is a little weird right now: 
- `timeout-handlers.test.ts` already contains broader parameter
threading tests, so this was the smallest change for now
  - this can be cleaned up in a later refactor


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Add `ignoreSelectors` to `observe()` so you can exclude elements from
the snapshot sent to the model. Brings `observe()` to parity with
`extract()`.

- **New Features**
- Support `observe({ ignoreSelectors: ["nav", ".cookie-banner"] })` to
skip elements and subtrees during snapshot capture.
- Plumbed through public types, internal handler params, and
`captureHybridSnapshot()` call path.
- Request schema and `packages/server-v3/openapi.v3.yaml` updated;
client serialization and validation covered by tests.
  - Published as a minor for `@browserbasehq/stagehand`.

<sup>Written for commit c7b538c04d439b13d28db843a12344529171219b.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-07 11:24:56 -07:00
Sam F
4a808c4378
STG-1806 Make model API key optional in OpenAPI (#2094)
## Summary
- remove ModelApiKey from the server-v3 OpenAPI global security
requirement
- update the checked-in v3 OpenAPI spec so Stainless no longer treats
model API key as required client auth

## Verification
- pnpm --filter @browserbasehq/stagehand-server-v3 run lint
- pnpm --filter @browserbasehq/stagehand-server-v3 run build

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Make the model API key optional by removing `ModelApiKey` from server v3
OpenAPI global security (STG-1806). Update the spec/generator in
`@browserbasehq/stagehand-server-v3` and add a patch changeset so
Stainless clients no longer require a model key.

- **Migration**
- Regenerate clients from the v3 OpenAPI (e.g., Stainless) to drop
`ModelApiKey` as required auth.

<sup>Written for commit 3c74893febe6a3501f26eaa700ba2290c02cad56.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: samfinton <samfinton@samLaptopm5.galaxus.box>
stagehand-server-v3/v3.6.6
2026-05-07 20:22:09 +02:00
Sean McGuire
86d3ec070e
[chore]: commit updated spec (#2095)
# why
- schema descriptions changed which means spec yaml files need to be
regenerated
- the regenerated yaml files were uncommitted
# what changed
- added the regenerated yaml files


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Regenerated OpenAPI specs for `packages/server-v3` and
`packages/server-v4` to align with schema description updates. In v4,
the `x-bb-project-id` header and related `projectId` fields are now
deprecated since API keys are project-scoped.

- **Migration**
  - Stop sending the `x-bb-project-id` header.
  - Omit `projectId` in v4 session payloads and ignore it in responses.

<sup>Written for commit 8dd382e1833337e8ea52021f8ca66f9284e01061.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-07 10:44:46 -07:00
Shrey Pandya
8d2f354142
[STG-1935] fix: strengthen observe element ID prompting (#2089)
Linear:
https://linear.app/browserbase/issue/STG-1935/strengthen-observe-element-id-prompting

## Summary

- keep the strict `number-number` observe element ID schema intact
- strengthen the observe system prompt to tell the model to copy the
full bracketed tree ID, including frame ordinal and hyphen
- strengthen the schema description with the concrete `[0-18372] ->
"0-18372", not "18372"` example
- add prompt regression coverage, main-frame `0-...` observe integration
coverage, and a patch changeset
- add `observe/observe_main_frame_element_ids` as a regression eval for
multiple main-frame targets

## Eval Coverage

Current bench eval count after this PR:

```text
146 total bench evals
12 observe bench evals
```

This PR adds one observe regression eval:
`observe/observe_main_frame_element_ids`.

## Validation

- `pnpm --filter @browserbasehq/stagehand run build`
- `pnpm --filter @browserbasehq/stagehand run test:core --
packages/core/dist/esm/tests/unit/prompt-observe-variables.test.js`
- `pnpm --filter @browserbasehq/stagehand run test:e2e --
packages/core/dist/esm/tests/integration/observe-element-id-format.spec.js`
- `pnpm --filter @browserbasehq/stagehand run lint`
- `pnpm --filter @browserbasehq/stagehand-evals run typecheck`
- `pnpm --filter @browserbasehq/stagehand-evals run test:unit --
tests/taskConfig.test.ts`
- `pnpm --filter @browserbasehq/stagehand-evals run lint`
- `env STAGEHAND_E2E_TARGET=BROWSERBASE pnpm exec tsx
/Users/shrey/Developer/scratchpad/stagehand-observe-bare-id-e2e/index.ts`
- `pnpm --filter @browserbasehq/stagehand-evals exec tsx cli.ts run
observe_main_frame_element_ids -e browserbase -t 1 -c 1 -m
anthropic/claude-sonnet-4-6`

Real Browserbase E2E used `anthropic/claude-sonnet-4-6` and verified raw
model IDs plus click success:

```text
Attempt 1 raw model elementId: 0-5011
Attempt 1 act success: true
Attempt 1 DOM clicked marker: yes
Attempt 2 raw model elementId: 0-5011
Attempt 2 act success: true
Attempt 2 DOM clicked marker: yes
Attempt 3 raw model elementId: 0-5011
Attempt 3 act success: true
Attempt 3 DOM clicked marker: yes
```

The new Browserbase bench eval also passed with
`anthropic/claude-sonnet-4-6`:

```text
observe/observe_main_frame_element_ids  anthropic/claude-sonnet-4-6  passed
Results: 1 passed, 0 failed (1 total)
Pass rate: 100%
```
2026-05-07 08:49:55 -07:00
Filip Michalsky
c4da2e2286
remove default temp (#2076)
# why

Stagehand v3 was setting hard-coded temperature defaults for standard
act, extract, and observe inference calls. This forced provider
  behavior instead of letting model/provider defaults apply.

  # what changed

Removed the hard-coded temperature: isGPT5 ? 1 : 0.1 defaults from
shared act, extract, and observe inference.

  Removed the unused GPT-5 temperature branching in that path.

Updated Groq and Cerebras clients to pass through options.temperature
directly instead of defaulting missing values to 0.7, while preserving
  explicit temperature: 0.

  Added regression tests covering:

  - act, extract, and observe no longer pass temperature by default
  - Groq/Cerebras do not inject default temperature
  - Groq/Cerebras preserve explicit zero temperature

  # test plan

  - pnpm --filter @browserbasehq/stagehand run typecheck
- pnpm --filter @browserbasehq/stagehand exec prettier --check
lib/inference.ts lib/v3/llm/GroqClient.ts lib/v3/llm/CerebrasClient.ts
tests/
unit/inference-temperature.test.ts
tests/unit/openai-compatible-temperature.test.ts
  - pnpm --filter @browserbasehq/stagehand run build:esm
- pnpm --filter @browserbasehq/stagehand run test:core --
packages/core/dist/esm/tests/unit/inference-temperature.test.js
packages/core/
dist/esm/tests/unit/openai-compatible-temperature.test.js
packages/core/dist/esm/tests/unit/aisdk-clients.test.js
packages/core/dist/esm/
    tests/unit/anthropic-cua-adaptive-thinking.test.js

---------

Co-authored-by: Filip Michalsky <filip-michalsky@users.noreply.github.com>
Co-authored-by: Miguel <36487034+miguelg719@users.noreply.github.com>
Co-authored-by: Chromie <miguel@browserbase.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-05-06 17:17:16 -07:00
Miguel
21c78b3a50
remove experimental requirement on agent variables (#2079) (#2080)
# why
`experimental: true` is no longer required for agent variable support

# what changed
- Removed the experimental gate on agent variables: dropped the
variables check in `validateExperimentalFeatures.ts` and the
@experimental JSDoc on `AgentExecuteOptionsBase.variables`.
- Added a variables field to the AgentExecuteOptions API schema
(`packages/core/lib/v3/types/public/api.ts`) and reflected it in the
generated `openapi.v3.yaml` (both request and output components).
- Updated docs to drop `experimental: true` from the Variables Note and
Basic Usage example (kept the "Non-CUA agents only" caveat).
- Added unit tests covering the new surface
- Added integration coverage on the login eval site: new variables
describe blocks in `act.test.ts` and `agentExecute.test.ts` (simple and
rich { value, description } forms, verified via CDP), and updated
`observe.test.ts`'s existing variables tests to use the same scenario.

# test plan
- [x] pnpm --filter @browserbasehq/stagehand-server-v3 typecheck —
passes.
- [x] pnpm exec prettier --check on the three test files — passes.
- [x] pnpm exec eslint on the three test files — passes. 

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Enable variables in v3 `agentExecute` without experimental mode, with
end-to-end substitution across `act`, `observe`, and agent tools.
Tightened `fillFormVision` so secrets are only substituted at type-time
and never leak via logs, replays, or tool outputs.

- **New Features**
- Added `variables` to `AgentExecuteOptionsSchema` and server OpenAPI
(`Variables`).
- Removed experimental gating for variables; CUA-mode rejection remains.
  - Updated JSDoc and docs to drop the experimental requirement.
- Added tests: unit (validation, client serialization, schema) and
integration for substitution in `act`, `observe`, and `agentExecute`
(simple and rich forms).

- **Bug Fixes**
- Updated `fillFormVision` to substitute variables only when typing and
keep placeholders in action logs, replays, and returned arguments.

<sup>Written for commit a6e9d621d396cab397bd5348c814629373693cf0.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Chromie <miguel@browserbase.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: tkattkat <tkat@tkat.net>
2026-05-06 11:51:41 -07:00
Sam F
76403819b8
[STG-1808] Deprecate Browserbase project ID (#2039)
# why
Browserbase API keys are project-scoped, so Stagehand should keep
accepting project IDs without requiring them.

# what changed
- Mark `browserbaseSessionCreateParams.projectId` deprecated in
Zod/OpenAPI.
- Make `BROWSERBASE_PROJECT_ID` nullable/deprecated in Stainless config.
- Use Browserbase-returned `projectId` in the local server instead of
`x-bb-project-id`.

```ts
projectId: z.string().optional().meta({ deprecated: true })
```

# test plan
- `pnpm --filter @browserbasehq/stagehand run lint`
- `pnpm --filter @browserbasehq/stagehand-server-v3 run lint`

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Deprecate Browserbase project ID per STG-1808; Stagehand now relies on
project-scoped API keys and the project ID returned by Browserbase. The
`x-bb-project-id` header, session `projectId` param, and
`BROWSERBASE_PROJECT_ID` config are deprecated but remain optional for
now.

- **Refactors**
- Marked `projectId` in session create params as deprecated in
Zod/OpenAPI; updated security scheme descriptions to reflect deprecation
of `x-bb-project-id`.
- Made `BROWSERBASE_PROJECT_ID` nullable and marked deprecated in
`stainless.yml`.
- Server now resolves `projectId` from the retrieved/created Browserbase
session; stops reading `x-bb-project-id` and adjusts validation
messages.

- **Migration**
- Stop sending `x-bb-project-id` and remove `BROWSERBASE_PROJECT_ID`;
they’re no longer needed.
- Avoid passing `projectId` in session creation; it’s accepted but
deprecated.

<sup>Written for commit 5ee90b860105739405927eec695060a1d1a066fc.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2039">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
stagehand-server-v3/v3.6.5
2026-05-06 19:11:05 +02:00
Miguel
886884bd70
Manual eval dispatch workflow (#2085)
# why

# what changed

# test plan


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds a manual GitHub Actions workflow to run evaluation suites on demand
with selectable model, agent mode, trials, and concurrency. Builds once,
runs chosen evals in a matrix, and posts Braintrust results and
summaries.

- **New Features**
- Run agent, act/extract/observe, or external agent benchmarks
(webvoyager, onlineMind2Web, webtailbench).
- Inputs: `model`, `agent_mode` (auto/dom/hybrid/cua), `trials` (default
1), `concurrency` (default 50).
- One build job; artifacts reused across eval jobs. Dynamic matrix built
via `jq`.
- Executes evals with `pnpm` + `turbo` against
`@browserbasehq/stagehand-evals`, with
OpenAI/Anthropic/Google/Braintrust/Browserbase env wired and region
selection.
- Publishes job summary with Braintrust link and primary score, and
uploads CTRF report + V8 coverage (fails if no summary).

<sup>Written for commit 8267815433cfbbac5988d1f6b827be62ea43bade.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-06 10:07:50 -07:00
Sean McGuire
0641d44a84
[feat]: add ignoreSelectors to extract() (#2084)
# why
- to allow users to specify selectors whose content should be omitted
from the LLM request inside of `extract()`
# what changed
- added `ignoreSelectors` to `extract()` so callers can exclude parts of
the page from the extraction snapshot
- `packages/core/lib/v3/understudy/a11y/snapshot/capture.ts`:
- added `resolveIgnoreSelectorRoots()` to resolve ignore selectors into
`frameId + backendNodeId`
- added `buildFrameExclusionIntervals()` to turn those roots into
per-frame subtree ranges
- updated `collectPerFrameMaps()` / `tryScopedSnapshot()` to skip
ignored nodes when building snapshot data
- `packages/core/lib/v3/understudy/a11y/snapshot/domTree.ts`:
- updated `buildSessionDomIndex()` to record dfs entry/exit positions
for each backend node so ignored subtrees can be filtered by interval
membership instead of having to re-traverse the tree
- `packages/core/lib/v3/understudy/a11y/snapshot/a11yTree.ts`: 
- updated `a11yForFrame()` to accept an ignore predicate and remove
ignored nodes from the outline (the stringified tree) and `urlMap`
- `packages/core/lib/v3/handlers/extractHandler.ts` and
`packages/core/lib/v3/v3.ts`:
- passed `ignoreSelectors` down through the extract path into
`captureHybridSnapshot()`
- kept `xpathMap` values based on the original dom structure and only
skipped ignored entries, so surviving selectors do not get renumbered

# test plan
- updated `packages/core/tests/unit/public-api/public-types.test.ts` to
cover the new `ignoreSelectors` option on `extract()`
- updated `packages/core/tests/unit/snapshot-a11y-resolvers.test.ts` to
cover ignored backend nodes being removed from the a11y outline and
`urlMap`, and to cover scoped snapshots dropping ignored xpath entries
- updated
`packages/core/tests/unit/snapshot-capture-orchestration.test.ts` to
cover the new capture flow, including filtered per-frame maps and merged
snapshot output when `ignoreSelectors` is used

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds `ignoreSelectors` to `extract()` so callers can exclude elements
and their subtrees from snapshots and the LLM input. Supports CSS and
XPath (with iframe hops) across shadow DOM and iframes, without
renumbering existing XPaths; also adds a fast path for scoped snapshots
and excludes child frame subtrees when an iframe host is ignored.

- **New Features**
- `extract({ ignoreSelectors: string[] })`: drops matching nodes and
descendants from DOM maps, a11y outline, and `urlMap`; `xpathMap` keeps
original values but omits ignored entries.
- Applies to full-page and scoped snapshots; filters resolved per frame,
excludes child frames when their iframe host is ignored, and enforces
DFS enter/exit intervals for speed.
- A11y tree accepts an ignore predicate and filters before building the
outline and links.
- Scoped snapshots return early when only `selector` is provided (no
`ignoreSelectors`) for faster capture.
- Plumbed through `extractHandler`, `v3`, and `captureHybridSnapshot()`;
public types and OpenAPI updated in `@browserbasehq/stagehand`; unit
tests added.

<sup>Written for commit 0ad2f67d8fe81d15945599c60351ac45f66f1277.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-06 10:05:31 -07:00
Filip Michalsky
1d176c466e
remove hardcoded agent temp (#2068)
# why                                                                
Reasoning models (e.g. `openai/gpt-5-mini`) emit `AI SDK Warning: The
"temperature" setting is not supported by this model` whenever the v3
agent
runs against them, because `V3AgentHandler` hardcoded `temperature: 1`
on every `generateText`/`streamText` call. There was no constructor or
agent-config option to override it, so users running reasoning models
with `experimental: true` had no way to silence the warning.
  # what changed                                                       
- Removed the hardcoded `temperature: 1` from both the non-streaming and
streaming agent paths in
`packages/core/lib/v3/handlers/v3AgentHandler.ts`.
- Agent calls now omit `temperature` entirely, so the AI SDK no longer
flags it as user-set on models that don't accept it. Models that do
accept
temperature still default to the AI SDK provider's own default.
- Added unit coverage in
`packages/core/tests/unit/agent-temperature.test.ts` asserting neither
code path passes a `temperature` to the LLM
client.
- Added a patch changeset
(`.changeset/remove-agent-temperature-default.md`).
  # test plan
- [ ] `pnpm --filter @browserbasehq/stagehand test agent-temperature` —
new unit tests pass
- [ ] Existing v3 agent unit suite stays green
- [ ] Manual: `agent.execute(...)` against `openai/gpt-5-mini` with
`experimental: true` — confirm no `"temperature" setting is not
supported`
warning in stderr
- [ ] Manual: same call against a non-reasoning model (e.g.
`openai/gpt-4.1-mini`) — confirm agent still completes a multi-step task
end-to-end
- [ ] Manual: hybrid-mode agent run on a reasoning model — verify
streaming path is also clean

---------

Co-authored-by: Filip Michalsky <filip-michalsky@users.noreply.github.com>
2026-05-06 06:09:32 -07:00
Sam F
f437f738d2
fix: preserve OOPIF frame registry parent edge (#2077)
# why

Stagehand could lose a known parent-child frame relationship after
adopting an OOPIF child session. In practice, that meant the registry
could forget that a child frame still belonged under the top-level page,
leaving later iframe resolution with an incomplete frame tree.

# what changed

Added a deterministic FrameRegistry unit test for the OOPIF adoption
ordering. Updated `seedFromFrameTree()` to preserve an already-known
parent when a child frame tree snapshot does not include parent
metadata.

# test plan

Before the fix, the new test failed with `expected null to be
'root-frame'`. After the fix, this passes:

`pnpm --dir packages/core run build:esm && pnpm --dir packages/core run
test:core --
packages/core/dist/esm/tests/unit/frame-registry-oopif-adoption.test.js`

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes a bug where the frame registry could lose a known parent-child
link after adopting an OOPIF child session. Preserves the parent edge
when seeding from a child frame tree that lacks parent metadata to keep
the iframe tree complete.

- Bug Fixes
- Updated `FrameRegistry.seedFromFrameTree()` to retain an existing
parent when `parentId` is missing and to sync child sets accordingly.
- Added a unit test for OOPIF adoption ordering and seeding to prevent
regressions, and a changeset for `@browserbasehq/stagehand` patch
release.

<sup>Written for commit 1e50e9c3ed4d56c158c1b694bbd04d2aac55821c.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: samfinton <samfinton@samLaptopm5.galaxus.box>
2026-05-05 18:12:02 +02:00
Sam F
1fa96130ab
[STG-1808] Use STAGEHAND_API_URL for Stagehand API client (#2040)
# why
Stagehand should standardize public docs and runtime overrides on
`STAGEHAND_API_URL` without breaking users who still export
`STAGEHAND_BASE_URL`.

# what changed
- Prefer `STAGEHAND_API_URL`, with `STAGEHAND_BASE_URL` as a deprecated
legacy fallback.
- Emit a one-time deprecation warning when the legacy fallback is used.
- Update server-v3 test env plumbing, CI, and docs to use
`STAGEHAND_API_URL`.
- Add patch changesets for core and server-v3.

```ts
const configuredBaseUrl =
  process.env.STAGEHAND_API_URL ?? process.env.STAGEHAND_BASE_URL;
```

# test plan
- `pnpm --filter @browserbasehq/stagehand run build:esm`
- `pnpm --filter @browserbasehq/stagehand run test:core --
packages/core/dist/esm/tests/unit/api-optional-model-api-key.test.js`
- `pnpm --filter @browserbasehq/stagehand run typecheck`
- `pnpm --filter @browserbasehq/stagehand-server-v3 run typecheck`
- `pnpm --filter @browserbasehq/stagehand run lint`
- `pnpm --filter @browserbasehq/stagehand-server-v3 run lint`
- `pnpm exec prettier --check .changeset/rude-tips-check.md
.github/workflows/ci.yml packages/docs/v3/sdk/java.mdx
packages/docs/v3/sdk/python.mdx turbo.json`
- `pnpm changeset status --since origin/main --verbose`
stagehand-server-v3/v3.6.4
2026-05-04 10:32:27 -07:00
Miguel
ff1f1c3651
Remove old trace (#2075)
# why

# what changed

# test plan


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Removed an accidental development transcript file from the repo to
reduce noise and avoid shipping non-code artifacts. No runtime or build
impact.

<sup>Written for commit 25c03785632e6ca0781b6d6dc8a2fbb58ea9daac.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-05-02 21:43:38 -07:00
Miguel
2eb9416470
Evals v2 (#2011)
# why
### **Evals CLI v2.0**
<img width="813" height="450" alt="Screenshot 2026-04-28 at 3 34 44 PM"
src="https://github.com/user-attachments/assets/69ef8dd4-b717-4548-9798-b1639febe753"
/>

# what changed

You can find this diagram in `ARCHITECTURE.md`
```mermaid
flowchart TB
    subgraph Entrypoints["Entrypoints and Distribution"]
        CLI["cli.ts<br/>single-shot argv"]
        REPL["tui/repl.ts<br/>interactive shell<br/>Esc abort support"]
        BuildCLI["scripts/build-cli.ts<br/>bundled dist/cli bin"]
        BuildESM["scripts/build-esm.ts<br/>dist/esm + assets + datasets"]
        LegacyCLI["cli-legacy.ts / index.eval.ts<br/>--legacy escape hatch"]
    end

    subgraph TUI["TUI Command Layer"]
        Help["help<br/>usage text"]
        List["list<br/>registry views"]
        ConfigCmd["config<br/>defaults + core tool setup"]
        NewTask["new<br/>task scaffold"]
        Experiments["experiments<br/>Braintrust inspect/compare"]
        RunCmd["run<br/>progress + dry-run plan"]
        Parse["parse.ts<br/>flag validation<br/>target shorthand<br/>env override scope"]
        ConfigFile[("evals.config.json<br/>defaults, core config,<br/>benchmark metadata")]
    end

    subgraph Registry["Task Registry and Target Resolution"]
        RuntimePaths["runtimePaths.ts<br/>source vs dist roots"]
        Discovery["framework/discovery.ts<br/>filesystem scan<br/>category overrides<br/>lazy/eager metadata"]
        Resolver["resolveTarget()<br/>tier, category, task,<br/>partial name"]
        CoreTasks[("core/tasks/*<br/>deterministic tool tasks")]
        BenchTasks[("tasks/bench/*<br/>act, extract, observe,<br/>agent, combination, suites")]
        DefineTask["framework/defineTask.ts<br/>defineCoreTask<br/>defineBenchTask"]
    end

    subgraph Planning["Run Planning"]
        ResolvedOptions["ResolvedRunOptions<br/>env, trials, concurrency,<br/>model, provider, api,<br/>harness, agent mode"]
        BenchPlanner["framework/benchPlanner.ts<br/>model matrix<br/>suite expansion<br/>metadata/tags"]
        TaskConfig["taskConfig.ts<br/>default models<br/>agent model entries"]
        Datasets[("datasets/*<br/>WebVoyager<br/>OnlineMind2Web<br/>WebTailBench")]
        DryRun["dry-run JSON<br/>tasks, skippedTasks,<br/>envOverrides, runOptions"]
    end

    subgraph Runner["Unified Runner"]
        RunEvals["framework/runner.ts<br/>generate testcases<br/>Braintrust Eval wrapper<br/>progress callbacks"]
        BraintrustAdapter["framework/braintrust.ts<br/>lazy import<br/>traced spans"]
        Cleanup["activeRunCleanup.ts<br/>signal cleanup<br/>registered session cleanup"]
        Scoring["scoring.ts<br/>passRate<br/>exactMatch<br/>errorMatch"]
        Summary["summary.ts<br/>local run summary"]
    end

    subgraph CoreTier["Core Tier Execution"]
        CoreContext["framework/context.ts<br/>buildCoreContext"]
        FixtureServer["core/fixtures<br/>local deterministic pages"]
        CoreTargets["core/targets<br/>local Chrome<br/>Browserbase CDP"]
        CoreTools["core/tools registry<br/>understudy_code<br/>playwright_code<br/>cdp_code<br/>MCP + browse_cli"]
        CoreAssertions["assertions + metrics<br/>adapter-backed results"]
    end

    subgraph BenchTier["Bench Tier Execution"]
        BenchRunner["framework/benchRunner.ts<br/>load task module<br/>execute bench task"]
        TaskLoader["framework/taskLoader.ts<br/>defineBenchTask + legacy<br/>module loader"]
        HarnessRegistry["framework/benchHarness.ts<br/>stagehand + claude_code<br/>(codex slot pending)"]
        InitV3["initV3.ts<br/>V3 session + agent"]
        AISDK["AISdkClientWrapped<br/>provider/model client"]
        ClaudeCodeRunner["framework/claudeCodeRunner.ts<br/>Claude Code SDK adapter<br/>prompt + result parse"]
        ClaudeCodeToolAdapter["framework/claudeCodeToolAdapter.ts<br/>tool-surface handoff<br/>browse_cli wrapper"]
        ExternalHarnessPlan["framework/externalHarnessPlan.ts<br/>webvoyager / onlineMind2Web /<br/>webtailbench task plan"]
        BenchSuites["suites/*<br/>dataset-backed suite cases"]
        BenchTaskDefs["bench task modules<br/>defineBenchTask or legacy fn"]
    end

    subgraph Reporting["Reporting and Inspection"]
        Progress["tui/progress.ts<br/>live task table"]
        ResultsView["tui/results.ts<br/>model summary<br/>results table"]
        ReportLib["lib/braintrust-report.ts<br/>experiment fetch<br/>comparison data"]
        RenderReport["scripts/render-braintrust-core-report.ts<br/>core report rendering"]
    end

    subgraph External["External Systems"]
        Braintrust["Braintrust<br/>experiments, spans,<br/>scores, comparisons"]
        Browserbase["Browserbase<br/>remote browser sessions"]
        LocalBrowser["Local Chromium<br/>fixture-backed sessions"]
        Providers["LLM Providers<br/>OpenAI, Anthropic,<br/>Google, others"]
        Stagehand["Stagehand package<br/>V3, AgentProvider,<br/>available models"]
        ClaudeCodeSDK["@anthropic-ai/claude-agent-sdk<br/>external Claude Code agent<br/>headless run"]
    end

    %% Entrypoints
    CLI --> Parse
    CLI --> RunCmd
    CLI --> List
    CLI --> ConfigCmd
    CLI --> Experiments
    CLI --> NewTask
    CLI --> Help
    REPL --> Parse
    REPL --> RunCmd
    BuildCLI --> CLI
    BuildESM --> RuntimePaths
    LegacyCLI --> LegacyRunner["legacy runner path<br/>pre-refactor behavior"]

    %% TUI and config
    ConfigCmd <--> ConfigFile
    Parse --> ConfigFile
    Parse --> ResolvedOptions
    Parse --> DryRun
    RunCmd --> ResolvedOptions
    RunCmd --> RuntimePaths
    RunCmd --> Discovery
    List --> Discovery
    NewTask --> CoreTasks
    NewTask --> BenchTasks
    Experiments --> ReportLib

    %% Registry
    RuntimePaths --> CoreTasks
    RuntimePaths --> BenchTasks
    CoreTasks --> Discovery
    BenchTasks --> Discovery
    DefineTask --> CoreTasks
    DefineTask --> BenchTasks
    Discovery --> Resolver
    Resolver --> RunCmd

    %% Planning
    RunCmd --> BenchPlanner
    RunCmd --> DryRun
    ResolvedOptions --> BenchPlanner
    TaskConfig --> BenchPlanner
    Datasets --> BenchSuites
    BenchSuites --> BenchPlanner
    BenchPlanner --> RunEvals
    Resolver --> RunEvals

    %% Runner
    RunCmd --> RunEvals
    RunEvals --> BraintrustAdapter
    RunEvals --> Cleanup
    RunEvals --> Scoring
    RunEvals --> Summary
    RunEvals --> CoreContext
    RunEvals --> BenchRunner
    RunEvals --> Progress
    Progress --> ResultsView

    %% Core path
    CoreContext --> FixtureServer
    CoreContext --> CoreTargets
    CoreContext --> CoreTools
    CoreContext --> CoreAssertions
    CoreTargets --> LocalBrowser
    CoreTargets --> Browserbase
    CoreTools --> LocalBrowser
    CoreTools --> Browserbase

    %% Bench path
    BenchRunner --> HarnessRegistry
    BenchRunner --> TaskLoader
    TaskLoader --> BenchTaskDefs
    HarnessRegistry --> InitV3
    InitV3 --> Stagehand
    InitV3 --> Browserbase
    InitV3 --> LocalBrowser
    HarnessRegistry --> AISDK
    AISDK --> Providers
    BenchRunner --> BenchTaskDefs
    BenchTaskDefs --> Stagehand
    BenchSuites --> Datasets

    %% Claude Code external harness path
    HarnessRegistry --> ClaudeCodeRunner
    HarnessRegistry --> ClaudeCodeToolAdapter
    HarnessRegistry --> ExternalHarnessPlan
    ClaudeCodeToolAdapter --> CoreTools
    ClaudeCodeToolAdapter --> CoreTargets
    ClaudeCodeRunner --> ClaudeCodeSDK
    ClaudeCodeSDK --> Providers

    %% Reporting and external
    BraintrustAdapter --> Braintrust
    ReportLib --> Braintrust
    RenderReport --> ReportLib
    Summary --> Braintrust
    Stagehand --> Providers

    %% Styling
    style Entrypoints fill:#263238,stroke:#111111,stroke-width:3px,color:#ffffff
    style TUI fill:#0d47a1,stroke:#082d69,stroke-width:3px,color:#ffffff
    style Registry fill:#1b5e20,stroke:#0d3010,stroke-width:3px,color:#ffffff
    style Planning fill:#6a1b9a,stroke:#4a148c,stroke-width:3px,color:#ffffff
    style Runner fill:#004d40,stroke:#00251f,stroke-width:3px,color:#ffffff
    style CoreTier fill:#bf360c,stroke:#7f2207,stroke-width:3px,color:#ffffff
    style BenchTier fill:#1565c0,stroke:#0d47a1,stroke-width:3px,color:#ffffff
    style Reporting fill:#4e342e,stroke:#2d1b17,stroke-width:3px,color:#ffffff
    style External fill:#7f0000,stroke:#4d0000,stroke-width:3px,color:#ffffff

    style CLI fill:#eceff1,stroke:#263238,stroke-width:2px,color:#000000
    style REPL fill:#eceff1,stroke:#263238,stroke-width:2px,color:#000000
    style BuildCLI fill:#cfd8dc,stroke:#263238,stroke-width:2px,color:#000000
    style BuildESM fill:#cfd8dc,stroke:#263238,stroke-width:2px,color:#000000
    style LegacyCLI fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px,color:#000000
    style LegacyRunner fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px,color:#000000

    style Parse fill:#bbdefb,stroke:#0d47a1,stroke-width:2px,color:#000000
    style RunCmd fill:#bbdefb,stroke:#0d47a1,stroke-width:2px,color:#000000
    style ConfigFile fill:#e3f2fd,stroke:#0d47a1,stroke-width:2px,color:#000000
    style Discovery fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px,color:#000000
    style Resolver fill:#a5d6a7,stroke:#1b5e20,stroke-width:2px,color:#000000
    style CoreTasks fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px,color:#000000
    style BenchTasks fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px,color:#000000

    style BenchPlanner fill:#e1bee7,stroke:#6a1b9a,stroke-width:2px,color:#000000
    style Datasets fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px,color:#000000
    style RunEvals fill:#b2dfdb,stroke:#004d40,stroke-width:2px,color:#000000
    style BraintrustAdapter fill:#e0f2f1,stroke:#004d40,stroke-width:2px,color:#000000
    style Cleanup fill:#e0f2f1,stroke:#004d40,stroke-width:2px,color:#000000

    style CoreContext fill:#ffe0b2,stroke:#bf360c,stroke-width:2px,color:#000000
    style CoreTargets fill:#ffcc80,stroke:#bf360c,stroke-width:2px,color:#000000
    style CoreTools fill:#ffcc80,stroke:#bf360c,stroke-width:2px,color:#000000
    style BenchRunner fill:#bbdefb,stroke:#1565c0,stroke-width:2px,color:#000000
    style HarnessRegistry fill:#90caf9,stroke:#1565c0,stroke-width:2px,color:#000000
    style InitV3 fill:#90caf9,stroke:#1565c0,stroke-width:2px,color:#000000
    style TaskLoader fill:#bbdefb,stroke:#1565c0,stroke-width:2px,color:#000000
    style ClaudeCodeRunner fill:#90caf9,stroke:#1565c0,stroke-width:2px,color:#000000
    style ClaudeCodeToolAdapter fill:#90caf9,stroke:#1565c0,stroke-width:2px,color:#000000
    style ExternalHarnessPlan fill:#bbdefb,stroke:#1565c0,stroke-width:2px,color:#000000

    style Braintrust fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000
    style Browserbase fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000
    style Providers fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000
    style Stagehand fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000
    style ClaudeCodeSDK fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000

```

# test plan


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Evals v2 rebuilds the suite into a core + bench system with
auto-discovered tasks, adapter-backed tools, a unified Braintrust
runner, and a stable CLI/TUI. It now stabilizes the `playwright_code`
and `cdp_code` tool surfaces, adds `browse_cli` and MCP adapters, ships
a Braintrust comparison report, fixes `simple_google_search`, guards
file-upload evals with cleanup-on-failure, and removes outdated planning
docs.

- **New Features**
- Core: deterministic tasks under `packages/evals/core/tasks/**`, shared
contracts/assertions/metrics, local fixture server, lazy `getEnv()`,
runner-provided Chrome targets (local, Browserbase) with
artifacts/cleanup and a shared cleanup registry.
- Tools: stable `playwright_code` and `cdp_code`; plus
`understudy_code`, `playwright_mcp`, `chrome_devtools_mcp`,
`browse_cli`.
- Bench: matrix planner and external-suite support (WebVoyager,
OnlineMind2Web, WebTailBench); harness registry with `stagehand` and
`claude_code`, Claude Code SDK boundary, and a `browse_cli` tool
handoff; bench runner propagates `sessionUrl`/`debugUrl`.
- CLI/TUI: stable `dist/cli` (`pnpm evals ...`) with `bin/evals`,
`experiments` command, startup warning suppression, quieter logs via
`EvalLogger` echo toggle; legacy CLI via `pnpm evals:old`.
- Reporting: Braintrust data layer (`lib/braintrust-report.ts`),
`report:core` script, and summaries that include the experiment URL and
scores.
- CI/Deps/Docs: defaults updated (`EVAL_AGENT_MODELS=
openai/gpt-5.4-mini, anthropic/claude-haiku-4-5,
google/gemini-3-flash-preview`; `EVAL_CATEGORIES` includes
`regression`); `.gitignore` ignores Playwright/MCP artifacts; version
bumped to `2.0.0`; add `playwright`, `@anthropic-ai/claude-agent-sdk`,
and `vitest`; updated README, architecture diagram, and `skills/browser`
guide.

- **Migration**
- Tasks are auto-discovered; remove task lists from `evals.config.json`.
- Bench tasks must default-export `defineBenchTask({ name })` under
`packages/evals/tasks/bench/**`; core tasks live under
`packages/evals/core/tasks/**` via `defineCoreTask`.
- Use the new CLI (`pnpm evals ...` or the `evals` bin); the legacy
interface remains via `pnpm evals:old`.
- Set `BROWSERBASE_API_KEY` (and optionally `BROWSERBASE_PROJECT_ID`);
set `CHROME_PATH` if auto-detect fails; `EVAL_ENV` is chosen at runtime.
- Filters/categories changed; remove `webbench` references — use
`b:webvoyager` or `b:onlineMind2Web`.

<sup>Written for commit 5a0056d5aba49219bd80c3b65032164ce0ca6611.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2011?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-05-01 15:32:24 -07:00
tkattkat
a87c1fc435
default agent to hybrid mode (#2047)
# why

Dom mode has been deprecated for a while, and is now being replaced with
hybrid as a default

# what changed

- When a hybrid capable model is provided, it is used with hybrid mode
by default
- When a non hybrid capable model is provided, it is used with dom mode 
- if dom or hybrid is explicitly provided, it will use the provided mode

# test plan
tested locally + wrote tests 

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Set the default v3 agent mode to hybrid, with automatic routing to dom
for models that aren’t hybrid-capable. Explicit mode settings are always
respected; CUA behavior is unchanged.

- **New Features**
- Auto-selects "hybrid" for models matching
`HYBRID_CAPABLE_MODEL_PATTERNS` (`gemini-3`, `claude`, `gpt-5.4`,
`gpt-5.5`); otherwise uses "dom".
- Routes using the provided agent model (string or config), or falls
back to the Stagehand-level model when undefined.
- Honors explicit `mode` (`dom`, `hybrid`, `cua`); if `mode` is unset,
deprecated `cua: true` enables CUA.
- Logs the resolved mode and warns when "hybrid" is used with
non-hybrid-capable models.
  - Adds unit tests for routing and handler fallbacks.

<sup>Written for commit 01e7d69945369cca735dff9ba088538420e344c9.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2047">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-04-30 14:40:25 -07:00
Miguel
9ff70dd26c
[chore] update supported CUA models (#2064) (#2065)
## What Changed

Adds three new models to the list of supported CUA (Computer Use Agent)
models:

- `openai/gpt-5.4-mini`
- `openai/gpt-5.5`
- `anthropic/claude-haiku-4-5`

---

## Files Changed

### `packages/core/lib/v3/types/public/agent.ts`

- Added the three models to the `AVAILABLE_CUA_MODELS` const array  
- This drives:
  - The `AvailableCuaModel` type
  - The runtime CUA model check in `resolveModel()`

### `packages/core/lib/v3/agent/AgentProvider.ts`

- Added model → provider mappings in `modelToAgentProviderMap`:
  - `gpt-5.4-mini` → `openai`
  - `gpt-5.5` → `openai`
  - `claude-haiku-4-5` → `anthropic`
- Ensures `AgentProvider.getAgentProvider()` resolves them correctly

### `packages/core/tests/unit/public-api/llm-and-agents.test.ts`

- Updated the `expectedModels` list in the type-level test to stay in
sync with the source of truth

---

## How It Was Tested

- ✅ Build passes (`pnpm turbo run build`)
- ✅ All **204 public-api unit tests** pass
- ✅ All **608 unit tests** pass
- ⚠️ 4 pre-existing failures in `flowlogger-eventstore.test.js`
(confirmed on base branch)

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds support for three new Computer Use Agent models and maps them to
the right providers. Apps can now use `openai/gpt-5.4-mini`,
`openai/gpt-5.5`, and `anthropic/claude-haiku-4-5` via the public API.

- **New Features**
- Registered the models in `AVAILABLE_CUA_MODELS` and mapped providers:
`gpt-5.4-mini` → `openai`, `gpt-5.5` → `openai`, `claude-haiku-4-5` →
`anthropic`.
  - Updated public API tests to include the new models.
- Added a changeset to release these additions as a patch in
`@browserbasehq/stagehand`.

<sup>Written for commit ec5b691fdf9e469e11b4d025a11a7ac32dbba0c5.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2065?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Chromie <miguel@browserbase.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-04-29 13:20:27 -07:00