17 Commits

Author SHA1 Message Date
Sam F
1fa96130ab
[STG-1808] Use STAGEHAND_API_URL for Stagehand API client (#2040)
# why
Stagehand should standardize public docs and runtime overrides on
`STAGEHAND_API_URL` without breaking users who still export
`STAGEHAND_BASE_URL`.

# what changed
- Prefer `STAGEHAND_API_URL`, with `STAGEHAND_BASE_URL` as a deprecated
legacy fallback.
- Emit a one-time deprecation warning when the legacy fallback is used.
- Update server-v3 test env plumbing, CI, and docs to use
`STAGEHAND_API_URL`.
- Add patch changesets for core and server-v3.

```ts
const configuredBaseUrl =
  process.env.STAGEHAND_API_URL ?? process.env.STAGEHAND_BASE_URL;
```

# test plan
- `pnpm --filter @browserbasehq/stagehand run build:esm`
- `pnpm --filter @browserbasehq/stagehand run test:core --
packages/core/dist/esm/tests/unit/api-optional-model-api-key.test.js`
- `pnpm --filter @browserbasehq/stagehand run typecheck`
- `pnpm --filter @browserbasehq/stagehand-server-v3 run typecheck`
- `pnpm --filter @browserbasehq/stagehand run lint`
- `pnpm --filter @browserbasehq/stagehand-server-v3 run lint`
- `pnpm exec prettier --check .changeset/rude-tips-check.md
.github/workflows/ci.yml packages/docs/v3/sdk/java.mdx
packages/docs/v3/sdk/python.mdx turbo.json`
- `pnpm changeset status --since origin/main --verbose`
2026-05-04 10:32:27 -07:00
Miguel
2eb9416470
Evals v2 (#2011)
# why
### **Evals CLI v2.0**
<img width="813" height="450" alt="Screenshot 2026-04-28 at 3 34 44 PM"
src="https://github.com/user-attachments/assets/69ef8dd4-b717-4548-9798-b1639febe753"
/>

# what changed

You can find this diagram in `ARCHITECTURE.md`
```mermaid
flowchart TB
    subgraph Entrypoints["Entrypoints and Distribution"]
        CLI["cli.ts<br/>single-shot argv"]
        REPL["tui/repl.ts<br/>interactive shell<br/>Esc abort support"]
        BuildCLI["scripts/build-cli.ts<br/>bundled dist/cli bin"]
        BuildESM["scripts/build-esm.ts<br/>dist/esm + assets + datasets"]
        LegacyCLI["cli-legacy.ts / index.eval.ts<br/>--legacy escape hatch"]
    end

    subgraph TUI["TUI Command Layer"]
        Help["help<br/>usage text"]
        List["list<br/>registry views"]
        ConfigCmd["config<br/>defaults + core tool setup"]
        NewTask["new<br/>task scaffold"]
        Experiments["experiments<br/>Braintrust inspect/compare"]
        RunCmd["run<br/>progress + dry-run plan"]
        Parse["parse.ts<br/>flag validation<br/>target shorthand<br/>env override scope"]
        ConfigFile[("evals.config.json<br/>defaults, core config,<br/>benchmark metadata")]
    end

    subgraph Registry["Task Registry and Target Resolution"]
        RuntimePaths["runtimePaths.ts<br/>source vs dist roots"]
        Discovery["framework/discovery.ts<br/>filesystem scan<br/>category overrides<br/>lazy/eager metadata"]
        Resolver["resolveTarget()<br/>tier, category, task,<br/>partial name"]
        CoreTasks[("core/tasks/*<br/>deterministic tool tasks")]
        BenchTasks[("tasks/bench/*<br/>act, extract, observe,<br/>agent, combination, suites")]
        DefineTask["framework/defineTask.ts<br/>defineCoreTask<br/>defineBenchTask"]
    end

    subgraph Planning["Run Planning"]
        ResolvedOptions["ResolvedRunOptions<br/>env, trials, concurrency,<br/>model, provider, api,<br/>harness, agent mode"]
        BenchPlanner["framework/benchPlanner.ts<br/>model matrix<br/>suite expansion<br/>metadata/tags"]
        TaskConfig["taskConfig.ts<br/>default models<br/>agent model entries"]
        Datasets[("datasets/*<br/>WebVoyager<br/>OnlineMind2Web<br/>WebTailBench")]
        DryRun["dry-run JSON<br/>tasks, skippedTasks,<br/>envOverrides, runOptions"]
    end

    subgraph Runner["Unified Runner"]
        RunEvals["framework/runner.ts<br/>generate testcases<br/>Braintrust Eval wrapper<br/>progress callbacks"]
        BraintrustAdapter["framework/braintrust.ts<br/>lazy import<br/>traced spans"]
        Cleanup["activeRunCleanup.ts<br/>signal cleanup<br/>registered session cleanup"]
        Scoring["scoring.ts<br/>passRate<br/>exactMatch<br/>errorMatch"]
        Summary["summary.ts<br/>local run summary"]
    end

    subgraph CoreTier["Core Tier Execution"]
        CoreContext["framework/context.ts<br/>buildCoreContext"]
        FixtureServer["core/fixtures<br/>local deterministic pages"]
        CoreTargets["core/targets<br/>local Chrome<br/>Browserbase CDP"]
        CoreTools["core/tools registry<br/>understudy_code<br/>playwright_code<br/>cdp_code<br/>MCP + browse_cli"]
        CoreAssertions["assertions + metrics<br/>adapter-backed results"]
    end

    subgraph BenchTier["Bench Tier Execution"]
        BenchRunner["framework/benchRunner.ts<br/>load task module<br/>execute bench task"]
        TaskLoader["framework/taskLoader.ts<br/>defineBenchTask + legacy<br/>module loader"]
        HarnessRegistry["framework/benchHarness.ts<br/>stagehand + claude_code<br/>(codex slot pending)"]
        InitV3["initV3.ts<br/>V3 session + agent"]
        AISDK["AISdkClientWrapped<br/>provider/model client"]
        ClaudeCodeRunner["framework/claudeCodeRunner.ts<br/>Claude Code SDK adapter<br/>prompt + result parse"]
        ClaudeCodeToolAdapter["framework/claudeCodeToolAdapter.ts<br/>tool-surface handoff<br/>browse_cli wrapper"]
        ExternalHarnessPlan["framework/externalHarnessPlan.ts<br/>webvoyager / onlineMind2Web /<br/>webtailbench task plan"]
        BenchSuites["suites/*<br/>dataset-backed suite cases"]
        BenchTaskDefs["bench task modules<br/>defineBenchTask or legacy fn"]
    end

    subgraph Reporting["Reporting and Inspection"]
        Progress["tui/progress.ts<br/>live task table"]
        ResultsView["tui/results.ts<br/>model summary<br/>results table"]
        ReportLib["lib/braintrust-report.ts<br/>experiment fetch<br/>comparison data"]
        RenderReport["scripts/render-braintrust-core-report.ts<br/>core report rendering"]
    end

    subgraph External["External Systems"]
        Braintrust["Braintrust<br/>experiments, spans,<br/>scores, comparisons"]
        Browserbase["Browserbase<br/>remote browser sessions"]
        LocalBrowser["Local Chromium<br/>fixture-backed sessions"]
        Providers["LLM Providers<br/>OpenAI, Anthropic,<br/>Google, others"]
        Stagehand["Stagehand package<br/>V3, AgentProvider,<br/>available models"]
        ClaudeCodeSDK["@anthropic-ai/claude-agent-sdk<br/>external Claude Code agent<br/>headless run"]
    end

    %% Entrypoints
    CLI --> Parse
    CLI --> RunCmd
    CLI --> List
    CLI --> ConfigCmd
    CLI --> Experiments
    CLI --> NewTask
    CLI --> Help
    REPL --> Parse
    REPL --> RunCmd
    BuildCLI --> CLI
    BuildESM --> RuntimePaths
    LegacyCLI --> LegacyRunner["legacy runner path<br/>pre-refactor behavior"]

    %% TUI and config
    ConfigCmd <--> ConfigFile
    Parse --> ConfigFile
    Parse --> ResolvedOptions
    Parse --> DryRun
    RunCmd --> ResolvedOptions
    RunCmd --> RuntimePaths
    RunCmd --> Discovery
    List --> Discovery
    NewTask --> CoreTasks
    NewTask --> BenchTasks
    Experiments --> ReportLib

    %% Registry
    RuntimePaths --> CoreTasks
    RuntimePaths --> BenchTasks
    CoreTasks --> Discovery
    BenchTasks --> Discovery
    DefineTask --> CoreTasks
    DefineTask --> BenchTasks
    Discovery --> Resolver
    Resolver --> RunCmd

    %% Planning
    RunCmd --> BenchPlanner
    RunCmd --> DryRun
    ResolvedOptions --> BenchPlanner
    TaskConfig --> BenchPlanner
    Datasets --> BenchSuites
    BenchSuites --> BenchPlanner
    BenchPlanner --> RunEvals
    Resolver --> RunEvals

    %% Runner
    RunCmd --> RunEvals
    RunEvals --> BraintrustAdapter
    RunEvals --> Cleanup
    RunEvals --> Scoring
    RunEvals --> Summary
    RunEvals --> CoreContext
    RunEvals --> BenchRunner
    RunEvals --> Progress
    Progress --> ResultsView

    %% Core path
    CoreContext --> FixtureServer
    CoreContext --> CoreTargets
    CoreContext --> CoreTools
    CoreContext --> CoreAssertions
    CoreTargets --> LocalBrowser
    CoreTargets --> Browserbase
    CoreTools --> LocalBrowser
    CoreTools --> Browserbase

    %% Bench path
    BenchRunner --> HarnessRegistry
    BenchRunner --> TaskLoader
    TaskLoader --> BenchTaskDefs
    HarnessRegistry --> InitV3
    InitV3 --> Stagehand
    InitV3 --> Browserbase
    InitV3 --> LocalBrowser
    HarnessRegistry --> AISDK
    AISDK --> Providers
    BenchRunner --> BenchTaskDefs
    BenchTaskDefs --> Stagehand
    BenchSuites --> Datasets

    %% Claude Code external harness path
    HarnessRegistry --> ClaudeCodeRunner
    HarnessRegistry --> ClaudeCodeToolAdapter
    HarnessRegistry --> ExternalHarnessPlan
    ClaudeCodeToolAdapter --> CoreTools
    ClaudeCodeToolAdapter --> CoreTargets
    ClaudeCodeRunner --> ClaudeCodeSDK
    ClaudeCodeSDK --> Providers

    %% Reporting and external
    BraintrustAdapter --> Braintrust
    ReportLib --> Braintrust
    RenderReport --> ReportLib
    Summary --> Braintrust
    Stagehand --> Providers

    %% Styling
    style Entrypoints fill:#263238,stroke:#111111,stroke-width:3px,color:#ffffff
    style TUI fill:#0d47a1,stroke:#082d69,stroke-width:3px,color:#ffffff
    style Registry fill:#1b5e20,stroke:#0d3010,stroke-width:3px,color:#ffffff
    style Planning fill:#6a1b9a,stroke:#4a148c,stroke-width:3px,color:#ffffff
    style Runner fill:#004d40,stroke:#00251f,stroke-width:3px,color:#ffffff
    style CoreTier fill:#bf360c,stroke:#7f2207,stroke-width:3px,color:#ffffff
    style BenchTier fill:#1565c0,stroke:#0d47a1,stroke-width:3px,color:#ffffff
    style Reporting fill:#4e342e,stroke:#2d1b17,stroke-width:3px,color:#ffffff
    style External fill:#7f0000,stroke:#4d0000,stroke-width:3px,color:#ffffff

    style CLI fill:#eceff1,stroke:#263238,stroke-width:2px,color:#000000
    style REPL fill:#eceff1,stroke:#263238,stroke-width:2px,color:#000000
    style BuildCLI fill:#cfd8dc,stroke:#263238,stroke-width:2px,color:#000000
    style BuildESM fill:#cfd8dc,stroke:#263238,stroke-width:2px,color:#000000
    style LegacyCLI fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px,color:#000000
    style LegacyRunner fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px,color:#000000

    style Parse fill:#bbdefb,stroke:#0d47a1,stroke-width:2px,color:#000000
    style RunCmd fill:#bbdefb,stroke:#0d47a1,stroke-width:2px,color:#000000
    style ConfigFile fill:#e3f2fd,stroke:#0d47a1,stroke-width:2px,color:#000000
    style Discovery fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px,color:#000000
    style Resolver fill:#a5d6a7,stroke:#1b5e20,stroke-width:2px,color:#000000
    style CoreTasks fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px,color:#000000
    style BenchTasks fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px,color:#000000

    style BenchPlanner fill:#e1bee7,stroke:#6a1b9a,stroke-width:2px,color:#000000
    style Datasets fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px,color:#000000
    style RunEvals fill:#b2dfdb,stroke:#004d40,stroke-width:2px,color:#000000
    style BraintrustAdapter fill:#e0f2f1,stroke:#004d40,stroke-width:2px,color:#000000
    style Cleanup fill:#e0f2f1,stroke:#004d40,stroke-width:2px,color:#000000

    style CoreContext fill:#ffe0b2,stroke:#bf360c,stroke-width:2px,color:#000000
    style CoreTargets fill:#ffcc80,stroke:#bf360c,stroke-width:2px,color:#000000
    style CoreTools fill:#ffcc80,stroke:#bf360c,stroke-width:2px,color:#000000
    style BenchRunner fill:#bbdefb,stroke:#1565c0,stroke-width:2px,color:#000000
    style HarnessRegistry fill:#90caf9,stroke:#1565c0,stroke-width:2px,color:#000000
    style InitV3 fill:#90caf9,stroke:#1565c0,stroke-width:2px,color:#000000
    style TaskLoader fill:#bbdefb,stroke:#1565c0,stroke-width:2px,color:#000000
    style ClaudeCodeRunner fill:#90caf9,stroke:#1565c0,stroke-width:2px,color:#000000
    style ClaudeCodeToolAdapter fill:#90caf9,stroke:#1565c0,stroke-width:2px,color:#000000
    style ExternalHarnessPlan fill:#bbdefb,stroke:#1565c0,stroke-width:2px,color:#000000

    style Braintrust fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000
    style Browserbase fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000
    style Providers fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000
    style Stagehand fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000
    style ClaudeCodeSDK fill:#ffcdd2,stroke:#7f0000,stroke-width:2px,color:#000000

```

# test plan


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Evals v2 rebuilds the suite into a core + bench system with
auto-discovered tasks, adapter-backed tools, a unified Braintrust
runner, and a stable CLI/TUI. It now stabilizes the `playwright_code`
and `cdp_code` tool surfaces, adds `browse_cli` and MCP adapters, ships
a Braintrust comparison report, fixes `simple_google_search`, guards
file-upload evals with cleanup-on-failure, and removes outdated planning
docs.

- **New Features**
- Core: deterministic tasks under `packages/evals/core/tasks/**`, shared
contracts/assertions/metrics, local fixture server, lazy `getEnv()`,
runner-provided Chrome targets (local, Browserbase) with
artifacts/cleanup and a shared cleanup registry.
- Tools: stable `playwright_code` and `cdp_code`; plus
`understudy_code`, `playwright_mcp`, `chrome_devtools_mcp`,
`browse_cli`.
- Bench: matrix planner and external-suite support (WebVoyager,
OnlineMind2Web, WebTailBench); harness registry with `stagehand` and
`claude_code`, Claude Code SDK boundary, and a `browse_cli` tool
handoff; bench runner propagates `sessionUrl`/`debugUrl`.
- CLI/TUI: stable `dist/cli` (`pnpm evals ...`) with `bin/evals`,
`experiments` command, startup warning suppression, quieter logs via
`EvalLogger` echo toggle; legacy CLI via `pnpm evals:old`.
- Reporting: Braintrust data layer (`lib/braintrust-report.ts`),
`report:core` script, and summaries that include the experiment URL and
scores.
- CI/Deps/Docs: defaults updated (`EVAL_AGENT_MODELS=
openai/gpt-5.4-mini, anthropic/claude-haiku-4-5,
google/gemini-3-flash-preview`; `EVAL_CATEGORIES` includes
`regression`); `.gitignore` ignores Playwright/MCP artifacts; version
bumped to `2.0.0`; add `playwright`, `@anthropic-ai/claude-agent-sdk`,
and `vitest`; updated README, architecture diagram, and `skills/browser`
guide.

- **Migration**
- Tasks are auto-discovered; remove task lists from `evals.config.json`.
- Bench tasks must default-export `defineBenchTask({ name })` under
`packages/evals/tasks/bench/**`; core tasks live under
`packages/evals/core/tasks/**` via `defineCoreTask`.
- Use the new CLI (`pnpm evals ...` or the `evals` bin); the legacy
interface remains via `pnpm evals:old`.
- Set `BROWSERBASE_API_KEY` (and optionally `BROWSERBASE_PROJECT_ID`);
set `CHROME_PATH` if auto-detect fails; `EVAL_ENV` is chosen at runtime.
- Filters/categories changed; remove `webbench` references — use
`b:webvoyager` or `b:onlineMind2Web`.

<sup>Written for commit 5a0056d5aba49219bd80c3b65032164ce0ca6611.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2011?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-05-01 15:32:24 -07:00
Sean McGuire
66f6dbf981
[chore]: simplify build for browser-side scripts (#1988)
# why

# what changed

# test plan

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Unifies the build for browser-injected DOM scripts by replacing four
generators with a single builder. Simplifies the pipeline and removes
redundant file copying, with no runtime changes expected.

- **Refactors**
- Added `scripts/build-dom-scripts.ts` to build all DOM-injected assets:
IIFE strings for `piercer.entry.ts` and
`rerenderMissingShadows.entry.ts`; bootstrap + per-function sources for
`locator` (`__stagehandLocatorScripts`) and `a11y`
(`__stagehandA11yScripts`); sources-only for `screenshot` scripts.
- Removed `genDomScripts.ts`, `genLocatorScripts.ts`,
`genScreenshotScripts.ts`, `genA11yScripts.ts`.
- Updated `packages/core/package.json` to a single `build-dom-scripts`
command and simplified `turbo.json` to one task with
`lib/v3/dom/build/**` outputs.
- Removed JS copy steps from `scripts/build-cjs.ts` and
`scripts/build-esm.ts`; generated TS emits compile with the rest of the
package.
- Updated `tsconfig.json` to include generated files (removed old
`gen*.ts` exclude).

- **Migration**
- Use `pnpm run build-dom-scripts`; the old `build-dom-scripts:*`
scripts are removed.

<sup>Written for commit ac3839794447064bb865a0b6070decb2f2518169.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1988">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-04-10 11:11:06 -07:00
Sam F
206fb14ac0
Recover server-v4 drizzle stack onto main (#1917)
## Why
The original server-v4 stacked PRs merged into each other after
`STG-1614` had already merged into `main`, so the env/test-harness work,
Drizzle foundation, and Drizzle schema/controller flow never actually
reached `main`.

This PR recovers that full stack onto current `main` in one branch.

## What Changed
- cherry-picked the three stacked squash-merge commits from `#1896`,
`#1897`, and `#1898` onto fresh `main`
- preserved the full `server-v4` env/test harness cleanup, Drizzle
foundation, DB smoke tests, schema/migration work, and `/v4/llms`
controller/service/repo flow
- added one small follow-up commit to normalize `pnpm-lock.yaml` against
current `main` after applying the recovered stack

## Test Plan
- `pnpm install`
- `pnpm --filter @browserbasehq/stagehand-server-v4 run lint`
- `pnpm --filter @browserbasehq/stagehand-server-v4 run test:unit`
- `pnpm --filter @browserbasehq/stagehand-server-v4 run build`
- `pnpm --filter @browserbasehq/stagehand-server-v4 run gen:openapi`

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Recovers the server‑v4 Drizzle stack onto `main`, restoring DB‑backed
`/v4/llms` and switching the public `LLMId` to UUID. Adds a typed env,
clean app bootstrap, and a local `pglite`/Postgres runtime with
migrations, tests, and a simpler source‑based test harness.

- **New Features**
- Drizzle foundation with `pglite` default and Postgres option; new
`db:*` scripts via `drizzle-kit`.
- Fastify `databasePlugin` and unified DB client; `app.ts` bootstrap
with health/readiness and Swagger/OpenAPI; env via `@t3-oss/env-core`.
- Tables/relations for LLMs and Stagehand; DB‑derived Zod schemas; first
migration; OpenAPI updated for UUID `LLMId` with strict format and
pattern.
- LLM module (repo/service/controller) backing `/v4/llms`; routes now
delegate to the controller; browser session create uses the service and
auto‑creates a system default LLM when `llmId` is missing.
- Tests run from source via `tsx`; test server script simplified;
`turbo` tasks cleaned up; schema cleanup (`z.url()`).

- **Migration**
  - Run: `pnpm --filter @browserbasehq/stagehand-server-v4 db:migrate`.
- Local dev uses `pglite` at `~/.stagehand/db/stagehand-v4`. For
Postgres set `STAGEHAND_DB_MODE=postgres` and `DATABASE_URL`.
- Update clients to treat `LLMId` as a UUID and regenerate from
`openapi.v4.yaml` if needed.

<sup>Written for commit 1fb861ed27739ad2330a4a0a6daf6e2c4951a575.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1917">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Nick Sweeting <github@sweeting.me>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-03-30 10:06:13 -07:00
Miguel
84473aa1ad
Revert unintentional test commit (#1850)
# why

# what changed

# test plan

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Reverts an unintended test commit by deleting
`packages/core/tests/unit/rerender-missing-shadows.test.ts`; test suite
returns to its previous state. Also updates `turbo.json` to pass through
`BROWSERBASE_FLOW_LOGS`; no production code changes.

<sup>Written for commit 49fdf8cadf812e9cc07c4037f3718b6aaa959377.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1850">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-03-18 15:46:47 -07:00
Sean McGuire
ac4539ab6b
[chore]: refactor & fix lint for browse CLI (#1821) 2026-03-12 17:44:59 -07:00
Nick Sweeting
e36ddd6f29
[STG-1536] Split packages/stagehand/server into separate server-v3 and server-v4 dirs (#1796)
# Follow-up Tasks

- [ ] Update stainless SDK custom code for all languages to pull new
`stagehand-server-v3-darwin-x64` binary names (`-v3-` added)

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Split the Stagehand API into `packages/server-v3` and
`packages/server-v4`, each with its own builds, tests, SEA binaries, and
release workflows. Delivers STG-1536 and lets us keep v3 stable while
iterating on v4; CI/test discovery and OpenAPI artifacts are versioned.

- **Refactors**
- Renamed the original server to `packages/server-v3`
(`@browserbasehq/stagehand-server-v3`); updated docs and runtime path
helpers (now synced across core/docs/evals and both servers), ESLint
globs/ignores, scripts/Turbo filters, tests, and Stainless to read
`packages/server-v3/openapi.v3.yaml`; v3 SEA binaries use
`stagehand-server-v3-*`.
- Added `packages/server-v4` (`@browserbasehq/stagehand-server-v4`) with
`/v4/**` routes, SSE streaming via `x-stream-response`, LRU/TTL
in-memory session store, health/readiness, logging/metrics,
`openapi.v4.yaml` + generator, SEA tooling, and v4 integration tests.
- CI: path filters, test discovery, and artifacts cover both versions;
added `stagehand-server-v4-release.yml` and
`stagehand-server-v4-sea-build.yml`; renamed v3 workflows; artifacts
include `packages/server-v3/**` and `packages/server-v4/**` dists and
OAS.

- **Migration**
- Replace `packages/server/**` refs with `packages/server-v3/**` or
`packages/server-v4/**`.
- Use new package filters and binary names:
`@browserbasehq/stagehand-server-v3` /
`@browserbasehq/stagehand-server-v4`; `stagehand-server-v3-*` /
`stagehand-server-v4-*`.
- Update OpenAPI consumers to `packages/server-v3/openapi.v3.yaml` or
`packages/server-v4/openapi.v4.yaml`.

<sup>Written for commit 2b9114ca8097d28ac1567fc67062b24f789a5c2e.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1796">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-03-09 17:32:57 -07:00
Nick Sweeting
4f8d2d5d85
Fix Windows SEA build (#1734)
# why

# what changed

# test plan

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes the Windows SEA build and cross‑platform path resolution
(including when run via tsx). CI builds, launches, and smoke‑tests
Linux, macOS, and Windows binaries (x64/arm64); CJS/ESM outputs are
stable.

- Bug Fixes
- Reusable SEA build workflow with a full OS/arch matrix (incl. Ubuntu
24.04 ARM and Windows 11 ARM), same‑arch launch check against /healthz,
selective artifact upload, and an include_sourcemaps toggle (dev/test
only; releases exclude sourcemaps).
- Centralized runtimePaths helpers across core/server/evals/docs to
replace import.meta/fileURLToPath, fix tsx path resolution, normalize
absolute paths, and correct supervisorClient CLI resolution and Windows
spawning.
- Tooling: split DOM script builds into four Turbo tasks for Windows;
switch to in‑process esbuild and Node‑invoked turbo/postject; stabilize
CJS/ESM via tsc emits plus small CJS wrappers; refine tsconfig includes;
add default NODE_ENV/BB_ENV in server env.

<sup>Written for commit b0b191f7b0411b6590a79ce3f10b3fba9435b4d7.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1734">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Touches build/CI and packaging code (SEA generation, path resolution)
across multiple packages, so failures could break releases or CI on
specific platforms despite being largely tooling-focused.
> 
> **Overview**
> Fixes cross-platform SEA building by moving `build-server-sea` to a
reusable workflow with a multi-OS/arch matrix (including ARM runners),
optional artifact/caching inputs, selective artifact upload, and a
same-arch smoke test that boots each binary and hits `/healthz`.
> 
> Standardizes path/root resolution via new `runtimePaths` helpers
(core/server/evals/docs) and replaces ad-hoc
`fileURLToPath(import.meta.url)`/`__dirname` usage across scripts,
tests, and `supervisorClient` (also tightening CLI resolution and SEA
detection).
> 
> Simplifies build/tooling: core CJS/ESM builds now rely on `tsc` emits
plus small CJS wrapper entrypoints (dropping esbuild bundling of
core/tests), DOM script generation is split into separate Turbo tasks,
server SEA build switches to in-process `esbuild` and Node-invoked
`turbo`/`postject`, and server env config now provides default
`NODE_ENV`/`BB_ENV` values.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
2e4a07a6d0b54eddb7fea10d079c8062ecba1d6b. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
2026-02-24 10:39:45 -08:00
Nick Sweeting
6329982da2
Fix typechecking for core/tests, fix SEA build for windows, speed up CI (#1721)
# why

# what changed

# test plan

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes TypeScript across core tests/scripts, finalizes the default
export/CLI, and makes the SEA build reliable on Windows. CI splits build
and lint, OpenAPI generation no longer runs Prettier, and builds run in
parallel across packages for faster pipelines.

- **Bug Fixes**
- Stricter TS across tests/scripts (tsconfig includes tests + CLI, added
@types/jsdom, switched to playwright-core); tightened types (child-frame
probes, DocumentReadyState, model utils, source-map Raw types); fixed
OpenAPI typing; removed cross-package imports in server tests; added
default-export parity test.
- Default export and CLI fixed: introduced StagehandDefault mirroring
named exports; ESM/CJS indices now export default; corrected
package.json CLI paths/types.
- SEA build works on Windows: build scripts use pnpm.cmd with better
spawn error handling.
- List parsing now works from raw source across core/server/evals and
maps cleanly to dist outputs.

- **Refactors**
- CI split into separate Build and Lint jobs; lint cancels the workflow
on failure; optional Turbo cache restore; discovery/runner steps no
longer depend on the build job.
- Parallelized builds: DOM script generators run concurrently; ESM/CJS
builds and server/evals build tasks run in parallel; tuned Turbo inputs
to reduce noisy rebuilds and improve caching.

<sup>Written for commit a25e17c2411b374e8d35c39f6120d242102ff8dc.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1721">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-02-20 17:09:39 -08:00
Nick Sweeting
50a6f574a3
Speed up turbo, package.json scripts, and CI configuration (#1712)
# why

# what changed

# test plan

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Speed up builds and tests by upgrading Turborepo, tightening cache keys,
and refactoring scripts to run from compiled dist (ESM/CJS). CI is
faster and more reliable, with ESM-first SEA builds for the server.

- **Refactors**
- Upgrade Turborepo to 2.8.10 and rewrite turbo.json with explicit
inputs/outputs for better caching.
- Stronger Turbo cache keys in CI (lockfiles + workspace + package.json
+ turbo.json + commit), workflow-scoped concurrency, and a 60s session
create timeout.
- Standardized CI via a composite setup action; build now runs before
lint.
- Build/test scripts resolve the repo root via fileURLToPath; tests run
from compiled dist; add Vitest config for CJS; CJS builds include
sourcemaps.
- Server: ESM-first SEA build and split server build into dist vs SEA
steps with a new build:server:dist script.
- Unified test scripts across packages; top-level test runs all suites,
and evals tests support source or dist CLI via --cli.

- **Migration**
  - Use Node 20.19+ or 22.12+ (enforced in core).
  - Run pnpm install to pick up the Turbo upgrade.
- Build before tests: pnpm -w build. For SEA tests, also run pnpm -w
--filter=@browserbasehq/stagehand-server run build:sea:esm.

<sup>Written for commit 843857c13fc4a3975c18b0e44002ae9f55e4dd66.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1712">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-02-19 15:35:37 -08:00
Nick Sweeting
a091affd2c
[chore]: add .js to the end of all imports (#1700)
# why

- removes the need to append .js to all imports during esm build process
- aligns us with the default standard for esm / esbuild js projects that
also need to support cjs
- this allows us to seamlessly support ESM, CJS, and TSX with no
remapping needed

# what changed

# test plan

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Standardized all internal imports and re-exports to include .js
extensions to enable native ESM resolution, while keeping CJS and TS
working without path remapping. Updated examples to use explicit
index.js entrypoints and environment-driven config for easier local
runs.

- **Refactors**
- Appended .js to all relative imports and barrel exports across core,
v3 agent, DOM runtimes/locator scripts, caches, handlers, external
clients, and index files.
- Switched examples to ../lib/v3/index.js, fixed nested re-export paths,
and added env-based config (BROWSERBASE_API_KEY, BROWSERBASE_PROJECT_ID,
MODEL_API_KEY) passed into Stagehand/model.
  - Updated dotenv to ^17.3.1 in dev deps. No public API changes.

- **Migration**
  - Most users importing from package entry points: no changes.
- If importing internal files locally (e.g., examples or custom
integrations), include .js in paths and prefer index.js for directories.

<sup>Written for commit d1d0547daafca47a06a8aa9124d7081c68042270.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1700">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-02-18 12:19:15 -08:00
Nick Sweeting
afbd08bb63
[STG-1232] Speed up PR Github Actions checks, add code coverage, fix e2e bb tests running not running in bb mode, remove tsup, generate unified re-used ESM and CJS builds with sourcemaps and types, switch CI to new Evals CLI (#1632)
# why

- unit tests, e2e integration tests, and evals were taking 10~15 minutes
per pr
- finding the specific failing eval or test was tedious and required
reading tons of log output
- we weren't leveraging pnpm or turbo caches or github action's native
chromium install
- tests were flaky because CDP would randomly fail to connect when
github actions runner was overloaded
- e2e bb tests were not actually running against bb, so it was
previously just double-running e2e:local and reporting it as bb
- tsup has been deprecated for a while and should be replaced by esbuild

# what changed

- parallelized all CI tests and broke out evals and tests into
individual matrix jobs
- created unified node+pnpm+turbo setup action that all other actions
call
- verified vanilla chromium is able to launch inside all the e2e local
jobs before running any of our own code to catch resource contention
flakyness
- add proper esm and csj builds with sourcemaps and types for aligned
coverage
- remove tsup dependency
- switch to new evals cli
- add coverage reporting
- fix some flaky tests needed to get everything passing reliably

<img width="1635" height="609" alt="Screenshot 2026-01-26 at 1 04 04 PM"
src="https://github.com/user-attachments/assets/24bb2ea9-e182-4cb3-9856-6a77efe921b7"
/>



<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Speeds up CI and makes tests stable by running Vitest/Playwright/Evals
on the built ESM dist with cached artifacts, verified runner Chromium,
weighted Browserbase routing, and merged coverage. Publishes a unified
ESM+CJS SDK with sourcemaps/types and SEA binaries from ESM; fixes
Browserbase e2e, switches to the new Evals CLI, and adds flaky‑test
reporting, addressing Linear STG‑1232.

- **New Features**
- CI: split pnpm/Turbo caches; composite actions (setup/caches, Chromium
launch check, region selection, CTRF/V8 uploads); reuse artifacts; use
Actions Chromium via CHROME_PATH with --no-sandbox; unique CTRF reports;
V8 coverage normalization/merge with cancellation safety; push triggers
enabled.
- Tests: run from dist/esm; env reporter; true Browserbase runs with
weighted/randomized regions; hard timeouts for LLM and session
create/CDP connect; faster multi‑click dispatch; higher timeouts and
targeted skips; cleanup helpers; fixes for init‑script,
wait‑for‑selector, and screenshot reliability.
- Build/SDK/SEA: export map (ESM import + CJS require) with
sourcemaps/types; SEA binaries built from ESM with sourcemaps and plain
logs in CI; removed tsup and dotenv; AISdk/CustomOpenAI moved to lib and
exported; getAISDKLanguageModel added; replay metrics now include
method/parameters/result/timestamps/cost.

- **Migration**
- Set STAGEHAND_BROWSER_TARGET=local|browserbase, STAGEHAND_ENV
(replaces TEST_ENV), CHROME_PATH; tune LLM_MAX_MS,
BROWSERBASE_SESSION_CREATE_MAX_MS, BROWSERBASE_CDP_CONNECT_MAX_MS;
optionally set BROWSERBASE_REGION_DISTRIBUTION.
- Remove dotenv; set env vars directly and import
@browserbasehq/stagehand via the export map (avoid dist paths).

<sup>Written for commit 5c76fecf6b9016f4ad5273d4143c3fe747b5d06f.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1632">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->







<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Ships an ESM‑first SDK with a CJS entry, unified sourcemaps, and types.
CI runs Vitest/Playwright and the new Evals CLI against the built ESM
dist using the runner’s Chromium (launch‑verified), with region‑weighted
Browserbase routing, normalized V8 coverage, unique CTRF artifacts, and
more stable tests (timeouts, Browserbase skips, understudy click/API
fixes).

Speed up CI by running Vitest and Playwright tests in parallel matrices,
add V8/CTRF coverage, and stabilize Browserbase/LLM flows with hard
timeouts and better cleanup. Server integration tests now use the
runner’s Chromium for reliability and are folded into the main CI.

- **Refactors**
- Parallelized core unit tests via discover-core-tests/core-unit-tests
(Vitest matrix, fail-fast).
- Parallelized E2E via discover-e2e-tests (Playwright local +
Browserbase) gated on core changes; weighted Browserbase region
selection; verify Chromium/CDP before tests; local uses runner Chromium
via CHROME_PATH and --no-sandbox; Playwright can emit JUnit when
CTRF_JUNIT_PATH is set.
- Unified Lint & Build with Turbo caching and shared artifacts;
tests/evals download and reuse; prepare script skips build in CI; SEA
builds support sourcemaps.
- Enforced configurable timeouts and parallelism via env (LLM_MAX_MS,
BROWSERBASE_CDP_CONNECT_MAX_MS, BROWSERBASE_SESSION_CREATE_MAX_MS,
LOCAL_SESSION_LIMIT_PER_E2E_TEST,
BROWSERBASE_SESSION_LIMIT_PER_E2E_TEST, EVAL_*); added LLM and
Browserbase hard timeouts; improved click event dispatch to reduce
remote flakiness.
- Explicitly close Browserbase sessions in tests and evals to free
capacity.
- Added composite actions: setup-node-pnpm-turbo,
verify-chromium-launch, select-browserbase-region, publish-ctrf-report
(JUnit→CTRF and evals), upload-v8-coverage.
- Renamed TEST_ENV to STAGEHAND_ENV across E2E; added an env reporter
for CI logs; removed the standalone stagehand-server tests workflow and
ran server tests in the unified CI with Node’s test runner and GitHub
Actions Chromium.


- **New Features**
- Build/Exports: ESM in dist/esm and CJS in dist/cjs via export map;
types from ESM; SEA binaries built from dist with sourcemaps; server SEA
job targets CJS; new build/test/coverage scripts; Node loader maps
package imports to dist/esm; ESM‑safe pathing (fileURLToPath);
ESLint/Prettier tweaks; dotenv removed.
- CI: Composite actions (setup-node-pnpm-turbo, verify-chromium-launch,
select-browserbase-region, upload-ctrf-report, upload-v8-coverage); uses
runner Chromium via CHROME_PATH with --no-sandbox and launch checks;
weighted Browserbase region selection; unique CTRF artifacts; V8
coverage normalized/merged with debug logs and graceful cancellation;
Playwright/Vitest discover tests in dist/esm; push/PR triggers enabled.
- Tests & API: Vitest/Playwright run on dist/esm; new env reporter and
closeV3 helper; tighter multi‑click CDP dispatch; higher timeouts and
targeted skips (e.g., console on Browserbase) reduce flakiness; local
launches honor CHROME_PATH; replay metrics now include
parameters/result/timestamps/cost with parser fixes; AISdk/CustomOpenAI
clients moved to lib/v3/external_clients and exported; env‑driven hard
timeouts for LLM ops and Browserbase session create/CDP connect;
understudy API usage fixes.

- **Migration**
- Set STAGEHAND_BROWSER_TARGET=local|browserbase and CHROME_PATH; tune
LLM_MAX_MS, BROWSERBASE_SESSION_CREATE_MAX_MS, and
BROWSERBASE_CDP_CONNECT_MAX_MS; optionally set
BROWSERBASE_REGION_DISTRIBUTION.
  - dotenv support is removed; set env vars explicitly in dev and CI.
- Import @browserbasehq/stagehand via the export map; do not reference
dist paths or prior tsup outputs.

<sup>Written for commit 8119145426a4e6e5cb614dee5b3f2d8fe41a65e9.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/1632">Review in
cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-02-17 17:45:49 -08:00
Sean McGuire
638efc7fea
[fix]: refresh agent cache on action failure (#1472) 2025-12-26 13:14:26 -08:00
Sean McGuire
4082b65e23
[chore]: fix e2e local tests in CI (#1323)
# why
- our `e2e:local` tests now require LLM API keys
- these are currently unavailable during the `e2e:local` step in CI,
which causes them to fail
# what changed
- added the relevant LLM API keys to the `env` in CI














<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes failing e2e:local tests in CI by skipping forks and passing the
required API keys to the job.

# Why:
- E2E local tests were failing due to missing secrets and forked PRs not
having access to them.

# What:
- Added job condition to run only on pushes and same-repo PRs (skip
forks).
- Passed required env secrets: OPENAI_API_KEY, ANTHROPIC_API_KEY,
GOOGLE_GENERATIVE_AI_API_KEY, BROWSERBASE_API_KEY,
BROWSERBASE_PROJECT_ID.
- Added ANTHROPIC_API_KEY to turbo.json env for the e2e:local task.

# Test Plan:
- Push a branch in this repo: e2e:local job runs and passes.
- Open a PR from this repo: job runs and external API calls succeed.
- Open a PR from a fork: e2e:local job is skipped.
- Run tests locally with required API keys set: tests pass.

<sup>Written for commit 7eb1ed9f45f51ee59e56266071d50e3f4c1d8a82.
Summary will update automatically on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2025-12-01 11:12:15 -08:00
Filip Michalsky
3c1c7cd2b5
first ci attempt for v3 (#1169)
# why
v3 routes were previously not on CI

# what changed
added CI runners for v3

# test plan
CI passes
2025-10-27 21:16:50 -07:00
Miguel
69b4efa6bc
typecheck (#1124)
# why

# what changed

# test plan

---------

Co-authored-by: Sean McGuire <seanmcguire1@outlook.com>
2025-10-10 13:43:14 -07:00
Miguel
2e02bed7cd
monorepo reorg (#1120)
# why

# what changed

# test plan

---------

Co-authored-by: Sean McGuire <seanmcguire1@outlook.com>
2025-10-10 12:29:49 -07:00