Compare commits

...

11 Commits

Author SHA1 Message Date
Kyle Jeong
1db5f1c193
feat: Add Claude Opus 4.8 CUA support (#2170)
## Summary
- Add `anthropic/claude-opus-4-8` to the CUA model whitelist.
- Route bare `claude-opus-4-8` model names to the Anthropic CUA client.
- Treat Claude Opus 4.8 as a Claude 4.6+ adaptive-thinking CUA model and
document it in the supported CUA model lists.

## Validation
- `pnpm --filter @browserbasehq/stagehand run typecheck`
- `pnpm run test:core --
packages/core/dist/esm/tests/unit/public-api/llm-and-agents.test.js
packages/core/dist/esm/tests/unit/anthropic-cua-adaptive-thinking.test.js`

Note: the focused test command passed 38 tests, then the test wrapper
printed a non-fatal CTRF conversion warning from `glob/minimatch` under
Node 22.
2026-05-28 11:57:04 -07:00
Sam F
ce21468147
STG-1746 Add SEA --version flag (#2167)
# why

The SEA binary needs a lightweight way to report the commit and
timestamp it was built from.

# what changed

Added a `--version` path that prints build metadata baked into the SEA
binary during build.
2026-05-28 19:01:37 +02:00
Sam F
49575d62f5
[STG-1756] forward Vertex model config (#2160)
## Why
Vertex model credentials need a typed public shape and must be forwarded
from the TypeScript client to API-backed Stagehand requests without
persisting secrets server-side.

## What changed
- Define a canonical Vertex model config shape using `provider:
"vertex"`, `auth`, and `providerOptions.vertex`, while preserving
existing generic model string/object support.
- Forward constructor model config through API-backed
act/observe/extract/agentExecute requests, with per-call model config
still taking precedence.
- Keep navigate/goto from exposing a public per-call model option, while
still allowing constructor config to bootstrap API-backed sessions
internally.
- Sync server-v3 request parsing with the hosted API approach: Zod-first
safe parsing, separate request model config from Stagehand init model
config, and clean SSE validation errors.
- Regenerate server-v3 OpenAPI from the Stagehand Zod source so
Stainless can pick up the canonical shape.

TODO before final hosted API merge: swap core to the Orca build produced
from this Stagehand change.

---------

Co-authored-by: samfinton <samfinton@samLaptopm5.local>
2026-05-27 21:41:59 +02:00
Sean McGuire
bf9d44989c
[chore]: bump ws dep (#2169)
# why
- bumps `ws` & a few other deps to versions with various security
patches
# what changed
- bumps direct `ws` dep from `^8.18.0` to `^8.21.0`
- bumps turbo from `^2.8.10` to `^2.9.15`
- adds monorepo-wide overrides for `qs` & `ws` which are transitive deps
brought in by mintlify & braintrust
- overrides were required here since a direct bump of braintrust &
mintlify did not resolve to upgraded `qs` & `ws` versions


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Upgrade `ws` to patched versions across the repo and add overrides for
transitive `ws` and `qs` to address security fixes. Also bump `turbo` to
the latest minor.

- **Dependencies**
  - `packages/cli` and `packages/core`: `ws` ^8.18.0 -> ^8.21.0
- Root overrides: `ws` ^8.20.1, `qs` ^6.15.2 (to override transitive
deps from `mintlify` and `braintrust`)
  - `turbo` ^2.8.10 -> ^2.9.15

<sup>Written for commit e8f6ff1f863aac475f754f7d80e83db0eea4ad09.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2169?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-05-27 11:50:35 -07:00
Sean McGuire
aff5528c8d
[chore]: update lockfile (#2168)
# why
- recent lockfile change was left out of a previous commit
# what changed
- ran pnpm i & added the lockfile


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Regenerated `pnpm-lock.yaml` that was missed in a prior commit to align
with current workspace dependencies. This removes stale entries and
ensures reproducible installs in CI and local dev.

<sup>Written for commit c0e79fec55c32dedb245976a3117d3f861f745d5.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2168?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-05-27 10:50:50 -07:00
Sam F
947e6e4249
Fix SDK API reference parameter labels (#2164)
## Summary
- add a Mintlify custom script that rewrites generated SDK API reference
schema labels from OpenAPI camelCase to SDK snake_case on Python and
Ruby pages
- keep code/pre/input/script/style content untouched so generated code
samples stay language-correct
- leave Java/Go/TypeScript routes unchanged for now

## Why
Mintlify renders these language-scoped API reference pages from the
Stainless documented OpenAPI spec, so visible schema labels use
wire-format camelCase even when the Python and Ruby SDK snippets use
snake_case.

## Validation
- `pnpm --dir packages/docs exec prettier --check
sdk-api-reference-labels.js`
- `node --check packages/docs/sdk-api-reference-labels.js`
- confirmed Mintlify embeds root-level docs JS by checking the existing
custom script in local/prod HTML and the new script in local HTML
- browser fixture served at `/v3/api-reference/{python,ruby,java}/...`:
Python/Ruby labels rewrote to snake_case, code blocks stayed camelCase,
and Java remained unchanged

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes mismatched parameter labels on Python and Ruby SDK API reference
pages by rewriting OpenAPI camelCase labels to SDK snake_case. Code
samples and inputs remain unchanged; other languages are unaffected.

- **Bug Fixes**
- Added `packages/docs/sdk-api-reference-labels.js` to relabel visible
schema fields for Python and Ruby only.
- Skips `code`, `pre`, `input`, `script`, `style`, and editable elements
to preserve language-correct snippets.
- Reacts to SPA route changes and DOM mutations to keep labels
consistent.
  - Leaves Java/Go/TypeScript pages unchanged.

<sup>Written for commit ea143a674cc1a8b55cf39351506e93ecf0d70016.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2164?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-05-27 10:23:50 +02:00
Miguel
e102a89c90
feat(verifier): record agent trajectories (#2131)
# Why

The new verifier needs richer evidence than a final screenshot,
especially for DOM and Hybrid agent modes where the important facts
often live in tool returns, ARIA snapshots, and per-step observations.
This PR adds trajectory recording without changing the verifier judgment
engine.

# What Changed

- Added typed agent events for screenshot, step-finished, step-observed,
and final-answer events.
- Added listener-gated post-step probes for screenshots and ARIA trees.
- Attached the settled post-turn probe to every tool call in a
DOM/Hybrid turn.
- Added CUA step evidence pairing and final answer capture.
- Added `TrajectoryRecorder` persistence and a smoke script for
trajectory shape and disk layout.

# Tests

- `pnpm --filter @browserbasehq/stagehand run typecheck`
- `pnpm --filter @browserbasehq/stagehand-evals run typecheck`
- `node --import tsx
packages/evals/scripts/verify-trajectory-recorder.ts`
- `git diff --check`

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 14:16:16 -07:00
Miguel
2cd60a34b0
feat(verifier): add verifier evaluator shell and types (#2157) 2026-05-22 12:38:21 -07:00
Nick Sweeting
295ac6d1ce
Delete packages/server-v4 entirely for now (#2151) 2026-05-21 17:45:26 -07:00
Miguel
278890b116
fix[evals] remove braintrust api key requirement (#2145) (#2153)
# why
Evals should be able to run without the braintrust dependency

# what changed
- Added `hasBraintrustApiKey()` and made `tracedSpan()` a passthrough
when no key.
- `AISdkClientWrapped` falls back to the plain `ai` SDK when
`braintrust` wrapping isn’t available; introduced a `WrappedAI` type and
lazy load keyed on `BRAINTRUST_API_KEY`.

# test plan
- [x] Added tests for key-present/absent paths and updated `core-runner`
test to set a dummy key; fixes for test stability and build.


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Make Braintrust optional for evals. Evals run without
`BRAINTRUST_API_KEY` and skip sending logs; behavior is unchanged when a
key is set.

- **Bug Fixes**
- Added `hasBraintrustApiKey()` and made `tracedSpan()` a passthrough
when no key.
- Runner now passes `noSendLogs: true` to `Eval` and skips `flush()`
without a key; sends logs and flushes when present.
- `AISdkClientWrapped` falls back to the plain `ai` SDK when
`braintrust` wrapping isn’t available; introduced a `WrappedAI` type and
lazy load keyed on `BRAINTRUST_API_KEY`.
- Added tests for key-present/absent paths and updated `core-runner`
test to set a dummy key; fixes for test stability and build.

<sup>Written for commit 482e49bac6dce88ea110aa9a4f06a50346617a06.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2153?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Chromie <miguel@browserbase.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-05-21 15:39:40 -07:00
Miguel
43548374b9
Workflow: publish eval results (#2093) 2026-05-21 13:08:12 -07:00
192 changed files with 5980 additions and 26043 deletions

View File

@ -0,0 +1,5 @@
---
"@browserbasehq/stagehand": patch
---
Add Claude Opus 4.8 to the supported CUA model whitelist.

View File

@ -0,0 +1,5 @@
---
"@browserbasehq/stagehand-server-v3": patch
---
Add SEA binary --version build metadata output.

View File

@ -0,0 +1,5 @@
---
"@browserbasehq/stagehand": patch
---
Add verifier trajectory, rubric, and evaluation-result types with normalized public naming.

View File

@ -0,0 +1,5 @@
---
"@browserbasehq/stagehand": patch
---
Capture verifier trajectory evidence from agent evidence callbacks for offline scoring.

View File

@ -0,0 +1,6 @@
---
"@browserbasehq/stagehand": minor
"@browserbasehq/stagehand-server-v3": minor
---
Forward constructor and request model configuration when initializing API-backed sessions.

View File

@ -193,6 +193,14 @@ jobs:
run: |
set +e
# The GitHub iOS app serializes `type: number` workflow_dispatch
# inputs as doubles ("1.0"/"50.0") regardless of whether the
# workflow needs an integer. The evals CLI's `parsePositiveInteger`
# (regex /^[0-9]+$/) rejects the decimal form, so strip a single
# trailing `.0` here before passing the values along.
EVAL_TRIALS="${EVAL_TRIALS%.0}"
EVAL_CONCURRENCY="${EVAL_CONCURRENCY%.0}"
agent_mode_args=()
if [[ -n "${EVAL_AGENT_MODE:-}" && "$EVAL_AGENT_MODE" != "auto" ]]; then
agent_mode_args=(--agent-mode "$EVAL_AGENT_MODE")

130
.github/workflows/publish-evals.yml vendored Normal file
View File

@ -0,0 +1,130 @@
name: Publish Evals
on:
workflow_dispatch:
inputs:
experiment:
description: Braintrust experiment name or UUID.
required: true
type: string
project:
description: Braintrust project containing the experiment.
required: false
default: stagehand
type: string
kv_key:
description: Upstash Redis key the UI reads.
required: false
default: stagehand:evals:latest
type: string
experiment_key_prefix:
description: Prefix for the secondary experiment-id key.
required: false
default: stagehand:evals:experiments
type: string
write_experiment_key:
description: Also write <experiment_key_prefix>:<experiment_id>.
required: false
default: true
type: boolean
dry_run:
description: Fetch and render payload without writing to Upstash.
required: false
default: false
type: boolean
permissions:
contents: read
env:
PUPPETEER_SKIP_DOWNLOAD: "1"
PLAYWRIGHT_SKIP_DOWNLOAD: "1"
TURBO_TELEMETRY_DISABLED: "1"
jobs:
publish:
name: Publish Evals UI Data
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
- uses: ./.github/actions/setup-node-pnpm-turbo
with:
use-prebuilt-artifacts: "false"
restore-turbo-cache: "false"
node-version: 20.x
- name: Publish Braintrust experiment to Upstash
id: publish
env:
BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
UPSTASH_REDIS_REST_URL: ${{ secrets.UPSTASH_REDIS_REST_URL }}
UPSTASH_REDIS_REST_TOKEN: ${{ secrets.UPSTASH_REDIS_REST_TOKEN }}
KV_REST_API_URL: ${{ secrets.KV_REST_API_URL }}
KV_REST_API_TOKEN: ${{ secrets.KV_REST_API_TOKEN }}
EXPERIMENT: ${{ inputs.experiment }}
PROJECT: ${{ inputs.project }}
KV_KEY: ${{ inputs.kv_key }}
EXPERIMENT_KEY_PREFIX: ${{ inputs.experiment_key_prefix }}
WRITE_EXPERIMENT_KEY: ${{ inputs.write_experiment_key }}
DRY_RUN: ${{ inputs.dry_run }}
shell: bash
run: |
set -euo pipefail
args=(
--experiment "$EXPERIMENT"
--project "$PROJECT"
--key "$KV_KEY"
--experiment-key-prefix "$EXPERIMENT_KEY_PREFIX"
--out evals-ui-data.json
)
if [[ "$WRITE_EXPERIMENT_KEY" != "true" ]]; then
args+=(--no-experiment-key)
fi
if [[ "$DRY_RUN" == "true" ]]; then
args+=(--dry-run)
fi
pnpm --filter @browserbasehq/stagehand-evals exec tsx \
scripts/publish-braintrust-ui-data.ts \
"${args[@]}" | tee publish-evals-ui-data-output.json
{
echo "summary<<EOF"
jq -r '
"Experiment: \(.experimentName)",
"Project: \(.projectName)",
"Keys:",
(.keys[] | " " + .),
"Passed: \(.summary.passed)/\(.summary.total)",
"Pass rate: \((.summary.passPercent | tonumber) | tostring)%",
"Dry run: \(.dryRun)"
' publish-evals-ui-data-output.json
echo "EOF"
} >> "$GITHUB_OUTPUT"
- name: Add publish summary
if: always()
env:
PUBLISH_SUMMARY: ${{ steps.publish.outputs.summary }}
shell: bash
run: |
if [ -n "${PUBLISH_SUMMARY:-}" ]; then
echo "### Published Evals UI Data" >> "$GITHUB_STEP_SUMMARY"
echo '```' >> "$GITHUB_STEP_SUMMARY"
printf '%s\n' "$PUBLISH_SUMMARY" >> "$GITHUB_STEP_SUMMARY"
echo '```' >> "$GITHUB_STEP_SUMMARY"
fi
- name: Upload generated payload
if: always()
uses: actions/upload-artifact@v4
with:
name: evals-ui-data
path: |
evals-ui-data.json
publish-evals-ui-data-output.json

View File

@ -56,7 +56,7 @@
"prettier": "^3.2.5",
"source-map": "^0.7.4",
"tsx": "^4.19.4",
"turbo": "^2.8.10",
"turbo": "^2.9.15",
"typescript": "5.8.3",
"typescript-eslint": "^8.56.1"
},
@ -78,7 +78,9 @@
"jws": "^4.0.1",
"lodash": "^4.18.0",
"minimatch": "^3.1.4",
"uuid": "^11.1.1"
"uuid": "^11.1.1",
"ws": "^8.20.1",
"qs": "^6.15.2"
}
},
"lint-staged": {

View File

@ -51,7 +51,7 @@
"node-html-markdown": "^1.3.0",
"pino": "^9.6.0",
"pino-pretty": "^13.0.0",
"ws": "^8.18.0"
"ws": "^8.21.0"
},
"devDependencies": {
"@types/node": "^20.11.30",

View File

@ -23,6 +23,7 @@ export const modelToAgentProviderMap: Record<string, AgentProviderType> = {
"claude-sonnet-4-5-20250929": "anthropic",
"claude-opus-4-5-20251101": "anthropic",
"claude-opus-4-6": "anthropic",
"claude-opus-4-8": "anthropic",
"claude-sonnet-4-6": "anthropic",
"claude-haiku-4-5": "anthropic",
"claude-haiku-4-5-20251001": "anthropic",

View File

@ -453,6 +453,7 @@ export class AnthropicCUAClient extends AgentClient {
// Check if this is a Claude 4.6+ model that supports adaptive thinking
const isAdaptiveThinkingModel = [
"claude-opus-4-8",
"claude-opus-4-6",
"claude-sonnet-4-6",
].includes(modelBase);

View File

@ -0,0 +1,73 @@
/**
* captureAriaTreeProbe — capture a truncated accessibility tree of the active
* page for use as tier-2 evidence in the trajectory recorder.
*
* Shared by v3AgentHandler and v3CuaAgentHandler. Listener-gated by the
* callers so ordinary agent runs (no TrajectoryRecorder attached) don't pay
* the cost.
*
* The a11y tree is the same payload the agent's `ariaTree` tool sees, but
* captured by the harness (not the agent) so the verifier has independent
* textual ground truth for grounding non-visual claims — prices, names,
* dates, list contents — without OCR'ing screenshots.
*
* Budget: defaults to ~8000 tokens (32k chars). Per-step a11y captures
* across a ~30-step trajectory at that cap sum to ~240k tokens total,
* which the verifier handles via per-criterion top-K selection. The cap
* is configurable via VERIFIER_ARIATREE_TOKEN_BUDGET so consumers can
* trade RAM/disk for fidelity. Truncated content is marked explicitly so
* the verifier knows it was clipped.
*/
import type { V3 } from "../../v3.js";
const APPROX_CHARS_PER_TOKEN = 4;
const DEFAULT_TOKEN_BUDGET = 8_000;
const DEFAULT_TIMEOUT_MS = 5_000;
interface CaptureAriaTreeOptions {
/** Soft cap on token count (chars/4 approximation). Default 8000. */
tokenBudget?: number;
/** Hard timeout on the capture. Default 5s. */
timeoutMs?: number;
}
/**
* Returns the truncated a11y tree as a plain string, or undefined when
* capture fails. Never throws — a11y capture is best-effort tier-2 evidence,
* not a hard requirement, so failures are silently absorbed (the verifier
* surfaces this via evidence_insufficient).
*/
export async function captureAriaTreeProbe(
v3: V3,
opts: CaptureAriaTreeOptions = {},
): Promise<string | undefined> {
const envBudget = parseInt(
process.env.VERIFIER_ARIATREE_TOKEN_BUDGET ?? "",
10,
);
const tokenBudget =
opts.tokenBudget ??
(Number.isFinite(envBudget) && envBudget > 0
? envBudget
: DEFAULT_TOKEN_BUDGET);
const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
const maxChars = tokenBudget * APPROX_CHARS_PER_TOKEN;
try {
// v3.extract() without a schema returns { pageText } where pageText is the
// rendered accessibility tree — same path the agent's ariaTree tool uses.
const result = await v3.extract({ timeout: timeoutMs });
const pageText = result?.pageText;
if (typeof pageText !== "string" || pageText.length === 0) return undefined;
if (pageText.length > maxChars) {
return (
pageText.slice(0, maxChars) +
`\n\n[CONTENT TRUNCATED at ~${tokenBudget} tokens — set VERIFIER_ARIATREE_TOKEN_BUDGET to raise]`
);
}
return pageText;
} catch {
return undefined;
}
}

View File

@ -0,0 +1,81 @@
import type { AgentEvidenceCallback } from "../../types/public/agentEvidenceEvents.js";
import type { LogLine } from "../../types/public/logs.js";
import type { V3 } from "../../v3.js";
import { captureAriaTreeProbe } from "./captureAriaTreeProbe.js";
interface CaptureProbeEvidenceOptions {
v3: V3;
url: string;
logger: (message: LogLine) => void;
warningMessage: string;
}
interface EmitPostStepProbeEvidenceOptions extends CaptureProbeEvidenceOptions {
evidenceCallback?: AgentEvidenceCallback;
}
function errorMessage(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
export async function captureProbeEvidence({
v3,
url,
logger,
warningMessage,
}: CaptureProbeEvidenceOptions): Promise<{
url: string;
screenshot?: Buffer;
ariaTree?: string;
}> {
let probeUrl = url;
let screenshot: Buffer | undefined;
try {
const page = await v3.context.awaitActivePage();
probeUrl = page.url();
screenshot = await page.screenshot({ fullPage: false });
} catch (e) {
logger({
category: "agent",
message: `${warningMessage}: ${errorMessage(e)}`,
level: 1,
});
}
const ariaTree = await captureAriaTreeProbe(v3);
return {
url: probeUrl,
...(screenshot ? { screenshot } : {}),
...(ariaTree !== undefined ? { ariaTree } : {}),
};
}
export async function emitPostStepProbeEvidence({
v3,
url,
evidenceCallback,
logger,
warningMessage,
}: EmitPostStepProbeEvidenceOptions): Promise<void> {
if (!evidenceCallback) return;
const probe = await captureProbeEvidence({
v3,
url,
logger,
warningMessage,
});
if (probe.screenshot) {
await evidenceCallback({
type: "screenshot",
screenshot: probe.screenshot,
url: probe.url,
evidenceRole: "probe",
});
}
await evidenceCallback({
type: "step_observed",
url: probe.url,
ariaTree: probe.ariaTree,
});
}

View File

@ -0,0 +1,76 @@
import type { AgentStepFinishedEvent } from "../../types/public/agentEvidenceEvents.js";
const ERROR_STRING_LIMIT = 1000;
function isRecord(value: unknown): value is Record<string, unknown> {
return value !== null && typeof value === "object" && !Array.isArray(value);
}
function hasOwn(value: Record<string, unknown>, key: string): boolean {
return Object.prototype.hasOwnProperty.call(value, key);
}
function normalizeError(value: unknown): string | undefined {
if (value === undefined || value === null || value === false) {
return undefined;
}
if (value instanceof Error) {
return value.message;
}
if (typeof value === "string") {
return value;
}
if (
typeof value === "number" ||
typeof value === "boolean" ||
typeof value === "bigint"
) {
return String(value);
}
let serialized: string;
try {
serialized = JSON.stringify(value) ?? String(value);
} catch {
serialized = String(value);
}
if (serialized.length <= ERROR_STRING_LIMIT) {
return serialized;
}
return `${serialized.slice(0, ERROR_STRING_LIMIT)}... [truncated]`;
}
function statusCandidates(toolResult: unknown): Record<string, unknown>[] {
if (!isRecord(toolResult)) {
return [];
}
const candidates = [toolResult];
const output = toolResult.output;
if (isRecord(output)) {
candidates.push(output);
}
return candidates;
}
export function inferToolOutput(
toolResult: unknown,
): AgentStepFinishedEvent["toolOutput"] {
const candidates = statusCandidates(toolResult);
const error = candidates
.map((candidate) =>
hasOwn(candidate, "error") ? normalizeError(candidate.error) : undefined,
)
.find((message): message is string => message !== undefined);
const successFalse = candidates.some(
(candidate) => candidate.success === false,
);
const isError = candidates.some((candidate) => Boolean(candidate.isError));
return {
ok: error === undefined && !isError && !successFalse,
result: toolResult,
error,
};
}

View File

@ -0,0 +1,27 @@
import type { AgentEvidenceCallback } from "../../types/public/agentEvidenceEvents.js";
import type { LogLine } from "../../types/public/logs.js";
// onEvidence is a user-supplied observability hook (trajectory recording,
// verifier capture, etc.). Wrap it once at the boundary where the handler
// receives it so a throwing user callback can never abort the agent loop —
// internal emit sites can then call the wrapped callback directly without
// per-site try/catch.
export function wrapEvidenceCallback(
callback: AgentEvidenceCallback | undefined,
logger: (message: LogLine) => void,
): AgentEvidenceCallback | undefined {
if (!callback) return undefined;
return async (event) => {
try {
await callback(event);
} catch (e) {
logger({
category: "agent",
message: `Warning: onEvidence callback failed for ${event.type}: ${
e instanceof Error ? e.message : String(e)
}`,
level: 1,
});
}
};
}

View File

@ -106,6 +106,8 @@ interface ClientSessionStartParams extends Api.SessionStartRequest {
* Optional: when omitted, requests are sent without the x-model-api-key header
* and the server is expected to handle model authentication on its own. */
modelApiKey?: string;
/** Default model config for later action requests. Not sent to /sessions/start. */
defaultModelConfig?: ModelConfiguration;
}
/**
@ -115,6 +117,11 @@ type ApiResponse<T> =
| { success: true; data: T }
| { success: false; message: string };
type PreparedModelConfig = { modelName: string; apiKey?: string } & Record<
string,
unknown
>;
/**
* Union of all API request body types for type-safe execute() calls
*/
@ -180,6 +187,7 @@ export class StagehandAPIClient {
private sessionId?: string;
private modelApiKey?: string;
private modelProvider?: string;
private defaultModelConfig?: PreparedModelConfig;
private region?: BrowserbaseRegion;
private logger: (message: LogLine) => void;
private fetchWithCookies;
@ -205,6 +213,7 @@ export class StagehandAPIClient {
async init({
modelName,
modelApiKey,
defaultModelConfig,
domSettleTimeoutMs,
verbose,
systemPrompt,
@ -218,6 +227,9 @@ export class StagehandAPIClient {
this.modelProvider = modelName?.includes("/")
? modelName.split("/")[0]
: undefined;
this.defaultModelConfig = defaultModelConfig
? this.prepareModelConfig(defaultModelConfig)
: undefined;
// Store the region for multi-region API URL resolution
this.region = browserbaseSessionCreateParams?.region;
@ -288,12 +300,18 @@ export class StagehandAPIClient {
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const { page: _, serverCache: enableCache, ...restOptions } = options;
serverCache = enableCache;
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
} else if (this.defaultModelConfig) {
restOptions.model = this.getDefaultModelConfig();
}
if (Object.keys(restOptions).length > 0) {
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
}
wireOptions = restOptions as unknown as Api.ActRequest["options"];
}
} else if (this.defaultModelConfig) {
wireOptions = {
model: this.getDefaultModelConfig(),
} as unknown as Api.ActRequest["options"];
}
// Build wire-format request body
@ -326,12 +344,18 @@ export class StagehandAPIClient {
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const { page: _, serverCache: enableCache, ...restOptions } = options;
serverCache = enableCache;
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
} else if (this.defaultModelConfig) {
restOptions.model = this.getDefaultModelConfig();
}
if (Object.keys(restOptions).length > 0) {
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
}
wireOptions = restOptions as unknown as Api.ExtractRequest["options"];
}
} else if (this.defaultModelConfig) {
wireOptions = {
model: this.getDefaultModelConfig(),
} as unknown as Api.ExtractRequest["options"];
}
// Build wire-format request body
@ -361,12 +385,18 @@ export class StagehandAPIClient {
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const { page: _, serverCache: enableCache, ...restOptions } = options;
serverCache = enableCache;
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
} else if (this.defaultModelConfig) {
restOptions.model = this.getDefaultModelConfig();
}
if (Object.keys(restOptions).length > 0) {
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
}
wireOptions = restOptions as unknown as Api.ObserveRequest["options"];
}
} else if (this.defaultModelConfig) {
wireOptions = {
model: this.getDefaultModelConfig(),
} as unknown as Api.ObserveRequest["options"];
}
// Build wire-format request body
@ -388,7 +418,22 @@ export class StagehandAPIClient {
options?: Api.NavigateRequest["options"],
frameId?: string,
): Promise<SerializableResponse | null> {
const requestBody: Api.NavigateRequest = { url, options, frameId };
const publicOptions = { ...(options ?? {}) } as NonNullable<
Api.NavigateRequest["options"]
> & { model?: unknown };
delete publicOptions.model;
const wireOptions = {
...publicOptions,
...(this.defaultModelConfig
? { model: this.getDefaultModelConfig() }
: {}),
};
const requestBody: Api.NavigateRequest = {
url,
options: Object.keys(wireOptions).length > 0 ? wireOptions : undefined,
frameId,
};
return this.execute<SerializableResponse | null>({
method: "navigate",
@ -425,7 +470,7 @@ export class StagehandAPIClient {
cua: agentConfig.mode === undefined ? agentConfig.cua : undefined,
model: agentConfig.model
? this.prepareModelConfig(agentConfig.model)
: undefined,
: this.getDefaultModelConfig(),
executionModel: agentConfig.executionModel
? this.prepareModelConfig(agentConfig.executionModel)
: undefined,
@ -605,40 +650,52 @@ export class StagehandAPIClient {
* In API mode, we only attempt to load an API key from env vars when the
* model provider differs from the one used to init the session.
*/
private prepareModelConfig(
model: ModelConfiguration,
): { modelName: string; apiKey?: string } & Record<string, unknown> {
private prepareModelConfig(model: ModelConfiguration): PreparedModelConfig {
if (typeof model === "string") {
// Extract provider from model string (e.g., "openai/gpt-5-nano" -> "openai")
const provider = model.includes("/") ? model.split("/")[0] : undefined;
const provider = this.getModelProvider(model);
const inheritedDefault =
provider && provider === this.modelProvider
? this.getDefaultModelConfig()
: undefined;
const apiKey =
provider && provider !== this.modelProvider
? (loadApiKeyFromEnv(provider, this.logger) ?? this.modelApiKey)
: this.modelApiKey;
return {
...inheritedDefault,
modelName: model,
...(apiKey ? { apiKey } : {}),
};
}
if (!model.apiKey) {
const provider = model.modelName?.includes("/")
? model.modelName.split("/")[0]
const provider = this.getModelProvider(model.modelName);
const inheritedDefault =
provider && provider === this.modelProvider
? this.getDefaultModelConfig()
: undefined;
const apiKey =
provider && provider !== this.modelProvider
? (loadApiKeyFromEnv(provider, this.logger) ?? this.modelApiKey)
: this.modelApiKey;
return {
...model,
...(apiKey ? { apiKey } : {}),
};
}
const apiKey =
!model.apiKey && provider && provider !== this.modelProvider
? (loadApiKeyFromEnv(provider, this.logger) ?? this.modelApiKey)
: !model.apiKey
? this.modelApiKey
: undefined;
return model as { modelName: string; apiKey: string } & Record<
string,
unknown
>;
return {
...inheritedDefault,
...model,
...(apiKey ? { apiKey } : {}),
} as PreparedModelConfig;
}
private getDefaultModelConfig(): PreparedModelConfig | undefined {
return this.defaultModelConfig
? ({ ...this.defaultModelConfig } as PreparedModelConfig)
: undefined;
}
private getModelProvider(modelName: string | undefined): string | undefined {
return modelName?.includes("/") ? modelName.split("/")[0] : undefined;
}
private consumeFinishedEventData<T>(): T | null {

View File

@ -31,6 +31,7 @@ import {
AgentModelConfig,
Variables,
} from "../types/public/agent.js";
import type { AgentEvidenceCallback } from "../types/public/agentEvidenceEvents.js";
import { HYBRID_CAPABLE_MODEL_PATTERNS } from "../types/private/agent.js";
import { V3FunctionName } from "../types/public/methods.js";
import { mapToolResultToActions } from "../agent/utils/actionMapping.js";
@ -41,6 +42,12 @@ import {
AgentAbortError,
} from "../types/public/sdkErrors.js";
import { handleDoneToolCall } from "../agent/utils/handleDoneToolCall.js";
import {
captureProbeEvidence,
emitPostStepProbeEvidence,
} from "../agent/utils/postStepProbeEvidence.js";
import { wrapEvidenceCallback } from "../agent/utils/wrapEvidenceCallback.js";
import { inferToolOutput } from "../agent/utils/toolOutputEvidence.js";
import {
CaptchaSolver,
CAPTCHA_SOLVED_MSG,
@ -51,6 +58,19 @@ function getErrorMessage(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
type FinalAnswerDraft = {
message: string;
output?: Record<string, unknown>;
};
interface StepHandlerOptions {
userCallback?:
| GenerateTextOnStepFinishCallback<ToolSet>
| StreamTextOnStepFinishCallback<ToolSet>;
evidenceCallback?: AgentEvidenceCallback;
onFinalAnswer?: (answer: FinalAnswerDraft) => void;
}
/**
* Prepends a system message with cache control to the messages array.
* The cache control providerOptions are used by Anthropic and ignored by other providers.
@ -244,9 +264,7 @@ export class V3AgentHandler {
private createStepHandler(
state: AgentState,
userCallback?:
| GenerateTextOnStepFinishCallback<ToolSet>
| StreamTextOnStepFinishCallback<ToolSet>,
{ userCallback, evidenceCallback, onFinalAnswer }: StepHandlerOptions,
) {
return async (event: StepResult<ToolSet>) => {
this.logger({
@ -255,6 +273,8 @@ export class V3AgentHandler {
level: 2,
});
let lastFinalAnswer: FinalAnswerDraft | undefined;
if (event.toolCalls && event.toolCalls.length > 0) {
for (let i = 0; i < event.toolCalls.length; i++) {
const toolCall = event.toolCalls[i];
@ -279,6 +299,13 @@ export class V3AgentHandler {
? `${allReasoning} ${doneReasoning}`.trim()
: allReasoning || "Task completed successfully";
}
lastFinalAnswer = {
message: state.finalMessage,
output:
typeof args?.output === "object" && args?.output !== null
? (args.output as Record<string, unknown>)
: undefined,
};
}
const mappedActions = mapToolResultToActions({
toolCallName: toolCall.toolName,
@ -292,8 +319,35 @@ export class V3AgentHandler {
action.timestamp = Date.now();
state.actions.push(action);
}
await evidenceCallback?.({
type: "step_finished",
actionName: toolCall.toolName,
actionArgs:
typeof args === "object" && args !== null
? (args as Record<string, unknown>)
: {},
reasoning: event.text ?? "",
toolOutput: inferToolOutput(toolResult),
});
}
state.currentPageUrl = (await this.v3.context.awaitActivePage()).url();
// Harness probe — one screenshot / a11y snapshot per AI SDK step.
// The recorder applies the probe to every step_finished received
// since the previous probe, so a multi-tool turn shares the same
// post-turn observation.
await emitPostStepProbeEvidence({
v3: this.v3,
url: state.currentPageUrl,
evidenceCallback,
logger: this.logger,
warningMessage: "Warning: harness probe failed",
});
}
if (lastFinalAnswer) {
onFinalAnswer?.(lastFinalAnswer);
}
if (userCallback) {
@ -321,6 +375,7 @@ export class V3AgentHandler {
completed: false,
currentPageUrl: "",
};
let finalAnswerFromDoneTool: FinalAnswerDraft | undefined;
let messages: ModelMessage[] = [];
let captchaSolver: CaptchaSolver | undefined;
@ -368,6 +423,11 @@ export class V3AgentHandler {
}
}
const evidenceCallback = wrapEvidenceCallback(
callbacks?.onEvidence,
this.logger,
);
const result = await this.llmClient.generateText({
model: wrappedModel,
messages: prependSystemMessage(systemPrompt, messages),
@ -379,7 +439,13 @@ export class V3AgentHandler {
callbacks?.prepareStep,
captchaSolver,
),
onStepFinish: this.createStepHandler(state, callbacks?.onStepFinish),
onStepFinish: this.createStepHandler(state, {
userCallback: callbacks?.onStepFinish,
evidenceCallback,
onFinalAnswer: (answer) => {
finalAnswerFromDoneTool = answer;
},
}),
abortSignal: preparedOptions.signal,
providerOptions: {
google: { mediaResolution: "MEDIA_RESOLUTION_HIGH" },
@ -396,6 +462,15 @@ export class V3AgentHandler {
preparedOptions.output,
this.logger,
);
const output = doneResult.output ?? finalAnswerFromDoneTool?.output;
await this.emitFinalEvidence(
state,
{
message: state.finalMessage,
output,
},
evidenceCallback,
);
return this.consolidateMetricsAndResult(
startTime,
@ -403,7 +478,7 @@ export class V3AgentHandler {
doneResult.messages,
result,
maxSteps,
doneResult.output,
output,
);
} catch (error) {
// Re-throw validation errors that should propagate to the caller
@ -449,6 +524,7 @@ export class V3AgentHandler {
// Highlight cursor defaults to true for hybrid mode, can be overridden
const shouldHighlightCursor =
streamOptions?.highlightCursor ?? this.mode === "hybrid";
let finalAnswerFromDoneTool: FinalAnswerDraft | undefined;
const {
options,
@ -503,6 +579,11 @@ export class V3AgentHandler {
rejectResult(error);
};
const evidenceCallback = wrapEvidenceCallback(
callbacks?.onEvidence,
this.logger,
);
let streamResult: ReturnType<typeof this.llmClient.streamText>;
try {
streamResult = this.llmClient.streamText({
@ -515,7 +596,13 @@ export class V3AgentHandler {
callbacks?.prepareStep,
captchaSolver,
),
onStepFinish: this.createStepHandler(state, callbacks?.onStepFinish),
onStepFinish: this.createStepHandler(state, {
userCallback: callbacks?.onStepFinish,
evidenceCallback,
onFinalAnswer: (answer) => {
finalAnswerFromDoneTool = answer;
},
}),
onError: (event) => {
captchaSolver?.dispose();
if (callbacks?.onError) {
@ -541,17 +628,29 @@ export class V3AgentHandler {
options.instruction,
options.output,
this.logger,
).then((doneResult) => {
const result = this.consolidateMetricsAndResult(
startTime,
state,
doneResult.messages,
event,
maxSteps,
doneResult.output,
);
resolveResult(result);
});
)
.then(async (doneResult) => {
const output =
doneResult.output ?? finalAnswerFromDoneTool?.output;
await this.emitFinalEvidence(
state,
{
message: state.finalMessage,
output,
},
evidenceCallback,
);
const result = this.consolidateMetricsAndResult(
startTime,
state,
doneResult.messages,
event,
maxSteps,
output,
);
resolveResult(result);
})
.catch(handleError);
},
onAbort: (event) => {
captchaSolver?.dispose();
@ -580,6 +679,26 @@ export class V3AgentHandler {
return agentStreamResult;
}
private async emitFinalEvidence(
state: AgentState,
finalAnswer: { message: string; output?: Record<string, unknown> },
evidenceCallback?: AgentEvidenceCallback,
): Promise<void> {
if (!evidenceCallback) return;
const observation = await captureProbeEvidence({
v3: this.v3,
url: state.currentPageUrl,
logger: this.logger,
warningMessage: "Warning: final harness probe failed",
});
await evidenceCallback({
type: "final_answer",
...finalAnswer,
observation,
});
}
private consolidateMetricsAndResult(
startTime: number,
state: AgentState,

View File

@ -7,6 +7,12 @@ import { GoogleCUAClient } from "../agent/GoogleCUAClient.js";
import { OpenAICUAClient } from "../agent/OpenAICUAClient.js";
import { mapKeyToPlaywright } from "../agent/utils/cuaKeyMapping.js";
import { ensureXPath } from "../agent/utils/xpath.js";
import {
captureProbeEvidence,
emitPostStepProbeEvidence,
} from "../agent/utils/postStepProbeEvidence.js";
import { wrapEvidenceCallback } from "../agent/utils/wrapEvidenceCallback.js";
import { inferToolOutput } from "../agent/utils/toolOutputEvidence.js";
import {
ActionExecutionResult,
AgentAction,
@ -16,6 +22,10 @@ import {
SafetyConfirmationHandler,
} from "../types/public/agent.js";
import { LogLine } from "../types/public/logs.js";
import type {
AgentEvidenceCallback,
AgentStepFinishedEvent,
} from "../types/public/agentEvidenceEvents.js";
import { type Action, V3FunctionName } from "../types/public/methods.js";
import { FlowLogger } from "../flowlogger/FlowLogger.js";
import { toTitleCase } from "../../utils.js";
@ -37,6 +47,8 @@ export class V3CuaAgentHandler {
private captchaSolver: CaptchaSolver | null = null;
private captchaClickGuardRemaining = 0;
private currentInstruction = "";
private lastAgentScreenshotUrl?: string;
private evidenceCallback?: AgentEvidenceCallback;
constructor(
v3: V3,
@ -76,6 +88,9 @@ export class V3CuaAgentHandler {
this.ensureNotClosed();
const page = await this.v3.context.awaitActivePage();
const screenshotBuffer = await page.screenshot({ fullPage: false });
await this.emitCuaScreenshot(screenshotBuffer, page.url());
return screenshotBuffer.toString("base64"); // base64 png
});
@ -119,7 +134,12 @@ export class V3CuaAgentHandler {
const waitBetween =
(this.options.clientOptions?.waitBetweenActions as number) ||
defaultDelay;
// Skip logging for screenshot actions - they're no-ops; the CUA client
// takes its own screenshot via screenshotProvider between API turns.
// Computed outside the try so the catch can still record a failed step.
const shouldLog = action.type !== "screenshot";
try {
let executionResult: ActionExecutionResult | undefined;
// Try to inject cursor before each action if enabled
if (this.highlightCursor) {
try {
@ -129,11 +149,8 @@ export class V3CuaAgentHandler {
}
}
await new Promise((r) => setTimeout(r, 300));
// Skip logging for screenshot actions - they're no-ops; the CUA client
// takes its own screenshot via screenshotProvider between API turns.
const shouldLog = action.type !== "screenshot";
if (shouldLog) {
await FlowLogger.runWithLogging(
executionResult = await FlowLogger.runWithLogging(
{
eventType: `V3Cua${toTitleCase(action.type)}`, // e.g. "V3CuaClick"
data: {
@ -145,10 +162,16 @@ export class V3CuaAgentHandler {
[action],
);
} else {
await this.executeAction(action);
executionResult = await this.executeAction(action);
}
action.timestamp = Date.now();
if (shouldLog && this.evidenceCallback) {
await this.emitCuaActionStep(
action,
inferToolOutput(executionResult ?? { success: true }),
);
}
await new Promise((r) => setTimeout(r, waitBetween));
} catch (error) {
@ -158,6 +181,30 @@ export class V3CuaAgentHandler {
message: `Error executing action ${action.type}: ${msg}`,
level: 0,
});
// Record the failed action as an ok:false step (with a best-effort
// post-failure probe, since a throwing action can still partially
// mutate the page) before rethrowing — otherwise the failure is
// dropped from the persisted trajectory. Evidence emission must never
// mask the original action error.
if (shouldLog && this.evidenceCallback) {
try {
await this.emitCuaActionStep(action, {
ok: false,
result: undefined,
error: msg,
});
} catch (evidenceError) {
this.logger({
category: "agent",
message: `Failed to record failed-action evidence: ${
evidenceError instanceof Error
? evidenceError.message
: String(evidenceError)
}`,
level: 1,
});
}
}
throw error;
}
});
@ -184,6 +231,11 @@ export class V3CuaAgentHandler {
: optionsOrInstruction;
this.setSafetyConfirmationHandler(options.callbacks?.onSafetyConfirmation);
this.evidenceCallback = wrapEvidenceCallback(
options.callbacks?.onEvidence,
this.logger,
);
this.lastAgentScreenshotUrl = undefined;
this.highlightCursor = options.highlightCursor !== false;
this.currentInstruction = options.instruction;
@ -239,7 +291,28 @@ export class V3CuaAgentHandler {
let result: AgentResult;
try {
result = await this.agent.execute({ options, logger: this.logger });
if (this.evidenceCallback) {
let finalUrl = "";
try {
finalUrl = (await this.v3.context.awaitActivePage()).url();
} catch {
finalUrl = this.lastAgentScreenshotUrl ?? "";
}
const observation = await captureProbeEvidence({
v3: this.v3,
url: finalUrl,
logger: this.logger,
warningMessage: "Warning: CUA final probe failed",
});
await this.evidenceCallback({
type: "final_answer",
message: result.message,
output: result.output,
observation,
});
}
} finally {
this.evidenceCallback = undefined;
this.captchaSolver?.dispose();
this.captchaSolver = null;
}
@ -658,6 +731,10 @@ export class V3CuaAgentHandler {
const screenshotBuffer = await page.screenshot({ fullPage: false });
const currentUrl = page.url();
// Mirror the same buffer the CUA client receives as agent evidence.
await this.emitCuaScreenshot(screenshotBuffer, currentUrl);
return await this.agentClient.captureScreenshot({
base64Image: screenshotBuffer.toString("base64"),
currentUrl,
@ -767,6 +844,75 @@ export class V3CuaAgentHandler {
}
}
/**
* Emit a pre-action CUA screenshot — the exact buffer the model received
* as input. Tier-1 evidence (agent-mirrored); the tier-2 probe is taken
* separately in emitCuaActionStep after the action runs, so the recorder
* can compare what the model saw against what the page actually showed
* once the keystrokes/clicks landed.
*/
private async emitCuaScreenshot(
screenshot: Buffer,
url: string,
): Promise<void> {
if (!this.evidenceCallback) return;
this.lastAgentScreenshotUrl = url;
await this.evidenceCallback({
type: "screenshot",
screenshot,
url,
evidenceRole: "agent",
});
}
private async emitCuaActionStep(
action: AgentAction,
toolOutput: AgentStepFinishedEvent["toolOutput"],
): Promise<void> {
let pageUrl =
typeof action.pageUrl === "string"
? action.pageUrl
: (this.lastAgentScreenshotUrl ?? "");
try {
pageUrl = (await this.v3.context.awaitActivePage()).url();
} catch {
// Keep the best pre-action URL fallback.
}
const actionArgs = Object.fromEntries(
Object.entries(action).filter(
([key]) => key !== "screenshot" && key !== "timestamp",
),
);
const reasoning =
typeof action.reasoning === "string"
? action.reasoning
: typeof action.action === "string"
? action.action
: "";
await this.evidenceCallback?.({
type: "step_finished",
actionName: String(action.type),
actionArgs,
reasoning,
toolOutput,
});
// Post-action tier-2 probe. The pre-action screenshot from
// screenshotProvider is what the model SAW; this one shows what the
// page actually LOOKS LIKE after the action ran. Without this the
// verifier has no visual evidence that keystrokes/clicks landed, and
// has to trust the action history alone.
await emitPostStepProbeEvidence({
v3: this.v3,
url: pageUrl,
evidenceCallback: this.evidenceCallback,
logger: this.logger,
warningMessage: "Warning: CUA post-action probe failed",
});
}
private async injectCursor(): Promise<void> {
try {
const page = await this.v3.context.awaitActivePage();

View File

@ -24,6 +24,16 @@ import { tool } from "ai";
import { getAISDKLanguageModel } from "./llm/LLMProvider.js";
import { __internalCreateInMemoryAgentCacheHandle } from "./cache/serverAgentCache.js";
import { maybeRunShutdownSupervisorFromArgv } from "./shutdown/supervisor.js";
import {
buildAgentEvidenceFromStepFinished,
loadTrajectoryFromDisk,
mergeAgentEvidence,
nextResultFilename,
normalizeRubric,
redactInlineImagePayloads,
shouldPersistTrajectory,
writeTrajectoryDir,
} from "./verifier/index.js";
export { V3 } from "./v3.js";
export { V3 as Stagehand } from "./v3.js";
@ -64,6 +74,36 @@ export type {
V3EvaluatorConstructorOptions,
V3EvaluatorOptions,
} from "../v3Evaluator.js";
export type {
Trajectory,
TrajectoryStep,
TrajectoryStatus,
TrajectoryUsage,
TaskSpec,
Rubric,
RubricCriterion,
AgentEvidence,
AgentEvidenceModality,
ProbeEvidence,
ToolOutput,
Verifier,
EvaluationResult,
CriterionScore,
FirstPointOfFailure,
TaskValidity,
VerifierFinding,
VerifierRawSteps,
} from "./verifier/index.js";
export {
buildAgentEvidenceFromStepFinished,
loadTrajectoryFromDisk,
mergeAgentEvidence,
nextResultFilename,
normalizeRubric,
redactInlineImagePayloads,
shouldPersistTrajectory,
writeTrajectoryDir,
} from "./verifier/index.js";
export { tool } from "ai";
export { getAISDKLanguageModel } from "./llm/LLMProvider.js";
export { __internalCreateInMemoryAgentCacheHandle } from "./cache/serverAgentCache.js";
@ -114,6 +154,14 @@ const StagehandDefault = {
toJsonSchema,
connectToMCPServer,
V3Evaluator,
buildAgentEvidenceFromStepFinished,
loadTrajectoryFromDisk,
mergeAgentEvidence,
nextResultFilename,
normalizeRubric,
redactInlineImagePayloads,
shouldPersistTrajectory,
writeTrajectoryDir,
tool,
getAISDKLanguageModel,
__internalCreateInMemoryAgentCacheHandle,

View File

@ -1,6 +1,5 @@
import type { LanguageModelV2Middleware } from "@ai-sdk/provider";
import {
ExperimentalNotConfiguredError,
UnsupportedAISDKModelProviderError,
UnsupportedModelError,
UnsupportedModelProviderError,
@ -70,6 +69,37 @@ const AISDKProvidersWithAPIKey: Record<string, AISDKCustomProvider> = {
gateway: createGateway,
};
type AISDKProviderClientOptions = ClientOptions & Record<string, unknown>;
export function toAISDKClientOptions(
subProvider: string,
clientOptions?: ClientOptions,
): AISDKProviderClientOptions | undefined {
if (!clientOptions || subProvider !== "vertex") {
return clientOptions as AISDKProviderClientOptions | undefined;
}
const { auth, providerOptions, ...rest } = clientOptions;
const vertexOptions = providerOptions?.vertex;
return {
...rest,
...(vertexOptions ?? {}),
...(auth?.type === "googleServiceAccount"
? {
googleAuthOptions: {
credentials: auth.credentials,
...(auth.scopes ? { scopes: auth.scopes } : {}),
...(auth.projectId ? { projectId: auth.projectId } : {}),
...(auth.universeDomain
? { universeDomain: auth.universeDomain }
: {}),
},
}
: {}),
};
}
const modelToProviderMap: { [key in AvailableModel]: ModelProvider } = {
"gpt-4.1": "openai",
"gpt-4.1-mini": "openai",
@ -106,9 +136,12 @@ export function getAISDKLanguageModel(
clientOptions?: ClientOptions,
middleware?: LanguageModelV2Middleware,
) {
const aiSdkClientOptions = toAISDKClientOptions(subProvider, clientOptions);
const hasValidOptions =
clientOptions &&
Object.values(clientOptions).some((v) => v !== undefined && v !== null);
aiSdkClientOptions &&
Object.values(aiSdkClientOptions).some(
(v) => v !== undefined && v !== null,
);
let model;
if (hasValidOptions) {
@ -119,7 +152,7 @@ export function getAISDKLanguageModel(
Object.keys(AISDKProvidersWithAPIKey),
);
}
const provider = creator(clientOptions);
const provider = creator(aiSdkClientOptions as ClientOptions);
model = provider(subModelName);
} else {
const provider = AISDKProviders[subProvider];
@ -163,13 +196,6 @@ export class LLMProvider {
const firstSlashIndex = modelName.indexOf("/");
const subProvider = modelName.substring(0, firstSlashIndex);
const subModelName = modelName.substring(firstSlashIndex + 1);
if (
subProvider === "vertex" &&
!options?.disableAPI &&
!options?.experimental
) {
throw new ExperimentalNotConfiguredError("Vertex provider");
}
const effectiveMiddleware = options?.middleware ?? this.middleware;
const languageModel = getAISDKLanguageModel(

View File

@ -15,6 +15,7 @@ import {
import { LogLine } from "./logs.js";
import { ClientOptions } from "./model.js";
import { StagehandZodObject } from "../../zodCompat.js";
import type { AgentEvidenceCallback } from "./agentEvidenceEvents.js";
// Re-export ModelMessage for consumers who want to use it for conversation continuation
export type { ModelMessage } from "ai";
@ -136,6 +137,11 @@ export interface AgentCallbacks {
onStepFinish?:
| GenerateTextOnStepFinishCallback<ToolSet>
| StreamTextOnStepFinishCallback<ToolSet>;
/**
* Callback called when Stagehand captures agent-run evidence such as
* screenshots, completed tool/action steps, or post-action observations.
*/
onEvidence?: AgentEvidenceCallback;
}
/**
@ -456,6 +462,7 @@ export const AVAILABLE_CUA_MODELS = [
"openai/computer-use-preview-2025-03-11",
"anthropic/claude-opus-4-5-20251101",
"anthropic/claude-opus-4-6",
"anthropic/claude-opus-4-8",
"anthropic/claude-sonnet-4-6",
"anthropic/claude-haiku-4-5",
"anthropic/claude-haiku-4-5-20251001",

View File

@ -0,0 +1,99 @@
/**
* Evidence events emitted through AgentExecuteOptions.callbacks.onEvidence.
*
* These events describe observations made by Stagehand during an agent run.
* They are emitted in temporal order; consumers should treat the stream as
* sequential. An agent-role screenshot applies to every subsequent
* step_finished until a newer agent-role screenshot replaces it — a CUA
* provider may choose multiple actions from a single screenshot, so each of
* those steps shares that frame. A step_observed/probe applies to all
* step_finished events received since the last probe. Verifier-specific
* storage and normalization live in the evals/verifier layers.
*/
export type AgentEvidenceRole = "probe" | "agent";
export type AgentEvidenceEvent =
| AgentScreenshotEvidenceEvent
| AgentStepFinishedEvent
| AgentStepObservedEvent
| AgentFinalAnswerEvent;
/**
* Screenshot captured during an agent run.
*
* In DOM/hybrid mode, post-tool screenshots are probe evidence. In CUA mode,
* screenshots captured by the screenshot provider are agent evidence because
* they are the exact bytes sent to the provider.
*/
export interface AgentScreenshotEvidenceEvent {
type: "screenshot";
/** PNG bytes from page.screenshot(). */
screenshot: Buffer;
/** Page URL at the time of capture. */
url: string;
/** Role this screenshot plays in downstream evidence collection. */
evidenceRole: AgentEvidenceRole;
}
/**
* One completed agent tool/action step.
*/
export interface AgentStepFinishedEvent {
type: "step_finished";
/** Name of the tool/action that ran, e.g. "act", "extract", "click". */
actionName: string;
/** Arguments passed to the tool/action. */
actionArgs: Record<string, unknown>;
/** Agent textual reasoning for the step, when available. */
reasoning: string;
/** Outcome of the tool/action as seen by Stagehand. */
toolOutput: {
ok: boolean;
/** Native return value from the tool/action. */
result: unknown;
error?: string;
};
}
/**
* Independent post-step browser observation. Emitted once per agent turn;
* consumers apply it to every step_finished received since the previous probe.
*/
export interface AgentStepObservedEvent {
type: "step_observed";
/** Page URL after the step's tool/action execution. */
url: string;
/** Accessibility tree snapshot, when captured. */
ariaTree?: string;
}
export interface AgentFinalObservation {
/** Page URL at the time of terminal capture. */
url: string;
/** PNG bytes from page.screenshot(), when capture succeeds. */
screenshot?: Buffer;
/** Accessibility tree snapshot, when captured. */
ariaTree?: string;
}
/** Final answer emitted by the agent, when available. */
export interface AgentFinalAnswerEvent {
type: "final_answer";
/** The agent's final summary message. */
message: string;
/** Optional structured output if the agent's output schema was set. */
output?: Record<string, unknown>;
/**
* Independent terminal browser observation captured after the agent finishes.
*
* This preserves the legacy verifier behavior of evaluating against a final
* page screenshot even when the last agent output is a final answer rather
* than a browser action.
*/
observation?: AgentFinalObservation;
}
export type AgentEvidenceCallback = (
event: AgentEvidenceEvent,
) => PromiseLike<void> | void;

View File

@ -73,9 +73,13 @@ export const GoogleServiceAccountCredentialsSchema = z
.strict()
.meta({ id: "GoogleServiceAccountCredentials" });
export const GoogleAuthOptionsSchema = z
export const GoogleServiceAccountAuthSchema = z
.object({
credentials: GoogleServiceAccountCredentialsSchema.optional().meta({
type: z.literal("googleServiceAccount").meta({
description:
"Use inline Google Cloud service account credentials for provider authentication",
}),
credentials: GoogleServiceAccountCredentialsSchema.meta({
description: "Google Cloud service account credentials",
}),
scopes: z
@ -92,18 +96,44 @@ export const GoogleAuthOptionsSchema = z
}),
})
.strict()
.meta({ id: "GoogleAuthOptions" });
.meta({ id: "GoogleServiceAccountAuth" });
export const ModelConfigObjectSchema = z
export const ModelAuthSchema = z
.discriminatedUnion("type", [GoogleServiceAccountAuthSchema])
.meta({ id: "ModelAuth" });
export const VertexProviderOptionsSchema = z
.object({
project: z.string().meta({
description: "Google Cloud project ID for Vertex AI models",
example: "my-gcp-project",
}),
location: z.string().meta({
description: "Google Cloud location for Vertex AI models",
example: "us-central1",
}),
baseURL: z.string().url().optional().meta({
description: "Base URL for the Vertex AI provider",
}),
headers: z.record(z.string(), z.string()).optional().meta({
description:
"Custom headers sent with every request to the Vertex AI provider",
}),
})
.strict()
.meta({ id: "VertexProviderOptions" });
export const ModelProviderOptionsSchema = z
.object({
vertex: VertexProviderOptionsSchema.meta({
description: "Vertex AI provider-specific settings",
}),
})
.strict()
.meta({ id: "ModelProviderOptions" });
const ModelConfigBaseSchema = z
.object({
provider: z
.enum(["openai", "anthropic", "google", "microsoft", "bedrock", "vertex"])
.optional()
.meta({
description:
"AI provider for the model (or provide a baseURL endpoint instead)",
example: "openai",
}),
modelName: z.string().meta({
description:
"Model name string with provider prefix (e.g., 'openai/gpt-5-nano')",
@ -121,19 +151,38 @@ export const ModelConfigObjectSchema = z
description:
"Custom headers sent with every request to the model provider",
}),
project: z.string().optional().meta({
description: "Google Cloud project ID for Vertex AI models",
example: "my-gcp-project",
}),
location: z.string().optional().meta({
description: "Google Cloud location for Vertex AI models",
example: "us-central1",
}),
googleAuthOptions: GoogleAuthOptionsSchema.optional().meta({
description:
"google-auth-library options used to authenticate Vertex AI models",
}),
})
.strict();
export const GenericModelConfigObjectSchema = ModelConfigBaseSchema.extend({
provider: z
.enum(["openai", "anthropic", "google", "microsoft", "bedrock"])
.optional()
.meta({
description:
"AI provider for the model (or provide a baseURL endpoint instead)",
example: "openai",
}),
})
.strict()
.meta({ id: "GenericModelConfigObject" });
export const VertexModelConfigObjectSchema = ModelConfigBaseSchema.extend({
provider: z.literal("vertex").meta({
description: "Vertex AI model provider",
}),
auth: ModelAuthSchema.meta({
description: "Vertex provider authentication configuration",
}),
providerOptions: ModelProviderOptionsSchema.meta({
description: "Vertex provider-specific model configuration",
}),
})
.strict()
.meta({ id: "VertexModelConfigObject" });
export const ModelConfigObjectSchema = z
.union([VertexModelConfigObjectSchema, GenericModelConfigObjectSchema])
.meta({ id: "ModelConfigObject" });
/** Model configuration */
@ -1161,6 +1210,21 @@ export const Operations = {
// Shared types
export type Action = z.infer<typeof ActionSchema>;
export type ModelConfig = z.infer<typeof ModelConfigSchema>;
export type GenericModelConfigObject = z.infer<
typeof GenericModelConfigObjectSchema
>;
export type VertexModelConfigObject = z.infer<
typeof VertexModelConfigObjectSchema
>;
export type GoogleServiceAccountCredentials = z.infer<
typeof GoogleServiceAccountCredentialsSchema
>;
export type GoogleServiceAccountAuth = z.infer<
typeof GoogleServiceAccountAuthSchema
>;
export type ModelAuth = z.infer<typeof ModelAuthSchema>;
export type VertexProviderOptions = z.infer<typeof VertexProviderOptionsSchema>;
export type ModelProviderOptions = z.infer<typeof ModelProviderOptionsSchema>;
export type BrowserConfig = z.infer<typeof BrowserConfigSchema>;
export type SessionIdParams = z.infer<typeof SessionIdParamsSchema>;

View File

@ -1,4 +1,5 @@
export * from "./agent.js";
export * from "./agentEvidenceEvents.js";
// Export api.ts under namespace to avoid conflicts with methods.ts types
export * as Api from "./api.js";
// Also export BrowserbaseRegion directly for convenience

View File

@ -1,5 +1,4 @@
import type { ClientOptions as AnthropicClientOptionsBase } from "@anthropic-ai/sdk";
import type { GoogleVertexProviderSettings as GoogleVertexProviderSettingsBase } from "@ai-sdk/google-vertex";
import type {
LanguageModelV2,
LanguageModelV2Middleware,
@ -31,19 +30,26 @@ export interface GoogleServiceAccountCredentials {
universe_domain?: string;
}
export interface GoogleVertexAuthOptions {
credentials?: GoogleServiceAccountCredentials;
export interface GoogleServiceAccountAuth {
type: "googleServiceAccount";
credentials: GoogleServiceAccountCredentials;
scopes?: string | string[];
projectId?: string;
universeDomain?: string;
}
export type GoogleVertexProviderSettings = Pick<
GoogleVertexProviderSettingsBase,
"project" | "location" | "headers" | "baseURL"
> & {
googleAuthOptions?: GoogleVertexAuthOptions;
};
export type ModelAuth = GoogleServiceAccountAuth;
export interface VertexProviderOptions {
project: string;
location: string;
baseURL?: string;
headers?: Record<string, string>;
}
export interface ModelProviderOptions {
vertex: VertexProviderOptions;
}
export type AnthropicJsonSchemaObject = {
definitions?: {
@ -113,13 +119,11 @@ export type ModelProvider =
*/
export type ThinkingEffort = "none" | "low" | "medium" | "high" | "max";
export type ClientOptions = (
| OpenAIClientOptions
| AnthropicClientOptions
| GoogleVertexProviderSettings
) & {
export type ClientOptions = (OpenAIClientOptions | AnthropicClientOptions) & {
apiKey?: string;
provider?: AgentProviderType;
auth?: ModelAuth;
providerOptions?: ModelProviderOptions;
baseURL?: string;
/** OpenAI organization ID */
organization?: string;

View File

@ -538,6 +538,26 @@ export class V3 {
);
}
private getApiDefaultModelConfig(): ModelConfiguration | undefined {
const clientOptions = Object.fromEntries(
Object.entries(this.modelClientOptions).filter(
([key, value]) => key !== "apiKey" && value !== undefined,
),
);
if (Object.keys(clientOptions).length === 0) {
return undefined;
}
return {
modelName: this.modelName,
...clientOptions,
...((this.modelClientOptions as { apiKey?: string }).apiKey
? { apiKey: (this.modelClientOptions as { apiKey?: string }).apiKey }
: {}),
} as ModelConfiguration;
}
private resolveLlmClient(model?: ModelConfiguration): LLMClient {
if (!model) {
return this.llmClient;
@ -1043,6 +1063,7 @@ export class V3 {
const { sessionId, available } = await this.apiClient.init({
modelName: this.modelName,
modelApiKey: this.modelClientOptions.apiKey,
defaultModelConfig: this.getApiDefaultModelConfig(),
domSettleTimeoutMs: this.domSettleTimeoutMs,
verbose: this.verbose,
systemPrompt: this.opts.systemPrompt,

View File

@ -0,0 +1,118 @@
import type { AgentStepFinishedEvent } from "../types/public/agentEvidenceEvents.js";
import type { AgentEvidence } from "./types.js";
export const REDACTED_INLINE_IMAGE = "[redacted inline image payload]";
const INLINE_IMAGE_KEYS = new Set(["screenshotBase64"]);
function shouldRedactBase64Key(key: string, actionName?: string): boolean {
return (
INLINE_IMAGE_KEYS.has(key) ||
(actionName === "screenshot" && key === "base64")
);
}
export function collectInlineImagePayloads(
value: unknown,
actionName?: string,
out: string[] = [],
): string[] {
if (!value || typeof value !== "object") return out;
if (Buffer.isBuffer(value)) return out;
if (Array.isArray(value)) {
for (const item of value) {
collectInlineImagePayloads(item, actionName, out);
}
return out;
}
for (const [key, nested] of Object.entries(value)) {
if (shouldRedactBase64Key(key, actionName) && typeof nested === "string") {
out.push(nested);
continue;
}
collectInlineImagePayloads(nested, actionName, out);
}
return out;
}
export function redactInlineImagePayloads(
value: unknown,
actionName?: string,
): unknown {
if (!value || typeof value !== "object") return value;
if (Buffer.isBuffer(value)) return value;
if (Array.isArray(value)) {
return value.map((item) => redactInlineImagePayloads(item, actionName));
}
const out: Record<string, unknown> = {};
for (const [key, nested] of Object.entries(value)) {
out[key] =
shouldRedactBase64Key(key, actionName) && typeof nested === "string"
? REDACTED_INLINE_IMAGE
: redactInlineImagePayloads(nested, actionName);
}
return out;
}
export function mergeAgentEvidence(
...parts: Array<AgentEvidence | undefined>
): AgentEvidence {
return {
modalities: parts.flatMap((p) => p?.modalities ?? []),
};
}
export function buildAgentEvidenceFromStepFinished(
event: AgentStepFinishedEvent,
): AgentEvidence {
const modalities: AgentEvidence["modalities"] = [];
if (event.reasoning) {
modalities.push({ type: "text", content: event.reasoning });
}
const result = event.toolOutput.result;
if (result === undefined || result === null) {
return { modalities };
}
if (typeof result === "string") {
modalities.push({ type: "text", content: result });
} else if (
typeof result === "number" ||
typeof result === "boolean" ||
typeof result === "bigint"
) {
modalities.push({ type: "text", content: String(result) });
} else if (Buffer.isBuffer(result)) {
modalities.push({
type: "image",
bytes: result,
mediaType: "image/png",
});
} else if (typeof result === "object") {
for (const imageBase64 of collectInlineImagePayloads(
result,
event.actionName,
)) {
try {
modalities.push({
type: "image",
bytes: Buffer.from(imageBase64, "base64"),
mediaType: "image/png",
});
} catch {
// Malformed base64; skip the image and keep the JSON modality.
}
}
modalities.push({
type: "json",
content: redactInlineImagePayloads(result, event.actionName),
});
}
return { modalities };
}

View File

@ -0,0 +1,37 @@
/**
* Public re-exports for the verifier subsystem.
*/
export type {
AgentEvidence,
AgentEvidenceModality,
CriterionScore,
EvaluationResult,
FirstPointOfFailure,
ProbeEvidence,
Rubric,
RubricCriterion,
TaskSpec,
TaskValidity,
ToolOutput,
Trajectory,
TrajectoryStatus,
TrajectoryStep,
TrajectoryUsage,
Verifier,
VerifierFinding,
VerifierRawSteps,
} from "./types.js";
export {
buildAgentEvidenceFromStepFinished,
collectInlineImagePayloads,
mergeAgentEvidence,
redactInlineImagePayloads,
REDACTED_INLINE_IMAGE,
} from "./evidenceNormalization.js";
export {
loadTrajectoryFromDisk,
nextResultFilename,
normalizeRubric,
shouldPersistTrajectory,
writeTrajectoryDir,
} from "./trajectory.js";

View File

@ -0,0 +1,391 @@
import fs from "node:fs/promises";
import path from "node:path";
import type {
AgentEvidenceModality,
ProbeEvidence,
Rubric,
Trajectory,
TrajectoryStep,
} from "./types.js";
import { redactInlineImagePayloads } from "./evidenceNormalization.js";
type RawRubricCriterion = {
criterion: unknown;
description: unknown;
max_points?: unknown;
maxPoints?: unknown;
condition?: unknown;
};
type RawRubric = {
items?: unknown;
};
type PersistedProbeEvidence = ProbeEvidence & {
screenshotPath?: string;
};
/**
* Convert dataset or generated rubric JSON into the public Stagehand shape.
* Snake-case dataset fields are accepted here so serialized quirks do not leak
* into the canonical rubric type.
*/
export function normalizeRubric(rubric: unknown): Rubric | undefined {
if (rubric == null) return undefined;
if (typeof rubric !== "object") {
throw new TypeError("Rubric must be an object");
}
const rawRubric = rubric as RawRubric;
if (!Array.isArray(rawRubric.items)) {
throw new TypeError("Rubric is missing an items array");
}
return {
items: rawRubric.items.map((item) => {
const criterion = normalizeRequiredString(item.criterion, "criterion");
const description = normalizeRequiredString(
item.description,
"description",
);
const maxPoints = normalizeMaxPoints(item);
if (typeof maxPoints !== "number" || !Number.isFinite(maxPoints)) {
throw new TypeError(
`Rubric criterion "${criterion}" is missing a numeric maxPoints value`,
);
}
return {
criterion,
description,
maxPoints,
...(typeof item.condition === "string" && {
condition: item.condition,
}),
};
}),
};
}
function normalizeRequiredString(value: unknown, fieldName: string): string {
if (typeof value === "string" && value.length) {
return value;
}
throw new TypeError(`Rubric criterion is missing a ${fieldName} value`);
}
function normalizeMaxPoints(item: RawRubricCriterion): unknown {
return item.maxPoints ?? item.max_points;
}
function normalizeResultLabel(label?: string): string {
return (label ?? `rescore-${new Date().toISOString()}`).replace(
/[^A-Za-z0-9._-]/g,
"_",
);
}
// ─────────────────────────────────────────────────────────────────────────────
// On-disk loader
// ─────────────────────────────────────────────────────────────────────────────
/**
* Hydrate a Trajectory from the on-disk directory layout written by
* TrajectoryRecorder.persist(). Used by the offline re-scoring CLI (`bench
* verify`) and by any consumer that wants to feed a saved trajectory back
* into V3Evaluator.verify() without running an agent.
*
* Reverses the recorder's serialization tweaks:
* - `probeEvidence.screenshotPath` → read file into `probeEvidence.screenshot`.
* - Image modalities in `agentEvidence.modalities` carry `imagePath` on
* disk instead of raw Buffer; legacy `bytesBase64` fixtures are also
* accepted.
*
* @param dir absolute or cwd-relative path to a `<run-id>/<task-id>/` directory.
*/
export async function loadTrajectoryFromDisk(dir: string): Promise<Trajectory> {
const fs = await import("node:fs/promises");
const path = await import("node:path");
const trajectoryDir = path.resolve(dir);
const trajectoryPath = path.join(trajectoryDir, "trajectory.json");
const raw = await fs.readFile(trajectoryPath, "utf8");
const parsed = JSON.parse(raw) as Trajectory & {
finalObservation?: PersistedProbeEvidence;
steps: Array<
TrajectoryStep & {
agentEvidence: {
modalities: Array<
| { type: "text"; content: string }
| {
type: "image";
mediaType: string;
// On-disk forms. Current writer externalizes bytes to
// imagePath; bytesBase64 is accepted for older fixtures.
bytes?: unknown;
bytesBase64?: string;
imagePath?: string;
}
| { type: "json"; content: unknown }
>;
};
probeEvidence: PersistedProbeEvidence;
}
>;
};
const resolveWithinTrajectoryDir = (
candidate: string,
fieldName = "screenshotPath",
): string => {
const resolved = path.resolve(trajectoryDir, candidate);
const relative = path.relative(trajectoryDir, resolved);
const outside =
relative === ".." ||
relative.startsWith(`..${path.sep}`) ||
path.isAbsolute(relative);
if (outside) {
throw new Error(
`Trajectory ${fieldName} escapes trajectory directory: ${candidate}`,
);
}
return resolved;
};
const hydrateProbeScreenshot = async (
probe: PersistedProbeEvidence | undefined,
): Promise<void> => {
if (probe?.screenshotPath && !probe.screenshot) {
const resolved = resolveWithinTrajectoryDir(probe.screenshotPath);
try {
probe.screenshot = await fs.readFile(resolved);
} catch {
// Missing screenshot file: leave probe.screenshot unset. The verifier's
// evidence_insufficient path will handle it.
}
}
};
for (const step of parsed.steps) {
// Rehydrate tier-2 probe screenshot from its on-disk file reference.
await hydrateProbeScreenshot(step.probeEvidence);
// Decode image modalities from disk references back to Buffer.
if (step.agentEvidence?.modalities) {
const modalities: AgentEvidenceModality[] = [];
for (const m of step.agentEvidence.modalities) {
// The on-disk shape carries imagePath/bytesBase64 instead of bytes,
// so we look through `unknown` rather than rely on the typed union.
const raw = m as unknown as {
bytesBase64?: string;
imagePath?: string;
};
if (m.type === "image" && typeof raw.bytesBase64 === "string") {
modalities.push({
type: "image" as const,
bytes: Buffer.from(raw.bytesBase64, "base64"),
mediaType: m.mediaType,
});
continue;
}
if (m.type === "image" && typeof raw.imagePath === "string") {
const resolved = resolveWithinTrajectoryDir(
raw.imagePath,
"imagePath",
);
try {
modalities.push({
type: "image" as const,
bytes: await fs.readFile(resolved),
mediaType: m.mediaType,
});
} catch {
// Missing agent image file: omit that image modality. The
// verifier's evidence_insufficient path will handle missing bytes.
}
continue;
}
modalities.push(m as AgentEvidenceModality);
}
step.agentEvidence.modalities = modalities;
}
}
await hydrateProbeScreenshot(parsed.finalObservation);
return parsed;
}
/**
* Build a `result*.json` filename for persisted evaluator output.
*
* Convention: the live run writes `result.json`; offline re-score attempts use
* a label-based name (e.g., `result_rescore-2026-05-11.json`) so they coexist
* without collisions and remain easy to diff.
*/
export function nextResultFilename(label?: string): string {
return `result_${normalizeResultLabel(label)}.json`;
}
/**
* Default persistence policy: explicit override, then env, then "on unless CI".
*/
export function shouldPersistTrajectory(
override: boolean | undefined,
): boolean {
if (override !== undefined) return override;
const env = process.env.VERIFIER_PERSIST_TRAJECTORIES?.toLowerCase();
if (env === "1" || env === "true") return true;
if (env === "0" || env === "false") return false;
return !process.env.CI;
}
/**
* Write the on-disk trajectory layout under `dir`:
*
* <dir>/
* ├── task_data.json
* ├── trajectory.json (screenshots referenced by path)
* ├── screenshots/
* │ ├── probe/<N>.png
* │ └── agent/<N>[_M].png
* ├── scores/ (empty; populated separately)
* └── core.log
*
* Image bytes are externalized to PNG files; the in-memory Trajectory is left
* untouched so callers can keep using it after persistence.
*/
export async function writeTrajectoryDir(
dir: string,
trajectory: Trajectory,
): Promise<void> {
await fs.mkdir(dir, { recursive: true });
await fs.mkdir(path.join(dir, "screenshots", "probe"), { recursive: true });
await fs.mkdir(path.join(dir, "screenshots", "agent"), { recursive: true });
const serializableSteps: unknown[] = [];
// A single post-turn probe is fanned across every step of a multi-tool turn,
// and a single agent screenshot is shared across every action a CUA provider
// chose from it, so the same Buffer is shared by reference. Dedupe by
// identity: write the PNG once and point every sharing step at the same file.
const probePathByBuffer = new Map<Buffer, string>();
const agentPathByBuffer = new Map<Buffer, string>();
for (const [i, step] of trajectory.steps.entries()) {
const probe: ProbeEvidence = { ...step.probeEvidence };
if (probe.screenshot) {
let relPath = probePathByBuffer.get(probe.screenshot);
if (!relPath) {
relPath = `screenshots/probe/${i + 1}.png`;
await fs.writeFile(path.join(dir, relPath), probe.screenshot);
probePathByBuffer.set(probe.screenshot, relPath);
}
probe.screenshotPath = relPath;
delete probe.screenshot;
}
const imageModalities = step.agentEvidence.modalities.filter(
(m) => m.type === "image",
);
const multipleImages = imageModalities.length > 1;
let imageSeq = 0;
const modalities: unknown[] = [];
for (const m of step.agentEvidence.modalities) {
if (m.type !== "image") {
modalities.push(
m.type === "json"
? {
...m,
content: redactInlineImagePayloads(m.content, step.actionName),
}
: m,
);
continue;
}
let relPath = agentPathByBuffer.get(m.bytes);
if (!relPath) {
const suffix = multipleImages ? `_${imageSeq}` : "";
relPath = `screenshots/agent/${i + 1}${suffix}.png`;
await fs.writeFile(path.join(dir, relPath), m.bytes);
agentPathByBuffer.set(m.bytes, relPath);
}
modalities.push({
type: "image",
imagePath: relPath,
mediaType: m.mediaType,
});
imageSeq += 1;
}
serializableSteps.push({
...step,
probeEvidence: probe,
agentEvidence: { modalities },
toolOutput: {
...step.toolOutput,
result: redactInlineImagePayloads(
step.toolOutput.result,
step.actionName,
),
},
});
}
const finalObservation: ProbeEvidence | undefined =
trajectory.finalObservation === undefined
? undefined
: { ...trajectory.finalObservation };
if (finalObservation?.screenshot) {
const relPath = "screenshots/probe/final.png";
await fs.writeFile(path.join(dir, relPath), finalObservation.screenshot);
finalObservation.screenshotPath = relPath;
delete finalObservation.screenshot;
}
// Image modalities carry imagePath instead of raw bytes on disk; cast
// through unknown rather than widen Trajectory's type contract.
const serialized = {
...trajectory,
steps: serializableSteps,
...(finalObservation ? { finalObservation } : {}),
} as unknown;
await fs.writeFile(
path.join(dir, "trajectory.json"),
JSON.stringify(serialized, null, 2),
);
await fs.writeFile(
path.join(dir, "task_data.json"),
JSON.stringify(
{
task: trajectory.task,
status: trajectory.status,
finalAnswer: trajectory.finalAnswer ?? null,
},
null,
2,
),
);
await fs.mkdir(path.join(dir, "scores"), { recursive: true });
await fs.writeFile(path.join(dir, "core.log"), coreLog(trajectory));
}
function coreLog(trajectory: Trajectory): string {
return (
trajectory.steps
.map((step, i) =>
JSON.stringify({
step: i,
action: step.actionName,
url: step.probeEvidence.url ?? null,
ok: step.toolOutput.ok,
reasoning: step.reasoning || undefined,
}),
)
.join("\n") + "\n"
);
}

View File

@ -0,0 +1,299 @@
/**
* Shared verifier types for trajectories, rubrics, evidence, and results.
*
* The verifier consumes saved trajectories instead of a live browser. DOM and
* Hybrid runs preserve tool-return text/JSON evidence, while CUA runs preserve
* screenshots sent to the provider plus independent harness probes.
*/
/** Token usage for one or more LLM calls. Matches AgentResult.usage shape. */
export interface TrajectoryUsage {
input_tokens: number;
output_tokens: number;
reasoning_tokens?: number;
cached_input_tokens?: number;
inference_time_ms?: number;
}
/** A single criterion in a Stagehand rubric. */
export interface RubricCriterion {
/** Short name of the criterion (e.g., "Add ground beef to cart"). */
criterion: string;
/** What to evaluate and how to award partial credit. */
description: string;
/** Maximum points for this criterion. */
maxPoints: number;
/**
* Applicability rule for situational criteria. When this condition is not
* met, the criterion is excluded from scoring rather than counted as failed.
*/
condition?: string;
}
/** A rubric — list of criteria for a task. */
export interface Rubric {
items: RubricCriterion[];
}
/**
* Spec for a single task being verified. Carried both at runtime and into the
* verifier alongside the trajectory.
*/
export interface TaskSpec {
/** Stable identifier (e.g., "united_13" for WebTailBench, task_id for Mind2Web). */
id: string;
/** Task instruction shown to the agent. */
instruction: string;
/** Starting URL, if any. */
initUrl?: string;
/** Rubric carried by the dataset or generated by a verifier backend. */
precomputedRubric?: Rubric;
/** Optional reference answer (set when dataset ships one). */
expectedAnswer?: string;
}
/**
* A single modality unit in tier-1 agent evidence. Mirrors the shape of
* ModelMessage content parts so we can reproduce what the LLM ingested.
*/
export type AgentEvidenceModality =
| { type: "text"; content: string }
| { type: "image"; bytes: Buffer; mediaType: string }
| { type: "json"; content: unknown };
/**
* Tier 1 — exactly the bytes/strings/objects the agent's LLM ingested as the
* tool result for this step.
*
* Modes:
* - CUA: usually a single image modality (the screenshot sent to the provider).
* - Hybrid: tool result with optional screenshotBase64 → one image + one text.
* - DOM: tool returns (extract JSON, ariaTree text, etc.) → text/json modalities.
*/
export interface AgentEvidence {
modalities: AgentEvidenceModality[];
}
/**
* Tier 2 — independent harness probes around this step.
*
* If a probe wasn't captured, the field is absent (not null).
*/
export interface ProbeEvidence {
/** URL after the step's tool execution. */
url?: string;
/**
* Bus screenshot captured after the step. Path on disk is preferred once
* persisted; in-memory Buffer is used during a live run.
*/
screenshot?: Buffer;
/** Reference to the persisted screenshot file under the trajectory dir. */
screenshotPath?: string;
/** Accessibility tree snapshot. */
ariaTree?: string;
/** Verifier-requested probes, keyed by criterion id. */
onDemand?: Record<string, unknown>;
}
/** Outcome of a single tool execution as seen by the harness. */
export interface ToolOutput {
ok: boolean;
/**
* The tool's return value. Same payload that flowed into agentEvidence
* modalities, but in its native shape (e.g., the extract result, the act
* describe-string) rather than serialized for the LLM.
*/
result: unknown;
error?: string;
}
/** One step in a trajectory: action + reasoning + evidence + outcome. */
export interface TrajectoryStep {
actionName: string;
actionArgs: Record<string, unknown>;
/** From AgentAction.reasoning. May be empty for tools that don't surface reasoning. */
reasoning: string;
agentEvidence: AgentEvidence;
probeEvidence: ProbeEvidence;
toolOutput: ToolOutput;
}
/** Terminal status of the agent run. */
export type TrajectoryStatus = "complete" | "aborted" | "stalled" | "error";
/**
* Full trajectory for one task run.
*
* The on-disk layout is one directory per task:
*
* .trajectories/<run-id>/<task-id>/
* ├── task_data.json — TaskSpec + result metadata
* ├── trajectory.json — this object, with screenshotPath instead of bytes
* ├── screenshots/ — step probe/agent images plus final observation
* ├── scores/
* │ └── result.json — Result from V3Evaluator.verify()
* └── core.log — captured action log
*/
export interface Trajectory {
task: TaskSpec;
steps: TrajectoryStep[];
finalAnswer?: string;
/** Terminal page observation captured after the agent finishes. */
finalObservation?: ProbeEvidence;
status: TrajectoryStatus;
usage: TrajectoryUsage;
}
/** Score for a single rubric criterion after evidence analysis + rescoring. */
export interface CriterionScore {
/** Matches RubricCriterion.criterion (the criterion's short name). */
criterion: string;
/** Maximum possible points for this criterion. */
maxPoints: number;
/**
* Points earned post-evidence-analysis (paper's post_image_earned_points).
* Null if the criterion was conditional and its condition wasn't met (excluded
* from both numerator and denominator in the process score).
*/
earnedPoints: number | null;
/** Verifier's explanation for the score. */
explanation: string;
/**
* True if the criterion is conditional and its condition was determined to
* be met. Absent for non-conditional criteria.
*/
conditionMet?: boolean;
/**
* Set when the verifier had no evidence to ground this criterion in either
* tier. Per paper §2, treated as uncontrollable failure → full credit, but
* surfaced here so dashboards can flag low-confidence results.
*/
evidenceInsufficient?: boolean;
}
/**
* First-point-of-failure analysis (paper Step 9a). Identifies the earliest
* step where the agent's trajectory went off-track, using a structured error
* taxonomy (7 top-level categories, 1.1–7.4 sub-codes).
*/
export interface FirstPointOfFailure {
stepIndex: number;
/** Sub-code from the error taxonomy (e.g., "2.3" for a specific hallucination type). */
errorCode: string;
/** Top-level category name (Selection, Hallucination, etc.). */
category: string;
/** Verifier's reasoning for selecting this point. */
description?: string;
}
/**
* Structured observation surfaced by the verifier that another agent or
* tooling could act on. Findings are emitted opportunistically by Step 8
* (outcome verification) when the verifier notices actionable patterns —
* repeated tool-call failures, ambiguous task specs, evidence gaps, etc.
*
* Not produced for every task: when nothing actionable surfaces, the
* `findings` array on the EvaluationResult is empty. Consumers should treat the
* field as advisory, not as part of the formal score.
*/
export interface VerifierFinding {
/**
* Category of the observation. Open-ended enum — additional categories may
* be added as verifier backends surface new failure modes.
*/
category:
| "agent_tool_usage" // agent's tool calls had repeated issues (misclicks, wrong args, retries)
| "agent_strategy" // higher-level planning / decision-making problems
| "rubric_quality" // criteria were overly strict, ambiguous, or contradictory
| "trajectory_capture" // gaps in evidence (missing screenshots, empty steps)
| "task_specification" // task instruction was ambiguous / under- or over-specified
| "verifier_uncertainty" // verifier itself couldn't confidently decide
| "other";
/** Impact: info (FYI), warning (worth investigating), blocking (broke the task). */
severity: "info" | "warning" | "blocking";
/** What the verifier noticed. Plain prose, grounded in evidence from the trajectory. */
description: string;
/**
* Optional concrete next action another agent could take. Should be
* specific enough that it can be acted on without further reasoning —
* e.g., "Try double_click instead of triple_click to clear placeholder
* text on this form field."
*/
suggestedAction?: string;
/** Step indices in the trajectory where this pattern showed up. */
relatedSteps?: number[];
}
/** Stable debugging summary emitted by verifier backends. */
export interface VerifierRawSteps {
backend?: "legacy" | "verifier";
primaryIntent?: string;
reasoning?: string;
rubricSource?: "precomputed" | "generated" | "none";
approach?: "a" | "b";
optionalsMode?: "folded" | "separate" | "skip";
totalEarned?: number;
totalMax?: number;
evidenceImages?: number;
evidenceTexts?: number;
evidenceOriginalScreenshots?: number;
legacyEvaluation?: string;
screenshotCount?: number;
}
/** Task-validity classification (paper Step 10). */
export interface TaskValidity {
/** True if the task is underspecified / has multiple valid interpretations. */
isAmbiguous: boolean;
/** True if the task is impossible / illegal / NSFW / otherwise infeasible. */
isInvalid: boolean;
/** Optional sub-codes from the task-classification taxonomy. */
ambiguityCodes?: string[];
invalidTaskCodes?: string[];
}
/**
* Evaluator output. Legacy evaluation may only populate outcome fields; richer
* verifier backends can also populate process scoring and diagnostics.
*
* Process and outcome are deliberately independent when both are present:
* an agent can follow the right steps but get blocked (high process, low
* outcome), or succeed through an unexpected path (variable process, high
* outcome).
*/
export interface EvaluationResult {
/** Did the agent accomplish the task from the user's perspective? */
outcomeSuccess: boolean;
/** Human-readable explanation for the outcome. */
explanation?: string;
/** Aggregated earned/max across applicable criteria, in [0, 1]. */
processScore?: number;
/** Per-criterion breakdown after rescoring. */
perCriterion?: CriterionScore[];
/** Step 9a — first step where the trajectory went off-track, if any. */
firstPointOfFailure?: FirstPointOfFailure;
/** Step 10 — task-itself ambiguity / validity. */
taskValidity?: TaskValidity;
/**
* Ids (RubricCriterion.criterion strings) of criteria where neither tier of
* evidence resolved the question. Treated as uncontrollable → full credit,
* but flagged here so consumers can decide whether to discount the score.
*/
evidenceInsufficient?: string[];
/**
* Structured observations from the verifier that a downstream tool or
* follow-up agent could act on. Opportunistic — empty when the verifier
* doesn't notice anything actionable. Not part of the score; advisory.
*/
findings?: VerifierFinding[];
/** Debugging summary from the active evaluator backend. */
rawSteps?: VerifierRawSteps;
}
/**
* Verifier interface. Implementations consume a Trajectory and return an
* EvaluationResult — they MUST NOT touch a live browser.
*/
export interface Verifier {
verify(trajectory: Trajectory): Promise<EvaluationResult>;
}

View File

@ -2,11 +2,20 @@ import type { AvailableModel, ClientOptions } from "./v3/types/public/model.js";
import type {
EvaluateOptions,
BatchAskOptions,
EvaluationResult,
EvaluationResult as LegacyEvaluationResult,
} from "./v3/types/private/evaluator.js";
import { V3 } from "./v3/v3.js";
import { StagehandInvalidArgumentError } from "./v3/types/public/sdkErrors.js";
import { LegacyV3Evaluator } from "./v3LegacyEvaluator.js";
import type {
Trajectory,
TaskSpec,
EvaluationResult,
Rubric,
Verifier,
AgentEvidenceModality,
VerifierFinding,
} from "./v3/verifier/index.js";
const EVALUATOR_BACKEND_ENV = "STAGEHAND_EVALUATOR_BACKEND";
const DEFAULT_EVALUATOR_BACKEND: V3EvaluatorBackend = "legacy";
@ -36,7 +45,7 @@ type NormalizedConstructorOptions = {
backend?: V3EvaluatorBackend;
};
export class V3Evaluator {
export class V3Evaluator implements Verifier {
private readonly backend: V3EvaluatorBackend;
private readonly legacyEvaluator: LegacyV3Evaluator;
@ -60,23 +69,77 @@ export class V3Evaluator {
);
}
async ask(options: EvaluateOptions): Promise<EvaluationResult> {
async ask(options: EvaluateOptions): Promise<LegacyEvaluationResult> {
return this.getLegacyBackend("ask").ask(options);
}
async batchAsk(options: BatchAskOptions): Promise<EvaluationResult[]> {
async batchAsk(options: BatchAskOptions): Promise<LegacyEvaluationResult[]> {
return this.getLegacyBackend("batchAsk").batchAsk(options);
}
async verify(trajectory: Trajectory): Promise<EvaluationResult> {
const taskSpec = assertVerifierInput(trajectory);
if (this.backend === "legacy") {
return this.verifyTrajectoryWithLegacyEvaluator(trajectory, taskSpec);
}
return this.unavailableVerifierBackend("verify");
}
async generateRubric(taskSpec: TaskSpec): Promise<Rubric> {
if (!taskSpec?.id) {
throw new StagehandInvalidArgumentError(
"TaskSpec.id is required for rubric generation",
);
}
if (this.backend === "verifier") {
return this.unavailableVerifierBackend("generateRubric");
}
return {
items: [legacyTaskCompletionCriterion(taskSpec)],
};
}
private getLegacyBackend(methodName: string): LegacyV3Evaluator {
if (this.backend === "legacy") {
return this.legacyEvaluator;
}
return this.unavailableVerifierBackend(methodName);
}
private unavailableVerifierBackend(methodName: string): never {
throw new StagehandInvalidArgumentError(
`V3Evaluator.${methodName}() was configured with ${EVALUATOR_BACKEND_ENV}=verifier, but the verifier backend is not available in this build. Use "legacy" or install the verifier backend PR.`,
);
}
private async verifyTrajectoryWithLegacyEvaluator(
trajectory: Trajectory,
taskSpec: TaskSpec,
): Promise<EvaluationResult> {
const screenshots = collectLegacyScreenshots(trajectory);
const agentReasoning = renderLegacyAgentReasoning(trajectory);
const answer = trajectory.finalAnswer;
if (!screenshots.length && !answer) {
return legacyInsufficientEvidenceResult(
"Legacy evaluator compatibility mode had no screenshots or final answer to evaluate.",
);
}
const result = await this.legacyEvaluator.ask({
question: taskSpec.instruction,
screenshot: screenshots.length ? screenshots : false,
answer,
agentReasoning,
});
return legacyEvaluationToResult(result, screenshots.length);
}
}
function normalizeConstructorOptions(
@ -120,3 +183,149 @@ function resolveEvaluatorBackend(
`Invalid ${EVALUATOR_BACKEND_ENV}="${configuredBackend}". Expected "legacy" or "verifier".`,
);
}
function assertVerifierInput(trajectory: Trajectory): TaskSpec {
if (!trajectory) {
throw new StagehandInvalidArgumentError(
"Trajectory is required for verification",
);
}
if (!trajectory.task?.id) {
throw new StagehandInvalidArgumentError(
"Trajectory.task.id is required for verification",
);
}
return trajectory.task;
}
function legacyTaskCompletionCriterion(taskSpec: TaskSpec) {
return {
criterion: "legacy-task-completion",
description: `Evaluate whether the task was completed successfully: ${taskSpec.instruction}`,
maxPoints: 1,
};
}
function collectLegacyScreenshots(trajectory: Trajectory): Buffer[] {
const screenshots: Buffer[] = [];
for (const step of trajectory.steps ?? []) {
if (Buffer.isBuffer(step.probeEvidence?.screenshot)) {
screenshots.push(step.probeEvidence.screenshot);
continue;
}
const agentImage = step.agentEvidence?.modalities?.find(
(
modality,
): modality is Extract<AgentEvidenceModality, { type: "image" }> =>
modality.type === "image" && Buffer.isBuffer(modality.bytes),
);
if (agentImage) {
screenshots.push(agentImage.bytes);
}
}
if (Buffer.isBuffer(trajectory.finalObservation?.screenshot)) {
screenshots.push(trajectory.finalObservation.screenshot);
}
return screenshots;
}
function renderLegacyAgentReasoning(
trajectory: Trajectory,
): string | undefined {
const stepLines = (trajectory.steps ?? []).map((step, i) => {
const status = step.toolOutput?.ok === false ? "Tool status: failed" : "";
const output = step.toolOutput?.error
? `Tool error: ${step.toolOutput.error}`
: `Tool output: ${stringifyForPrompt(step.toolOutput?.result)}`;
return [
`Step ${i}: ${step.actionName}`,
step.reasoning ? `Reasoning: ${step.reasoning}` : undefined,
status || undefined,
output,
]
.filter(Boolean)
.join("\n");
});
if (!stepLines.length) {
return undefined;
}
return truncateForPrompt(
`Agent trajectory:\n${stepLines.join("\n\n")}`,
16000,
);
}
function stringifyForPrompt(value: unknown): string {
if (typeof value === "string") {
return value;
}
try {
const serialized = JSON.stringify(value);
return serialized ?? String(value);
} catch {
return String(value);
}
}
function truncateForPrompt(value: string, maxLength: number): string {
if (value.length <= maxLength) {
return value;
}
return `${value.slice(0, maxLength)}... [truncated]`;
}
function legacyEvaluationToResult(
result: LegacyEvaluationResult,
screenshotCount: number,
): EvaluationResult {
const outcomeSuccess = result.evaluation === "YES";
const invalid = result.evaluation === "INVALID";
const findings: VerifierFinding[] = invalid
? [
{
category: "verifier_uncertainty",
severity: "warning",
description: result.reasoning,
},
]
: [];
return {
outcomeSuccess,
explanation: result.reasoning,
...(findings.length ? { findings } : {}),
rawSteps: {
backend: "legacy",
legacyEvaluation: result.evaluation,
screenshotCount,
},
};
}
function legacyInsufficientEvidenceResult(reason: string): EvaluationResult {
return {
outcomeSuccess: false,
explanation: reason,
findings: [
{
category: "trajectory_capture",
severity: "blocking",
description: reason,
},
],
rawSteps: {
backend: "legacy",
legacyEvaluation: "INVALID",
screenshotCount: 0,
},
};
}

View File

@ -74,6 +74,7 @@ export class LegacyV3Evaluator {
if (Array.isArray(screenshot)) {
return this._evaluateWithMultipleScreenshots({
question,
answer,
screenshots: screenshot,
systemPrompt,
agentReasoning,
@ -224,12 +225,14 @@ export class LegacyV3Evaluator {
private async _evaluateWithMultipleScreenshots(options: {
question: string;
answer?: string;
screenshots: Buffer[];
systemPrompt?: string;
agentReasoning?: string;
}): Promise<EvaluationResult> {
const {
question,
answer,
screenshots,
agentReasoning,
systemPrompt = `You are an expert evaluator that confidently returns YES or NO given a question and multiple screenshots showing the progression of a task.
@ -272,6 +275,9 @@ export class LegacyV3Evaluator {
? `Question: ${question}\n\nAgent's reasoning and actions throughout the task:\n${agentReasoning}\n\nI'm providing ${screenshots.length} screenshots showing the progression of the task. Please analyze both the agent's reasoning and all screenshots to determine if the task was completed successfully.`
: `${question}\n\nI'm providing ${screenshots.length} screenshots showing the progression of the task. Please analyze all of them to determine if the task was completed successfully.`,
},
...(answer
? [{ type: "text" as const, text: `the answer is ${answer}` }]
: []),
...imageContents,
],
},

View File

@ -90,7 +90,7 @@
"pino": "^9.6.0",
"pino-pretty": "^13.0.0",
"uuid": "^11.1.1",
"ws": "^8.18.0",
"ws": "^8.21.0",
"zod-to-json-schema": "^3.25.0"
},
"optionalDependencies": {

View File

@ -60,6 +60,7 @@ class FakeCuaClient {
public contextNotes: string[] = [];
public preStepHook?: () => Promise<void>;
public actionHandler?: (action: Record<string, unknown>) => Promise<void>;
public screenshotProvider?: () => Promise<string>;
public executeImpl = vi.fn(async (options: unknown) => {
void options;
return {
@ -72,7 +73,9 @@ class FakeCuaClient {
public captureScreenshot = vi.fn(async () => null);
public setViewport = vi.fn();
public setCurrentUrl = vi.fn();
public setScreenshotProvider = vi.fn();
public setScreenshotProvider = vi.fn((provider: () => Promise<string>) => {
this.screenshotProvider = provider;
});
public setSafetyConfirmationHandler = vi.fn();
setActionHandler(
@ -247,7 +250,6 @@ describe("agent captcha hooks", () => {
context: {
awaitActivePage: async () => page,
},
bus: { emit: vi.fn() },
isCaptchaAutoSolveEnabled: true,
isAdvancedStealth: false,
configuredViewport: { width: 1288, height: 711 },
@ -316,7 +318,6 @@ describe("agent captcha hooks", () => {
context: {
awaitActivePage: async () => page,
},
bus: { emit: vi.fn() },
isCaptchaAutoSolveEnabled: true,
isAdvancedStealth: false,
configuredViewport: { width: 1288, height: 711 },
@ -392,7 +393,6 @@ describe("agent captcha hooks", () => {
context: {
awaitActivePage: async () => page,
},
bus: { emit: vi.fn() },
isCaptchaAutoSolveEnabled: true,
isAdvancedStealth: false,
configuredViewport: { width: 1288, height: 711 },
@ -474,7 +474,6 @@ describe("v3 cua handler screenshot behavior", () => {
context: {
awaitActivePage: async () => page,
},
bus: { emit: vi.fn() },
isCaptchaAutoSolveEnabled: false,
isAdvancedStealth: false,
configuredViewport: { width: 1288, height: 711 },
@ -504,4 +503,113 @@ describe("v3 cua handler screenshot behavior", () => {
// the CUA client takes a single screenshot after all actions itself.
expect(screenshotSpy).not.toHaveBeenCalled();
});
it("still returns provider screenshots when screenshot evidence callbacks fail", async () => {
const screenshotBase64 = Buffer.from("fake-image").toString("base64");
const onEvidence = vi.fn(async (event: { type: string }) => {
if (event.type === "screenshot") {
throw new Error("recorder failed");
}
});
fakeCuaClient.executeImpl = vi.fn(async () => {
await expect(fakeCuaClient.screenshotProvider?.()).resolves.toBe(
screenshotBase64,
);
return {
success: true,
message: "ok",
actions: [],
completed: true,
};
});
const handler = new V3CuaAgentHandler(
{
context: {
awaitActivePage: async () => page,
},
isCaptchaAutoSolveEnabled: false,
isAdvancedStealth: false,
configuredViewport: { width: 1288, height: 711 },
isAgentReplayActive: () => false,
updateMetrics: vi.fn(),
} as never,
logger,
{
modelName: "openai/gpt-5.4",
clientOptions: { waitBetweenActions: 1 },
} as never,
);
await handler.execute({
instruction: "describe the page",
highlightCursor: false,
callbacks: { onEvidence },
});
expect(onEvidence).toHaveBeenCalledWith(
expect.objectContaining({ type: "screenshot" }),
);
expect(
logs.some((line) =>
line.message.includes("onEvidence callback failed for screenshot"),
),
).toBe(true);
});
it("records a failed action as step_finished {ok:false} and rethrows the original error", async () => {
const events: Array<{ type: string; [k: string]: unknown }> = [];
const onEvidence = vi.fn(async (event: { type: string }) => {
events.push(event as { type: string });
});
fakeCuaClient.executeImpl = vi.fn(async () => {
await fakeCuaClient.actionHandler?.({
type: "click",
button: "left",
x: 5,
y: 9,
});
return { success: true, message: "ok", actions: [], completed: true };
});
const handler = new V3CuaAgentHandler(
{
context: {
awaitActivePage: async () => page,
},
isCaptchaAutoSolveEnabled: false,
isAdvancedStealth: false,
configuredViewport: { width: 1288, height: 711 },
isAgentReplayActive: () => false,
updateMetrics: vi.fn(),
} as never,
logger,
{
modelName: "openai/gpt-5.4",
clientOptions: { waitBetweenActions: 1 },
} as never,
);
vi.spyOn(
handler as unknown as {
executeAction: (action: Record<string, unknown>) => Promise<unknown>;
},
"executeAction",
).mockRejectedValue(new Error("click failed"));
await expect(
handler.execute({
instruction: "click the thing",
highlightCursor: false,
callbacks: { onEvidence },
}),
).rejects.toThrow("click failed");
const stepFinished = events.find((e) => e.type === "step_finished");
expect(stepFinished).toMatchObject({
actionName: "click",
toolOutput: { ok: false, error: "click failed" },
});
});
});

View File

@ -36,7 +36,30 @@ describe("AnthropicCUAClient adaptive thinking", () => {
});
});
describe("Claude 4.6 models (adaptive thinking)", () => {
describe("Claude 4.6+ models (adaptive thinking)", () => {
it("should use thinking.type: 'adaptive' for claude-opus-4-8 when thinkingEffort is set", async () => {
const client = new AnthropicCUAClient(
"anthropic",
"claude-opus-4-8",
undefined,
{
apiKey: "test-key",
thinkingEffort: "high",
},
);
client.setViewport(1280, 720);
await client.getAction([{ role: "user", content: "test" }]);
expect(mockCreate).toHaveBeenCalledWith(
expect.objectContaining({
thinking: { type: "adaptive" },
output_config: { effort: "high" },
temperature: 1,
}),
);
});
it("should use thinking.type: 'adaptive' for claude-opus-4-6 when thinkingEffort is set", async () => {
const client = new AnthropicCUAClient(
"anthropic",

View File

@ -0,0 +1,342 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { StagehandAPIClient } from "../../lib/v3/api.js";
import type { ModelConfiguration } from "../../lib/v3/types/public/model.js";
const vertexModel = {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
} as unknown as ModelConfiguration;
function createClientWithExecuteMock() {
const client = new StagehandAPIClient({
apiKey: "bb-test",
logger: vi.fn(),
});
const executeMock = vi.fn().mockResolvedValue({});
(
client as unknown as {
execute: typeof executeMock;
}
).execute = executeMock;
return { client, executeMock };
}
describe("StagehandAPIClient default model config", () => {
let originalFetch: typeof globalThis.fetch;
beforeEach(() => {
originalFetch = globalThis.fetch;
globalThis.fetch = vi.fn().mockResolvedValue(
new Response(
JSON.stringify({
success: true,
data: { sessionId: "sess-default-model", available: true },
}),
{
status: 200,
headers: { "Content-Type": "application/json" },
},
),
);
});
afterEach(() => {
globalThis.fetch = originalFetch;
vi.restoreAllMocks();
});
it("sends constructor Vertex model config on navigate bootstrap, act, observe, extract, and agent execute requests", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "vertex/gemini-2.5-flash",
defaultModelConfig: vertexModel,
});
await client.goto("https://example.com");
await client.act({ input: "click the login button" });
await client.observe({ instruction: "find the login button" });
await client.extract({ instruction: "extract the headline" });
await client.agentExecute({ mode: "dom" }, "click the login button");
expect(executeMock).toHaveBeenNthCalledWith(
1,
expect.objectContaining({
method: "navigate",
args: expect.objectContaining({
options: {
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
},
}),
}),
);
expect(executeMock).toHaveBeenNthCalledWith(
2,
expect.objectContaining({
method: "act",
args: expect.objectContaining({
options: {
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
},
}),
}),
);
expect(executeMock).toHaveBeenNthCalledWith(
3,
expect.objectContaining({
method: "observe",
args: expect.objectContaining({
options: expect.objectContaining({
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
}),
}),
}),
);
expect(executeMock).toHaveBeenNthCalledWith(
4,
expect.objectContaining({
method: "extract",
args: expect.objectContaining({
options: expect.objectContaining({
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
}),
}),
}),
);
expect(executeMock).toHaveBeenNthCalledWith(
5,
expect.objectContaining({
method: "agentExecute",
args: expect.objectContaining({
agentConfig: expect.objectContaining({
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
}),
}),
}),
);
});
it("prefers a per-call model over the constructor default model config", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "vertex/gemini-2.5-flash",
defaultModelConfig: vertexModel,
});
await client.act({
input: "click the login button",
options: {
model: {
modelName: "openai/gpt-4.1-mini",
apiKey: "sk-per-call",
},
},
});
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
args: expect.objectContaining({
options: expect.objectContaining({
model: {
modelName: "openai/gpt-4.1-mini",
apiKey: "sk-per-call",
},
}),
}),
}),
);
});
it("inherits constructor Vertex auth for a same-provider string model override", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "vertex/gemini-2.5-flash",
defaultModelConfig: vertexModel,
});
await client.observe({
instruction: "find the login button",
options: {
model: "vertex/gemini-2.0-flash",
},
});
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
method: "observe",
args: expect.objectContaining({
options: expect.objectContaining({
model: expect.objectContaining({
modelName: "vertex/gemini-2.0-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
}),
}),
}),
);
});
it("does not treat model as a public navigate option", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "openai/gpt-4.1-mini",
modelApiKey: "sk-header-only",
});
await client.goto("https://example.com", {
model: {
modelName: "openai/gpt-4o",
apiKey: "sk-per-call",
},
} as unknown as Parameters<StagehandAPIClient["goto"]>[1]);
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
method: "navigate",
args: expect.objectContaining({
options: undefined,
}),
}),
);
});
it("keeps constructor bootstrap config when a caller passes an unsupported navigate model", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "vertex/gemini-2.5-flash",
defaultModelConfig: vertexModel,
});
await client.goto("https://example.com", {
model: {
modelName: "openai/gpt-4o",
apiKey: "sk-per-call",
},
} as unknown as Parameters<StagehandAPIClient["goto"]>[1]);
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
method: "navigate",
args: expect.objectContaining({
options: {
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
},
}),
}),
);
});
it("does not add a body model config when the constructor only provides a model API key", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "openai/gpt-4.1-mini",
modelApiKey: "sk-header-only",
});
await client.act({ input: "click the login button" });
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
args: expect.objectContaining({
options: undefined,
}),
}),
);
});
});

View File

@ -73,9 +73,8 @@ describe("API model config schemas", () => {
const vertexModel = {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
project: "test-gcp-project",
location: "us-central1",
googleAuthOptions: {
auth: {
type: "googleServiceAccount",
credentials: {
type: "service_account",
project_id: "test-gcp-project",
@ -89,6 +88,12 @@ describe("API model config schemas", () => {
projectId: "test-gcp-project",
universeDomain: "googleapis.com",
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
};
it("preserves Vertex auth params for act requests", () => {
@ -111,15 +116,20 @@ describe("API model config schemas", () => {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
project: "test-gcp-project",
location: "us-central1",
googleAuthOptions: {
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
},
},
});
@ -131,7 +141,11 @@ describe("API model config schemas", () => {
if (typeof parsedModel !== "object" || parsedModel === null) {
throw new Error("Expected object model config");
}
expect(parsedModel.googleAuthOptions).toEqual({
if (!("auth" in parsedModel)) {
throw new Error("Expected Vertex auth config");
}
expect(parsedModel.auth).toEqual({
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
@ -140,6 +154,67 @@ describe("API model config schemas", () => {
});
});
it("requires explicit Vertex provider configs to include service account auth and provider options", () => {
for (const model of [
{
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
},
{
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
},
]) {
const result = Api.ActRequestSchema.safeParse({
input: "click the search button",
options: { model },
});
expect(result.success).toBe(false);
}
});
it("requires provider vertex when passing Vertex auth and provider options", () => {
const result = Api.ActRequestSchema.safeParse({
input: "click the search button",
options: {
model: {
modelName: "vertex/gemini-2.5-flash",
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
},
},
});
expect(result.success).toBe(false);
});
it("preserves Vertex auth params for agent model configs", () => {
const result = Api.AgentExecuteRequestSchema.safeParse({
agentConfig: {
@ -158,6 +233,20 @@ describe("API model config schemas", () => {
});
it("rejects Vertex key file paths and external credential sources", () => {
const topLevelLegacyShape = Api.ActRequestSchema.safeParse({
input: "click the search button",
options: {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
project: "test-gcp-project",
location: "us-central1",
googleAuthOptions: {},
},
},
});
expect(topLevelLegacyShape.success).toBe(false);
for (const blockedAuthOptions of [
{ keyFilename: "/etc/passwd" },
{ keyFile: "/etc/passwd" },
@ -169,7 +258,7 @@ describe("API model config schemas", () => {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
googleAuthOptions: blockedAuthOptions,
auth: blockedAuthOptions,
},
},
});
@ -183,7 +272,8 @@ describe("API model config schemas", () => {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
googleAuthOptions: {
auth: {
type: "googleServiceAccount",
credentials: {
type: "external_account",
credential_source: {

View File

@ -1,5 +1,9 @@
import { describe, expect, it } from "vitest";
import { getAISDKLanguageModel } from "../../lib/v3/llm/LLMProvider.js";
import {
getAISDKLanguageModel,
LLMProvider,
toAISDKClientOptions,
} from "../../lib/v3/llm/LLMProvider.js";
describe("getAISDKLanguageModel", () => {
describe("ollama provider", () => {
@ -68,3 +72,67 @@ describe("getAISDKLanguageModel", () => {
});
});
});
describe("LLMProvider", () => {
it("allows Vertex models without experimental mode", () => {
const provider = new LLMProvider(() => {});
expect(() =>
provider.getClient(
"vertex/gemini-2.5-flash" as never,
{
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "stagehand@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\ntest\n-----END PRIVATE KEY-----\n",
},
},
providerOptions: {
vertex: {
project: "test-project",
location: "us-central1",
},
},
} as never,
{ experimental: false, disableAPI: false },
),
).not.toThrow();
});
it("adapts canonical Vertex auth into AI SDK googleAuthOptions", () => {
expect(
toAISDKClientOptions("vertex", {
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "stagehand@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\ntest\n-----END PRIVATE KEY-----\n",
},
projectId: "test-project",
universeDomain: "googleapis.com",
},
providerOptions: {
vertex: {
project: "test-project",
location: "us-central1",
},
},
}),
).toEqual({
project: "test-project",
location: "us-central1",
googleAuthOptions: {
credentials: {
client_email: "stagehand@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\ntest\n-----END PRIVATE KEY-----\n",
},
projectId: "test-project",
universeDomain: "googleapis.com",
},
});
});
});

View File

@ -32,6 +32,8 @@ const publicApiShape = {
V3: Stagehand.V3,
V3Evaluator: Stagehand.V3Evaluator,
V3FunctionName: Stagehand.V3FunctionName,
buildAgentEvidenceFromStepFinished:
Stagehand.buildAgentEvidenceFromStepFinished,
connectToMCPServer: Stagehand.connectToMCPServer,
default: StagehandDefaultExport,
defaultExtractSchema: Stagehand.defaultExtractSchema,
@ -43,16 +45,23 @@ const publicApiShape = {
isZod4Schema: Stagehand.isZod4Schema,
jsonSchemaToZod: Stagehand.jsonSchemaToZod,
loadApiKeyFromEnv: Stagehand.loadApiKeyFromEnv,
loadTrajectoryFromDisk: Stagehand.loadTrajectoryFromDisk,
mergeAgentEvidence: Stagehand.mergeAgentEvidence,
localBrowserLaunchOptionsSchema: Stagehand.localBrowserLaunchOptionsSchema,
modelToAgentProviderMap: Stagehand.modelToAgentProviderMap,
nextResultFilename: Stagehand.nextResultFilename,
normalizeRubric: Stagehand.normalizeRubric,
pageTextSchema: Stagehand.pageTextSchema,
providerEnvVarMap: Stagehand.providerEnvVarMap,
redactInlineImagePayloads: Stagehand.redactInlineImagePayloads,
shouldPersistTrajectory: Stagehand.shouldPersistTrajectory,
toGeminiSchema: Stagehand.toGeminiSchema,
toJsonSchema: Stagehand.toJsonSchema,
tool: Stagehand.tool,
transformSchema: Stagehand.transformSchema,
trimTrailingTextNode: Stagehand.trimTrailingTextNode,
validateZodSchema: Stagehand.validateZodSchema,
writeTrajectoryDir: Stagehand.writeTrajectoryDir,
...publicErrorTypes,
} as const;

View File

@ -3,17 +3,30 @@ import * as Stagehand from "@browserbasehq/stagehand";
describe("LLM and Agents public API types", () => {
describe("ModelConfiguration", () => {
it("accepts Vertex headers in model config", () => {
const googleConfig = {
modelName: "google/gemini-3-flash-preview",
project: "test-project",
location: "global",
it("accepts Vertex provider options in model config", () => {
const vertexConfig = {
provider: "vertex",
modelName: "vertex/gemini-3-flash-preview",
headers: {
"X-Goog-Priority": "high",
},
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-project",
location: "global",
},
},
} satisfies Stagehand.ModelConfiguration;
void googleConfig;
void vertexConfig;
});
});
@ -44,6 +57,7 @@ describe("LLM and Agents public API types", () => {
"openai/gpt-5.5",
"anthropic/claude-opus-4-5-20251101",
"anthropic/claude-opus-4-6",
"anthropic/claude-opus-4-8",
"anthropic/claude-sonnet-4-6",
"anthropic/claude-haiku-4-5",
"anthropic/claude-haiku-4-5-20251001",
@ -61,6 +75,12 @@ describe("LLM and Agents public API types", () => {
>();
void expectedModels; // Mark as used to satisfy ESLint
});
it("includes Claude Opus 4.8 at runtime", () => {
expect(Stagehand.AVAILABLE_CUA_MODELS).toContain(
"anthropic/claude-opus-4-8",
);
});
});
describe("AgentProvider", () => {
@ -274,6 +294,12 @@ describe("LLM and Agents public API types", () => {
typeof Stagehand.modelToAgentProviderMap
>().toExtend<ExpectedModelToAgentProviderMap>();
});
it("routes Claude Opus 4.8 to Anthropic", () => {
expect(Stagehand.modelToAgentProviderMap["claude-opus-4-8"]).toBe(
"anthropic",
);
});
});
describe("Response", () => {

View File

@ -134,6 +134,18 @@ describe("V3 Core public API types", () => {
>();
});
it("has verifier facade methods", () => {
expectTypeOf<V3EvaluatorInstance["verify"]>().toExtend<
(
trajectory: Stagehand.Trajectory,
taskSpec: Stagehand.TaskSpec,
) => Promise<Stagehand.EvaluationResult>
>();
expectTypeOf<V3EvaluatorInstance["generateRubric"]>().toExtend<
(taskSpec: Stagehand.TaskSpec) => Promise<Stagehand.Rubric>
>();
});
it("accepts legacy evaluator backend options", () => {
const mockV3 = {} as Stagehand.Stagehand;
expectTypeOf<typeof Stagehand.V3Evaluator>().toBeConstructibleWith(
@ -143,35 +155,6 @@ describe("V3 Core public API types", () => {
} satisfies Stagehand.V3EvaluatorConstructorOptions,
);
});
it("rejects verifier backend before the verifier PR is installed", async () => {
const evaluator = new Stagehand.V3Evaluator({} as Stagehand.Stagehand, {
backend: "verifier",
});
await expect(
evaluator.ask({ question: "Was the task completed?" }),
).rejects.toThrow(
"STAGEHAND_EVALUATOR_BACKEND=verifier, but the verifier backend is not available",
);
});
it("rejects invalid evaluator backend env values", () => {
const previousBackend = process.env.STAGEHAND_EVALUATOR_BACKEND;
process.env.STAGEHAND_EVALUATOR_BACKEND = "not-a-backend";
try {
expect(
() => new Stagehand.V3Evaluator({} as Stagehand.Stagehand),
).toThrow('Invalid STAGEHAND_EVALUATOR_BACKEND="not-a-backend"');
} finally {
if (previousBackend === undefined) {
delete process.env.STAGEHAND_EVALUATOR_BACKEND;
} else {
process.env.STAGEHAND_EVALUATOR_BACKEND = previousBackend;
}
}
});
});
describe("V3FunctionName", () => {

View File

@ -0,0 +1,47 @@
import { describe, expect, it } from "vitest";
import { inferToolOutput } from "../../lib/v3/agent/utils/toolOutputEvidence.js";
describe("inferToolOutput", () => {
it.each<[string, unknown, boolean, string | undefined]>([
[
"preserves raw results while normalizing top-level failure status",
{ success: false },
false,
undefined,
],
[
"normalizes one-level AI SDK output wrappers",
{
toolCallId: "call-1",
output: { success: false, error: { message: "not found" } },
},
false,
'{"message":"not found"}',
],
[
"handles isError and non-string errors",
{ isError: true, error: new Error("bad input") },
false,
"bad input",
],
[
"normalizes non-json error values",
{ error: Symbol("bad input") },
false,
"Symbol(bad input)",
],
[
"does not recursively treat page data as tool status",
{ data: { success: false, error: "page field" } },
true,
undefined,
],
])("%s", (_, result, ok, error) => {
expect(inferToolOutput(result)).toEqual({
ok,
result,
error,
});
});
});

View File

@ -0,0 +1,230 @@
import { describe, expect, it, vi } from "vitest";
import { V3Evaluator } from "../../lib/v3Evaluator.js";
import type { V3 } from "../../lib/v3/v3.js";
import type { TaskSpec, Trajectory } from "../../lib/v3/verifier/index.js";
describe("V3Evaluator verifier facade", () => {
it("rejects verifier backend before the verifier PR is installed", async () => {
const evaluator = new V3Evaluator({} as V3, {
backend: "verifier",
});
await expect(
evaluator.ask({ question: "Was the task completed?" }),
).rejects.toThrow(
"STAGEHAND_EVALUATOR_BACKEND=verifier, but the verifier backend is not available",
);
});
it("rejects verify when the verifier backend is selected before the verifier PR is installed", async () => {
const taskSpec: TaskSpec = {
id: "verifier-unavailable",
instruction: "Complete the task",
};
const evaluator = new V3Evaluator({} as V3, {
backend: "verifier",
});
await expect(evaluator.verify(makeTrajectory(taskSpec))).rejects.toThrow(
"STAGEHAND_EVALUATOR_BACKEND=verifier, but the verifier backend is not available",
);
});
it("rejects generateRubric when the verifier backend is selected before the verifier PR is installed", async () => {
const evaluator = new V3Evaluator({} as V3, {
backend: "verifier",
});
await expect(
evaluator.generateRubric({
id: "rubric-unavailable",
instruction: "Complete the task",
}),
).rejects.toThrow(
"STAGEHAND_EVALUATOR_BACKEND=verifier, but the verifier backend is not available",
);
});
it("maps legacy YES evaluations with trajectory screenshots to a successful result", async () => {
const taskSpec: TaskSpec = {
id: "success",
instruction: "Complete the task",
};
const screenshot = Buffer.from("screenshot");
const trajectory = makeTrajectory(taskSpec, {
screenshot,
finalAnswer: "The task is complete.",
});
const ask = vi.fn().mockResolvedValue({
evaluation: "YES",
reasoning: "The screenshot shows completion.",
});
const evaluator = new V3Evaluator({} as V3, {
backend: "legacy",
});
Object.defineProperty(evaluator, "legacyEvaluator", {
value: { ask },
});
const result = await evaluator.verify(trajectory);
expect(ask).toHaveBeenCalledWith(
expect.objectContaining({
question: taskSpec.instruction,
screenshot: [screenshot],
answer: "The task is complete.",
}),
);
expect(result.outcomeSuccess).toBe(true);
expect(result.explanation).toBe("The screenshot shows completion.");
expect(result.processScore).toBeUndefined();
expect(result.perCriterion).toBeUndefined();
});
it("passes final observation screenshots to the legacy verifier adapter", async () => {
const taskSpec: TaskSpec = {
id: "final-observation",
instruction: "Complete the task",
};
const finalScreenshot = Buffer.from("final screenshot");
const trajectory = {
...makeTrajectory(taskSpec),
finalObservation: {
url: "https://example.com/done",
screenshot: finalScreenshot,
},
};
const ask = vi.fn().mockResolvedValue({
evaluation: "YES",
reasoning: "The final screenshot shows completion.",
});
const evaluator = new V3Evaluator({} as V3, {
backend: "legacy",
});
Object.defineProperty(evaluator, "legacyEvaluator", {
value: { ask },
});
await evaluator.verify(trajectory);
expect(ask).toHaveBeenCalledWith(
expect.objectContaining({
screenshot: [finalScreenshot],
}),
);
});
it("keeps legacy tool output detail until the overall reasoning budget is reached", async () => {
const taskSpec: TaskSpec = {
id: "reasoning-budget",
instruction: "Complete the task",
};
const longToolOutput = "x".repeat(3000);
const ask = vi.fn().mockResolvedValue({
evaluation: "YES",
reasoning: "The trajectory shows completion.",
});
const evaluator = new V3Evaluator({} as V3, {
backend: "legacy",
});
Object.defineProperty(evaluator, "legacyEvaluator", {
value: { ask },
});
await evaluator.verify(
makeTrajectory(taskSpec, {
finalAnswer: "The task is complete.",
toolResult: longToolOutput,
}),
);
const firstCall = ask.mock.calls[0]?.[0];
expect(firstCall?.agentReasoning).toContain(longToolOutput);
expect(firstCall?.agentReasoning).not.toContain("Final answer:");
expect(firstCall?.answer).toBe("The task is complete.");
});
it("returns an evidence-insufficient legacy result for empty trajectories", async () => {
const taskSpec: TaskSpec = {
id: "empty",
instruction: "Complete the task",
};
const evaluator = new V3Evaluator({} as V3, {
backend: "legacy",
});
const result = await evaluator.verify(makeEmptyTrajectory(taskSpec));
expect(result).toMatchObject({
outcomeSuccess: false,
explanation:
"Legacy evaluator compatibility mode had no screenshots or final answer to evaluate.",
rawSteps: {
backend: "legacy",
legacyEvaluation: "INVALID",
screenshotCount: 0,
},
});
expect(result.processScore).toBeUndefined();
expect(result.perCriterion).toBeUndefined();
});
it("rejects invalid evaluator backend env values", () => {
const previousBackend = process.env.STAGEHAND_EVALUATOR_BACKEND;
process.env.STAGEHAND_EVALUATOR_BACKEND = "not-a-backend";
try {
expect(() => new V3Evaluator({} as V3)).toThrow(
'Invalid STAGEHAND_EVALUATOR_BACKEND="not-a-backend"',
);
} finally {
if (previousBackend === undefined) {
delete process.env.STAGEHAND_EVALUATOR_BACKEND;
} else {
process.env.STAGEHAND_EVALUATOR_BACKEND = previousBackend;
}
}
});
});
function makeEmptyTrajectory(taskSpec: TaskSpec): Trajectory {
return {
task: taskSpec,
steps: [],
status: "complete",
usage: {
input_tokens: 0,
output_tokens: 0,
},
};
}
function makeTrajectory(
taskSpec: TaskSpec,
options: {
screenshot?: Buffer;
finalAnswer?: string;
toolResult?: unknown;
} = {},
): Trajectory {
return {
...makeEmptyTrajectory(taskSpec),
steps: [
{
actionName: "act",
actionArgs: {},
reasoning: "I completed the task.",
agentEvidence: { modalities: [] },
probeEvidence: options.screenshot
? { screenshot: options.screenshot }
: {},
toolOutput: {
ok: true,
result: options.toolResult ?? "done",
},
},
],
finalAnswer: options.finalAnswer,
};
}

View File

@ -0,0 +1,63 @@
import { describe, expect, it } from "vitest";
import {
buildAgentEvidenceFromStepFinished,
REDACTED_INLINE_IMAGE,
} from "../../lib/v3/verifier/evidenceNormalization.js";
describe("buildAgentEvidenceFromStepFinished", () => {
it("captures primitive tool results as text evidence", () => {
const evidence = buildAgentEvidenceFromStepFinished({
type: "step_finished",
actionName: "check",
actionArgs: {},
reasoning: "",
toolOutput: { ok: true, result: false },
});
expect(evidence.modalities).toEqual([{ type: "text", content: "false" }]);
});
it("lifts inline screenshot payloads into image evidence and redacts JSON", () => {
const inlineScreenshot =
Buffer.from("inline screenshot").toString("base64");
const evidence = buildAgentEvidenceFromStepFinished({
type: "step_finished",
actionName: "click",
actionArgs: { describe: "Open fare details" },
reasoning: "",
toolOutput: {
ok: true,
result: {
output: {
success: true,
describe: "Open fare details",
screenshotBase64: inlineScreenshot,
},
},
},
});
const [imageModality, jsonModality] = evidence.modalities;
expect(JSON.stringify(evidence)).not.toContain(inlineScreenshot);
expect(jsonModality).toMatchObject({
type: "json",
content: {
output: {
screenshotBase64: REDACTED_INLINE_IMAGE,
},
},
});
expect(imageModality).toMatchObject({
type: "image",
mediaType: "image/png",
});
if (imageModality?.type === "image") {
expect(imageModality.bytes).toEqual(
Buffer.from(inlineScreenshot, "base64"),
);
}
});
});

View File

@ -0,0 +1,252 @@
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import {
loadTrajectoryFromDisk,
nextResultFilename,
normalizeRubric,
writeTrajectoryDir,
} from "../../lib/v3/verifier/trajectory.js";
describe("verifier trajectory utilities", () => {
it("normalizes serialized empty earned points out of public rubrics", () => {
expect(
normalizeRubric({
items: [
{
criterion: "Criterion",
description: "Description",
max_points: 1,
earned_points: "",
},
],
}),
).toEqual({
items: [
{
criterion: "Criterion",
description: "Description",
maxPoints: 1,
},
],
});
});
it("round-trips serialized snake_case rubrics to public camelCase rubrics", () => {
expect(
normalizeRubric({
items: [
{
criterion: "Criterion",
description: "Description",
max_points: 3,
earned_points: "2",
condition: "Only if relevant",
justification: "Partial credit.",
},
],
}),
).toEqual({
items: [
{
criterion: "Criterion",
description: "Description",
maxPoints: 3,
condition: "Only if relevant",
},
],
});
});
it("loads trajectory screenshots and image modalities from disk", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "stagehand-verifier-"));
const screenshot = Buffer.from("probe screenshot");
const finalScreenshot = Buffer.from("final screenshot");
const agentImage = Buffer.from("agent image");
await writeFile(path.join(dir, "screenshot_1.png"), screenshot);
await writeFile(path.join(dir, "final.png"), finalScreenshot);
await mkdir(path.join(dir, "screenshots", "agent"), { recursive: true });
await writeFile(
path.join(dir, "screenshots", "agent", "1.png"),
agentImage,
);
await writeFile(
path.join(dir, "trajectory.json"),
JSON.stringify({
task: { id: "task", instruction: "Do the task" },
status: "complete",
usage: { input_tokens: 0, output_tokens: 0 },
steps: [
{
actionName: "act",
actionArgs: {},
reasoning: "",
agentEvidence: {
modalities: [
{
type: "image",
mediaType: "image/png",
imagePath: "screenshots/agent/1.png",
},
],
},
probeEvidence: { screenshotPath: "screenshot_1.png" },
toolOutput: { ok: true, result: null },
},
],
finalObservation: {
url: "https://example.com/done",
screenshotPath: "final.png",
},
}),
);
const trajectory = await loadTrajectoryFromDisk(dir);
const modality = trajectory.steps[0].agentEvidence.modalities[0];
expect(trajectory.steps[0].probeEvidence.screenshot).toEqual(screenshot);
expect(trajectory.finalObservation?.screenshot).toEqual(finalScreenshot);
expect(modality.type).toBe("image");
if (modality.type === "image") {
expect(modality.bytes).toEqual(agentImage);
}
});
it("loads legacy base64 image modalities from disk", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "stagehand-verifier-"));
const agentImage = Buffer.from("legacy agent image");
await writeFile(
path.join(dir, "trajectory.json"),
JSON.stringify({
task: { id: "task", instruction: "Do the task" },
status: "complete",
usage: { input_tokens: 0, output_tokens: 0 },
steps: [
{
actionName: "act",
actionArgs: {},
reasoning: "",
agentEvidence: {
modalities: [
{
type: "image",
mediaType: "image/png",
bytesBase64: agentImage.toString("base64"),
},
],
},
probeEvidence: {},
toolOutput: { ok: true, result: null },
},
],
}),
);
const trajectory = await loadTrajectoryFromDisk(dir);
const modality = trajectory.steps[0].agentEvidence.modalities[0];
expect(modality.type).toBe("image");
if (modality.type === "image") {
expect(modality.bytes).toEqual(agentImage);
}
});
it("redacts inline screenshot payloads when writing trajectories", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "stagehand-verifier-"));
const inlineScreenshot =
Buffer.from("inline screenshot").toString("base64");
await writeTrajectoryDir(dir, {
task: { id: "task", instruction: "Do the task" },
status: "complete",
usage: { input_tokens: 0, output_tokens: 0 },
finalObservation: {
url: "https://example.com/done",
screenshot: Buffer.from("final screenshot"),
},
steps: [
{
actionName: "click",
actionArgs: {},
reasoning: "",
agentEvidence: {
modalities: [
{
type: "json",
content: {
output: {
success: true,
screenshotBase64: inlineScreenshot,
},
},
},
],
},
probeEvidence: {},
toolOutput: {
ok: true,
result: {
output: {
success: true,
screenshotBase64: inlineScreenshot,
},
},
},
},
],
});
const raw = await readFile(path.join(dir, "trajectory.json"), "utf8");
const trajectory = JSON.parse(raw);
expect(raw).not.toContain(inlineScreenshot);
expect(
trajectory.steps[0].agentEvidence.modalities[0].content.output
.screenshotBase64,
).toBe("[redacted inline image payload]");
expect(trajectory.steps[0].toolOutput.result.output.screenshotBase64).toBe(
"[redacted inline image payload]",
);
expect(trajectory.finalObservation.screenshotPath).toBe(
"screenshots/probe/final.png",
);
await expect(
readFile(path.join(dir, "screenshots", "probe", "final.png")),
).resolves.toEqual(Buffer.from("final screenshot"));
});
it("rejects screenshot paths outside the trajectory directory", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "stagehand-verifier-"));
await writeFile(
path.join(dir, "trajectory.json"),
JSON.stringify({
task: { id: "task", instruction: "Do the task" },
status: "complete",
usage: { input_tokens: 0, output_tokens: 0 },
steps: [
{
actionName: "act",
actionArgs: {},
reasoning: "",
agentEvidence: { modalities: [] },
probeEvidence: { screenshotPath: "../../../etc/passwd" },
toolOutput: { ok: true, result: null },
},
],
}),
);
await expect(loadTrajectoryFromDisk(dir)).rejects.toThrow(
"escapes trajectory directory",
);
});
it("sanitizes result filename labels", () => {
expect(nextResultFilename("rescore / task:one?")).toBe(
"result_rescore___task_one_.json",
);
});
});

View File

@ -0,0 +1,202 @@
// Re-label generated OpenAPI schema fields on SDK-flavored API reference pages.
//
// Mintlify renders these pages from the OpenAPI wire schema, so body fields use
// camelCase even when the selected SDK accepts snake_case parameters. Stainless
// already emits language-correct code samples; this small client-side patch keeps
// the visible parameter labels aligned with the Python and Ruby SDK snippets.
(function () {
const SNAKE_CASE_PARAM_LABELS = {
acceptDownloads: "accept_downloads",
actTimeoutMs: "act_timeout_ms",
actionDescription: "action_description",
actionId: "action_id",
advancedStealth: "advanced_stealth",
agentConfig: "agent_config",
apiKey: "api_key",
backendNodeId: "backend_node_id",
baseURL: "base_url",
blockAds: "block_ads",
browserSettings: "browser_settings",
browserbaseAPIKey: "browserbase_api_key",
browserbaseProjectID: "browserbase_project_id",
browserbaseSessionCreateParams: "browserbase_session_create_params",
browserbaseSessionID: "browserbase_session_id",
cacheEntry: "cache_entry",
cacheKey: "cache_key",
captchaImageSelector: "captcha_image_selector",
captchaInputSelector: "captcha_input_selector",
cdpHeaders: "cdp_headers",
cdpUrl: "cdp_url",
cdpURL: "cdp_url",
chromiumSandbox: "chromium_sandbox",
clientLanguage: "client_language",
connectTimeoutMs: "connect_timeout_ms",
deviceScaleFactor: "device_scale_factor",
domSettleTimeoutMs: "dom_settle_timeout_ms",
domainPattern: "domain_pattern",
downloadsPath: "downloads_path",
endTime: "end_time",
executablePath: "executable_path",
executeOptions: "execute_options",
executionModel: "execution_model",
extensionId: "extension_id",
extensionID: "extension_id",
frameId: "frame_id",
frameID: "frame_id",
googleAuthOptions: "google_auth_options",
hasTouch: "has_touch",
highlightCursor: "highlight_cursor",
httpVersion: "http_version",
ignoreDefaultArgs: "ignore_default_args",
ignoreHTTPSErrors: "ignore_https_errors",
ignoreSelectors: "ignore_selectors",
inputTokens: "input_tokens",
keepAlive: "keep_alive",
launchOptions: "launch_options",
logSession: "log_session",
maxHeight: "max_height",
maxSteps: "max_steps",
maxWidth: "max_width",
minHeight: "min_height",
minWidth: "min_width",
modelAPIKey: "model_api_key",
modelName: "model_name",
operatingSystems: "operating_systems",
outputTokens: "output_tokens",
pageText: "page_text",
pageUrl: "page_url",
preserveUserDataDir: "preserve_user_data_dir",
projectId: "project_id",
projectID: "project_id",
recordSession: "record_session",
rememberMe: "remember_me",
selfHeal: "self_heal",
sessionId: "session_id",
shouldCache: "should_cache",
solveCaptchas: "solve_captchas",
streamResponse: "stream_response",
systemPrompt: "system_prompt",
taskCompleted: "task_completed",
timeMs: "time_ms",
tokenUsage: "token_usage",
toolTimeout: "tool_timeout",
universeDomain: "universe_domain",
useSearch: "use_search",
userDataDir: "user_data_dir",
userMetadata: "user_metadata",
waitForCaptchaSolves: "wait_for_captcha_solves",
waitUntil: "wait_until",
xStreamResponse: "x_stream_response",
};
const SDK_PARAM_LABELS = {
python: SNAKE_CASE_PARAM_LABELS,
ruby: SNAKE_CASE_PARAM_LABELS,
};
const CONTENT_SCOPE_SELECTOR = [
"#content-area",
"#api-playground-input",
"api-section",
"field",
].join(",");
const SKIP_SELECTOR = [
"code",
"input",
"pre",
"script",
"style",
"textarea",
'[contenteditable="true"]',
].join(",");
let rewriteTimer = null;
let lastPath = window.location.pathname;
function currentSdkLanguage() {
const match = window.location.pathname.match(
/^\/v3\/api-reference\/([^/]+)(?:\/|$)/,
);
return match ? match[1] : null;
}
function escapeRegExp(value) {
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
function buildPattern(labels) {
return new RegExp(
`\\b(${Object.keys(labels).map(escapeRegExp).join("|")})\\b`,
"g",
);
}
function shouldRewriteTextNode(node, pattern) {
if (!node.nodeValue) return false;
const parent = node.parentElement;
if (!parent) return false;
if (parent.closest(SKIP_SELECTOR)) return false;
if (!parent.closest(CONTENT_SCOPE_SELECTOR)) return false;
pattern.lastIndex = 0;
return pattern.test(node.nodeValue);
}
function rewriteLabels(root) {
const labels = SDK_PARAM_LABELS[currentSdkLanguage()];
if (!labels) return;
const pattern = buildPattern(labels);
const walker = document.createTreeWalker(
root || document.body,
NodeFilter.SHOW_TEXT,
{
acceptNode(node) {
return shouldRewriteTextNode(node, pattern)
? NodeFilter.FILTER_ACCEPT
: NodeFilter.FILTER_REJECT;
},
},
);
const nodes = [];
while (walker.nextNode()) nodes.push(walker.currentNode);
for (const node of nodes) {
pattern.lastIndex = 0;
node.nodeValue = node.nodeValue.replace(pattern, (name) => labels[name]);
}
}
function scheduleRewrite(root) {
if (rewriteTimer) window.clearTimeout(rewriteTimer);
rewriteTimer = window.setTimeout(() => {
rewriteTimer = null;
window.requestAnimationFrame(() => rewriteLabels(root || document.body));
}, 50);
}
scheduleRewrite();
const observer = new MutationObserver((mutations) => {
for (const mutation of mutations) {
for (const node of mutation.addedNodes) {
if (node.nodeType === Node.ELEMENT_NODE) {
scheduleRewrite(node);
return;
}
}
}
});
observer.observe(document.body, { childList: true, subtree: true });
window.setInterval(() => {
if (window.location.pathname !== lastPath) {
lastPath = window.location.pathname;
scheduleRewrite();
}
}, 250);
})();

View File

@ -669,6 +669,7 @@ const agent = stagehand.agent({
| Anthropic | `anthropic/claude-sonnet-4-5-20250929` |
| Anthropic | `anthropic/claude-opus-4-5-20251101` |
| Anthropic | `anthropic/claude-opus-4-6` |
| Anthropic | `anthropic/claude-opus-4-8` |
| Google | `google/gemini-2.5-computer-use-preview-10-2025` |
| Google | `google/gemini-3-flash-preview` |
| Google | `google/gemini-3-pro-preview` |

View File

@ -72,6 +72,7 @@ interface AgentInstance {
- `"anthropic/claude-sonnet-4-5-20250929"`
- `"anthropic/claude-opus-4-5-20251101"`
- `"anthropic/claude-opus-4-6"`
- `"anthropic/claude-opus-4-8"`
- `"google/gemini-2.5-computer-use-preview-10-2025"`
- `"google/gemini-3-flash-preview"`
- `"google/gemini-3-pro-preview"`
@ -867,4 +868,4 @@ The following errors may be thrown by the `agent()` method:
- **StagehandDefaultError** - General execution error with detailed message
- **AgentAbortError** - Thrown when agent execution is cancelled via an `AbortSignal`
- **StreamingCallbacksInNonStreamingModeError** - Thrown when streaming-only callbacks (`onChunk`, `onFinish`, `onError`, `onAbort`) are used without `stream: true`
- **ExperimentalNotConfiguredError** - Thrown when experimental features (callbacks, signal, messages, streaming) are used without `experimental: true` in Stagehand constructor
- **ExperimentalNotConfiguredError** - Thrown when experimental features (callbacks, signal, messages, streaming) are used without `experimental: true` in Stagehand constructor

View File

@ -1,5 +1,9 @@
let braintrustPromise: Promise<typeof import("braintrust")> | undefined;
export function hasBraintrustApiKey(): boolean {
return Boolean(process.env.BRAINTRUST_API_KEY);
}
export function loadBraintrust(): Promise<typeof import("braintrust")> {
braintrustPromise ??= import("braintrust");
return braintrustPromise;
@ -9,6 +13,9 @@ export async function tracedSpan<T>(
fn: () => Promise<T>,
options: { name: string },
): Promise<T> {
if (!hasBraintrustApiKey()) {
return fn();
}
const { traced } = await loadBraintrust();
return traced(fn, options);
}

View File

@ -22,7 +22,11 @@ import type { Testcase, EvalInput } from "../types/evals.js";
import { generateBenchTestcases } from "./benchPlanner.js";
import { DEFAULT_BENCH_HARNESS, type Harness } from "./benchTypes.js";
import { executeBenchTask } from "./benchRunner.js";
import { loadBraintrust, tracedSpan } from "./braintrust.js";
import {
hasBraintrustApiKey,
loadBraintrust,
tracedSpan,
} from "./braintrust.js";
import { onceAsync, registerActiveRunCleanup } from "./activeRunCleanup.js";
import { loadTaskModuleFromPath } from "./taskLoader.js";
@ -352,6 +356,7 @@ export async function runEvals(
: [exactMatch, errorMatch];
const { Eval, flush } = await loadBraintrust();
const sendLogs = hasBraintrustApiKey();
// Aggressive abort: when the caller flips signal.reason to "aggressive",
// close every active session so any in-flight task throws on its next
@ -440,10 +445,13 @@ export async function runEvals(
{
progress: silentBraintrustProgress,
reporter: silentBraintrustReporter,
...(sendLogs ? {} : { noSendLogs: true }),
},
);
await flush();
if (sendLogs) {
await flush();
}
const summaryResults = evalResult.results.map((result) => {
const output =

View File

@ -0,0 +1,243 @@
import fs from "node:fs/promises";
import path from "node:path";
import {
buildAgentEvidenceFromStepFinished,
mergeAgentEvidence,
redactInlineImagePayloads,
shouldPersistTrajectory,
writeTrajectoryDir,
} from "@browserbasehq/stagehand";
import type {
AgentEvidence,
AgentEvidenceEvent,
AgentFinalAnswerEvent,
AgentScreenshotEvidenceEvent,
AgentStepFinishedEvent,
AgentStepObservedEvent,
ProbeEvidence,
TaskSpec,
Trajectory,
TrajectoryStatus,
TrajectoryStep,
TrajectoryUsage,
EvaluationResult,
} from "@browserbasehq/stagehand";
export interface TrajectoryRecorderOptions {
taskSpec: TaskSpec;
/**
* Root directory under which trajectory dirs are written. Each task run
* gets a subdirectory named by runId/task.id.
* Defaults to `<cwd>/.trajectories`.
*/
outputRoot?: string;
/** Run identifier (e.g., ISO timestamp + env). Defaults to a fresh timestamp. */
runId?: string;
/**
* Override the env-gated persistence default. `true` always persists,
* `false` never does, `undefined` defers to VERIFIER_PERSIST_TRAJECTORIES.
*/
persist?: boolean;
}
export interface TrajectoryFinishOptions {
status: TrajectoryStatus;
finalAnswer?: string;
usage?: Partial<TrajectoryUsage>;
}
const ZERO_USAGE: TrajectoryUsage = {
input_tokens: 0,
output_tokens: 0,
};
export class TrajectoryRecorder {
private readonly taskSpec: TaskSpec;
private readonly runId: string;
private readonly outputDir: string;
private readonly persistEnabled: boolean;
// Steps are appended in arrival order on each step_finished event.
private readonly steps: TrajectoryStep[] = [];
// The most recent agent-role screenshot. It applies to every step_finished
// until a newer agent-role screenshot replaces it — a CUA provider can pick
// multiple actions from one screenshot, so each of those steps must carry
// that same tier-1 frame. (It is NOT cleared on consume; it is only replaced
// by a newer screenshot, or wiped on cancel().)
private latestAgentScreenshot?: Buffer;
// The most recent probe-role screenshot waits for the matching step_observed.
private pendingProbeScreenshot?: Buffer;
// Steps that haven't yet had a probe attached. The next step_observed fans
// out to all of them (one probe per agent turn, N tool calls per turn).
private stepsAwaitingProbe: number[] = [];
private finalAnswerEvent?: AgentFinalAnswerEvent;
private finalObservation?: ProbeEvidence;
private onScreenshot(e: AgentScreenshotEvidenceEvent): void {
if (e.evidenceRole === "agent") {
this.latestAgentScreenshot = e.screenshot;
} else {
this.pendingProbeScreenshot = e.screenshot;
}
}
private onStepFinished(e: AgentStepFinishedEvent): void {
const modalities: AgentEvidence["modalities"] = [];
if (this.latestAgentScreenshot) {
modalities.push({
type: "image",
bytes: this.latestAgentScreenshot,
mediaType: "image/png",
});
}
const merged = mergeAgentEvidence(
{ modalities },
buildAgentEvidenceFromStepFinished(e),
);
// Intentionally not cleared here: the same agent screenshot applies to
// every step in a batched CUA turn. It's replaced when a newer agent
// screenshot arrives (onScreenshot) or wiped on cancel().
this.stepsAwaitingProbe.push(this.steps.length);
this.steps.push({
actionName: e.actionName,
actionArgs: e.actionArgs,
reasoning: e.reasoning,
agentEvidence: merged,
probeEvidence: {},
toolOutput: {
...e.toolOutput,
result: redactInlineImagePayloads(e.toolOutput.result, e.actionName),
},
});
}
private onStepObserved(e: AgentStepObservedEvent): void {
const probe: ProbeEvidence = { url: e.url };
if (this.pendingProbeScreenshot)
probe.screenshot = this.pendingProbeScreenshot;
if (e.ariaTree !== undefined) probe.ariaTree = e.ariaTree;
for (const idx of this.stepsAwaitingProbe) {
this.steps[idx].probeEvidence = probe;
}
this.stepsAwaitingProbe = [];
this.pendingProbeScreenshot = undefined;
}
private onFinalAnswer(e: AgentFinalAnswerEvent): void {
this.finalAnswerEvent = e;
if (e.observation) {
this.finalObservation = {
url: e.observation.url,
...(e.observation.screenshot
? { screenshot: e.observation.screenshot }
: {}),
...(e.observation.ariaTree !== undefined
? { ariaTree: e.observation.ariaTree }
: {}),
};
}
}
constructor(opts: TrajectoryRecorderOptions) {
this.taskSpec = opts.taskSpec;
this.runId = opts.runId ?? new Date().toISOString().replace(/[:.]/g, "-");
const root = opts.outputRoot ?? path.join(process.cwd(), ".trajectories");
this.outputDir = path.join(root, this.runId, opts.taskSpec.id);
this.persistEnabled = shouldPersistTrajectory(opts.persist);
}
/** Ingest an evidence callback event from agent.execute(). */
record(event: AgentEvidenceEvent): void {
switch (event.type) {
case "screenshot":
this.onScreenshot(event);
break;
case "step_finished":
this.onStepFinished(event);
break;
case "step_observed":
this.onStepObserved(event);
break;
case "final_answer":
this.onFinalAnswer(event);
break;
}
}
/**
* Detach listeners, assemble the Trajectory, and (if persistence is on)
* write the on-disk layout. Idempotent.
*/
async finish(opts: TrajectoryFinishOptions): Promise<Trajectory> {
const trajectory: Trajectory = {
task: this.taskSpec,
steps: this.steps,
finalAnswer: opts.finalAnswer ?? this.finalAnswerEvent?.message,
...(this.finalObservation
? { finalObservation: this.finalObservation }
: {}),
status: opts.status,
usage: { ...ZERO_USAGE, ...(opts.usage ?? {}) },
};
if (this.persistEnabled) {
await writeTrajectoryDir(this.outputDir, trajectory);
}
return trajectory;
}
/** Throw away in-memory state without writing to disk. Used on early abort. */
cancel(): void {
this.steps.length = 0;
this.latestAgentScreenshot = undefined;
this.pendingProbeScreenshot = undefined;
this.stepsAwaitingProbe = [];
this.finalAnswerEvent = undefined;
this.finalObservation = undefined;
}
/** Where the trajectory dir lives (whether or not it was persisted). */
get directory(): string {
return this.outputDir;
}
/** Whether this recorder wrote the trajectory directory on finish(). */
get persisted(): boolean {
return this.persistEnabled;
}
/**
* Persist evaluator result next to the trajectory. No-op when trajectory
* persistence is disabled.
*/
async persistResult(
result: EvaluationResult,
filename = "result.json",
): Promise<void> {
if (!this.persistEnabled) return;
const scoresDir = path.join(this.outputDir, "scores");
await fs.mkdir(scoresDir, { recursive: true });
await fs.writeFile(
path.join(scoresDir, filename),
JSON.stringify(result, null, 2),
);
const taskDataPath = path.join(this.outputDir, "task_data.json");
let taskData: Record<string, unknown>;
try {
taskData = JSON.parse(await fs.readFile(taskDataPath, "utf8")) as Record<
string,
unknown
>;
} catch {
taskData = { task: this.taskSpec };
}
await fs.writeFile(
taskDataPath,
JSON.stringify({ ...taskData, result }, null, 2),
);
}
}

View File

@ -21,16 +21,18 @@ import {
toJsonSchema,
} from "@browserbasehq/stagehand";
let wrappedAiPromise:
| Promise<ReturnType<(typeof import("braintrust"))["wrapAISDK"]>>
| undefined;
type WrappedAI = ReturnType<(typeof import("braintrust"))["wrapAISDK"]>;
async function loadWrappedAISDK(): Promise<
ReturnType<(typeof import("braintrust"))["wrapAISDK"]>
> {
wrappedAiPromise ??= import("braintrust").then(({ wrapAISDK }) =>
wrapAISDK(ai),
);
let wrappedAiPromise: Promise<WrappedAI> | undefined;
async function loadWrappedAISDK(): Promise<WrappedAI> {
wrappedAiPromise ??= (async () => {
if (!process.env.BRAINTRUST_API_KEY) {
return ai as unknown as WrappedAI;
}
const { wrapAISDK } = await import("braintrust");
return wrapAISDK(ai);
})();
return wrappedAiPromise;
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,62 @@
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import { hasBraintrustApiKey, tracedSpan } from "../../framework/braintrust.js";
/**
* Verify that braintrust.ts helpers and AISdkClientWrapped work without
* BRAINTRUST_API_KEY. The runner-level tests live in a separate file
* (braintrust-runner-nolog.test.ts) because they need to vi.mock the
* braintrust module entirely.
*/
describe("braintrust.ts helpers without BRAINTRUST_API_KEY", () => {
const originalKey = process.env.BRAINTRUST_API_KEY;
beforeEach(() => {
delete process.env.BRAINTRUST_API_KEY;
});
afterEach(() => {
if (originalKey !== undefined) {
process.env.BRAINTRUST_API_KEY = originalKey;
} else {
delete process.env.BRAINTRUST_API_KEY;
}
});
it("hasBraintrustApiKey returns false when BRAINTRUST_API_KEY is unset", () => {
expect(hasBraintrustApiKey()).toBe(false);
});
it("hasBraintrustApiKey returns true when BRAINTRUST_API_KEY is set", () => {
process.env.BRAINTRUST_API_KEY = "test-key-123";
expect(hasBraintrustApiKey()).toBe(true);
});
it("tracedSpan calls fn directly without Braintrust when key is absent", async () => {
const fn = vi.fn(async () => 42);
const result = await tracedSpan(fn, { name: "test-span" });
expect(fn).toHaveBeenCalledOnce();
expect(result).toBe(42);
});
});
describe("AISdkClientWrapped without BRAINTRUST_API_KEY", () => {
const originalKey = process.env.BRAINTRUST_API_KEY;
beforeEach(() => {
delete process.env.BRAINTRUST_API_KEY;
});
afterEach(() => {
if (originalKey !== undefined) {
process.env.BRAINTRUST_API_KEY = originalKey;
} else {
delete process.env.BRAINTRUST_API_KEY;
}
});
it("AISdkClientWrapped class is importable without BRAINTRUST_API_KEY", async () => {
const mod = await import("../../lib/AISdkClientWrapped.js");
expect(mod.AISdkClientWrapped).toBeDefined();
});
});

View File

@ -0,0 +1,121 @@
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
/**
* Verify that the runner passes noSendLogs to Braintrust Eval and skips
* flush() when BRAINTRUST_API_KEY is absent. This file lives separately
* because it needs to vi.mock the braintrust module at the top level.
*/
vi.mock("playwright", () => ({
chromium: {},
}));
const mockEval = vi.fn<
(
name: string,
options: unknown,
evalOptions?: unknown,
) => Promise<{
results: unknown[];
summary: { experimentName: string; scores: Record<string, unknown> };
}>
>(async () => ({
results: [],
summary: { experimentName: "test", scores: {} },
}));
const mockFlush = vi.fn(async () => {});
let mockHasKey = false;
vi.mock("../../framework/braintrust.js", () => ({
hasBraintrustApiKey: () => mockHasKey,
loadBraintrust: async () => ({
Eval: mockEval,
flush: mockFlush,
}),
tracedSpan: async <T>(fn: () => Promise<T>) => fn(),
}));
describe("runner.ts skips Braintrust logging when API key is absent", () => {
const originalKey = process.env.BRAINTRUST_API_KEY;
beforeEach(() => {
delete process.env.BRAINTRUST_API_KEY;
mockEval.mockClear();
mockFlush.mockClear();
mockHasKey = false;
});
afterEach(() => {
if (originalKey !== undefined) {
process.env.BRAINTRUST_API_KEY = originalKey;
} else {
delete process.env.BRAINTRUST_API_KEY;
}
});
it("passes noSendLogs: true to Eval when BRAINTRUST_API_KEY is unset", async () => {
mockHasKey = false;
const { runEvals } = await import("../../framework/runner.js");
const task = {
name: "test-task",
tier: "bench" as const,
primaryCategory: "extract",
categories: ["extract"],
tags: [] as string[],
filePath: "/fake.ts",
isLegacy: false,
};
await runEvals({
tasks: [task],
registry: {
tasks: [task],
byName: new Map([[task.name, task]]),
byTier: new Map([["bench", [task]]]),
byCategory: new Map([["extract", [task]]]),
},
trials: 1,
});
expect(mockEval).toHaveBeenCalledTimes(1);
const evalOptions = mockEval.mock.calls[0][2];
expect(evalOptions).toHaveProperty("noSendLogs", true);
// flush should NOT be called
expect(mockFlush).not.toHaveBeenCalled();
});
it("does NOT pass noSendLogs when BRAINTRUST_API_KEY is set", async () => {
mockHasKey = true;
const { runEvals } = await import("../../framework/runner.js");
const task = {
name: "test-task",
tier: "bench" as const,
primaryCategory: "extract",
categories: ["extract"],
tags: [] as string[],
filePath: "/fake.ts",
isLegacy: false,
};
await runEvals({
tasks: [task],
registry: {
tasks: [task],
byName: new Map([[task.name, task]]),
byTier: new Map([["bench", [task]]]),
byCategory: new Map([["extract", [task]]]),
},
trials: 1,
});
expect(mockEval).toHaveBeenCalledTimes(1);
const evalOptions = mockEval.mock.calls[0][2];
expect(evalOptions).not.toHaveProperty("noSendLogs");
// flush SHOULD be called
expect(mockFlush).toHaveBeenCalledTimes(1);
});
});

View File

@ -65,6 +65,9 @@ function makeRegistry(tasks: DiscoveredTask[]): TaskRegistry {
beforeEach(() => {
originalCi = process.env.CI;
delete process.env.CI;
// Set a dummy API key so the runner sends logs and calls flush(),
// which this test asserts.
process.env.BRAINTRUST_API_KEY = "test-key";
tracedNames.length = 0;
evalMock.mockReset();
flushMock.mockClear();
@ -79,6 +82,7 @@ afterEach(() => {
} else {
process.env.CI = originalCi;
}
delete process.env.BRAINTRUST_API_KEY;
while (tempDirs.length > 0) {
const dir = tempDirs.pop();

View File

@ -0,0 +1,247 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import type { TaskSpec } from "@browserbasehq/stagehand";
import { TrajectoryRecorder } from "../../framework/trajectoryRecorder.js";
const tempDirs: string[] = [];
afterEach(async () => {
while (tempDirs.length > 0) {
const dir = tempDirs.pop();
if (dir) await fs.rm(dir, { recursive: true, force: true });
}
});
function makeTempDir(): Promise<string> {
return fs
.mkdtemp(path.join(os.tmpdir(), "trajectory-recorder-"))
.then((dir) => {
tempDirs.push(dir);
return dir;
});
}
function makeTaskSpec(): TaskSpec {
return {
id: "recorder-task",
instruction: "Compare economy and business fares.",
initUrl: "https://example.com",
};
}
function recordSimpleStep(recorder: TrajectoryRecorder, screenshot: Buffer) {
recorder.record({
type: "screenshot",
screenshot,
url: "https://example.com/search",
evidenceRole: "agent",
});
recorder.record({
type: "step_finished",
actionName: "act",
actionArgs: { instruction: "Search fares" },
reasoning: "Search for fares.",
toolOutput: { ok: true, result: "done" },
});
recorder.record({
type: "screenshot",
screenshot,
url: "https://example.com/search",
evidenceRole: "probe",
});
recorder.record({
type: "step_observed",
url: "https://example.com/search",
});
}
function recordFinalAnswer(
recorder: TrajectoryRecorder,
opts: { message: string; screenshot: Buffer; ariaTree?: string },
): void {
recorder.record({
type: "final_answer",
message: opts.message,
observation: {
url: "https://example.com/complete",
screenshot: opts.screenshot,
...(opts.ariaTree !== undefined ? { ariaTree: opts.ariaTree } : {}),
},
});
}
describe("TrajectoryRecorder", () => {
it("assembles ordered callback events into trajectory steps", async () => {
const recorder = new TrajectoryRecorder({
taskSpec: makeTaskSpec(),
persist: false,
});
const screenshot = Buffer.from("screen-1");
const staleScreenshot = Buffer.from("stale-screen");
const probeScreenshot = Buffer.from("probe-screen");
recorder.record({
type: "screenshot",
screenshot: staleScreenshot,
url: "https://example.com/stale",
evidenceRole: "agent",
});
recorder.record({
type: "screenshot",
screenshot,
url: "https://example.com/search",
evidenceRole: "agent",
});
recorder.record({
type: "step_finished",
actionName: "click",
actionArgs: { describe: "Open fares" },
reasoning: "Open fare details.",
toolOutput: { ok: true, result: "opened" },
});
recorder.record({
type: "step_finished",
actionName: "extract",
actionArgs: { instruction: "Read fares" },
reasoning: "Read visible fare cells.",
toolOutput: {
ok: true,
result: { economy: "$100", business: "$250" },
},
});
recorder.record({
type: "screenshot",
screenshot: probeScreenshot,
url: "https://example.com/search",
evidenceRole: "probe",
});
recorder.record({
type: "step_observed",
url: "https://example.com/search",
ariaTree: "RootWebArea\nStaticText: Economy $100",
});
recordFinalAnswer(recorder, {
message: "Business is $150 more than economy.",
screenshot: Buffer.from("final-screen"),
ariaTree: "RootWebArea\nStaticText: Complete",
});
const trajectory = await recorder.finish({
status: "complete",
usage: { input_tokens: 10, output_tokens: 5 },
});
expect(trajectory.steps).toHaveLength(2);
expect(trajectory.steps[0]).toMatchObject({
actionName: "click",
probeEvidence: {
url: "https://example.com/search",
ariaTree: "RootWebArea\nStaticText: Economy $100",
},
});
expect(trajectory.steps[1]).toMatchObject({
actionName: "extract",
toolOutput: { ok: true, result: { economy: "$100", business: "$250" } },
});
expect(trajectory.steps[0].probeEvidence.screenshot).toEqual(
probeScreenshot,
);
expect(trajectory.steps[1].probeEvidence.screenshot).toEqual(
probeScreenshot,
);
expect(trajectory.steps[0].agentEvidence.modalities).toEqual(
expect.arrayContaining([
{ type: "image", bytes: screenshot, mediaType: "image/png" },
{ type: "text", content: "Open fare details." },
]),
);
// Both actions were chosen from the same agent screenshot (one screenshot,
// two step_finished), so the second step must carry that frame too.
expect(trajectory.steps[1].agentEvidence.modalities).toEqual(
expect.arrayContaining([
{ type: "image", bytes: screenshot, mediaType: "image/png" },
]),
);
expect(trajectory.finalAnswer).toBe("Business is $150 more than economy.");
expect(trajectory.finalObservation).toMatchObject({
url: "https://example.com/complete",
ariaTree: "RootWebArea\nStaticText: Complete",
});
expect(trajectory.finalObservation?.screenshot).toEqual(
Buffer.from("final-screen"),
);
});
it("persists trajectory files and evaluator results", async () => {
const outputRoot = await makeTempDir();
const recorder = new TrajectoryRecorder({
taskSpec: makeTaskSpec(),
outputRoot,
runId: "run-1",
persist: true,
});
const screenshot = Buffer.from("screen-1");
recordSimpleStep(recorder, screenshot);
recordFinalAnswer(recorder, {
message: "Complete.",
screenshot: Buffer.from("final-screen"),
});
await recorder.finish({ status: "complete" });
await recorder.persistResult({
outcomeSuccess: true,
explanation: "The task was completed.",
});
const taskDir = path.join(outputRoot, "run-1", "recorder-task");
await expect(fs.readdir(taskDir)).resolves.toEqual(
expect.arrayContaining([
"core.log",
"scores",
"screenshots",
"task_data.json",
"trajectory.json",
]),
);
await expect(
fs.readFile(path.join(taskDir, "screenshots", "probe", "1.png")),
).resolves.toEqual(screenshot);
await expect(
fs.readFile(path.join(taskDir, "screenshots", "probe", "final.png")),
).resolves.toEqual(Buffer.from("final-screen"));
await expect(
fs.readFile(path.join(taskDir, "screenshots", "agent", "1.png")),
).resolves.toEqual(screenshot);
await expect(
fs.readFile(path.join(taskDir, "scores", "result.json"), "utf8"),
).resolves.toContain('"outcomeSuccess": true');
const trajectory = JSON.parse(
await fs.readFile(path.join(taskDir, "trajectory.json"), "utf8"),
);
expect(trajectory.steps[0].probeEvidence.screenshotPath).toBe(
"screenshots/probe/1.png",
);
expect(trajectory.finalObservation.screenshotPath).toBe(
"screenshots/probe/final.png",
);
expect(trajectory.steps[0].agentEvidence.modalities).toContainEqual({
type: "image",
imagePath: "screenshots/agent/1.png",
mediaType: "image/png",
});
const taskData = JSON.parse(
await fs.readFile(path.join(taskDir, "task_data.json"), "utf8"),
);
expect(taskData.result).toMatchObject({
outcomeSuccess: true,
explanation: "The task was completed.",
});
});
});

View File

@ -202,9 +202,14 @@ components:
- private_key
- client_email
additionalProperties: false
GoogleAuthOptions:
GoogleServiceAccountAuth:
type: object
properties:
type:
description: Use inline Google Cloud service account credentials for provider
authentication
type: string
const: googleServiceAccount
credentials:
description: Google Cloud service account credentials
$ref: "#/components/schemas/GoogleServiceAccountCredentials"
@ -221,21 +226,51 @@ components:
universeDomain:
description: Google Cloud universe domain
type: string
required:
- type
- credentials
additionalProperties: false
ModelConfigObject:
ModelAuth:
oneOf:
- $ref: "#/components/schemas/GoogleServiceAccountAuth"
VertexProviderOptions:
type: object
properties:
provider:
description: AI provider for the model (or provide a baseURL endpoint instead)
example: openai
project:
description: Google Cloud project ID for Vertex AI models
example: my-gcp-project
type: string
enum:
- openai
- anthropic
- google
- microsoft
- bedrock
- vertex
location:
description: Google Cloud location for Vertex AI models
example: us-central1
type: string
baseURL:
description: Base URL for the Vertex AI provider
type: string
format: uri
headers:
description: Custom headers sent with every request to the Vertex AI provider
type: object
propertyNames:
type: string
additionalProperties:
type: string
required:
- project
- location
additionalProperties: false
ModelProviderOptions:
type: object
properties:
vertex:
description: Vertex AI provider-specific settings
$ref: "#/components/schemas/VertexProviderOptions"
required:
- vertex
additionalProperties: false
GenericModelConfigObject:
type: object
properties:
modelName:
description: Model name string with provider prefix (e.g., 'openai/gpt-5-nano')
example: openai/gpt-5.4-mini
@ -256,19 +291,62 @@ components:
type: string
additionalProperties:
type: string
project:
description: Google Cloud project ID for Vertex AI models
example: my-gcp-project
provider:
description: AI provider for the model (or provide a baseURL endpoint instead)
example: openai
type: string
location:
description: Google Cloud location for Vertex AI models
example: us-central1
type: string
googleAuthOptions:
description: google-auth-library options used to authenticate Vertex AI models
$ref: "#/components/schemas/GoogleAuthOptions"
enum:
- openai
- anthropic
- google
- microsoft
- bedrock
required:
- modelName
additionalProperties: false
VertexModelConfigObject:
type: object
properties:
modelName:
description: Model name string with provider prefix (e.g., 'openai/gpt-5-nano')
example: openai/gpt-5.4-mini
type: string
apiKey:
description: API key for the model provider
example: sk-some-openai-api-key
type: string
baseURL:
description: Base URL for the model provider
example: https://api.openai.com/v1
type: string
format: uri
headers:
description: Custom headers sent with every request to the model provider
type: object
propertyNames:
type: string
additionalProperties:
type: string
provider:
description: Vertex AI model provider
type: string
const: vertex
auth:
description: Vertex provider authentication configuration
$ref: "#/components/schemas/ModelAuth"
providerOptions:
description: Vertex provider-specific model configuration
$ref: "#/components/schemas/ModelProviderOptions"
required:
- modelName
- provider
- auth
- providerOptions
additionalProperties: false
ModelConfigObject:
anyOf:
- $ref: "#/components/schemas/VertexModelConfigObject"
- $ref: "#/components/schemas/GenericModelConfigObject"
ModelConfig:
$ref: "#/components/schemas/ModelConfigObject"
Action:
@ -1319,56 +1397,6 @@ components:
acceptDownloads:
type: boolean
additionalProperties: false
ModelConfigObjectOutput:
type: object
properties:
provider:
description: AI provider for the model (or provide a baseURL endpoint instead)
example: openai
type: string
enum:
- openai
- anthropic
- google
- microsoft
- bedrock
- vertex
modelName:
description: Model name string with provider prefix (e.g., 'openai/gpt-5-nano')
example: openai/gpt-5.4-mini
type: string
apiKey:
description: API key for the model provider
example: sk-some-openai-api-key
type: string
baseURL:
description: Base URL for the model provider
example: https://api.openai.com/v1
type: string
format: uri
headers:
description: Custom headers sent with every request to the model provider
type: object
propertyNames:
type: string
additionalProperties:
type: string
project:
description: Google Cloud project ID for Vertex AI models
example: my-gcp-project
type: string
location:
description: Google Cloud location for Vertex AI models
example: us-central1
type: string
googleAuthOptions:
description: google-auth-library options used to authenticate Vertex AI models
$ref: "#/components/schemas/GoogleAuthOptions"
required:
- modelName
additionalProperties: false
ModelConfigOutput:
$ref: "#/components/schemas/ModelConfigObjectOutput"
ActionOutput:
description: Action object returned by observe and used by act
type: object
@ -1633,7 +1661,7 @@ components:
description: Model configuration object or model name string (e.g.,
'openai/gpt-5-nano')
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
variables:
description: Variables to substitute in the action instruction. Accepts flat
@ -1693,7 +1721,7 @@ components:
description: Model configuration object or model name string (e.g.,
'openai/gpt-5-nano')
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
timeout:
description: Timeout in ms for the extraction
@ -1738,7 +1766,7 @@ components:
description: Model configuration object or model name string (e.g.,
'openai/gpt-5-nano')
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
variables:
description: Variables whose names are exposed to the model so observe() returns
@ -1801,7 +1829,7 @@ components:
description: Model configuration object or model name string (e.g.,
'openai/gpt-5-nano')
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
systemPrompt:
description: Custom system prompt for the agent
@ -1825,7 +1853,7 @@ components:
agent tools). If not specified, inherits from the main model
configuration.
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
additionalProperties: false
AgentUsageOutput:

View File

@ -12,7 +12,7 @@
* SEA_INCLUDE_SOURCEMAPS.
* Example: pnpm run build:sea:cjs -- --target-platform=linux --target-arch=arm64
*/
import { spawnSync } from "node:child_process";
import { execFileSync, spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import fs from "node:fs";
import os from "node:os";
@ -87,6 +87,17 @@ const includeSourcemaps = parseBoolean(
false,
);
const gitOutput = (args: string[]) =>
execFileSync("git", args, { cwd: repoDir, encoding: "utf8" }).trim();
const buildCommit = gitOutput(["rev-parse", "HEAD"]);
const seaVersion = [
`commit: ${buildCommit}`,
`timestamp: ${gitOutput(["show", "-s", "--format=%cI", buildCommit])}`,
].join("\n");
const seaBuildDefine = {
__STAGEHAND_SERVER_VERSION__: JSON.stringify(seaVersion),
};
const run = (cmd: string, args: string[], opts: { cwd?: string } = {}) => {
const result = spawnSync(cmd, args, { stdio: "inherit", ...opts });
if (result.error) {
@ -273,6 +284,7 @@ const buildCjsBundle = () => {
platform: "node",
format: "cjs",
outfile: bundlePath,
define: seaBuildDefine,
logLevel: "warning",
absWorkingDir: repoDir,
});
@ -295,6 +307,7 @@ const buildEsmBundle = () => {
format: "esm",
treeShaking: false,
outfile: appBundlePath,
define: seaBuildDefine,
alias: {
"@browserbasehq/stagehand": `${repoDir}/packages/core/dist/esm/index.js`,
},
@ -412,6 +425,15 @@ const { pathToFileURL } = require("node:url");
const bundleBase64 = ${JSON.stringify(appBytes.toString("base64"))};
const bundleLength = ${appBytes.length};
const bundleHash = ${JSON.stringify(bundleHash)};
const versionOutput = ${JSON.stringify(seaVersion)};
const argv = process.argv.slice(1);
const normalizedArgv = argv[0]?.startsWith("--") ? argv : argv.slice(1);
if (normalizedArgv.includes("--version")) {
console.log(versionOutput);
process.exit(0);
}
const cacheRoot =
process.env.STAGEHAND_SEA_CACHE_DIR ||

View File

@ -44,7 +44,12 @@ async function main() {
LocalBrowserLaunchOptions: Api.LocalBrowserLaunchOptionsSchema,
GoogleServiceAccountCredentials:
Api.GoogleServiceAccountCredentialsSchema,
GoogleAuthOptions: Api.GoogleAuthOptionsSchema,
GoogleServiceAccountAuth: Api.GoogleServiceAccountAuthSchema,
ModelAuth: Api.ModelAuthSchema,
VertexProviderOptions: Api.VertexProviderOptionsSchema,
ModelProviderOptions: Api.ModelProviderOptionsSchema,
GenericModelConfigObject: Api.GenericModelConfigObjectSchema,
VertexModelConfigObject: Api.VertexModelConfigObjectSchema,
ModelConfigObject: Api.ModelConfigObjectSchema,
ModelConfig: Api.ModelConfigSchema,
Action: Api.ActionSchema,

View File

@ -208,10 +208,17 @@ export class InMemorySessionStore implements SessionStore {
const options: V3Options = {
env: isBrowserbase ? "BROWSERBASE" : "LOCAL",
model: {
modelName: params.modelName,
apiKey: ctx.modelApiKey,
},
model: ctx.requestModelConfig
? {
...ctx.requestModelConfig,
...(ctx.requestModelConfig.apiKey
? {}
: { apiKey: ctx.modelApiKey }),
}
: {
modelName: params.modelName,
apiKey: ctx.modelApiKey,
},
verbose: params.verbose,
systemPrompt: params.systemPrompt,
selfHeal: params.selfHeal,

View File

@ -67,6 +67,8 @@ export interface CreateSessionParams {
export interface RequestContext {
/** Model API key (from x-model-api-key header) */
modelApiKey?: string;
/** Structured model config from the current request body */
requestModelConfig?: Api.ModelConfig;
/** Logger function for this request */
logger?: (message: LogLine) => void;
}

View File

@ -1,4 +1,6 @@
import type { FastifyRequest } from "fastify";
import { Api } from "@browserbasehq/stagehand";
import { z } from "zod/v4";
import { MissingHeaderError } from "../types/error.js";
@ -39,41 +41,134 @@ export const getOptionalHeader = (
return headerValue;
};
const emptyStringSchema = z.literal("").transform((): undefined => undefined);
const requiredStringSchema = z.string().min(1);
const optionalStringSchema = z
.union([emptyStringSchema, z.string()])
.optional();
const requestModelInputSchema = z
.union([
emptyStringSchema,
requiredStringSchema.transform(
(modelName): Api.ModelConfig => ({ modelName }),
),
Api.ModelConfigSchema,
])
.optional();
const requestModelCarrierSchema = z
.object({
model: requestModelInputSchema,
})
.passthrough();
const unparsedModelCarrierSchema = z
.object({
model: z.unknown().optional(),
})
.passthrough();
const requestModelEnvelopeSchema = z
.object({
options: requestModelCarrierSchema.optional(),
agentConfig: unparsedModelCarrierSchema.optional(),
modelName: optionalStringSchema,
})
.passthrough();
const stagehandInitModelEnvelopeSchema = z
.object({
options: requestModelCarrierSchema.optional(),
agentConfig: requestModelCarrierSchema.optional(),
modelName: optionalStringSchema,
})
.passthrough();
const requestModelConfigSchema = z
.object({
model: Api.ModelConfigSchema.optional(),
modelName: optionalStringSchema,
apiKey: optionalStringSchema,
})
.strict();
export type RequestModelConfig = z.infer<typeof requestModelConfigSchema>;
type RequestModelConfigResult =
| { success: true; data: RequestModelConfig }
| { success: false; error: z.ZodError };
/**
* Extracts model name from request body, supporting V3 structure.
* - V3: body.options.model.modelName
* Extracts request-level model config with precedence.
*
* Model name:
* 1. body.options.model.modelName or body.options.model string
* 2. Legacy body.modelName fallback
*
* API key:
* 1. body.options.model.apiKey
* 2. x-model-api-key header
*
* agentConfig.model is parsed separately for Stagehand initialization. Its
* credentials are scoped to the agent main model and must not become the
* request-level API key fallback used by action/execution models.
*/
export function getModelName(request: FastifyRequest): string | undefined {
const body = request.body as Record<string, unknown> | undefined;
const options = body?.options as Record<string, unknown> | undefined;
const model = options?.model as Record<string, unknown> | undefined;
if (typeof model?.modelName === "string" && model.modelName) {
return model.modelName;
export function getRequestModelConfig(
request: FastifyRequest,
): RequestModelConfigResult {
const bodyResult = requestModelEnvelopeSchema.safeParse(request.body ?? {});
if (!bodyResult.success) {
return bodyResult;
}
if (typeof body?.modelName === "string" && body.modelName) {
return body.modelName;
const body = bodyResult.data;
const model = body.options?.model;
const configResult = requestModelConfigSchema.safeParse({
model,
modelName: model?.modelName ?? body.modelName,
apiKey: model?.apiKey ?? getOptionalHeader(request, "x-model-api-key"),
});
if (!configResult.success) {
return configResult;
}
return undefined;
return configResult;
}
/**
* Extracts the model API key with precedence:
* 1. Per-request body apiKey (V3: body.options.model.apiKey)
* 2. Per-request header x-model-api-key
* Extracts the structured model config used when creating a Stagehand instance
* for this request. This can read agentConfig.model for agentExecute startup,
* but it does not promote agent model credentials to the request-level API key.
*/
export function getModelApiKey(request: FastifyRequest): string | undefined {
const body = request.body as Record<string, unknown> | undefined;
const options = body?.options as Record<string, unknown> | undefined;
const model = options?.model as Record<string, unknown> | undefined;
if (typeof model?.apiKey === "string" && model.apiKey) {
return model.apiKey;
export function getStagehandInitModelConfig(
request: FastifyRequest,
requestModelConfig?: RequestModelConfig,
): RequestModelConfigResult {
const baseConfigResult = requestModelConfig
? requestModelConfigSchema.safeParse(requestModelConfig)
: getRequestModelConfig(request);
if (!baseConfigResult.success) {
return baseConfigResult;
}
return getOptionalHeader(request, "x-model-api-key");
const baseConfig = baseConfigResult.data;
if (baseConfig.model) {
return baseConfigResult;
}
const bodyResult = stagehandInitModelEnvelopeSchema.safeParse(
request.body ?? {},
);
if (!bodyResult.success) {
return bodyResult;
}
const agentModel = bodyResult.data.agentConfig?.model;
const configResult = requestModelConfigSchema.safeParse({
model: agentModel,
modelName: agentModel?.modelName ?? baseConfig.modelName,
apiKey: baseConfig.apiKey,
});
if (!configResult.success) {
return configResult;
}
return configResult;
}
/**

View File

@ -6,10 +6,11 @@ import { z } from "zod/v4";
import { AppError } from "./errorHandler.js";
import {
getModelApiKey,
getModelName,
getOptionalHeader,
getRequestModelConfig,
getStagehandInitModelConfig,
shouldRespondWithSSE,
type RequestModelConfig,
} from "./header.js";
import { error, success } from "./response.js";
import { getSessionStore } from "./sessionStoreManager.js";
@ -35,6 +36,13 @@ type StreamEventName =
| "error";
type StreamPayloadType = "system" | "log";
function formatZodIssues(err: z.ZodError) {
return err.issues.map((issue) => ({
path: issue.path[0] ?? "unknown",
message: issue.message,
}));
}
export async function createStreamingResponse<TV3>({
sessionId,
request,
@ -44,7 +52,6 @@ export async function createStreamingResponse<TV3>({
operation,
}: StreamingResponseOptions<TV3>) {
const shouldStreamResponse = shouldRespondWithSSE(request);
const modelApiKey = getModelApiKey(request);
const sessionStore = getSessionStore();
const sessionConfig = await sessionStore.getSessionConfig(sessionId);
@ -127,9 +134,45 @@ export async function createStreamingResponse<TV3>({
const actionId = v4();
sendData("starting", "system", { status: "starting" });
const sendZodValidationError = (err: z.ZodError) => {
const validationIssues = formatZodIssues(err);
if (shouldStreamResponse) {
sendData("error", "system", {
status: "error",
error: validationIssues,
});
reply.raw.end();
return reply;
}
return reply.status(StatusCodes.BAD_REQUEST).send({
error: validationIssues,
});
};
const requestModelConfigResult = getRequestModelConfig(request);
if (requestModelConfigResult.success === false) {
return sendZodValidationError(requestModelConfigResult.error);
}
const requestModelConfig: RequestModelConfig = requestModelConfigResult.data;
const stagehandInitModelConfigResult = getStagehandInitModelConfig(
request,
requestModelConfig,
);
if (stagehandInitModelConfigResult.success === false) {
return sendZodValidationError(stagehandInitModelConfigResult.error);
}
const stagehandInitModelConfig = stagehandInitModelConfigResult.data;
const modelApiKey = requestModelConfig.apiKey;
const parsedRequestModelConfig = stagehandInitModelConfig.model?.modelName
? stagehandInitModelConfig.model
: undefined;
const requestContext: RequestContext = {
modelApiKey,
requestModelConfig: parsedRequestModelConfig,
logger: shouldStreamResponse
? (message) => {
sendData("running", "log", { status: "running", message });
@ -177,7 +220,7 @@ export async function createStreamingResponse<TV3>({
operation: operation ?? "operation",
sessionId,
browserType,
modelName: getModelName(request),
modelName: requestModelConfig.modelName,
hasModelApiKey: Boolean(modelApiKey),
hasBrowserbaseApiKey: Boolean(browserbaseApiKey),
hasBrowserbaseProjectId: Boolean(browserbaseProjectId),

View File

@ -8,7 +8,10 @@ import { z } from "zod/v4";
import { authMiddleware } from "../../../lib/auth.js";
import { withErrorHandling } from "../../../lib/errorHandler.js";
import { getModelApiKey, getOptionalHeader } from "../../../lib/header.js";
import {
getOptionalHeader,
getRequestModelConfig,
} from "../../../lib/header.js";
import { error, success } from "../../../lib/response.js";
import { getSessionStore } from "../../../lib/sessionStoreManager.js";
import { AISDK_PROVIDERS } from "../../../types/model.js";
@ -110,11 +113,30 @@ const startRouteHandler: RouteHandler = withErrorHandling(
}
const browserType = browser?.type ?? "browserbase";
const localBrowserLaunchOptions =
browserType === "local" && browser?.launchOptions && !browser?.cdpUrl
? browser.launchOptions
: undefined;
let bbApiKey: string | undefined;
let browserbaseProjectId: string | undefined;
let browserbaseSessionId: string | undefined;
let connectUrl: string | undefined;
let localBrowserModelApiKey: string | undefined;
if (localBrowserLaunchOptions) {
const requestModelConfigResult = getRequestModelConfig(request);
if (requestModelConfigResult.success === false) {
return reply.status(StatusCodes.BAD_REQUEST).send({
error: requestModelConfigResult.error.issues.map((issue) => ({
path: issue.path[0] ?? "unknown",
message: issue.message,
})),
});
}
localBrowserModelApiKey = requestModelConfigResult.data.apiKey;
}
if (browserType === "browserbase") {
bbApiKey = getOptionalHeader(request, "x-bb-api-key");
@ -210,12 +232,11 @@ const startRouteHandler: RouteHandler = withErrorHandling(
// For local browsers with launchOptions (no explicit cdpUrl), eagerly
// initialize the browser so we can return the actual CDP URL
let finalCdpUrl = connectUrl ?? session.cdpUrl ?? "";
if (browserType === "local" && browser?.launchOptions && !browser?.cdpUrl) {
const modelApiKey = getModelApiKey(request);
if (localBrowserLaunchOptions) {
try {
const stagehand = await sessionStore.getOrCreateStagehand(
session.sessionId,
{ modelApiKey },
{ modelApiKey: localBrowserModelApiKey },
);
finalCdpUrl = stagehand.connectURL();
} catch (err) {
@ -226,12 +247,12 @@ const startRouteHandler: RouteHandler = withErrorHandling(
browserType,
chromePathEnv: process.env.CHROME_PATH,
launchOptions: {
executablePath: browser.launchOptions.executablePath,
argsCount: browser.launchOptions.args?.length ?? 0,
headless: browser.launchOptions.headless,
hasUserDataDir: Boolean(browser.launchOptions.userDataDir),
port: browser.launchOptions.port,
connectTimeoutMs: browser.launchOptions.connectTimeoutMs,
executablePath: localBrowserLaunchOptions.executablePath,
argsCount: localBrowserLaunchOptions.args?.length ?? 0,
headless: localBrowserLaunchOptions.headless,
hasUserDataDir: Boolean(localBrowserLaunchOptions.userDataDir),
port: localBrowserLaunchOptions.port,
connectTimeoutMs: localBrowserLaunchOptions.connectTimeoutMs,
},
},
"Failed to initialize local browser session in /v1/sessions/start",

View File

@ -1,15 +1,23 @@
import { __internalMaybeRunShutdownSupervisorFromArgv } from "@browserbasehq/stagehand";
declare const __STAGEHAND_SERVER_VERSION__: string;
// if SEA binary is launched with --supervisor, it will run the shutdown supervisor only
const argv = process.argv.slice(1);
const normalizedArgv = argv[0]?.startsWith("--") ? argv : argv.slice(1);
// otherwise, start the stagehand/server
if (!__internalMaybeRunShutdownSupervisorFromArgv(normalizedArgv)) {
// Force flow logs on before the server module loads Stagehand core.
process.env.BROWSERBASE_FLOW_LOGS = "1";
void import("./server.js").catch((err) => {
if (normalizedArgv.includes("--version")) {
console.log(__STAGEHAND_SERVER_VERSION__);
process.exit(0);
}
void import("@browserbasehq/stagehand")
.then(async ({ __internalMaybeRunShutdownSupervisorFromArgv }) => {
if (__internalMaybeRunShutdownSupervisorFromArgv(normalizedArgv)) {
return;
}
process.env.BROWSERBASE_FLOW_LOGS = "1";
await import("./server.js");
})
.catch((err) => {
console.error("Failed to start server:", err);
process.exit(1);
});
}

View File

@ -0,0 +1,120 @@
import assert from "node:assert/strict";
import { describe, it } from "node:test";
import type { FastifyRequest } from "fastify";
import {
getRequestModelConfig,
getStagehandInitModelConfig,
type RequestModelConfig,
} from "../../src/lib/header.js";
function createRequest({
body,
headers = {},
}: {
body?: Record<string, unknown>;
headers?: Record<string, string>;
}): FastifyRequest {
return {
body,
headers,
} as FastifyRequest;
}
function assertSuccess(
result: ReturnType<typeof getRequestModelConfig>,
): RequestModelConfig {
if (result.success === false) {
throw result.error;
}
assert.equal(result.success, true);
return result.data;
}
describe("getRequestModelConfig", () => {
it("preserves a Vertex model config from an action request so auth fields reach session initialization", () => {
const model = {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
};
const request = createRequest({
body: {
options: { model },
},
headers: {
"x-model-api-key": "sk-header",
},
});
const config = assertSuccess(getRequestModelConfig(request));
assert.equal(config.modelName, "vertex/gemini-2.5-flash");
assert.equal(config.apiKey, "sk-header");
assert.deepEqual(config.model, model);
});
it("does not read agentConfig.model as a request-level model config", () => {
const request = createRequest({
body: {
agentConfig: { model: "google/gemini-2.5-flash" },
},
});
assert.deepEqual(assertSuccess(getRequestModelConfig(request)), {
model: undefined,
modelName: undefined,
apiKey: undefined,
});
});
it("normalizes an agent string model into the session bootstrap model config", () => {
const request = createRequest({
body: {
agentConfig: { model: "google/gemini-2.5-flash" },
},
});
assert.deepEqual(assertSuccess(getStagehandInitModelConfig(request)), {
model: { modelName: "google/gemini-2.5-flash" },
modelName: "google/gemini-2.5-flash",
apiKey: undefined,
});
});
it("returns a validation result instead of throwing when agent bootstrap model config is invalid", () => {
const request = createRequest({
body: {
agentConfig: {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
},
},
},
});
assert.deepEqual(assertSuccess(getRequestModelConfig(request)), {
model: undefined,
modelName: undefined,
apiKey: undefined,
});
const result = getStagehandInitModelConfig(request);
assert.equal(result.success, false);
});
});

View File

@ -1,31 +0,0 @@
# @browserbasehq/stagehand-server-v4
## 3.6.4
### Patch Changes
- [#2098](https://github.com/browserbase/stagehand/pull/2098) [`a783b99`](https://github.com/browserbase/stagehand/commit/a783b99fb947968b685050314bd1df256d7a1f5a) Thanks [@seanmcguire12](https://github.com/seanmcguire12)! - bump transitive deps to patched versions
- Updated dependencies [[`21c78b3`](https://github.com/browserbase/stagehand/commit/21c78b3a50fd20cbec7ca8aa5f766f55e17b0f78), [`0641d44`](https://github.com/browserbase/stagehand/commit/0641d44a849062f5f7ce6a36a34ee95f9840efaa), [`f437f73`](https://github.com/browserbase/stagehand/commit/f437f738d23951cf460a30d3d285d1eba4c78ea2), [`a783b99`](https://github.com/browserbase/stagehand/commit/a783b99fb947968b685050314bd1df256d7a1f5a), [`8d2f354`](https://github.com/browserbase/stagehand/commit/8d2f3541427ca7c9c6d9a831601a6a5babc48502), [`a11603d`](https://github.com/browserbase/stagehand/commit/a11603d09d80f5e2fc341d154a0b90fe9fa48d1c), [`a87c1fc`](https://github.com/browserbase/stagehand/commit/a87c1fc435be83dbf14eab9edc6c421454ef7be4), [`26e6c96`](https://github.com/browserbase/stagehand/commit/26e6c960ca2894dc459ca40c9f31eb01e6d92053), [`1d176c4`](https://github.com/browserbase/stagehand/commit/1d176c466e25eb0cb03d9986b51d5cdb35a2e56b), [`1fa9613`](https://github.com/browserbase/stagehand/commit/1fa96130abbee2197a4e7f208878d06cba10c70b), [`9ff70dd`](https://github.com/browserbase/stagehand/commit/9ff70dd26cf4e03dce00ddcdc2d3b5e8d116781c), [`7640381`](https://github.com/browserbase/stagehand/commit/76403819b8f33d8d2670b6bea521a76f5ecc274e)]:
- @browserbasehq/stagehand@3.4.0
## 3.6.3
### Patch Changes
- Updated dependencies [[`732b384`](https://github.com/browserbase/stagehand/commit/732b3840f1631210dad2c720ec567e15e69ed304), [`20b601d`](https://github.com/browserbase/stagehand/commit/20b601dc8779a1bacf66abb68ebdf276a238e5db), [`8543c11`](https://github.com/browserbase/stagehand/commit/8543c11f6b4816f5c94fefdc8083f3616b987b22), [`14b64ec`](https://github.com/browserbase/stagehand/commit/14b64ec7d226a3eb0ce1c19937f14581e61b7a85), [`a500de1`](https://github.com/browserbase/stagehand/commit/a500de15cc010db3e42a7a05b7bcc92d2a9ad1d8), [`e471d2e`](https://github.com/browserbase/stagehand/commit/e471d2e89d41bac4e9b907ee9c0d7adc36828104), [`8f7192c`](https://github.com/browserbase/stagehand/commit/8f7192cee912268a125caf53bbaf2c6ba0f0947f)]:
- @browserbasehq/stagehand@3.3.0
## 3.6.2
### Patch Changes
- Updated dependencies [[`144e18e`](https://github.com/browserbase/stagehand/commit/144e18e9e0334ad3bc23aea6f4f7e181e0e6b9f0), [`d3c3736`](https://github.com/browserbase/stagehand/commit/d3c3736c579a78dfee8f7ad0ed4a299bfad70c41), [`5c889df`](https://github.com/browserbase/stagehand/commit/5c889dfef9659a1f57f4de641c2d4e79208a159a), [`a1ab39e`](https://github.com/browserbase/stagehand/commit/a1ab39e7aa805ac739d7bfa39dce9a2680bb1b17), [`f3fe7ce`](https://github.com/browserbase/stagehand/commit/f3fe7ce59743be8b5bdd070f749af7a9c6e84f19), [`5fb9785`](https://github.com/browserbase/stagehand/commit/5fb9785357fb724274bc615a226fe13c93d5ccb2), [`f5d1f1f`](https://github.com/browserbase/stagehand/commit/f5d1f1ff8072fa4acf979bc63f491c775a48991d), [`8bf5db8`](https://github.com/browserbase/stagehand/commit/8bf5db8eb8c69ce46fb8467d4216befe3a247f5b), [`6dc2276`](https://github.com/browserbase/stagehand/commit/6dc2276144ed48456a2b3e6525c5be47fa4eda18)]:
- @browserbasehq/stagehand@3.2.1
## 3.6.1
### Patch Changes
- Updated dependencies [[`505e8c6`](https://github.com/browserbase/stagehand/commit/505e8c6736f3706328dbc8df670c49a018058388), [`2f43ffa`](https://github.com/browserbase/stagehand/commit/2f43ffac11778152d17e4c44405770cc32c3ec8c), [`63ee247`](https://github.com/browserbase/stagehand/commit/63ee247ac6bf2992046d4f6b2759f46b15643e36), [`7dc35f5`](https://github.com/browserbase/stagehand/commit/7dc35f5e25689e6518d68b25ef71536d2781c8aa), [`335cf47`](https://github.com/browserbase/stagehand/commit/335cf4730e73bce33e92331d04bda4b0fd42685d), [`6ba0a1d`](https://github.com/browserbase/stagehand/commit/6ba0a1db7fc2d5d5a2f8927b1417d8f1d15eda10), [`4ff3bb8`](https://github.com/browserbase/stagehand/commit/4ff3bb831a6ef6e2d57148e7afb68ea8d23e395d), [`c27054b`](https://github.com/browserbase/stagehand/commit/c27054bbd0508431ade91d655f89efc87bbf5867), [`2abf5b9`](https://github.com/browserbase/stagehand/commit/2abf5b90f1e2bb1442509ef3a686b6128c9cdcf6), [`7817fcc`](https://github.com/browserbase/stagehand/commit/7817fcc315eee4455ce04567cf56c9ec801caf0b), [`7390508`](https://github.com/browserbase/stagehand/commit/73905088c5ed5923d276da9cce2efd0a0a3a46eb), [`611f43a`](https://github.com/browserbase/stagehand/commit/611f43ac8d4c580216d55d2b217c14a9a9c11013), [`521a10e`](https://github.com/browserbase/stagehand/commit/521a10e3698fc5631e219947bc90dad0f8bddaa8), [`2402a3c`](https://github.com/browserbase/stagehand/commit/2402a3c4d50270391b3e6440f4385cdcf5e1eb64)]:
- @browserbasehq/stagehand@3.2.0

View File

@ -1,36 +0,0 @@
# Stagehand API
The Stagehand is a powerful service that provides a RESTful interface for browser automation and session management using the Browserbase platform. It enables recording, playback, and manipulation of browser sessions with a focus on reliability and performance.
## 📋 Prerequisites
To run the Stagehand API locally, ensure you have the following installed:
- Node.js
- pnpm
## 🛠 Installation
1. Clone the repository:
```bash
git clone https://github.com/browserbase/stagehand/
cd stagehand/packages/server-v4
```
2. Install dependencies:
```bash
pnpm install
```
3. Set up environment variables:
```bash
cp .env.example .env
```
4. Configure your `.env` file with the environment variables required by `src/lib/env.ts` (BB environment, API base URLs, etc.).
5. `pnpm dev`

View File

@ -1,31 +0,0 @@
import path from "node:path";
import { defineConfig } from "drizzle-kit";
import { constants } from "./src/constants";
import { env } from "./src/env";
const drizzleConfig =
env.STAGEHAND_DB_MODE === "postgres"
? {
dialect: "postgresql" as const,
dbCredentials: {
url: env.DATABASE_URL,
},
}
: {
dialect: "postgresql" as const,
driver: "pglite" as const,
dbCredentials: {
url: path.resolve(
env.BROWSERBASE_CONFIG_DIR,
...constants.paths.pgliteDataDirSegments,
),
},
};
export default defineConfig({
out: "./drizzle",
schema: "./src/db/schema/index.ts",
...drizzleConfig,
strict: true,
verbose: true,
});

View File

@ -1,82 +0,0 @@
CREATE TYPE "llm_session_status" AS ENUM('disconnected', 'idle', 'thinking', 'permanent-error', 'ratelimited');--> statement-breakpoint
CREATE TYPE "llm_source" AS ENUM('user', 'system-default');--> statement-breakpoint
CREATE TYPE "stagehand_browser_session_status" AS ENUM('running', 'terminated');--> statement-breakpoint
CREATE TYPE "stagehand_step_operation" AS ENUM('act', 'extract', 'observe', 'agent');--> statement-breakpoint
CREATE TABLE "llm_calls" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
"llm_session_id" uuid NOT NULL,
"sent_at" timestamp with time zone DEFAULT now() NOT NULL,
"received_at" timestamp with time zone,
"prompt" text NOT NULL,
"expected_response_schema" jsonb,
"response" jsonb,
"error" jsonb,
"usage" jsonb,
"model" text NOT NULL
);
--> statement-breakpoint
CREATE TABLE "llm_configs" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
"source" "llm_source" DEFAULT 'user'::"llm_source" NOT NULL,
"display_name" text,
"model_name" text NOT NULL,
"base_url" text,
"system_prompt" text,
"provider_options" jsonb,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE "llm_sessions" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
"copied_template_id" uuid,
"forked_session_id" uuid,
"project_id" uuid NOT NULL,
"browser_session_id" uuid,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL,
"connected_at" timestamp with time zone,
"disconnected_at" timestamp with time zone,
"last_request_at" timestamp with time zone,
"last_response_at" timestamp with time zone,
"last_error_at" timestamp with time zone,
"last_error_message" text,
"status" "llm_session_status" NOT NULL,
"model" text NOT NULL,
"base_url" text,
"options" jsonb,
"extra_http_headers" jsonb,
"system_prompt" text,
"tokens_input" integer DEFAULT 0 NOT NULL,
"tokens_output" integer DEFAULT 0 NOT NULL,
"tokens_reasoning" integer DEFAULT 0 NOT NULL,
"tokens_cached_input" integer DEFAULT 0 NOT NULL,
"tokens_total" integer DEFAULT 0 NOT NULL
);
--> statement-breakpoint
CREATE TABLE "stagehand_browser_sessions" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
"project_id" uuid NOT NULL,
"browserbase_session_id" uuid,
"cdp_url" text NOT NULL,
"status" "stagehand_browser_session_status" NOT NULL,
"default_llm_session_id" uuid NOT NULL
);
--> statement-breakpoint
CREATE TABLE "stagehand_steps" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
"stagehand_browser_session_id" uuid NOT NULL,
"operation" "stagehand_step_operation" NOT NULL,
"llm_template_id" uuid NOT NULL,
"llm_session_id" uuid,
"params" jsonb NOT NULL,
"result" jsonb
);
--> statement-breakpoint
ALTER TABLE "llm_calls" ADD CONSTRAINT "llm_calls_llm_session_id_llm_sessions_id_fkey" FOREIGN KEY ("llm_session_id") REFERENCES "llm_sessions"("id") ON DELETE CASCADE;--> statement-breakpoint
ALTER TABLE "llm_sessions" ADD CONSTRAINT "llm_sessions_copied_template_id_fkey" FOREIGN KEY ("copied_template_id") REFERENCES "llm_sessions"("id") ON DELETE SET NULL;--> statement-breakpoint
ALTER TABLE "llm_sessions" ADD CONSTRAINT "llm_sessions_forked_session_id_fkey" FOREIGN KEY ("forked_session_id") REFERENCES "llm_sessions"("id") ON DELETE SET NULL;--> statement-breakpoint
ALTER TABLE "stagehand_browser_sessions" ADD CONSTRAINT "stagehand_browser_sessions_s0BMZHuHZkxx_fkey" FOREIGN KEY ("default_llm_session_id") REFERENCES "llm_sessions"("id") ON DELETE RESTRICT;--> statement-breakpoint
ALTER TABLE "stagehand_steps" ADD CONSTRAINT "stagehand_steps_kqGcAyNQHgqC_fkey" FOREIGN KEY ("stagehand_browser_session_id") REFERENCES "stagehand_browser_sessions"("id") ON DELETE CASCADE;--> statement-breakpoint
ALTER TABLE "stagehand_steps" ADD CONSTRAINT "stagehand_steps_llm_template_id_llm_sessions_id_fkey" FOREIGN KEY ("llm_template_id") REFERENCES "llm_sessions"("id") ON DELETE RESTRICT;--> statement-breakpoint
ALTER TABLE "stagehand_steps" ADD CONSTRAINT "stagehand_steps_llm_session_id_llm_sessions_id_fkey" FOREIGN KEY ("llm_session_id") REFERENCES "llm_sessions"("id") ON DELETE SET NULL;

View File

@ -1,979 +0,0 @@
{
"version": "8",
"dialect": "postgres",
"id": "742a2758-ab12-40d4-90e0-7ebada53ba93",
"prevIds": [
"00000000-0000-0000-0000-000000000000"
],
"ddl": [
{
"values": [
"disconnected",
"idle",
"thinking",
"permanent-error",
"ratelimited"
],
"name": "llm_session_status",
"entityType": "enums",
"schema": "public"
},
{
"values": [
"user",
"system-default"
],
"name": "llm_source",
"entityType": "enums",
"schema": "public"
},
{
"values": [
"running",
"terminated"
],
"name": "stagehand_browser_session_status",
"entityType": "enums",
"schema": "public"
},
{
"values": [
"act",
"extract",
"observe",
"agent"
],
"name": "stagehand_step_operation",
"entityType": "enums",
"schema": "public"
},
{
"isRlsEnabled": false,
"name": "llm_calls",
"entityType": "tables",
"schema": "public"
},
{
"isRlsEnabled": false,
"name": "llm_configs",
"entityType": "tables",
"schema": "public"
},
{
"isRlsEnabled": false,
"name": "llm_sessions",
"entityType": "tables",
"schema": "public"
},
{
"isRlsEnabled": false,
"name": "stagehand_browser_sessions",
"entityType": "tables",
"schema": "public"
},
{
"isRlsEnabled": false,
"name": "stagehand_steps",
"entityType": "tables",
"schema": "public"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "gen_random_uuid()",
"generated": null,
"identity": null,
"name": "id",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "llm_session_id",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "now()",
"generated": null,
"identity": null,
"name": "sent_at",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "received_at",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "text",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "prompt",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "jsonb",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "expected_response_schema",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "jsonb",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "response",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "jsonb",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "error",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "jsonb",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "usage",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "text",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "model",
"entityType": "columns",
"schema": "public",
"table": "llm_calls"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "gen_random_uuid()",
"generated": null,
"identity": null,
"name": "id",
"entityType": "columns",
"schema": "public",
"table": "llm_configs"
},
{
"type": "llm_source",
"typeSchema": "public",
"notNull": true,
"dimensions": 0,
"default": "'user'",
"generated": null,
"identity": null,
"name": "source",
"entityType": "columns",
"schema": "public",
"table": "llm_configs"
},
{
"type": "text",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "display_name",
"entityType": "columns",
"schema": "public",
"table": "llm_configs"
},
{
"type": "text",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "model_name",
"entityType": "columns",
"schema": "public",
"table": "llm_configs"
},
{
"type": "text",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "base_url",
"entityType": "columns",
"schema": "public",
"table": "llm_configs"
},
{
"type": "text",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "system_prompt",
"entityType": "columns",
"schema": "public",
"table": "llm_configs"
},
{
"type": "jsonb",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "provider_options",
"entityType": "columns",
"schema": "public",
"table": "llm_configs"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "now()",
"generated": null,
"identity": null,
"name": "created_at",
"entityType": "columns",
"schema": "public",
"table": "llm_configs"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "now()",
"generated": null,
"identity": null,
"name": "updated_at",
"entityType": "columns",
"schema": "public",
"table": "llm_configs"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "gen_random_uuid()",
"generated": null,
"identity": null,
"name": "id",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "copied_template_id",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "forked_session_id",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "project_id",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "browser_session_id",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "now()",
"generated": null,
"identity": null,
"name": "created_at",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "now()",
"generated": null,
"identity": null,
"name": "updated_at",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "connected_at",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "disconnected_at",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "last_request_at",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "last_response_at",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "timestamp with time zone",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "last_error_at",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "text",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "last_error_message",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "llm_session_status",
"typeSchema": "public",
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "status",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "text",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "model",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "text",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "base_url",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "jsonb",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "options",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "jsonb",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "extra_http_headers",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "text",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "system_prompt",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "integer",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "0",
"generated": null,
"identity": null,
"name": "tokens_input",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "integer",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "0",
"generated": null,
"identity": null,
"name": "tokens_output",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "integer",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "0",
"generated": null,
"identity": null,
"name": "tokens_reasoning",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "integer",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "0",
"generated": null,
"identity": null,
"name": "tokens_cached_input",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "integer",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "0",
"generated": null,
"identity": null,
"name": "tokens_total",
"entityType": "columns",
"schema": "public",
"table": "llm_sessions"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "gen_random_uuid()",
"generated": null,
"identity": null,
"name": "id",
"entityType": "columns",
"schema": "public",
"table": "stagehand_browser_sessions"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "project_id",
"entityType": "columns",
"schema": "public",
"table": "stagehand_browser_sessions"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "browserbase_session_id",
"entityType": "columns",
"schema": "public",
"table": "stagehand_browser_sessions"
},
{
"type": "text",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "cdp_url",
"entityType": "columns",
"schema": "public",
"table": "stagehand_browser_sessions"
},
{
"type": "stagehand_browser_session_status",
"typeSchema": "public",
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "status",
"entityType": "columns",
"schema": "public",
"table": "stagehand_browser_sessions"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "default_llm_session_id",
"entityType": "columns",
"schema": "public",
"table": "stagehand_browser_sessions"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": "gen_random_uuid()",
"generated": null,
"identity": null,
"name": "id",
"entityType": "columns",
"schema": "public",
"table": "stagehand_steps"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "stagehand_browser_session_id",
"entityType": "columns",
"schema": "public",
"table": "stagehand_steps"
},
{
"type": "stagehand_step_operation",
"typeSchema": "public",
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "operation",
"entityType": "columns",
"schema": "public",
"table": "stagehand_steps"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "llm_template_id",
"entityType": "columns",
"schema": "public",
"table": "stagehand_steps"
},
{
"type": "uuid",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "llm_session_id",
"entityType": "columns",
"schema": "public",
"table": "stagehand_steps"
},
{
"type": "jsonb",
"typeSchema": null,
"notNull": true,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "params",
"entityType": "columns",
"schema": "public",
"table": "stagehand_steps"
},
{
"type": "jsonb",
"typeSchema": null,
"notNull": false,
"dimensions": 0,
"default": null,
"generated": null,
"identity": null,
"name": "result",
"entityType": "columns",
"schema": "public",
"table": "stagehand_steps"
},
{
"nameExplicit": false,
"columns": [
"llm_session_id"
],
"schemaTo": "public",
"tableTo": "llm_sessions",
"columnsTo": [
"id"
],
"onUpdate": "NO ACTION",
"onDelete": "CASCADE",
"name": "llm_calls_llm_session_id_llm_sessions_id_fkey",
"entityType": "fks",
"schema": "public",
"table": "llm_calls"
},
{
"nameExplicit": true,
"columns": [
"copied_template_id"
],
"schemaTo": "public",
"tableTo": "llm_sessions",
"columnsTo": [
"id"
],
"onUpdate": "NO ACTION",
"onDelete": "SET NULL",
"name": "llm_sessions_copied_template_id_fkey",
"entityType": "fks",
"schema": "public",
"table": "llm_sessions"
},
{
"nameExplicit": true,
"columns": [
"forked_session_id"
],
"schemaTo": "public",
"tableTo": "llm_sessions",
"columnsTo": [
"id"
],
"onUpdate": "NO ACTION",
"onDelete": "SET NULL",
"name": "llm_sessions_forked_session_id_fkey",
"entityType": "fks",
"schema": "public",
"table": "llm_sessions"
},
{
"nameExplicit": false,
"columns": [
"default_llm_session_id"
],
"schemaTo": "public",
"tableTo": "llm_sessions",
"columnsTo": [
"id"
],
"onUpdate": "NO ACTION",
"onDelete": "RESTRICT",
"name": "stagehand_browser_sessions_s0BMZHuHZkxx_fkey",
"entityType": "fks",
"schema": "public",
"table": "stagehand_browser_sessions"
},
{
"nameExplicit": false,
"columns": [
"stagehand_browser_session_id"
],
"schemaTo": "public",
"tableTo": "stagehand_browser_sessions",
"columnsTo": [
"id"
],
"onUpdate": "NO ACTION",
"onDelete": "CASCADE",
"name": "stagehand_steps_kqGcAyNQHgqC_fkey",
"entityType": "fks",
"schema": "public",
"table": "stagehand_steps"
},
{
"nameExplicit": false,
"columns": [
"llm_template_id"
],
"schemaTo": "public",
"tableTo": "llm_sessions",
"columnsTo": [
"id"
],
"onUpdate": "NO ACTION",
"onDelete": "RESTRICT",
"name": "stagehand_steps_llm_template_id_llm_sessions_id_fkey",
"entityType": "fks",
"schema": "public",
"table": "stagehand_steps"
},
{
"nameExplicit": false,
"columns": [
"llm_session_id"
],
"schemaTo": "public",
"tableTo": "llm_sessions",
"columnsTo": [
"id"
],
"onUpdate": "NO ACTION",
"onDelete": "SET NULL",
"name": "stagehand_steps_llm_session_id_llm_sessions_id_fkey",
"entityType": "fks",
"schema": "public",
"table": "stagehand_steps"
},
{
"columns": [
"id"
],
"nameExplicit": false,
"name": "llm_calls_pkey",
"schema": "public",
"table": "llm_calls",
"entityType": "pks"
},
{
"columns": [
"id"
],
"nameExplicit": false,
"name": "llm_configs_pkey",
"schema": "public",
"table": "llm_configs",
"entityType": "pks"
},
{
"columns": [
"id"
],
"nameExplicit": false,
"name": "llm_sessions_pkey",
"schema": "public",
"table": "llm_sessions",
"entityType": "pks"
},
{
"columns": [
"id"
],
"nameExplicit": false,
"name": "stagehand_browser_sessions_pkey",
"schema": "public",
"table": "stagehand_browser_sessions",
"entityType": "pks"
},
{
"columns": [
"id"
],
"nameExplicit": false,
"name": "stagehand_steps_pkey",
"schema": "public",
"table": "stagehand_steps",
"entityType": "pks"
}
],
"renames": []
}

File diff suppressed because it is too large Load Diff

View File

@ -1,72 +0,0 @@
{
"name": "@browserbasehq/stagehand-server-v4",
"version": "3.6.4",
"description": "Stagehand API server v4",
"type": "module",
"private": true,
"scripts": {
"build": "pnpm --filter @browserbasehq/stagehand-server-v4 run --parallel \"/^(build:server:dist|gen:openapi|build:sea:esm)$/\"",
"db:generate": "drizzle-kit generate --config ./drizzle.config.ts",
"db:migrate": "drizzle-kit migrate --config ./drizzle.config.ts",
"db:push": "drizzle-kit push --config ./drizzle.config.ts",
"db:studio": "drizzle-kit studio --config ./drizzle.config.ts",
"start": "tsx src/server.ts",
"dev": "tsx watch src/server.ts",
"build:server:dist": "pnpm -w --dir ../.. exec tsc -p packages/server-v4/tsconfig.json && pnpm -w --dir ../.. exec tsc-alias -p packages/server-v4/tsconfig.json",
"build:sea:esm": "tsx scripts/build-sea.ts --mode=esm",
"build:sea:cjs": "tsx scripts/build-sea.ts --mode=cjs",
"lint": "cd ../.. && prettier --check packages/server-v4 && cd packages/server-v4 && eslint . && pnpm run typecheck",
"typecheck": "pnpm -w --dir ../.. exec tsc -p packages/server-v4/tsconfig.json --noEmit",
"test": "pnpm -w --dir ../.. exec turbo run test:server --filter=@browserbasehq/stagehand-server-v4 --",
"test:server": "tsx scripts/test-server.ts",
"test:unit": "pnpm run test:server -- packages/server-v4/test/unit",
"test:integration": "pnpm run test:server -- packages/server-v4/test/integration",
"test:integration:local": "STAGEHAND_SERVER_TARGET=local pnpm run test:server -- packages/server-v4/test/integration",
"test:integration:sea": "STAGEHAND_SERVER_TARGET=sea pnpm run test:server -- packages/server-v4/test/integration",
"gen:openapi": "tsx scripts/gen-openapi.ts"
},
"dependencies": {
"@browserbasehq/sdk": "^2.10.0",
"@browserbasehq/stagehand": "workspace:*",
"@electric-sql/pglite": "0.4.2",
"@fastify/cors": "^11.0.1",
"@fastify/swagger": "^9.6.1",
"@fastify/swagger-ui": "^5.2.6",
"@t3-oss/env-core": "^0.13.11",
"drizzle-orm": "1.0.0-beta.20",
"fastify": "^5.8.5",
"fastify-metrics": "^12.1.0",
"fastify-plugin": "^4.5.1",
"fastify-zod-openapi": "^5.5.0",
"http-status-codes": "^2.3.0",
"pino": "^9.7.0",
"pino-pretty": "^11.3.0",
"playwright": "^1.55.1",
"postgres": "^3.4.8",
"uuid": "^11.1.1",
"zod": "^4.2.1"
},
"devDependencies": {
"@types/node": "22.13.1",
"drizzle-kit": "1.0.0-beta.19",
"eslint": "10.0.2",
"eslint-plugin-security": "^3.0.1",
"openai": "4.87.1",
"postject": "1.0.0-alpha.6",
"prettier": "^3.2.5",
"source-map": "^0.7.4",
"tsc-alias": "^1.8.10",
"tsx": "*",
"vite": "^7.3.3",
"vitest": "^4.1.6"
},
"repository": {
"type": "git",
"url": "git+https://github.com/browserbase/stagehand.git",
"directory": "packages/server-v4"
},
"bugs": {
"url": "https://github.com/browserbase/stagehand/issues"
},
"homepage": "https://stagehand.dev"
}

View File

@ -1,521 +0,0 @@
#!/usr/bin/env node
/**
* Build SEA binary from ESM (test) or CJS (release) bundles.
*
* Prereqs:
* - CJS mode: runs core CJS build via Turbo if dist is missing.
* - ESM mode: core dist/esm available (pnpm run build:esm).
* - postject installed; tar available for non-Windows downloads.
*
* Args: --mode=esm|cjs --target-platform=<platform> --target-arch=<arch> --binary-name=<name>
* Env: SEA_BUILD_MODE, SEA_TARGET_PLATFORM, SEA_TARGET_ARCH, SEA_BINARY_NAME,
* SEA_INCLUDE_SOURCEMAPS.
* Example: pnpm run build:sea:cjs -- --target-platform=linux --target-arch=arm64
*/
import { spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import https from "node:https";
import { pathToFileURL } from "node:url";
import esbuild from "esbuild";
import { getRepoRootDir } from "./runtimePaths.js";
const repoDir = getRepoRootDir();
const seaFuse = "NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2";
const argValue = (name: string) => {
const prefix = `--${name}=`;
for (let i = 0; i < process.argv.length; i++) {
const arg = process.argv[i];
if (arg === `--${name}` && process.argv[i + 1]) return process.argv[i + 1];
if (arg.startsWith(prefix)) return arg.slice(prefix.length);
}
return undefined;
};
const mode = (
argValue("mode") ??
process.env.SEA_BUILD_MODE ??
"esm"
).toLowerCase();
const parseBoolean = (
value: string | undefined,
fallback: boolean,
): boolean => {
if (value === undefined) return fallback;
const normalized = value.toLowerCase();
if (
normalized === "1" ||
normalized === "true" ||
normalized === "yes" ||
normalized === "on"
) {
return true;
}
if (
normalized === "0" ||
normalized === "false" ||
normalized === "no" ||
normalized === "off"
) {
return false;
}
throw new Error(
`Invalid boolean value "${value}" for --include-sourcemaps / SEA_INCLUDE_SOURCEMAPS`,
);
};
const targetPlatform =
argValue("target-platform") ??
argValue("platform") ??
process.env.SEA_TARGET_PLATFORM ??
process.platform;
const targetArch =
argValue("target-arch") ??
argValue("arch") ??
process.env.SEA_TARGET_ARCH ??
process.arch;
const binaryName =
argValue("binary-name") ??
process.env.SEA_BINARY_NAME ??
`stagehand-server-v4-${targetPlatform}-${targetArch}${targetPlatform === "win32" ? ".exe" : ""}`;
const includeSourcemaps = parseBoolean(
argValue("include-sourcemaps") ?? process.env.SEA_INCLUDE_SOURCEMAPS,
false,
);
const run = (cmd: string, args: string[], opts: { cwd?: string } = {}) => {
const result = spawnSync(cmd, args, { stdio: "inherit", ...opts });
if (result.error) {
throw new Error(
`Command failed to start: ${cmd} ${args.join(" ")}\n${String(result.error)}`,
);
}
if (result.status !== 0) {
throw new Error(`Command failed: ${cmd} ${args.join(" ")}`);
}
};
const runNodeScript = (
scriptPath: string,
args: string[],
opts: { cwd?: string } = {},
) => run(process.execPath, [scriptPath, ...args], opts);
const resolveFirstExisting = (paths: string[]): string => {
for (const candidate of paths) {
if (fs.existsSync(candidate)) return candidate;
}
throw new Error(`Missing tool script. Tried: ${paths.join(", ")}`);
};
const runOptional = (
cmd: string,
args: string[],
opts: { cwd?: string } = {},
) => {
spawnSync(cmd, args, { stdio: "ignore", ...opts });
};
const hasSeaFuse = (binaryPath: string): boolean => {
try {
return fs.readFileSync(binaryPath).includes(Buffer.from(seaFuse));
} catch {
return false;
}
};
const download = (url: string, dest: string): Promise<void> =>
new Promise((resolve, reject) => {
https
.get(url, (res) => {
if (res.statusCode && res.statusCode >= 300 && res.statusCode < 400) {
const location = res.headers.location;
if (!location) {
reject(new Error(`Redirect without location: ${url}`));
return;
}
res.resume();
download(location, dest).then(resolve, reject);
return;
}
if (res.statusCode !== 200) {
reject(new Error(`Download failed (${res.statusCode}) ${url}`));
res.resume();
return;
}
const file = fs.createWriteStream(dest);
const fail = (error: Error) => {
file.destroy();
reject(error);
};
res.on("error", fail);
file.on("error", fail);
file.on("finish", () => {
file.close((closeError) => {
if (closeError) {
reject(closeError);
return;
}
resolve();
});
});
res.pipe(file);
})
.on("error", reject);
});
const resolveNodeBinary = async (): Promise<string> => {
if (targetPlatform !== process.platform) {
throw new Error(
`Cross-platform builds are not supported. Host=${process.platform}, target=${targetPlatform}`,
);
}
if (targetArch === process.arch && hasSeaFuse(process.execPath)) {
return process.execPath;
}
if (targetArch === process.arch) {
console.warn(
`Current Node binary at ${process.execPath} does not include ${seaFuse}; falling back to the official ${process.version} distribution for SEA injection.`,
);
}
const version = process.version;
const distPlatform = targetPlatform === "win32" ? "win" : targetPlatform;
const archiveBase = `node-${version}-${distPlatform}-${targetArch}`;
const archiveExt = distPlatform === "win" ? "zip" : "tar.xz";
const tmpRoot = `${os.tmpdir()}/stagehand-server-v4-sea/${archiveBase}`;
const archivePath = `${tmpRoot}/${archiveBase}.${archiveExt}`;
const extractRoot = `${tmpRoot}/${archiveBase}`;
const binaryPath =
distPlatform === "win"
? `${extractRoot}/node.exe`
: `${extractRoot}/bin/node`;
if (fs.existsSync(binaryPath)) {
if (!hasSeaFuse(binaryPath)) {
throw new Error(
`Node binary at ${binaryPath} does not include ${seaFuse}; unable to build SEA binary. Delete ${tmpRoot} and retry.`,
);
}
return binaryPath;
}
fs.mkdirSync(tmpRoot, { recursive: true });
if (!fs.existsSync(archivePath)) {
const url = `https://nodejs.org/dist/${version}/${archiveBase}.${archiveExt}`;
await download(url, archivePath);
}
if (archiveExt === "zip") {
if (process.platform !== "win32") {
throw new Error("Windows binaries must be built on Windows runners.");
}
run("powershell", [
"-Command",
`Expand-Archive -Path '${archivePath}' -DestinationPath '${tmpRoot}' -Force`,
]);
} else {
run("tar", ["-xf", archivePath, "-C", tmpRoot]);
}
if (!fs.existsSync(binaryPath)) {
throw new Error(`Missing Node binary at ${binaryPath}`);
}
if (!hasSeaFuse(binaryPath)) {
throw new Error(
`Node binary at ${binaryPath} does not include ${seaFuse}; unable to build SEA binary. Delete ${tmpRoot} and retry.`,
);
}
return binaryPath;
};
const writeSeaConfig = (
mainPath: string,
outputPath: string,
execArgvExtension?: string,
) => {
const configPath = `${repoDir}/packages/server-v4/dist/sea/sea-config-${mode}.json`;
const config = {
main: path
.relative(`${repoDir}/packages/server-v4`, mainPath)
.replaceAll("\\", "/"),
output: path
.relative(`${repoDir}/packages/server-v4`, outputPath)
.replaceAll("\\", "/"),
...(execArgvExtension ? { execArgvExtension } : {}),
};
fs.writeFileSync(configPath, JSON.stringify(config, null, 2));
return configPath;
};
const buildCjsBundle = () => {
const turboBin = resolveFirstExisting([
`${repoDir}/node_modules/turbo/bin/turbo`,
]);
runNodeScript(
turboBin,
["run", "build:cjs", "--filter", "@browserbasehq/stagehand"],
{
cwd: repoDir,
},
);
fs.mkdirSync(`${repoDir}/packages/server-v4/dist/sea`, { recursive: true });
const bundlePath = `${repoDir}/packages/server-v4/dist/sea/bundle.cjs`;
esbuild.buildSync({
entryPoints: ["packages/server-v4/src/sea-entry.ts"],
bundle: true,
platform: "node",
format: "cjs",
outfile: bundlePath,
logLevel: "warning",
absWorkingDir: repoDir,
});
return bundlePath;
};
const buildEsmBundle = () => {
if (!fs.existsSync(`${repoDir}/packages/core/dist/esm/index.js`)) {
throw new Error(
`Missing ${repoDir}/packages/core/dist/esm/index.js. Run pnpm run build:esm first.`,
);
}
fs.mkdirSync(`${repoDir}/packages/server-v4/dist/sea`, { recursive: true });
const appBundlePath = `${repoDir}/packages/server-v4/dist/app.mjs`;
esbuild.buildSync({
entryPoints: ["packages/server-v4/src/sea-entry.ts"],
bundle: true,
platform: "node",
format: "esm",
treeShaking: false,
outfile: appBundlePath,
alias: {
"@browserbasehq/stagehand": `${repoDir}/packages/core/dist/esm/index.js`,
},
sourcemap: includeSourcemaps ? "inline" : false,
sourcesContent: includeSourcemaps,
...(includeSourcemaps ? { sourceRoot: repoDir } : {}),
banner: {
js: 'import { createRequire as __createRequire } from "node:module"; const require = __createRequire(import.meta.url);',
},
logLevel: "warning",
absWorkingDir: repoDir,
});
const appSource = fs.readFileSync(appBundlePath, "utf8");
let finalAppSource = appSource;
if (includeSourcemaps) {
const mapMatch = appSource.match(
/sourceMappingURL=data:application\/json;base64,([A-Za-z0-9+/=]+)\s*$/,
);
if (!mapMatch) {
throw new Error("Missing inline sourcemap in dist/app.mjs");
}
const mapJson = Buffer.from(mapMatch[1], "base64").toString("utf8");
const map = JSON.parse(mapJson) as {
sourceRoot?: string;
sources: string[];
sourcesContent?: string[];
};
const toPosix = (value: string) => value.replaceAll("\\", "/");
const fileUrlToPathSafe = (value: string) => {
const parsed = new URL(value);
let pathname = decodeURIComponent(parsed.pathname);
if (/^\/[A-Za-z]:/.test(pathname)) {
pathname = pathname.slice(1);
}
return pathname;
};
const toRepoRelative = (source: string) => {
let sourcePath = source;
if (source.startsWith("file://")) {
sourcePath = fileUrlToPathSafe(source);
}
if (path.isAbsolute(sourcePath)) {
const normalizedSourcePath = toPosix(sourcePath);
if (normalizedSourcePath.startsWith(`${repoDir}/`)) {
return toPosix(path.relative(repoDir, normalizedSourcePath));
}
return normalizedSourcePath;
}
if (sourcePath.startsWith("../src/")) {
const rel = sourcePath.slice("../src/".length);
return `packages/server-v4/src/${rel}`;
}
if (sourcePath.startsWith("../../core/")) {
const rel = sourcePath.slice("../../core/".length);
return `packages/core/${rel}`;
}
if (sourcePath.startsWith("../../../node_modules/")) {
const rel = sourcePath.slice("../../../node_modules/".length);
return `node_modules/${rel}`;
}
if (sourcePath.startsWith("src/")) {
const rel = sourcePath.slice("src/".length);
return `packages/server-v4/src/${rel}`;
}
if (sourcePath.startsWith("../node_modules/")) {
const rel = sourcePath.slice("../node_modules/".length);
return `node_modules/${rel}`;
}
if (sourcePath.startsWith("../core/")) {
const rel = sourcePath.slice("../core/".length);
return `packages/core/${rel}`;
}
if (sourcePath.startsWith("core/")) {
return `packages/core/${sourcePath.slice("core/".length)}`;
}
if (
sourcePath.startsWith("packages/") ||
sourcePath.startsWith("node_modules/")
) {
return toPosix(sourcePath);
}
const resolved = toPosix(
path.resolve(`${repoDir}/packages/server-v4`, sourcePath),
);
if (resolved.startsWith(`${repoDir}/`)) {
return toPosix(path.relative(repoDir, resolved));
}
return toPosix(sourcePath);
};
map.sourceRoot = pathToFileURL(`${repoDir}/`).href;
map.sources = map.sources.map(toRepoRelative);
const updatedMap = Buffer.from(JSON.stringify(map)).toString("base64");
finalAppSource = appSource.replace(mapMatch[1], updatedMap);
fs.writeFileSync(appBundlePath, finalAppSource);
}
const appBytes = Buffer.from(finalAppSource);
const bundleHash = createHash("sha256")
.update(appBytes)
.digest("hex")
.slice(0, 12);
const bootstrapPath = `${repoDir}/packages/server-v4/dist/sea/sea-bootstrap.cjs`;
const bootstrap = `/* eslint-disable */
const fs = require("node:fs");
const os = require("node:os");
const { pathToFileURL } = require("node:url");
const bundleBase64 = ${JSON.stringify(appBytes.toString("base64"))};
const bundleLength = ${appBytes.length};
const bundleHash = ${JSON.stringify(bundleHash)};
const cacheRoot =
process.env.STAGEHAND_SEA_CACHE_DIR ||
\`\${os.tmpdir()}/stagehand-server-v4-sea\`;
const cacheDir = \`\${cacheRoot}/\${bundleHash}\`;
const appPath = \`\${cacheDir}/app.mjs\`;
fs.mkdirSync(cacheDir, { recursive: true });
let needsWrite = true;
try {
const stat = fs.statSync(appPath);
needsWrite = stat.size !== bundleLength;
} catch {}
if (needsWrite) {
const tmpPath =
\`\${cacheDir}/app.mjs.tmp-\${process.pid}-\${Date.now().toString(16)}\`;
fs.writeFileSync(tmpPath, Buffer.from(bundleBase64, "base64"));
try {
fs.renameSync(tmpPath, appPath);
} catch (err) {
if (!fs.existsSync(appPath)) throw err;
}
try {
fs.chmodSync(appPath, 0o500);
} catch {}
}
(async () => {
await import(pathToFileURL(appPath).href);
})().catch((err) => {
console.error(err);
process.exitCode = 1;
});
`;
fs.writeFileSync(bootstrapPath, bootstrap);
return bootstrapPath;
};
const main = async () => {
fs.mkdirSync(`${repoDir}/packages/server-v4/dist/sea`, { recursive: true });
let mainPath: string;
let execArgvExtension: string | undefined;
if (mode === "cjs") {
mainPath = buildCjsBundle();
} else if (mode === "esm") {
mainPath = buildEsmBundle();
execArgvExtension = "cli";
} else {
throw new Error(`Unknown SEA build mode: ${mode}`);
}
const seaConfigPath = writeSeaConfig(
mainPath,
`${repoDir}/packages/server-v4/dist/sea/sea-prep.blob`,
execArgvExtension,
);
run("node", ["--experimental-sea-config", seaConfigPath], {
cwd: `${repoDir}/packages/server-v4`,
});
if (!fs.existsSync(`${repoDir}/packages/server-v4/dist/sea/sea-prep.blob`)) {
throw new Error(
`Missing ${repoDir}/packages/server-v4/dist/sea/sea-prep.blob; SEA blob generation failed.`,
);
}
const nodeBinary = await resolveNodeBinary();
const outPath = `${repoDir}/packages/server-v4/dist/sea/${binaryName}`;
fs.copyFileSync(nodeBinary, outPath);
if (targetPlatform !== "win32") {
fs.chmodSync(outPath, 0o755);
}
if (targetPlatform === "darwin") {
runOptional("codesign", ["--remove-signature", outPath]);
}
const postjectCliPath = resolveFirstExisting([
`${repoDir}/packages/server-v4/node_modules/postject/dist/cli.js`,
`${repoDir}/node_modules/postject/dist/cli.js`,
]);
const postjectArgs = [
outPath,
"NODE_SEA_BLOB",
`${repoDir}/packages/server-v4/dist/sea/sea-prep.blob`,
"--sentinel-fuse",
seaFuse,
];
if (targetPlatform === "darwin") {
postjectArgs.push("--macho-segment-name", "NODE_SEA");
}
runNodeScript(postjectCliPath, postjectArgs, {
cwd: `${repoDir}/packages/server-v4`,
});
if (targetPlatform === "darwin") {
runOptional("codesign", ["--sign", "-", outPath]);
}
};
main().catch((err) => {
console.error(err instanceof Error ? err.message : String(err));
process.exit(1);
});

View File

@ -1,126 +0,0 @@
import { writeFile } from "node:fs/promises";
import path from "node:path";
import { getCurrentDirPath } from "./runtimePaths.js";
import fastify from "fastify";
import fastifySwagger from "@fastify/swagger";
import {
fastifyZodOpenApiPlugin,
fastifyZodOpenApiTransformers,
serializerCompiler,
validatorCompiler,
type FastifyZodOpenApiTypeProvider,
} from "fastify-zod-openapi";
import { Api } from "@browserbasehq/stagehand";
import { browserSessionOpenApiComponents } from "../src/schemas/v4/browserSession.js";
import { llmOpenApiComponents } from "../src/schemas/v4/llm.js";
import { pageOpenApiComponents } from "../src/schemas/v4/page.js";
import { browserSessionRoutes } from "../src/routes/v4/browsersession/routes.js";
import { llmRoutes } from "../src/routes/v4/llms/routes.js";
import { pageRoutes } from "../src/routes/v4/page/routes.js";
// Routes
import healthcheckRoute from "../src/routes/healthcheck.js";
import readinessRoute from "../src/routes/readiness.js";
const OUTPUT_PATH = path.resolve(getCurrentDirPath(), "../openapi.v4.yaml");
async function main() {
const app = fastify({
logger: false,
}).withTypeProvider<FastifyZodOpenApiTypeProvider>();
app.setValidatorCompiler(validatorCompiler);
app.setSerializerCompiler(serializerCompiler);
// Register all API schemas as components so fastify-zod-openapi can create $ref links
const components = {
schemas: {
...browserSessionOpenApiComponents.schemas,
...llmOpenApiComponents.schemas,
...pageOpenApiComponents.schemas,
},
};
await app.register(fastifyZodOpenApiPlugin, { components });
await app.register(fastifySwagger, {
openapi: {
info: {
title: "Stagehand API v4",
version: "4.0.0",
description: `Stagehand SDK for AI browser automation [ALPHA]. This API allows clients to
execute browser automation tasks remotely on the Browserbase cloud.
Create a browser session with /browsersession, then use that id with page routes.
Responses are streamed using Server-Sent Events (SSE) when the
\`x-stream-response: true\` header is provided.
This SDK is currently ALPHA software and is not production ready!
Please try it and give us your feedback, stay tuned for upcoming release announcements!`,
contact: {
name: "Browserbase",
url: "https://browserbase.com",
},
},
openapi: "3.1.0",
servers: [
{
url: "https://api.stagehand.browserbase.com",
},
],
components: {
securitySchemes: Api.openApiSecuritySchemes,
links: Api.openApiLinks,
},
security: [
{ BrowserbaseApiKey: [], BrowserbaseProjectId: [], ModelApiKey: [] },
],
},
...fastifyZodOpenApiTransformers,
});
await app.register(
(instance, _opts, done) => {
for (const route of browserSessionRoutes) {
instance.route(route);
}
for (const route of llmRoutes) {
instance.route(route);
}
for (const route of pageRoutes) {
instance.route(route);
}
done();
},
{ prefix: "/v4" },
);
app.route(healthcheckRoute);
app.route(readinessRoute);
await app.ready();
const yaml = app.swagger({ yaml: true });
// Mintlify expects OpenAPI version fields to be strings, so quote them here.
const fixedYaml = yaml
.replace(/^openapi:\s*(?!['"])([^#\s]+)\s*$/m, 'openapi: "$1"')
.replace(/^ {2}version:\s*(?!['"])([^#\s]+)\s*$/m, ' version: "$1"')
.replace(
"description: Wait for captcha solves (deprecated, v2 only)",
"description: Wait for captcha solves",
)
.replace(
"description: Timeout in ms for act operations (deprecated, v2 only)",
"description: Timeout in ms for act operations",
);
await writeFile(OUTPUT_PATH, fixedYaml, "utf8");
await app.close();
console.log(`OpenAPI spec written to ${OUTPUT_PATH}`);
}
main().catch((err) => {
console.error(err);
process.exit(1);
});

View File

@ -1,122 +0,0 @@
/**
* Keep this file in sync with:
* - /packages/core/lib/v3/runtimePaths.ts
* - /packages/server-v4/scripts/runtimePaths.ts
* - /packages/evals/runtimePaths.ts
* - /packages/docs/scripts/runtimePaths.js
*/
import path from "node:path";
import { fileURLToPath } from "node:url";
import { createRequire } from "node:module";
const PACKAGE_SEGMENT = "/packages/server-v4/";
const EVAL_FRAMES = new Set(["[eval]", "[eval]-wrapper"]);
const INTERNAL_FRAME_NAMES = new Set([
"readCallsites",
"readCallsitePath",
"resolveCallerFilePath",
"getCurrentFilePath",
"getCurrentDirPath",
"getRepoRootDir",
"getPackageRootDir",
"createRequireFromCaller",
"isMainModule",
]);
const normalizePath = (value: string): string => {
const input = value.startsWith("file://") ? fileURLToPath(value) : value;
return path.resolve(input).replaceAll("\\", "/");
};
const readCallsites = (): NodeJS.CallSite[] => {
const previousPrepare = Error.prepareStackTrace;
try {
Error.prepareStackTrace = (_, stack) => stack;
return (
(new Error().stack as unknown as NodeJS.CallSite[] | undefined) ?? []
);
} finally {
Error.prepareStackTrace = previousPrepare;
}
};
type CallSiteWithScriptName = NodeJS.CallSite & {
getScriptNameOrSourceURL?: () => string | null;
};
const readCallsitePath = (callsite: NodeJS.CallSite): string | null => {
const callsiteWithScript = callsite as CallSiteWithScriptName;
const rawPath =
callsite.getFileName() ?? callsiteWithScript.getScriptNameOrSourceURL?.();
if (!rawPath) return null;
if (rawPath.startsWith("node:")) return null;
if (EVAL_FRAMES.has(rawPath)) return null;
return normalizePath(rawPath);
};
const isInternalCallsite = (callsite: NodeJS.CallSite): boolean => {
const functionName = callsite.getFunctionName();
if (functionName && INTERNAL_FRAME_NAMES.has(functionName)) return true;
const methodName = callsite.getMethodName();
if (methodName && INTERNAL_FRAME_NAMES.has(methodName)) return true;
const callsiteString = callsite.toString();
for (const frameName of INTERNAL_FRAME_NAMES) {
if (callsiteString.includes(`${frameName} (`)) return true;
if (callsiteString.includes(`.${frameName} (`)) return true;
}
return false;
};
const resolveCallerFilePath = (): string => {
const packageCandidates: string[] = [];
const fallbackCandidates: string[] = [];
for (const callsite of readCallsites()) {
const filePath = readCallsitePath(callsite);
if (!filePath) continue;
if (isInternalCallsite(callsite)) continue;
if (filePath.includes(PACKAGE_SEGMENT)) {
packageCandidates.push(filePath);
continue;
}
fallbackCandidates.push(filePath);
}
const packageCandidate = packageCandidates[0];
if (packageCandidate) return packageCandidate;
const fallbackCandidate = fallbackCandidates[0];
if (fallbackCandidate) return fallbackCandidate;
throw new Error("Unable to resolve caller file path.");
};
export const getCurrentFilePath = (): string => resolveCallerFilePath();
export const getCurrentDirPath = (): string =>
path.dirname(getCurrentFilePath());
export const getRepoRootDir = (): string => {
const currentFilePath = getCurrentFilePath();
const index = currentFilePath.lastIndexOf(PACKAGE_SEGMENT);
if (index === -1) {
throw new Error(
`Unable to determine repo root from ${currentFilePath} (missing ${PACKAGE_SEGMENT}).`,
);
}
return currentFilePath.slice(0, index);
};
export const getPackageRootDir = (): string =>
`${getRepoRootDir()}${PACKAGE_SEGMENT.slice(0, -1)}`;
export const createRequireFromCaller = () =>
createRequire(getCurrentFilePath());
export const isMainModule = (): boolean => {
const entryScript = process.argv.at(1);
if (!entryScript) return false;
return normalizePath(entryScript) === getCurrentFilePath();
};

View File

@ -1,384 +0,0 @@
/**
* Server unit + integration tests on source TS tests + local/SEA/remote server targets.
*
* Prereqs:
* - pnpm run build for local/SEA integration (packages/server-v4/dist/server.js).
* - SEA integration still requires build:sea when STAGEHAND_SERVER_TARGET=sea.
*
* Args: [test paths...] -- [node --test args...] | --list (prints JSON matrix)
* Env: STAGEHAND_SERVER_TARGET=sea|local|remote, STAGEHAND_BASE_URL, SEA_BINARY_NAME,
* NODE_TEST_CONSOLE_REPORTER, NODE_TEST_REPORTER, NODE_TEST_REPORTER_DESTINATION,
* NODE_V8_COVERAGE; writes CTRF to ctrf/node-test-*.xml by default.
* Example: STAGEHAND_SERVER_TARGET=sea pnpm run test:server -- packages/server-v4/test/integration/v4/page.test.ts
*/
import fs from "node:fs";
import path from "node:path";
import { spawn, spawnSync } from "node:child_process";
import { getRepoRootDir } from "./runtimePaths.js";
const ensureParentDir = (filePath: string) => {
fs.mkdirSync(path.dirname(filePath), { recursive: true });
};
const splitArgs = (args: string[]) => {
const tokens = [...args];
while (tokens[0] === "--") {
tokens.shift();
}
const leadingExtra: string[] = [];
while (tokens.length > 0 && tokens[0].startsWith("-")) {
const arg = tokens.shift();
if (!arg) break;
if (arg === "--") break;
leadingExtra.push(arg);
if (
!arg.includes("=") &&
tokens[0] &&
tokens[0] !== "--" &&
!tokens[0].startsWith("-")
) {
leadingExtra.push(tokens.shift() as string);
}
}
while (tokens[0] === "--") {
tokens.shift();
}
const separatorIndex = tokens.indexOf("--");
return {
paths: separatorIndex === -1 ? tokens : tokens.slice(0, separatorIndex),
extra: [
...leadingExtra,
...(separatorIndex === -1 ? [] : tokens.slice(separatorIndex + 1)),
],
};
};
const toSafeName = (name: string) => name.replace(/[\\/]/g, "-");
const collectFiles = (dir: string, suffix: string) => {
const results: string[] = [];
const walk = (current: string) => {
for (const entry of fs.readdirSync(current, { withFileTypes: true })) {
const full = `${current}/${entry.name}`;
if (entry.isDirectory()) {
walk(full);
} else if (entry.isFile() && entry.name.endsWith(suffix)) {
results.push(full);
}
}
};
if (fs.existsSync(dir)) walk(dir);
return results.sort();
};
const repoRoot = getRepoRootDir();
const writeCtrfFromJunit = (junitPath: string, tool: string) => {
if (!fs.existsSync(junitPath)) return;
const stat = fs.statSync(junitPath);
if (stat.size === 0) return;
const ctrfPath = junitPath.match(/\.xml$/i)
? junitPath.replace(/\.xml$/i, ".json")
: `${junitPath}.json`;
const result = spawnSync(
"pnpm",
["exec", "junit-to-ctrf", junitPath, "-o", ctrfPath, "-t", tool],
{ stdio: "inherit", cwd: repoRoot },
);
if (result.status !== 0) {
console.warn(`CTRF conversion failed for ${junitPath}.`);
}
};
const testsDir = `${repoRoot}/packages/server-v4/test`;
const unitTestsDir = `${testsDir}/unit`;
const integrationTestsDir = `${testsDir}/integration`;
const resolveRepoRelative = (value: string) =>
path.isAbsolute(value) ? value : path.resolve(repoRoot, value);
const stripNodeReporterArgs = (argsList: string[]) => {
const filtered: string[] = [];
let removed = false;
for (let i = 0; i < argsList.length; i++) {
const arg = argsList[i];
if (
arg === "--test-reporter" ||
arg.startsWith("--test-reporter=") ||
arg === "--test-reporter-destination" ||
arg.startsWith("--test-reporter-destination=")
) {
removed = true;
if (
(arg === "--test-reporter" || arg === "--test-reporter-destination") &&
argsList[i + 1]
) {
i += 1;
}
continue;
}
filtered.push(arg);
}
return { filtered, removed };
};
const toTestName = (testPath: string, root: string) => {
const abs = resolveRepoRelative(testPath);
const rel = path.relative(root, abs).replaceAll("\\", "/");
if (!rel.startsWith("..")) {
return rel.replace(/\.test\.(js|ts)$/i, "");
}
return path.basename(abs).replace(/\.test\.(js|ts)$/i, "");
};
const rawArgs = process.argv.slice(2);
const listRequested = rawArgs.includes("--list");
if (listRequested) {
const unitTests = collectFiles(unitTestsDir, ".test.ts").map((file) => {
const name = path.basename(file, ".test.ts");
return {
path: path.relative(repoRoot, file).replaceAll("\\", "/"),
name,
safe_name: toSafeName(name),
};
});
const integrationTests = collectFiles(integrationTestsDir, ".test.ts").map(
(file) => {
const rel = path
.relative(integrationTestsDir, file)
.replaceAll("\\", "/")
.replace(/\.test\.ts$/, "");
return {
path: path.relative(repoRoot, file).replaceAll("\\", "/"),
name: rel,
safe_name: toSafeName(rel),
};
},
);
console.log(JSON.stringify([...unitTests, ...integrationTests]));
process.exit(0);
}
const { paths, extra } = splitArgs(rawArgs);
const { filtered: extraArgs, removed: removedReporterOverride } =
stripNodeReporterArgs(extra);
if (removedReporterOverride) {
console.warn(
"Ignoring node --test reporter overrides to preserve console + JUnit output.",
);
}
if (!fs.existsSync(testsDir)) {
console.error(
"Missing packages/server-v4/test. Check the repository layout.",
);
process.exit(1);
}
const serverTarget = (
process.env.STAGEHAND_SERVER_TARGET ?? "sea"
).toLowerCase();
const explicitBaseUrl = process.env.STAGEHAND_BASE_URL;
const baseUrl = explicitBaseUrl ?? "http://stagehand-api.localhost:3107";
if (serverTarget === "remote" && !explicitBaseUrl) {
console.error("Missing STAGEHAND_BASE_URL for remote server target.");
process.exit(1);
}
if (
serverTarget === "local" &&
!fs.existsSync(`${repoRoot}/packages/server-v4/dist/server.js`)
) {
console.error(
"Missing packages/server-v4/dist/server.js. Run pnpm run build first.",
);
process.exit(1);
}
const parsedBaseUrl = new URL(baseUrl);
const port =
parsedBaseUrl.port || (parsedBaseUrl.protocol === "https:" ? "443" : "80");
process.env.PORT = port;
process.env.STAGEHAND_API_URL = baseUrl;
process.env.BB_ENV = process.env.BB_ENV ?? "local";
const baseNodeOptions = "--enable-source-maps";
const nodeOptions = [process.env.NODE_OPTIONS, baseNodeOptions]
.filter(Boolean)
.join(" ");
const allPaths =
paths.length > 0
? paths.map(resolveRepoRelative)
: [
...collectFiles(unitTestsDir, ".test.ts"),
...collectFiles(integrationTestsDir, ".test.ts"),
];
const unitPaths = allPaths.filter((p) =>
p.replaceAll("\\", "/").includes("/packages/server-v4/test/unit/"),
);
const integrationPaths = allPaths.filter((p) =>
p.replaceAll("\\", "/").includes("/packages/server-v4/test/integration/"),
);
const singlePath = allPaths.length === 1 ? allPaths[0] : null;
const coverageSuffix =
singlePath && singlePath.startsWith(unitTestsDir)
? `server-unit/${path.basename(singlePath).replace(/\.test\.ts$/, "")}`
: singlePath && singlePath.startsWith(integrationTestsDir)
? `server-integration/${path
.relative(integrationTestsDir, singlePath)
.replace(/\.test\.ts$/, "")
.replaceAll("\\", "/")}`
: "server";
const coverageRoot = resolveRepoRelative(
process.env.NODE_V8_COVERAGE ?? `${repoRoot}/coverage/${coverageSuffix}`,
);
const testsCoverage = `${coverageRoot}/tests`;
const serverCoverage = `${coverageRoot}/server`;
fs.mkdirSync(testsCoverage, { recursive: true });
fs.mkdirSync(serverCoverage, { recursive: true });
const consoleReporter = process.env.NODE_TEST_CONSOLE_REPORTER ?? "spec";
const defaultReporter = process.env.NODE_TEST_REPORTER ?? "junit";
const envDestination = process.env.NODE_TEST_REPORTER_DESTINATION
? resolveRepoRelative(process.env.NODE_TEST_REPORTER_DESTINATION)
: null;
const reporterArgsFor = (kind: "unit" | "integration", testName?: string) => {
const destination =
envDestination ??
`${repoRoot}/ctrf/${kind === "unit" ? "server-unit" : "server-integration"}/${testName ? `${testName}.xml` : "all.xml"}`;
ensureParentDir(destination);
return {
args: [
`--test-reporter=${consoleReporter}`,
`--test-reporter=${defaultReporter}`,
"--test-reporter-destination=stdout",
`--test-reporter-destination=${destination}`,
],
destination,
};
};
const runNodeTests = (files: string[], reporterArgs: string[]) =>
spawnSync(
process.execPath,
["--import=tsx", "--test", ...extraArgs, ...reporterArgs, ...files],
{
stdio: "inherit",
env: {
...process.env,
NODE_OPTIONS: nodeOptions,
NODE_V8_COVERAGE: testsCoverage,
},
},
);
const waitForServer = async (url: string, timeoutMs = 30_000) => {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
try {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 2_000);
const res = await fetch(url, { signal: controller.signal });
clearTimeout(timer);
if (res.ok) return true;
} catch {
// retry
}
await new Promise((resolve) => setTimeout(resolve, 1_000));
}
return false;
};
const startServer = async () => {
if (serverTarget === "remote") return null;
if (serverTarget === "local") {
return spawn(
process.execPath,
[`${repoRoot}/packages/server-v4/dist/server.js`],
{
stdio: "inherit",
env: {
...process.env,
NODE_ENV: "development",
NODE_OPTIONS: nodeOptions,
NODE_V8_COVERAGE: serverCoverage,
},
},
);
}
const defaultName = `stagehand-server-v4-${process.platform}-${process.arch}${process.platform === "win32" ? ".exe" : ""}`;
const seaBinary = `${repoRoot}/packages/server-v4/dist/sea/${process.env.SEA_BINARY_NAME ?? defaultName}`;
if (!fs.existsSync(seaBinary)) {
console.error(`SEA binary not found at ${seaBinary}`);
process.exit(1);
}
return spawn(seaBinary, ["--node-options=--no-lazy --enable-source-maps"], {
stdio: "inherit",
env: {
...process.env,
NODE_ENV: "production",
NODE_V8_COVERAGE: serverCoverage,
STAGEHAND_SEA_CACHE_DIR:
process.env.STAGEHAND_SEA_CACHE_DIR ?? `${repoRoot}/.stagehand-sea`,
},
});
};
let serverProc: ReturnType<typeof spawn> | null = null;
let status = 0;
if (unitPaths.length > 0) {
const unitName =
unitPaths.length === 1 ? toTestName(unitPaths[0], unitTestsDir) : undefined;
const reporter = reporterArgsFor("unit", unitName);
const result = runNodeTests(unitPaths, reporter.args);
status = result.status ?? 1;
writeCtrfFromJunit(reporter.destination, "node-test");
}
if (status === 0 && integrationPaths.length > 0) {
serverProc = await startServer();
const ready = await waitForServer(`${process.env.STAGEHAND_API_URL}/healthz`);
if (!ready) {
console.error("Server failed to start within 30 seconds.");
status = 1;
} else {
const integrationName =
integrationPaths.length === 1
? toTestName(integrationPaths[0], integrationTestsDir)
: undefined;
const reporter = reporterArgsFor("integration", integrationName);
const result = runNodeTests(integrationPaths, reporter.args);
status = result.status ?? 1;
writeCtrfFromJunit(reporter.destination, "node-test");
}
}
if (serverProc) {
serverProc.kill("SIGTERM");
await new Promise<void>((resolve) => {
if (serverProc?.exitCode !== null) return resolve();
const timer = setTimeout(resolve, 10_000);
serverProc?.once("exit", () => {
clearTimeout(timer);
resolve();
});
});
await new Promise((resolve) => setTimeout(resolve, 5_000));
}
process.exit(status);

View File

@ -1,143 +0,0 @@
import fastify from "fastify";
import path from "node:path";
import fastifySwagger from "@fastify/swagger";
import fastifySwaggerUI from "@fastify/swagger-ui";
import {
fastifyZodOpenApiPlugin,
fastifyZodOpenApiTransformers,
serializerCompiler,
validatorCompiler,
type FastifyZodOpenApiTypeProvider,
} from "fastify-zod-openapi";
import { StatusCodes } from "http-status-codes";
import { constants } from "./constants.js";
import { databasePlugin } from "./db/plugin.js";
import { env } from "./env.js";
import { llmModulePlugin } from "./plugins/llmModule.js";
import healthcheckRoute from "./routes/healthcheck.js";
import readinessRoute from "./routes/readiness.js";
import { browserSessionRoutesPlugin } from "./routes/v4/browsersession/routes.js";
import { llmRoutesPlugin } from "./routes/v4/llms/routes.js";
import { pageRoutesPlugin } from "./routes/v4/page/routes.js";
import { browserSessionOpenApiComponents } from "./schemas/v4/browserSession.js";
import { llmOpenApiComponents } from "./schemas/v4/llm.js";
import { pageOpenApiComponents } from "./schemas/v4/page.js";
export const buildApp = async () => {
const pgliteDataDir = path.resolve(
env.BROWSERBASE_CONFIG_DIR,
...constants.paths.pgliteDataDirSegments,
);
const app = fastify({
logger: true,
return503OnClosing: false,
});
// Allow requests with `Content-Type: application/json` and an empty body (0 bytes).
// Some clients always send the header even when there is no request body (e.g. /end).
const defaultJsonParser = app.getDefaultJsonParser("error", "error");
app.addContentTypeParser<string>(
"application/json",
{ parseAs: "string" },
(request, body, done) => {
if (body === "" || (Buffer.isBuffer(body) && body.length === 0)) {
done(null, {});
return;
}
void defaultJsonParser(request, body, done);
},
);
app.setValidatorCompiler(validatorCompiler);
app.setSerializerCompiler(serializerCompiler);
await app.register(databasePlugin, {
database:
env.STAGEHAND_DB_MODE === "postgres"
? {
mode: "postgres",
databaseUrl: env.DATABASE_URL,
}
: {
mode: "pglite",
dataDir: pgliteDataDir,
},
migrateOnStartup: env.STAGEHAND_DB_MODE === "pglite",
});
await app.register(llmModulePlugin);
await app.register(fastifyZodOpenApiPlugin, {
components: {
schemas: {
...browserSessionOpenApiComponents.schemas,
...llmOpenApiComponents.schemas,
...pageOpenApiComponents.schemas,
},
},
});
await app.register(fastifySwagger, {
openapi: {
info: {
title: "Stagehand API",
version: "3.0.5",
},
openapi: "3.1.0",
tags: [
{
name: "browserSession",
description: "Browser session lifecycle and browser-scoped actions",
},
{
name: "llm",
description: "Reusable llm configuration resources",
},
{
name: "page",
description: "Page-scoped actions and action history endpoints",
},
],
},
...fastifyZodOpenApiTransformers,
});
if (env.NODE_ENV === "development") {
await app.register(fastifySwaggerUI, {
routePrefix: "/documentation",
});
}
app.setErrorHandler((error, _request, reply) => {
const statusCode = (error as { validation?: unknown[] }).validation
? StatusCodes.BAD_REQUEST
: ((error as { statusCode?: number }).statusCode ??
StatusCodes.INTERNAL_SERVER_ERROR);
const errorMessage = (error as { validation?: unknown[] }).validation
? "Request validation failed"
: error instanceof Error
? error.message
: String(error);
reply.status(statusCode).send({
error:
statusCode === Number(StatusCodes.INTERNAL_SERVER_ERROR)
? "Internal Server Error"
: errorMessage,
statusCode,
});
});
const appWithTypes = app.withTypeProvider<FastifyZodOpenApiTypeProvider>();
await appWithTypes.register(browserSessionRoutesPlugin, { prefix: "/v4" });
await appWithTypes.register(llmRoutesPlugin, { prefix: "/v4" });
await appWithTypes.register(pageRoutesPlugin, { prefix: "/v4" });
appWithTypes.route(healthcheckRoute);
appWithTypes.route(readinessRoute);
return app;
};

View File

@ -1,39 +0,0 @@
/**
* Shared constants for server-v4 runtime configuration and path conventions.
* @constant
*/
export const constants = {
llm: {
/**
* The display name used when the server materializes a system default LLM.
* @constant
*/
defaultDisplayName: "Default LLM",
/**
* The default model name used for system-generated LLM configs.
* @constant
*/
defaultModelName: "openai/gpt-4.1-nano",
},
paths: {
/**
* The default local config directory name under the user's home directory.
* @constant
*/
defaultConfigDirName: ".stagehand",
/**
* The relative path segments used for the local persistent PGlite database.
* @constant
*/
pgliteDataDirSegments: ["db", "stagehand-v4"] as const,
},
urls: {
/**
* TODO: Replace this placeholder once the real v4 database URL has been provisioned.
* @constant
*/
defaultDatabaseUrl: "postgresql://example.com/stagehand_v4",
},
} as const;

View File

@ -1,71 +0,0 @@
import type { FastifyReply, FastifyRequest } from "fastify";
import { StatusCodes } from "http-status-codes";
import {
LLMListResponseSchema,
LLMResponseSchema,
type LLMCreateRequest,
type LLMIdParams,
type LLMUpdateRequest,
} from "../../schemas/v4/llm.js";
import type { LlmService } from "../../services/llm/llm.service.js";
export class LlmController {
constructor(private readonly llmService: LlmService) {}
async list(_request: FastifyRequest, reply: FastifyReply) {
const llms = await this.llmService.listLlms();
return reply.status(StatusCodes.OK).send(
LLMListResponseSchema.parse({
success: true,
data: {
llms,
},
}),
);
}
async create(request: FastifyRequest, reply: FastifyReply) {
const body = request.body as LLMCreateRequest;
const llm = await this.llmService.createLlm(body);
return reply.status(StatusCodes.OK).send(
LLMResponseSchema.parse({
success: true,
data: {
llm,
},
}),
);
}
async get(request: FastifyRequest, reply: FastifyReply) {
const { id } = request.params as LLMIdParams;
const llm = await this.llmService.getLlm(id);
return reply.status(StatusCodes.OK).send(
LLMResponseSchema.parse({
success: true,
data: {
llm,
},
}),
);
}
async update(request: FastifyRequest, reply: FastifyReply) {
const { id } = request.params as LLMIdParams;
const body = request.body as LLMUpdateRequest;
const llm = await this.llmService.updateLlm(id, body);
return reply.status(StatusCodes.OK).send(
LLMResponseSchema.parse({
success: true,
data: {
llm,
},
}),
);
}
}

View File

@ -1,123 +0,0 @@
import fs from "node:fs";
import { fileURLToPath } from "node:url";
import { PGlite } from "@electric-sql/pglite";
import {
type PgliteDatabase,
drizzle as drizzlePglite,
} from "drizzle-orm/pglite";
import { migrate as migratePglite } from "drizzle-orm/pglite/migrator";
import {
drizzle as drizzlePostgres,
type PostgresJsDatabase,
} from "drizzle-orm/postgres-js";
import { migrate as migratePostgres } from "drizzle-orm/postgres-js/migrator";
import postgres from "postgres";
import * as relations from "./schema/relations.js";
import * as schema from "./schema/index.js";
export type DatabaseSchema = typeof schema;
export type DatabaseRelations = typeof relations.relations;
export type PostgresDatabaseClient = PostgresJsDatabase<
DatabaseSchema,
DatabaseRelations
>;
export type PgliteDatabaseClient = PgliteDatabase<
DatabaseSchema,
DatabaseRelations
>;
export type DatabaseClient = PostgresDatabaseClient | PgliteDatabaseClient;
export type DatabaseDriver = ReturnType<typeof postgres> | PGlite;
export type PostgresDatabaseTransaction = Parameters<
Parameters<PostgresDatabaseClient["transaction"]>[0]
>[0];
export type PgliteDatabaseTransaction = Parameters<
Parameters<PgliteDatabaseClient["transaction"]>[0]
>[0];
export type DatabaseExecutor =
| DatabaseClient
| PostgresDatabaseTransaction
| PgliteDatabaseTransaction;
export type DatabaseRuntimeConfig =
| {
mode: "postgres";
databaseUrl: string;
}
| {
mode: "pglite";
dataDir: string;
};
interface BaseDatabaseConnection {
client: DatabaseDriver;
db: DatabaseClient;
mode: DatabaseRuntimeConfig["mode"];
migrate: () => Promise<void>;
ping: () => Promise<void>;
close: () => Promise<void>;
}
export type DatabaseConnection =
| (BaseDatabaseConnection & {
mode: "postgres";
client: ReturnType<typeof postgres>;
db: PostgresDatabaseClient;
})
| (BaseDatabaseConnection & {
mode: "pglite";
client: PGlite;
db: PgliteDatabaseClient;
});
const migrationsFolder = fileURLToPath(
new URL("../../drizzle", import.meta.url),
);
export const createDatabaseConnection = async (
config: DatabaseRuntimeConfig,
): Promise<DatabaseConnection> => {
if (config.mode === "postgres") {
// Supabase pooled Postgres connections do not support prepared statements.
const client = postgres(config.databaseUrl, { prepare: false });
const db = drizzlePostgres({
client,
schema,
relations: relations.relations,
});
return {
mode: "postgres",
client,
db,
migrate: async () => {
await migratePostgres(db, { migrationsFolder });
},
ping: async () => {
await db.execute("select 1");
},
close: async () => {
await client.end();
},
};
}
fs.mkdirSync(config.dataDir, { recursive: true });
const client = new PGlite(config.dataDir);
await client.waitReady;
const db = drizzlePglite({ client, schema, relations: relations.relations });
return {
mode: "pglite",
client,
db,
migrate: async () => {
await migratePglite(db, { migrationsFolder });
},
ping: async () => {
await db.execute("select 1");
},
close: async () => {
await client.close();
},
};
};

View File

@ -1,36 +0,0 @@
import fp from "fastify-plugin";
import {
type DatabaseRuntimeConfig,
createDatabaseConnection,
} from "./client.js";
export interface DatabasePluginOptions {
database: DatabaseRuntimeConfig;
migrateOnStartup?: boolean;
}
export const databasePlugin = fp<DatabasePluginOptions>(
async (app, { database, migrateOnStartup }) => {
app.decorate("db", null);
app.decorate("dbClient", null);
app.decorate("hasDatabase", false);
const connection = await createDatabaseConnection(database);
if (migrateOnStartup) {
await connection.migrate();
}
await connection.ping();
app.db = connection.db;
app.dbClient = connection.client;
app.hasDatabase = true;
app.addHook("onClose", async () => {
await connection.close();
});
},
{
name: "database-plugin",
},
);

View File

@ -1,14 +0,0 @@
export { llmCallsTable as llmCalls } from "./llmCalls.js";
export { llmConfigsTable as llmConfigs, llmSourceEnum } from "./llmConfigs.js";
export {
llmSessionStatusEnum,
llmSessionsTable as llmSessions,
} from "./llmSessions.js";
export {
stagehandBrowserSessionStatusEnum,
stagehandBrowserSessionsTable as stagehandBrowserSessions,
} from "./stagehandBrowserSessions.js";
export {
stagehandStepOperationEnum,
stagehandStepsTable as stagehandSteps,
} from "./stagehandSteps.js";

View File

@ -1,30 +0,0 @@
import { jsonb, pgTable, text, timestamp, uuid } from "drizzle-orm/pg-core";
import { llmSessionsTable } from "./llmSessions.js";
import type { DatabaseJsonValue } from "./types.js";
export const llmCallsTable = pgTable("llm_calls", {
id: uuid("id").defaultRandom().primaryKey(),
llmSessionId: uuid("llm_session_id")
.notNull()
.references(() => llmSessionsTable.id, {
onDelete: "cascade",
}),
sentAt: timestamp("sent_at", {
withTimezone: true,
mode: "string",
})
.notNull()
.defaultNow(),
receivedAt: timestamp("received_at", {
withTimezone: true,
mode: "string",
}),
prompt: text("prompt").notNull(),
expectedResponseSchema: jsonb(
"expected_response_schema",
).$type<DatabaseJsonValue | null>(),
response: jsonb("response").$type<DatabaseJsonValue | null>(),
error: jsonb("error").$type<DatabaseJsonValue | null>(),
usage: jsonb("usage").$type<DatabaseJsonValue | null>(),
model: text("model").notNull(),
});

View File

@ -1,35 +0,0 @@
import { sql } from "drizzle-orm";
import {
jsonb,
pgEnum,
pgTable,
text,
timestamp,
uuid,
} from "drizzle-orm/pg-core";
import type { LLMProviderOptions } from "./types.js";
export const llmSourceEnum = pgEnum("llm_source", ["user", "system-default"]);
export const llmConfigsTable = pgTable("llm_configs", {
id: uuid("id").defaultRandom().primaryKey(),
source: llmSourceEnum("source").notNull().default("user"),
displayName: text("display_name"),
modelName: text("model_name").notNull(),
baseUrl: text("base_url"),
systemPrompt: text("system_prompt"),
providerOptions: jsonb("provider_options").$type<LLMProviderOptions | null>(),
createdAt: timestamp("created_at", {
withTimezone: true,
mode: "string",
})
.notNull()
.defaultNow(),
updatedAt: timestamp("updated_at", {
withTimezone: true,
mode: "string",
})
.notNull()
.defaultNow()
.$onUpdateFn(() => sql`now()`),
});

View File

@ -1,91 +0,0 @@
import { sql } from "drizzle-orm";
import {
foreignKey,
integer,
jsonb,
pgEnum,
pgTable,
text,
timestamp,
uuid,
} from "drizzle-orm/pg-core";
import type { DatabaseJsonValue, ExtraHttpHeaders } from "./types.js";
export const llmSessionStatusEnum = pgEnum("llm_session_status", [
"disconnected",
"idle",
"thinking",
"permanent-error",
"ratelimited",
]);
export const llmSessionsTable = pgTable(
"llm_sessions",
{
id: uuid("id").defaultRandom().primaryKey(),
copiedTemplateId: uuid("copied_template_id"),
forkedSessionId: uuid("forked_session_id"),
projectId: uuid("project_id").notNull(),
// Nullable on the table so template sessions can exist independently of a browser session.
browserSessionId: uuid("browser_session_id"),
createdAt: timestamp("created_at", {
withTimezone: true,
mode: "string",
})
.notNull()
.defaultNow(),
updatedAt: timestamp("updated_at", {
withTimezone: true,
mode: "string",
})
.notNull()
.defaultNow()
.$onUpdateFn(() => sql`now()`),
connectedAt: timestamp("connected_at", {
withTimezone: true,
mode: "string",
}),
disconnectedAt: timestamp("disconnected_at", {
withTimezone: true,
mode: "string",
}),
lastRequestAt: timestamp("last_request_at", {
withTimezone: true,
mode: "string",
}),
lastResponseAt: timestamp("last_response_at", {
withTimezone: true,
mode: "string",
}),
lastErrorAt: timestamp("last_error_at", {
withTimezone: true,
mode: "string",
}),
lastErrorMessage: text("last_error_message"),
status: llmSessionStatusEnum("status").notNull(),
model: text("model").notNull(),
baseUrl: text("base_url"),
options: jsonb("options").$type<DatabaseJsonValue | null>(),
extraHttpHeaders: jsonb(
"extra_http_headers",
).$type<ExtraHttpHeaders | null>(),
systemPrompt: text("system_prompt"),
tokensInput: integer("tokens_input").notNull().default(0),
tokensOutput: integer("tokens_output").notNull().default(0),
tokensReasoning: integer("tokens_reasoning").notNull().default(0),
tokensCachedInput: integer("tokens_cached_input").notNull().default(0),
tokensTotal: integer("tokens_total").notNull().default(0),
},
(table) => [
foreignKey({
columns: [table.copiedTemplateId],
foreignColumns: [table.id],
name: "llm_sessions_copied_template_id_fkey",
}).onDelete("set null"),
foreignKey({
columns: [table.forkedSessionId],
foreignColumns: [table.id],
name: "llm_sessions_forked_session_id_fkey",
}).onDelete("set null"),
],
);

View File

@ -1,86 +0,0 @@
import { defineRelations } from "drizzle-orm";
import * as schema from "./index.js";
export const relations = defineRelations(schema, (r) => ({
llmConfigs: {},
llmSessions: {
copiedTemplate: r.one.llmSessions({
from: r.llmSessions.copiedTemplateId,
to: r.llmSessions.id,
alias: "copiedTemplate",
}),
forkedSession: r.one.llmSessions({
from: r.llmSessions.forkedSessionId,
to: r.llmSessions.id,
alias: "forkedSession",
}),
browserSession: r.one.stagehandBrowserSessions({
from: r.llmSessions.browserSessionId,
to: r.stagehandBrowserSessions.id,
}),
calls: r.many.llmCalls({
from: r.llmSessions.id,
to: r.llmCalls.llmSessionId,
}),
childCopiedSessions: r.many.llmSessions({
from: r.llmSessions.id,
to: r.llmSessions.copiedTemplateId,
alias: "copiedTemplate",
}),
childForkedSessions: r.many.llmSessions({
from: r.llmSessions.id,
to: r.llmSessions.forkedSessionId,
alias: "forkedSession",
}),
stagehandStepsAsTemplate: r.many.stagehandSteps({
from: r.llmSessions.id,
to: r.stagehandSteps.llmTemplateId,
alias: "llmTemplate",
}),
stagehandStepsAsSession: r.many.stagehandSteps({
from: r.llmSessions.id,
to: r.stagehandSteps.llmSessionId,
alias: "llmSession",
}),
defaultBrowserSessions: r.many.stagehandBrowserSessions({
from: r.llmSessions.id,
to: r.stagehandBrowserSessions.defaultLLMSessionId,
}),
},
llmCalls: {
llmSession: r.one.llmSessions({
from: r.llmCalls.llmSessionId,
to: r.llmSessions.id,
}),
},
stagehandBrowserSessions: {
defaultLLMSession: r.one.llmSessions({
from: r.stagehandBrowserSessions.defaultLLMSessionId,
to: r.llmSessions.id,
}),
llmSessions: r.many.llmSessions({
from: r.stagehandBrowserSessions.id,
to: r.llmSessions.browserSessionId,
}),
steps: r.many.stagehandSteps({
from: r.stagehandBrowserSessions.id,
to: r.stagehandSteps.stagehandBrowserSessionId,
}),
},
stagehandSteps: {
browserSession: r.one.stagehandBrowserSessions({
from: r.stagehandSteps.stagehandBrowserSessionId,
to: r.stagehandBrowserSessions.id,
}),
llmTemplate: r.one.llmSessions({
from: r.stagehandSteps.llmTemplateId,
to: r.llmSessions.id,
alias: "llmTemplate",
}),
llmSession: r.one.llmSessions({
from: r.stagehandSteps.llmSessionId,
to: r.llmSessions.id,
alias: "llmSession",
}),
},
}));

View File

@ -1,23 +0,0 @@
import { pgEnum, pgTable, text, uuid } from "drizzle-orm/pg-core";
import { llmSessionsTable } from "./llmSessions.js";
export const stagehandBrowserSessionStatusEnum = pgEnum(
"stagehand_browser_session_status",
["running", "terminated"],
);
export const stagehandBrowserSessionsTable = pgTable(
"stagehand_browser_sessions",
{
id: uuid("id").defaultRandom().primaryKey(),
projectId: uuid("project_id").notNull(),
browserbaseSessionId: uuid("browserbase_session_id"),
cdpUrl: text("cdp_url").notNull(),
status: stagehandBrowserSessionStatusEnum("status").notNull(),
defaultLLMSessionId: uuid("default_llm_session_id")
.notNull()
.references(() => llmSessionsTable.id, {
onDelete: "restrict",
}),
},
);

View File

@ -1,31 +0,0 @@
import { jsonb, pgEnum, pgTable, uuid } from "drizzle-orm/pg-core";
import { llmSessionsTable } from "./llmSessions.js";
import { stagehandBrowserSessionsTable } from "./stagehandBrowserSessions.js";
import type { DatabaseJsonValue } from "./types.js";
export const stagehandStepOperationEnum = pgEnum("stagehand_step_operation", [
"act",
"extract",
"observe",
"agent",
]);
export const stagehandStepsTable = pgTable("stagehand_steps", {
id: uuid("id").defaultRandom().primaryKey(),
stagehandBrowserSessionId: uuid("stagehand_browser_session_id")
.notNull()
.references(() => stagehandBrowserSessionsTable.id, {
onDelete: "cascade",
}),
operation: stagehandStepOperationEnum("operation").notNull(),
llmTemplateId: uuid("llm_template_id")
.notNull()
.references(() => llmSessionsTable.id, {
onDelete: "restrict",
}),
llmSessionId: uuid("llm_session_id").references(() => llmSessionsTable.id, {
onDelete: "set null",
}),
params: jsonb("params").$type<DatabaseJsonValue>().notNull(),
result: jsonb("result").$type<DatabaseJsonValue | null>(),
});

View File

@ -1,16 +0,0 @@
export type DatabaseJsonValue =
| string
| number
| boolean
| null
| { [key: string]: DatabaseJsonValue }
| DatabaseJsonValue[];
export interface LLMProviderOptions {
temperature?: number;
organization?: string;
project?: string;
location?: string;
}
export type ExtraHttpHeaders = Record<string, string>;

View File

@ -1,249 +0,0 @@
import { createSchemaFactory } from "drizzle-orm/zod";
import { z } from "zod/v4";
import { llmCallsTable } from "./llmCalls.js";
import { llmConfigsTable, llmSourceEnum } from "./llmConfigs.js";
import { llmSessionStatusEnum, llmSessionsTable } from "./llmSessions.js";
import {
stagehandBrowserSessionStatusEnum,
stagehandBrowserSessionsTable,
} from "./stagehandBrowserSessions.js";
import {
stagehandStepOperationEnum,
stagehandStepsTable,
} from "./stagehandSteps.js";
import type { DatabaseJsonValue, ExtraHttpHeaders } from "./types.js";
export const DatabaseJsonValueSchema: z.ZodType<DatabaseJsonValue> = z.lazy(
() =>
z.union([
z.string(),
z.number(),
z.boolean(),
z.null(),
z.array(DatabaseJsonValueSchema),
z.record(z.string(), DatabaseJsonValueSchema),
]),
);
export const DatabaseTimestampSchema = z
.string()
.refine((value) => !Number.isNaN(Date.parse(value)), {
message: "Invalid timestamp",
});
export const LLMProviderOptionsSchema = z
.object({
temperature: z.number().optional(),
organization: z.string().optional(),
project: z.string().optional(),
location: z.string().optional(),
})
.strict();
export const ExtraHttpHeadersSchema: z.ZodType<ExtraHttpHeaders> = z.record(
z.string(),
z.string(),
);
const { createInsertSchema, createSelectSchema, createUpdateSchema } =
createSchemaFactory({
zodInstance: z,
});
export const llmSourceSchema = createSelectSchema(llmSourceEnum);
export const llmSessionStatusSchema = createSelectSchema(llmSessionStatusEnum);
export const stagehandBrowserSessionStatusSchema = createSelectSchema(
stagehandBrowserSessionStatusEnum,
);
export const stagehandStepOperationSchema = createSelectSchema(
stagehandStepOperationEnum,
);
export const llmConfigSelectSchema = createSelectSchema(llmConfigsTable, {
id: z.uuid(),
baseUrl: z.url().nullable(),
providerOptions: LLMProviderOptionsSchema.nullable(),
createdAt: DatabaseTimestampSchema,
updatedAt: DatabaseTimestampSchema,
}).strict();
export const llmConfigInsertSchema = createInsertSchema(llmConfigsTable, {
id: z.uuid().optional(),
source: llmSourceSchema.optional(),
baseUrl: z.url().nullable().optional(),
providerOptions: LLMProviderOptionsSchema.nullable().optional(),
createdAt: DatabaseTimestampSchema.optional(),
updatedAt: DatabaseTimestampSchema.optional(),
}).strict();
export const llmConfigUpdateSchema = createUpdateSchema(llmConfigsTable, {
id: z.uuid().optional(),
source: llmSourceSchema.optional(),
baseUrl: z.url().nullable().optional(),
providerOptions: LLMProviderOptionsSchema.nullable().optional(),
createdAt: DatabaseTimestampSchema.optional(),
updatedAt: DatabaseTimestampSchema.optional(),
}).strict();
export const llmSessionSelectSchema = createSelectSchema(llmSessionsTable, {
id: z.uuid(),
copiedTemplateId: z.uuid().nullable(),
forkedSessionId: z.uuid().nullable(),
projectId: z.uuid(),
browserSessionId: z.uuid().nullable(),
createdAt: DatabaseTimestampSchema,
updatedAt: DatabaseTimestampSchema,
connectedAt: DatabaseTimestampSchema.nullable(),
disconnectedAt: DatabaseTimestampSchema.nullable(),
lastRequestAt: DatabaseTimestampSchema.nullable(),
lastResponseAt: DatabaseTimestampSchema.nullable(),
lastErrorAt: DatabaseTimestampSchema.nullable(),
options: DatabaseJsonValueSchema.nullable(),
extraHttpHeaders: ExtraHttpHeadersSchema.nullable(),
baseUrl: z.url().nullable(),
}).strict();
export const llmSessionInsertSchema = createInsertSchema(llmSessionsTable, {
id: z.uuid().optional(),
copiedTemplateId: z.uuid().nullable().optional(),
forkedSessionId: z.uuid().nullable().optional(),
projectId: z.uuid(),
browserSessionId: z.uuid().nullable().optional(),
createdAt: DatabaseTimestampSchema.optional(),
updatedAt: DatabaseTimestampSchema.optional(),
connectedAt: DatabaseTimestampSchema.nullable().optional(),
disconnectedAt: DatabaseTimestampSchema.nullable().optional(),
lastRequestAt: DatabaseTimestampSchema.nullable().optional(),
lastResponseAt: DatabaseTimestampSchema.nullable().optional(),
lastErrorAt: DatabaseTimestampSchema.nullable().optional(),
options: DatabaseJsonValueSchema.nullable().optional(),
extraHttpHeaders: ExtraHttpHeadersSchema.nullable().optional(),
baseUrl: z.url().nullable().optional(),
}).strict();
export const llmSessionUpdateSchema = createUpdateSchema(llmSessionsTable, {
id: z.uuid().optional(),
copiedTemplateId: z.uuid().nullable().optional(),
forkedSessionId: z.uuid().nullable().optional(),
projectId: z.uuid().optional(),
browserSessionId: z.uuid().nullable().optional(),
createdAt: DatabaseTimestampSchema.optional(),
updatedAt: DatabaseTimestampSchema.optional(),
connectedAt: DatabaseTimestampSchema.nullable().optional(),
disconnectedAt: DatabaseTimestampSchema.nullable().optional(),
lastRequestAt: DatabaseTimestampSchema.nullable().optional(),
lastResponseAt: DatabaseTimestampSchema.nullable().optional(),
lastErrorAt: DatabaseTimestampSchema.nullable().optional(),
options: DatabaseJsonValueSchema.nullable().optional(),
extraHttpHeaders: ExtraHttpHeadersSchema.nullable().optional(),
baseUrl: z.url().nullable().optional(),
}).strict();
export const llmCallSelectSchema = createSelectSchema(llmCallsTable, {
id: z.uuid(),
llmSessionId: z.uuid(),
sentAt: DatabaseTimestampSchema,
receivedAt: DatabaseTimestampSchema.nullable(),
expectedResponseSchema: DatabaseJsonValueSchema.nullable(),
response: DatabaseJsonValueSchema.nullable(),
error: DatabaseJsonValueSchema.nullable(),
usage: DatabaseJsonValueSchema.nullable(),
}).strict();
export const llmCallInsertSchema = createInsertSchema(llmCallsTable, {
id: z.uuid().optional(),
llmSessionId: z.uuid(),
sentAt: DatabaseTimestampSchema.optional(),
receivedAt: DatabaseTimestampSchema.nullable().optional(),
expectedResponseSchema: DatabaseJsonValueSchema.nullable().optional(),
response: DatabaseJsonValueSchema.nullable().optional(),
error: DatabaseJsonValueSchema.nullable().optional(),
usage: DatabaseJsonValueSchema.nullable().optional(),
}).strict();
export const llmCallUpdateSchema = createUpdateSchema(llmCallsTable, {
id: z.uuid().optional(),
llmSessionId: z.uuid().optional(),
sentAt: DatabaseTimestampSchema.optional(),
receivedAt: DatabaseTimestampSchema.nullable().optional(),
expectedResponseSchema: DatabaseJsonValueSchema.nullable().optional(),
response: DatabaseJsonValueSchema.nullable().optional(),
error: DatabaseJsonValueSchema.nullable().optional(),
usage: DatabaseJsonValueSchema.nullable().optional(),
}).strict();
export const stagehandBrowserSessionSelectSchema = createSelectSchema(
stagehandBrowserSessionsTable,
{
id: z.uuid(),
projectId: z.uuid(),
browserbaseSessionId: z.uuid().nullable(),
defaultLLMSessionId: z.uuid(),
},
).strict();
export const stagehandBrowserSessionInsertSchema = createInsertSchema(
stagehandBrowserSessionsTable,
{
id: z.uuid().optional(),
projectId: z.uuid(),
browserbaseSessionId: z.uuid().nullable().optional(),
defaultLLMSessionId: z.uuid(),
},
).strict();
export const stagehandBrowserSessionUpdateSchema = createUpdateSchema(
stagehandBrowserSessionsTable,
{
id: z.uuid().optional(),
projectId: z.uuid().optional(),
browserbaseSessionId: z.uuid().nullable().optional(),
defaultLLMSessionId: z.uuid().optional(),
},
).strict();
export const stagehandStepSelectSchema = createSelectSchema(
stagehandStepsTable,
{
id: z.uuid(),
stagehandBrowserSessionId: z.uuid(),
llmTemplateId: z.uuid(),
llmSessionId: z.uuid().nullable(),
params: DatabaseJsonValueSchema,
result: DatabaseJsonValueSchema.nullable(),
},
).strict();
export const stagehandStepInsertSchema = createInsertSchema(
stagehandStepsTable,
{
id: z.uuid().optional(),
stagehandBrowserSessionId: z.uuid(),
llmTemplateId: z.uuid(),
llmSessionId: z.uuid().nullable().optional(),
params: DatabaseJsonValueSchema,
result: DatabaseJsonValueSchema.nullable().optional(),
},
).strict();
export const stagehandStepUpdateSchema = createUpdateSchema(
stagehandStepsTable,
{
id: z.uuid().optional(),
stagehandBrowserSessionId: z.uuid().optional(),
llmTemplateId: z.uuid().optional(),
llmSessionId: z.uuid().nullable().optional(),
params: DatabaseJsonValueSchema.optional(),
result: DatabaseJsonValueSchema.nullable().optional(),
},
).strict();
export type LLMConfigSelect = z.infer<typeof llmConfigSelectSchema>;
export type LLMConfigInsert = z.infer<typeof llmConfigInsertSchema>;
export type LLMConfigUpdate = z.infer<typeof llmConfigUpdateSchema>;
export type LLMSessionSelect = z.infer<typeof llmSessionSelectSchema>;
export type LLMCallSelect = z.infer<typeof llmCallSelectSchema>;
export type StagehandBrowserSessionSelect = z.infer<
typeof stagehandBrowserSessionSelectSchema
>;
export type StagehandStepSelect = z.infer<typeof stagehandStepSelectSchema>;

View File

@ -1,51 +0,0 @@
import os from "node:os";
import path from "node:path";
import { createEnv } from "@t3-oss/env-core";
import { z } from "zod";
import { constants } from "./constants.js";
export type DatabaseMode = "postgres" | "pglite";
const defaultConfigDir = path.resolve(
os.homedir(),
constants.paths.defaultConfigDirName,
);
export const parseEnvironment = (runtimeEnv: NodeJS.ProcessEnv) => {
const parsedEnv = createEnv({
clientPrefix: "",
client: {},
server: {
BROWSERBASE_CONFIG_DIR: z
.string()
.min(1)
.transform((dir) => path.resolve(dir))
.default(defaultConfigDir),
NODE_ENV: z
.enum(["development", "test", "production"])
.default("development"),
PORT: z.coerce.number().int().positive().default(3000),
STAGEHAND_DB_MODE: z.enum(["postgres", "pglite"]).default("pglite"),
DATABASE_URL: z.url().default(constants.urls.defaultDatabaseUrl),
},
runtimeEnvStrict: {
BROWSERBASE_CONFIG_DIR: runtimeEnv.BROWSERBASE_CONFIG_DIR,
NODE_ENV: runtimeEnv.NODE_ENV,
PORT: runtimeEnv.PORT,
STAGEHAND_DB_MODE: runtimeEnv.STAGEHAND_DB_MODE,
DATABASE_URL: runtimeEnv.DATABASE_URL,
},
emptyStringAsUndefined: true,
});
if (
parsedEnv.STAGEHAND_DB_MODE === "postgres" &&
parsedEnv.DATABASE_URL === constants.urls.defaultDatabaseUrl
) {
throw new Error("DATABASE_URL must be set when STAGEHAND_DB_MODE=postgres");
}
return parsedEnv;
};
export const env = parseEnvironment(process.env);

View File

@ -1,30 +0,0 @@
import fp from "fastify-plugin";
import { LlmController } from "../controllers/v4/llm.controller.js";
import { LlmConfigRepository } from "../repositories/llm/llmConfig.repository.js";
import { LlmService } from "../services/llm/llm.service.js";
export const llmModulePlugin = fp(
async (app) => {
if (!app.db) {
throw new Error(
"Database plugin must be registered before llmModulePlugin",
);
}
const llmConfigRepository = new LlmConfigRepository(app.db);
const llmService = new LlmService({
db: app.db,
llmConfigRepository,
});
const llmController = new LlmController(llmService);
app.decorate("llmConfigRepository", llmConfigRepository);
app.decorate("llmService", llmService);
app.decorate("llmController", llmController);
},
{
name: "llm-module-plugin",
dependencies: ["database-plugin"],
},
);

View File

@ -1,48 +0,0 @@
import { desc, eq } from "drizzle-orm";
import type { DatabaseExecutor } from "../../db/client.js";
import { llmConfigs } from "../../db/schema/index.js";
import type { LLMConfigSelect } from "../../db/schema/zod.js";
type LlmConfigInsertRow = typeof llmConfigs.$inferInsert;
type LlmConfigUpdateRow = Partial<LlmConfigInsertRow>;
export class LlmConfigRepository {
constructor(private readonly db: DatabaseExecutor) {}
async list(): Promise<LLMConfigSelect[]> {
return this.db
.select()
.from(llmConfigs)
.orderBy(desc(llmConfigs.createdAt), desc(llmConfigs.id));
}
async getById(id: string): Promise<LLMConfigSelect | null> {
const [llm] = await this.db
.select()
.from(llmConfigs)
.where(eq(llmConfigs.id, id))
.limit(1);
return llm ?? null;
}
async create(values: LlmConfigInsertRow): Promise<LLMConfigSelect> {
const [llm] = await this.db.insert(llmConfigs).values(values).returning();
return llm;
}
async updateById(
id: string,
values: LlmConfigUpdateRow,
): Promise<LLMConfigSelect | null> {
const [llm] = await this.db
.update(llmConfigs)
.set(values)
.where(eq(llmConfigs.id, id))
.returning();
return llm ?? null;
}
}

View File

@ -1,26 +0,0 @@
import type { RouteOptions } from "fastify";
import { z } from "zod/v4";
import type { FastifyZodOpenApiSchema } from "fastify-zod-openapi";
const healthcheckRoute: RouteOptions = {
method: "GET",
url: "/healthz",
logLevel: "silent",
schema: {
hide: true, // Hide from OpenAPI spec - utility endpoint
response: {
200: z
.object({
status: z.string(),
timestamp: z.string(),
})
.strict(),
},
} satisfies FastifyZodOpenApiSchema,
handler: async () => ({
status: "ok",
timestamp: new Date().toISOString(),
}),
};
export default healthcheckRoute;

View File

@ -1,53 +0,0 @@
import type { RouteOptions } from "fastify";
import { StatusCodes } from "http-status-codes";
import { z } from "zod/v4";
import type { FastifyZodOpenApiSchema } from "fastify-zod-openapi";
// Server readiness state management
let isReady = false;
/**
* Get the current readiness state of the server
* @returns {boolean} Whether the server is ready to accept requests
*/
export const getIsReady = (): boolean => {
return isReady;
};
/**
* Mark the server as ready to accept requests
*/
export const setReady = (): void => {
isReady = true;
};
/**
* Mark the server as not ready to accept requests
* Used during graceful shutdown to stop accepting new requests
*/
export const setUnready = (): void => {
isReady = false;
};
const readinessRoute: RouteOptions = {
method: "GET",
url: "/readyz",
logLevel: "silent",
schema: {
hide: true, // Hide from OpenAPI spec - utility endpoint
response: {
200: z.string(),
503: z.string(),
},
} satisfies FastifyZodOpenApiSchema,
handler: async (_request, reply) => {
if (!isReady) {
return reply
.code(StatusCodes.SERVICE_UNAVAILABLE)
.send("Service Unavailable");
}
return reply.code(StatusCodes.OK).send("Ready");
},
};
export default readinessRoute;

View File

@ -1,46 +0,0 @@
import type { RouteHandlerMethod, RouteOptions } from "fastify";
import { StatusCodes } from "http-status-codes";
import type { FastifyZodOpenApiSchema } from "fastify-zod-openapi";
import {
BrowserSessionEndRequestSchema,
BrowserSessionHeadersSchema,
BrowserSessionIdParamsSchema,
BrowserSessionResponseSchema,
BrowserSessionV4ErrorResponseSchema,
type BrowserSessionIdParams,
} from "../../../../schemas/v4/browserSession.js";
import { endBrowserSession } from "../../stubState.js";
const endBrowserSessionHandler: RouteHandlerMethod = async (request, reply) => {
const { id } = request.params as BrowserSessionIdParams;
const browserSession = endBrowserSession(id);
return reply.status(StatusCodes.OK).send(
BrowserSessionResponseSchema.parse({
success: true,
data: {
browserSession,
},
}),
);
};
const endBrowserSessionRoute: RouteOptions = {
method: "POST",
url: "/browsersession/:id/end",
schema: {
operationId: "BrowserSessionEnd",
summary: "End a browser session",
headers: BrowserSessionHeadersSchema,
params: BrowserSessionIdParamsSchema,
body: BrowserSessionEndRequestSchema,
response: {
200: BrowserSessionResponseSchema,
404: BrowserSessionV4ErrorResponseSchema,
},
} satisfies FastifyZodOpenApiSchema,
handler: endBrowserSessionHandler,
};
export default endBrowserSessionRoute;

View File

@ -1,44 +0,0 @@
import type { RouteHandlerMethod, RouteOptions } from "fastify";
import { StatusCodes } from "http-status-codes";
import type { FastifyZodOpenApiSchema } from "fastify-zod-openapi";
import {
BrowserSessionHeadersSchema,
BrowserSessionIdParamsSchema,
BrowserSessionResponseSchema,
BrowserSessionV4ErrorResponseSchema,
type BrowserSessionIdParams,
} from "../../../../schemas/v4/browserSession.js";
import { getBrowserSession } from "../../stubState.js";
const getBrowserSessionHandler: RouteHandlerMethod = async (request, reply) => {
const { id } = request.params as BrowserSessionIdParams;
const browserSession = getBrowserSession(id);
return reply.status(StatusCodes.OK).send(
BrowserSessionResponseSchema.parse({
success: true,
data: {
browserSession,
},
}),
);
};
const getBrowserSessionRoute: RouteOptions = {
method: "GET",
url: "/browsersession/:id",
schema: {
operationId: "BrowserSessionStatus",
summary: "Get browser session status",
headers: BrowserSessionHeadersSchema,
params: BrowserSessionIdParamsSchema,
response: {
200: BrowserSessionResponseSchema,
404: BrowserSessionV4ErrorResponseSchema,
},
} satisfies FastifyZodOpenApiSchema,
handler: getBrowserSessionHandler,
};
export default getBrowserSessionRoute;

View File

@ -1,51 +0,0 @@
import type { RouteHandlerMethod, RouteOptions } from "fastify";
import { StatusCodes } from "http-status-codes";
import type { FastifyZodOpenApiSchema } from "fastify-zod-openapi";
import {
BrowserSessionHeadersSchema,
BrowserSessionIdParamsSchema,
BrowserSessionResponseSchema,
BrowserSessionUpdateRequestSchema,
BrowserSessionV4ErrorResponseSchema,
type BrowserSessionIdParams,
type BrowserSessionUpdateRequest,
} from "../../../../schemas/v4/browserSession.js";
import { updateBrowserSession } from "../../stubState.js";
const updateBrowserSessionHandler: RouteHandlerMethod = async (
request,
reply,
) => {
const { id } = request.params as BrowserSessionIdParams;
const body = request.body as BrowserSessionUpdateRequest;
const browserSession = updateBrowserSession(id, body);
return reply.status(StatusCodes.OK).send(
BrowserSessionResponseSchema.parse({
success: true,
data: {
browserSession,
},
}),
);
};
const updateBrowserSessionRoute: RouteOptions = {
method: "PATCH",
url: "/browsersession/:id",
schema: {
operationId: "BrowserSessionUpdate",
summary: "Update a browser session",
headers: BrowserSessionHeadersSchema,
params: BrowserSessionIdParamsSchema,
body: BrowserSessionUpdateRequestSchema,
response: {
200: BrowserSessionResponseSchema,
404: BrowserSessionV4ErrorResponseSchema,
},
} satisfies FastifyZodOpenApiSchema,
handler: updateBrowserSessionHandler,
};
export default updateBrowserSessionRoute;

Some files were not shown because too many files have changed in this diff Show More