Compare commits

..

7 Commits

Author SHA1 Message Date
Kyle Jeong
1db5f1c193
feat: Add Claude Opus 4.8 CUA support (#2170)
## Summary
- Add `anthropic/claude-opus-4-8` to the CUA model whitelist.
- Route bare `claude-opus-4-8` model names to the Anthropic CUA client.
- Treat Claude Opus 4.8 as a Claude 4.6+ adaptive-thinking CUA model and
document it in the supported CUA model lists.

## Validation
- `pnpm --filter @browserbasehq/stagehand run typecheck`
- `pnpm run test:core --
packages/core/dist/esm/tests/unit/public-api/llm-and-agents.test.js
packages/core/dist/esm/tests/unit/anthropic-cua-adaptive-thinking.test.js`

Note: the focused test command passed 38 tests, then the test wrapper
printed a non-fatal CTRF conversion warning from `glob/minimatch` under
Node 22.
2026-05-28 11:57:04 -07:00
Sam F
ce21468147
STG-1746 Add SEA --version flag (#2167)
# why

The SEA binary needs a lightweight way to report the commit and
timestamp it was built from.

# what changed

Added a `--version` path that prints build metadata baked into the SEA
binary during build.
2026-05-28 19:01:37 +02:00
Sam F
49575d62f5
[STG-1756] forward Vertex model config (#2160)
## Why
Vertex model credentials need a typed public shape and must be forwarded
from the TypeScript client to API-backed Stagehand requests without
persisting secrets server-side.

## What changed
- Define a canonical Vertex model config shape using `provider:
"vertex"`, `auth`, and `providerOptions.vertex`, while preserving
existing generic model string/object support.
- Forward constructor model config through API-backed
act/observe/extract/agentExecute requests, with per-call model config
still taking precedence.
- Keep navigate/goto from exposing a public per-call model option, while
still allowing constructor config to bootstrap API-backed sessions
internally.
- Sync server-v3 request parsing with the hosted API approach: Zod-first
safe parsing, separate request model config from Stagehand init model
config, and clean SSE validation errors.
- Regenerate server-v3 OpenAPI from the Stagehand Zod source so
Stainless can pick up the canonical shape.

TODO before final hosted API merge: swap core to the Orca build produced
from this Stagehand change.

---------

Co-authored-by: samfinton <samfinton@samLaptopm5.local>
2026-05-27 21:41:59 +02:00
Sean McGuire
bf9d44989c
[chore]: bump ws dep (#2169)
# why
- bumps `ws` & a few other deps to versions with various security
patches
# what changed
- bumps direct `ws` dep from `^8.18.0` to `^8.21.0`
- bumps turbo from `^2.8.10` to `^2.9.15`
- adds monorepo-wide overrides for `qs` & `ws` which are transitive deps
brought in by mintlify & braintrust
- overrides were required here since a direct bump of braintrust &
mintlify did not resolve to upgraded `qs` & `ws` versions


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Upgrade `ws` to patched versions across the repo and add overrides for
transitive `ws` and `qs` to address security fixes. Also bump `turbo` to
the latest minor.

- **Dependencies**
  - `packages/cli` and `packages/core`: `ws` ^8.18.0 -> ^8.21.0
- Root overrides: `ws` ^8.20.1, `qs` ^6.15.2 (to override transitive
deps from `mintlify` and `braintrust`)
  - `turbo` ^2.8.10 -> ^2.9.15

<sup>Written for commit e8f6ff1f863aac475f754f7d80e83db0eea4ad09.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2169?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-05-27 11:50:35 -07:00
Sean McGuire
aff5528c8d
[chore]: update lockfile (#2168)
# why
- recent lockfile change was left out of a previous commit
# what changed
- ran pnpm i & added the lockfile


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Regenerated `pnpm-lock.yaml` that was missed in a prior commit to align
with current workspace dependencies. This removes stale entries and
ensures reproducible installs in CI and local dev.

<sup>Written for commit c0e79fec55c32dedb245976a3117d3f861f745d5.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2168?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-05-27 10:50:50 -07:00
Sam F
947e6e4249
Fix SDK API reference parameter labels (#2164)
## Summary
- add a Mintlify custom script that rewrites generated SDK API reference
schema labels from OpenAPI camelCase to SDK snake_case on Python and
Ruby pages
- keep code/pre/input/script/style content untouched so generated code
samples stay language-correct
- leave Java/Go/TypeScript routes unchanged for now

## Why
Mintlify renders these language-scoped API reference pages from the
Stainless documented OpenAPI spec, so visible schema labels use
wire-format camelCase even when the Python and Ruby SDK snippets use
snake_case.

## Validation
- `pnpm --dir packages/docs exec prettier --check
sdk-api-reference-labels.js`
- `node --check packages/docs/sdk-api-reference-labels.js`
- confirmed Mintlify embeds root-level docs JS by checking the existing
custom script in local/prod HTML and the new script in local HTML
- browser fixture served at `/v3/api-reference/{python,ruby,java}/...`:
Python/Ruby labels rewrote to snake_case, code blocks stayed camelCase,
and Java remained unchanged

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes mismatched parameter labels on Python and Ruby SDK API reference
pages by rewriting OpenAPI camelCase labels to SDK snake_case. Code
samples and inputs remain unchanged; other languages are unaffected.

- **Bug Fixes**
- Added `packages/docs/sdk-api-reference-labels.js` to relabel visible
schema fields for Python and Ruby only.
- Skips `code`, `pre`, `input`, `script`, `style`, and editable elements
to preserve language-correct snippets.
- Reacts to SPA route changes and DOM mutations to keep labels
consistent.
  - Leaves Java/Go/TypeScript pages unchanged.

<sup>Written for commit ea143a674cc1a8b55cf39351506e93ecf0d70016.
Summary will update on new commits. <a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2164?utm_source=github">Review
in cubic</a></sup>

<!-- End of auto-generated description by cubic. -->
2026-05-27 10:23:50 +02:00
Miguel
e102a89c90
feat(verifier): record agent trajectories (#2131)
# Why

The new verifier needs richer evidence than a final screenshot,
especially for DOM and Hybrid agent modes where the important facts
often live in tool returns, ARIA snapshots, and per-step observations.
This PR adds trajectory recording without changing the verifier judgment
engine.

# What Changed

- Added typed agent events for screenshot, step-finished, step-observed,
and final-answer events.
- Added listener-gated post-step probes for screenshots and ARIA trees.
- Attached the settled post-turn probe to every tool call in a
DOM/Hybrid turn.
- Added CUA step evidence pairing and final answer capture.
- Added `TrajectoryRecorder` persistence and a smoke script for
trajectory shape and disk layout.

# Tests

- `pnpm --filter @browserbasehq/stagehand run typecheck`
- `pnpm --filter @browserbasehq/stagehand-evals run typecheck`
- `node --import tsx
packages/evals/scripts/verify-trajectory-recorder.ts`
- `git diff --check`

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 14:16:16 -07:00
50 changed files with 2065 additions and 2058 deletions

View File

@ -0,0 +1,5 @@
---
"@browserbasehq/stagehand": patch
---
Add Claude Opus 4.8 to the supported CUA model whitelist.

View File

@ -0,0 +1,5 @@
---
"@browserbasehq/stagehand-server-v3": patch
---
Add SEA binary --version build metadata output.

View File

@ -0,0 +1,6 @@
---
"@browserbasehq/stagehand": minor
"@browserbasehq/stagehand-server-v3": minor
---
Forward constructor and request model configuration when initializing API-backed sessions.

View File

@ -56,7 +56,7 @@
"prettier": "^3.2.5",
"source-map": "^0.7.4",
"tsx": "^4.19.4",
"turbo": "^2.8.10",
"turbo": "^2.9.15",
"typescript": "5.8.3",
"typescript-eslint": "^8.56.1"
},
@ -78,7 +78,9 @@
"jws": "^4.0.1",
"lodash": "^4.18.0",
"minimatch": "^3.1.4",
"uuid": "^11.1.1"
"uuid": "^11.1.1",
"ws": "^8.20.1",
"qs": "^6.15.2"
}
},
"lint-staged": {

View File

@ -51,7 +51,7 @@
"node-html-markdown": "^1.3.0",
"pino": "^9.6.0",
"pino-pretty": "^13.0.0",
"ws": "^8.18.0"
"ws": "^8.21.0"
},
"devDependencies": {
"@types/node": "^20.11.30",

View File

@ -23,6 +23,7 @@ export const modelToAgentProviderMap: Record<string, AgentProviderType> = {
"claude-sonnet-4-5-20250929": "anthropic",
"claude-opus-4-5-20251101": "anthropic",
"claude-opus-4-6": "anthropic",
"claude-opus-4-8": "anthropic",
"claude-sonnet-4-6": "anthropic",
"claude-haiku-4-5": "anthropic",
"claude-haiku-4-5-20251001": "anthropic",

View File

@ -453,6 +453,7 @@ export class AnthropicCUAClient extends AgentClient {
// Check if this is a Claude 4.6+ model that supports adaptive thinking
const isAdaptiveThinkingModel = [
"claude-opus-4-8",
"claude-opus-4-6",
"claude-sonnet-4-6",
].includes(modelBase);

View File

@ -1,55 +0,0 @@
import type { AgentScreenshotEvidenceEvent } from "../../types/public/agentEvidenceEvents.js";
export interface PairedCuaActionStep {
stepIndex: number;
replayScreenshot?: AgentScreenshotEvidenceEvent;
}
export class CuaEvidenceStepTracker {
private nextStepIndex = 0;
private latestScreenshot?: AgentScreenshotEvidenceEvent;
private latestScreenshotConsumed = true;
reset(): void {
this.nextStepIndex = 0;
this.latestScreenshot = undefined;
this.latestScreenshotConsumed = true;
}
recordScreenshot(
screenshot: Buffer,
url: string,
): AgentScreenshotEvidenceEvent {
const event: AgentScreenshotEvidenceEvent = {
type: "screenshot",
stepIndex: this.nextStepIndex++,
screenshot,
url,
evidenceRole: "agent",
};
this.latestScreenshot = event;
this.latestScreenshotConsumed = false;
return event;
}
pairAction(): PairedCuaActionStep {
if (this.latestScreenshot && !this.latestScreenshotConsumed) {
this.latestScreenshotConsumed = true;
return { stepIndex: this.latestScreenshot.stepIndex };
}
const stepIndex = this.nextStepIndex++;
if (this.latestScreenshot) {
return {
stepIndex,
replayScreenshot: { ...this.latestScreenshot, stepIndex },
};
}
return { stepIndex };
}
get latestScreenshotUrl(): string | undefined {
return this.latestScreenshot?.url;
}
}

View File

@ -11,7 +11,6 @@ interface CaptureProbeEvidenceOptions {
}
interface EmitPostStepProbeEvidenceOptions extends CaptureProbeEvidenceOptions {
stepIndices: number | number[];
evidenceCallback?: AgentEvidenceCallback;
}
@ -53,7 +52,6 @@ export async function captureProbeEvidence({
export async function emitPostStepProbeEvidence({
v3,
stepIndices,
url,
evidenceCallback,
logger,
@ -61,30 +59,23 @@ export async function emitPostStepProbeEvidence({
}: EmitPostStepProbeEvidenceOptions): Promise<void> {
if (!evidenceCallback) return;
const indices = Array.isArray(stepIndices) ? stepIndices : [stepIndices];
if (indices.length === 0) return;
const probe = await captureProbeEvidence({
v3,
url,
logger,
warningMessage,
});
for (const stepIndex of indices) {
if (probe.screenshot) {
await evidenceCallback({
type: "screenshot",
stepIndex,
screenshot: probe.screenshot,
url: probe.url,
evidenceRole: "probe",
});
}
if (probe.screenshot) {
await evidenceCallback({
type: "step_observed",
stepIndex,
type: "screenshot",
screenshot: probe.screenshot,
url: probe.url,
ariaTree: probe.ariaTree,
evidenceRole: "probe",
});
}
await evidenceCallback({
type: "step_observed",
url: probe.url,
ariaTree: probe.ariaTree,
});
}

View File

@ -0,0 +1,27 @@
import type { AgentEvidenceCallback } from "../../types/public/agentEvidenceEvents.js";
import type { LogLine } from "../../types/public/logs.js";
// onEvidence is a user-supplied observability hook (trajectory recording,
// verifier capture, etc.). Wrap it once at the boundary where the handler
// receives it so a throwing user callback can never abort the agent loop —
// internal emit sites can then call the wrapped callback directly without
// per-site try/catch.
export function wrapEvidenceCallback(
callback: AgentEvidenceCallback | undefined,
logger: (message: LogLine) => void,
): AgentEvidenceCallback | undefined {
if (!callback) return undefined;
return async (event) => {
try {
await callback(event);
} catch (e) {
logger({
category: "agent",
message: `Warning: onEvidence callback failed for ${event.type}: ${
e instanceof Error ? e.message : String(e)
}`,
level: 1,
});
}
};
}

View File

@ -106,6 +106,8 @@ interface ClientSessionStartParams extends Api.SessionStartRequest {
* Optional: when omitted, requests are sent without the x-model-api-key header
* and the server is expected to handle model authentication on its own. */
modelApiKey?: string;
/** Default model config for later action requests. Not sent to /sessions/start. */
defaultModelConfig?: ModelConfiguration;
}
/**
@ -115,6 +117,11 @@ type ApiResponse<T> =
| { success: true; data: T }
| { success: false; message: string };
type PreparedModelConfig = { modelName: string; apiKey?: string } & Record<
string,
unknown
>;
/**
* Union of all API request body types for type-safe execute() calls
*/
@ -180,6 +187,7 @@ export class StagehandAPIClient {
private sessionId?: string;
private modelApiKey?: string;
private modelProvider?: string;
private defaultModelConfig?: PreparedModelConfig;
private region?: BrowserbaseRegion;
private logger: (message: LogLine) => void;
private fetchWithCookies;
@ -205,6 +213,7 @@ export class StagehandAPIClient {
async init({
modelName,
modelApiKey,
defaultModelConfig,
domSettleTimeoutMs,
verbose,
systemPrompt,
@ -218,6 +227,9 @@ export class StagehandAPIClient {
this.modelProvider = modelName?.includes("/")
? modelName.split("/")[0]
: undefined;
this.defaultModelConfig = defaultModelConfig
? this.prepareModelConfig(defaultModelConfig)
: undefined;
// Store the region for multi-region API URL resolution
this.region = browserbaseSessionCreateParams?.region;
@ -288,12 +300,18 @@ export class StagehandAPIClient {
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const { page: _, serverCache: enableCache, ...restOptions } = options;
serverCache = enableCache;
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
} else if (this.defaultModelConfig) {
restOptions.model = this.getDefaultModelConfig();
}
if (Object.keys(restOptions).length > 0) {
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
}
wireOptions = restOptions as unknown as Api.ActRequest["options"];
}
} else if (this.defaultModelConfig) {
wireOptions = {
model: this.getDefaultModelConfig(),
} as unknown as Api.ActRequest["options"];
}
// Build wire-format request body
@ -326,12 +344,18 @@ export class StagehandAPIClient {
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const { page: _, serverCache: enableCache, ...restOptions } = options;
serverCache = enableCache;
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
} else if (this.defaultModelConfig) {
restOptions.model = this.getDefaultModelConfig();
}
if (Object.keys(restOptions).length > 0) {
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
}
wireOptions = restOptions as unknown as Api.ExtractRequest["options"];
}
} else if (this.defaultModelConfig) {
wireOptions = {
model: this.getDefaultModelConfig(),
} as unknown as Api.ExtractRequest["options"];
}
// Build wire-format request body
@ -361,12 +385,18 @@ export class StagehandAPIClient {
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const { page: _, serverCache: enableCache, ...restOptions } = options;
serverCache = enableCache;
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
} else if (this.defaultModelConfig) {
restOptions.model = this.getDefaultModelConfig();
}
if (Object.keys(restOptions).length > 0) {
if (restOptions.model) {
restOptions.model = this.prepareModelConfig(restOptions.model);
}
wireOptions = restOptions as unknown as Api.ObserveRequest["options"];
}
} else if (this.defaultModelConfig) {
wireOptions = {
model: this.getDefaultModelConfig(),
} as unknown as Api.ObserveRequest["options"];
}
// Build wire-format request body
@ -388,7 +418,22 @@ export class StagehandAPIClient {
options?: Api.NavigateRequest["options"],
frameId?: string,
): Promise<SerializableResponse | null> {
const requestBody: Api.NavigateRequest = { url, options, frameId };
const publicOptions = { ...(options ?? {}) } as NonNullable<
Api.NavigateRequest["options"]
> & { model?: unknown };
delete publicOptions.model;
const wireOptions = {
...publicOptions,
...(this.defaultModelConfig
? { model: this.getDefaultModelConfig() }
: {}),
};
const requestBody: Api.NavigateRequest = {
url,
options: Object.keys(wireOptions).length > 0 ? wireOptions : undefined,
frameId,
};
return this.execute<SerializableResponse | null>({
method: "navigate",
@ -425,7 +470,7 @@ export class StagehandAPIClient {
cua: agentConfig.mode === undefined ? agentConfig.cua : undefined,
model: agentConfig.model
? this.prepareModelConfig(agentConfig.model)
: undefined,
: this.getDefaultModelConfig(),
executionModel: agentConfig.executionModel
? this.prepareModelConfig(agentConfig.executionModel)
: undefined,
@ -605,40 +650,52 @@ export class StagehandAPIClient {
* In API mode, we only attempt to load an API key from env vars when the
* model provider differs from the one used to init the session.
*/
private prepareModelConfig(
model: ModelConfiguration,
): { modelName: string; apiKey?: string } & Record<string, unknown> {
private prepareModelConfig(model: ModelConfiguration): PreparedModelConfig {
if (typeof model === "string") {
// Extract provider from model string (e.g., "openai/gpt-5-nano" -> "openai")
const provider = model.includes("/") ? model.split("/")[0] : undefined;
const provider = this.getModelProvider(model);
const inheritedDefault =
provider && provider === this.modelProvider
? this.getDefaultModelConfig()
: undefined;
const apiKey =
provider && provider !== this.modelProvider
? (loadApiKeyFromEnv(provider, this.logger) ?? this.modelApiKey)
: this.modelApiKey;
return {
...inheritedDefault,
modelName: model,
...(apiKey ? { apiKey } : {}),
};
}
if (!model.apiKey) {
const provider = model.modelName?.includes("/")
? model.modelName.split("/")[0]
const provider = this.getModelProvider(model.modelName);
const inheritedDefault =
provider && provider === this.modelProvider
? this.getDefaultModelConfig()
: undefined;
const apiKey =
provider && provider !== this.modelProvider
? (loadApiKeyFromEnv(provider, this.logger) ?? this.modelApiKey)
: this.modelApiKey;
return {
...model,
...(apiKey ? { apiKey } : {}),
};
}
const apiKey =
!model.apiKey && provider && provider !== this.modelProvider
? (loadApiKeyFromEnv(provider, this.logger) ?? this.modelApiKey)
: !model.apiKey
? this.modelApiKey
: undefined;
return model as { modelName: string; apiKey: string } & Record<
string,
unknown
>;
return {
...inheritedDefault,
...model,
...(apiKey ? { apiKey } : {}),
} as PreparedModelConfig;
}
private getDefaultModelConfig(): PreparedModelConfig | undefined {
return this.defaultModelConfig
? ({ ...this.defaultModelConfig } as PreparedModelConfig)
: undefined;
}
private getModelProvider(modelName: string | undefined): string | undefined {
return modelName?.includes("/") ? modelName.split("/")[0] : undefined;
}
private consumeFinishedEventData<T>(): T | null {

View File

@ -46,6 +46,7 @@ import {
captureProbeEvidence,
emitPostStepProbeEvidence,
} from "../agent/utils/postStepProbeEvidence.js";
import { wrapEvidenceCallback } from "../agent/utils/wrapEvidenceCallback.js";
import { inferToolOutput } from "../agent/utils/toolOutputEvidence.js";
import {
CaptchaSolver,
@ -263,16 +264,8 @@ export class V3AgentHandler {
private createStepHandler(
state: AgentState,
{
userCallback,
evidenceCallback,
onFinalAnswer,
}: StepHandlerOptions,
{ userCallback, evidenceCallback, onFinalAnswer }: StepHandlerOptions,
) {
// Monotonic step counter scoped to this execute() call. Each tool call in
// the agent loop becomes one trajectory step. The counter feeds stepIndex
// on evidence callback events.
let stepCounter = 0;
return async (event: StepResult<ToolSet>) => {
this.logger({
category: "agent",
@ -280,7 +273,6 @@ export class V3AgentHandler {
level: 2,
});
const stepIndicesInTurn: number[] = [];
let lastFinalAnswer: FinalAnswerDraft | undefined;
if (event.toolCalls && event.toolCalls.length > 0) {
@ -328,12 +320,8 @@ export class V3AgentHandler {
state.actions.push(action);
}
const stepIndex = stepCounter++;
stepIndicesInTurn.push(stepIndex);
const finishedAt = new Date().toISOString();
await evidenceCallback?.({
type: "step_finished",
stepIndex,
actionName: toolCall.toolName,
actionArgs:
typeof args === "object" && args !== null
@ -341,19 +329,16 @@ export class V3AgentHandler {
: {},
reasoning: event.text ?? "",
toolOutput: inferToolOutput(toolResult),
finishedAt,
});
}
state.currentPageUrl = (await this.v3.context.awaitActivePage()).url();
// Harness probe — take a single screenshot / a11y snapshot per AI SDK
// step and attach it to every tool call in that turn. The observation
// reflects the settled page state after the batch of tool calls; this
// is more faithful than dropping probe evidence for all but the last
// tool call, while still avoiding per-tool screenshot overhead.
// Harness probe — one screenshot / a11y snapshot per AI SDK step.
// The recorder applies the probe to every step_finished received
// since the previous probe, so a multi-tool turn shares the same
// post-turn observation.
await emitPostStepProbeEvidence({
v3: this.v3,
stepIndices: stepIndicesInTurn,
url: state.currentPageUrl,
evidenceCallback,
logger: this.logger,
@ -438,6 +423,11 @@ export class V3AgentHandler {
}
}
const evidenceCallback = wrapEvidenceCallback(
callbacks?.onEvidence,
this.logger,
);
const result = await this.llmClient.generateText({
model: wrappedModel,
messages: prependSystemMessage(systemPrompt, messages),
@ -449,16 +439,13 @@ export class V3AgentHandler {
callbacks?.prepareStep,
captchaSolver,
),
onStepFinish: this.createStepHandler(
state,
{
userCallback: callbacks?.onStepFinish,
evidenceCallback: callbacks?.onEvidence,
onFinalAnswer: (answer) => {
finalAnswerFromDoneTool = answer;
},
onStepFinish: this.createStepHandler(state, {
userCallback: callbacks?.onStepFinish,
evidenceCallback,
onFinalAnswer: (answer) => {
finalAnswerFromDoneTool = answer;
},
),
}),
abortSignal: preparedOptions.signal,
providerOptions: {
google: { mediaResolution: "MEDIA_RESOLUTION_HIGH" },
@ -482,7 +469,7 @@ export class V3AgentHandler {
message: state.finalMessage,
output,
},
callbacks?.onEvidence,
evidenceCallback,
);
return this.consolidateMetricsAndResult(
@ -592,6 +579,11 @@ export class V3AgentHandler {
rejectResult(error);
};
const evidenceCallback = wrapEvidenceCallback(
callbacks?.onEvidence,
this.logger,
);
let streamResult: ReturnType<typeof this.llmClient.streamText>;
try {
streamResult = this.llmClient.streamText({
@ -604,16 +596,13 @@ export class V3AgentHandler {
callbacks?.prepareStep,
captchaSolver,
),
onStepFinish: this.createStepHandler(
state,
{
userCallback: callbacks?.onStepFinish,
evidenceCallback: callbacks?.onEvidence,
onFinalAnswer: (answer) => {
finalAnswerFromDoneTool = answer;
},
onStepFinish: this.createStepHandler(state, {
userCallback: callbacks?.onStepFinish,
evidenceCallback,
onFinalAnswer: (answer) => {
finalAnswerFromDoneTool = answer;
},
),
}),
onError: (event) => {
captchaSolver?.dispose();
if (callbacks?.onError) {
@ -649,7 +638,7 @@ export class V3AgentHandler {
message: state.finalMessage,
output,
},
callbacks?.onEvidence,
evidenceCallback,
);
const result = this.consolidateMetricsAndResult(
startTime,

View File

@ -11,7 +11,7 @@ import {
captureProbeEvidence,
emitPostStepProbeEvidence,
} from "../agent/utils/postStepProbeEvidence.js";
import { CuaEvidenceStepTracker } from "../agent/utils/cuaEvidenceStepTracker.js";
import { wrapEvidenceCallback } from "../agent/utils/wrapEvidenceCallback.js";
import { inferToolOutput } from "../agent/utils/toolOutputEvidence.js";
import {
ActionExecutionResult,
@ -22,7 +22,10 @@ import {
SafetyConfirmationHandler,
} from "../types/public/agent.js";
import { LogLine } from "../types/public/logs.js";
import type { AgentEvidenceCallback } from "../types/public/agentEvidenceEvents.js";
import type {
AgentEvidenceCallback,
AgentStepFinishedEvent,
} from "../types/public/agentEvidenceEvents.js";
import { type Action, V3FunctionName } from "../types/public/methods.js";
import { FlowLogger } from "../flowlogger/FlowLogger.js";
import { toTitleCase } from "../../utils.js";
@ -44,7 +47,7 @@ export class V3CuaAgentHandler {
private captchaSolver: CaptchaSolver | null = null;
private captchaClickGuardRemaining = 0;
private currentInstruction = "";
private readonly cuaEvidenceSteps = new CuaEvidenceStepTracker();
private lastAgentScreenshotUrl?: string;
private evidenceCallback?: AgentEvidenceCallback;
constructor(
@ -86,7 +89,7 @@ export class V3CuaAgentHandler {
const page = await this.v3.context.awaitActivePage();
const screenshotBuffer = await page.screenshot({ fullPage: false });
await this.emitCuaScreenshotNonFatal(screenshotBuffer, page.url());
await this.emitCuaScreenshot(screenshotBuffer, page.url());
return screenshotBuffer.toString("base64"); // base64 png
});
@ -131,9 +134,12 @@ export class V3CuaAgentHandler {
const waitBetween =
(this.options.clientOptions?.waitBetweenActions as number) ||
defaultDelay;
// Skip logging for screenshot actions - they're no-ops; the CUA client
// takes its own screenshot via screenshotProvider between API turns.
// Computed outside the try so the catch can still record a failed step.
const shouldLog = action.type !== "screenshot";
try {
let executionResult: ActionExecutionResult | undefined;
const startedAt = new Date().toISOString();
// Try to inject cursor before each action if enabled
if (this.highlightCursor) {
try {
@ -143,9 +149,6 @@ export class V3CuaAgentHandler {
}
}
await new Promise((r) => setTimeout(r, 300));
// Skip logging for screenshot actions - they're no-ops; the CUA client
// takes its own screenshot via screenshotProvider between API turns.
const shouldLog = action.type !== "screenshot";
if (shouldLog) {
executionResult = await FlowLogger.runWithLogging(
{
@ -163,8 +166,11 @@ export class V3CuaAgentHandler {
}
action.timestamp = Date.now();
if (shouldLog) {
await this.emitCuaActionStep(action, executionResult, startedAt);
if (shouldLog && this.evidenceCallback) {
await this.emitCuaActionStep(
action,
inferToolOutput(executionResult ?? { success: true }),
);
}
await new Promise((r) => setTimeout(r, waitBetween));
@ -175,6 +181,30 @@ export class V3CuaAgentHandler {
message: `Error executing action ${action.type}: ${msg}`,
level: 0,
});
// Record the failed action as an ok:false step (with a best-effort
// post-failure probe, since a throwing action can still partially
// mutate the page) before rethrowing — otherwise the failure is
// dropped from the persisted trajectory. Evidence emission must never
// mask the original action error.
if (shouldLog && this.evidenceCallback) {
try {
await this.emitCuaActionStep(action, {
ok: false,
result: undefined,
error: msg,
});
} catch (evidenceError) {
this.logger({
category: "agent",
message: `Failed to record failed-action evidence: ${
evidenceError instanceof Error
? evidenceError.message
: String(evidenceError)
}`,
level: 1,
});
}
}
throw error;
}
});
@ -201,8 +231,11 @@ export class V3CuaAgentHandler {
: optionsOrInstruction;
this.setSafetyConfirmationHandler(options.callbacks?.onSafetyConfirmation);
this.evidenceCallback = options.callbacks?.onEvidence;
this.cuaEvidenceSteps.reset();
this.evidenceCallback = wrapEvidenceCallback(
options.callbacks?.onEvidence,
this.logger,
);
this.lastAgentScreenshotUrl = undefined;
this.highlightCursor = options.highlightCursor !== false;
this.currentInstruction = options.instruction;
@ -263,7 +296,7 @@ export class V3CuaAgentHandler {
try {
finalUrl = (await this.v3.context.awaitActivePage()).url();
} catch {
finalUrl = this.cuaEvidenceSteps.latestScreenshotUrl ?? "";
finalUrl = this.lastAgentScreenshotUrl ?? "";
}
const observation = await captureProbeEvidence({
v3: this.v3,
@ -700,7 +733,7 @@ export class V3CuaAgentHandler {
const currentUrl = page.url();
// Mirror the same buffer the CUA client receives as agent evidence.
await this.emitCuaScreenshotNonFatal(screenshotBuffer, currentUrl);
await this.emitCuaScreenshot(screenshotBuffer, currentUrl);
return await this.agentClient.captureScreenshot({
base64Image: screenshotBuffer.toString("base64"),
@ -822,49 +855,34 @@ export class V3CuaAgentHandler {
screenshot: Buffer,
url: string,
): Promise<void> {
await this.evidenceCallback?.(
this.cuaEvidenceSteps.recordScreenshot(screenshot, url),
);
}
private async emitCuaScreenshotNonFatal(
screenshot: Buffer,
url: string,
): Promise<void> {
try {
await this.emitCuaScreenshot(screenshot, url);
} catch (e) {
this.logger({
category: "agent",
message: `Warning: CUA screenshot evidence callback failed: ${
e instanceof Error ? e.message : String(e)
}`,
level: 1,
});
}
if (!this.evidenceCallback) return;
this.lastAgentScreenshotUrl = url;
await this.evidenceCallback({
type: "screenshot",
screenshot,
url,
evidenceRole: "agent",
});
}
private async emitCuaActionStep(
action: AgentAction,
result: ActionExecutionResult | undefined,
startedAt: string,
toolOutput: AgentStepFinishedEvent["toolOutput"],
): Promise<void> {
let pageUrl =
typeof action.pageUrl === "string"
? action.pageUrl
: (this.cuaEvidenceSteps.latestScreenshotUrl ?? "");
: (this.lastAgentScreenshotUrl ?? "");
try {
pageUrl = (await this.v3.context.awaitActivePage()).url();
} catch {
// Keep the best pre-action URL fallback.
}
const { stepIndex, replayScreenshot } = this.cuaEvidenceSteps.pairAction();
if (replayScreenshot) {
await this.evidenceCallback?.(replayScreenshot);
}
const actionArgs = Object.fromEntries(
Object.entries(action).filter(([key]) => key !== "screenshot"),
Object.entries(action).filter(
([key]) => key !== "screenshot" && key !== "timestamp",
),
);
const reasoning =
typeof action.reasoning === "string"
@ -875,13 +893,10 @@ export class V3CuaAgentHandler {
await this.evidenceCallback?.({
type: "step_finished",
stepIndex,
actionName: String(action.type),
actionArgs,
reasoning,
toolOutput: inferToolOutput(result ?? { success: true }),
startedAt,
finishedAt: new Date().toISOString(),
toolOutput,
});
// Post-action tier-2 probe. The pre-action screenshot from
@ -891,7 +906,6 @@ export class V3CuaAgentHandler {
// has to trust the action history alone.
await emitPostStepProbeEvidence({
v3: this.v3,
stepIndices: stepIndex,
url: pageUrl,
evidenceCallback: this.evidenceCallback,
logger: this.logger,

View File

@ -1,6 +1,5 @@
import type { LanguageModelV2Middleware } from "@ai-sdk/provider";
import {
ExperimentalNotConfiguredError,
UnsupportedAISDKModelProviderError,
UnsupportedModelError,
UnsupportedModelProviderError,
@ -70,6 +69,37 @@ const AISDKProvidersWithAPIKey: Record<string, AISDKCustomProvider> = {
gateway: createGateway,
};
type AISDKProviderClientOptions = ClientOptions & Record<string, unknown>;
export function toAISDKClientOptions(
subProvider: string,
clientOptions?: ClientOptions,
): AISDKProviderClientOptions | undefined {
if (!clientOptions || subProvider !== "vertex") {
return clientOptions as AISDKProviderClientOptions | undefined;
}
const { auth, providerOptions, ...rest } = clientOptions;
const vertexOptions = providerOptions?.vertex;
return {
...rest,
...(vertexOptions ?? {}),
...(auth?.type === "googleServiceAccount"
? {
googleAuthOptions: {
credentials: auth.credentials,
...(auth.scopes ? { scopes: auth.scopes } : {}),
...(auth.projectId ? { projectId: auth.projectId } : {}),
...(auth.universeDomain
? { universeDomain: auth.universeDomain }
: {}),
},
}
: {}),
};
}
const modelToProviderMap: { [key in AvailableModel]: ModelProvider } = {
"gpt-4.1": "openai",
"gpt-4.1-mini": "openai",
@ -106,9 +136,12 @@ export function getAISDKLanguageModel(
clientOptions?: ClientOptions,
middleware?: LanguageModelV2Middleware,
) {
const aiSdkClientOptions = toAISDKClientOptions(subProvider, clientOptions);
const hasValidOptions =
clientOptions &&
Object.values(clientOptions).some((v) => v !== undefined && v !== null);
aiSdkClientOptions &&
Object.values(aiSdkClientOptions).some(
(v) => v !== undefined && v !== null,
);
let model;
if (hasValidOptions) {
@ -119,7 +152,7 @@ export function getAISDKLanguageModel(
Object.keys(AISDKProvidersWithAPIKey),
);
}
const provider = creator(clientOptions);
const provider = creator(aiSdkClientOptions as ClientOptions);
model = provider(subModelName);
} else {
const provider = AISDKProviders[subProvider];
@ -163,13 +196,6 @@ export class LLMProvider {
const firstSlashIndex = modelName.indexOf("/");
const subProvider = modelName.substring(0, firstSlashIndex);
const subModelName = modelName.substring(firstSlashIndex + 1);
if (
subProvider === "vertex" &&
!options?.disableAPI &&
!options?.experimental
) {
throw new ExperimentalNotConfiguredError("Vertex provider");
}
const effectiveMiddleware = options?.middleware ?? this.middleware;
const languageModel = getAISDKLanguageModel(

View File

@ -462,6 +462,7 @@ export const AVAILABLE_CUA_MODELS = [
"openai/computer-use-preview-2025-03-11",
"anthropic/claude-opus-4-5-20251101",
"anthropic/claude-opus-4-6",
"anthropic/claude-opus-4-8",
"anthropic/claude-sonnet-4-6",
"anthropic/claude-haiku-4-5",
"anthropic/claude-haiku-4-5-20251001",

View File

@ -2,7 +2,12 @@
* Evidence events emitted through AgentExecuteOptions.callbacks.onEvidence.
*
* These events describe observations made by Stagehand during an agent run.
* They are intentionally transport-level callback payloads; verifier-specific
* They are emitted in temporal order; consumers should treat the stream as
* sequential. An agent-role screenshot applies to every subsequent
* step_finished until a newer agent-role screenshot replaces it — a CUA
* provider may choose multiple actions from a single screenshot, so each of
* those steps shares that frame. A step_observed/probe applies to all
* step_finished events received since the last probe. Verifier-specific
* storage and normalization live in the evals/verifier layers.
*/
@ -23,14 +28,12 @@ export type AgentEvidenceEvent =
*/
export interface AgentScreenshotEvidenceEvent {
type: "screenshot";
/** Zero-based index of the step this screenshot corresponds to. */
stepIndex: number;
/** PNG bytes from page.screenshot(). */
screenshot: Buffer;
/** Page URL at the time of capture. */
url: string;
/** Role this screenshot plays in downstream evidence collection. */
evidenceRole?: AgentEvidenceRole;
evidenceRole: AgentEvidenceRole;
}
/**
@ -38,7 +41,6 @@ export interface AgentScreenshotEvidenceEvent {
*/
export interface AgentStepFinishedEvent {
type: "step_finished";
stepIndex: number;
/** Name of the tool/action that ran, e.g. "act", "extract", "click". */
actionName: string;
/** Arguments passed to the tool/action. */
@ -52,24 +54,18 @@ export interface AgentStepFinishedEvent {
result: unknown;
error?: string;
};
/** ISO 8601 timestamp at which the step's tool execution started, when available. */
startedAt?: string;
/** ISO 8601 timestamp at which the step finished. */
finishedAt: string;
}
/**
* Independent post-step browser observation.
* Independent post-step browser observation. Emitted once per agent turn;
* consumers apply it to every step_finished received since the previous probe.
*/
export interface AgentStepObservedEvent {
type: "step_observed";
stepIndex: number;
/** Page URL after the step's tool/action execution. */
url: string;
/** Accessibility tree snapshot, when captured. */
ariaTree?: string;
/** Viewport scroll context, when captured. */
scroll?: { top: number; pageHeight: number };
}
export interface AgentFinalObservation {
@ -79,8 +75,6 @@ export interface AgentFinalObservation {
screenshot?: Buffer;
/** Accessibility tree snapshot, when captured. */
ariaTree?: string;
/** Viewport scroll context, when captured. */
scroll?: { top: number; pageHeight: number };
}
/** Final answer emitted by the agent, when available. */

View File

@ -73,9 +73,13 @@ export const GoogleServiceAccountCredentialsSchema = z
.strict()
.meta({ id: "GoogleServiceAccountCredentials" });
export const GoogleAuthOptionsSchema = z
export const GoogleServiceAccountAuthSchema = z
.object({
credentials: GoogleServiceAccountCredentialsSchema.optional().meta({
type: z.literal("googleServiceAccount").meta({
description:
"Use inline Google Cloud service account credentials for provider authentication",
}),
credentials: GoogleServiceAccountCredentialsSchema.meta({
description: "Google Cloud service account credentials",
}),
scopes: z
@ -92,18 +96,44 @@ export const GoogleAuthOptionsSchema = z
}),
})
.strict()
.meta({ id: "GoogleAuthOptions" });
.meta({ id: "GoogleServiceAccountAuth" });
export const ModelConfigObjectSchema = z
export const ModelAuthSchema = z
.discriminatedUnion("type", [GoogleServiceAccountAuthSchema])
.meta({ id: "ModelAuth" });
export const VertexProviderOptionsSchema = z
.object({
project: z.string().meta({
description: "Google Cloud project ID for Vertex AI models",
example: "my-gcp-project",
}),
location: z.string().meta({
description: "Google Cloud location for Vertex AI models",
example: "us-central1",
}),
baseURL: z.string().url().optional().meta({
description: "Base URL for the Vertex AI provider",
}),
headers: z.record(z.string(), z.string()).optional().meta({
description:
"Custom headers sent with every request to the Vertex AI provider",
}),
})
.strict()
.meta({ id: "VertexProviderOptions" });
export const ModelProviderOptionsSchema = z
.object({
vertex: VertexProviderOptionsSchema.meta({
description: "Vertex AI provider-specific settings",
}),
})
.strict()
.meta({ id: "ModelProviderOptions" });
const ModelConfigBaseSchema = z
.object({
provider: z
.enum(["openai", "anthropic", "google", "microsoft", "bedrock", "vertex"])
.optional()
.meta({
description:
"AI provider for the model (or provide a baseURL endpoint instead)",
example: "openai",
}),
modelName: z.string().meta({
description:
"Model name string with provider prefix (e.g., 'openai/gpt-5-nano')",
@ -121,19 +151,38 @@ export const ModelConfigObjectSchema = z
description:
"Custom headers sent with every request to the model provider",
}),
project: z.string().optional().meta({
description: "Google Cloud project ID for Vertex AI models",
example: "my-gcp-project",
}),
location: z.string().optional().meta({
description: "Google Cloud location for Vertex AI models",
example: "us-central1",
}),
googleAuthOptions: GoogleAuthOptionsSchema.optional().meta({
description:
"google-auth-library options used to authenticate Vertex AI models",
}),
})
.strict();
export const GenericModelConfigObjectSchema = ModelConfigBaseSchema.extend({
provider: z
.enum(["openai", "anthropic", "google", "microsoft", "bedrock"])
.optional()
.meta({
description:
"AI provider for the model (or provide a baseURL endpoint instead)",
example: "openai",
}),
})
.strict()
.meta({ id: "GenericModelConfigObject" });
export const VertexModelConfigObjectSchema = ModelConfigBaseSchema.extend({
provider: z.literal("vertex").meta({
description: "Vertex AI model provider",
}),
auth: ModelAuthSchema.meta({
description: "Vertex provider authentication configuration",
}),
providerOptions: ModelProviderOptionsSchema.meta({
description: "Vertex provider-specific model configuration",
}),
})
.strict()
.meta({ id: "VertexModelConfigObject" });
export const ModelConfigObjectSchema = z
.union([VertexModelConfigObjectSchema, GenericModelConfigObjectSchema])
.meta({ id: "ModelConfigObject" });
/** Model configuration */
@ -1161,6 +1210,21 @@ export const Operations = {
// Shared types
export type Action = z.infer<typeof ActionSchema>;
export type ModelConfig = z.infer<typeof ModelConfigSchema>;
export type GenericModelConfigObject = z.infer<
typeof GenericModelConfigObjectSchema
>;
export type VertexModelConfigObject = z.infer<
typeof VertexModelConfigObjectSchema
>;
export type GoogleServiceAccountCredentials = z.infer<
typeof GoogleServiceAccountCredentialsSchema
>;
export type GoogleServiceAccountAuth = z.infer<
typeof GoogleServiceAccountAuthSchema
>;
export type ModelAuth = z.infer<typeof ModelAuthSchema>;
export type VertexProviderOptions = z.infer<typeof VertexProviderOptionsSchema>;
export type ModelProviderOptions = z.infer<typeof ModelProviderOptionsSchema>;
export type BrowserConfig = z.infer<typeof BrowserConfigSchema>;
export type SessionIdParams = z.infer<typeof SessionIdParamsSchema>;

View File

@ -1,5 +1,4 @@
import type { ClientOptions as AnthropicClientOptionsBase } from "@anthropic-ai/sdk";
import type { GoogleVertexProviderSettings as GoogleVertexProviderSettingsBase } from "@ai-sdk/google-vertex";
import type {
LanguageModelV2,
LanguageModelV2Middleware,
@ -31,19 +30,26 @@ export interface GoogleServiceAccountCredentials {
universe_domain?: string;
}
export interface GoogleVertexAuthOptions {
credentials?: GoogleServiceAccountCredentials;
export interface GoogleServiceAccountAuth {
type: "googleServiceAccount";
credentials: GoogleServiceAccountCredentials;
scopes?: string | string[];
projectId?: string;
universeDomain?: string;
}
export type GoogleVertexProviderSettings = Pick<
GoogleVertexProviderSettingsBase,
"project" | "location" | "headers" | "baseURL"
> & {
googleAuthOptions?: GoogleVertexAuthOptions;
};
export type ModelAuth = GoogleServiceAccountAuth;
export interface VertexProviderOptions {
project: string;
location: string;
baseURL?: string;
headers?: Record<string, string>;
}
export interface ModelProviderOptions {
vertex: VertexProviderOptions;
}
export type AnthropicJsonSchemaObject = {
definitions?: {
@ -113,13 +119,11 @@ export type ModelProvider =
*/
export type ThinkingEffort = "none" | "low" | "medium" | "high" | "max";
export type ClientOptions = (
| OpenAIClientOptions
| AnthropicClientOptions
| GoogleVertexProviderSettings
) & {
export type ClientOptions = (OpenAIClientOptions | AnthropicClientOptions) & {
apiKey?: string;
provider?: AgentProviderType;
auth?: ModelAuth;
providerOptions?: ModelProviderOptions;
baseURL?: string;
/** OpenAI organization ID */
organization?: string;

View File

@ -538,6 +538,26 @@ export class V3 {
);
}
private getApiDefaultModelConfig(): ModelConfiguration | undefined {
const clientOptions = Object.fromEntries(
Object.entries(this.modelClientOptions).filter(
([key, value]) => key !== "apiKey" && value !== undefined,
),
);
if (Object.keys(clientOptions).length === 0) {
return undefined;
}
return {
modelName: this.modelName,
...clientOptions,
...((this.modelClientOptions as { apiKey?: string }).apiKey
? { apiKey: (this.modelClientOptions as { apiKey?: string }).apiKey }
: {}),
} as ModelConfiguration;
}
private resolveLlmClient(model?: ModelConfiguration): LLMClient {
if (!model) {
return this.llmClient;
@ -1043,6 +1063,7 @@ export class V3 {
const { sessionId, available } = await this.apiClient.init({
modelName: this.modelName,
modelApiKey: this.modelClientOptions.apiKey,
defaultModelConfig: this.getApiDefaultModelConfig(),
domSettleTimeoutMs: this.domSettleTimeoutMs,
verbose: this.verbose,
systemPrompt: this.opts.systemPrompt,

View File

@ -253,7 +253,6 @@ export function shouldPersistTrajectory(
* ├── screenshots/
* │ ├── probe/<N>.png
* │ └── agent/<N>[_M].png
* ├── times.json
* ├── scores/ (empty; populated separately)
* └── core.log
*
@ -269,11 +268,21 @@ export async function writeTrajectoryDir(
await fs.mkdir(path.join(dir, "screenshots", "agent"), { recursive: true });
const serializableSteps: unknown[] = [];
for (const step of trajectory.steps) {
// A single post-turn probe is fanned across every step of a multi-tool turn,
// and a single agent screenshot is shared across every action a CUA provider
// chose from it, so the same Buffer is shared by reference. Dedupe by
// identity: write the PNG once and point every sharing step at the same file.
const probePathByBuffer = new Map<Buffer, string>();
const agentPathByBuffer = new Map<Buffer, string>();
for (const [i, step] of trajectory.steps.entries()) {
const probe: ProbeEvidence = { ...step.probeEvidence };
if (probe.screenshot) {
const relPath = `screenshots/probe/${step.index + 1}.png`;
await fs.writeFile(path.join(dir, relPath), probe.screenshot);
let relPath = probePathByBuffer.get(probe.screenshot);
if (!relPath) {
relPath = `screenshots/probe/${i + 1}.png`;
await fs.writeFile(path.join(dir, relPath), probe.screenshot);
probePathByBuffer.set(probe.screenshot, relPath);
}
probe.screenshotPath = relPath;
delete probe.screenshot;
}
@ -296,9 +305,13 @@ export async function writeTrajectoryDir(
);
continue;
}
const suffix = multipleImages ? `_${imageSeq}` : "";
const relPath = `screenshots/agent/${step.index + 1}${suffix}.png`;
await fs.writeFile(path.join(dir, relPath), m.bytes);
let relPath = agentPathByBuffer.get(m.bytes);
if (!relPath) {
const suffix = multipleImages ? `_${imageSeq}` : "";
relPath = `screenshots/agent/${i + 1}${suffix}.png`;
await fs.writeFile(path.join(dir, relPath), m.bytes);
agentPathByBuffer.set(m.bytes, relPath);
}
modalities.push({
type: "image",
imagePath: relPath,
@ -357,19 +370,6 @@ export async function writeTrajectoryDir(
),
);
await fs.writeFile(
path.join(dir, "times.json"),
JSON.stringify(
{
timing: trajectory.timing,
usage: trajectory.usage,
stepCount: trajectory.steps.length,
},
null,
2,
),
);
await fs.mkdir(path.join(dir, "scores"), { recursive: true });
await fs.writeFile(path.join(dir, "core.log"), coreLog(trajectory));
}
@ -377,15 +377,13 @@ export async function writeTrajectoryDir(
function coreLog(trajectory: Trajectory): string {
return (
trajectory.steps
.map((step) =>
.map((step, i) =>
JSON.stringify({
step: step.index,
step: i,
action: step.actionName,
url: step.probeEvidence.url ?? null,
ok: step.toolOutput.ok,
reasoning: step.reasoning || undefined,
startedAt: step.startedAt,
finishedAt: step.finishedAt,
}),
)
.join("\n") + "\n"

View File

@ -89,8 +89,6 @@ export interface ProbeEvidence {
screenshot?: Buffer;
/** Reference to the persisted screenshot file under the trajectory dir. */
screenshotPath?: string;
/** Viewport scroll context. Lets the verifier reason about whether the agent saw the full page. */
scroll?: { top: number; pageHeight: number };
/** Accessibility tree snapshot. */
ariaTree?: string;
/** Verifier-requested probes, keyed by criterion id. */
@ -111,7 +109,6 @@ export interface ToolOutput {
/** One step in a trajectory: action + reasoning + evidence + outcome. */
export interface TrajectoryStep {
index: number;
actionName: string;
actionArgs: Record<string, unknown>;
/** From AgentAction.reasoning. May be empty for tools that don't surface reasoning. */
@ -119,10 +116,6 @@ export interface TrajectoryStep {
agentEvidence: AgentEvidence;
probeEvidence: ProbeEvidence;
toolOutput: ToolOutput;
/** ISO 8601 timestamp when the step's tool execution started. */
startedAt: string;
/** ISO 8601 timestamp when the step's tool execution finished. */
finishedAt: string;
}
/** Terminal status of the agent run. */
@ -139,8 +132,7 @@ export type TrajectoryStatus = "complete" | "aborted" | "stalled" | "error";
* ├── screenshots/ — step probe/agent images plus final observation
* ├── scores/
* │ └── result.json — Result from V3Evaluator.verify()
* ├── core.log — captured action log
* └── times.json — step timing + token usage
* └── core.log — captured action log
*/
export interface Trajectory {
task: TaskSpec;
@ -150,7 +142,6 @@ export interface Trajectory {
finalObservation?: ProbeEvidence;
status: TrajectoryStatus;
usage: TrajectoryUsage;
timing: { startedAt: string; endedAt: string };
}
/** Score for a single rubric criterion after evidence analysis + rescoring. */

View File

@ -237,13 +237,13 @@ function collectLegacyScreenshots(trajectory: Trajectory): Buffer[] {
function renderLegacyAgentReasoning(
trajectory: Trajectory,
): string | undefined {
const stepLines = (trajectory.steps ?? []).map((step) => {
const stepLines = (trajectory.steps ?? []).map((step, i) => {
const status = step.toolOutput?.ok === false ? "Tool status: failed" : "";
const output = step.toolOutput?.error
? `Tool error: ${step.toolOutput.error}`
: `Tool output: ${stringifyForPrompt(step.toolOutput?.result)}`;
return [
`Step ${step.index}: ${step.actionName}`,
`Step ${i}: ${step.actionName}`,
step.reasoning ? `Reasoning: ${step.reasoning}` : undefined,
status || undefined,
output,

View File

@ -90,7 +90,7 @@
"pino": "^9.6.0",
"pino-pretty": "^13.0.0",
"uuid": "^11.1.1",
"ws": "^8.18.0",
"ws": "^8.21.0",
"zod-to-json-schema": "^3.25.0"
},
"optionalDependencies": {

View File

@ -553,8 +553,63 @@ describe("v3 cua handler screenshot behavior", () => {
);
expect(
logs.some((line) =>
line.message.includes("CUA screenshot evidence callback failed"),
line.message.includes("onEvidence callback failed for screenshot"),
),
).toBe(true);
});
it("records a failed action as step_finished {ok:false} and rethrows the original error", async () => {
const events: Array<{ type: string; [k: string]: unknown }> = [];
const onEvidence = vi.fn(async (event: { type: string }) => {
events.push(event as { type: string });
});
fakeCuaClient.executeImpl = vi.fn(async () => {
await fakeCuaClient.actionHandler?.({
type: "click",
button: "left",
x: 5,
y: 9,
});
return { success: true, message: "ok", actions: [], completed: true };
});
const handler = new V3CuaAgentHandler(
{
context: {
awaitActivePage: async () => page,
},
isCaptchaAutoSolveEnabled: false,
isAdvancedStealth: false,
configuredViewport: { width: 1288, height: 711 },
isAgentReplayActive: () => false,
updateMetrics: vi.fn(),
} as never,
logger,
{
modelName: "openai/gpt-5.4",
clientOptions: { waitBetweenActions: 1 },
} as never,
);
vi.spyOn(
handler as unknown as {
executeAction: (action: Record<string, unknown>) => Promise<unknown>;
},
"executeAction",
).mockRejectedValue(new Error("click failed"));
await expect(
handler.execute({
instruction: "click the thing",
highlightCursor: false,
callbacks: { onEvidence },
}),
).rejects.toThrow("click failed");
const stepFinished = events.find((e) => e.type === "step_finished");
expect(stepFinished).toMatchObject({
actionName: "click",
toolOutput: { ok: false, error: "click failed" },
});
});
});

View File

@ -36,7 +36,30 @@ describe("AnthropicCUAClient adaptive thinking", () => {
});
});
describe("Claude 4.6 models (adaptive thinking)", () => {
describe("Claude 4.6+ models (adaptive thinking)", () => {
it("should use thinking.type: 'adaptive' for claude-opus-4-8 when thinkingEffort is set", async () => {
const client = new AnthropicCUAClient(
"anthropic",
"claude-opus-4-8",
undefined,
{
apiKey: "test-key",
thinkingEffort: "high",
},
);
client.setViewport(1280, 720);
await client.getAction([{ role: "user", content: "test" }]);
expect(mockCreate).toHaveBeenCalledWith(
expect.objectContaining({
thinking: { type: "adaptive" },
output_config: { effort: "high" },
temperature: 1,
}),
);
});
it("should use thinking.type: 'adaptive' for claude-opus-4-6 when thinkingEffort is set", async () => {
const client = new AnthropicCUAClient(
"anthropic",

View File

@ -0,0 +1,342 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { StagehandAPIClient } from "../../lib/v3/api.js";
import type { ModelConfiguration } from "../../lib/v3/types/public/model.js";
const vertexModel = {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
} as unknown as ModelConfiguration;
function createClientWithExecuteMock() {
const client = new StagehandAPIClient({
apiKey: "bb-test",
logger: vi.fn(),
});
const executeMock = vi.fn().mockResolvedValue({});
(
client as unknown as {
execute: typeof executeMock;
}
).execute = executeMock;
return { client, executeMock };
}
describe("StagehandAPIClient default model config", () => {
let originalFetch: typeof globalThis.fetch;
beforeEach(() => {
originalFetch = globalThis.fetch;
globalThis.fetch = vi.fn().mockResolvedValue(
new Response(
JSON.stringify({
success: true,
data: { sessionId: "sess-default-model", available: true },
}),
{
status: 200,
headers: { "Content-Type": "application/json" },
},
),
);
});
afterEach(() => {
globalThis.fetch = originalFetch;
vi.restoreAllMocks();
});
it("sends constructor Vertex model config on navigate bootstrap, act, observe, extract, and agent execute requests", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "vertex/gemini-2.5-flash",
defaultModelConfig: vertexModel,
});
await client.goto("https://example.com");
await client.act({ input: "click the login button" });
await client.observe({ instruction: "find the login button" });
await client.extract({ instruction: "extract the headline" });
await client.agentExecute({ mode: "dom" }, "click the login button");
expect(executeMock).toHaveBeenNthCalledWith(
1,
expect.objectContaining({
method: "navigate",
args: expect.objectContaining({
options: {
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
},
}),
}),
);
expect(executeMock).toHaveBeenNthCalledWith(
2,
expect.objectContaining({
method: "act",
args: expect.objectContaining({
options: {
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
},
}),
}),
);
expect(executeMock).toHaveBeenNthCalledWith(
3,
expect.objectContaining({
method: "observe",
args: expect.objectContaining({
options: expect.objectContaining({
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
}),
}),
}),
);
expect(executeMock).toHaveBeenNthCalledWith(
4,
expect.objectContaining({
method: "extract",
args: expect.objectContaining({
options: expect.objectContaining({
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
}),
}),
}),
);
expect(executeMock).toHaveBeenNthCalledWith(
5,
expect.objectContaining({
method: "agentExecute",
args: expect.objectContaining({
agentConfig: expect.objectContaining({
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
}),
}),
}),
);
});
it("prefers a per-call model over the constructor default model config", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "vertex/gemini-2.5-flash",
defaultModelConfig: vertexModel,
});
await client.act({
input: "click the login button",
options: {
model: {
modelName: "openai/gpt-4.1-mini",
apiKey: "sk-per-call",
},
},
});
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
args: expect.objectContaining({
options: expect.objectContaining({
model: {
modelName: "openai/gpt-4.1-mini",
apiKey: "sk-per-call",
},
}),
}),
}),
);
});
it("inherits constructor Vertex auth for a same-provider string model override", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "vertex/gemini-2.5-flash",
defaultModelConfig: vertexModel,
});
await client.observe({
instruction: "find the login button",
options: {
model: "vertex/gemini-2.0-flash",
},
});
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
method: "observe",
args: expect.objectContaining({
options: expect.objectContaining({
model: expect.objectContaining({
modelName: "vertex/gemini-2.0-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
}),
}),
}),
);
});
it("does not treat model as a public navigate option", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "openai/gpt-4.1-mini",
modelApiKey: "sk-header-only",
});
await client.goto("https://example.com", {
model: {
modelName: "openai/gpt-4o",
apiKey: "sk-per-call",
},
} as unknown as Parameters<StagehandAPIClient["goto"]>[1]);
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
method: "navigate",
args: expect.objectContaining({
options: undefined,
}),
}),
);
});
it("keeps constructor bootstrap config when a caller passes an unsupported navigate model", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "vertex/gemini-2.5-flash",
defaultModelConfig: vertexModel,
});
await client.goto("https://example.com", {
model: {
modelName: "openai/gpt-4o",
apiKey: "sk-per-call",
},
} as unknown as Parameters<StagehandAPIClient["goto"]>[1]);
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
method: "navigate",
args: expect.objectContaining({
options: {
model: expect.objectContaining({
modelName: "vertex/gemini-2.5-flash",
auth: expect.objectContaining({
type: "googleServiceAccount",
}),
providerOptions: {
vertex: expect.objectContaining({
project: "test-gcp-project",
location: "us-central1",
}),
},
}),
},
}),
}),
);
});
it("does not add a body model config when the constructor only provides a model API key", async () => {
const { client, executeMock } = createClientWithExecuteMock();
await client.init({
modelName: "openai/gpt-4.1-mini",
modelApiKey: "sk-header-only",
});
await client.act({ input: "click the login button" });
expect(executeMock).toHaveBeenCalledWith(
expect.objectContaining({
args: expect.objectContaining({
options: undefined,
}),
}),
);
});
});

View File

@ -73,9 +73,8 @@ describe("API model config schemas", () => {
const vertexModel = {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
project: "test-gcp-project",
location: "us-central1",
googleAuthOptions: {
auth: {
type: "googleServiceAccount",
credentials: {
type: "service_account",
project_id: "test-gcp-project",
@ -89,6 +88,12 @@ describe("API model config schemas", () => {
projectId: "test-gcp-project",
universeDomain: "googleapis.com",
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
};
it("preserves Vertex auth params for act requests", () => {
@ -111,15 +116,20 @@ describe("API model config schemas", () => {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
project: "test-gcp-project",
location: "us-central1",
googleAuthOptions: {
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
},
},
});
@ -131,7 +141,11 @@ describe("API model config schemas", () => {
if (typeof parsedModel !== "object" || parsedModel === null) {
throw new Error("Expected object model config");
}
expect(parsedModel.googleAuthOptions).toEqual({
if (!("auth" in parsedModel)) {
throw new Error("Expected Vertex auth config");
}
expect(parsedModel.auth).toEqual({
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
@ -140,6 +154,67 @@ describe("API model config schemas", () => {
});
});
it("requires explicit Vertex provider configs to include service account auth and provider options", () => {
for (const model of [
{
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
},
{
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
},
]) {
const result = Api.ActRequestSchema.safeParse({
input: "click the search button",
options: { model },
});
expect(result.success).toBe(false);
}
});
it("requires provider vertex when passing Vertex auth and provider options", () => {
const result = Api.ActRequestSchema.safeParse({
input: "click the search button",
options: {
model: {
modelName: "vertex/gemini-2.5-flash",
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
},
},
});
expect(result.success).toBe(false);
});
it("preserves Vertex auth params for agent model configs", () => {
const result = Api.AgentExecuteRequestSchema.safeParse({
agentConfig: {
@ -158,6 +233,20 @@ describe("API model config schemas", () => {
});
it("rejects Vertex key file paths and external credential sources", () => {
const topLevelLegacyShape = Api.ActRequestSchema.safeParse({
input: "click the search button",
options: {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
project: "test-gcp-project",
location: "us-central1",
googleAuthOptions: {},
},
},
});
expect(topLevelLegacyShape.success).toBe(false);
for (const blockedAuthOptions of [
{ keyFilename: "/etc/passwd" },
{ keyFile: "/etc/passwd" },
@ -169,7 +258,7 @@ describe("API model config schemas", () => {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
googleAuthOptions: blockedAuthOptions,
auth: blockedAuthOptions,
},
},
});
@ -183,7 +272,8 @@ describe("API model config schemas", () => {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
googleAuthOptions: {
auth: {
type: "googleServiceAccount",
credentials: {
type: "external_account",
credential_source: {

View File

@ -1,55 +0,0 @@
import { describe, expect, it } from "vitest";
import { CuaEvidenceStepTracker } from "../../lib/v3/agent/utils/cuaEvidenceStepTracker.js";
describe("CuaEvidenceStepTracker", () => {
it("pairs a fresh provider screenshot with the next action", () => {
const tracker = new CuaEvidenceStepTracker();
const screenshot = Buffer.from("screen");
const event = tracker.recordScreenshot(screenshot, "https://example.com");
const paired = tracker.pairAction();
expect(event).toMatchObject({
type: "screenshot",
stepIndex: 0,
evidenceRole: "agent",
url: "https://example.com",
});
expect(paired).toEqual({ stepIndex: 0 });
});
it("allocates an action step without screenshot evidence", () => {
const tracker = new CuaEvidenceStepTracker();
expect(tracker.pairAction()).toEqual({ stepIndex: 0 });
});
it("replays the latest consumed screenshot for later actions", () => {
const tracker = new CuaEvidenceStepTracker();
const screenshot = Buffer.from("screen");
tracker.recordScreenshot(screenshot, "https://example.com/start");
tracker.pairAction();
const paired = tracker.pairAction();
expect(paired.stepIndex).toBe(1);
expect(paired.replayScreenshot).toMatchObject({
type: "screenshot",
stepIndex: 1,
evidenceRole: "agent",
url: "https://example.com/start",
});
expect(paired.replayScreenshot?.screenshot).toEqual(screenshot);
});
it("resets step allocation and pending screenshot state", () => {
const tracker = new CuaEvidenceStepTracker();
tracker.recordScreenshot(Buffer.from("screen"), "https://example.com");
tracker.reset();
expect(tracker.pairAction()).toEqual({ stepIndex: 0 });
expect(tracker.latestScreenshotUrl).toBeUndefined();
});
});

View File

@ -1,5 +1,9 @@
import { describe, expect, it } from "vitest";
import { getAISDKLanguageModel } from "../../lib/v3/llm/LLMProvider.js";
import {
getAISDKLanguageModel,
LLMProvider,
toAISDKClientOptions,
} from "../../lib/v3/llm/LLMProvider.js";
describe("getAISDKLanguageModel", () => {
describe("ollama provider", () => {
@ -68,3 +72,67 @@ describe("getAISDKLanguageModel", () => {
});
});
});
describe("LLMProvider", () => {
it("allows Vertex models without experimental mode", () => {
const provider = new LLMProvider(() => {});
expect(() =>
provider.getClient(
"vertex/gemini-2.5-flash" as never,
{
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "stagehand@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\ntest\n-----END PRIVATE KEY-----\n",
},
},
providerOptions: {
vertex: {
project: "test-project",
location: "us-central1",
},
},
} as never,
{ experimental: false, disableAPI: false },
),
).not.toThrow();
});
it("adapts canonical Vertex auth into AI SDK googleAuthOptions", () => {
expect(
toAISDKClientOptions("vertex", {
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "stagehand@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\ntest\n-----END PRIVATE KEY-----\n",
},
projectId: "test-project",
universeDomain: "googleapis.com",
},
providerOptions: {
vertex: {
project: "test-project",
location: "us-central1",
},
},
}),
).toEqual({
project: "test-project",
location: "us-central1",
googleAuthOptions: {
credentials: {
client_email: "stagehand@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\ntest\n-----END PRIVATE KEY-----\n",
},
projectId: "test-project",
universeDomain: "googleapis.com",
},
});
});
});

View File

@ -3,17 +3,30 @@ import * as Stagehand from "@browserbasehq/stagehand";
describe("LLM and Agents public API types", () => {
describe("ModelConfiguration", () => {
it("accepts Vertex headers in model config", () => {
const googleConfig = {
modelName: "google/gemini-3-flash-preview",
project: "test-project",
location: "global",
it("accepts Vertex provider options in model config", () => {
const vertexConfig = {
provider: "vertex",
modelName: "vertex/gemini-3-flash-preview",
headers: {
"X-Goog-Priority": "high",
},
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-project",
location: "global",
},
},
} satisfies Stagehand.ModelConfiguration;
void googleConfig;
void vertexConfig;
});
});
@ -44,6 +57,7 @@ describe("LLM and Agents public API types", () => {
"openai/gpt-5.5",
"anthropic/claude-opus-4-5-20251101",
"anthropic/claude-opus-4-6",
"anthropic/claude-opus-4-8",
"anthropic/claude-sonnet-4-6",
"anthropic/claude-haiku-4-5",
"anthropic/claude-haiku-4-5-20251001",
@ -61,6 +75,12 @@ describe("LLM and Agents public API types", () => {
>();
void expectedModels; // Mark as used to satisfy ESLint
});
it("includes Claude Opus 4.8 at runtime", () => {
expect(Stagehand.AVAILABLE_CUA_MODELS).toContain(
"anthropic/claude-opus-4-8",
);
});
});
describe("AgentProvider", () => {
@ -274,6 +294,12 @@ describe("LLM and Agents public API types", () => {
typeof Stagehand.modelToAgentProviderMap
>().toExtend<ExpectedModelToAgentProviderMap>();
});
it("routes Claude Opus 4.8 to Anthropic", () => {
expect(Stagehand.modelToAgentProviderMap["claude-opus-4-8"]).toBe(
"anthropic",
);
});
});
describe("Response", () => {

View File

@ -3,56 +3,45 @@ import { describe, expect, it } from "vitest";
import { inferToolOutput } from "../../lib/v3/agent/utils/toolOutputEvidence.js";
describe("inferToolOutput", () => {
it("preserves raw results while normalizing top-level failure status", () => {
const result = { success: false };
it.each<[string, unknown, boolean, string | undefined]>([
[
"preserves raw results while normalizing top-level failure status",
{ success: false },
false,
undefined,
],
[
"normalizes one-level AI SDK output wrappers",
{
toolCallId: "call-1",
output: { success: false, error: { message: "not found" } },
},
false,
'{"message":"not found"}',
],
[
"handles isError and non-string errors",
{ isError: true, error: new Error("bad input") },
false,
"bad input",
],
[
"normalizes non-json error values",
{ error: Symbol("bad input") },
false,
"Symbol(bad input)",
],
[
"does not recursively treat page data as tool status",
{ data: { success: false, error: "page field" } },
true,
undefined,
],
])("%s", (_, result, ok, error) => {
expect(inferToolOutput(result)).toEqual({
ok: false,
ok,
result,
error: undefined,
});
});
it("normalizes one-level AI SDK output wrappers", () => {
const result = {
toolCallId: "call-1",
output: { success: false, error: { message: "not found" } },
};
expect(inferToolOutput(result)).toEqual({
ok: false,
result,
error: '{"message":"not found"}',
});
});
it("handles isError and non-string errors", () => {
const result = { isError: true, error: new Error("bad input") };
expect(inferToolOutput(result)).toEqual({
ok: false,
result,
error: "bad input",
});
});
it("normalizes non-json error values", () => {
const result = { error: Symbol("bad input") };
expect(inferToolOutput(result)).toEqual({
ok: false,
result,
error: "Symbol(bad input)",
});
});
it("does not recursively treat page data as tool status", () => {
const result = { data: { success: false, error: "page field" } };
expect(inferToolOutput(result)).toEqual({
ok: true,
result,
error: undefined,
error,
});
});
});

View File

@ -197,10 +197,6 @@ function makeEmptyTrajectory(taskSpec: TaskSpec): Trajectory {
input_tokens: 0,
output_tokens: 0,
},
timing: {
startedAt: new Date(0).toISOString(),
endedAt: new Date(0).toISOString(),
},
};
}
@ -216,7 +212,6 @@ function makeTrajectory(
...makeEmptyTrajectory(taskSpec),
steps: [
{
index: 0,
actionName: "act",
actionArgs: {},
reasoning: "I completed the task.",
@ -228,8 +223,6 @@ function makeTrajectory(
ok: true,
result: options.toolResult ?? "done",
},
startedAt: new Date(0).toISOString(),
finishedAt: new Date(0).toISOString(),
},
],
finalAnswer: options.finalAnswer,

View File

@ -1,20 +1,63 @@
import { describe, expect, it } from "vitest";
import { buildAgentEvidenceFromStepFinished } from "../../lib/v3/verifier/evidenceNormalization.js";
import {
buildAgentEvidenceFromStepFinished,
REDACTED_INLINE_IMAGE,
} from "../../lib/v3/verifier/evidenceNormalization.js";
describe("buildAgentEvidenceFromStepFinished", () => {
it("captures primitive tool results as text evidence", () => {
const evidence = buildAgentEvidenceFromStepFinished({
type: "step_finished",
stepIndex: 0,
actionName: "check",
actionArgs: {},
reasoning: "",
toolOutput: { ok: true, result: false },
startedAt: new Date(0).toISOString(),
finishedAt: new Date(1).toISOString(),
});
expect(evidence.modalities).toEqual([{ type: "text", content: "false" }]);
});
it("lifts inline screenshot payloads into image evidence and redacts JSON", () => {
const inlineScreenshot =
Buffer.from("inline screenshot").toString("base64");
const evidence = buildAgentEvidenceFromStepFinished({
type: "step_finished",
actionName: "click",
actionArgs: { describe: "Open fare details" },
reasoning: "",
toolOutput: {
ok: true,
result: {
output: {
success: true,
describe: "Open fare details",
screenshotBase64: inlineScreenshot,
},
},
},
});
const [imageModality, jsonModality] = evidence.modalities;
expect(JSON.stringify(evidence)).not.toContain(inlineScreenshot);
expect(jsonModality).toMatchObject({
type: "json",
content: {
output: {
screenshotBase64: REDACTED_INLINE_IMAGE,
},
},
});
expect(imageModality).toMatchObject({
type: "image",
mediaType: "image/png",
});
if (imageModality?.type === "image") {
expect(imageModality.bytes).toEqual(
Buffer.from(inlineScreenshot, "base64"),
);
}
});
});

View File

@ -79,13 +79,8 @@ describe("verifier trajectory utilities", () => {
task: { id: "task", instruction: "Do the task" },
status: "complete",
usage: { input_tokens: 0, output_tokens: 0 },
timing: {
startedAt: new Date(0).toISOString(),
endedAt: new Date(0).toISOString(),
},
steps: [
{
index: 0,
actionName: "act",
actionArgs: {},
reasoning: "",
@ -100,8 +95,6 @@ describe("verifier trajectory utilities", () => {
},
probeEvidence: { screenshotPath: "screenshot_1.png" },
toolOutput: { ok: true, result: null },
startedAt: new Date(0).toISOString(),
finishedAt: new Date(0).toISOString(),
},
],
finalObservation: {
@ -131,13 +124,8 @@ describe("verifier trajectory utilities", () => {
task: { id: "task", instruction: "Do the task" },
status: "complete",
usage: { input_tokens: 0, output_tokens: 0 },
timing: {
startedAt: new Date(0).toISOString(),
endedAt: new Date(0).toISOString(),
},
steps: [
{
index: 0,
actionName: "act",
actionArgs: {},
reasoning: "",
@ -152,8 +140,6 @@ describe("verifier trajectory utilities", () => {
},
probeEvidence: {},
toolOutput: { ok: true, result: null },
startedAt: new Date(0).toISOString(),
finishedAt: new Date(0).toISOString(),
},
],
}),
@ -177,17 +163,12 @@ describe("verifier trajectory utilities", () => {
task: { id: "task", instruction: "Do the task" },
status: "complete",
usage: { input_tokens: 0, output_tokens: 0 },
timing: {
startedAt: new Date(0).toISOString(),
endedAt: new Date(0).toISOString(),
},
finalObservation: {
url: "https://example.com/done",
screenshot: Buffer.from("final screenshot"),
},
steps: [
{
index: 0,
actionName: "click",
actionArgs: {},
reasoning: "",
@ -214,8 +195,6 @@ describe("verifier trajectory utilities", () => {
},
},
},
startedAt: new Date(0).toISOString(),
finishedAt: new Date(0).toISOString(),
},
],
});
@ -247,21 +226,14 @@ describe("verifier trajectory utilities", () => {
task: { id: "task", instruction: "Do the task" },
status: "complete",
usage: { input_tokens: 0, output_tokens: 0 },
timing: {
startedAt: new Date(0).toISOString(),
endedAt: new Date(0).toISOString(),
},
steps: [
{
index: 0,
actionName: "act",
actionArgs: {},
reasoning: "",
agentEvidence: { modalities: [] },
probeEvidence: { screenshotPath: "../../../etc/passwd" },
toolOutput: { ok: true, result: null },
startedAt: new Date(0).toISOString(),
finishedAt: new Date(0).toISOString(),
},
],
}),

View File

@ -0,0 +1,202 @@
// Re-label generated OpenAPI schema fields on SDK-flavored API reference pages.
//
// Mintlify renders these pages from the OpenAPI wire schema, so body fields use
// camelCase even when the selected SDK accepts snake_case parameters. Stainless
// already emits language-correct code samples; this small client-side patch keeps
// the visible parameter labels aligned with the Python and Ruby SDK snippets.
(function () {
const SNAKE_CASE_PARAM_LABELS = {
acceptDownloads: "accept_downloads",
actTimeoutMs: "act_timeout_ms",
actionDescription: "action_description",
actionId: "action_id",
advancedStealth: "advanced_stealth",
agentConfig: "agent_config",
apiKey: "api_key",
backendNodeId: "backend_node_id",
baseURL: "base_url",
blockAds: "block_ads",
browserSettings: "browser_settings",
browserbaseAPIKey: "browserbase_api_key",
browserbaseProjectID: "browserbase_project_id",
browserbaseSessionCreateParams: "browserbase_session_create_params",
browserbaseSessionID: "browserbase_session_id",
cacheEntry: "cache_entry",
cacheKey: "cache_key",
captchaImageSelector: "captcha_image_selector",
captchaInputSelector: "captcha_input_selector",
cdpHeaders: "cdp_headers",
cdpUrl: "cdp_url",
cdpURL: "cdp_url",
chromiumSandbox: "chromium_sandbox",
clientLanguage: "client_language",
connectTimeoutMs: "connect_timeout_ms",
deviceScaleFactor: "device_scale_factor",
domSettleTimeoutMs: "dom_settle_timeout_ms",
domainPattern: "domain_pattern",
downloadsPath: "downloads_path",
endTime: "end_time",
executablePath: "executable_path",
executeOptions: "execute_options",
executionModel: "execution_model",
extensionId: "extension_id",
extensionID: "extension_id",
frameId: "frame_id",
frameID: "frame_id",
googleAuthOptions: "google_auth_options",
hasTouch: "has_touch",
highlightCursor: "highlight_cursor",
httpVersion: "http_version",
ignoreDefaultArgs: "ignore_default_args",
ignoreHTTPSErrors: "ignore_https_errors",
ignoreSelectors: "ignore_selectors",
inputTokens: "input_tokens",
keepAlive: "keep_alive",
launchOptions: "launch_options",
logSession: "log_session",
maxHeight: "max_height",
maxSteps: "max_steps",
maxWidth: "max_width",
minHeight: "min_height",
minWidth: "min_width",
modelAPIKey: "model_api_key",
modelName: "model_name",
operatingSystems: "operating_systems",
outputTokens: "output_tokens",
pageText: "page_text",
pageUrl: "page_url",
preserveUserDataDir: "preserve_user_data_dir",
projectId: "project_id",
projectID: "project_id",
recordSession: "record_session",
rememberMe: "remember_me",
selfHeal: "self_heal",
sessionId: "session_id",
shouldCache: "should_cache",
solveCaptchas: "solve_captchas",
streamResponse: "stream_response",
systemPrompt: "system_prompt",
taskCompleted: "task_completed",
timeMs: "time_ms",
tokenUsage: "token_usage",
toolTimeout: "tool_timeout",
universeDomain: "universe_domain",
useSearch: "use_search",
userDataDir: "user_data_dir",
userMetadata: "user_metadata",
waitForCaptchaSolves: "wait_for_captcha_solves",
waitUntil: "wait_until",
xStreamResponse: "x_stream_response",
};
const SDK_PARAM_LABELS = {
python: SNAKE_CASE_PARAM_LABELS,
ruby: SNAKE_CASE_PARAM_LABELS,
};
const CONTENT_SCOPE_SELECTOR = [
"#content-area",
"#api-playground-input",
"api-section",
"field",
].join(",");
const SKIP_SELECTOR = [
"code",
"input",
"pre",
"script",
"style",
"textarea",
'[contenteditable="true"]',
].join(",");
let rewriteTimer = null;
let lastPath = window.location.pathname;
function currentSdkLanguage() {
const match = window.location.pathname.match(
/^\/v3\/api-reference\/([^/]+)(?:\/|$)/,
);
return match ? match[1] : null;
}
function escapeRegExp(value) {
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
function buildPattern(labels) {
return new RegExp(
`\\b(${Object.keys(labels).map(escapeRegExp).join("|")})\\b`,
"g",
);
}
function shouldRewriteTextNode(node, pattern) {
if (!node.nodeValue) return false;
const parent = node.parentElement;
if (!parent) return false;
if (parent.closest(SKIP_SELECTOR)) return false;
if (!parent.closest(CONTENT_SCOPE_SELECTOR)) return false;
pattern.lastIndex = 0;
return pattern.test(node.nodeValue);
}
function rewriteLabels(root) {
const labels = SDK_PARAM_LABELS[currentSdkLanguage()];
if (!labels) return;
const pattern = buildPattern(labels);
const walker = document.createTreeWalker(
root || document.body,
NodeFilter.SHOW_TEXT,
{
acceptNode(node) {
return shouldRewriteTextNode(node, pattern)
? NodeFilter.FILTER_ACCEPT
: NodeFilter.FILTER_REJECT;
},
},
);
const nodes = [];
while (walker.nextNode()) nodes.push(walker.currentNode);
for (const node of nodes) {
pattern.lastIndex = 0;
node.nodeValue = node.nodeValue.replace(pattern, (name) => labels[name]);
}
}
function scheduleRewrite(root) {
if (rewriteTimer) window.clearTimeout(rewriteTimer);
rewriteTimer = window.setTimeout(() => {
rewriteTimer = null;
window.requestAnimationFrame(() => rewriteLabels(root || document.body));
}, 50);
}
scheduleRewrite();
const observer = new MutationObserver((mutations) => {
for (const mutation of mutations) {
for (const node of mutation.addedNodes) {
if (node.nodeType === Node.ELEMENT_NODE) {
scheduleRewrite(node);
return;
}
}
}
});
observer.observe(document.body, { childList: true, subtree: true });
window.setInterval(() => {
if (window.location.pathname !== lastPath) {
lastPath = window.location.pathname;
scheduleRewrite();
}
}, 250);
})();

View File

@ -669,6 +669,7 @@ const agent = stagehand.agent({
| Anthropic | `anthropic/claude-sonnet-4-5-20250929` |
| Anthropic | `anthropic/claude-opus-4-5-20251101` |
| Anthropic | `anthropic/claude-opus-4-6` |
| Anthropic | `anthropic/claude-opus-4-8` |
| Google | `google/gemini-2.5-computer-use-preview-10-2025` |
| Google | `google/gemini-3-flash-preview` |
| Google | `google/gemini-3-pro-preview` |

View File

@ -72,6 +72,7 @@ interface AgentInstance {
- `"anthropic/claude-sonnet-4-5-20250929"`
- `"anthropic/claude-opus-4-5-20251101"`
- `"anthropic/claude-opus-4-6"`
- `"anthropic/claude-opus-4-8"`
- `"google/gemini-2.5-computer-use-preview-10-2025"`
- `"google/gemini-3-flash-preview"`
- `"google/gemini-3-pro-preview"`
@ -867,4 +868,4 @@ The following errors may be thrown by the `agent()` method:
- **StagehandDefaultError** - General execution error with detailed message
- **AgentAbortError** - Thrown when agent execution is cancelled via an `AbortSignal`
- **StreamingCallbacksInNonStreamingModeError** - Thrown when streaming-only callbacks (`onChunk`, `onFinish`, `onError`, `onAbort`) are used without `stream: true`
- **ExperimentalNotConfiguredError** - Thrown when experimental features (callbacks, signal, messages, streaming) are used without `experimental: true` in Stagehand constructor
- **ExperimentalNotConfiguredError** - Thrown when experimental features (callbacks, signal, messages, streaming) are used without `experimental: true` in Stagehand constructor

View File

@ -23,18 +23,6 @@ import type {
EvaluationResult,
} from "@browserbasehq/stagehand";
interface PartialStep {
index: number;
actionName: string;
actionArgs: Record<string, unknown>;
reasoning: string;
agentEvidence: AgentEvidence;
probeEvidence: ProbeEvidence;
toolOutput: { ok: boolean; result: unknown; error?: string };
startedAt: string;
finishedAt: string;
}
export interface TrajectoryRecorderOptions {
taskSpec: TaskSpec;
/**
@ -69,69 +57,71 @@ export class TrajectoryRecorder {
private readonly outputDir: string;
private readonly persistEnabled: boolean;
// Events can arrive out-of-order across step indices; same-step events all
// fire in one microtask.
private readonly partialSteps = new Map<number, Partial<PartialStep>>();
// Steps are appended in arrival order on each step_finished event.
private readonly steps: TrajectoryStep[] = [];
// The most recent agent-role screenshot. It applies to every step_finished
// until a newer agent-role screenshot replaces it — a CUA provider can pick
// multiple actions from one screenshot, so each of those steps must carry
// that same tier-1 frame. (It is NOT cleared on consume; it is only replaced
// by a newer screenshot, or wiped on cancel().)
private latestAgentScreenshot?: Buffer;
// The most recent probe-role screenshot waits for the matching step_observed.
private pendingProbeScreenshot?: Buffer;
// Steps that haven't yet had a probe attached. The next step_observed fans
// out to all of them (one probe per agent turn, N tool calls per turn).
private stepsAwaitingProbe: number[] = [];
private finalAnswerEvent?: AgentFinalAnswerEvent;
private finalObservation?: ProbeEvidence;
private startedAt = "";
private endedAt = "";
private onScreenshot(e: AgentScreenshotEvidenceEvent): void {
const partial = this.ensurePartial(e.stepIndex);
// Default to probe when the emit site doesn't tag a role: matches
// v3AgentHandler's post-step screenshot. For CUA the pre-action shot is
// NOT a probe — emitCuaActionStep fills that role post-action.
const role = e.evidenceRole ?? "probe";
if (role === "probe") {
const probe: ProbeEvidence = { ...(partial.probeEvidence ?? {}) };
probe.screenshot = e.screenshot;
probe.url = e.url;
partial.probeEvidence = probe;
} else if (!partial.probeEvidence?.url) {
// Capture URL even for tier-1-only events; a later post-action URL
// can still overwrite it.
partial.probeEvidence = {
...(partial.probeEvidence ?? {}),
url: e.url,
};
}
if (role === "agent") {
partial.agentEvidence = mergeAgentEvidence(partial.agentEvidence, {
modalities: [
{ type: "image", bytes: e.screenshot, mediaType: "image/png" },
],
});
if (e.evidenceRole === "agent") {
this.latestAgentScreenshot = e.screenshot;
} else {
this.pendingProbeScreenshot = e.screenshot;
}
}
private onStepFinished(e: AgentStepFinishedEvent): void {
const partial = this.ensurePartial(e.stepIndex);
partial.actionName = e.actionName;
partial.actionArgs = e.actionArgs;
partial.reasoning = e.reasoning;
partial.toolOutput = {
...e.toolOutput,
result: redactInlineImagePayloads(e.toolOutput.result, e.actionName),
};
partial.startedAt = e.startedAt ?? e.finishedAt;
partial.finishedAt = e.finishedAt;
partial.agentEvidence = mergeAgentEvidence(
partial.agentEvidence,
const modalities: AgentEvidence["modalities"] = [];
if (this.latestAgentScreenshot) {
modalities.push({
type: "image",
bytes: this.latestAgentScreenshot,
mediaType: "image/png",
});
}
const merged = mergeAgentEvidence(
{ modalities },
buildAgentEvidenceFromStepFinished(e),
);
// Intentionally not cleared here: the same agent screenshot applies to
// every step in a batched CUA turn. It's replaced when a newer agent
// screenshot arrives (onScreenshot) or wiped on cancel().
this.stepsAwaitingProbe.push(this.steps.length);
this.steps.push({
actionName: e.actionName,
actionArgs: e.actionArgs,
reasoning: e.reasoning,
agentEvidence: merged,
probeEvidence: {},
toolOutput: {
...e.toolOutput,
result: redactInlineImagePayloads(e.toolOutput.result, e.actionName),
},
});
}
private onStepObserved(e: AgentStepObservedEvent): void {
const partial = this.ensurePartial(e.stepIndex);
const probe: ProbeEvidence = { ...(partial.probeEvidence ?? {}) };
probe.url = e.url;
const probe: ProbeEvidence = { url: e.url };
if (this.pendingProbeScreenshot)
probe.screenshot = this.pendingProbeScreenshot;
if (e.ariaTree !== undefined) probe.ariaTree = e.ariaTree;
if (e.scroll !== undefined) probe.scroll = e.scroll;
partial.probeEvidence = probe;
for (const idx of this.stepsAwaitingProbe) {
this.steps[idx].probeEvidence = probe;
}
this.stepsAwaitingProbe = [];
this.pendingProbeScreenshot = undefined;
}
private onFinalAnswer(e: AgentFinalAnswerEvent): void {
@ -145,9 +135,6 @@ export class TrajectoryRecorder {
...(e.observation.ariaTree !== undefined
? { ariaTree: e.observation.ariaTree }
: {}),
...(e.observation.scroll !== undefined
? { scroll: e.observation.scroll }
: {}),
};
}
}
@ -160,15 +147,8 @@ export class TrajectoryRecorder {
this.persistEnabled = shouldPersistTrajectory(opts.persist);
}
/** Mark the beginning of collection. Call once before agent.execute(). */
start(): void {
if (this.startedAt) return;
this.startedAt = new Date().toISOString();
}
/** Ingest an evidence callback event from agent.execute(). */
record(event: AgentEvidenceEvent): void {
if (!this.startedAt) this.start();
switch (event.type) {
case "screenshot":
this.onScreenshot(event);
@ -190,20 +170,15 @@ export class TrajectoryRecorder {
* write the on-disk layout. Idempotent.
*/
async finish(opts: TrajectoryFinishOptions): Promise<Trajectory> {
if (!this.startedAt) this.start();
this.endedAt = new Date().toISOString();
const steps = this.assembleSteps();
const trajectory: Trajectory = {
task: this.taskSpec,
steps,
steps: this.steps,
finalAnswer: opts.finalAnswer ?? this.finalAnswerEvent?.message,
...(this.finalObservation
? { finalObservation: this.finalObservation }
: {}),
status: opts.status,
usage: { ...ZERO_USAGE, ...(opts.usage ?? {}) },
timing: { startedAt: this.startedAt, endedAt: this.endedAt },
};
if (this.persistEnabled) {
@ -215,7 +190,10 @@ export class TrajectoryRecorder {
/** Throw away in-memory state without writing to disk. Used on early abort. */
cancel(): void {
this.partialSteps.clear();
this.steps.length = 0;
this.latestAgentScreenshot = undefined;
this.pendingProbeScreenshot = undefined;
this.stepsAwaitingProbe = [];
this.finalAnswerEvent = undefined;
this.finalObservation = undefined;
}
@ -262,44 +240,4 @@ export class TrajectoryRecorder {
JSON.stringify({ ...taskData, result }, null, 2),
);
}
private ensurePartial(stepIndex: number): Partial<PartialStep> {
let p = this.partialSteps.get(stepIndex);
if (!p) {
p = { index: stepIndex };
this.partialSteps.set(stepIndex, p);
}
return p;
}
private assembleSteps(): TrajectoryStep[] {
const out: TrajectoryStep[] = [];
const indices = [...this.partialSteps.keys()].sort((a, b) => a - b);
for (const i of indices) {
const p = this.partialSteps.get(i)!;
if (
p.actionName === undefined ||
p.toolOutput === undefined ||
p.startedAt === undefined ||
p.finishedAt === undefined
) {
// Provider-only screenshot refreshes are transport evidence for the
// next CUA action. If no action arrives for this index, there is no
// completed trajectory step to persist.
continue;
}
out.push({
index: i,
actionName: p.actionName,
actionArgs: p.actionArgs ?? {},
reasoning: p.reasoning ?? "",
agentEvidence: p.agentEvidence ?? { modalities: [] },
probeEvidence: p.probeEvidence ?? {},
toolOutput: p.toolOutput,
startedAt: p.startedAt,
finishedAt: p.finishedAt,
});
}
return out;
}
}

View File

@ -30,44 +30,81 @@ function makeTaskSpec(): TaskSpec {
id: "recorder-task",
instruction: "Compare economy and business fares.",
initUrl: "https://example.com",
precomputedRubric: {
items: [
{
criterion: "Report fare delta",
description: "Report the difference between two fares.",
maxPoints: 1,
},
],
},
};
}
function recordSimpleStep(recorder: TrajectoryRecorder, screenshot: Buffer) {
recorder.record({
type: "screenshot",
screenshot,
url: "https://example.com/search",
evidenceRole: "agent",
});
recorder.record({
type: "step_finished",
actionName: "act",
actionArgs: { instruction: "Search fares" },
reasoning: "Search for fares.",
toolOutput: { ok: true, result: "done" },
});
recorder.record({
type: "screenshot",
screenshot,
url: "https://example.com/search",
evidenceRole: "probe",
});
recorder.record({
type: "step_observed",
url: "https://example.com/search",
});
}
function recordFinalAnswer(
recorder: TrajectoryRecorder,
opts: { message: string; screenshot: Buffer; ariaTree?: string },
): void {
recorder.record({
type: "final_answer",
message: opts.message,
observation: {
url: "https://example.com/complete",
screenshot: opts.screenshot,
...(opts.ariaTree !== undefined ? { ariaTree: opts.ariaTree } : {}),
},
});
}
describe("TrajectoryRecorder", () => {
it("assembles trajectory evidence from callback events", async () => {
it("assembles ordered callback events into trajectory steps", async () => {
const recorder = new TrajectoryRecorder({
taskSpec: makeTaskSpec(),
persist: false,
});
const screenshot = Buffer.from("screen-1");
const staleScreenshot = Buffer.from("stale-screen");
const probeScreenshot = Buffer.from("probe-screen");
recorder.start();
recorder.record({
type: "screenshot",
stepIndex: 0,
screenshot: staleScreenshot,
url: "https://example.com/stale",
evidenceRole: "agent",
});
recorder.record({
type: "screenshot",
screenshot,
url: "https://example.com/search",
evidenceRole: "agent",
});
recorder.record({
type: "screenshot",
stepIndex: 0,
screenshot,
url: "https://example.com/search",
evidenceRole: "probe",
type: "step_finished",
actionName: "click",
actionArgs: { describe: "Open fares" },
reasoning: "Open fare details.",
toolOutput: { ok: true, result: "opened" },
});
recorder.record({
type: "step_finished",
stepIndex: 0,
actionName: "extract",
actionArgs: { instruction: "Read fares" },
reasoning: "Read visible fare cells.",
@ -75,23 +112,22 @@ describe("TrajectoryRecorder", () => {
ok: true,
result: { economy: "$100", business: "$250" },
},
startedAt: new Date(0).toISOString(),
finishedAt: new Date(0).toISOString(),
});
recorder.record({
type: "screenshot",
screenshot: probeScreenshot,
url: "https://example.com/search",
evidenceRole: "probe",
});
recorder.record({
type: "step_observed",
stepIndex: 0,
url: "https://example.com/search",
ariaTree: "RootWebArea\nStaticText: Economy $100",
});
recorder.record({
type: "final_answer",
recordFinalAnswer(recorder, {
message: "Business is $150 more than economy.",
observation: {
url: "https://example.com/checkout",
screenshot: Buffer.from("final-screen"),
ariaTree: "RootWebArea\nStaticText: Complete",
},
screenshot: Buffer.from("final-screen"),
ariaTree: "RootWebArea\nStaticText: Complete",
});
const trajectory = await recorder.finish({
@ -99,33 +135,40 @@ describe("TrajectoryRecorder", () => {
usage: { input_tokens: 10, output_tokens: 5 },
});
expect(trajectory.steps).toHaveLength(1);
expect(trajectory.steps).toHaveLength(2);
expect(trajectory.steps[0]).toMatchObject({
index: 0,
actionName: "extract",
actionArgs: { instruction: "Read fares" },
reasoning: "Read visible fare cells.",
toolOutput: {
ok: true,
result: { economy: "$100", business: "$250" },
},
actionName: "click",
probeEvidence: {
url: "https://example.com/search",
ariaTree: "RootWebArea\nStaticText: Economy $100",
},
});
expect(trajectory.steps[0].probeEvidence.screenshot).toEqual(screenshot);
expect(trajectory.steps[0].startedAt).toBe(new Date(0).toISOString());
expect(trajectory.steps[1]).toMatchObject({
actionName: "extract",
toolOutput: { ok: true, result: { economy: "$100", business: "$250" } },
});
expect(trajectory.steps[0].probeEvidence.screenshot).toEqual(
probeScreenshot,
);
expect(trajectory.steps[1].probeEvidence.screenshot).toEqual(
probeScreenshot,
);
expect(trajectory.steps[0].agentEvidence.modalities).toEqual(
expect.arrayContaining([
{ type: "image", bytes: screenshot, mediaType: "image/png" },
{ type: "text", content: "Read visible fare cells." },
{ type: "json", content: { economy: "$100", business: "$250" } },
{ type: "text", content: "Open fare details." },
]),
);
// Both actions were chosen from the same agent screenshot (one screenshot,
// two step_finished), so the second step must carry that frame too.
expect(trajectory.steps[1].agentEvidence.modalities).toEqual(
expect.arrayContaining([
{ type: "image", bytes: screenshot, mediaType: "image/png" },
]),
);
expect(trajectory.finalAnswer).toBe("Business is $150 more than economy.");
expect(trajectory.finalObservation).toMatchObject({
url: "https://example.com/checkout",
url: "https://example.com/complete",
ariaTree: "RootWebArea\nStaticText: Complete",
});
expect(trajectory.finalObservation?.screenshot).toEqual(
@ -143,43 +186,10 @@ describe("TrajectoryRecorder", () => {
});
const screenshot = Buffer.from("screen-1");
recorder.start();
recorder.record({
type: "screenshot",
stepIndex: 0,
screenshot,
url: "https://example.com/search",
evidenceRole: "agent",
});
recorder.record({
type: "screenshot",
stepIndex: 0,
screenshot,
url: "https://example.com/search",
evidenceRole: "probe",
});
recorder.record({
type: "step_finished",
stepIndex: 0,
actionName: "act",
actionArgs: { instruction: "Search fares" },
reasoning: "Search for fares.",
toolOutput: { ok: true, result: "done" },
startedAt: new Date(0).toISOString(),
finishedAt: new Date(0).toISOString(),
});
recorder.record({
type: "step_observed",
stepIndex: 0,
url: "https://example.com/search",
});
recorder.record({
type: "final_answer",
recordSimpleStep(recorder, screenshot);
recordFinalAnswer(recorder, {
message: "Complete.",
observation: {
url: "https://example.com/complete",
screenshot: Buffer.from("final-screen"),
},
screenshot: Buffer.from("final-screen"),
});
await recorder.finish({ status: "complete" });
@ -195,7 +205,6 @@ describe("TrajectoryRecorder", () => {
"scores",
"screenshots",
"task_data.json",
"times.json",
"trajectory.json",
]),
);
@ -235,93 +244,4 @@ describe("TrajectoryRecorder", () => {
explanation: "The task was completed.",
});
});
it("normalizes missing step startedAt to finishedAt", async () => {
const recorder = new TrajectoryRecorder({
taskSpec: makeTaskSpec(),
persist: false,
});
const finishedAt = new Date(1).toISOString();
recorder.record({
type: "step_finished",
stepIndex: 0,
actionName: "extract",
actionArgs: { instruction: "Read fares" },
reasoning: "",
toolOutput: { ok: true, result: false },
finishedAt,
});
const trajectory = await recorder.finish({ status: "complete" });
expect(trajectory.steps[0].startedAt).toBe(finishedAt);
expect(trajectory.steps[0].finishedAt).toBe(finishedAt);
});
it("lifts inline screenshot payloads into image evidence and redacts JSON", async () => {
const inlineScreenshot =
Buffer.from("inline screenshot").toString("base64");
const recorder = new TrajectoryRecorder({
taskSpec: makeTaskSpec(),
persist: false,
});
recorder.record({
type: "step_finished",
stepIndex: 0,
actionName: "click",
actionArgs: { describe: "Open fare details" },
reasoning: "Click the fare details button.",
toolOutput: {
ok: true,
result: {
output: {
success: true,
describe: "Open fare details",
screenshotBase64: inlineScreenshot,
},
},
},
startedAt: new Date(0).toISOString(),
finishedAt: new Date(0).toISOString(),
});
const trajectory = await recorder.finish({ status: "complete" });
const step = trajectory.steps[0];
const rawTrajectory = JSON.stringify(trajectory);
const imageModalities = step.agentEvidence.modalities.filter(
(m) => m.type === "image",
);
const jsonModality = step.agentEvidence.modalities.find(
(m) => m.type === "json",
);
expect(rawTrajectory).not.toContain(inlineScreenshot);
expect(step.toolOutput.result).toMatchObject({
output: {
success: true,
describe: "Open fare details",
screenshotBase64: "[redacted inline image payload]",
},
});
expect(jsonModality).toMatchObject({
type: "json",
content: {
output: {
screenshotBase64: "[redacted inline image payload]",
},
},
});
expect(imageModalities).toHaveLength(1);
expect(imageModalities[0]).toMatchObject({
type: "image",
mediaType: "image/png",
});
if (imageModalities[0].type === "image") {
expect(imageModalities[0].bytes).toEqual(
Buffer.from(inlineScreenshot, "base64"),
);
}
});
});

View File

@ -202,9 +202,14 @@ components:
- private_key
- client_email
additionalProperties: false
GoogleAuthOptions:
GoogleServiceAccountAuth:
type: object
properties:
type:
description: Use inline Google Cloud service account credentials for provider
authentication
type: string
const: googleServiceAccount
credentials:
description: Google Cloud service account credentials
$ref: "#/components/schemas/GoogleServiceAccountCredentials"
@ -221,21 +226,51 @@ components:
universeDomain:
description: Google Cloud universe domain
type: string
required:
- type
- credentials
additionalProperties: false
ModelConfigObject:
ModelAuth:
oneOf:
- $ref: "#/components/schemas/GoogleServiceAccountAuth"
VertexProviderOptions:
type: object
properties:
provider:
description: AI provider for the model (or provide a baseURL endpoint instead)
example: openai
project:
description: Google Cloud project ID for Vertex AI models
example: my-gcp-project
type: string
enum:
- openai
- anthropic
- google
- microsoft
- bedrock
- vertex
location:
description: Google Cloud location for Vertex AI models
example: us-central1
type: string
baseURL:
description: Base URL for the Vertex AI provider
type: string
format: uri
headers:
description: Custom headers sent with every request to the Vertex AI provider
type: object
propertyNames:
type: string
additionalProperties:
type: string
required:
- project
- location
additionalProperties: false
ModelProviderOptions:
type: object
properties:
vertex:
description: Vertex AI provider-specific settings
$ref: "#/components/schemas/VertexProviderOptions"
required:
- vertex
additionalProperties: false
GenericModelConfigObject:
type: object
properties:
modelName:
description: Model name string with provider prefix (e.g., 'openai/gpt-5-nano')
example: openai/gpt-5.4-mini
@ -256,19 +291,62 @@ components:
type: string
additionalProperties:
type: string
project:
description: Google Cloud project ID for Vertex AI models
example: my-gcp-project
provider:
description: AI provider for the model (or provide a baseURL endpoint instead)
example: openai
type: string
location:
description: Google Cloud location for Vertex AI models
example: us-central1
type: string
googleAuthOptions:
description: google-auth-library options used to authenticate Vertex AI models
$ref: "#/components/schemas/GoogleAuthOptions"
enum:
- openai
- anthropic
- google
- microsoft
- bedrock
required:
- modelName
additionalProperties: false
VertexModelConfigObject:
type: object
properties:
modelName:
description: Model name string with provider prefix (e.g., 'openai/gpt-5-nano')
example: openai/gpt-5.4-mini
type: string
apiKey:
description: API key for the model provider
example: sk-some-openai-api-key
type: string
baseURL:
description: Base URL for the model provider
example: https://api.openai.com/v1
type: string
format: uri
headers:
description: Custom headers sent with every request to the model provider
type: object
propertyNames:
type: string
additionalProperties:
type: string
provider:
description: Vertex AI model provider
type: string
const: vertex
auth:
description: Vertex provider authentication configuration
$ref: "#/components/schemas/ModelAuth"
providerOptions:
description: Vertex provider-specific model configuration
$ref: "#/components/schemas/ModelProviderOptions"
required:
- modelName
- provider
- auth
- providerOptions
additionalProperties: false
ModelConfigObject:
anyOf:
- $ref: "#/components/schemas/VertexModelConfigObject"
- $ref: "#/components/schemas/GenericModelConfigObject"
ModelConfig:
$ref: "#/components/schemas/ModelConfigObject"
Action:
@ -1319,56 +1397,6 @@ components:
acceptDownloads:
type: boolean
additionalProperties: false
ModelConfigObjectOutput:
type: object
properties:
provider:
description: AI provider for the model (or provide a baseURL endpoint instead)
example: openai
type: string
enum:
- openai
- anthropic
- google
- microsoft
- bedrock
- vertex
modelName:
description: Model name string with provider prefix (e.g., 'openai/gpt-5-nano')
example: openai/gpt-5.4-mini
type: string
apiKey:
description: API key for the model provider
example: sk-some-openai-api-key
type: string
baseURL:
description: Base URL for the model provider
example: https://api.openai.com/v1
type: string
format: uri
headers:
description: Custom headers sent with every request to the model provider
type: object
propertyNames:
type: string
additionalProperties:
type: string
project:
description: Google Cloud project ID for Vertex AI models
example: my-gcp-project
type: string
location:
description: Google Cloud location for Vertex AI models
example: us-central1
type: string
googleAuthOptions:
description: google-auth-library options used to authenticate Vertex AI models
$ref: "#/components/schemas/GoogleAuthOptions"
required:
- modelName
additionalProperties: false
ModelConfigOutput:
$ref: "#/components/schemas/ModelConfigObjectOutput"
ActionOutput:
description: Action object returned by observe and used by act
type: object
@ -1633,7 +1661,7 @@ components:
description: Model configuration object or model name string (e.g.,
'openai/gpt-5-nano')
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
variables:
description: Variables to substitute in the action instruction. Accepts flat
@ -1693,7 +1721,7 @@ components:
description: Model configuration object or model name string (e.g.,
'openai/gpt-5-nano')
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
timeout:
description: Timeout in ms for the extraction
@ -1738,7 +1766,7 @@ components:
description: Model configuration object or model name string (e.g.,
'openai/gpt-5-nano')
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
variables:
description: Variables whose names are exposed to the model so observe() returns
@ -1801,7 +1829,7 @@ components:
description: Model configuration object or model name string (e.g.,
'openai/gpt-5-nano')
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
systemPrompt:
description: Custom system prompt for the agent
@ -1825,7 +1853,7 @@ components:
agent tools). If not specified, inherits from the main model
configuration.
anyOf:
- $ref: "#/components/schemas/ModelConfigOutput"
- $ref: "#/components/schemas/ModelConfig"
- type: string
additionalProperties: false
AgentUsageOutput:

View File

@ -12,7 +12,7 @@
* SEA_INCLUDE_SOURCEMAPS.
* Example: pnpm run build:sea:cjs -- --target-platform=linux --target-arch=arm64
*/
import { spawnSync } from "node:child_process";
import { execFileSync, spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import fs from "node:fs";
import os from "node:os";
@ -87,6 +87,17 @@ const includeSourcemaps = parseBoolean(
false,
);
const gitOutput = (args: string[]) =>
execFileSync("git", args, { cwd: repoDir, encoding: "utf8" }).trim();
const buildCommit = gitOutput(["rev-parse", "HEAD"]);
const seaVersion = [
`commit: ${buildCommit}`,
`timestamp: ${gitOutput(["show", "-s", "--format=%cI", buildCommit])}`,
].join("\n");
const seaBuildDefine = {
__STAGEHAND_SERVER_VERSION__: JSON.stringify(seaVersion),
};
const run = (cmd: string, args: string[], opts: { cwd?: string } = {}) => {
const result = spawnSync(cmd, args, { stdio: "inherit", ...opts });
if (result.error) {
@ -273,6 +284,7 @@ const buildCjsBundle = () => {
platform: "node",
format: "cjs",
outfile: bundlePath,
define: seaBuildDefine,
logLevel: "warning",
absWorkingDir: repoDir,
});
@ -295,6 +307,7 @@ const buildEsmBundle = () => {
format: "esm",
treeShaking: false,
outfile: appBundlePath,
define: seaBuildDefine,
alias: {
"@browserbasehq/stagehand": `${repoDir}/packages/core/dist/esm/index.js`,
},
@ -412,6 +425,15 @@ const { pathToFileURL } = require("node:url");
const bundleBase64 = ${JSON.stringify(appBytes.toString("base64"))};
const bundleLength = ${appBytes.length};
const bundleHash = ${JSON.stringify(bundleHash)};
const versionOutput = ${JSON.stringify(seaVersion)};
const argv = process.argv.slice(1);
const normalizedArgv = argv[0]?.startsWith("--") ? argv : argv.slice(1);
if (normalizedArgv.includes("--version")) {
console.log(versionOutput);
process.exit(0);
}
const cacheRoot =
process.env.STAGEHAND_SEA_CACHE_DIR ||

View File

@ -44,7 +44,12 @@ async function main() {
LocalBrowserLaunchOptions: Api.LocalBrowserLaunchOptionsSchema,
GoogleServiceAccountCredentials:
Api.GoogleServiceAccountCredentialsSchema,
GoogleAuthOptions: Api.GoogleAuthOptionsSchema,
GoogleServiceAccountAuth: Api.GoogleServiceAccountAuthSchema,
ModelAuth: Api.ModelAuthSchema,
VertexProviderOptions: Api.VertexProviderOptionsSchema,
ModelProviderOptions: Api.ModelProviderOptionsSchema,
GenericModelConfigObject: Api.GenericModelConfigObjectSchema,
VertexModelConfigObject: Api.VertexModelConfigObjectSchema,
ModelConfigObject: Api.ModelConfigObjectSchema,
ModelConfig: Api.ModelConfigSchema,
Action: Api.ActionSchema,

View File

@ -208,10 +208,17 @@ export class InMemorySessionStore implements SessionStore {
const options: V3Options = {
env: isBrowserbase ? "BROWSERBASE" : "LOCAL",
model: {
modelName: params.modelName,
apiKey: ctx.modelApiKey,
},
model: ctx.requestModelConfig
? {
...ctx.requestModelConfig,
...(ctx.requestModelConfig.apiKey
? {}
: { apiKey: ctx.modelApiKey }),
}
: {
modelName: params.modelName,
apiKey: ctx.modelApiKey,
},
verbose: params.verbose,
systemPrompt: params.systemPrompt,
selfHeal: params.selfHeal,

View File

@ -67,6 +67,8 @@ export interface CreateSessionParams {
export interface RequestContext {
/** Model API key (from x-model-api-key header) */
modelApiKey?: string;
/** Structured model config from the current request body */
requestModelConfig?: Api.ModelConfig;
/** Logger function for this request */
logger?: (message: LogLine) => void;
}

View File

@ -1,4 +1,6 @@
import type { FastifyRequest } from "fastify";
import { Api } from "@browserbasehq/stagehand";
import { z } from "zod/v4";
import { MissingHeaderError } from "../types/error.js";
@ -39,41 +41,134 @@ export const getOptionalHeader = (
return headerValue;
};
const emptyStringSchema = z.literal("").transform((): undefined => undefined);
const requiredStringSchema = z.string().min(1);
const optionalStringSchema = z
.union([emptyStringSchema, z.string()])
.optional();
const requestModelInputSchema = z
.union([
emptyStringSchema,
requiredStringSchema.transform(
(modelName): Api.ModelConfig => ({ modelName }),
),
Api.ModelConfigSchema,
])
.optional();
const requestModelCarrierSchema = z
.object({
model: requestModelInputSchema,
})
.passthrough();
const unparsedModelCarrierSchema = z
.object({
model: z.unknown().optional(),
})
.passthrough();
const requestModelEnvelopeSchema = z
.object({
options: requestModelCarrierSchema.optional(),
agentConfig: unparsedModelCarrierSchema.optional(),
modelName: optionalStringSchema,
})
.passthrough();
const stagehandInitModelEnvelopeSchema = z
.object({
options: requestModelCarrierSchema.optional(),
agentConfig: requestModelCarrierSchema.optional(),
modelName: optionalStringSchema,
})
.passthrough();
const requestModelConfigSchema = z
.object({
model: Api.ModelConfigSchema.optional(),
modelName: optionalStringSchema,
apiKey: optionalStringSchema,
})
.strict();
export type RequestModelConfig = z.infer<typeof requestModelConfigSchema>;
type RequestModelConfigResult =
| { success: true; data: RequestModelConfig }
| { success: false; error: z.ZodError };
/**
* Extracts model name from request body, supporting V3 structure.
* - V3: body.options.model.modelName
* Extracts request-level model config with precedence.
*
* Model name:
* 1. body.options.model.modelName or body.options.model string
* 2. Legacy body.modelName fallback
*
* API key:
* 1. body.options.model.apiKey
* 2. x-model-api-key header
*
* agentConfig.model is parsed separately for Stagehand initialization. Its
* credentials are scoped to the agent main model and must not become the
* request-level API key fallback used by action/execution models.
*/
export function getModelName(request: FastifyRequest): string | undefined {
const body = request.body as Record<string, unknown> | undefined;
const options = body?.options as Record<string, unknown> | undefined;
const model = options?.model as Record<string, unknown> | undefined;
if (typeof model?.modelName === "string" && model.modelName) {
return model.modelName;
export function getRequestModelConfig(
request: FastifyRequest,
): RequestModelConfigResult {
const bodyResult = requestModelEnvelopeSchema.safeParse(request.body ?? {});
if (!bodyResult.success) {
return bodyResult;
}
if (typeof body?.modelName === "string" && body.modelName) {
return body.modelName;
const body = bodyResult.data;
const model = body.options?.model;
const configResult = requestModelConfigSchema.safeParse({
model,
modelName: model?.modelName ?? body.modelName,
apiKey: model?.apiKey ?? getOptionalHeader(request, "x-model-api-key"),
});
if (!configResult.success) {
return configResult;
}
return undefined;
return configResult;
}
/**
* Extracts the model API key with precedence:
* 1. Per-request body apiKey (V3: body.options.model.apiKey)
* 2. Per-request header x-model-api-key
* Extracts the structured model config used when creating a Stagehand instance
* for this request. This can read agentConfig.model for agentExecute startup,
* but it does not promote agent model credentials to the request-level API key.
*/
export function getModelApiKey(request: FastifyRequest): string | undefined {
const body = request.body as Record<string, unknown> | undefined;
const options = body?.options as Record<string, unknown> | undefined;
const model = options?.model as Record<string, unknown> | undefined;
if (typeof model?.apiKey === "string" && model.apiKey) {
return model.apiKey;
export function getStagehandInitModelConfig(
request: FastifyRequest,
requestModelConfig?: RequestModelConfig,
): RequestModelConfigResult {
const baseConfigResult = requestModelConfig
? requestModelConfigSchema.safeParse(requestModelConfig)
: getRequestModelConfig(request);
if (!baseConfigResult.success) {
return baseConfigResult;
}
return getOptionalHeader(request, "x-model-api-key");
const baseConfig = baseConfigResult.data;
if (baseConfig.model) {
return baseConfigResult;
}
const bodyResult = stagehandInitModelEnvelopeSchema.safeParse(
request.body ?? {},
);
if (!bodyResult.success) {
return bodyResult;
}
const agentModel = bodyResult.data.agentConfig?.model;
const configResult = requestModelConfigSchema.safeParse({
model: agentModel,
modelName: agentModel?.modelName ?? baseConfig.modelName,
apiKey: baseConfig.apiKey,
});
if (!configResult.success) {
return configResult;
}
return configResult;
}
/**

View File

@ -6,10 +6,11 @@ import { z } from "zod/v4";
import { AppError } from "./errorHandler.js";
import {
getModelApiKey,
getModelName,
getOptionalHeader,
getRequestModelConfig,
getStagehandInitModelConfig,
shouldRespondWithSSE,
type RequestModelConfig,
} from "./header.js";
import { error, success } from "./response.js";
import { getSessionStore } from "./sessionStoreManager.js";
@ -35,6 +36,13 @@ type StreamEventName =
| "error";
type StreamPayloadType = "system" | "log";
function formatZodIssues(err: z.ZodError) {
return err.issues.map((issue) => ({
path: issue.path[0] ?? "unknown",
message: issue.message,
}));
}
export async function createStreamingResponse<TV3>({
sessionId,
request,
@ -44,7 +52,6 @@ export async function createStreamingResponse<TV3>({
operation,
}: StreamingResponseOptions<TV3>) {
const shouldStreamResponse = shouldRespondWithSSE(request);
const modelApiKey = getModelApiKey(request);
const sessionStore = getSessionStore();
const sessionConfig = await sessionStore.getSessionConfig(sessionId);
@ -127,9 +134,45 @@ export async function createStreamingResponse<TV3>({
const actionId = v4();
sendData("starting", "system", { status: "starting" });
const sendZodValidationError = (err: z.ZodError) => {
const validationIssues = formatZodIssues(err);
if (shouldStreamResponse) {
sendData("error", "system", {
status: "error",
error: validationIssues,
});
reply.raw.end();
return reply;
}
return reply.status(StatusCodes.BAD_REQUEST).send({
error: validationIssues,
});
};
const requestModelConfigResult = getRequestModelConfig(request);
if (requestModelConfigResult.success === false) {
return sendZodValidationError(requestModelConfigResult.error);
}
const requestModelConfig: RequestModelConfig = requestModelConfigResult.data;
const stagehandInitModelConfigResult = getStagehandInitModelConfig(
request,
requestModelConfig,
);
if (stagehandInitModelConfigResult.success === false) {
return sendZodValidationError(stagehandInitModelConfigResult.error);
}
const stagehandInitModelConfig = stagehandInitModelConfigResult.data;
const modelApiKey = requestModelConfig.apiKey;
const parsedRequestModelConfig = stagehandInitModelConfig.model?.modelName
? stagehandInitModelConfig.model
: undefined;
const requestContext: RequestContext = {
modelApiKey,
requestModelConfig: parsedRequestModelConfig,
logger: shouldStreamResponse
? (message) => {
sendData("running", "log", { status: "running", message });
@ -177,7 +220,7 @@ export async function createStreamingResponse<TV3>({
operation: operation ?? "operation",
sessionId,
browserType,
modelName: getModelName(request),
modelName: requestModelConfig.modelName,
hasModelApiKey: Boolean(modelApiKey),
hasBrowserbaseApiKey: Boolean(browserbaseApiKey),
hasBrowserbaseProjectId: Boolean(browserbaseProjectId),

View File

@ -8,7 +8,10 @@ import { z } from "zod/v4";
import { authMiddleware } from "../../../lib/auth.js";
import { withErrorHandling } from "../../../lib/errorHandler.js";
import { getModelApiKey, getOptionalHeader } from "../../../lib/header.js";
import {
getOptionalHeader,
getRequestModelConfig,
} from "../../../lib/header.js";
import { error, success } from "../../../lib/response.js";
import { getSessionStore } from "../../../lib/sessionStoreManager.js";
import { AISDK_PROVIDERS } from "../../../types/model.js";
@ -110,11 +113,30 @@ const startRouteHandler: RouteHandler = withErrorHandling(
}
const browserType = browser?.type ?? "browserbase";
const localBrowserLaunchOptions =
browserType === "local" && browser?.launchOptions && !browser?.cdpUrl
? browser.launchOptions
: undefined;
let bbApiKey: string | undefined;
let browserbaseProjectId: string | undefined;
let browserbaseSessionId: string | undefined;
let connectUrl: string | undefined;
let localBrowserModelApiKey: string | undefined;
if (localBrowserLaunchOptions) {
const requestModelConfigResult = getRequestModelConfig(request);
if (requestModelConfigResult.success === false) {
return reply.status(StatusCodes.BAD_REQUEST).send({
error: requestModelConfigResult.error.issues.map((issue) => ({
path: issue.path[0] ?? "unknown",
message: issue.message,
})),
});
}
localBrowserModelApiKey = requestModelConfigResult.data.apiKey;
}
if (browserType === "browserbase") {
bbApiKey = getOptionalHeader(request, "x-bb-api-key");
@ -210,12 +232,11 @@ const startRouteHandler: RouteHandler = withErrorHandling(
// For local browsers with launchOptions (no explicit cdpUrl), eagerly
// initialize the browser so we can return the actual CDP URL
let finalCdpUrl = connectUrl ?? session.cdpUrl ?? "";
if (browserType === "local" && browser?.launchOptions && !browser?.cdpUrl) {
const modelApiKey = getModelApiKey(request);
if (localBrowserLaunchOptions) {
try {
const stagehand = await sessionStore.getOrCreateStagehand(
session.sessionId,
{ modelApiKey },
{ modelApiKey: localBrowserModelApiKey },
);
finalCdpUrl = stagehand.connectURL();
} catch (err) {
@ -226,12 +247,12 @@ const startRouteHandler: RouteHandler = withErrorHandling(
browserType,
chromePathEnv: process.env.CHROME_PATH,
launchOptions: {
executablePath: browser.launchOptions.executablePath,
argsCount: browser.launchOptions.args?.length ?? 0,
headless: browser.launchOptions.headless,
hasUserDataDir: Boolean(browser.launchOptions.userDataDir),
port: browser.launchOptions.port,
connectTimeoutMs: browser.launchOptions.connectTimeoutMs,
executablePath: localBrowserLaunchOptions.executablePath,
argsCount: localBrowserLaunchOptions.args?.length ?? 0,
headless: localBrowserLaunchOptions.headless,
hasUserDataDir: Boolean(localBrowserLaunchOptions.userDataDir),
port: localBrowserLaunchOptions.port,
connectTimeoutMs: localBrowserLaunchOptions.connectTimeoutMs,
},
},
"Failed to initialize local browser session in /v1/sessions/start",

View File

@ -1,15 +1,23 @@
import { __internalMaybeRunShutdownSupervisorFromArgv } from "@browserbasehq/stagehand";
declare const __STAGEHAND_SERVER_VERSION__: string;
// if SEA binary is launched with --supervisor, it will run the shutdown supervisor only
const argv = process.argv.slice(1);
const normalizedArgv = argv[0]?.startsWith("--") ? argv : argv.slice(1);
// otherwise, start the stagehand/server
if (!__internalMaybeRunShutdownSupervisorFromArgv(normalizedArgv)) {
// Force flow logs on before the server module loads Stagehand core.
process.env.BROWSERBASE_FLOW_LOGS = "1";
void import("./server.js").catch((err) => {
if (normalizedArgv.includes("--version")) {
console.log(__STAGEHAND_SERVER_VERSION__);
process.exit(0);
}
void import("@browserbasehq/stagehand")
.then(async ({ __internalMaybeRunShutdownSupervisorFromArgv }) => {
if (__internalMaybeRunShutdownSupervisorFromArgv(normalizedArgv)) {
return;
}
process.env.BROWSERBASE_FLOW_LOGS = "1";
await import("./server.js");
})
.catch((err) => {
console.error("Failed to start server:", err);
process.exit(1);
});
}

View File

@ -0,0 +1,120 @@
import assert from "node:assert/strict";
import { describe, it } from "node:test";
import type { FastifyRequest } from "fastify";
import {
getRequestModelConfig,
getStagehandInitModelConfig,
type RequestModelConfig,
} from "../../src/lib/header.js";
function createRequest({
body,
headers = {},
}: {
body?: Record<string, unknown>;
headers?: Record<string, string>;
}): FastifyRequest {
return {
body,
headers,
} as FastifyRequest;
}
function assertSuccess(
result: ReturnType<typeof getRequestModelConfig>,
): RequestModelConfig {
if (result.success === false) {
throw result.error;
}
assert.equal(result.success, true);
return result.data;
}
describe("getRequestModelConfig", () => {
it("preserves a Vertex model config from an action request so auth fields reach session initialization", () => {
const model = {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
auth: {
type: "googleServiceAccount",
credentials: {
client_email: "vertex@example.iam.gserviceaccount.com",
private_key:
"-----BEGIN PRIVATE KEY-----\\ntest\\n-----END PRIVATE KEY-----",
},
},
providerOptions: {
vertex: {
project: "test-gcp-project",
location: "us-central1",
},
},
};
const request = createRequest({
body: {
options: { model },
},
headers: {
"x-model-api-key": "sk-header",
},
});
const config = assertSuccess(getRequestModelConfig(request));
assert.equal(config.modelName, "vertex/gemini-2.5-flash");
assert.equal(config.apiKey, "sk-header");
assert.deepEqual(config.model, model);
});
it("does not read agentConfig.model as a request-level model config", () => {
const request = createRequest({
body: {
agentConfig: { model: "google/gemini-2.5-flash" },
},
});
assert.deepEqual(assertSuccess(getRequestModelConfig(request)), {
model: undefined,
modelName: undefined,
apiKey: undefined,
});
});
it("normalizes an agent string model into the session bootstrap model config", () => {
const request = createRequest({
body: {
agentConfig: { model: "google/gemini-2.5-flash" },
},
});
assert.deepEqual(assertSuccess(getStagehandInitModelConfig(request)), {
model: { modelName: "google/gemini-2.5-flash" },
modelName: "google/gemini-2.5-flash",
apiKey: undefined,
});
});
it("returns a validation result instead of throwing when agent bootstrap model config is invalid", () => {
const request = createRequest({
body: {
agentConfig: {
model: {
provider: "vertex",
modelName: "vertex/gemini-2.5-flash",
},
},
},
});
assert.deepEqual(assertSuccess(getRequestModelConfig(request)), {
model: undefined,
modelName: undefined,
apiKey: undefined,
});
const result = getStagehandInitModelConfig(request);
assert.equal(result.success, false);
});
});

1280
pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff