…steps
# why
- Evaluator often is too strict in its evaluation, resulting in false
positives
- max step limits in evals are very brittle and can cause false
negatives on tasks that exceed the current limit
- Evaluator can sometimes view screenshots as a single source of truth
resulting in false negatives. Providing the agents reasoning alongside
these heps mitigate this
# what changed
- paramaterized max steps to allow for easy configuration across all
evals through env
- adjusted evaluator prompting
- added "agent reasoning" param which can be used to pass in the agents
reasoning alongside bulk screenshots in a format evaluator can
understand well
# test plan
tested locally
* Adding Groq inference support (#536)
* Adding Groq inference support
* Add Groq model Llama 3.3 70B Versatile
* Add Groq Llama 3.3 70B SpecdDec model and form filling example
- Added new Groq model "groq-llama-3.3-70b-specdec" to supported models
- Created a new example script for form filling with sensible data using Groq
- Updated model type definitions and provider mappings
* Remove Groq form filling example script
* Add Groq model type to LLMClient
* Add Groq client to LLMProvider
* changeset
---------
Co-authored-by: Sankalp Gunturi <sgunturi@protonmail.com>
* Adding Cerebras inference support (#502)
* Added support for Cerebras models
Full function calling support for both llama-3.3-70b and llama-3.1-8b models
(llama 3.1 8b struggles with tools)
* Added higher performance model as default for example
* Changed Cerebras model prefixes to include provider
* Update CerebrasClient.ts
* Update LLMProvider.ts
* Update LLMClient.ts
* Update model.ts
* Update LLMClient.ts
* Update LLMProvider.ts
* Update LLMProvider.ts
---------
Co-authored-by: Anirudh Kamath <github@kamath.io>
* changeset
---------
Co-authored-by: kwt00 <116334641+kwt00@users.noreply.github.com>
* add api class & types
* simplify stagehand method calls
* revert example
* changeset
* use new syntax in evals
* new extract eval
* allow no observe input
* add api param to constructor
* offload to api
* refresh page after goto and act
* convert zod schema
* use env api url
* add systemprompt to init
* update api responses
* throw error when using api w/ local
* update stream header
* add end call
* fix end & stream
* rm changeset
* revert stagehand config
* Update .env.example
* changeset
* add gh actions - WIP
* update dependency package ffor ci
* inject env to gh action
* install playwright deps in ci
* added headless var to env variables
* add headless env injection to evals
* adding more logging to evals
* updated workflows
* use brainstrust gh action
* running gh action
* run all tasks
* uncomment tasks
* turn off costar eval
* add costar back to actions
* error out costar eval
* typo fix
* add gh action for braintrust
* move permissions up in the file
* change logs to JSONL format
* remove undefined function from evals
* fixing braintrust
* revert logs back to strings
* updatge
* update
* trim the DOM
---------
Co-authored-by: Filip Michalsky <filip_michalsky@g.harvard.edu>
* add abstraction layer for llm to extend to different LLMs
* remove instructor
* antrhopic wip
* antrhopic is working
---------
Co-authored-by: Filip Michalsky <filip_michalsky@g.harvard.edu>