## Summary Two things arrive together because the second is what made the first usable. **Run raven from a container.** A Dockerfile builds the page, the Python environment and the engine into one image; `cd docker && docker compose up` serves it. nginx serves the built page and proxies the RPC socket to the engine over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a second container would have no address to dial. Compose publishes on 127.0.0.1, and the entrypoint prints a one-time sign-in link on boot. **A first run is no longer a dead end.** The gateway used to refuse to start until a provider was configured, while being the thing that serves the page where a provider gets configured. It now starts, says what is missing, and picks up a key saved in Settings on the next turn without a restart. The onboarding overlay no longer takes the page either: New chat, the model chip and Send send the reader to Settings > Models instead, and an empty model list is marked on the control that fills it. Also carried: the provider catalogue registry (capabilities, modalities and context windows per model), the reworked Settings > Models pane, and four fixes found in review of that work -- a sibling catalogue answering 200 with a non-catalogue body no longer crashes the whole fetch; adding a tag no longer erases the label and description already stored for that model; `authenticated` no longer demands a bearer token from a local provider reached by address; and registry rows are keyed by the spelling every lookup produces, which recovers 593 rows that no reader could reach. ## Type - [ ] Fix - [x] Feature - [x] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification Commands run on the branch head, with their results: - `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped - `npx vitest run` in `ui-web` -- 1573 passed (101 files) - `npx vitest run` in `ui-tui` -- 2037 passed (142 files) - `uv run ruff check raven scripts tests` and `ruff format --check` -- clean - `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean - `npm run lint:rpc` in `ui-tui` -- generated.ts in sync - `make check-source-language`, `make check-large-files` -- clean Each of the four review fixes was checked by reverting it and watching a named test go red, then restoring it. The container image itself has not been built by CI or by me -- no Docker daemon was available on the machine this was written on. It has been built and run by the author of the branch. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk Two deliberate security trade-offs in the container deployment, both documented in `docker/README.md`: - nginx rewrites the `Origin` header, because the engine accepts only an Origin naming its own loopback port and no browser reaching a published port can send one. This drops a defence-in-depth layer. The session cookie is SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite ignores ports, so another page on localhost counts as same-site. - `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to every page load, so anyone who can reach the published port is signed in. It is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any deployment reachable from elsewhere. These are two settings that must agree, and nothing warns when they do not. Behaviour changes outside the container: the gateway now starts without a configured provider rather than exiting 1, and the first-run onboarding overlay no longer opens by default (`?onboard=1` still forces it). Rollback is by revert; nothing here migrates state or changes an on-disk format. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A
Raven Benchmarks
This directory holds evaluation harnesses that are deliberately decoupled
from the runtime package. They are not imported by raven/ and are
excluded from the wheel build — keep it that way.
Use this area for reproducible evaluation work: capability suites, agent comparisons, context stress tests, and proactivity runs that should not ship as part of the end-user CLI package.
Layout
benchmarks/
├── appworld/ AppWorld agent benchmark + evolver plugin
│ ├── agent_cli.py One-task subject agent (drives AgentLoop)
│ ├── batch.py Batch scorer: N tasks x K trials, resumable
│ └── evolve/ evolver BenchBundle plugin (entry.py)
│ + designer/diagnosis/sandbox/precheck glue
│
├── pinchbench/ Context / AgentLoop capability benchmark
│ ├── tasks/ 23 task_*.md cards (YAML frontmatter + sections)
│ ├── direct/ Drives AgentLoop.process_direct() per task
│ ├── bot_runner/ Drives full gateway + channel path per task
│ ├── assets/ Task-specific workspace files
│ └── results/ Run outputs (gitignored)
│
├── clawbench/ ClawBench streaming benchmark adapter
│ ├── stream.py Drives AgentLoop.process_direct() across one session
│ ├── run.sh Shell wrapper
│ └── README.md Setup and run instructions
│
├── proactivity_eval/ Sentinel / Proactive Engine benchmark
│ ├── data/ 5 sub-benchmarks: cases / pbench / longrun /
│ │ simulation / timeline + reward_data
│ └── runners/ run.py (unified entry) + judge.py +
│ *_scorecard.py + per-agent adapters
│
└── README.md This file
skill_evals/ will land here in Phase 1 once raven/skill_forge/evals/
is folded into memory_engine/skill/.
Running
Model and tool configuration
The benchmark runners can use the normal ~/.raven/config.json, or the
environment overrides below. Never commit real keys.
For an OpenAI-compatible gateway using OpenRouter-style environment names:
export OPENROUTER_API_KEY="..."
export OPENROUTER_API_BASE="https://openrouter.ai/api/v1"
export RAVEN_BENCH_PROVIDER="custom"
export RAVEN_BENCH_MODEL="deepseek-v4-flash"
Optional web tools:
export SERPER_API_KEY="..."
export JINA_API_KEY="..."
Equivalent ~/.raven/config.json:
{
"agents": {
"defaults": {
"provider": "custom",
"model": "deepseek-v4-flash",
"maxToolIterations": 40,
"contextWindowTokens": 65536
}
},
"providers": {
"custom": {
"apiKey": "YOUR_API_KEY",
"apiBase": "YOUR_OPENAI_COMPATIBLE_API_BASE"
}
},
"tools": {
"web": {
"jinaApiKey": "YOUR_JINA_KEY",
"search": {
"apiKey": "YOUR_SERPER_KEY"
}
}
}
}
PinchBench (Direct mode):
./benchmarks/pinchbench/direct/run.sh \
--model deepseek-v4-flash \
--provider custom \
--api-base "$OPENROUTER_API_BASE" \
--api-key "$OPENROUTER_API_KEY" \
--suite task_00_sanity
PinchBench (Bot mode):
./benchmarks/pinchbench/bot_runner/run.sh --suite automated-only
ClawBench (first 80 tasks, one streaming session):
git clone https://github.com/claw-bench/claw-bench ../claw-bench
export CLAW_BENCH_ROOT="$PWD/../claw-bench"
./benchmarks/clawbench/run.sh \
--clawbench-root "$CLAW_BENCH_ROOT" \
--limit 80 \
--session-id clawbench-stream-raven-80 \
--max-iterations 40
ClawBench with Curator context engine:
./benchmarks/clawbench/run.sh \
--clawbench-root "$CLAW_BENCH_ROOT" \
--limit 80 \
--session-id clawbench-stream-raven-curator-80 \
--context-engine curator \
--curator-model deepseek-v4-flash \
--max-iterations 40
Proactivity Eval (longrun, 30-day persona):
uv run python benchmarks/proactivity_eval/runners/run.py \
--agent raven --mode sentinel --benchmark longrun \
--persona dev-01
Relation to runtime
The runtime (raven/) never statically imports from benchmarks/ — this
is the "independent eval track" principle. The reverse is allowed and
expected: benchmarks import raven.agent, raven.providers, etc. directly.
One scoped exception: evolver loads its bench plugins from here by
registry name at launch (benchmarks.appworld.evolve.entry:build), inserting
the subject repo root on sys.path first. It is lazy, opt-in, and only works
from a repo checkout — evolution needs the git repo as its subject anyway, so
nothing in the installed wheel depends on this directory.
Two unit tests under tests/ reach into benchmarks/proactivity_eval/runners/
via sys.path injection because they exercise the synthesizer / prompt-loader
code that lives there (it is benchmark code, not runtime code, and there is
no production substitute to test against).