0xKT 8716ec525c feat(docker): run raven in a docker container and make a first run configurable from the page
## Summary

Two things arrive together because the second is what made the first usable.

**Run raven from a container.** A Dockerfile builds the page, the Python
environment and the engine into one image; `cd docker && docker compose up`
serves it. nginx serves the built page and proxies the RPC socket to the engine
over the loopback they share -- the engine binds 127.0.0.1 only, so a proxy in a
second container would have no address to dial. Compose publishes on 127.0.0.1,
and the entrypoint prints a one-time sign-in link on boot.

**A first run is no longer a dead end.** The gateway used to refuse to start
until a provider was configured, while being the thing that serves the page
where a provider gets configured. It now starts, says what is missing, and picks
up a key saved in Settings on the next turn without a restart. The onboarding
overlay no longer takes the page either: New chat, the model chip and Send send
the reader to Settings > Models instead, and an empty model list is marked on
the control that fills it.

Also carried: the provider catalogue registry (capabilities, modalities and
context windows per model), the reworked Settings > Models pane, and four fixes
found in review of that work -- a sibling catalogue answering 200 with a
non-catalogue body no longer crashes the whole fetch; adding a tag no longer
erases the label and description already stored for that model; `authenticated`
no longer demands a bearer token from a local provider reached by address; and
registry rows are keyed by the spelling every lookup produces, which recovers
593 rows that no reader could reach.

## Type

- [ ] Fix
- [x] Feature
- [x] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Commands run on the branch head, with their results:

- `uv run pytest tests -q -k "provider or registry or catalog or model or rpc or gateway or docker or i18n"` -- 4389 passed, 46 skipped
- `npx vitest run` in `ui-web` -- 1573 passed (101 files)
- `npx vitest run` in `ui-tui` -- 2037 passed (142 files)
- `uv run ruff check raven scripts tests` and `ruff format --check` -- clean
- `npx tsc --noEmit` in `ui-web` and `npm run type-check` in `ui-tui` -- clean
- `npm run lint:rpc` in `ui-tui` -- generated.ts in sync
- `make check-source-language`, `make check-large-files` -- clean

Each of the four review fixes was checked by reverting it and watching a named
test go red, then restoring it.

The container image itself has not been built by CI or by me -- no Docker daemon
was available on the machine this was written on. It has been built and run by
the author of the branch.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

Two deliberate security trade-offs in the container deployment, both documented
in `docker/README.md`:

- nginx rewrites the `Origin` header, because the engine accepts only an Origin
  naming its own loopback port and no browser reaching a published port can send
  one. This drops a defence-in-depth layer. The session cookie is
  SameSite=Strict, so an unauthenticated socket is still refused -- but SameSite
  ignores ports, so another page on localhost counts as same-site.
- `RAVEN_AUTO_LOGIN=1` is the default: nginx attaches the session cookie to
  every page load, so anyone who can reach the published port is signed in. It
  is contained by Compose publishing on 127.0.0.1 only. Set it to 0 for any
  deployment reachable from elsewhere. These are two settings that must agree,
  and nothing warns when they do not.

Behaviour changes outside the container: the gateway now starts without a
configured provider rather than exiting 1, and the first-run onboarding overlay
no longer opens by default (`?onboard=1` still forces it).

Rollback is by revert; nothing here migrates state or changes an on-disk format.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A
2026-09-11 03:27:52 +00:00
..

Raven Benchmarks

This directory holds evaluation harnesses that are deliberately decoupled from the runtime package. They are not imported by raven/ and are excluded from the wheel build — keep it that way.

Use this area for reproducible evaluation work: capability suites, agent comparisons, context stress tests, and proactivity runs that should not ship as part of the end-user CLI package.

Layout

benchmarks/
├── appworld/           AppWorld agent benchmark + evolver plugin
│   ├── agent_cli.py       One-task subject agent (drives AgentLoop)
│   ├── batch.py           Batch scorer: N tasks x K trials, resumable
│   └── evolve/            evolver BenchBundle plugin (entry.py)
│                          + designer/diagnosis/sandbox/precheck glue
│
├── pinchbench/         Context / AgentLoop capability benchmark
│   ├── tasks/             23 task_*.md cards (YAML frontmatter + sections)
│   ├── direct/            Drives AgentLoop.process_direct() per task
│   ├── bot_runner/        Drives full gateway + channel path per task
│   ├── assets/            Task-specific workspace files
│   └── results/           Run outputs (gitignored)
│
├── clawbench/          ClawBench streaming benchmark adapter
│   ├── stream.py          Drives AgentLoop.process_direct() across one session
│   ├── run.sh             Shell wrapper
│   └── README.md          Setup and run instructions
│
├── proactivity_eval/   Sentinel / Proactive Engine benchmark
│   ├── data/              5 sub-benchmarks: cases / pbench / longrun /
│   │                      simulation / timeline + reward_data
│   └── runners/           run.py (unified entry) + judge.py +
│                          *_scorecard.py + per-agent adapters
│
└── README.md           This file

skill_evals/ will land here in Phase 1 once raven/skill_forge/evals/ is folded into memory_engine/skill/.

Running

Model and tool configuration

The benchmark runners can use the normal ~/.raven/config.json, or the environment overrides below. Never commit real keys.

For an OpenAI-compatible gateway using OpenRouter-style environment names:

export OPENROUTER_API_KEY="..."
export OPENROUTER_API_BASE="https://openrouter.ai/api/v1"
export RAVEN_BENCH_PROVIDER="custom"
export RAVEN_BENCH_MODEL="deepseek-v4-flash"

Optional web tools:

export SERPER_API_KEY="..."
export JINA_API_KEY="..."

Equivalent ~/.raven/config.json:

{
  "agents": {
    "defaults": {
      "provider": "custom",
      "model": "deepseek-v4-flash",
      "maxToolIterations": 40,
      "contextWindowTokens": 65536
    }
  },
  "providers": {
    "custom": {
      "apiKey": "YOUR_API_KEY",
      "apiBase": "YOUR_OPENAI_COMPATIBLE_API_BASE"
    }
  },
  "tools": {
    "web": {
      "jinaApiKey": "YOUR_JINA_KEY",
      "search": {
        "apiKey": "YOUR_SERPER_KEY"
      }
    }
  }
}

PinchBench (Direct mode):

./benchmarks/pinchbench/direct/run.sh \
    --model deepseek-v4-flash \
    --provider custom \
    --api-base "$OPENROUTER_API_BASE" \
    --api-key "$OPENROUTER_API_KEY" \
    --suite task_00_sanity

PinchBench (Bot mode):

./benchmarks/pinchbench/bot_runner/run.sh --suite automated-only

ClawBench (first 80 tasks, one streaming session):

git clone https://github.com/claw-bench/claw-bench ../claw-bench
export CLAW_BENCH_ROOT="$PWD/../claw-bench"

./benchmarks/clawbench/run.sh \
    --clawbench-root "$CLAW_BENCH_ROOT" \
    --limit 80 \
    --session-id clawbench-stream-raven-80 \
    --max-iterations 40

ClawBench with Curator context engine:

./benchmarks/clawbench/run.sh \
    --clawbench-root "$CLAW_BENCH_ROOT" \
    --limit 80 \
    --session-id clawbench-stream-raven-curator-80 \
    --context-engine curator \
    --curator-model deepseek-v4-flash \
    --max-iterations 40

Proactivity Eval (longrun, 30-day persona):

uv run python benchmarks/proactivity_eval/runners/run.py \
    --agent raven --mode sentinel --benchmark longrun \
    --persona dev-01

Relation to runtime

The runtime (raven/) never statically imports from benchmarks/ — this is the "independent eval track" principle. The reverse is allowed and expected: benchmarks import raven.agent, raven.providers, etc. directly.

One scoped exception: evolver loads its bench plugins from here by registry name at launch (benchmarks.appworld.evolve.entry:build), inserting the subject repo root on sys.path first. It is lazy, opt-in, and only works from a repo checkout — evolution needs the git repo as its subject anyway, so nothing in the installed wheel depends on this directory.

Two unit tests under tests/ reach into benchmarks/proactivity_eval/runners/ via sys.path injection because they exercise the synthesizer / prompt-loader code that lives there (it is benchmark code, not runtime code, and there is no production substitute to test against).