Raven-1/tests/test_everos_server.py
zhanghui 2b3cce70fa
feat(*): upgrade everos to 1.4.1, lift the windows gate, guard the embedding width (#791)
## Summary

EverOS 1.4.0 runs natively on Windows and 1.4.1 pins `openai` below 3,
so the memory plugin's exact pin moves from `everos[multimodal]==1.2.3`
to `==1.4.1`. The lock follows: `pyarrow` 24.0.0 -> 25.0.1 (EverOS's new
floor), `everalgo-boundary` 0.2.0 -> 0.2.1, `everalgo-core` 0.4.0 ->
0.3.0 (EverOS now pins its everalgo layer with `==`), and `msvc-runtime`
on `win32` only. `lancedb` stays at 0.34.0. Every internal everos symbol
the adapter reaches is present in 1.4.1 with the same signature, so the
adapter itself is unchanged. The plugin manifest and package version
move to 1.4.0.

**The Windows gate is gone.** `everos_platform_note` (#737) and its five
call sites, `ServiceState.UNSUPPORTED`, the import.scan refusal (#750),
the wizard's WSL notice and its zh catalogue line are removed;
`_everos_executable` looks for `everos.exe` on `win32`. Spawning,
probing and reusing the server were already portable. Identifying and
stopping a stale server was not: `_cmdline_of` asked `ps`, and the
marker `everos server start` never matched a Windows command line
(`...\Scripts\everos.exe server start`). The command line now comes from
WMI through PowerShell on `win32` and the marker accepts both shapes;
`_listening_port` asks the TCP table through PowerShell there, walking
the launcher's descendants: the socket belongs to the base interpreter
two launchers below `everos.exe`. The server is spawned in a process
group of its own (the Windows counterpart of `start_new_session`, which
also keeps a Ctrl-C at the gateway off it) and stopped with Ctrl-Break,
which uvicorn takes as a shutdown; one that ignores it for ten seconds,
or one no group can be addressed to, gets `TerminateProcess` as before.
And because Windows cannot replace a running executable, the upgrade
helper stops whatever still runs from under the tool environment before
it installs (the server outlives the gateway by design); without that,
`uv` failed on `Scripts\everos.exe` with os error 32. Recorded in
`docs/memory-plugin-architecture.md` section 7.4.

**Two guards, found while verifying against a copy of a real 1.2.x
store:**

- `set_embedding_endpoint` measures the model's vector width before
writing the pin and refuses anything narrower than the 1024-wide memory
index (`REQUIRED_EMBEDDING_DIMENSIONS`), but only where an EverOS
backend will read the pin: the `everos-memory` distribution is installed
and the config names `everos` as the memory backend (an absent key is
the schema default; an explicit `null` or another backend is not gated,
and a knowledge base sizes itself to whatever width the model returns).
The wizard's own check imports the same constant and probe. A
768-dimension model pinned through the settings page had left every
memory store and search answering 500 about a mismatched width, with
nothing on the page saying why. A probe that cannot reach the provider
is not a verdict: the pin is written and a warning logged. The two RPC
writers run the (now network-touching) writer in a thread.
- `ensure_everos_server` reads the running server's `/health` version
and, for a root raven owns, sends a mismatch with the installed everos
through the same precheck / stop / spawn chain a rotated credential
takes (`stale_reason`). Raven used to reuse whatever answered on the
port, so an upgrade left the old server serving until something
unrelated restarted it. A root the user manages is never touched.

**The wire contract is pinned.** Every request body the plugin and the
memory page send (`/add` with a tool exchange, `/flush`, the four
`/search` shapes, the four `/get` kinds) is validated against everos
1.4.1's own request models in `TestRequestBodiesMatchEverosModels`, so a
schema move on the next upgrade fails in the suite rather than as a 422
in a gateway log.

**A six-angle review then hardened the degraded paths** (turn survival
under every EverOS failure, Windows lifecycle, upgrade paths, the
embedding guard, the wire contract, concurrency). The rule: memory
degrades with a notice and never blocks a turn, a start or an upgrade.

- A gateway starts the server again when it finds nothing listening (or
a probe that timed out) and nothing holding the OME lock: only for a
root raven owns, never over a child of its own, at most once per thirty
seconds. The only spawn used to be in `start()`, so a server that went
away left a running gateway without memory until restarted by hand. A
stop that was sent and did not finish is not adopted (uvicorn closes the
port at once and finishes what it had), so the version-mismatch restart
after an upgrade cannot leave the gateway on a dying server.
- A command-line lookup that failed (no `ps`, PowerShell blocked, WMI
wedged, timeout) is distinct from a process that is gone: the stop keeps
waiting, the lock names nothing, and a settings save over a server that
answers but cannot be identified is reported as not applied. A losing
spawn no longer overwrites the live server's pidfile. `stop_pid` counts
wall time and takes a grace; on Windows a backend drains the server it
started at `stop()`.
- The upgrade helper asks again after `Stop-Process` and refuses to run
`uv` while anything from the environment survives (an elevated or
another user's process): `--force` removes the environment before
writing and would have left every file but the one that could not go.
- An embedding pin narrower than the index is measured at backend start
(once per pin per process) and withheld from the spawn: EverOS runs
keyword recall and keeps storing, and the notice names the model and
width. This covers a pin written around the write-time check (before
memory was on, unreachable at save time, edited by hand). The probe no
longer raises on an odd response, times out at ten seconds, honours
`plugins.disabled`; `migrate_roles` runs off the event loop it had put a
provider round-trip onto.
- The memory page uses the `/api/v2` routes (v1 is EverOS's legacy
alias), asks `/health` before a search the way the chat adapter does,
and shows the server's own sentence on a refusal. The recalled profile
is rendered as lines rather than a `namespace(...)` repr with its
evidence fields; a tool-call-only row stores empty content, not
`"None"`; `top_k` stays within 1..100; an empty query asks nothing; a
failed capability probe is not cached. The wizard accepts a model wider
than 1024 (EverOS keeps the first 1024, as the settings page already
allowed) and re-prompts on the host's refusal instead of a traceback.

`docs/memory-plugin-architecture.md` section 7 records the upgrade (7.4)
the way 7.3 recorded 1.2.1 -> 1.2.3.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Unit and RPC tests over every touched area:

```
env -u SERPER_API_KEY uv run --frozen --python 3.12 --extra dev pytest -q tests/test_everos_server.py tests/test_everos_backend.py tests/test_cli_onboard_commands.py tests/test_everos_discover.py tests/test_rpc_settings.py tests/test_everos_config.py tests/test_everos_plugin_discovery.py tests/test_cli_plugin_commands.py tests/test_core_plugin_stack.py tests/test_rpc_memory.py tests/test_rpc_import_sync.py tests/test_everos_http_adapter.py tests/test_memory_backend_protocol.py tests/test_memory_backend_contract.py tests/test_plugin_boundary.py tests/test_i18n_boundary.py tests/test_config_update*.py
# 1312 passed, 1 skipped (inotify is Linux-only) on the first commit; 960 passed after the review round; full suite 26214 passed, 2 failed, 120 skipped, the 2 failing on an untouched github/main checkout too (LibreOffice path, background child env)
uv run --frozen --python 3.12 --extra dev ruff check <changed files>   # All checks passed!
uv run --frozen --python 3.12 --extra dev ruff format --check <changed files>   # 20 files already formatted
```

Against a 46 MB copy of a store written by everos 1.2.x (568 episodes,
5528 atomic facts, seven LanceDB tables), never touching the live one:

- `everos server start` on 1.4.1 opens it with no schema complaint;
`/health` reports `version 1.4.1`, `cascade.healthy true`.
- The same four `/api/v2/memory/search` requests on 1.2.3 and 1.4.1
return the same ids in the same order; BM25 scores drift in the third
decimal place.
- `raven agent -m` on this branch spawns everos 1.4.1, a "remember this"
turn lands in `episodes/episode-2026-09-24.md` and `.atomic_facts/`, a
fresh one-shot session recalls it, and a third session recalls episodes
written under 1.2.x months earlier.
- With a 1.2.3 server left on the port, the next `raven agent -m` logs
`runs everos 1.2.3 while this raven installs 1.4.1; restarting`, the old
pid exits and `/health` reports 1.4.1. The 1.2.3 server had opened the
index 1.4.1 wrote with no complaint.
- Windows 11 box, this branch checked out and synced: `raven serve`
spawns `everos.exe` natively, the Memory page shows its four tabs
instead of the platform sentence, a chat turn lands as an episode
(`/add` 200, `/flush` 200, cascade `upserted=1`), and after a memory
role change the next start logs `holds credentials raven has since
changed; restarting`, the old everos pid exits and the new one answers
`/health` 1.4.1.
- The settings page refuses `BAAI/bge-base-en-v1.5` on deepinfra with
`returns 768-dimension vectors and the memory index is 1024 wide` (real
probe), and the pin stays unchanged.
- Adapter import smoke against 1.4.1: all 13 internal symbols present,
signatures unchanged.
- Windows 11 box, second round: with the server up, `uv pip install
--reinstall --no-deps everos==1.4.1` fails on `Scripts\everos.exe` (os
error 32, exit 2); after the helper's sweep (`Stopped what was still
running from the old install (pid ...)`) the same command succeeds and
the relaunched gateway spawns a fresh 1.4.1. A stop from the spawning
console runs the full uvicorn shutdown (`Shutting down` ... `Finished
server process`) in 3 s; a stop from another console falls back and the
process is gone in 2 s. `lock_holder(root).port` reads 18997 (launcher
-> venv python -> base python).
- `tests/test_everos_server.py tests/test_updates_upgrade.py`: 261
passed, 1 skipped; `tests/test_everos_backend.py -k RequestBodies`: 7
passed.
- Review round (`env -u SERPER_API_KEY uv run --frozen --python 3.12
--extra dev pytest` over the everos, config, settings, memory page,
import, onboard, plugin-stack, boundary, cycle-budget and contract
files): 1225 passed, 1 skipped; `test_everos_server.py` 124,
`test_everos_backend.py` 188.
- Windows 11 box, review round: with the gateway serving, the everos
launcher was terminated by hand (`Stop-Process`, nothing on 18997); the
next turn answered without memory (`recall failed ...
state=unresponsive`), and six seconds after it began the gateway logged
`started everos server` / `everos server ready`, the pidfile named the
new pid and `/health` answered 1.4.1. The first attempt did not fire
because a dead port there takes 2-4 s to be refused, longer than the 1 s
probe; the respawn now fires on a timed-out probe as well, guarded by
the lock holder. The memory page's four counts and a search on the Cases
tab went through the v2 routes.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- Windows installs that used to read "long-term memory is not available
on Windows yet" now spawn EverOS, and a stale server is identified and
replaced there too. The stop there is Ctrl-Break with a ten-second
fallback to `TerminateProcess`. An upgrade on Windows first stops every
process whose executable lives under the tool environment: raven's own
gateway, sub-agents and server, nothing outside that directory.
- An embedding pin narrower than 1024 dimensions is refused at the
settings page and the wizard, with the width in the message, only for an
install whose memory runs on EverOS (plugin installed, backend `everos`
or the default). Knowledge-only installs, memory off, and other backends
keep any width. Existing pins are not touched.
- An owned EverOS server whose version differs from the installed
package is restarted on the next session start. A root the user manages
is never restarted.
- A gateway may now start an EverOS server on its own when its probe
finds nothing listening and nothing holding the lock (owned roots only,
at most every thirty seconds). On Windows a backend stops the server it
started when it shuts down (Ctrl-Break, sixty seconds to drain), so a
settings change or an upgrade there restarts the server rather than
leaving it. An upgrade on Windows refuses to proceed while a process
from the tool environment cannot be stopped, and says which pid.
- An embedding pin narrower than 1024 that reached the config around the
settings check is no longer handed to EverOS: recall runs on keywords
with a notice naming the model, until a 1024-dimension model is pinned.
- Rollback: revert the commit and `uv sync`. `lancedb` did not move, so
an index written by 1.4.1 opens under 1.2.3 (exercised on the copy).

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: gloryfromca <23442919+gloryfromca@users.noreply.github.com>
Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-25 19:33:18 +08:00

2161 lines
92 KiB
Python

"""Tests for raven_everos.server."""
from __future__ import annotations
import asyncio
import os
import shutil
import sys
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import MagicMock, patch
import pytest
import raven_everos.server as everos_server
from raven_everos.server import (
EverosNotConfiguredError,
EverosStillStartingError,
_everos_executable,
ensure_everos_server,
)
def _make_executable(path):
path.write_text("#!/bin/sh\n")
path.chmod(0o755)
return path
class TestEverosExecutable:
"""The interpreter's own directory wins over PATH.
``uv tool install`` exposes only the requested package's entry points, so a
released install has ``raven`` on PATH and ``everos`` only inside the tool
venv. Preferring the sibling also avoids picking up an everos from an
unrelated environment whose version does not match raven's pin.
"""
def test_prefers_interpreter_sibling(self, tmp_path, monkeypatch) -> None:
venv_bin = tmp_path / "venv" / "bin"
venv_bin.mkdir(parents=True)
_make_executable(venv_bin / "everos")
monkeypatch.setattr("sys.executable", str(venv_bin / "python3"))
monkeypatch.setenv("PATH", "")
assert _everos_executable() == str(venv_bin / "everos")
def test_falls_back_to_path(self, tmp_path, monkeypatch) -> None:
venv_bin = tmp_path / "venv" / "bin"
venv_bin.mkdir(parents=True)
path_dir = tmp_path / "elsewhere"
path_dir.mkdir()
_make_executable(path_dir / "everos")
monkeypatch.setattr("sys.executable", str(venv_bin / "python3"))
monkeypatch.setenv("PATH", str(path_dir))
assert _everos_executable() == str(path_dir / "everos")
def test_sibling_beats_path_when_both_exist(self, tmp_path, monkeypatch) -> None:
venv_bin = tmp_path / "venv" / "bin"
venv_bin.mkdir(parents=True)
_make_executable(venv_bin / "everos")
path_dir = tmp_path / "elsewhere"
path_dir.mkdir()
_make_executable(path_dir / "everos")
monkeypatch.setattr("sys.executable", str(venv_bin / "python3"))
monkeypatch.setenv("PATH", str(path_dir))
assert _everos_executable() == str(venv_bin / "everos")
def test_sibling_without_exec_bit_is_skipped(self, tmp_path, monkeypatch) -> None:
venv_bin = tmp_path / "venv" / "bin"
venv_bin.mkdir(parents=True)
(venv_bin / "everos").write_text("not executable\n")
(venv_bin / "everos").chmod(0o644)
path_dir = tmp_path / "elsewhere"
path_dir.mkdir()
_make_executable(path_dir / "everos")
monkeypatch.setattr("sys.executable", str(venv_bin / "python3"))
monkeypatch.setenv("PATH", str(path_dir))
assert _everos_executable() == str(path_dir / "everos")
def test_on_windows_the_sibling_is_everos_exe(self, tmp_path, monkeypatch) -> None:
scripts = tmp_path / "venv" / "Scripts"
scripts.mkdir(parents=True)
_make_executable(scripts / "everos.exe")
monkeypatch.setattr("sys.platform", "win32")
monkeypatch.setattr("sys.executable", str(scripts / "python.exe"))
monkeypatch.setenv("PATH", "")
assert _everos_executable() == str(scripts / "everos.exe")
def test_missing_everywhere_names_the_interpreter_dir(self, tmp_path, monkeypatch) -> None:
venv_bin = tmp_path / "venv" / "bin"
venv_bin.mkdir(parents=True)
monkeypatch.setattr("sys.executable", str(venv_bin / "python3"))
monkeypatch.setenv("PATH", "")
with pytest.raises(RuntimeError, match=str(venv_bin)):
_everos_executable()
@pytest.fixture
def everos_toml(tmp_path, monkeypatch):
"""Redirect the EverOS config read/write to a throwaway root raven owns."""
from raven_everos import config as ue
root = tmp_path / ".everos"
monkeypatch.setattr(ue, "everos_root", lambda: root)
monkeypatch.setattr(ue, "everos_owned", lambda: True)
return root / "everos.toml"
def _live_child():
"""A Popen stand-in that is still running (``poll()`` returns None)."""
proc = MagicMock()
proc.poll.return_value = None
return proc
def _pin_llm_role(cfg, *, model="mem-llm", api_key="k"):
"""Make the llm role count as configured, the way raven records it now.
The gate asks raven's own config for a model and a provider, then asks that
provider for a usable credential. So "no api key" is expressed here as a
provider that has none -- not as a toml section missing a field, which is
the shape that stopped being read.
"""
import json
from raven import home as raven_home
provider = {"apiBase": "https://mem.test/v1"}
if api_key:
provider["apiKey"] = api_key
path = cfg.parent.parent / "config.json"
path.write_text(
json.dumps(
{
"providers": {"deepseek": provider},
"plugins": {"config": {"everos-memory": {"llm": {"model": model, "provider": "deepseek"}}}},
}
),
encoding="utf-8",
)
raven_home.set_config_path(path)
return path
@pytest.fixture(autouse=True)
def _deterministic_inotify_gate(request, monkeypatch):
"""Spawn tests must not depend on the host's inotify headroom.
``_inotify_gate`` scans procfs and refuses the spawn when the per-user
cap is exhausted -- real behaviour that ``TestInotifyGate`` covers
explicitly, and a wall the other tests should not have to pass on a host
that happens to be near the cap.
"""
if request.cls is TestInotifyGate:
return
monkeypatch.setattr("raven_everos.server._inotify_gate", lambda: None)
class TestInotifyGate:
"""The inotify headroom check keeps a spawn that cannot survive off the host."""
@pytest.fixture
def _logs(self, tmp_path, monkeypatch):
monkeypatch.setattr("raven_everos.server.get_logs_dir", lambda: tmp_path)
return tmp_path / "everos-server.log"
def test_usage_is_none_when_the_cap_is_unreadable(self, monkeypatch, tmp_path) -> None:
monkeypatch.setattr(everos_server, "_INOTIFY_LIMIT_PATH", tmp_path / "missing")
assert everos_server._inotify_usage() is None
def test_usage_counts_same_uid_instances(self, monkeypatch, tmp_path) -> None:
cap = tmp_path / "max"
cap.write_text("128\n", encoding="ascii")
monkeypatch.setattr(everos_server, "_INOTIFY_LIMIT_PATH", cap)
proc = tmp_path / "proc"
(proc / "111" / "fd").mkdir(parents=True)
(proc / "111" / "fd" / "3").symlink_to("anon_inode:inotify")
(proc / "111" / "fd" / "4").symlink_to("socket:[123]")
(proc / "222" / "fd").mkdir(parents=True)
(proc / "222" / "fd" / "5").symlink_to("anon_inode:inotify")
(proc / "111" / "fd" / "9").symlink_to(proc / "vanished")
(proc / "333").symlink_to(proc / "vanished")
(proc / "947").mkdir()
(proc / "947" / "fd").write_text("not a dir", encoding="ascii")
(proc / "notaproc" / "fd").mkdir(parents=True)
monkeypatch.setattr(everos_server, "_PROC_ROOT", proc)
assert everos_server._inotify_usage() == (2, 128)
def test_usage_ignores_other_users_instances(self, monkeypatch, tmp_path) -> None:
cap = tmp_path / "max"
cap.write_text("128\n", encoding="ascii")
monkeypatch.setattr(everos_server, "_INOTIFY_LIMIT_PATH", cap)
proc = tmp_path / "proc"
(proc / "111" / "fd").mkdir(parents=True)
(proc / "111" / "fd" / "3").symlink_to("anon_inode:inotify")
monkeypatch.setattr(everos_server, "_PROC_ROOT", proc)
monkeypatch.setattr(everos_server.os, "getuid", lambda: 424242)
assert everos_server._inotify_usage() == (0, 128)
def test_raise_limit_writes_a_floor_and_reports_it(self, monkeypatch, tmp_path) -> None:
cap = tmp_path / "max"
cap.write_text("128\n", encoding="ascii")
monkeypatch.setattr(everos_server, "_INOTIFY_LIMIT_PATH", cap)
assert everos_server._try_raise_inotify_limit() == 1024
assert cap.read_text(encoding="ascii") == "1024"
def test_raise_limit_is_none_when_the_write_is_refused(self, monkeypatch) -> None:
class _RefusingCap:
def read_text(self, *a, **kw):
return "128"
def write_text(self, *a, **kw):
raise OSError("read-only /proc/sys")
monkeypatch.setattr(everos_server, "_INOTIFY_LIMIT_PATH", _RefusingCap())
assert everos_server._try_raise_inotify_limit() is None
def test_raise_limit_is_none_when_the_cap_is_unreadable(self, monkeypatch, tmp_path) -> None:
cap_dir = tmp_path / "maxdir"
cap_dir.mkdir()
monkeypatch.setattr(everos_server, "_INOTIFY_LIMIT_PATH", cap_dir)
assert everos_server._try_raise_inotify_limit() is None
def test_gate_is_open_with_headroom(self, monkeypatch) -> None:
monkeypatch.setattr(everos_server, "_inotify_usage", lambda: (10, 128))
monkeypatch.setattr(everos_server, "_try_raise_inotify_limit", lambda: pytest.fail("must not raise"))
assert everos_server._inotify_gate() is None
def test_gate_is_open_when_the_kernel_cannot_be_measured(self, monkeypatch) -> None:
monkeypatch.setattr(everos_server, "_inotify_usage", lambda: None)
assert everos_server._inotify_gate() is None
def test_gate_raises_the_cap_when_privileged(self, monkeypatch) -> None:
calls = {"n": 0}
def usage():
calls["n"] += 1
return (128, 128) if calls["n"] == 1 else (128, 1024)
monkeypatch.setattr(everos_server, "_inotify_usage", usage)
raised = []
monkeypatch.setattr(everos_server, "_try_raise_inotify_limit", lambda: raised.append(1) or 1024)
assert everos_server._inotify_gate() is None
assert raised == [1]
@pytest.mark.parametrize(
("usage", "expected"),
[((128, 128), "1024"), ((2048, 2048), "4096")],
ids=["under-the-floor", "at-or-above-the-floor"],
)
def test_gate_returns_the_fix_text_when_raising_fails(self, monkeypatch, usage, expected) -> None:
monkeypatch.setattr(everos_server, "_inotify_usage", lambda: usage)
monkeypatch.setattr(everos_server, "_try_raise_inotify_limit", lambda: None)
msg = everos_server._inotify_gate()
assert msg is not None
used, limit = usage
assert f"{used} of the {limit}" in msg
assert f"sudo sysctl -w fs.inotify.max_user_instances={expected}" in msg
assert "/etc/sysctl.d/99-inotify-limits.conf" in msg
@pytest.mark.asyncio
async def test_an_exhausted_cap_refuses_to_spawn(self, everos_toml, monkeypatch) -> None:
_pin_llm_role(everos_toml)
monkeypatch.setattr(
"raven_everos.server._inotify_gate",
lambda: "no inotify room; run sudo sysctl.",
)
spawn = MagicMock()
monkeypatch.setattr("raven_everos.server._start_server_if_unlocked", spawn)
with patch("raven_everos.server._probe_health", return_value=False):
with pytest.raises(RuntimeError, match="no inotify room"):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
spawn.assert_not_called()
@pytest.mark.asyncio
async def test_the_gate_is_consulted_before_a_spawn(self, everos_toml, monkeypatch) -> None:
_pin_llm_role(everos_toml)
gate = MagicMock(return_value=None)
monkeypatch.setattr("raven_everos.server._inotify_gate", gate)
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: _live_child(),
)
with patch("raven_everos.server._probe_health", side_effect=[False, True]):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
gate.assert_called_once()
@pytest.mark.asyncio
async def test_inotify_log_failure_carries_the_fix(self, everos_toml, _logs, monkeypatch) -> None:
_pin_llm_role(everos_toml)
_logs.parent.mkdir(parents=True, exist_ok=True)
_logs.write_text(
"OSError: [Errno 24] inotify instance limit reached\nApplication startup failed. Exiting.\n",
encoding="utf-8",
)
dead = MagicMock()
dead.poll.return_value = 3
dead.returncode = 3
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: dead,
)
monkeypatch.setattr(
"raven_everos.server._inotify_gate",
MagicMock(side_effect=[None, "inotify fix: sudo sysctl -w fs.inotify.max_user_instances=1024"]),
)
with patch("raven_everos.server._probe_health", return_value=False):
with pytest.raises(RuntimeError, match="inotify.*sudo sysctl"):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
@pytest.mark.asyncio
async def test_unrelated_failure_keeps_the_gate_out(self, everos_toml, _logs, monkeypatch) -> None:
_pin_llm_role(everos_toml)
_logs.parent.mkdir(parents=True, exist_ok=True)
_logs.write_text("EngineLockHeldError: held\n", encoding="utf-8")
dead = MagicMock()
dead.poll.return_value = 3
dead.returncode = 3
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: dead,
)
gate = MagicMock(return_value=None)
monkeypatch.setattr("raven_everos.server._inotify_gate", gate)
with patch("raven_everos.server._probe_health", return_value=False):
with pytest.raises(RuntimeError, match="EngineLockHeldError") as exc:
await ensure_everos_server("http://localhost:18791", timeout=5.0)
gate.assert_called_once()
assert "fs.inotify" not in str(exc.value)
class TestSpawnPreflight:
"""A server with no LLM credentials cannot survive startup, so do not spawn.
EverOS builds its LLM client eagerly during startup and fails outright when
credentials are missing. Spawning regardless burned a full poll timeout on a
process that had already exited and buried the reason in the server log.
"""
@pytest.mark.asyncio
async def test_missing_api_key_does_not_spawn(self, everos_toml, monkeypatch) -> None:
_pin_llm_role(everos_toml, api_key="")
spawned = []
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: spawned.append(1),
)
with patch("raven_everos.server._probe_health", return_value=False):
with pytest.raises(EverosNotConfiguredError, match="memory LLM is not configured"):
await ensure_everos_server("http://localhost:18791", timeout=1.0)
assert spawned == [], "spawned a server that could never finish starting"
@pytest.mark.asyncio
async def test_missing_section_entirely_does_not_spawn(self, everos_toml, monkeypatch) -> None:
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_toml.write_text('[memory]\ntimezone = "UTC"\n', encoding="utf-8")
spawned = []
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: spawned.append(1),
)
with patch("raven_everos.server._probe_health", return_value=False):
with pytest.raises(EverosNotConfiguredError):
await ensure_everos_server("http://localhost:18791", timeout=1.0)
assert spawned == []
@pytest.mark.asyncio
async def test_a_running_server_on_current_credentials_is_adopted(self, everos_toml) -> None:
"""No wait, no restart, no credential probe -- it is already what raven holds.
This used to assert something stronger and wrong: that a /health 200
proved the LLM client was good, so raven need not look further. It does
not. EverOS builds its clients at boot and /health never touches one, so
a key rotated after the boot leaves this probe green and the server
unusable. What raven may skip is the *probe*; what it may not skip is
comparing the credentials, which is the record this now writes first.
"""
_pin_llm_role(everos_toml)
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
waits: list[int] = []
stop = MagicMock()
# _speaks_our_api is stubbed too: a healthy address is also probed for the
# API prefix, and leaving that unstubbed makes the outcome depend on
# whether the developer's machine happens to have EverOS on this port.
with (
patch("raven_everos.server._probe_health", return_value=True),
patch("raven_everos.server._speaks_our_api", return_value=True),
patch("raven_everos.server.stop_for_reload", stop),
):
await ensure_everos_server("http://localhost:18791", on_wait=lambda: waits.append(1))
assert waits == [], "narrated a wait that never happened"
stop.assert_not_called()
@pytest.mark.asyncio
async def test_configured_llm_reaches_the_spawn(self, everos_toml, tmp_path, monkeypatch) -> None:
_pin_llm_role(everos_toml)
spawned = []
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: spawned.append(1),
)
monkeypatch.setattr(
"raven_everos.server.get_logs_dir",
lambda: tmp_path,
)
with patch("raven_everos.server._probe_health", side_effect=[False, True]):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
assert spawned == [1]
class TestTheRootDescribesItsOwnAddress:
"""The address goes into ``<root>/everos.toml``, not onto the command line.
A ``--port`` override left the file describing an address nobody listened on,
which is how the wizard came to probe one port while the backend talked to
another.
"""
@pytest.fixture(autouse=True)
def _no_real_spawn(self, tmp_path, monkeypatch):
monkeypatch.setattr("raven_everos.server.get_logs_dir", lambda: tmp_path)
monkeypatch.setattr("raven_everos.server.get_data_dir", lambda: tmp_path)
monkeypatch.setattr(
"raven_everos.server._everos_executable",
lambda: "/bin/true",
)
self.spawned: list[list[str]] = []
class _Proc:
pid = 4242
def poll(self):
return None
def _popen(argv, **_kw):
self.spawned.append(argv)
return _Proc()
monkeypatch.setattr("subprocess.Popen", _popen)
def test_the_declared_address_is_written_before_the_child_starts(self, everos_toml) -> None:
import tomllib
from raven_everos.server import _start_server_if_unlocked
_pin_llm_role(everos_toml)
_start_server_if_unlocked("http://localhost:18791")
with everos_toml.open("rb") as fh:
api = tomllib.load(fh)["api"]
assert api == {"host": "localhost", "port": 18791}
def test_the_child_gets_root_and_no_port(self, everos_toml) -> None:
from raven_everos.server import _start_server_if_unlocked
_pin_llm_role(everos_toml)
_start_server_if_unlocked("http://localhost:18791")
argv = self.spawned[0]
assert "--port" not in argv, "a command-line port would override the toml again"
assert "--root" in argv
assert argv[argv.index("--root") + 1] == str(everos_toml.parent)
def test_the_started_server_is_recorded(self, everos_toml, tmp_path) -> None:
import json
from raven_everos.server import _start_server_if_unlocked
_pin_llm_role(everos_toml)
_start_server_if_unlocked("http://localhost:18791")
record = json.loads((tmp_path / "everos-server.pid").read_text())
assert record["pid"] == 4242
assert record["root"] == str(everos_toml.parent)
def test_a_pidfile_naming_a_live_server_survives_a_second_start(self, everos_toml, tmp_path, monkeypatch) -> None:
"""Two starts inside one boot window: the lock covers the spawn, not
the boot, so the second child dies on the OME lock. On Windows the
pidfile is the only way back to the first server, and the loser must
not put its own pid there."""
import json
from raven_everos import server as _server
_pin_llm_role(everos_toml)
_server._write_pidfile(77, base_url="http://localhost:18791", root=everos_toml.parent)
monkeypatch.setattr(_server, "_is_everos_server", lambda pid: pid == 77)
_server._start_server_if_unlocked("http://localhost:18791")
assert json.loads((tmp_path / "everos-server.pid").read_text())["pid"] == 77
def test_a_pidfile_naming_a_dead_server_is_replaced(self, everos_toml, tmp_path, monkeypatch) -> None:
import json
from raven_everos import server as _server
_pin_llm_role(everos_toml)
_server._write_pidfile(77, base_url="http://localhost:18791", root=everos_toml.parent)
monkeypatch.setattr(_server, "_is_everos_server", lambda pid: False)
_server._start_server_if_unlocked("http://localhost:18791")
assert json.loads((tmp_path / "everos-server.pid").read_text())["pid"] == 4242
class TestThePrimitivesAgainstTheRealOS:
"""The four OS-touching helpers, unmocked.
Everything above stubs these out to stay deterministic, which leaves the
layer where a wrong assumption about flock or ``ps`` would not be caught by
anything. These run them for real.
"""
def test_an_untouched_root_reads_as_free(self, tmp_path) -> None:
from raven_everos.server import ome_lock_held
assert ome_lock_held(tmp_path / "never-started") is False
@pytest.mark.skipif(sys.platform != "linux", reason="inotify is Linux-only")
def test_zero_watch_instances_count_against_the_cap(self) -> None:
import ctypes
from raven_everos import server as _server
libc = ctypes.CDLL(None, use_errno=True)
libc.inotify_init.restype = ctypes.c_int
def mine() -> int:
"""This process's own instances, by the predicate _inotify_usage
applies across every same-UID process.
Counting ours rather than the system total is what makes the delta
reliable: the suite runs under xdist, and a neighbouring worker
closing an unrelated instance between the two snapshots would
cancel out one of the descriptors opened here.
"""
found = 0
for entry in (_server._PROC_ROOT / "self" / "fd").iterdir():
try:
if os.readlink(entry) == "anon_inode:inotify":
found += 1
except OSError:
continue
return found
before_mine = mine()
before = _server._inotify_usage()
assert before is not None
fds = []
try:
for _ in range(2):
fd = libc.inotify_init()
if fd < 0:
pytest.skip("per-user inotify cap is exhausted on this host")
fds.append(fd)
after_mine = mine()
after = _server._inotify_usage()
finally:
for fd in fds:
libc.close(fd)
assert after_mine == before_mine + 2
assert after is not None
assert after[1] == before[1]
def test_a_held_ome_lock_is_detected(self, tmp_path) -> None:
"""The signal the whole "served elsewhere" state rests on.
Also pins the documented flock caveat: the lock lives on the open file
description, so a holder in *this* process collides with itself. That is
why only the wizard and doctor -- which hold no engine -- may ask.
"""
from raven.utils.portable_lock import file_lock
from raven_everos.server import ome_lock_held
lock = tmp_path / "root" / ".index" / "sqlite" / "ome.db.lock"
lock.parent.mkdir(parents=True)
lock.touch()
with file_lock(lock, blocking=False):
assert ome_lock_held(tmp_path / "root") is True
assert ome_lock_held(tmp_path / "root") is False, "the lock outlived its holder"
def test_ps_does_not_mistake_this_process_for_a_server(self) -> None:
"""The pid-reuse guard, run against a real ``ps``.
A pidfile names a number, not a process; without this check a recycled
pid would take a SIGTERM meant for an everos server.
"""
from raven_everos.server import _is_everos_server
assert _is_everos_server(os.getpid()) is False
@pytest.mark.skipif(
shutil.which("ps") is None,
reason="no ps on PATH: _is_everos_server cannot read a command line without it",
)
def test_ps_recognises_a_process_whose_command_line_says_everos(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""The marker is found even though it sits past the 80th column.
``COLUMNS`` is pinned rather than inherited, and that is the whole test.
``ps`` truncates to it, the marker sits at the end of the command line
behind an absolute interpreter path, and so whether this passed used to
depend on how deep the checkout was: green in a clone at
``~/Raven``, red in a git worktree, whose extra
``.claude/worktrees/<name>/`` pushed the marker past column 80. A test
that a deeper directory can turn red was testing the path, not the code.
The command line is padded so the marker is past column 80 wherever the
suite runs. The bug is in `_is_everos_server`, which reads a real `ps`,
so it cannot be asserted against a stub.
"""
import subprocess
import sys
from raven_everos.server import _is_everos_server
# A stand-in whose command line carries the marker `ps` looks for; the
# point is that the parsing works against real `ps` output, not that this
# is an everos build.
#
# Deliberately not `sh -c "sleep 5 # marker"`: a shell handed a single
# simple command execs it directly, replacing its own command line and
# dropping the marker -- which made this pass alone and fail in a full
# run. A python interpreter never rewrites its argv, so the marker stays.
monkeypatch.setenv("COLUMNS", "80")
# Padded so the marker is past column 80 on any machine, rather than only
# where the interpreter path happens to be long enough. Pinning COLUMNS
# without this would leave the case vacuous in a shallow clone -- which is
# the failure this test is being rewritten out of, in the other direction.
padding = "p" * 120
proc = subprocess.Popen(
[sys.executable, "-c", f"import time; time.sleep(5) # {padding} everos server start --root /x"]
)
try:
assert _is_everos_server(proc.pid) is True
finally:
proc.terminate()
proc.wait(timeout=5)
def test_a_dead_pid_is_not_a_server(self) -> None:
import subprocess
from raven_everos.server import _is_everos_server
proc = subprocess.Popen(["sh", "-c", "exit 0"])
proc.wait(timeout=5)
assert _is_everos_server(proc.pid) is False
def test_health_probe_reads_a_real_response(self, tmp_path) -> None:
"""``_probe_health`` against a real socket: 200 is up, 500 is not, and a
closed port is not an exception the caller has to handle."""
import http.server
import threading
from raven_everos.server import _probe_health
class _Handler(http.server.BaseHTTPRequestHandler):
status = 200
def do_GET(self): # noqa: N802 - stdlib callback name
self.send_response(self.status)
self.end_headers()
def log_message(self, *_a):
return
server = http.server.HTTPServer(("127.0.0.1", 0), _Handler)
port = server.server_address[1]
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
assert _probe_health(f"http://127.0.0.1:{port}") is True
_Handler.status = 503
assert _probe_health(f"http://127.0.0.1:{port}") is False
finally:
server.shutdown()
server.server_close()
thread.join(timeout=5)
# Nothing listening any more: a refused connection is False, not a raise.
assert _probe_health(f"http://127.0.0.1:{port}") is False
def test_the_wait_budget_stays_small() -> None:
"""The timeout is a budget for how long a user waits, not a failure detector.
Once the poll loop watches the child's exit code, a boot that cannot succeed
is caught in about a second whatever this value is -- so the only thing left
to size is the wait, and overrunning it is cheap: the child keeps booting and
the next session finds it. Pinned because the number reads like a safety
margin and invites being raised back.
"""
import inspect
default = inspect.signature(ensure_everos_server).parameters["timeout"].default
assert default == 10.0
class TestDeadChildDetection:
""" "Still booting" and "already dead" are different states.
The Popen handle used to be discarded, so a child that exited in under a
second still cost the caller the full poll timeout -- 30s per session, with
the reason buried in the server log.
"""
@pytest.fixture
def _logs(self, tmp_path, monkeypatch):
monkeypatch.setattr("raven_everos.server.get_logs_dir", lambda: tmp_path)
return tmp_path / "everos-server.log"
@pytest.mark.asyncio
async def test_exited_child_fails_immediately(self, everos_toml, _logs, monkeypatch) -> None:
_pin_llm_role(everos_toml)
probes = []
def _probe(*_a, **_kw):
probes.append(1)
return False
dead = MagicMock()
dead.poll.return_value = 1
dead.returncode = 1
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: dead,
)
with patch("raven_everos.server._probe_health", side_effect=_probe):
with pytest.raises(RuntimeError, match="exited with code 1"):
await ensure_everos_server("http://localhost:18791", timeout=30.0)
# Asserting on the probe count rather than wall-clock keeps this
# deterministic: one pre-loop probe plus one inside the loop, versus the
# 61 a full 30s budget would have cost.
assert len(probes) == 2
@pytest.mark.asyncio
async def test_failure_carries_the_reason_from_the_log(self, everos_toml, _logs, monkeypatch) -> None:
_pin_llm_role(everos_toml)
_logs.parent.mkdir(parents=True, exist_ok=True)
_logs.write_text(
"some uvicorn noise\n"
"Traceback (most recent call last):\n"
' File "engine.py", line 554, in _acquire_lock\n'
"everos.infra.ome.exceptions.EngineLockHeldError: another OfflineEngine "
"instance already holds /tmp/ome.db.lock\n"
"Application startup failed. Exiting.\n",
encoding="utf-8",
)
dead = MagicMock()
dead.poll.return_value = 3
dead.returncode = 3
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: dead,
)
with patch("raven_everos.server._probe_health", return_value=False):
with pytest.raises(RuntimeError, match="EngineLockHeldError"):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
@pytest.mark.asyncio
async def test_live_child_still_gets_the_full_budget(self, everos_toml, _logs, monkeypatch) -> None:
"""A slow first boot is what the timeout exists for; do not cut it short."""
_pin_llm_role(everos_toml)
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: _live_child(),
)
probes = []
def _probe(*_a, **_kw):
probes.append(1)
return False
with patch("raven_everos.server._probe_health", side_effect=_probe):
with pytest.raises(EverosStillStartingError, match="is still starting"):
await ensure_everos_server("http://localhost:18791", timeout=1.0)
# One pre-loop probe plus one per 0.5s poll interval across a 1s budget.
assert len(probes) == 3, "timeout budget was not spent polling"
@pytest.mark.asyncio
async def test_another_process_spawning_is_not_treated_as_dead(self, everos_toml, _logs, monkeypatch) -> None:
"""No handle means someone else holds the startup lock, not a dead child."""
_pin_llm_role(everos_toml)
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: None,
)
with patch("raven_everos.server._probe_health", side_effect=[False, True]):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
@pytest.mark.asyncio
async def test_missing_log_degrades_to_no_detail(self, everos_toml, _logs, monkeypatch) -> None:
_pin_llm_role(everos_toml)
dead = MagicMock()
dead.poll.return_value = 2
dead.returncode = 2
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: dead,
)
with patch("raven_everos.server._probe_health", return_value=False):
with pytest.raises(RuntimeError, match="exited with code 2"):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
class TestEnsureEverosServer:
@pytest.mark.asyncio
async def test_server_already_running(self) -> None:
mock_response = MagicMock()
mock_response.status_code = 200
with (
patch("raven_everos.server._probe_health", return_value=True),
patch("raven_everos.server._speaks_our_api", return_value=True),
):
await ensure_everos_server("http://localhost:18791")
@pytest.mark.asyncio
async def test_a_live_server_that_cannot_serve_our_api_is_refused(self) -> None:
"""Health is not a handshake.
An older EverOS answers /health perfectly and 404s every call the client
makes afterwards. Adopting it is what made that failure silent -- a
store failure is swallowed per turn, so the pairing ran for days looking
healthy while nothing was written and nothing recalled.
"""
with (
patch("raven_everos.server._probe_health", return_value=True),
patch("raven_everos.server._speaks_our_api", return_value=False),
patch("raven_everos.server._start_server_if_unlocked") as start,
):
with pytest.raises(RuntimeError, match="too old for this raven"):
await ensure_everos_server("http://localhost:18791")
# Refused, not worked around: starting a second one on the same port
# would only fail to bind.
start.assert_not_called()
@pytest.mark.asyncio
async def test_auto_start_on_connection_error(self, tmp_path, everos_toml) -> None:
_pin_llm_role(everos_toml)
call_count = 0
def probe_side_effect(*_args, **_kwargs):
nonlocal call_count
call_count += 1
return call_count >= 2
with (
patch(
"raven_everos.server._probe_health",
side_effect=probe_side_effect,
),
patch(
"raven_everos.server._start_server_if_unlocked",
return_value=_live_child(),
) as mock_start,
patch(
"raven_everos.server.get_logs_dir",
return_value=tmp_path,
),
):
await ensure_everos_server("http://localhost:18791", timeout=10.0)
mock_start.assert_called_once()
@pytest.mark.asyncio
async def test_timeout_raises(self, tmp_path, everos_toml) -> None:
"""No child of ours to watch (another process holds the startup lock),
and the address never turns healthy, so the budget is spent and the
message says the process is still up."""
_pin_llm_role(everos_toml)
with (
patch(
"raven_everos.server._probe_health",
return_value=False,
),
patch(
"raven_everos.server._start_server_if_unlocked",
return_value=None,
),
patch(
"raven_everos.server.get_logs_dir",
return_value=tmp_path,
),
pytest.raises(EverosStillStartingError, match="is still starting"),
):
await ensure_everos_server("http://localhost:18791", timeout=0.05)
def test_port_extraction(self) -> None:
from raven_everos.server import _extract_port
assert _extract_port("http://localhost:18791") == "18791"
assert _extract_port("http://127.0.0.1:9999") == "9999"
assert _extract_port("http://localhost") == "80"
class TestProbeClassification:
"""A probe answers *why* it failed, not just that it did.
Collapsing every failure into ``False`` hid the one distinction the caller
has to act on: a refused connection is instant and means nobody is
listening, while a timeout costs the full budget and means something *is*
listening but not answering. Retrying the first is free; retrying the second
charges the user again every turn.
"""
def test_ok_on_200(self) -> None:
from raven_everos.server import ProbeVerdict, probe_health
resp = MagicMock(status_code=200)
with patch("httpx.get", return_value=resp):
assert probe_health("http://localhost:18791") is ProbeVerdict.OK
def test_refused_is_distinct_from_timeout(self) -> None:
import httpx
from raven_everos.server import ProbeVerdict, probe_health
with patch("httpx.get", side_effect=httpx.ConnectError("refused")):
assert probe_health("http://localhost:18791") is ProbeVerdict.REFUSED
with patch("httpx.get", side_effect=httpx.ConnectTimeout("slow")):
assert probe_health("http://localhost:18791") is ProbeVerdict.TIMEOUT
with patch("httpx.get", side_effect=httpx.ReadTimeout("hung")):
assert probe_health("http://localhost:18791") is ProbeVerdict.TIMEOUT
def test_non_200_is_error_not_refused(self) -> None:
from raven_everos.server import ProbeVerdict, probe_health
resp = MagicMock(status_code=503)
with patch("httpx.get", return_value=resp):
assert probe_health("http://localhost:18791") is ProbeVerdict.ERROR
def test_unexpected_exception_is_error(self) -> None:
from raven_everos.server import ProbeVerdict, probe_health
with patch("httpx.get", side_effect=ValueError("garbage")):
assert probe_health("http://localhost:18791") is ProbeVerdict.ERROR
def test_bool_wrapper_stays_truthful(self) -> None:
"""``_probe_health`` keeps its bool contract for the callers that only
need liveness. Returning the enum from it directly would be a silent
bug: every enum member is truthy, so ``if _probe_health(...)`` would
pass on a refused connection."""
import httpx
from raven_everos.server import _probe_health
resp = MagicMock(status_code=200)
with patch("httpx.get", return_value=resp):
assert _probe_health("http://localhost:18791") is True
with patch("httpx.get", side_effect=httpx.ConnectError("refused")):
assert _probe_health("http://localhost:18791") is False
with patch("httpx.get", side_effect=httpx.ReadTimeout("hung")):
assert _probe_health("http://localhost:18791") is False
def test_probe_budget_is_one_second(self) -> None:
"""Loopback: if it does not answer in a second it is not answering.
The probe runs out of band now, but it still has to finish quickly --
a probe that outlives the turn it was kicked from is a leak, not a
check.
"""
from raven_everos.server import _PROBE_TIMEOUT_S
assert _PROBE_TIMEOUT_S == 1.0
class TestLockHolderLookup:
"""Who holds this root's data, and where is it actually listening?
The pidfile answers "did raven start it", which is a record raven keeps
about itself -- lose the file and a server raven did start reads as
foreign. The lock answers "who is serving this data" from the OS, which is
the question that actually blocks a start, and it stays true across a lost
pidfile, a moved config dir, and a rename.
Finding the holder's listening port turns the one unrecoverable state --
the lock is taken and the declared address is silent -- into a recoverable
one: connect to where it really is instead of killing it.
"""
def test_returns_none_when_lock_file_absent(self, tmp_path) -> None:
from raven_everos.server import lock_holder
assert lock_holder(tmp_path) is None
def test_lsof_pid_is_verified_against_the_cmdline(self, tmp_path) -> None:
"""A pid alone is not enough: the number may have been recycled onto an
unrelated process, and signalling that is the accident this guards."""
from raven_everos import server as _server
lock = tmp_path / ".index" / "sqlite" / "ome.db.lock"
lock.parent.mkdir(parents=True)
lock.touch()
with (
patch.object(_server, "_lock_holder_pid", return_value=4242),
patch.object(_server, "_cmdline_of", return_value="/usr/bin/vim notes.txt"),
):
assert _server.lock_holder(tmp_path) is None
def test_reports_the_port_the_holder_actually_listens_on(self, tmp_path) -> None:
from raven_everos import server as _server
lock = tmp_path / ".index" / "sqlite" / "ome.db.lock"
lock.parent.mkdir(parents=True)
lock.touch()
cmdline = f"/venv/bin/python /venv/bin/everos server start --root {tmp_path}"
with (
patch.object(_server, "_lock_holder_pid", return_value=4242),
patch.object(_server, "_cmdline_of", return_value=cmdline),
patch.object(_server, "_listening_port", return_value=39999),
):
holder = _server.lock_holder(tmp_path)
assert holder is not None
assert holder.pid == 4242
assert holder.port == 39999
def test_a_holder_with_no_listening_port_is_still_reported(self, tmp_path) -> None:
"""``everos demo`` and an embedded engine hold the lock without serving
HTTP. Reporting them with ``port=None`` is what lets the wizard say
"something has your data but there is nowhere to connect" instead of
silently deciding nobody is there."""
from raven_everos import server as _server
lock = tmp_path / ".index" / "sqlite" / "ome.db.lock"
lock.parent.mkdir(parents=True)
lock.touch()
cmdline = f"/venv/bin/python /venv/bin/everos server start --root {tmp_path}"
with (
patch.object(_server, "_lock_holder_pid", return_value=4242),
patch.object(_server, "_cmdline_of", return_value=cmdline),
patch.object(_server, "_listening_port", return_value=None),
):
holder = _server.lock_holder(tmp_path)
assert holder is not None
assert holder.port is None
def test_falls_back_to_the_pidfile_when_the_os_cannot_answer(self, tmp_path) -> None:
from raven_everos import server as _server
lock = tmp_path / ".index" / "sqlite" / "ome.db.lock"
lock.parent.mkdir(parents=True)
lock.touch()
cmdline = f"/venv/bin/python /venv/bin/everos server start --root {tmp_path}"
with (
patch.object(_server, "_lsof_lock_pid", return_value=None),
patch.object(_server, "_proc_locks_pid", return_value=None),
patch.object(_server, "_read_pidfile", return_value={"pid": 77, "root": str(tmp_path)}),
patch.object(_server, "_cmdline_of", return_value=cmdline),
patch.object(_server, "_listening_port", return_value=18791),
):
holder = _server.lock_holder(tmp_path)
assert holder is not None
assert holder.pid == 77
class TestTheWrittenAddressIsVerified:
"""Writing ``[api]`` and then spawning assumes the write landed.
Dropping ``--port`` made the toml the single authority on where a server
listens; that only holds if the file really says what raven thinks it
says. An unverified write turns a failed edit into a server on the wrong
port -- exactly the drift the change was meant to end.
"""
def test_readback_mismatch_refuses_to_spawn(self, tmp_path, monkeypatch) -> None:
from raven_everos import server as _server
root = tmp_path / "everos"
root.mkdir()
(root / "everos.toml").write_text('[api]\nhost = "127.0.0.1"\nport = 8000\n')
monkeypatch.setattr(_server, "_everos_executable", lambda: "/bin/true")
monkeypatch.setattr("raven_everos.config.everos_root", lambda: root)
# A write that silently does not take: the readback is the only thing
# standing between this and a server bound to 8000 while raven probes 18791.
monkeypatch.setattr("raven_everos.config.set_everos_api", lambda **kw: None)
with pytest.raises(RuntimeError, match="did not take effect"):
_server._start_server_if_unlocked("http://localhost:18791")
class TestWindowsProcessIdentification:
"""Native Windows has no ``ps``: the command line comes from WMI through
PowerShell, and it prints the executable quoted with its extension, which
the bare marker never matched."""
_CMDLINE = '"C:\\py\\python.exe" "C:\\venv\\Scripts\\everos.exe" server start --root C:\\home\\everos-root'
def test_the_command_line_is_asked_of_wmi(self, monkeypatch) -> None:
import subprocess
from raven_everos.server import _cmdline_of, _is_everos_server
seen: list[list[str]] = []
def fake_run(argv, **kwargs):
seen.append(argv)
return subprocess.CompletedProcess(argv, 0, stdout=self._CMDLINE + "\r\n", stderr="")
monkeypatch.setattr(sys, "platform", "win32")
monkeypatch.setattr(subprocess, "run", fake_run)
assert _cmdline_of(4242) == self._CMDLINE
assert _is_everos_server(4242) is True, "the quoted .exe shape is still an everos server"
assert seen[0][0] == "powershell" and "ProcessId = 4242" in seen[0][-1]
def test_the_posix_shape_still_matches(self) -> None:
from raven_everos.server import _SERVER_CMDLINE_RE
assert _SERVER_CMDLINE_RE.search("/Users/x/.venv/bin/python3 /Users/x/.venv/bin/everos server start --root /r")
assert _SERVER_CMDLINE_RE.search("/usr/bin/vim everos-notes server start.txt") is None
def test_a_process_that_is_gone_reads_as_no_command_line(self, monkeypatch) -> None:
import subprocess
from raven_everos.server import _cmdline_of, _is_everos_server
monkeypatch.setattr(sys, "platform", "win32")
monkeypatch.setattr(
subprocess, "run", lambda argv, **kw: subprocess.CompletedProcess(argv, 0, stdout="", stderr="")
)
assert _cmdline_of(4242) == ""
assert _is_everos_server(4242) is False
def test_the_listening_port_comes_from_the_tcp_table(self, monkeypatch) -> None:
"""No lsof, no /proc: the TCP table is asked, for the launcher and its
descendants -- the socket belongs to the base interpreter two launchers
down from ``everos.exe``."""
import subprocess
from raven_everos.server import _listening_port
seen: list[list[str]] = []
def fake_run(argv, **kwargs):
seen.append(argv)
return subprocess.CompletedProcess(argv, 0, stdout="18791\r\n", stderr="")
monkeypatch.setattr(sys, "platform", "win32")
monkeypatch.setattr(subprocess, "run", fake_run)
assert _listening_port(4242) == 18791
script = seen[0][-1]
assert "$ids = @(4242)" in script and "ParentProcessId" in script and "Get-NetTCPConnection" in script
def test_no_listener_reads_as_no_port(self, monkeypatch) -> None:
import subprocess
from raven_everos.server import _listening_port
monkeypatch.setattr(sys, "platform", "win32")
monkeypatch.setattr(
subprocess, "run", lambda argv, **kw: subprocess.CompletedProcess(argv, 0, stdout="", stderr="")
)
assert _listening_port(4242) is None
def test_a_lookup_that_failed_is_not_a_process_that_is_gone(self, monkeypatch) -> None:
"""PowerShell missing or blocked, WMI wedged, the timeout hit: the
answer is unknown, and a stop keeps waiting rather than declaring the
server gone, deleting the pidfile and starting a second one against
the lock the first still holds."""
import subprocess
from raven_everos.server import _cmdline_of, _is_everos_server
def refuse(argv, **kwargs):
raise OSError("powershell is not available")
monkeypatch.setattr(sys, "platform", "win32")
monkeypatch.setattr(subprocess, "run", refuse)
assert _cmdline_of(4242) is None
assert _is_everos_server(4242) is None
def test_a_powershell_that_exits_with_an_error_is_also_unknown(self, monkeypatch) -> None:
import subprocess
from raven_everos.server import _cmdline_of
monkeypatch.setattr(sys, "platform", "win32")
monkeypatch.setattr(
subprocess, "run", lambda argv, **kw: subprocess.CompletedProcess(argv, 1, stdout="", stderr="blocked")
)
assert _cmdline_of(4242) is None
class TestStoppingOnWindows:
"""Windows has no SIGTERM. The request is Ctrl-Break to the process group
the spawn created; a server that ignores it, or one no group can be
addressed to, gets TerminateProcess -- the stop every server there had."""
@pytest.fixture(autouse=True)
def _windows(self, monkeypatch, tmp_path) -> None:
monkeypatch.setattr(sys, "platform", "win32")
# CTRL_BREAK_EVENT's value there; the attribute does not exist here.
monkeypatch.setattr(everos_server, "_GRACEFUL_SIGNAL", 21)
monkeypatch.setattr(everos_server, "_WINDOWS_GRACE_S", 0.0)
monkeypatch.setattr(everos_server.time, "sleep", lambda _s: None)
monkeypatch.setattr(everos_server, "_pidfile_path", lambda: tmp_path / "everos-server.pid")
def test_ctrl_break_first_and_terminate_only_when_ignored(self, monkeypatch) -> None:
signalled: list[int] = []
monkeypatch.setattr(everos_server.os, "kill", lambda _pid, sig: signalled.append(sig))
alive = iter([True, True, False])
monkeypatch.setattr(everos_server, "_is_everos_server", lambda _p: next(alive, False))
assert everos_server.stop_pid(4242) is everos_server.StopOutcome.STOPPED
assert signalled == [21, everos_server.signal.SIGTERM]
def test_a_server_that_acts_on_ctrl_break_is_never_terminated(self, monkeypatch) -> None:
signalled: list[int] = []
monkeypatch.setattr(everos_server.os, "kill", lambda _pid, sig: signalled.append(sig))
monkeypatch.setattr(everos_server, "_is_everos_server", lambda _p: False)
assert everos_server.stop_pid(4242) is everos_server.StopOutcome.STOPPED
assert signalled == [21]
def test_a_server_outside_any_group_is_terminated_at_once(self, monkeypatch) -> None:
"""An older raven's server, spawned without a group: Ctrl-Break has no
address, and the stop must not be reported as undeliverable."""
signalled: list[int] = []
def kill(_pid, sig):
signalled.append(sig)
if sig == 21:
raise OSError("not a process group")
monkeypatch.setattr(everos_server.os, "kill", kill)
monkeypatch.setattr(everos_server, "_is_everos_server", lambda _p: False)
assert everos_server.stop_pid(4242) is everos_server.StopOutcome.STOPPED
assert signalled == [21, everos_server.signal.SIGTERM]
def test_an_unknown_answer_keeps_the_stop_waiting(self, monkeypatch) -> None:
signalled: list[int] = []
monkeypatch.setattr(everos_server.os, "kill", lambda _pid, sig: signalled.append(sig))
monkeypatch.setattr(everos_server, "_is_everos_server", lambda _p: None)
assert everos_server.stop_pid(4242, timeout=0.05) is everos_server.StopOutcome.STILL_DRAINING
assert everos_server.signal.SIGTERM in signalled, "the bounded fallback still fires"
def test_the_child_gets_a_process_group_of_its_own(self, monkeypatch) -> None:
import subprocess
monkeypatch.setattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0x200, raising=False)
assert everos_server._spawn_kwargs() == {"creationflags": 0x200}
def test_a_posix_child_gets_a_session_of_its_own(self, monkeypatch) -> None:
monkeypatch.setattr(sys, "platform", "linux")
assert everos_server._spawn_kwargs() == {"start_new_session": True}
class TestParsingLsofListenOutput:
"""Real ``lsof`` rows, because the shape is what the parser got wrong.
The address is neither the last field -- ``(LISTEN)`` is -- nor at a fixed
index, since COMMAND is padded to the widest value in the output. Taking
the last field silently found no port at all, so a server that was plainly
listening looked like a lock holder serving nothing.
"""
def test_ipv4_row(self) -> None:
from raven_everos.server import _parse_listen_port
out = (
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n"
"python3.1 94107 admin 32u IPv4 0x88b36e3af566b544 0t0 TCP 127.0.0.1:31995 (LISTEN)\n"
)
assert _parse_listen_port(out) == 31995
def test_ipv6_row(self) -> None:
from raven_everos.server import _parse_listen_port
out = (
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n"
"everos 4242 admin 11u IPv6 0x1e2 0t0 TCP [::1]:18791 (LISTEN)\n"
)
assert _parse_listen_port(out) == 18791
def test_wildcard_row(self) -> None:
from raven_everos.server import _parse_listen_port
out = (
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n"
"everos 4242 admin 11u IPv4 0x1e2 0t0 TCP *:8000 (LISTEN)\n"
)
assert _parse_listen_port(out) == 8000
def test_header_only_is_no_port(self) -> None:
from raven_everos.server import _parse_listen_port
assert _parse_listen_port("COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n") is None
def test_empty_output_is_no_port(self) -> None:
from raven_everos.server import _parse_listen_port
assert _parse_listen_port("") is None
class TestParsingProcLocks:
"""Rows captured from a real Linux /proc/locks (kernel 6.8, Ubuntu).
This branch cannot run on the development machine, so the format is pinned
from a live capture rather than from the documentation. The waiter row is
the reason: a process blocked on the same lock is listed with a "->" prefix
that shifts every field right by one, and a parser that read the pid
positionally without noticing would hand back the waiter. The caller of
``lock_holder`` goes on to signal what it is told.
"""
@staticmethod
def _parse(rows: str, inode: int, tmp_path, monkeypatch) -> int | None:
from raven_everos import server as _server
lock = tmp_path / "ome.db.lock"
lock.touch()
monkeypatch.setattr(Path, "read_text", lambda self, **kw: rows)
monkeypatch.setattr(Path, "exists", lambda self: True)
monkeypatch.setattr(_server.Path, "stat", lambda self, **kw: SimpleNamespace(st_ino=inode))
return _server._proc_locks_pid(lock)
def test_a_holder_row(self, tmp_path, monkeypatch) -> None:
rows = "3: FLOCK ADVISORY WRITE 1550263 fc:00:4980767 0 EOF\n"
assert self._parse(rows, 4980767, tmp_path, monkeypatch) == 1550263
def test_a_waiter_never_wins_whatever_the_order(self, tmp_path, monkeypatch) -> None:
holder = "2: FLOCK ADVISORY WRITE 1550263 fc:00:4980768 0 EOF"
waiter = "2: -> FLOCK ADVISORY WRITE 1550264 fc:00:4980768 0 EOF"
for rows in (f"{holder}\n{waiter}\n", f"{waiter}\n{holder}\n"):
assert self._parse(rows, 4980768, tmp_path, monkeypatch) == 1550263
def test_an_unrelated_inode_is_not_matched(self, tmp_path, monkeypatch) -> None:
rows = "3: FLOCK ADVISORY WRITE 1550263 fc:00:4980767 0 EOF\n"
assert self._parse(rows, 999999, tmp_path, monkeypatch) is None
class TestStoppingWhatTheLockNamed:
"""The pidfile finds the target for a stop; the lock finds it for a lookup.
Letting those disagree defeats the reason the lock lookup exists. A server
raven started and then lost the pidfile for is exactly the case the lock was
added to recover, and a caller that identifies the holder that way and then
stops "the recorded server" gets NOT_OURS for a process it just named.
"""
def test_a_pid_from_the_lock_can_be_stopped(self, monkeypatch) -> None:
from raven_everos import server as _server
signalled: list[int] = []
monkeypatch.setattr(_server.os, "kill", lambda pid, sig: signalled.append(pid))
# First poll says still alive, second says gone.
alive = iter([True, False])
monkeypatch.setattr(_server, "_is_everos_server", lambda _p: next(alive, False))
monkeypatch.setattr(_server.time, "sleep", lambda _s: None)
monkeypatch.setattr(_server, "_pidfile_path", lambda: Path("/nonexistent/everos-server.pid"))
assert _server.stop_pid(4242) is _server.StopOutcome.STOPPED
assert signalled == [4242]
def test_an_undeliverable_signal_is_reported(self, monkeypatch) -> None:
from raven_everos import server as _server
def _boom(_pid, _sig):
raise PermissionError("not yours")
monkeypatch.setattr(_server.os, "kill", _boom)
assert _server.stop_pid(4242) is _server.StopOutcome.SIGNAL_FAILED
def test_a_process_that_will_not_go_is_reported_as_draining(self, monkeypatch) -> None:
from raven_everos import server as _server
monkeypatch.setattr(_server.os, "kill", lambda _p, _s: None)
monkeypatch.setattr(_server, "_is_everos_server", lambda _p: True)
monkeypatch.setattr(_server.time, "sleep", lambda _s: None)
assert _server.stop_pid(4242, timeout=0.05) is _server.StopOutcome.STILL_DRAINING
class TestAnUnwritableRootFailsAsAStartFailure:
"""A root raven cannot write to must read as "the server did not start".
The address is written into the root before the child is spawned, so a
directory that refuses the write raises OSError from deep inside the start
path. Every caller guards with ``except RuntimeError`` -- the wizard, the
backend -- because that is what "could not start" has always been. An
OSError walked past all of them and ended the wizard on a traceback, in a
situation the user can act on: fix the permissions and run it again.
"""
@pytest.mark.skipif(os.geteuid() == 0, reason="chmod 0o555 does not block root")
def test_it_surfaces_as_runtime_error(self, tmp_path, monkeypatch) -> None:
from raven_everos import server as _server
root = tmp_path / "everos"
root.mkdir()
(root / "everos.toml").write_text('[api]\nhost = "127.0.0.1"\nport = 8000\n')
root.chmod(0o555)
monkeypatch.setattr(_server, "_everos_executable", lambda: "/bin/true")
monkeypatch.setattr("raven_everos.config.everos_root", lambda: root)
monkeypatch.setattr("raven_everos.config.everos_owned", lambda: True)
try:
with pytest.raises(RuntimeError) as caught:
_server._start_server_if_unlocked("http://localhost:18791")
finally:
root.chmod(0o755)
assert "everos.toml" in str(caught.value) or "write" in str(caught.value).lower()
class TestFindingTheHolderWithoutLsof:
"""lsof is optional on Linux, and the port lookup assumed it is not.
``_proc_locks_pid`` exists because minimal container images routinely omit
lsof; ``_listening_port`` then used lsof and nothing else. Without it
``LockHolder.port`` is None, and None is documented as "holds the lock but
serves no HTTP" -- so a perfectly healthy raven-managed server is described
as a squatter to be stopped.
Preferring lsof for the holder lookup had a second effect: on any Linux box
that has lsof the /proc/locks branch never runs, including the one it was
validated on.
"""
def test_the_port_comes_from_proc_net_tcp_when_lsof_is_gone(self, monkeypatch) -> None:
from raven_everos import server as _server
# /proc/net/tcp: local_address is hex ip:port; 0x4967 == 18791.
rows = (
" sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode\n"
" 0: 0100007F:4967 00000000:0000 0A 00000000:00000000 00:00000000 00000000 1000 0 4980767 1 ...\n"
)
monkeypatch.setattr(_server, "_lsof_listening_port", lambda _p: None)
monkeypatch.setattr(_server, "_proc_net_rows", lambda: rows)
monkeypatch.setattr(_server, "_socket_inodes_of", lambda _p: {4980767})
assert _server._listening_port(4242) == 18791
def test_lsof_is_still_used_when_present(self, monkeypatch) -> None:
from raven_everos import server as _server
monkeypatch.setattr(_server, "_lsof_listening_port", lambda _p: 31995)
monkeypatch.setattr(_server, "_proc_net_rows", lambda: pytest.fail("went to /proc with lsof available"))
assert _server._listening_port(4242) == 31995
def test_proc_locks_is_preferred_on_linux(self, monkeypatch) -> None:
"""The holder/waiter distinction /proc/locks makes is worth ten lines of
comment; asking lsof first threw it away wherever lsof exists."""
from raven_everos import server as _server
monkeypatch.setattr(_server, "_proc_locks_pid", lambda _l: 111)
monkeypatch.setattr(_server, "_lsof_lock_pid", lambda _l: pytest.fail("asked lsof first"))
assert _server._lock_holder_pid(Path("/x/ome.db.lock"), Path("/x")) == 111
def test_lsof_answers_when_proc_locks_cannot(self, monkeypatch, tmp_path) -> None:
from raven_everos import server as _server
monkeypatch.setattr(_server, "_proc_locks_pid", lambda _l: None)
monkeypatch.setattr(_server, "_lsof_lock_pid", lambda _l: 222)
monkeypatch.setattr(_server, "_read_pidfile", lambda: None)
assert _server._lock_holder_pid(tmp_path / "ome.db.lock", tmp_path) == 222
class TestChildEnv:
"""What the spawned server is handed, which is the only thing it reads."""
def test_an_endpoint_the_operator_exported_is_left_alone(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""A complete endpoint in the environment is the operator's, and raven
neither replaces it nor blanks it.
Blanking is the live risk now: raven emits every role it manages, unheld
ones empty, so a role it stays silent about has to be skipped whole
rather than sent as "". The provenance set is what tells somebody else's
export from raven's own binding.
"""
for key, value in (
("EVEROS_EMBEDDING__MODEL", "theirs/model"),
("EVEROS_EMBEDDING__BASE_URL", "https://theirs/v1"),
("EVEROS_EMBEDDING__API_KEY", "sk-theirs"),
):
monkeypatch.setenv(key, value)
monkeypatch.setattr("raven_everos.config._BOUND_HERE", set())
env = everos_server._child_env()
assert env["EVEROS_EMBEDDING__MODEL"] == "theirs/model"
assert env["EVEROS_EMBEDDING__API_KEY"] == "sk-theirs"
def test_a_fragment_is_replaced_whole_rather_than_topped_up(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""Three variables from two sources is the one outcome worth avoiding:
it is neither operator's endpoint and fails in a way that names neither.
A fragment is not an endpoint, so it does not count as the operator
having taken the role over -- raven speaks for it, whole.
"""
monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "half-an-endpoint")
monkeypatch.delenv("EVEROS_EMBEDDING__BASE_URL", raising=False)
monkeypatch.delenv("EVEROS_EMBEDDING__API_KEY", raising=False)
monkeypatch.setattr(
"raven_everos.config.everos_env",
lambda: {
"EVEROS_EMBEDDING__MODEL": "ravens/model",
"EVEROS_EMBEDDING__BASE_URL": "https://ravens/v1",
"EVEROS_EMBEDDING__API_KEY": "sk-raven",
},
)
env = everos_server._child_env()
assert env["EVEROS_EMBEDDING__MODEL"] == "ravens/model"
assert env["EVEROS_EMBEDDING__BASE_URL"] == "https://ravens/v1"
def test_a_role_raven_holds_nothing_for_is_blanked_not_omitted(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""An empty value is what suppresses a section left behind in
everos.toml. Omitting the key would let the stale section -- old model,
old key -- come back into force the moment raven stopped naming one."""
monkeypatch.delenv("EVEROS_RERANK__MODEL", raising=False)
monkeypatch.setattr("raven_everos.config.role_pin", lambda _s: None)
monkeypatch.setattr("raven_everos.config.resolve_role", lambda _s: None)
env = everos_server._child_env()
assert env["EVEROS_RERANK__MODEL"] == ""
def test_a_port_inherited_from_ravens_own_environment_is_dropped(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""EverOS resolves env above the toml, so an inherited EVEROS_API__PORT
would outrank the [api] section that is meant to be the sole authority
on where a server for this root listens."""
monkeypatch.setenv("EVEROS_API__PORT", "9999")
assert "EVEROS_API__PORT" not in everos_server._child_env()
class TestRestartingForAConfigChange:
"""The chain that makes a saved model the one memory actually runs.
EverOS builds its model clients at startup, so a write alone changes
nothing until the process is replaced. What the order buys is the state
nobody can recover from: stopped and not started. Spawning happens once per
session, in ``EverosBackend.start()`` -- so a restart that stops a healthy
server and then fails leaves the machine with no memory service and nothing
that would bring one back.
"""
@staticmethod
def _wire(monkeypatch, *, precheck=None, stop=None, ensure=None):
"""Record what the chain reached, and let each step be posed."""
seen: dict[str, list] = {"stop": [], "ensure": [], "result": []}
monkeypatch.setattr(everos_server, "precheck_spawn", lambda: precheck)
def _stop(root):
seen["stop"].append(root)
return stop
monkeypatch.setattr(everos_server, "stop_for_reload", _stop)
async def _ensure(base_url, **kw):
seen["ensure"].append(base_url)
if isinstance(ensure, Exception):
raise ensure
return None
monkeypatch.setattr(everos_server, "ensure_everos_server", _ensure)
return seen
async def _run(self, seen):
await everos_server.restart_for_config_change(
"/tmp/root",
"http://127.0.0.1:18791",
on_result=lambda ok, err: seen["result"].append((ok, err)),
)
async def test_a_failing_precheck_leaves_the_old_server_running(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""The whole reason the precheck runs first. A stop here would end with
no server and no spawner, which no later save recovers from."""
seen = self._wire(monkeypatch, precheck="memory LLM is not configured")
await self._run(seen)
assert seen["stop"] == []
assert seen["ensure"] == []
assert seen["result"] == [(False, "memory LLM is not configured")]
async def test_a_draining_stop_does_not_fall_through_to_the_spawn(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""The old server is still up, so ``ensure_everos_server`` would find
the address answering, adopt it, and report success for a configuration
that never took -- this function's own bug by a different door."""
seen = self._wire(monkeypatch, stop=everos_server.StopOutcome.STILL_DRAINING)
await self._run(seen)
assert seen["ensure"] == []
assert seen["result"] == [(False, "it is still finishing memory work")]
async def test_nothing_running_still_starts_one(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""No holder is not a failure: the person just configured memory, and
starting what they configured is what the save meant."""
seen = self._wire(monkeypatch, stop=None)
monkeypatch.setattr(everos_server, "_probe_health", lambda _url: False)
await self._run(seen)
assert seen["ensure"] == ["http://127.0.0.1:18791"]
assert seen["result"] == [(True, None)]
async def test_a_server_that_answers_but_cannot_be_named_is_not_reported_applied(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A lookup that failed, or a pidfile lost on Windows: something serves
the address and the lock cannot say what. Adopting it would report the
save as applied over a server still running the old configuration."""
seen = self._wire(monkeypatch, stop=None)
monkeypatch.setattr(everos_server, "_probe_health", lambda _url: True)
await self._run(seen)
assert seen["ensure"] == []
assert seen["result"] == [
(
False,
"the process serving it could not be identified, so it was not restarted; restart the memory service yourself to pick this up.",
)
]
async def test_a_startup_failure_reaches_the_caller_verbatim(self, monkeypatch: pytest.MonkeyPatch) -> None:
seen = self._wire(
monkeypatch,
stop=everos_server.StopOutcome.STOPPED,
ensure=RuntimeError("EverOS server exited with code 1 while starting"),
)
await self._run(seen)
ok, err = seen["result"][0]
assert ok is False
assert "exited with code 1" in (err or "")
async def test_success_reports_itself(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""Reported, not inferred. A page told only about failures has nothing
that clears the banner an earlier failure left standing."""
seen = self._wire(monkeypatch, stop=everos_server.StopOutcome.STOPPED)
await self._run(seen)
assert seen["result"] == [(True, None)]
class TestThePiecesTheChainIsMadeOf:
"""`precheck_spawn` and `stop_for_reload` asked directly.
Every case above poses them through the chain, which is where the ordering
lives -- but neither is only the chain's. The wizard calls the stop, and
both are exported for anyone who has to know whether a spawn could work
before spending a stop on finding out.
"""
def test_the_precheck_names_the_missing_llm(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr("raven_everos.config.everos_role_configured", lambda _s: False)
answer = everos_server.precheck_spawn()
assert answer and "memory LLM is not configured" in answer
def test_the_precheck_answers_none_when_a_spawn_could_work(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr("raven_everos.config.everos_role_configured", lambda _s: True)
monkeypatch.setattr(everos_server, "_inotify_gate", lambda: None)
assert everos_server.precheck_spawn() is None
def test_stopping_a_root_nothing_is_serving_is_not_a_failure(
self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""None, not an outcome: there was nothing to stop, which is what lets
the chain go on and start one."""
monkeypatch.setattr(everos_server, "lock_holder", lambda _root, **_kw: None)
assert everos_server.stop_for_reload(tmp_path) is None
def test_stopping_asks_the_pid_the_lock_named(self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""Not the pidfile's: asking that would report NOT_OURS about the very
process the lock just identified, which is the state the lock lookup
exists to get out of."""
asked: list[int] = []
monkeypatch.setattr(everos_server, "lock_holder", lambda _root, **_kw: SimpleNamespace(pid=4321))
monkeypatch.setattr(
everos_server, "stop_pid", lambda pid: asked.append(pid) or everos_server.StopOutcome.STOPPED
)
assert everos_server.stop_for_reload(tmp_path) is everos_server.StopOutcome.STOPPED
assert asked == [4321]
async def test_a_chain_cancelled_mid_flight_says_so(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""A gateway shutting down during a restart. Swallowed, the page keeps
whatever the last run put on its banner and the next session inherits a
claim nobody can check."""
monkeypatch.setattr(everos_server, "precheck_spawn", lambda: None)
monkeypatch.setattr(everos_server, "stop_for_reload", lambda _root: everos_server.StopOutcome.STOPPED)
async def _ensure(*a: object, **kw: object) -> None:
raise asyncio.CancelledError
monkeypatch.setattr(everos_server, "ensure_everos_server", _ensure)
seen: list = []
with pytest.raises(asyncio.CancelledError):
await everos_server.restart_for_config_change(
"/tmp/root", "http://127.0.0.1:18791", on_result=lambda ok, err: seen.append((ok, err))
)
assert seen == [], "the chain itself reports nothing on cancellation; its caller does"
class TestARootTheUserManagesIsNeverStopped:
"""The restart chain asks about ownership before it stops anything.
`_require_owned` sits on the write primitives so a new caller cannot opt
out of it. The restart is a new caller and it is not a write primitive, so
it did: the embedding role is exempt from the ownership gate by design, and
a save of it on a user-managed root reached `stop_for_reload` and SIGTERMed
the server that root belongs to. `everos_owned`'s own docstring is the
rule -- "record the address, never touch the config, never start or stop
the process".
"""
@pytest.mark.asyncio
async def test_an_unowned_root_stops_nothing_and_says_so(self, monkeypatch: pytest.MonkeyPatch) -> None:
from raven_everos import server as everos_server
touched: list[str] = []
monkeypatch.setattr("raven_everos.config.everos_owned", lambda: False)
monkeypatch.setattr(everos_server, "precheck_spawn", lambda: touched.append("precheck") or "")
monkeypatch.setattr(everos_server, "stop_for_reload", lambda _root: touched.append("stop"))
async def _never(*a, **k):
touched.append("ensure")
monkeypatch.setattr(everos_server, "ensure_everos_server", _never)
seen: list[tuple[bool, str | None]] = []
await everos_server.restart_for_config_change(
"/tmp/theirs", "http://127.0.0.1:18791", on_result=lambda ok, why: seen.append((ok, why))
)
assert touched == [], f"the chain touched the user's server: {touched}"
assert len(seen) == 1
ok, why = seen[0]
assert ok is False
assert why is not None and "never starts or stops it" in why
@pytest.mark.asyncio
async def test_an_owned_root_still_restarts(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""The control: the guard refuses the unowned root and nothing else."""
from raven_everos import server as everos_server
touched: list[str] = []
monkeypatch.setattr("raven_everos.config.everos_owned", lambda: True)
monkeypatch.setattr(everos_server, "precheck_spawn", lambda: touched.append("precheck") or "")
monkeypatch.setattr(
everos_server,
"stop_for_reload",
lambda _root: touched.append("stop") or everos_server.StopOutcome.STOPPED,
)
async def _ensure(*a, **k):
touched.append("ensure")
monkeypatch.setattr(everos_server, "ensure_everos_server", _ensure)
seen: list[tuple[bool, str | None]] = []
await everos_server.restart_for_config_change(
"/tmp/ours", "http://127.0.0.1:18791", on_result=lambda ok, why: seen.append((ok, why))
)
assert touched == ["precheck", "stop", "ensure"]
assert seen == [(True, None)]
class TestRoleDigest:
"""Whether the server already running was handed what raven holds now.
The gap this closes: EverOS builds its model clients once, in the API
lifespan, and raven hands it credentials only as environment variables at
the spawn. Everything that edits a provider afterwards -- the models page,
``raven provider set``, a hand-edited config.json -- changes the file and
not the process, and /health cannot see the difference because it never
touches a model. So memory goes on failing against a revoked key with every
surface green.
"""
@pytest.fixture(autouse=True)
def _no_ambient_role_env(self, monkeypatch) -> None:
"""An exported role outranks raven and is skipped by ``everos_env``.
A developer who has one in their shell would otherwise be running
different cases than CI, silently.
"""
for section in ("LLM", "EMBEDDING", "RERANK", "MULTIMODAL"):
for field in ("MODEL", "BASE_URL", "API_KEY", "DIMENSIONS", "PROVIDER"):
monkeypatch.delenv(f"EVEROS_{section}__{field}", raising=False)
from raven_everos import config as ue
ue._BOUND_HERE.clear()
def test_a_rotated_key_changes_the_digest(self, everos_toml) -> None:
"""The whole premise: the digest has to move when the credential does."""
from raven_everos.config import role_env_digest
cfg = _pin_llm_role(everos_toml, api_key="old-key")
before = role_env_digest()
_pin_llm_role(everos_toml, api_key="new-key")
assert cfg.exists()
assert role_env_digest() != before
def test_the_same_config_digests_the_same(self, everos_toml) -> None:
"""Or every session would restart the server it just started."""
from raven_everos.config import role_env_digest
_pin_llm_role(everos_toml)
assert role_env_digest() == role_env_digest()
def test_a_recorded_digest_reads_as_unchanged(self, everos_toml) -> None:
_pin_llm_role(everos_toml)
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
assert everos_server.roles_changed_since_spawn(everos_toml.parent) is False
def test_a_rotated_key_reads_as_changed(self, everos_toml) -> None:
"""The reported bug, at the layer that can see it."""
_pin_llm_role(everos_toml, api_key="old-key")
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
_pin_llm_role(everos_toml, api_key="new-key")
assert everos_server.roles_changed_since_spawn(everos_toml.parent) is True
def test_no_record_reads_as_changed(self, everos_toml) -> None:
"""An install that predates the record is exactly the one nobody compared."""
_pin_llm_role(everos_toml)
everos_toml.parent.mkdir(parents=True, exist_ok=True)
assert everos_server.roles_changed_since_spawn(everos_toml.parent) is True
def test_a_root_raven_does_not_own_is_left_alone(self, everos_toml, monkeypatch) -> None:
"""raven records that server's address and never starts or stops it."""
_pin_llm_role(everos_toml)
everos_toml.parent.mkdir(parents=True, exist_ok=True)
monkeypatch.setattr("raven_everos.config.everos_owned", lambda: False)
assert everos_server.roles_changed_since_spawn(everos_toml.parent) is False
def test_an_unwritable_root_does_not_fail_the_spawn(self, everos_toml, tmp_path) -> None:
"""Bookkeeping must not cost a server that otherwise started fine."""
_pin_llm_role(everos_toml)
everos_server.record_role_digest(tmp_path / "no" / "such" / "root")
class TestStaleServerRestart:
"""``ensure_everos_server`` is the door every session passes through.
A healthy server is adopted only when it holds the credentials raven holds
now and runs the everos raven installs now; either being stale sends it
through precheck, stop and spawn.
"""
@pytest.fixture(autouse=True)
def _no_ambient_role_env(self, monkeypatch) -> None:
for section in ("LLM", "EMBEDDING", "RERANK", "MULTIMODAL"):
for field in ("MODEL", "BASE_URL", "API_KEY", "DIMENSIONS", "PROVIDER"):
monkeypatch.delenv(f"EVEROS_{section}__{field}", raising=False)
from raven_everos import config as ue
ue._BOUND_HERE.clear()
@pytest.mark.asyncio
async def test_a_server_from_before_the_upgrade_is_replaced(self, everos_toml, tmp_path, monkeypatch) -> None:
"""Raven reuses whatever answers on the port, so moving the everos pin
left the old server serving until something unrelated restarted it --
every surface green, the new adapter talking to the old substrate. The
running version is read from ``/health``, so a server nothing recorded
a digest for is covered too."""
_pin_llm_role(everos_toml)
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
monkeypatch.setattr(everos_server, "installed_everos_version", lambda: "1.4.1")
monkeypatch.setattr(everos_server, "running_everos_version", lambda _url: "1.2.3")
order: list[str] = []
monkeypatch.setattr(everos_server, "precheck_spawn", lambda: order.append("precheck") or "")
monkeypatch.setattr(
everos_server,
"stop_for_reload",
lambda _root: order.append("stop") or everos_server.StopOutcome.STOPPED,
)
monkeypatch.setattr("raven_everos.server._start_server_if_unlocked", lambda *a, **kw: order.append("spawn"))
monkeypatch.setattr("raven_everos.server.get_logs_dir", lambda: tmp_path)
with (
patch("raven_everos.server._probe_health", side_effect=[True, True]),
patch("raven_everos.server._speaks_our_api", return_value=True),
):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
assert order == ["precheck", "stop", "spawn"]
@pytest.mark.asyncio
async def test_a_server_on_the_installed_version_is_adopted(self, everos_toml, monkeypatch) -> None:
_pin_llm_role(everos_toml)
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
monkeypatch.setattr(everos_server, "installed_everos_version", lambda: "1.4.1")
monkeypatch.setattr(everos_server, "running_everos_version", lambda _url: "1.4.1")
stop = MagicMock()
monkeypatch.setattr(everos_server, "stop_for_reload", stop)
with (
patch("raven_everos.server._probe_health", return_value=True),
patch("raven_everos.server._speaks_our_api", return_value=True),
):
assert await ensure_everos_server("http://localhost:18791", timeout=5.0) is None
stop.assert_not_called()
@pytest.mark.asyncio
async def test_a_version_that_cannot_be_read_is_not_a_verdict(self, everos_toml, monkeypatch) -> None:
"""A ``/health`` without a version says nothing about staleness; the
credential comparison alone decides, and here it says adopt."""
_pin_llm_role(everos_toml)
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
monkeypatch.setattr(everos_server, "installed_everos_version", lambda: "1.4.1")
monkeypatch.setattr(everos_server, "running_everos_version", lambda _url: None)
stop = MagicMock()
monkeypatch.setattr(everos_server, "stop_for_reload", stop)
with (
patch("raven_everos.server._probe_health", return_value=True),
patch("raven_everos.server._speaks_our_api", return_value=True),
):
assert await ensure_everos_server("http://localhost:18791", timeout=5.0) is None
stop.assert_not_called()
def test_the_running_version_is_read_from_health(self, monkeypatch, real_running_everos_version) -> None:
import httpx
seen: list[str] = []
def fake_get(url, timeout):
seen.append(url)
return httpx.Response(200, json={"status": "ok", "version": "1.2.3"})
monkeypatch.setattr(httpx, "get", fake_get)
assert real_running_everos_version("http://localhost:18791") == "1.2.3"
assert seen == ["http://localhost:18791/health"]
@pytest.mark.parametrize(
"answer",
[
lambda url, timeout: (_ for _ in ()).throw(ConnectionError("gone")),
lambda url, timeout: __import__("httpx").Response(200, json={"status": "ok"}),
lambda url, timeout: __import__("httpx").Response(200, json=["not", "a", "dict"]),
],
)
def test_a_version_that_cannot_be_read_is_none(self, monkeypatch, answer, real_running_everos_version) -> None:
import httpx
monkeypatch.setattr(httpx, "get", answer)
assert real_running_everos_version("http://localhost:18791") is None
def test_the_installed_version_is_the_package_metadata(self, monkeypatch) -> None:
import importlib.metadata as md
assert everos_server.installed_everos_version() == md.version("everos")
def missing(name):
raise md.PackageNotFoundError(name)
monkeypatch.setattr(md, "version", missing)
assert everos_server.installed_everos_version() is None
def test_a_root_the_user_manages_is_never_called_stale(self, everos_toml, monkeypatch) -> None:
"""Their server, their restart: the version gap is reported nowhere here."""
monkeypatch.setattr("raven_everos.config.everos_owned", lambda: False)
monkeypatch.setattr(everos_server, "installed_everos_version", lambda: "1.4.1")
monkeypatch.setattr(everos_server, "running_everos_version", lambda _url: "1.2.3")
assert everos_server.stale_reason("http://localhost:18791", everos_toml.parent) is None
@pytest.mark.asyncio
async def test_a_stale_credential_stops_and_respawns(self, everos_toml, tmp_path, monkeypatch) -> None:
_pin_llm_role(everos_toml, api_key="old-key")
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
_pin_llm_role(everos_toml, api_key="new-key")
order: list[str] = []
monkeypatch.setattr(everos_server, "precheck_spawn", lambda: order.append("precheck") or "")
monkeypatch.setattr(
everos_server,
"stop_for_reload",
lambda _root: order.append("stop") or everos_server.StopOutcome.STOPPED,
)
monkeypatch.setattr(
"raven_everos.server._start_server_if_unlocked",
lambda *a, **kw: order.append("spawn"),
)
monkeypatch.setattr("raven_everos.server.get_logs_dir", lambda: tmp_path)
# Healthy on the first probe (the adopt check), healthy again once the
# replacement has booted.
with (
patch("raven_everos.server._probe_health", side_effect=[True, True]),
patch("raven_everos.server._speaks_our_api", return_value=True),
):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
assert order == ["precheck", "stop", "spawn"], "precheck must run before the stop"
@pytest.mark.asyncio
async def test_a_replacement_that_cannot_boot_keeps_the_old_server(self, everos_toml, monkeypatch) -> None:
"""Stopped-and-not-started is the one genuinely bad state.
A stale key still serves most of what memory asks; no server serves
none of it.
"""
_pin_llm_role(everos_toml, api_key="old-key")
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
_pin_llm_role(everos_toml, api_key="new-key")
stop = MagicMock()
spawn = MagicMock()
monkeypatch.setattr(everos_server, "precheck_spawn", lambda: "no inotify room")
monkeypatch.setattr(everos_server, "stop_for_reload", stop)
monkeypatch.setattr("raven_everos.server._start_server_if_unlocked", spawn)
with (
patch("raven_everos.server._probe_health", return_value=True),
patch("raven_everos.server._speaks_our_api", return_value=True),
):
assert await ensure_everos_server("http://localhost:18791", timeout=5.0) is None
stop.assert_not_called()
spawn.assert_not_called()
@pytest.mark.asyncio
async def test_a_stop_that_does_not_finish_is_not_adopted(self, everos_toml, monkeypatch) -> None:
"""The request is out and cannot be taken back: the server has closed its
port and is finishing what it had. Reported as ready, a gateway would
sit on it without memory until it was restarted by hand; refused, the
probes start a new one once the old one is gone."""
_pin_llm_role(everos_toml, api_key="old-key")
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
_pin_llm_role(everos_toml, api_key="new-key")
spawn = MagicMock()
monkeypatch.setattr(everos_server, "precheck_spawn", lambda: "")
monkeypatch.setattr(everos_server, "stop_for_reload", lambda _root: everos_server.StopOutcome.STILL_DRAINING)
monkeypatch.setattr("raven_everos.server._start_server_if_unlocked", spawn)
with (
patch("raven_everos.server._probe_health", return_value=True),
patch("raven_everos.server._speaks_our_api", return_value=True),
pytest.raises(RuntimeError, match="still shutting down"),
):
await ensure_everos_server("http://localhost:18791", timeout=5.0)
spawn.assert_not_called()
@pytest.mark.asyncio
async def test_a_stop_that_cannot_name_the_process_keeps_the_old_server(self, everos_toml, monkeypatch) -> None:
"""``None`` is a failure here, and is not one in ``restart_for_config_change``.
That one may run with nothing serving. This one has just probed a
healthy server two lines up, so ``None`` means the lock could not name
it -- and spawning anyway leaves the old process holding the port while
the poll below finds its ``/health`` and calls the restart a success.
Found by driving a real spawn: reading ``None`` as "nothing to stop"
left the old server running and reported success.
"""
_pin_llm_role(everos_toml, api_key="old-key")
everos_toml.parent.mkdir(parents=True, exist_ok=True)
everos_server.record_role_digest(everos_toml.parent)
_pin_llm_role(everos_toml, api_key="new-key")
spawn = MagicMock()
monkeypatch.setattr(everos_server, "precheck_spawn", lambda: "")
monkeypatch.setattr(everos_server, "stop_for_reload", lambda _root: None)
monkeypatch.setattr("raven_everos.server._start_server_if_unlocked", spawn)
with (
patch("raven_everos.server._probe_health", return_value=True),
patch("raven_everos.server._speaks_our_api", return_value=True),
):
assert await ensure_everos_server("http://localhost:18791", timeout=5.0) is None
spawn.assert_not_called()
@pytest.mark.asyncio
async def test_the_respawn_records_the_new_digest(self, everos_toml, tmp_path, monkeypatch) -> None:
"""Or the next session restarts a server that is already correct."""
from raven_everos.config import role_env_digest
_pin_llm_role(everos_toml, api_key="new-key")
root = everos_toml.parent
root.mkdir(parents=True, exist_ok=True)
monkeypatch.setattr("raven_everos.server._write_pidfile", lambda *a, **kw: None)
monkeypatch.setattr("raven_everos.server.set_everos_api", lambda **kw: None, raising=False)
everos_server.record_role_digest(root)
assert (root / everos_server._ROLE_DIGEST_FILE).read_text(encoding="utf-8") == role_env_digest()