Raven-1/tests/test_litellm_setup.py
Dizhan Xue 916761ed4c
perf(*): trim the resident memory of the gateway and the web supervisor (#954)
## Summary

A gateway, and the supervisor `raven web` leaves running beside it, held
memory from startup on for things they never use or may never be asked
to use. Two of them are no longer kept, and three are loaded on first
use. On a key-less gateway with a warm home and no terminal, as `raven
web` runs it, PSS 90 seconds after start goes from 257.2 MiB to 211.8
MiB, and the supervisor's from 77.3 MiB to 35.2 MiB. Nothing on the
wire, on disk or in the page changes shape.

- **The supervisor no longer imports the command table** (77.3 to 35.2
MiB). It only restarts its gateway, but it reached its loop through
typer and the table. `raven/cli/entry.py` now hands the argv that `raven
web` and `system.upgrade` start it with, `web --supervise --port N` with
N in ASCII digits, to `serve_commands.supervise_from_argv`, which calls
`_web` with the arguments the `web` command passes; a refusal ends the
process with the code the command would have exited with. Any other argv
is parsed by the table as before. Two process-wide settings stay ahead
of the fast path in the entry point: the TLS switch, as before, and,
moved there from the table's import, the switch of a Windows process's
stdout and stderr to UTF-8 with `PYTHONIOENCODING` set for its children.
The table's own setup, the saved language and the config migration
notices, is skipped with it: the supervisor prints no translated text
and never reads the config.
- **Two tokenizer copies left by litellm's import are freed** (25.3
MiB). `litellm.utils` keeps the dict it parses from the bundled
Anthropic tokenizer beside the string it re-serializes from it, and only
the string is read: the Claude tokenizer is built from it. cl100k's
registered encoding keeps a Python dict of its ranks beside the copy
tiktoken's core encodes with; tiktoken reads the Python one only to
pickle an encoding outside the registry, since a registered one pickles
by name. `import_litellm` now drops the dict, only while the string is
there, and the Python ranks. A failure is logged at debug level and
leaves the import standing.
- **A process whose terminal cannot be asked for its colours finds the
console's host modules on its first console command** (6.2 MiB).
Registering the console feature imported every CLI module to find the
ones that host console commands. `cli_socket.register_cli` now also
accepts a function, run once on the first `console_hosts()`. That covers
a process with no terminal, as the gateway `raven web` detaches has
none, and Windows, SSH, screen or tmux, and CI, which
`_theme._can_probe` rules out. Where the terminal can be asked, as from
`raven tui`, the host modules are still found at registration:
`raven.cli._styles` asks the terminal for its colours as it is imported,
and that question has to come before the Ink screen starts.
- **The RPC models build each validator on first use** (6.0 MiB).
`_Strict`, the base every model in `raven/rpc/models.py` derives from,
sets `defer_build=True`. The reason is a comment rather than the
docstring, because pydantic copies the docstring into the JSON schema.
- **Loading the deck engine's plugin no longer loads python-pptx** (7.9
MiB). `plugins-dist/ppt-engine/raven_ppt/services/state.py` imported the
template service, and with it python-pptx, so a gateway with the plugin
installed held it even with the plugin's config slice off, which is the
default. The service is now imported where a deck is read. With the
slice enabled, python-pptx is still loaded by the check that decks can
be made.

The upgrade regression tests also use the native launcher filename
(`raven.exe` on Windows, `raven` elsewhere), so the complete
relaunch-argv and supervisor-handoff assertions run on both platforms.
This follow-up changes test expectations only.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [x] Other

## Verification

The original author's Linux performance and suite measurements below
were recorded before the Windows test-fixture follow-up and rebase. The
seven original patches remain equivalent after rebasing; the Windows
measurements below are from the final tree. No new memory or
live-provider measurements are claimed.

- **Memory:** an isolated gateway with a key-less config and
`memory.backend` null, started without a terminal on an exported copy of
the base commit and of this head, each on its own home warmed by an
earlier start, base and head runs alternating; PSS from
`/proc/self/smaps_rollup` 90 seconds after start. The base read 257.2
MiB and this head 211.8 MiB, each within 0.1 MiB from run to run. The
figure on each change above is what taking that change back out of this
head, alone, puts back; the four add up to the difference. The
supervisor's figures are its PSS under `raven web` started on each tree.
- **Behaviour, base against this head on the same trees:**
- Token ids and counts over the repo's docs, the `raven/agent` sources
and the message catalogs, through raven's two estimators, cl100k
directly and `litellm.token_counter` for Claude, GPT, GLM, DeepSeek and
Gemini model names, with decoding, copying and pickling the encoding:
identical.
- The console: the same host modules, found at registration on the base
and on the first lookup here. On a pseudo-terminal the colour query is
written during registration on both.
- Every model in `raven/rpc/models.py`: validation and serialization
schemas, fields, and the outcome of validating the same probe payloads,
identical. On the base all but eight were built at import, here none.
Sixteen threads using every model for the first time at once: no error,
one set of answers.
- The deck engine's plugin imports without python-pptx here; on the base
it loads it.
- `raven web` end to end: it starts and serves the page; after a SIGKILL
of the gateway the supervisor brings it back on the same port with
`RAVEN_SERVE_PORT_STRICT=1`; `raven web --stop` leaves no process and no
`web.json`; `web.log` carries the same lines, and the gateway's
environment is the same key by key.
- A streamed turn and a tool-calling turn on glm-5.3-flash through a
live config's LiteLLMProvider: the same reply, the same tool call, the
same prompt usage, and the same Raven estimate.
- **New tests, against the base code:** of the 33 new or changed cases,
29 fail there. Four pass there by design: the token counts, a near-miss
argv that goes to the table, a process whose terminal can be asked
finding the host modules at registration, and the streams left alone off
Windows. All 33 pass here.
- **Mutation:** each construct was removed or bent one at a time, with
the backup outside the tree, and each mutant failed the tests written
for it:
- the supervisor: no fast path, `isdigit` as the port check, extra
arguments accepted, a refusal swallowed, every refusal ending with code
1, the port ignored;
- the Windows streams: no switch, the switch after the fast path, no
`PYTHONIOENCODING`, the switch off Windows too, strict errors, no guard
around the switch;
- the tokenizer copies: none freed, the dict dropped without its string,
the string dropped too, the ranks kept (in this process and in a fresh
one), no guard around the freeing;
- the console: the finder run on every lookup, a reset keeping a finder
that never ran, lazy everywhere, eager everywhere (in this process and
in a fresh one), registration importing one of the finder's modules;
- the validators built at import; the template service imported with the
state module.
- **Directly affected and neighbouring files:** `uv run --frozen
--python 3.12 --all-extras pytest -q -p no:cacheprovider
--idle-ceiling-strict` over the seven changed test files and
`tests/test_ppt_engine_state.py`, `tests/test_ppt_engine_prepare.py`,
`tests/test_rpc_console.py`, `tests/test_cli_tui_commands.py`,
`tests/test_cli_gateway_commands.py`,
`tests/test_rpc_contract_shapes.py`, `tests/test_cli_smoke.py`,
`tests/test_updates_upgrade.py` and `tests/test_security_tls.py`. Result
at this head: 1522 passed.
- **Full suite at this head:** `uv run --frozen --python 3.12
--all-extras pytest -q -p no:cacheprovider` (repo addopts: `-n 4`,
integration and e2e deselected; proxy variables unset). Result: 3
failed, 31663 passed, 164 skipped, 2 xfailed. The three are this box's
environment and fail the same way on an exported copy of the base
commit:
`tests/test_install_script.py::test_resolve_node_dir_answers_each_case_it_exists_for`,
`tests/test_knowledge_parser_excel.py::test_a_legacy_workbook_is_converted_and_read`
(no LibreOffice) and
`tests/test_subagent_node_runtime.py::test_what_cannot_be_read_names_nothing`
(run as root).
- **Exported trees:** the same suite on exported copies of the base
commit and of this head, compared case by case: every case they share
has the same outcome, apart from two timing-sensitive cases that failed
in the base run and pass in this head's run and on repeated runs of both
trees:
`tests/test_subagent_acp.py::test_an_interrupted_first_turn_still_binds_its_session`
and
`tests/test_subagent_cli_remote.py::test_a_cancel_ends_the_agent_s_whole_process_tree_on_the_machine`.
- **Diff coverage:** `uv run --frozen --python 3.12 python
scripts/coverage_gate.py diff --base-ref origin/main --threshold 90`
over the neighbouring run's coverage. Result: 100.00%, 50 of 50
executable changed lines. The gate measures `raven/`;
`tests/test_ppt_engine_*.py` executes both imports moved in the deck
engine.
- **Lint and gates:** `make lint-python lint-imports lint-deps
lint-types check-large-files check-source-language`, and `commitlint
--from origin/main --to HEAD --config commitlint.config.cjs` plus
`scripts/check_commit_messages.py origin/main..HEAD`. Result: all exit
0; import contracts 10 kept, 0 broken.

**Native Windows follow-up (2026-10-15):** Windows 10.0.26300, Python
3.13.12, `PYTHONUTF8=1`, isolated Raven/pytest homes and a dedicated
worktree. Setup: `uv sync --frozen --python 3.13 --all-packages --extra
dev`. Imports were verified against the tested worktree. The same frozen
sync was repeated after the rebase picked up the memory-plugin
dependency update.

- Before the fix, `uv run --frozen --no-sync pytest
tests/test_rpc_system.py -n 0 --tb=short` returned **3 failed, 40
passed**. The gateway-hosted handoff, standalone handoff and dev-flag
regression each expected `raven` where native Windows correctly selected
`raven.exe`.
- After the fix, `uv run --frozen --no-sync pytest
tests/test_rpc_system.py
tests/test_cli_serve_commands.py::TestTheSupervisorStartsWithoutTheCommandTable
tests/test_cli_system_ca.py -n 0 --idle-ceiling-strict --tb=short`
returned **66 passed, 4 skipped**.
- On the final rebased tree, the following command returned **631
passed, 4 skipped**, exit 0 with the idle ceiling enforced. The four
skips require the Linux certificate store; none are counted as Windows
passes. JUnit output paths are omitted from these commands.

```powershell
uv run --frozen --no-sync pytest tests/test_rpc_system.py tests/test_cli_serve_commands.py::TestTheSupervisorStartsWithoutTheCommandTable tests/test_cli_system_ca.py tests/test_litellm_setup.py tests/test_ppt_engine_plugin.py::test_loading_the_plugin_leaves_python_pptx_for_the_first_deck tests/test_ppt_engine_state.py tests/test_rpc_cli_dispatch.py tests/test_rpc_schema_match.py -n 0 --idle-ceiling-strict --tb=short
```

- `uv run --frozen --no-sync ruff check raven evolver agents
plugins-dist tests scripts docs-site` and the matching `ruff format
--check`: passed (2,337 files formatted).
- `uv run --frozen --no-sync lint-imports`: 10 contracts kept. `uv run
--frozen --no-sync ty check raven/cli/_console_feature.py
raven/cli/commands.py raven/cli/entry.py raven/cli/serve_commands.py
raven/providers/litellm_setup.py raven/rpc/cli_socket.py
raven/rpc/models.py
plugins-dist/ppt-engine/raven_ppt/services/state.py`: passed.
- `uv run --frozen --no-sync pre-commit run --from-ref origin/main
--to-ref HEAD`, `node node_modules/@commitlint/cli/cli.js --from
origin/main --to HEAD --config commitlint.config.cjs`, `uv run --frozen
--no-sync python scripts/check_commit_messages.py origin/main..HEAD`,
`uv run --frozen --no-sync python scripts/check_large_files.py
origin/main...HEAD`, `uv run --frozen --no-sync python
scripts/check_source_language.py origin/main...HEAD`, and `git diff
--check origin/main...HEAD`: passed.

The test correction retains the complete executable path, argv,
credentials, delayed shutdown and supervisor-handoff assertions, without
adding a skip. Upgrade spawning and the supervisor loop are substituted
in these tests. No real upgrade or resident restart, live provider call,
Windows memory measurement, full Windows suite or new Linux/macOS run
was performed by this follow-up.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

No user-facing docs change, because no command, flag, setting or file
format does.

## Risk

User-visible behaviour changes:

- **The first console command in a process whose terminal cannot be
asked** imports the finder's modules then, once, on the event loop and
under the console's dispatch lock, instead of at startup: about 0.1 to
0.2 seconds on this box. A CLI module that fails to import then fails
each console command, where before it failed the process's startup.
- **Each RPC model builds its validator the first time it is used**, so
a method's first call pays for its own models, and a model whose fields
pydantic cannot build a validator for fails there rather than when
`raven.rpc.models` is imported.
- **An unhandled exception in the supervisor** reaches `web.log` as
Python's plain traceback rather than typer's formatted one.
- **The deck engine's template service is imported at the first deck
read** instead of when the plugin loads. A python-pptx that cannot be
imported therefore no longer stops the plugin from loading: with its
config slice enabled, the check that decks can be made withholds the
deck tools, and the turn hook's note on deck state, which logs an
unreadable state at debug level and goes on, meets the import error
there.
- **Code in the same process that reads the private `_mergeable_ranks`
off the registered cl100k encoding**, as the extension example in
tiktoken's README does, gets an `AttributeError` once raven has imported
litellm. In raven, its plugins and the locked dependencies, only
tiktoken's educational module does.

Left as today:

- **Any other supervisor argv**, for example one typed by hand with
`--port=N` or the flags reordered, goes through the command table and
keeps today's memory.
- **Freeing the tokenizer copies relies on litellm's and tiktoken's
internals.** The tests read each copy where it lives, check in a fresh
interpreter that litellm's own import registers the encoding whose ranks
are freed, and pin the token counts, so a release that moves a copy,
starts reading one or stops registering the encoding fails them.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

Rollback is a revert. Nothing written to disk or sent over the wire
changes.

## Related Issues

N/A
2026-10-15 11:24:20 +08:00

352 lines
13 KiB
Python

"""Unit tests for ``import_litellm`` -- banner, level restore, TTY detach."""
import io
import logging
import os
import sys
import threading
import pytest
from raven.providers import litellm_setup
from raven.providers.litellm_setup import import_litellm
_LITELLM_LOGGERS = ("LiteLLM", "LiteLLM Router", "LiteLLM Proxy")
@pytest.fixture(autouse=True)
def _first_import_again(monkeypatch):
"""Every case here is about what the first import in a process does, and
litellm is long imported by the time the suite reaches this file. Forgetting
the finished pass makes the body run for the case, as it did before the
pass was remembered."""
monkeypatch.setattr(litellm_setup, "_READY", None)
def _tty_handlers(name: str) -> list[logging.Handler]:
logger = logging.getLogger(name)
return [h for h in logger.handlers if getattr(h, "stream", None) in (sys.stderr, sys.stdout)]
def test_import_litellm_disables_banner() -> None:
module = import_litellm()
assert module.suppress_debug_info is True
def test_import_litellm_restores_logger_levels() -> None:
"""The import-time level bump must not persist, or runtime DEBUG would stop
propagating to the file sink."""
for name in _LITELLM_LOGGERS:
logging.getLogger(name).setLevel(logging.DEBUG)
import_litellm()
for name in _LITELLM_LOGGERS:
assert logging.getLogger(name).level == logging.DEBUG
def test_import_litellm_is_idempotent() -> None:
first = import_litellm()
second = import_litellm()
assert first is second
def test_a_call_after_a_finished_import_runs_none_of_the_setup(monkeypatch) -> None:
"""The window resolver reaches here once per model, so the pass is paid once
per process, not once per call: re-running it per model put a third of a
1095-model catalogue's time into this function."""
import_litellm()
entered: list[str] = []
monkeypatch.setattr(litellm_setup, "_point_oauth_tokens_at_raven", lambda: entered.append("setup"))
for _ in range(3):
import_litellm()
assert entered == []
def test_import_litellm_points_copilot_tokens_at_raven(
tmp_path,
monkeypatch,
) -> None:
"""LiteLLM's authenticator reads this in ``__init__`` and creates the
directory, so it has to be set before litellm is imported at all."""
monkeypatch.setattr("pathlib.Path.home", lambda: tmp_path)
monkeypatch.delenv("GITHUB_COPILOT_TOKEN_DIR", raising=False)
import_litellm()
assert os.environ["GITHUB_COPILOT_TOKEN_DIR"] == str(tmp_path / ".raven" / "oauth" / "github_copilot")
def test_import_litellm_keeps_an_explicit_copilot_dir(tmp_path, monkeypatch) -> None:
monkeypatch.setenv("GITHUB_COPILOT_TOKEN_DIR", str(tmp_path / "mine"))
import_litellm()
assert os.environ["GITHUB_COPILOT_TOKEN_DIR"] == str(tmp_path / "mine")
def test_import_litellm_pins_the_model_cost_map_to_the_installed_wheel(monkeypatch) -> None:
"""litellm reads this while fetching the catalogue in ``__init__``, so it
has to be published before litellm is imported at all -- and left unset it
puts an HTTP round trip on every raven startup path."""
monkeypatch.delenv("LITELLM_LOCAL_MODEL_COST_MAP", raising=False)
import_litellm()
assert os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] == "True"
def test_import_litellm_keeps_an_explicit_model_cost_map_setting(monkeypatch) -> None:
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "False")
import_litellm()
assert os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] == "False"
def test_import_litellm_detaches_terminal_handlers() -> None:
"""Every litellm import in raven is deferred, so the handler litellm installs
lands after the CLI has already stripped terminal handlers. Left attached, it
writes every record -- DEBUG included, since the stdlib intercept sets the
root level to 0 -- over whatever is on screen."""
for name in _LITELLM_LOGGERS:
logging.getLogger(name).addHandler(logging.StreamHandler(sys.stderr))
import_litellm()
for name in _LITELLM_LOGGERS:
assert _tty_handlers(name) == []
def test_import_litellm_keeps_non_terminal_handlers() -> None:
"""Only the terminal is off limits: a file or buffer sink still gets records."""
sink = logging.StreamHandler(io.StringIO())
logging.getLogger("LiteLLM").addHandler(sink)
try:
import_litellm()
assert sink in logging.getLogger("LiteLLM").handlers
finally:
logging.getLogger("LiteLLM").removeHandler(sink)
# --- Rows raven vouches for itself --------------------------------------------------
#
# 2026-09-11: DeepSeek's own id for V4.1-Flash, ``deepseek/deepseek-flash``, was
# in no LiteLLM table, so the window resolver fell back to 65,536 for a
# 1,048,576-token model and the cost recorder wrote 0. The row is registered
# into the live table at import, where the window, the output ceiling and the
# price are all read from.
def test_import_litellm_registers_the_deepseek_flash_row() -> None:
litellm = import_litellm()
row = litellm.model_cost["deepseek/deepseek-flash"]
assert row["max_input_tokens"] == 1_048_576
assert row["max_output_tokens"] == 384_000
assert row["litellm_provider"] == "deepseek"
# Peak-hour rates per token; off-peak is half, which a flat table cannot say.
assert row["input_cost_per_token"] == 0.30 / 1_000_000
assert row["output_cost_per_token"] == 1.20 / 1_000_000
def test_the_registered_row_reaches_the_window_and_ceiling_resolvers() -> None:
from raven.providers import rates
import_litellm()
assert rates.resolve_context_window("deepseek/deepseek-flash") == 1_048_576
assert rates.declared_max_output_tokens("deepseek/deepseek-flash") == 384_000
# The row's ceiling is where the resolver starts, not what it returns: a
# declaration this far inside the window still outruns any one reply, so
# what a request carries is the per-iteration bound.
assert rates.resolve_max_output_tokens("deepseek/deepseek-flash") == rates.MAX_OUTPUT_TOKENS_PER_ITERATION
def test_a_row_the_installed_catalogue_already_has_is_left_alone(monkeypatch) -> None:
"""A newer LiteLLM that learned the id wins over raven's copy."""
from raven.providers import litellm_setup
litellm = import_litellm()
theirs = {"max_input_tokens": 1, "litellm_provider": "deepseek", "mode": "chat"}
monkeypatch.setitem(litellm.model_cost, "deepseek/deepseek-flash", theirs)
litellm_setup._register_raven_model_rows(litellm)
assert litellm.model_cost["deepseek/deepseek-flash"] is theirs
def test_warm_up_in_background_imports_on_a_daemon_thread(monkeypatch) -> None:
"""The boot paths call this so the first request that needs litellm does
not pay the import; the work happens off the calling thread."""
from raven.providers import litellm_setup
seen: list[str] = []
monkeypatch.setattr(litellm_setup, "import_litellm", lambda: seen.append(threading.current_thread().name))
thread = litellm_setup.warm_up_in_background()
thread.join(timeout=5)
assert not thread.is_alive()
assert thread.daemon
assert seen == ["litellm-warmup"]
def test_warm_up_in_background_swallows_an_import_failure(monkeypatch, caplog) -> None:
"""A broken install must not take the boot thread down with it; the
on-demand import is what reports the failure to the caller that needs it."""
from raven.providers import litellm_setup
def broken() -> None:
raise ModuleNotFoundError("No module named 'litellm'")
monkeypatch.setattr(litellm_setup, "import_litellm", broken)
with caplog.at_level(logging.ERROR, logger="raven.providers.litellm_setup"):
thread = litellm_setup.warm_up_in_background()
thread.join(timeout=5)
assert not thread.is_alive()
assert "warm-up failed" in caplog.text
def test_a_second_caller_during_the_import_does_not_leave_the_levels_raised(monkeypatch) -> None:
"""The levels are raised for the duration of the import and put back after.
That only holds one thread at a time: a caller arriving inside the window
reads WARNING as the level to restore and restores it after the first caller
has put the real one back, and litellm's DEBUG and INFO stop reaching the
file sink until the process restarts. A server warms the import in the
background while it serves, so the window is an ordinary one.
"""
from raven.providers import litellm_setup
loggers = [logging.getLogger(name) for name in _LITELLM_LOGGERS]
for logger in loggers:
logger.setLevel(logging.INFO)
inside = threading.Event()
finish = threading.Event()
register = litellm_setup._register_raven_model_rows
def hold(module):
inside.set()
finish.wait(10)
register(module)
monkeypatch.setattr(litellm_setup, "_register_raven_model_rows", hold)
first = threading.Thread(target=import_litellm, name="first")
first.start()
assert inside.wait(10), "the first import never reached the seam"
second = threading.Thread(target=import_litellm, name="second")
second.start()
second.join(0.2) # long enough to read a level, were it allowed to
finish.set()
first.join(10)
second.join(10)
assert not first.is_alive() and not second.is_alive()
assert [logger.level for logger in loggers] == [logging.INFO] * 3
# litellm's import leaves two tokenizer copies resident that nothing reads: the
# dict its utils module loads Anthropic's tokenizer JSON into, kept beside the
# string the Claude tokenizer is built from, and the Python copy of cl100k's
# ranks, kept beside the one tiktoken's core encodes with.
_TEXT = "malloc_trim returns freed arenas"
def test_import_litellm_frees_both_tokenizer_copies() -> None:
import litellm.utils
import tiktoken
import_litellm()
assert litellm.utils.json_data is None
assert isinstance(litellm.utils.claude_json_str, str)
assert "_mergeable_ranks" not in vars(tiktoken.get_encoding("cl100k_base"))
# Where tiktoken keeps the ranks, read off an encoding outside the registry: a
# release that moved them would leave the line above passing with nothing freed.
unregistered = tiktoken.Encoding(name="unregistered", pat_str=r"\S+", mergeable_ranks={b"a": 0}, special_tokens={})
assert "_mergeable_ranks" in vars(unregistered)
@pytest.mark.slow
def test_litellm_s_own_import_registers_the_encoding_whose_ranks_it_frees(tmp_path) -> None:
"""The ranks are freed only if cl100k is registered by the time the import
returns, and litellm's import does that today only through its token counter.
In this process an earlier test may have registered it first, so the import is
checked alone, in a fresh interpreter."""
import subprocess
probe = (
"import tiktoken.registry as registry\n"
"from raven.providers.litellm_setup import import_litellm\n"
"import_litellm()\n"
"encoding = registry.ENCODINGS.get('cl100k_base')\n"
"print('registered by the import:', encoding is not None)\n"
"print('ranks left:', encoding is not None and '_mergeable_ranks' in vars(encoding))\n"
)
done = subprocess.run(
[sys.executable, "-P", "-c", probe], cwd=tmp_path, capture_output=True, text=True, timeout=120
)
assert "registered by the import: True" in done.stdout, done.stdout + done.stderr
assert "ranks left: False" in done.stdout, done.stdout + done.stderr
def test_every_tokenizer_counts_the_same_without_them() -> None:
"""Anthropic's tokenizer, built from the string, counts this text as 7; cl100k
counts it as 5 through raven's estimate, through litellm, and through a copy of
the encoding made by pickling, which a registered encoding does by name."""
import pickle
import tiktoken
from raven.utils.tokens import estimate_message_tokens
litellm = import_litellm()
encoding = tiktoken.get_encoding("cl100k_base")
assert encoding.encode(_TEXT) == [16561, 71125, 4780, 28634, 98767]
assert pickle.loads(pickle.dumps(encoding)).encode(_TEXT) == [16561, 71125, 4780, 28634, 98767]
assert estimate_message_tokens({"role": "user", "content": _TEXT}) == 5
assert litellm.token_counter(model="gpt-4", text=_TEXT) == 5
assert litellm.token_counter(model="claude-sonnet-4-5", text=_TEXT) == 7
def test_a_tokenizer_built_from_the_dict_itself_keeps_it(monkeypatch) -> None:
"""The dict goes only where its string form exists, since the string is what the
tokenizer is built from; a litellm that read the dict instead keeps it."""
import types
data = {"model": {"vocab": {}}}
utils = types.ModuleType("litellm.utils")
utils.json_data = data
monkeypatch.setitem(sys.modules, "litellm.utils", utils)
litellm_setup._drop_unread_tokenizer_copies()
assert utils.json_data is data
def test_a_failure_to_free_them_leaves_the_import_standing(monkeypatch) -> None:
"""Both sit in other libraries' undocumented internals, so freeing them is best effort."""
def moved() -> None:
raise AttributeError("a later release moved them")
monkeypatch.setattr(litellm_setup, "_drop_unread_tokenizer_copies", moved)
litellm = import_litellm()
assert litellm is sys.modules["litellm"]
assert litellm_setup._READY is litellm