## Summary Nothing in this repo could see a signature change reach its callers. Ruff reads one file at a time, the import contracts read edges rather than calls, and the suite only covers lines it executes. The consequence was concrete: a required `sandbox_home` argument added to `BoxliteExecutor` reached `plugins-dist/design-engine`'s render path with CI green, and a reviewer -- not a gate -- found it. `ty` answers that question. This adds it as a pinned dev dependency, a `make lint-types` target and a CI step, with **110 of its 133 rules live**: nine switched off with their measured backlog recorded beside each, fourteen shipped disabled upstream and untouched. Getting there meant clearing 81 diagnostics. This PR folds what were three stacked PRs (#422 and #423, now closed); each rule's story is a commit of its own. ### 1. The gate, at 108 rules (`chore(*): gate call correctness...`) 21 diagnostics to clear, five of them real: | Where | What | |---|---| | `evolver/orchestrator/production.py` | `design_of` annotated as taking two parameters while its docstring, its only implementation and its one call site all use three | | `raven/cli/provider_commands.py` | `dict[str, callable]` -- the builtin function, not a type, so the annotation asserted nothing | | `raven/config/schema.py` | `Config` is a `BaseSettings` but declared `env_prefix` / `env_nested_delimiter` through pydantic's `ConfigDict`, which does not carry those keys | | `raven/spine/delivery.py` | `StreamDelta.conversation_id` is optional and was used as a `dict[str, str]` key; a delta without one stored a `None` entry the close path then read as "no open stream", losing that stream's done marker | | `research_flow/support/harness_text.py` | a predicate table typed its matchers as `object`, so nothing checked they are callable | The rest were unproven invariants and annotations wider than the code: three late-bound callables asserted where a comment used to carry the claim, one closure binding the callable it guarded instead of re-reading it, four `hasattr` probes that now check what the call actually needs, and a contextmanager yielding the value it just set. ### 2. `unresolved-import` (`fix(*): resolve the imports the type gate could not`) 21 diagnostics; three were stale module paths nothing had run into: - `HarnessNode.current_core_version()` read `raven.__core_version__`, a module that exists neither here nor in the wheel `scripts/build_core_wheel.py` produces. The read sat inside `try/except ImportError`, so the except branch was the only branch and **every node `store.py` saved recorded `core_version="unknown"`**. It reads `raven.__version__` now and returns `0.1.13`. - `raven/agent/loop/wiring.py` imported `SkillForgeRouterConfig` from `raven.config.schema`; it lives in `raven.config.raven`. Under `TYPE_CHECKING`, so the annotation simply meant nothing. - three scripts imported `raven.context_engine.base` and `raven.memory_engine.base`, neither of which exists. **Two failed at module import** -- running them was not possible. The other eighteen were the environment: the three agent products keep their plugin package beside them, so their roots are declared in `[tool.ty.environment]`; `lxml` ships a compiled `etree` with no type information, so `lxml-stubs` is a dev dependency (real types rather than five silenced lines); `uno` and `com.sun.star.*` are suppressed per line, since that worker runs inside LibreOffice's own interpreter. ### 3. `invalid-return-type` (`fix(*): make the declared return types true`) 39 diagnostics, and **one stale annotation held eight of them**: `run_turn` and the `_run_turn` behind it were declared `-> LoopOutcome` while all three of their returns build a `TurnOutcome` -- a different dataclass with no field in common. Nothing broke, because the spine worker reads what it actually gets and `LoopOutcome` never leaves `_run_agent_loop`. The annotation was the only thing wrong. The other 31: nine annotations narrower than the code (five `return tuple(<genexpr>)` typed as fixed-arity tuples, an inner `one()` still declaring the two-tuple it returned before a third element was added, `image_block`'s TypedDict), five declarations refusing a value the function returns (four tool `execute` overrides saying `-> str` while returning `ToolResult`), seven invariants that lived in a `# type: ignore` comment, and ten honest widenings. ### 4. What the gates then caught in this PR's own work - The qq factory guard passed locally only because an earlier test had left an event loop behind. Reproduced the CI state (`asyncio.set_event_loop(None)`), then made the test async so the loop is its own. - `invalid-return-type`, while still disabled, hid a regression introduced here: the qq factory returning the class instead of an instance. Found by diffing the rule histogram, which is why that rule is on now. - The diff-coverage gate named four lines in `raven/routing/cache.py` -- the asserts this work had just added, sitting inside a `try` whose `except Exception` would have swallowed the `AssertionError` and served the stale cache. `_do_refresh` returns its data now; the asserts, their noqa and the swallowing all go with it. `child_scope`, `QQChannel.start()` and `BenchmarkCache` had no tests at all and have them now. Every one was mutation-checked. ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [x] CI / tooling - [ ] Refactor - [ ] Other ## Verification - `make lint-types` -> All checks passed; 81 diagnostics cleared across the three rule sets - Gate proven live three times, once per rule: a missing argument, an import of a module that does not exist, and a function returning a str under `-> int` each turn it red, and removing the probe turns it green - `uv run ruff check` + `ruff format --check` over all six trees -> clean - `uv run lint-imports` -> 10 kept, 0 broken; `deptry raven` (all extras) -> no issues; `uv lock --check` -> resolved - `make check-source-language` -> exit 0; `commitlint` -> exit 0 - `make coverage` locally: **22050 passed, 45 skipped, 4 failed**; diff coverage 92.59% - All four failures are **pre-existing on `main`**, checked by a real checkout in the same environment where they fail identically: two `test_the_products_tool_face_*` launcher tests, `test_a_ceiling_as_large_as_the_window_still_leaves_room_for_history` (`reserved_tools = 6044 > 6_000`), and `test_a_missing_key_is_said_once_rather_than_per_query` (environment-dependent: it asserts `SERPER_API_KEY` is absent) - Rule counts measured with both `--extra dev` and `--all-extras` -- identical, so this gate's verdict does not depend on which optional packages are installed - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk Behaviour changes are confined to the fixes listed above, and each replaces an error path with a correct one: nodes are stamped with the real raven version instead of `"unknown"`, two scripts that could not start now start, the stream bookkeeping no longer stores a `None` key, and a cache refresh returns what it fetched. Everything else is a declaration moved to match what the function already returned. `tests/` is out of the gate's scope (113 diagnostics under the same rules). Tests are executed, so a broken call there is already loud; this gate is for code CI does not run. Bringing them in is sized, separate work. Rollback: revert the whole PR to drop the gate and its fixes together, or delete the `[tool.ty]` block, the `lint-types` target and its CI step to keep the fixes and drop only the gate. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
83 lines
2.8 KiB
Python
83 lines
2.8 KiB
Python
"""What BenchmarkCache.load() hands back on each of its four routes.
|
|
|
|
The class had no test. Its two refresh-success routes are the ones that decide
|
|
whether a router sees fresh numbers or yesterday's, and both used to read the
|
|
attribute back after the refresh had set it rather than using what the refresh
|
|
returned -- a shape that could only be spelled with a suppression.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import time
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from raven.routing import cache as cache_mod
|
|
from raven.routing.types import ModelBenchmark, ModelTaskScore
|
|
|
|
|
|
def _bench(model: str, score: float) -> ModelBenchmark:
|
|
return ModelBenchmark(
|
|
model=model,
|
|
provider="acme",
|
|
overall_score=score,
|
|
speed=1.0,
|
|
cost=1.0,
|
|
task_scores=[ModelTaskScore(task_id="t", score=score, max_score=100.0)],
|
|
submission_id=f"sub-{model}",
|
|
)
|
|
|
|
|
|
def _write_cache(path: Path, data: dict[str, ModelBenchmark], fetched_at: float) -> None:
|
|
raw = cache_mod._serialize(data)
|
|
raw["fetched_at"] = fetched_at
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
path.write_text(json.dumps(raw), encoding="utf-8")
|
|
|
|
|
|
@pytest.fixture
|
|
def fresh_from_api(monkeypatch):
|
|
"""Whatever the API would return, without going near it."""
|
|
data = {"acme/new": _bench("acme/new", 90.0)}
|
|
|
|
async def _build():
|
|
return data
|
|
|
|
monkeypatch.setattr(cache_mod, "build_benchmark_data", _build)
|
|
return data
|
|
|
|
|
|
async def test_no_cache_on_disk_serves_what_the_refresh_returned(tmp_path, fresh_from_api) -> None:
|
|
cache = cache_mod.BenchmarkCache(cache_path=tmp_path / "benchmark-cache.json")
|
|
|
|
assert await cache.load() == fresh_from_api
|
|
|
|
|
|
async def test_a_stale_cache_is_replaced_by_the_refresh_rather_than_served(tmp_path, fresh_from_api) -> None:
|
|
"""Stale means "try the API first"; the disk copy is the fallback, not the answer."""
|
|
path = tmp_path / "benchmark-cache.json"
|
|
_write_cache(path, {"acme/old": _bench("acme/old", 10.0)}, time.time() - cache_mod.CACHE_TTL_S - 1)
|
|
cache = cache_mod.BenchmarkCache(cache_path=path)
|
|
|
|
assert await cache.load() == fresh_from_api
|
|
|
|
|
|
async def test_a_stale_cache_is_served_when_the_refresh_raises(tmp_path, monkeypatch) -> None:
|
|
"""The contrast to the test above: only a failed refresh makes the disk copy
|
|
the answer, and yesterday's numbers beat no numbers."""
|
|
stale = {"acme/old": _bench("acme/old", 10.0)}
|
|
path = tmp_path / "benchmark-cache.json"
|
|
_write_cache(path, stale, time.time() - cache_mod.CACHE_TTL_S - 1)
|
|
|
|
async def _fail():
|
|
raise RuntimeError("the benchmark endpoint is down")
|
|
|
|
monkeypatch.setattr(cache_mod, "build_benchmark_data", _fail)
|
|
cache = cache_mod.BenchmarkCache(cache_path=path)
|
|
|
|
loaded = await cache.load()
|
|
|
|
assert list(loaded) == ["acme/old"]
|