## Summary The unit job takes twelve minutes on one runner. The first cut of this PR took the production waits out of the tests, which made the suite a quarter faster on a laptop and moved CI by 14 s (pytest 720.95 s on main, 706.57 s here): the runner is three to four times slower than a laptop on CPU-bound tests, and two of the twelve minutes are the four xdist workers importing every test module. So the PR now attacks the wall clock from every side that CI's own numbers pointed at. - `ci(*)`: the unit job runs as four shards. pytest gains `--shard K/N` in `tests/conftest.py`, decided in `pytest_ignore_collect` so a shard never imports the other shards' files; test files are dealt round-robin over their sorted list so the slow families that sort together (the ppt engine's) are spread out. Each shard writes coverage data and no report; a `coverage` job downloads the four data files, combines them, writes `coverage.json` and `coverage.xml`, and runs the gates that used to follow the tests. - `perf(ppt-engine)`: `menu()` caches its reading of a template, capped at eight entries. One bind reads the bound template and then every other bundled template for the borrow offer; with ten templates bundled, every bind evicted what the next one needed and re-measured all of them with freetype. The cap is 32 and a hit moves its entry to the back. On this PR's first CI run the template-tool file alone was 257 s of the over-ceiling list, its tests 10 to 26 s each. - `test(*)`: an autouse fixture collapses the provider chat retry ladder, the loop's LLM-error ladder and the memory store backoff to a millisecond per step, and the steer hook grace to one poll. It keeps each tuple's length because both ladders count attempts with `len()`; the steps are a millisecond rather than zero because the store pipeline waits with `wait_for(stopping.wait(), timeout=delay)`, and a zero timeout cancels that wait before it runs. A test that proves a timing property opts out with the new `production_timing` marker. Ten tests spent 257 s of the suite's worker time in those waits; none asserted on the time. - `test(*)`: a wall-clock ceiling. Setup plus call over 2 s on a test marked neither `slow` nor `production_timing` is listed at the end of the run; `--wall-clock-ceiling-strict` fails an otherwise green session on it. Warning only here; CI flips to strict once the remaining tail is marked. - `test(*)`: every test created two numbered temp directories, and pytest picks each number by scanning the base temp dir; the fake home is now a sibling of `tmp_path`. `MPLCONFIGDIR` points at a git-ignored dir in the checkout, because the fake home changed per test and matplotlib rebuilt its font list from the system fonts on every run. - `refactor(agent)`: the outer LLM-error retry ladder default `(15, 30, 60)` lived in three places; it is one constant in `raven/config/schema.py` now, read by both `RecoveryLimits` and `_ladder`. Behaviour unchanged. - `test(updates)`: the first sharded run turned up an order dependency. `beta_channel` binds `ReleaseInfo` from `raven.updates.upgrade` at import, the update-notice tests stand a fake in for that module through `sys.modules`, and `_refresh` imports `beta_channel` lazily; a worker that imported `beta_channel` for the first time inside that test got an `ImportError` swallowed as a failed refresh. The test module now imports `beta_channel` at the top. - `ci(*)`: the PR diff gate named every `.py` under `plugins-dist` as production code while coverage measures only `raven` and `raven_everos`, so a change to the ppt or design engine could never appear in the report and the gate refused every such PR on sight; the first sharded run hit it on `menu.py`. The pathspec now names `plugins-dist/everos-memory`, and a test holds every production pathspec to a package some `--cov` flag in the Makefile measures. - Two slow tests were not waits: the sandbox second-cancel test cancelled one tick too early (two pending cancels coalesce into one `CancelledError`) and now waits, under a one second timeout, for the kill to land; the two thousand-session dispatch tests wrote a span per refused enqueue through a store that opens its file per span, so tracing is off for them. Follow-up on the product side: `raven/tracing/store.py` should reuse its handle. The suite has 30 upper-bound assertions of the form `assert elapsed < X`; shortened delays can only make those looser. ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [x] CI / tooling - [ ] Refactor - [ ] Other ## Verification Whole suite, `uv run --frozen --all-extras pytest -q` (macOS, 4 workers), 22051 passed / 43 skipped each time: | tree | wall clock | unmarked tests over 2 s | |---|---|---| | main before this branch | 372.82 s | (no ceiling yet) | | after the waits were taken out | 279.65 s | 63 | | this branch | 224.25 s | 39 | The shards partition the suite exactly (`uv run --frozen --all-extras pytest --collect-only -q -n 0 --shard K/4`, K = 1..4): | whole | shard 1 | shard 2 | shard 3 | shard 4 | union | ids in two shards | |---|---|---|---|---|---|---| | 22095 | 5058 | 5555 | 5955 | 5527 | 22095 | 0 | 196 / 195 / 194 / 194 test files per shard. A shard collects in 7 s where the whole suite takes 23 s on the same machine. `--ignore` still works under `--shard`: with four `--ignore` flags on shard 1, the ignored file that shard owns is gone from its collection (5034 ids against 5058). Combining loses nothing. Two small runs with `--cov=raven --cov=raven_everos --cov-branch --cov-report=` and `COVERAGE_FILE` set, then `make coverage-combine COVERAGE_DATA_DIR=<dir>`, against one `make coverage` of the whole suite on main: | report | files | statements | branches | |---|---|---|---| | one run of the whole suite | 704 | 82634 | 25548 | | two shards combined | 704 | 82636 | 25548 | Same file set; the two extra statements are the new constant and its import. A source package's never-imported files reach the combined report through the shard that imported it, as they reached the single run. The menu cache, the three files that bind templates (`uv run --frozen --all-extras pytest -q tests/test_ppt_engine_template_tool.py tests/test_ppt_engine_outline_checks.py tests/test_ppt_engine_default_templates.py`): 101 s of worker time in the last profile, now `196 passed in 11.84s` with no test over 2.9 s. The files that call `menu()` directly: `219 passed in 8.06s`. The font cache, the formula render test at `-n 0`: 8.91 s cold, 0.69 s on the next run. Seven home-sensitive files: `484 passed in 5.30s`. Ceiling guard, both modes, with the ceiling forced low so it has something to list: ``` uv run --frozen --all-extras pytest -q -n 0 --wall-clock-ceiling=0.01 tests/test_import_cycle_budget.py tests/test_updates_install_guard.py uv run --frozen --all-extras pytest -q -n 2 --wall-clock-ceiling=0.01 --wall-clock-ceiling-strict tests/test_updates_install_guard.py ``` First: 28 passed, five listed, exit 0. Second: 24 passed, four listed, exit 1 (the controller flips the status; workers do not). `make lint-python` clean, `make lint-imports` passes. The CI run on this PR is the measurement that matters: four `unit` shards and one `coverage gates` job. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed (none: CI plumbing, a cache cap, test setup) ## Risk - The coverage gates now run in a job after the four shards. A red shard skips the gates, a deliberate trade: combining three of four data files would report a coverage drop that is not one, and the PR is red either way. The gate scripts and their thresholds are unchanged. - The diff gate no longer sees `plugins-dist/ppt-engine` or `plugins-dist/design-engine` at all, as it could never judge them; measuring them (`--cov=raven_ppt --cov=raven_design`, a new ratchet baseline) is the follow-up that would bring them under the gate. - `--shard` is opt-in; a plain `pytest` run collects everything as before. Shard data artifacts are kept one day, the combined reports 30 days as before. - No product behaviour change: the ladder default is the same tuple read from one place; the menu cap only changes how long the cache keeps entries. - Test-only risk: a test that implicitly relied on a production-length retry delay would see different timing. The full suite passed with the fixture on, and a single test can opt out with `production_timing`. - Rollback: revert the PR. The unit job goes back to one runner with the gates inline; both markers are additive. - [x] Security impact considered (no security surface touched) - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
257 lines
8.9 KiB
Python
257 lines
8.9 KiB
Python
"""Unit tests for coverage summaries, diff selection, and ratchet enforcement."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import tomllib
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from scripts.coverage_gate import (
|
|
OMITTED_PATHS,
|
|
PRODUCTION_PATHSPECS,
|
|
calculate_diff_coverage,
|
|
check_baseline_update,
|
|
check_ratchet,
|
|
create_baseline,
|
|
git_diff_inputs,
|
|
load_coverage,
|
|
parse_changed_lines,
|
|
)
|
|
|
|
|
|
def _file_summary(
|
|
*, statements: int, covered_lines: int, branches: int = 0, covered_branches: int = 0
|
|
) -> dict[str, int | float]:
|
|
denominator = statements + branches
|
|
numerator = covered_lines + covered_branches
|
|
return {
|
|
"covered_lines": covered_lines,
|
|
"num_statements": statements,
|
|
"percent_covered": 100.0 if denominator == 0 else numerator * 100.0 / denominator,
|
|
"percent_covered_display": "0",
|
|
"missing_lines": statements - covered_lines,
|
|
"excluded_lines": 0,
|
|
"num_branches": branches,
|
|
"num_partial_branches": 0,
|
|
"covered_branches": covered_branches,
|
|
"missing_branches": branches - covered_branches,
|
|
}
|
|
|
|
|
|
def _report(
|
|
*,
|
|
statements: int = 10,
|
|
covered_lines: int = 8,
|
|
branches: int = 4,
|
|
covered_branches: int = 3,
|
|
) -> dict:
|
|
summary = _file_summary(
|
|
statements=statements,
|
|
covered_lines=covered_lines,
|
|
branches=branches,
|
|
covered_branches=covered_branches,
|
|
)
|
|
return {
|
|
"meta": {"branch_coverage": True},
|
|
"files": {
|
|
"raven/example.py": {
|
|
"executed_lines": list(range(1, covered_lines + 1)),
|
|
"missing_lines": list(range(covered_lines + 1, statements + 1)),
|
|
"excluded_lines": [],
|
|
"executed_branches": [],
|
|
"missing_branches": [],
|
|
"summary": summary,
|
|
}
|
|
},
|
|
"totals": summary,
|
|
}
|
|
|
|
|
|
def test_load_coverage_rejects_non_coverage_json(tmp_path: Path) -> None:
|
|
path = tmp_path / "report.json"
|
|
path.write_text(json.dumps({"files": []}), encoding="utf-8")
|
|
|
|
with pytest.raises(ValueError, match="not a coverage.py JSON report"):
|
|
load_coverage(path)
|
|
|
|
|
|
def test_documented_omissions_match_run_and_report_configuration() -> None:
|
|
with Path("pyproject.toml").open("rb") as handle:
|
|
coverage_config = tomllib.load(handle)["tool"]["coverage"]
|
|
|
|
assert set(coverage_config["run"]["omit"]) == OMITTED_PATHS
|
|
assert set(coverage_config["report"]["omit"]) == OMITTED_PATHS
|
|
|
|
|
|
def test_parse_changed_lines_uses_only_new_hunk_ranges() -> None:
|
|
diff = """diff --git a/raven/example.py b/raven/example.py
|
|
--- a/raven/example.py
|
|
+++ b/raven/example.py
|
|
@@ -2,3 +2,4 @@
|
|
+replacement
|
|
@@ -20,0 +22,2 @@
|
|
+first
|
|
+second
|
|
diff --git a/tests/test_example.py b/tests/test_example.py
|
|
--- a/tests/test_example.py
|
|
+++ b/tests/test_example.py
|
|
@@ -1 +1 @@
|
|
-old
|
|
+new
|
|
"""
|
|
|
|
assert parse_changed_lines(diff) == {
|
|
"raven/example.py": {2, 3, 4, 5, 22, 23},
|
|
"tests/test_example.py": {1},
|
|
}
|
|
|
|
|
|
def test_parse_changed_lines_ignores_deleted_file_hunks() -> None:
|
|
diff = """diff --git a/raven/removed.py b/raven/removed.py
|
|
--- a/raven/removed.py
|
|
+++ /dev/null
|
|
@@ -1,2 +0,0 @@
|
|
-gone
|
|
-also gone
|
|
diff --git a/raven/kept.py b/raven/kept.py
|
|
--- a/raven/kept.py
|
|
+++ b/raven/kept.py
|
|
@@ -3 +3 @@
|
|
+changed
|
|
"""
|
|
|
|
assert parse_changed_lines(diff) == {"raven/kept.py": {3}}
|
|
|
|
|
|
def test_diff_coverage_ignores_comments_deletions_and_non_executable_lines() -> None:
|
|
report = _report()
|
|
changed = {"raven/example.py": {2, 9, 10, 11, 12}}
|
|
|
|
result = calculate_diff_coverage(report, changed)
|
|
|
|
assert result.executable == {("raven/example.py", 2), ("raven/example.py", 9), ("raven/example.py", 10)}
|
|
assert result.uncovered == {("raven/example.py", 9), ("raven/example.py", 10)}
|
|
assert result.percent == pytest.approx(100 / 3)
|
|
|
|
|
|
def test_diff_coverage_flags_new_production_file_at_zero_percent() -> None:
|
|
report = _report(covered_lines=0)
|
|
|
|
result = calculate_diff_coverage(report, {"raven/example.py": set(range(1, 11))}, ["raven/example.py"])
|
|
|
|
assert result.zero_coverage_files == ("raven/example.py",)
|
|
assert result.percent == 0.0
|
|
|
|
|
|
def test_diff_coverage_fails_closed_for_unmeasured_production_file() -> None:
|
|
result = calculate_diff_coverage(_report(), {"raven/new_module.py": {1}})
|
|
|
|
assert result.unmeasured_files == ("raven/new_module.py",)
|
|
|
|
|
|
def test_diff_coverage_allows_documented_omitted_entry_point() -> None:
|
|
result = calculate_diff_coverage(_report(), {"raven/__main__.py": {1}})
|
|
|
|
assert result.unmeasured_files == ()
|
|
|
|
|
|
def test_git_diff_inputs_includes_top_level_raven_files(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
def git(*args: str) -> None:
|
|
subprocess.run(["git", *args], cwd=tmp_path, check=True, capture_output=True, text=True)
|
|
|
|
git("init")
|
|
git("config", "user.email", "test@example.com")
|
|
git("config", "user.name", "test")
|
|
(tmp_path / "README.md").write_text("base\n", encoding="utf-8")
|
|
git("add", "README.md")
|
|
git("commit", "-m", "base")
|
|
base = subprocess.run(
|
|
["git", "rev-parse", "HEAD"], cwd=tmp_path, check=True, capture_output=True, text=True
|
|
).stdout.strip()
|
|
|
|
(tmp_path / "raven").mkdir()
|
|
(tmp_path / "raven" / "__init__.py").write_text("value = 1\n", encoding="utf-8")
|
|
(tmp_path / "raven" / "pkg").mkdir()
|
|
(tmp_path / "raven" / "pkg" / "mod.py").write_text("def run():\n return 1\n", encoding="utf-8")
|
|
git("add", "-A")
|
|
git("commit", "-m", "add raven sources")
|
|
|
|
monkeypatch.chdir(tmp_path)
|
|
changed, added = git_diff_inputs(base, "HEAD")
|
|
|
|
assert "raven/__init__.py" in changed
|
|
assert "raven/pkg/mod.py" in changed
|
|
assert "raven/__init__.py" in added
|
|
|
|
|
|
def test_create_baseline_records_separate_line_and_branch_metrics() -> None:
|
|
baseline = create_baseline(_report(), "abc123")
|
|
|
|
assert baseline["reference_commit"] == "abc123"
|
|
assert baseline["python"] == f"{sys.version_info.major}.{sys.version_info.minor}"
|
|
assert baseline["totals"]["line_percent"] == 80.0
|
|
assert baseline["totals"]["branch_percent"] == 75.0
|
|
assert baseline["files"]["raven/example.py"]["covered_lines"] == 8
|
|
|
|
|
|
def test_baseline_update_cannot_lower_either_metric(capsys: pytest.CaptureFixture[str]) -> None:
|
|
previous = create_baseline(_report(), "base")
|
|
proposed = create_baseline(_report(covered_lines=9, covered_branches=2), "head")
|
|
|
|
assert check_baseline_update(proposed, previous) is False
|
|
output = capsys.readouterr().out
|
|
assert "line_percent: proposed=90.000000% target=80.000000%" in output
|
|
assert "branch_percent: proposed=50.000000% target=75.000000%" in output
|
|
assert "may not lower" in output
|
|
|
|
|
|
def test_baseline_update_accepts_equal_or_higher_metrics() -> None:
|
|
previous = create_baseline(_report(), "base")
|
|
proposed = create_baseline(_report(covered_lines=9, covered_branches=4), "head")
|
|
|
|
assert check_baseline_update(proposed, previous) is True
|
|
|
|
|
|
def test_ratchet_allows_small_tolerance(capsys: pytest.CaptureFixture[str]) -> None:
|
|
baseline = create_baseline(_report(statements=10_000, covered_lines=8_001), "abc123")
|
|
current = _report(statements=10_000, covered_lines=8_000)
|
|
|
|
assert check_ratchet(current, baseline, tolerance=0.05, limit=10) is True
|
|
assert "delta=-0.01pp" in capsys.readouterr().out
|
|
|
|
|
|
def test_ratchet_reports_total_and_largest_file_decline(capsys: pytest.CaptureFixture[str]) -> None:
|
|
baseline = create_baseline(_report(), "abc123")
|
|
current = _report(covered_lines=7, covered_branches=2)
|
|
|
|
assert check_ratchet(current, baseline, tolerance=0.05, limit=10) is False
|
|
output = capsys.readouterr().out
|
|
assert "Ratchet line: current=70.00% baseline=80.00% delta=-10.00pp" in output
|
|
assert "Ratchet branch: current=50.00% baseline=75.00% delta=-25.00pp" in output
|
|
assert "raven/example.py" in output
|
|
|
|
|
|
def test_the_diff_gate_claims_only_trees_that_coverage_measures() -> None:
|
|
"""A changed file under an unmeasured tree can never appear in the report,
|
|
so a production pathspec over one fails every PR that touches it: the gate
|
|
once named all of plugins-dist while coverage measured only raven_everos."""
|
|
repo = Path(__file__).resolve().parents[1]
|
|
measured = set(re.findall(r"--cov=(\S+)", (repo / "Makefile").read_text(encoding="utf-8")))
|
|
for spec in PRODUCTION_PATHSPECS:
|
|
if spec.startswith(":(glob,exclude)"):
|
|
continue
|
|
root = repo / spec.removeprefix(":(glob)").split("/**", 1)[0]
|
|
if not root.is_dir():
|
|
pytest.fail(f"{spec} names a directory that is not in the checkout: {root}")
|
|
if (root / "__init__.py").is_file():
|
|
packages = {root.name}
|
|
else:
|
|
packages = {child.name for child in root.iterdir() if (child / "__init__.py").is_file()}
|
|
assert packages, f"{spec} names no importable package"
|
|
assert packages <= measured, f"{spec} covers {packages - measured}, which no --cov flag measures"
|