Raven-1/tests/test_coverage_gate.py
KT a5f84b7164
ci(*): run unit tests as four shards and speed up the suite (#413)
## Summary

The unit job takes twelve minutes on one runner. The first cut of this
PR took the production waits out of the tests, which made the suite a
quarter faster on a laptop and moved CI by 14 s (pytest 720.95 s on
main, 706.57 s here): the runner is three to four times slower than a
laptop on CPU-bound tests, and two of the twelve minutes are the four
xdist workers importing every test module. So the PR now attacks the
wall clock from every side that CI's own numbers pointed at.

- `ci(*)`: the unit job runs as four shards. pytest gains `--shard K/N`
in `tests/conftest.py`, decided in `pytest_ignore_collect` so a shard
never imports the other shards' files; test files are dealt round-robin
over their sorted list so the slow families that sort together (the ppt
engine's) are spread out. Each shard writes coverage data and no report;
a `coverage` job downloads the four data files, combines them, writes
`coverage.json` and `coverage.xml`, and runs the gates that used to
follow the tests.
- `perf(ppt-engine)`: `menu()` caches its reading of a template, capped
at eight entries. One bind reads the bound template and then every other
bundled template for the borrow offer; with ten templates bundled, every
bind evicted what the next one needed and re-measured all of them with
freetype. The cap is 32 and a hit moves its entry to the back. On this
PR's first CI run the template-tool file alone was 257 s of the
over-ceiling list, its tests 10 to 26 s each.
- `test(*)`: an autouse fixture collapses the provider chat retry
ladder, the loop's LLM-error ladder and the memory store backoff to a
millisecond per step, and the steer hook grace to one poll. It keeps
each tuple's length because both ladders count attempts with `len()`;
the steps are a millisecond rather than zero because the store pipeline
waits with `wait_for(stopping.wait(), timeout=delay)`, and a zero
timeout cancels that wait before it runs. A test that proves a timing
property opts out with the new `production_timing` marker. Ten tests
spent 257 s of the suite's worker time in those waits; none asserted on
the time.
- `test(*)`: a wall-clock ceiling. Setup plus call over 2 s on a test
marked neither `slow` nor `production_timing` is listed at the end of
the run; `--wall-clock-ceiling-strict` fails an otherwise green session
on it. Warning only here; CI flips to strict once the remaining tail is
marked.
- `test(*)`: every test created two numbered temp directories, and
pytest picks each number by scanning the base temp dir; the fake home is
now a sibling of `tmp_path`. `MPLCONFIGDIR` points at a git-ignored dir
in the checkout, because the fake home changed per test and matplotlib
rebuilt its font list from the system fonts on every run.
- `refactor(agent)`: the outer LLM-error retry ladder default `(15, 30,
60)` lived in three places; it is one constant in
`raven/config/schema.py` now, read by both `RecoveryLimits` and
`_ladder`. Behaviour unchanged.
- `test(updates)`: the first sharded run turned up an order dependency.
`beta_channel` binds `ReleaseInfo` from `raven.updates.upgrade` at
import, the update-notice tests stand a fake in for that module through
`sys.modules`, and `_refresh` imports `beta_channel` lazily; a worker
that imported `beta_channel` for the first time inside that test got an
`ImportError` swallowed as a failed refresh. The test module now imports
`beta_channel` at the top.
- `ci(*)`: the PR diff gate named every `.py` under `plugins-dist` as
production code while coverage measures only `raven` and `raven_everos`,
so a change to the ppt or design engine could never appear in the report
and the gate refused every such PR on sight; the first sharded run hit
it on `menu.py`. The pathspec now names `plugins-dist/everos-memory`,
and a test holds every production pathspec to a package some `--cov`
flag in the Makefile measures.
- Two slow tests were not waits: the sandbox second-cancel test
cancelled one tick too early (two pending cancels coalesce into one
`CancelledError`) and now waits, under a one second timeout, for the
kill to land; the two thousand-session dispatch tests wrote a span per
refused enqueue through a store that opens its file per span, so tracing
is off for them. Follow-up on the product side: `raven/tracing/store.py`
should reuse its handle.

The suite has 30 upper-bound assertions of the form `assert elapsed <
X`; shortened delays can only make those looser.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [x] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Whole suite, `uv run --frozen --all-extras pytest -q` (macOS, 4
workers), 22051 passed / 43 skipped each time:

| tree | wall clock | unmarked tests over 2 s |
|---|---|---|
| main before this branch | 372.82 s | (no ceiling yet) |
| after the waits were taken out | 279.65 s | 63 |
| this branch | 224.25 s | 39 |

The shards partition the suite exactly (`uv run --frozen --all-extras
pytest --collect-only -q -n 0 --shard K/4`, K = 1..4):

| whole | shard 1 | shard 2 | shard 3 | shard 4 | union | ids in two
shards |
|---|---|---|---|---|---|---|
| 22095 | 5058 | 5555 | 5955 | 5527 | 22095 | 0 |

196 / 195 / 194 / 194 test files per shard. A shard collects in 7 s
where the whole suite takes 23 s on the same machine. `--ignore` still
works under `--shard`: with four `--ignore` flags on shard 1, the
ignored file that shard owns is gone from its collection (5034 ids
against 5058).

Combining loses nothing. Two small runs with `--cov=raven
--cov=raven_everos --cov-branch --cov-report=` and `COVERAGE_FILE` set,
then `make coverage-combine COVERAGE_DATA_DIR=<dir>`, against one `make
coverage` of the whole suite on main:

| report | files | statements | branches |
|---|---|---|---|
| one run of the whole suite | 704 | 82634 | 25548 |
| two shards combined | 704 | 82636 | 25548 |

Same file set; the two extra statements are the new constant and its
import. A source package's never-imported files reach the combined
report through the shard that imported it, as they reached the single
run.

The menu cache, the three files that bind templates (`uv run --frozen
--all-extras pytest -q tests/test_ppt_engine_template_tool.py
tests/test_ppt_engine_outline_checks.py
tests/test_ppt_engine_default_templates.py`): 101 s of worker time in
the last profile, now `196 passed in 11.84s` with no test over 2.9 s.
The files that call `menu()` directly: `219 passed in 8.06s`.

The font cache, the formula render test at `-n 0`: 8.91 s cold, 0.69 s
on the next run. Seven home-sensitive files: `484 passed in 5.30s`.

Ceiling guard, both modes, with the ceiling forced low so it has
something to list:

```
uv run --frozen --all-extras pytest -q -n 0 --wall-clock-ceiling=0.01 tests/test_import_cycle_budget.py tests/test_updates_install_guard.py
uv run --frozen --all-extras pytest -q -n 2 --wall-clock-ceiling=0.01 --wall-clock-ceiling-strict tests/test_updates_install_guard.py
```

First: 28 passed, five listed, exit 0. Second: 24 passed, four listed,
exit 1 (the controller flips the status; workers do not).

`make lint-python` clean, `make lint-imports` passes. The CI run on this
PR is the measurement that matters: four `unit` shards and one `coverage
gates` job.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed (none: CI
plumbing, a cache cap, test setup)

## Risk

- The coverage gates now run in a job after the four shards. A red shard
skips the gates, a deliberate trade: combining three of four data files
would report a coverage drop that is not one, and the PR is red either
way. The gate scripts and their thresholds are unchanged.
- The diff gate no longer sees `plugins-dist/ppt-engine` or
`plugins-dist/design-engine` at all, as it could never judge them;
measuring them (`--cov=raven_ppt --cov=raven_design`, a new ratchet
baseline) is the follow-up that would bring them under the gate.
- `--shard` is opt-in; a plain `pytest` run collects everything as
before. Shard data artifacts are kept one day, the combined reports 30
days as before.
- No product behaviour change: the ladder default is the same tuple read
from one place; the menu cap only changes how long the cache keeps
entries.
- Test-only risk: a test that implicitly relied on a production-length
retry delay would see different timing. The full suite passed with the
fixture on, and a single test can opt out with `production_timing`.
- Rollback: revert the PR. The unit job goes back to one runner with the
gates inline; both markers are additive.

- [x] Security impact considered (no security surface touched)
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-15 16:00:49 +08:00

257 lines
8.9 KiB
Python

"""Unit tests for coverage summaries, diff selection, and ratchet enforcement."""
from __future__ import annotations
import json
import re
import subprocess
import sys
import tomllib
from pathlib import Path
import pytest
from scripts.coverage_gate import (
OMITTED_PATHS,
PRODUCTION_PATHSPECS,
calculate_diff_coverage,
check_baseline_update,
check_ratchet,
create_baseline,
git_diff_inputs,
load_coverage,
parse_changed_lines,
)
def _file_summary(
*, statements: int, covered_lines: int, branches: int = 0, covered_branches: int = 0
) -> dict[str, int | float]:
denominator = statements + branches
numerator = covered_lines + covered_branches
return {
"covered_lines": covered_lines,
"num_statements": statements,
"percent_covered": 100.0 if denominator == 0 else numerator * 100.0 / denominator,
"percent_covered_display": "0",
"missing_lines": statements - covered_lines,
"excluded_lines": 0,
"num_branches": branches,
"num_partial_branches": 0,
"covered_branches": covered_branches,
"missing_branches": branches - covered_branches,
}
def _report(
*,
statements: int = 10,
covered_lines: int = 8,
branches: int = 4,
covered_branches: int = 3,
) -> dict:
summary = _file_summary(
statements=statements,
covered_lines=covered_lines,
branches=branches,
covered_branches=covered_branches,
)
return {
"meta": {"branch_coverage": True},
"files": {
"raven/example.py": {
"executed_lines": list(range(1, covered_lines + 1)),
"missing_lines": list(range(covered_lines + 1, statements + 1)),
"excluded_lines": [],
"executed_branches": [],
"missing_branches": [],
"summary": summary,
}
},
"totals": summary,
}
def test_load_coverage_rejects_non_coverage_json(tmp_path: Path) -> None:
path = tmp_path / "report.json"
path.write_text(json.dumps({"files": []}), encoding="utf-8")
with pytest.raises(ValueError, match="not a coverage.py JSON report"):
load_coverage(path)
def test_documented_omissions_match_run_and_report_configuration() -> None:
with Path("pyproject.toml").open("rb") as handle:
coverage_config = tomllib.load(handle)["tool"]["coverage"]
assert set(coverage_config["run"]["omit"]) == OMITTED_PATHS
assert set(coverage_config["report"]["omit"]) == OMITTED_PATHS
def test_parse_changed_lines_uses_only_new_hunk_ranges() -> None:
diff = """diff --git a/raven/example.py b/raven/example.py
--- a/raven/example.py
+++ b/raven/example.py
@@ -2,3 +2,4 @@
+replacement
@@ -20,0 +22,2 @@
+first
+second
diff --git a/tests/test_example.py b/tests/test_example.py
--- a/tests/test_example.py
+++ b/tests/test_example.py
@@ -1 +1 @@
-old
+new
"""
assert parse_changed_lines(diff) == {
"raven/example.py": {2, 3, 4, 5, 22, 23},
"tests/test_example.py": {1},
}
def test_parse_changed_lines_ignores_deleted_file_hunks() -> None:
diff = """diff --git a/raven/removed.py b/raven/removed.py
--- a/raven/removed.py
+++ /dev/null
@@ -1,2 +0,0 @@
-gone
-also gone
diff --git a/raven/kept.py b/raven/kept.py
--- a/raven/kept.py
+++ b/raven/kept.py
@@ -3 +3 @@
+changed
"""
assert parse_changed_lines(diff) == {"raven/kept.py": {3}}
def test_diff_coverage_ignores_comments_deletions_and_non_executable_lines() -> None:
report = _report()
changed = {"raven/example.py": {2, 9, 10, 11, 12}}
result = calculate_diff_coverage(report, changed)
assert result.executable == {("raven/example.py", 2), ("raven/example.py", 9), ("raven/example.py", 10)}
assert result.uncovered == {("raven/example.py", 9), ("raven/example.py", 10)}
assert result.percent == pytest.approx(100 / 3)
def test_diff_coverage_flags_new_production_file_at_zero_percent() -> None:
report = _report(covered_lines=0)
result = calculate_diff_coverage(report, {"raven/example.py": set(range(1, 11))}, ["raven/example.py"])
assert result.zero_coverage_files == ("raven/example.py",)
assert result.percent == 0.0
def test_diff_coverage_fails_closed_for_unmeasured_production_file() -> None:
result = calculate_diff_coverage(_report(), {"raven/new_module.py": {1}})
assert result.unmeasured_files == ("raven/new_module.py",)
def test_diff_coverage_allows_documented_omitted_entry_point() -> None:
result = calculate_diff_coverage(_report(), {"raven/__main__.py": {1}})
assert result.unmeasured_files == ()
def test_git_diff_inputs_includes_top_level_raven_files(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
def git(*args: str) -> None:
subprocess.run(["git", *args], cwd=tmp_path, check=True, capture_output=True, text=True)
git("init")
git("config", "user.email", "test@example.com")
git("config", "user.name", "test")
(tmp_path / "README.md").write_text("base\n", encoding="utf-8")
git("add", "README.md")
git("commit", "-m", "base")
base = subprocess.run(
["git", "rev-parse", "HEAD"], cwd=tmp_path, check=True, capture_output=True, text=True
).stdout.strip()
(tmp_path / "raven").mkdir()
(tmp_path / "raven" / "__init__.py").write_text("value = 1\n", encoding="utf-8")
(tmp_path / "raven" / "pkg").mkdir()
(tmp_path / "raven" / "pkg" / "mod.py").write_text("def run():\n return 1\n", encoding="utf-8")
git("add", "-A")
git("commit", "-m", "add raven sources")
monkeypatch.chdir(tmp_path)
changed, added = git_diff_inputs(base, "HEAD")
assert "raven/__init__.py" in changed
assert "raven/pkg/mod.py" in changed
assert "raven/__init__.py" in added
def test_create_baseline_records_separate_line_and_branch_metrics() -> None:
baseline = create_baseline(_report(), "abc123")
assert baseline["reference_commit"] == "abc123"
assert baseline["python"] == f"{sys.version_info.major}.{sys.version_info.minor}"
assert baseline["totals"]["line_percent"] == 80.0
assert baseline["totals"]["branch_percent"] == 75.0
assert baseline["files"]["raven/example.py"]["covered_lines"] == 8
def test_baseline_update_cannot_lower_either_metric(capsys: pytest.CaptureFixture[str]) -> None:
previous = create_baseline(_report(), "base")
proposed = create_baseline(_report(covered_lines=9, covered_branches=2), "head")
assert check_baseline_update(proposed, previous) is False
output = capsys.readouterr().out
assert "line_percent: proposed=90.000000% target=80.000000%" in output
assert "branch_percent: proposed=50.000000% target=75.000000%" in output
assert "may not lower" in output
def test_baseline_update_accepts_equal_or_higher_metrics() -> None:
previous = create_baseline(_report(), "base")
proposed = create_baseline(_report(covered_lines=9, covered_branches=4), "head")
assert check_baseline_update(proposed, previous) is True
def test_ratchet_allows_small_tolerance(capsys: pytest.CaptureFixture[str]) -> None:
baseline = create_baseline(_report(statements=10_000, covered_lines=8_001), "abc123")
current = _report(statements=10_000, covered_lines=8_000)
assert check_ratchet(current, baseline, tolerance=0.05, limit=10) is True
assert "delta=-0.01pp" in capsys.readouterr().out
def test_ratchet_reports_total_and_largest_file_decline(capsys: pytest.CaptureFixture[str]) -> None:
baseline = create_baseline(_report(), "abc123")
current = _report(covered_lines=7, covered_branches=2)
assert check_ratchet(current, baseline, tolerance=0.05, limit=10) is False
output = capsys.readouterr().out
assert "Ratchet line: current=70.00% baseline=80.00% delta=-10.00pp" in output
assert "Ratchet branch: current=50.00% baseline=75.00% delta=-25.00pp" in output
assert "raven/example.py" in output
def test_the_diff_gate_claims_only_trees_that_coverage_measures() -> None:
"""A changed file under an unmeasured tree can never appear in the report,
so a production pathspec over one fails every PR that touches it: the gate
once named all of plugins-dist while coverage measured only raven_everos."""
repo = Path(__file__).resolve().parents[1]
measured = set(re.findall(r"--cov=(\S+)", (repo / "Makefile").read_text(encoding="utf-8")))
for spec in PRODUCTION_PATHSPECS:
if spec.startswith(":(glob,exclude)"):
continue
root = repo / spec.removeprefix(":(glob)").split("/**", 1)[0]
if not root.is_dir():
pytest.fail(f"{spec} names a directory that is not in the checkout: {root}")
if (root / "__init__.py").is_file():
packages = {root.name}
else:
packages = {child.name for child in root.iterdir() if (child / "__init__.py").is_file()}
assert packages, f"{spec} names no importable package"
assert packages <= measured, f"{spec} covers {packages - measured}, which no --cov flag measures"