Raven-1/scripts/coverage_gate.py
KT a5f84b7164
ci(*): run unit tests as four shards and speed up the suite (#413)
## Summary

The unit job takes twelve minutes on one runner. The first cut of this
PR took the production waits out of the tests, which made the suite a
quarter faster on a laptop and moved CI by 14 s (pytest 720.95 s on
main, 706.57 s here): the runner is three to four times slower than a
laptop on CPU-bound tests, and two of the twelve minutes are the four
xdist workers importing every test module. So the PR now attacks the
wall clock from every side that CI's own numbers pointed at.

- `ci(*)`: the unit job runs as four shards. pytest gains `--shard K/N`
in `tests/conftest.py`, decided in `pytest_ignore_collect` so a shard
never imports the other shards' files; test files are dealt round-robin
over their sorted list so the slow families that sort together (the ppt
engine's) are spread out. Each shard writes coverage data and no report;
a `coverage` job downloads the four data files, combines them, writes
`coverage.json` and `coverage.xml`, and runs the gates that used to
follow the tests.
- `perf(ppt-engine)`: `menu()` caches its reading of a template, capped
at eight entries. One bind reads the bound template and then every other
bundled template for the borrow offer; with ten templates bundled, every
bind evicted what the next one needed and re-measured all of them with
freetype. The cap is 32 and a hit moves its entry to the back. On this
PR's first CI run the template-tool file alone was 257 s of the
over-ceiling list, its tests 10 to 26 s each.
- `test(*)`: an autouse fixture collapses the provider chat retry
ladder, the loop's LLM-error ladder and the memory store backoff to a
millisecond per step, and the steer hook grace to one poll. It keeps
each tuple's length because both ladders count attempts with `len()`;
the steps are a millisecond rather than zero because the store pipeline
waits with `wait_for(stopping.wait(), timeout=delay)`, and a zero
timeout cancels that wait before it runs. A test that proves a timing
property opts out with the new `production_timing` marker. Ten tests
spent 257 s of the suite's worker time in those waits; none asserted on
the time.
- `test(*)`: a wall-clock ceiling. Setup plus call over 2 s on a test
marked neither `slow` nor `production_timing` is listed at the end of
the run; `--wall-clock-ceiling-strict` fails an otherwise green session
on it. Warning only here; CI flips to strict once the remaining tail is
marked.
- `test(*)`: every test created two numbered temp directories, and
pytest picks each number by scanning the base temp dir; the fake home is
now a sibling of `tmp_path`. `MPLCONFIGDIR` points at a git-ignored dir
in the checkout, because the fake home changed per test and matplotlib
rebuilt its font list from the system fonts on every run.
- `refactor(agent)`: the outer LLM-error retry ladder default `(15, 30,
60)` lived in three places; it is one constant in
`raven/config/schema.py` now, read by both `RecoveryLimits` and
`_ladder`. Behaviour unchanged.
- `test(updates)`: the first sharded run turned up an order dependency.
`beta_channel` binds `ReleaseInfo` from `raven.updates.upgrade` at
import, the update-notice tests stand a fake in for that module through
`sys.modules`, and `_refresh` imports `beta_channel` lazily; a worker
that imported `beta_channel` for the first time inside that test got an
`ImportError` swallowed as a failed refresh. The test module now imports
`beta_channel` at the top.
- `ci(*)`: the PR diff gate named every `.py` under `plugins-dist` as
production code while coverage measures only `raven` and `raven_everos`,
so a change to the ppt or design engine could never appear in the report
and the gate refused every such PR on sight; the first sharded run hit
it on `menu.py`. The pathspec now names `plugins-dist/everos-memory`,
and a test holds every production pathspec to a package some `--cov`
flag in the Makefile measures.
- Two slow tests were not waits: the sandbox second-cancel test
cancelled one tick too early (two pending cancels coalesce into one
`CancelledError`) and now waits, under a one second timeout, for the
kill to land; the two thousand-session dispatch tests wrote a span per
refused enqueue through a store that opens its file per span, so tracing
is off for them. Follow-up on the product side: `raven/tracing/store.py`
should reuse its handle.

The suite has 30 upper-bound assertions of the form `assert elapsed <
X`; shortened delays can only make those looser.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [x] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Whole suite, `uv run --frozen --all-extras pytest -q` (macOS, 4
workers), 22051 passed / 43 skipped each time:

| tree | wall clock | unmarked tests over 2 s |
|---|---|---|
| main before this branch | 372.82 s | (no ceiling yet) |
| after the waits were taken out | 279.65 s | 63 |
| this branch | 224.25 s | 39 |

The shards partition the suite exactly (`uv run --frozen --all-extras
pytest --collect-only -q -n 0 --shard K/4`, K = 1..4):

| whole | shard 1 | shard 2 | shard 3 | shard 4 | union | ids in two
shards |
|---|---|---|---|---|---|---|
| 22095 | 5058 | 5555 | 5955 | 5527 | 22095 | 0 |

196 / 195 / 194 / 194 test files per shard. A shard collects in 7 s
where the whole suite takes 23 s on the same machine. `--ignore` still
works under `--shard`: with four `--ignore` flags on shard 1, the
ignored file that shard owns is gone from its collection (5034 ids
against 5058).

Combining loses nothing. Two small runs with `--cov=raven
--cov=raven_everos --cov-branch --cov-report=` and `COVERAGE_FILE` set,
then `make coverage-combine COVERAGE_DATA_DIR=<dir>`, against one `make
coverage` of the whole suite on main:

| report | files | statements | branches |
|---|---|---|---|
| one run of the whole suite | 704 | 82634 | 25548 |
| two shards combined | 704 | 82636 | 25548 |

Same file set; the two extra statements are the new constant and its
import. A source package's never-imported files reach the combined
report through the shard that imported it, as they reached the single
run.

The menu cache, the three files that bind templates (`uv run --frozen
--all-extras pytest -q tests/test_ppt_engine_template_tool.py
tests/test_ppt_engine_outline_checks.py
tests/test_ppt_engine_default_templates.py`): 101 s of worker time in
the last profile, now `196 passed in 11.84s` with no test over 2.9 s.
The files that call `menu()` directly: `219 passed in 8.06s`.

The font cache, the formula render test at `-n 0`: 8.91 s cold, 0.69 s
on the next run. Seven home-sensitive files: `484 passed in 5.30s`.

Ceiling guard, both modes, with the ceiling forced low so it has
something to list:

```
uv run --frozen --all-extras pytest -q -n 0 --wall-clock-ceiling=0.01 tests/test_import_cycle_budget.py tests/test_updates_install_guard.py
uv run --frozen --all-extras pytest -q -n 2 --wall-clock-ceiling=0.01 --wall-clock-ceiling-strict tests/test_updates_install_guard.py
```

First: 28 passed, five listed, exit 0. Second: 24 passed, four listed,
exit 1 (the controller flips the status; workers do not).

`make lint-python` clean, `make lint-imports` passes. The CI run on this
PR is the measurement that matters: four `unit` shards and one `coverage
gates` job.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed (none: CI
plumbing, a cache cap, test setup)

## Risk

- The coverage gates now run in a job after the four shards. A red shard
skips the gates, a deliberate trade: combining three of four data files
would report a coverage drop that is not one, and the PR is red either
way. The gate scripts and their thresholds are unchanged.
- The diff gate no longer sees `plugins-dist/ppt-engine` or
`plugins-dist/design-engine` at all, as it could never judge them;
measuring them (`--cov=raven_ppt --cov=raven_design`, a new ratchet
baseline) is the follow-up that would bring them under the gate.
- `--shard` is opt-in; a plain `pytest` run collects everything as
before. Shard data artifacts are kept one day, the combined reports 30
days as before.
- No product behaviour change: the ladder default is the same tuple read
from one place; the menu cap only changes how long the cache keeps
entries.
- Test-only risk: a test that implicitly relied on a production-length
retry delay would see different timing. The full suite passed with the
fixture on, and a single test can opt out with `production_timing`.
- Rollback: revert the PR. The unit job goes back to one runner with the
gates inline; both markers are additive.

- [x] Security impact considered (no security surface touched)
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
2026-09-15 16:00:49 +08:00

365 lines
15 KiB
Python

"""Coverage reporting and governance gates shared by CI and local development."""
from __future__ import annotations
import argparse
import json
import re
import shutil
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Iterable
SCHEMA_VERSION = 1
DEFAULT_TOLERANCE = 0.05
# Glob magic so ** matches zero-or-more path components; the default pathspec
# raven/**/*.py drops top-level files like raven/__init__.py because plain * spans
# slashes and the middle separator then has nothing to match.
PRODUCTION_PATHSPECS = (
":(glob)raven/**/*.py",
# The everos memory plugin beside the host wheel is production code too;
# without this a change there dodges the diff gate. Only that plugin: the
# gate can judge what coverage measures (--cov=raven --cov=raven_everos), and
# a changed file in an unmeasured tree would fail every PR that touches it.
":(glob)plugins-dist/everos-memory/**/*.py",
# Scaffold templates are copied out by the scaffolder, never imported by
# the runtime; they are data with a .py spelling, not production code.
":(glob,exclude)raven/templates/agents_scaffold/**",
)
DEFAULT_DIFF_THRESHOLD = 90.0
OMITTED_PATHS = {
"raven/__main__.py",
"evolver/__main__.py",
}
@dataclass(frozen=True)
class Metrics:
statements: int
covered_lines: int
branches: int
covered_branches: int
@property
def line_percent(self) -> float:
return _percent(self.covered_lines, self.statements)
@property
def branch_percent(self) -> float:
return _percent(self.covered_branches, self.branches)
@dataclass(frozen=True)
class DiffCoverage:
executable: frozenset[tuple[str, int]]
uncovered: frozenset[tuple[str, int]]
zero_coverage_files: tuple[str, ...]
unmeasured_files: tuple[str, ...]
@property
def percent(self) -> float:
return _percent(len(self.executable) - len(self.uncovered), len(self.executable))
def _percent(covered: int, total: int) -> float:
return 100.0 if total == 0 else covered * 100.0 / total
def _metrics(summary: dict[str, Any]) -> Metrics:
return Metrics(
statements=int(summary["num_statements"]),
covered_lines=int(summary["covered_lines"]),
branches=int(summary["num_branches"]),
covered_branches=int(summary["covered_branches"]),
)
def load_coverage(path: Path) -> dict[str, Any]:
with path.open(encoding="utf-8") as handle:
report = json.load(handle)
if not isinstance(report.get("files"), dict) or not isinstance(report.get("totals"), dict):
raise ValueError(f"{path} is not a coverage.py JSON report")
return report
def print_summary(report: dict[str, Any], limit: int) -> None:
totals = _metrics(report["totals"])
print(f"Line coverage: {totals.line_percent:.2f}% ({totals.covered_lines}/{totals.statements})")
print(f"Branch coverage: {totals.branch_percent:.2f}% ({totals.covered_branches}/{totals.branches})")
print(f"Lowest {limit} production files by combined coverage:")
ranked = sorted(
(
(float(data["summary"]["percent_covered"]), name, _metrics(data["summary"]))
for name, data in report["files"].items()
if int(data["summary"]["num_statements"]) > 0
),
key=lambda item: (item[0], item[1]),
)
for combined, name, metrics in ranked[:limit]:
print(f" {combined:6.2f}% line {metrics.line_percent:6.2f}% branch {metrics.branch_percent:6.2f}% {name}")
def create_baseline(report: dict[str, Any], commit: str) -> dict[str, Any]:
files = {
name: _metrics_payload(_metrics(data["summary"]))
for name, data in sorted(report["files"].items())
if int(data["summary"]["num_statements"]) > 0
}
return {
"schema_version": SCHEMA_VERSION,
"source": "raven",
"python": f"{sys.version_info.major}.{sys.version_info.minor}",
"reference_commit": commit,
"totals": _metrics_payload(_metrics(report["totals"])),
"files": files,
}
def _metrics_payload(metrics: Metrics) -> dict[str, int | float]:
return {
"statements": metrics.statements,
"covered_lines": metrics.covered_lines,
"line_percent": round(metrics.line_percent, 6),
"branches": metrics.branches,
"covered_branches": metrics.covered_branches,
"branch_percent": round(metrics.branch_percent, 6),
}
def write_baseline(report: dict[str, Any], output: Path, commit: str) -> None:
output.write_text(json.dumps(create_baseline(report, commit), indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(f"Wrote coverage baseline candidate to {output}")
def check_baseline_update(current: dict[str, Any], previous: dict[str, Any]) -> bool:
if current.get("schema_version") != SCHEMA_VERSION or previous.get("schema_version") != SCHEMA_VERSION:
raise ValueError("current and target-branch baselines must use the supported schema")
passed = True
for metric in ("line_percent", "branch_percent"):
value = float(current["totals"][metric])
target = float(previous["totals"][metric])
delta = value - target
print(f"Baseline {metric}: proposed={value:.6f}% target={target:.6f}% delta={delta:+.6f}pp")
if value < target:
passed = False
if passed:
print("Coverage baseline update is monotonic.")
else:
print("Coverage baseline files may not lower line or branch coverage.")
return passed
def check_ratchet(report: dict[str, Any], baseline: dict[str, Any], tolerance: float, limit: int) -> bool:
if baseline.get("schema_version") != SCHEMA_VERSION:
raise ValueError(f"unsupported baseline schema: {baseline.get('schema_version')!r}")
current = _metrics(report["totals"])
expected = baseline["totals"]
checks = (
("line", current.line_percent, float(expected["line_percent"])),
("branch", current.branch_percent, float(expected["branch_percent"])),
)
failed = [(name, value, target) for name, value, target in checks if value + tolerance < target]
for name, value, target in checks:
delta = value - target
print(f"Ratchet {name}: current={value:.2f}% baseline={target:.2f}% delta={delta:+.2f}pp")
if not failed:
print(f"Coverage ratchet passed with {tolerance:.2f}pp tolerance.")
return True
print("Largest per-file coverage declines:")
declines = _file_declines(report, baseline)
for decline, name, line_delta, branch_delta in declines[:limit]:
print(f" {decline:6.2f}pp line {line_delta:+7.2f}pp branch {branch_delta:+7.2f}pp {name}")
return False
def _file_declines(report: dict[str, Any], baseline: dict[str, Any]) -> list[tuple[float, str, float, float]]:
declines: list[tuple[float, str, float, float]] = []
current_files = report["files"]
for name, expected in baseline["files"].items():
data = current_files.get(name)
if data is None:
continue
current = _metrics(data["summary"])
line_delta = current.line_percent - float(expected["line_percent"])
branch_delta = current.branch_percent - float(expected["branch_percent"])
decline = max(-line_delta, -branch_delta, 0.0)
if decline > 0:
declines.append((decline, name, line_delta, branch_delta))
return sorted(declines, key=lambda item: (-item[0], item[1]))
_HUNK_RE = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")
def parse_changed_lines(diff: str) -> dict[str, set[int]]:
"""Return new-side hunk lines; deleted lines never enter the denominator."""
changed: dict[str, set[int]] = {}
path: str | None = None
for line in diff.splitlines():
if line.startswith("+++ "):
# Reset on a non-b/ target (e.g. +++ /dev/null) so a deleted file's
# hunks are not misattributed to the previously seen path.
path = line[6:] if line.startswith("+++ b/") else None
if path is not None:
changed.setdefault(path, set())
continue
match = _HUNK_RE.match(line)
if path is None or match is None:
continue
start = int(match.group(1))
count = int(match.group(2) or "1")
changed[path].update(range(start, start + count))
return changed
def calculate_diff_coverage(
report: dict[str, Any], changed: dict[str, set[int]], added_files: Iterable[str] = ()
) -> DiffCoverage:
"""Intersect changed lines with coverage.py statements to ignore comments and blanks."""
executable: set[tuple[str, int]] = set()
uncovered: set[tuple[str, int]] = set()
zero_coverage_files: list[str] = []
unmeasured_files: list[str] = []
added = set(added_files)
for name, lines in changed.items():
data = report["files"].get(name)
if data is None:
if name not in OMITTED_PATHS:
unmeasured_files.append(name)
continue
executed = set(data["executed_lines"])
missing = set(data["missing_lines"])
statements = executed | missing
for line in lines & statements:
item = (name, line)
executable.add(item)
if line in missing:
uncovered.add(item)
if name in added and statements and not executed:
zero_coverage_files.append(name)
return DiffCoverage(
frozenset(executable),
frozenset(uncovered),
tuple(sorted(zero_coverage_files)),
tuple(sorted(unmeasured_files)),
)
def _git(*args: str) -> str:
executable = shutil.which("git")
if executable is None:
raise OSError("git is required for coverage diff checks")
completed = subprocess.run([executable, *args], check=True, capture_output=True, text=True)
return completed.stdout
def load_baseline_from_ref(base_ref: str, path: Path) -> dict[str, Any] | None:
_git("rev-parse", "--verify", f"{base_ref}^{{commit}}")
try:
content = _git("show", f"{base_ref}:{path.as_posix()}")
except subprocess.CalledProcessError:
return None
return json.loads(content)
def git_diff_inputs(base_ref: str, head: str | None) -> tuple[dict[str, set[int]], set[str]]:
"""Resolve the merge base and collect Python changes for CI or the local worktree."""
merge_base = _git("merge-base", base_ref, "HEAD").strip()
end = head or ""
range_args = [merge_base, end] if end else [merge_base]
diff = _git("diff", "--unified=0", "--no-color", "--diff-filter=ACMR", *range_args, "--", *PRODUCTION_PATHSPECS)
names = _git("diff", "--name-only", "--diff-filter=A", *range_args, "--", *PRODUCTION_PATHSPECS)
return parse_changed_lines(diff), {name for name in names.splitlines() if name}
def check_diff(report: dict[str, Any], base_ref: str, head: str | None, threshold: float) -> bool:
changed, added = git_diff_inputs(base_ref, head)
result = calculate_diff_coverage(report, changed, added)
covered = len(result.executable) - len(result.uncovered)
print(f"Diff coverage: {result.percent:.2f}% ({covered}/{len(result.executable)} executable changed lines)")
if result.uncovered:
print("Uncovered changed lines:")
for name, line in sorted(result.uncovered):
print(f" {name}:{line}")
if result.zero_coverage_files:
print("New production files with 0% line coverage:")
for name in result.zero_coverage_files:
print(f" {name}")
if result.unmeasured_files:
print("Changed production files missing from the coverage report:")
for name in result.unmeasured_files:
print(f" {name}")
passed = result.percent + 1e-9 >= threshold and not result.zero_coverage_files and not result.unmeasured_files
if passed:
print(f"Diff coverage passed the {threshold:.2f}% threshold.")
else:
print(f"Diff coverage failed the {threshold:.2f}% threshold.")
return passed
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--coverage-json", type=Path, default=Path("coverage.json"))
subparsers = parser.add_subparsers(dest="command", required=True)
summary = subparsers.add_parser("summary")
summary.add_argument("--limit", type=int, default=10)
baseline = subparsers.add_parser("baseline")
baseline.add_argument("--output", type=Path, default=Path("coverage-baseline-candidate.json"))
baseline.add_argument("--commit", default="HEAD")
baseline_check = subparsers.add_parser("baseline-check")
baseline_check.add_argument("--baseline", type=Path, default=Path(".github/coverage-baseline.json"))
baseline_check.add_argument("--base-ref", required=True)
ratchet = subparsers.add_parser("ratchet")
ratchet.add_argument("--baseline", type=Path, default=Path(".github/coverage-baseline.json"))
ratchet.add_argument("--tolerance", type=float, default=DEFAULT_TOLERANCE)
ratchet.add_argument("--limit", type=int, default=10)
diff = subparsers.add_parser("diff")
diff.add_argument("--base-ref", required=True)
diff.add_argument("--head", default=None)
diff.add_argument("--threshold", type=float, default=DEFAULT_DIFF_THRESHOLD)
return parser
def main(argv: list[str] | None = None) -> int:
args = _parser().parse_args(argv)
try:
if args.command == "baseline-check":
with args.baseline.open(encoding="utf-8") as handle:
current_baseline = json.load(handle)
previous_baseline = load_baseline_from_ref(args.base_ref, args.baseline)
if previous_baseline is None:
print(f"No coverage baseline exists on {args.base_ref}; accepting the initial bootstrap baseline.")
return 0
return 0 if check_baseline_update(current_baseline, previous_baseline) else 1
report = load_coverage(args.coverage_json)
if args.command == "summary":
print_summary(report, args.limit)
return 0
if args.command == "baseline":
commit = _git("rev-parse", args.commit).strip()
write_baseline(report, args.output, commit)
return 0
if args.command == "ratchet":
with args.baseline.open(encoding="utf-8") as handle:
baseline = json.load(handle)
return 0 if check_ratchet(report, baseline, args.tolerance, args.limit) else 1
if args.command == "diff":
return 0 if check_diff(report, args.base_ref, args.head, args.threshold) else 1
except (OSError, ValueError, KeyError, json.JSONDecodeError, subprocess.CalledProcessError) as exc:
print(f"coverage gate error: {exc}", file=sys.stderr)
return 2
raise AssertionError(f"unhandled command: {args.command}")
if __name__ == "__main__":
raise SystemExit(main())