## Summary Adds `everos-cloud-memory`, a separately distributed memory plugin for EverOS Cloud, and wires the host so a second memory plugin is a first-class choice. The plugin (`plugins-dist/everos-cloud-memory`) is a thin REST client over httpx: one API key (the config slice `plugins.config.everos-cloud-memory.api_key`, or `EVEROS_CLOUD_API_KEY`), owner ids from the host's `ServiceLocator` (never from the slice), every add in `agent` mode, and a flush only when a caller marks the conversation over (`metadata.flush` / `metadata.is_final`: the sub-agent handoff and the importer's final batch) and once at stop. It ships its own onboard screen (env or on-file key used without a prompt; 401 / 403 / unreachable get their own sentences with Re-enter or Skip), `delete` answers False (the cloud deletes by scope, nothing calls it), and `health()` classifies seven answers for `raven doctor` and the settings card. Host changes, each guarded so `memory.backend` `everos` and `null` behave exactly as before: - `raven onboard` step 4 asks "Which memory backend?" when more than one plugin contributes a screen (preselects the current backend, `Off` clears it, cancel exits 1 and writes nothing); with one plugin the step is byte-identical to today. - The memory page reads whichever backend `memory.backend` names: the cloud slice's key goes out as a Bearer header and the `/health` probe is skipped for the cloud; a missing distribution is named in the page's note. - `settings.everos` reports `reason: "other_backend"` for any non-EverOS backend, so the Model page hides the three EverOS-only roles (embedding stays: knowledge bases use that pin); `settings.everosSet` refuses those three sections for another backend; `settings.set` admits `plugins.config.<id>.<field>` against the plugin's manifest through the same admission door channels use; new `settings.everosCloud` feeds the card (key source, endpoint, health). - Provider-key and main-model saves no longer restart the local EverOS when another backend is active. - ui-web: an EverOS Cloud card (status chip, key row, endpoint) and the `settings.everosCloud` data behind it. No page mounts the card: it drew on Model settings until the owner ruled that a memory backend's endpoint and credential answer neither question that page asks, and where it belongs is the Memory section's to say once that section has a design. Until then the key is entered through `raven onboard` or `EVEROS_CLOUD_API_KEY`. The fixture gate pins the four optional fields the offline library never sends. - Gates: the plugin boundary test scans both memory plugins, coverage and the release workflow build the second wheel, deptry's ignore list names the plugin beside `raven_everos`, and a conftest fixture keeps the second plugin out of entry-point discovery for the existing onboarding tests (they describe a one-plugin install). A second topic rides along, because it is what makes the hosted backend reachable from an agent product at all. Every product under `agents/` pinned its memory at the stock EverOS address and only `raven-design` read the host's, so the other four addressed whatever answered on 18791 -- the wizard moves a managed server when the port is taken, and a server the user runs themselves is wherever they put it. `inherit_memory_backend` in `raven/config/product_render.py` now carries `memory.backend`, the endpoint, and (only when the product holds none itself) the credential from the host into every render, beside the host denials and plugin opt-outs that already travel there; `raven-design`'s private copy is deleted and the five product configs stop naming an address. Identity never travels, which is what keeps five products apart inside one store. `memory.backend: null` travels like any other value, because that null is how an operator turns long-term memory off and a product that kept writing would be the one surface ignoring them; the product's own default stands only when the host carries no key at all. Which store memories live in is not a product's choice either: the host reads a sub-agent's memories back through its own backend, so a product writing anywhere else left that record empty with nothing reporting it. `owned` is forced off whenever the local backend is in force, because a product infers ownership from a slice naming no root -- the shape all five ship -- and so answered a host that was merely not serving by starting a second server against a root of its own, stranding every memory written so far. Decisions worth a reviewer's eye: the fake cloud (`tests/_everos_cloud_fake.py`) honours or refuses every field the backend can send and serves both an in-process transport and an HTTP face for the real-host acceptance; the live tests (`tests/integration/test_everos_cloud_real_cloud.py`) skip without `EVEROS_CLOUD_API_KEY`; run with a real key on 2026-10-09 all six pass against api.evermind.ai (round trip, get by session, empty flush, 401 wording, probe with and without a key, and extraction without a flush within two minutes), so the documentation and the service agree on everything the plugin sends. Design, acceptance cases and plan are under `docs/specs/2026-10-08-everos-cloud-memory-*.md` and `docs/plans/`. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed Rebased onto main `182dacac0` on 2026-10-13. What that rebase and the four commits on top were verified with: - Full suite, `uv run --frozen --python 3.12 --extra dev pytest -q` -> **29748 passed, 8 failed** in 4m10s. All eight fail identically with these four commits' sources reverted in-tree at the same base, so none of them is this branch's: the two `test_the_products_tool_face_*` cases in the code and oncall launchers (an extra `web_search` in the visible tool set), `test_agent_loop_token_budget`, `test_rpc_files::test_a_host_without_libreoffice_says_so`, `test_ppt_engine_image_search`, `test_shell_background`, and the two `test_simulation_*` cases. - Real-host before and after for the inheritance, driving the launcher the host spawns (`agents/raven-research/run.py`) against an isolated `RAVEN_HOME` whose EverOS address is `19701`, not the stock port. Before: the product rendered `base_url: http://localhost:18791` with no `owned` key, ignored the host, and started an EverOS of its own -- an orphaned `everos server start --root <its own state root>/everos` holding 18791, which is the stranding this change exists to stop. After: it rendered `{"base_url": "http://localhost:19701", "owned": false}` and said so itself, "the EverOS you manage is not running at http://localhost:19701 ... Raven does not start or stop it", with `ps` and `lsof` confirming nothing was started. On the cloud the product rendered `memory.backend: everos-cloud` with the host's key and left the out-of-force local slice untouched, and `raven doctor` over that rendered config answered `everos-cloud: ok https://api.evermind.ai`, so the credential reached the service. Identity stayed `raven-research-ng` throughout, and a fingerprint of the operator's own EverOS root was identical before and after. - The first CI round on this head found one that the local gates did not: `tests/test_import_cycle_budget.py`. `inherit_memory_backend` read the backend-to-distribution pairing from `raven.core.plugin_stack`, and that edge from `raven.config` into `raven.core` -- which imports `raven.config` back -- took mutual pairs from 13 to 14 and packages inside a cycle from 18 to 22. `make lint-imports` keeps ten contracts and none of them covers that pair, so it was green throughout. The pairing now lives in `raven/config/raven.py` beside the two fields it joins (`MemoryConfig.backend` names a contribution, `PluginsConfig.config` is keyed by the distribution), which `raven.core` may import and which imports nothing back; `plugin_stack` re-exports the two names its own callers read from it. An intermediate commit put it in `raven.contracts` instead, which broke the cycle but took the papers ceiling from 3700 to 3719 lines -- `tests/test_kernel_budget.py` says a new paper is a reviewed change to that ceiling, so it moved again rather than raising it. - The review round on this head raised one non-blocking finding, fixed rather than carried forward: the inheritance read `memory.backend` with `.get()`, which collapses an explicit `null` into an absent key, so a host with long-term memory off left every packaged agent on its own shipped backend and still writing. Reproduced by calling the function directly before the change and after, now read on presence, and pinned by a case whose mutation (restoring the `.get()`) turns it and nothing else red. - One ordering problem found while checking the above, pre-existing and not fixed here: `tests/test_rpc_settings.py` run before `tests/test_cli_onboard_commands.py` leaves `raven.home._current_config_path` pointing at a dead tmp config, so the onboard cases write somewhere else. With main's own copies of both files and main's conftest in the tree it is 57 failures; with this branch's, 19. Which cases turn red changes run to run as xdist distributes them differently. - `cd ui-web && npx vitest run` -> 876 files, 3728 passed, 0 failed; `npm run lint` -> 0 errors (4 warnings, all in files this change does not touch); `npm run gen:check` -> in sync (228 methods); `npm run type-check` -> clean; `cd ui-tui && npm run lint:i18n && npm run lint:rpc` -> in sync - `make lint-python` / `lint-imports` (10 contracts kept) / `lint-types` / `lint-deps` -> all pass; `uv lock --check` -> exit 0 with both sides' new packages present; `scripts/check_source_language.py` and `scripts/check_large_files.py` on `github/main..HEAD` -> exit 0; commitlint over every commit -> ok - Semantic-conflict audit of the rebase, beyond what the suite answers: 21 of the branch's 66 files are also in main's 499, and every one of main's changes in them is additive and elsewhere in the file. The one that could have broken silently is main's new `vision.model` / `vision.provider` simple keys meeting this branch's new `plugins.config.<id>.<field>` branch in `settings_set`; nothing in the suite exercises `vision.*` through that method, so it was run directly -> `{'applied': True}` and `{"model": "..."}` on disk. The subtraction decisions were re-checked against the new base: no new caller of a memory backend's `delete`, main's new onboard cases are about TLS and not the memory step, `inherit_everos_address` exists nowhere, and `DEFAULT_EVEROS_BASE_URL` is still defined. - Mutation runs on the new code, restored afterwards: dropping `owned: false`, letting the host's credential overwrite the product's own, not writing the host's backend name, and carrying every memory slice instead of the one in force each turn their own cases red; making the cloud card render nothing turns 4 of its 5 new cases red and leaves the fifth ("draws nothing when not selected") green. From the earlier round, on base `6d509a8c5`: - `uv run --frozen --python 3.12 --extra dev pytest -p no:cacheprovider -q tests/test_everos_cloud_fake.py tests/test_everos_cloud_backend.py tests/test_everos_cloud_backend_contract.py tests/test_everos_cloud_onboard.py tests/test_everos_cloud_discover.py tests/test_cli_onboard_commands.py tests/test_rpc_memory.py tests/test_rpc_settings.py tests/test_plugin_boundary.py tests/test_rpc_schema_match.py tests/test_rpc_registration.py tests/test_release_plugin_list.py tests/integration/test_everos_cloud_real_cloud.py` -> 1001 passed, 6 skipped (the six live tests, no key) - `cd ui-web && npx vitest run` -> 214 files, 3211 passed; `npm run lint` -> 0 errors (4 warnings, all in files this change does not touch); `npm run gen:check` -> in sync (207 methods); `cd ui-tui && npm run lint:rpc && npm run lint:i18n` -> in sync - `make lint` (ruff, ruff format, ui tsc, deptry) -> ok; `make lint-types` -> All checks passed; `make build-ui` -> boot snapshots match; `scripts/check_large_files.py` and `scripts/check_source_language.py` on `6d509a8c5..HEAD` -> exit 0; commitlint and `scripts/check_commit_messages.py` on each commit -> ok - Mutation runs per change (restored afterwards, `git diff` clean): a host import of the plugin turns the boundary test red; `import raven_everos` in the plugin turns the import-isolation test red; removing the chooser's cancel exit, the Bearer header, the restart guard, the slice-admission guard, the Roles filter and the fake's two fixes each turn their own test red - Real-host acceptance (64 cases, `docs/specs/2026-10-08-everos-cloud-memory-acceptance.md`) run against a fake EverOS Cloud: the wizard in tmux (chooser, key typed / from env / on file, 401 / 403 / unreachable wording, back, Off, Ctrl-C, non-interactive, base-tree transcript byte-identical with one plugin), `raven agent` turns (user and agent-track recall, agent-mode adds, 4 s recall bound, a 300 KB message dropped with a warning, a sub-agent trace stored under its own ids, flushed and read back, one exit flush per unflushed session), `raven import run` (3 adds then one flush; a hung flush retried 4x and the exit sweep defeated), `raven doctor` under seven cloud conditions (seven sentences, exit 2 only for the five faults) and with the distribution hidden, and the web page through playwright-cli (memory page counts and the fake's episode, empty and refused stores, the Model page hiding the three EverOS-only roles while the embedding row stays, the cloud card rotating a key and showing the 401 wording, three refused settings.set frames over the gateway WebSocket, the local everos / null / mem0 backends drawing the page as before, and a provider-key save that restarts the local EverOS on the base tree and does not on this branch). Evidence and the per-case verdicts are kept outside the repo. The live service was reached with a real key after the acceptance run: six tests, six passes, the service's replies archived with the evidence. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes User-visible changes: the wizard's step 4 chooser (only when two memory plugins are installed); Model settings hides the three EverOS-only role rows when `memory.backend` is not `everos`/`null` and draws no cloud card; the memory page sends a Bearer header for the cloud backend. Two behaviour changes for agent products, both on the `agents/` lane only. An agent product now follows the host's `memory.backend` and endpoint instead of its own, which is a fix for the four products that never followed the host and a change for `raven-design`, which followed the endpoint but not the backend. And a product no longer starts an EverOS server of its own: where it used to answer a host that was not serving by starting one against an empty root of its own, it now reports long-term memory as unavailable, which is what the host does. Rollback for this part is the one commit; the product configs it edits are shipped files, not user config. The plugin is not installed by default (its wheel is built by the release workflow; nothing changes for an install without it). Rollback: revert the squash commit; the config schema gains no new key (the slice is a plugin slice), so an older raven reads a config written by this one. Credentials: the cloud key is written only by `set_plugin_config_fields` into the plugin's slice; the settings page's key row goes through `settings.set` with a manifest-declared field, and `settings.everosCloud` reports `api_key_set` / `key_source`, never the value. ## Related Issues N/A --------- Co-authored-by: gloryfromca <23442919+gloryfromca@users.noreply.github.com> Co-authored-by: Claude (claude-fable-5-1) <noreply@anthropic.com>
366 lines
15 KiB
Python
366 lines
15 KiB
Python
"""Coverage reporting and governance gates shared by CI and local development."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import json
|
|
import re
|
|
import shutil
|
|
import subprocess
|
|
import sys
|
|
from dataclasses import dataclass
|
|
from pathlib import Path
|
|
from typing import Any, Iterable
|
|
|
|
SCHEMA_VERSION = 1
|
|
DEFAULT_TOLERANCE = 0.05
|
|
# Glob magic so ** matches zero-or-more path components; the default pathspec
|
|
# raven/**/*.py drops top-level files like raven/__init__.py because plain * spans
|
|
# slashes and the middle separator then has nothing to match.
|
|
PRODUCTION_PATHSPECS = (
|
|
":(glob)raven/**/*.py",
|
|
# The everos memory plugin beside the host wheel is production code too;
|
|
# without this a change there dodges the diff gate. Only that plugin: the
|
|
# gate can judge what coverage measures (--cov=raven --cov=raven_everos), and
|
|
# a changed file in an unmeasured tree would fail every PR that touches it.
|
|
":(glob)plugins-dist/everos-memory/**/*.py",
|
|
":(glob)plugins-dist/everos-cloud-memory/**/*.py",
|
|
# Scaffold templates are copied out by the scaffolder, never imported by
|
|
# the runtime; they are data with a .py spelling, not production code.
|
|
":(glob,exclude)raven/templates/agents_scaffold/**",
|
|
)
|
|
DEFAULT_DIFF_THRESHOLD = 90.0
|
|
OMITTED_PATHS = {
|
|
"raven/__main__.py",
|
|
"evolver/__main__.py",
|
|
}
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Metrics:
|
|
statements: int
|
|
covered_lines: int
|
|
branches: int
|
|
covered_branches: int
|
|
|
|
@property
|
|
def line_percent(self) -> float:
|
|
return _percent(self.covered_lines, self.statements)
|
|
|
|
@property
|
|
def branch_percent(self) -> float:
|
|
return _percent(self.covered_branches, self.branches)
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class DiffCoverage:
|
|
executable: frozenset[tuple[str, int]]
|
|
uncovered: frozenset[tuple[str, int]]
|
|
zero_coverage_files: tuple[str, ...]
|
|
unmeasured_files: tuple[str, ...]
|
|
|
|
@property
|
|
def percent(self) -> float:
|
|
return _percent(len(self.executable) - len(self.uncovered), len(self.executable))
|
|
|
|
|
|
def _percent(covered: int, total: int) -> float:
|
|
return 100.0 if total == 0 else covered * 100.0 / total
|
|
|
|
|
|
def _metrics(summary: dict[str, Any]) -> Metrics:
|
|
return Metrics(
|
|
statements=int(summary["num_statements"]),
|
|
covered_lines=int(summary["covered_lines"]),
|
|
branches=int(summary["num_branches"]),
|
|
covered_branches=int(summary["covered_branches"]),
|
|
)
|
|
|
|
|
|
def load_coverage(path: Path) -> dict[str, Any]:
|
|
with path.open(encoding="utf-8") as handle:
|
|
report = json.load(handle)
|
|
if not isinstance(report.get("files"), dict) or not isinstance(report.get("totals"), dict):
|
|
raise ValueError(f"{path} is not a coverage.py JSON report")
|
|
return report
|
|
|
|
|
|
def print_summary(report: dict[str, Any], limit: int) -> None:
|
|
totals = _metrics(report["totals"])
|
|
print(f"Line coverage: {totals.line_percent:.2f}% ({totals.covered_lines}/{totals.statements})")
|
|
print(f"Branch coverage: {totals.branch_percent:.2f}% ({totals.covered_branches}/{totals.branches})")
|
|
print(f"Lowest {limit} production files by combined coverage:")
|
|
ranked = sorted(
|
|
(
|
|
(float(data["summary"]["percent_covered"]), name, _metrics(data["summary"]))
|
|
for name, data in report["files"].items()
|
|
if int(data["summary"]["num_statements"]) > 0
|
|
),
|
|
key=lambda item: (item[0], item[1]),
|
|
)
|
|
for combined, name, metrics in ranked[:limit]:
|
|
print(f" {combined:6.2f}% line {metrics.line_percent:6.2f}% branch {metrics.branch_percent:6.2f}% {name}")
|
|
|
|
|
|
def create_baseline(report: dict[str, Any], commit: str) -> dict[str, Any]:
|
|
files = {
|
|
name: _metrics_payload(_metrics(data["summary"]))
|
|
for name, data in sorted(report["files"].items())
|
|
if int(data["summary"]["num_statements"]) > 0
|
|
}
|
|
return {
|
|
"schema_version": SCHEMA_VERSION,
|
|
"source": "raven",
|
|
"python": f"{sys.version_info.major}.{sys.version_info.minor}",
|
|
"reference_commit": commit,
|
|
"totals": _metrics_payload(_metrics(report["totals"])),
|
|
"files": files,
|
|
}
|
|
|
|
|
|
def _metrics_payload(metrics: Metrics) -> dict[str, int | float]:
|
|
return {
|
|
"statements": metrics.statements,
|
|
"covered_lines": metrics.covered_lines,
|
|
"line_percent": round(metrics.line_percent, 6),
|
|
"branches": metrics.branches,
|
|
"covered_branches": metrics.covered_branches,
|
|
"branch_percent": round(metrics.branch_percent, 6),
|
|
}
|
|
|
|
|
|
def write_baseline(report: dict[str, Any], output: Path, commit: str) -> None:
|
|
output.write_text(json.dumps(create_baseline(report, commit), indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
|
print(f"Wrote coverage baseline candidate to {output}")
|
|
|
|
|
|
def check_baseline_update(current: dict[str, Any], previous: dict[str, Any]) -> bool:
|
|
if current.get("schema_version") != SCHEMA_VERSION or previous.get("schema_version") != SCHEMA_VERSION:
|
|
raise ValueError("current and target-branch baselines must use the supported schema")
|
|
passed = True
|
|
for metric in ("line_percent", "branch_percent"):
|
|
value = float(current["totals"][metric])
|
|
target = float(previous["totals"][metric])
|
|
delta = value - target
|
|
print(f"Baseline {metric}: proposed={value:.6f}% target={target:.6f}% delta={delta:+.6f}pp")
|
|
if value < target:
|
|
passed = False
|
|
if passed:
|
|
print("Coverage baseline update is monotonic.")
|
|
else:
|
|
print("Coverage baseline files may not lower line or branch coverage.")
|
|
return passed
|
|
|
|
|
|
def check_ratchet(report: dict[str, Any], baseline: dict[str, Any], tolerance: float, limit: int) -> bool:
|
|
if baseline.get("schema_version") != SCHEMA_VERSION:
|
|
raise ValueError(f"unsupported baseline schema: {baseline.get('schema_version')!r}")
|
|
current = _metrics(report["totals"])
|
|
expected = baseline["totals"]
|
|
checks = (
|
|
("line", current.line_percent, float(expected["line_percent"])),
|
|
("branch", current.branch_percent, float(expected["branch_percent"])),
|
|
)
|
|
failed = [(name, value, target) for name, value, target in checks if value + tolerance < target]
|
|
for name, value, target in checks:
|
|
delta = value - target
|
|
print(f"Ratchet {name}: current={value:.2f}% baseline={target:.2f}% delta={delta:+.2f}pp")
|
|
if not failed:
|
|
print(f"Coverage ratchet passed with {tolerance:.2f}pp tolerance.")
|
|
return True
|
|
|
|
print("Largest per-file coverage declines:")
|
|
declines = _file_declines(report, baseline)
|
|
for decline, name, line_delta, branch_delta in declines[:limit]:
|
|
print(f" {decline:6.2f}pp line {line_delta:+7.2f}pp branch {branch_delta:+7.2f}pp {name}")
|
|
return False
|
|
|
|
|
|
def _file_declines(report: dict[str, Any], baseline: dict[str, Any]) -> list[tuple[float, str, float, float]]:
|
|
declines: list[tuple[float, str, float, float]] = []
|
|
current_files = report["files"]
|
|
for name, expected in baseline["files"].items():
|
|
data = current_files.get(name)
|
|
if data is None:
|
|
continue
|
|
current = _metrics(data["summary"])
|
|
line_delta = current.line_percent - float(expected["line_percent"])
|
|
branch_delta = current.branch_percent - float(expected["branch_percent"])
|
|
decline = max(-line_delta, -branch_delta, 0.0)
|
|
if decline > 0:
|
|
declines.append((decline, name, line_delta, branch_delta))
|
|
return sorted(declines, key=lambda item: (-item[0], item[1]))
|
|
|
|
|
|
_HUNK_RE = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")
|
|
|
|
|
|
def parse_changed_lines(diff: str) -> dict[str, set[int]]:
|
|
"""Return new-side hunk lines; deleted lines never enter the denominator."""
|
|
changed: dict[str, set[int]] = {}
|
|
path: str | None = None
|
|
for line in diff.splitlines():
|
|
if line.startswith("+++ "):
|
|
# Reset on a non-b/ target (e.g. +++ /dev/null) so a deleted file's
|
|
# hunks are not misattributed to the previously seen path.
|
|
path = line[6:] if line.startswith("+++ b/") else None
|
|
if path is not None:
|
|
changed.setdefault(path, set())
|
|
continue
|
|
match = _HUNK_RE.match(line)
|
|
if path is None or match is None:
|
|
continue
|
|
start = int(match.group(1))
|
|
count = int(match.group(2) or "1")
|
|
changed[path].update(range(start, start + count))
|
|
return changed
|
|
|
|
|
|
def calculate_diff_coverage(
|
|
report: dict[str, Any], changed: dict[str, set[int]], added_files: Iterable[str] = ()
|
|
) -> DiffCoverage:
|
|
"""Intersect changed lines with coverage.py statements to ignore comments and blanks."""
|
|
executable: set[tuple[str, int]] = set()
|
|
uncovered: set[tuple[str, int]] = set()
|
|
zero_coverage_files: list[str] = []
|
|
unmeasured_files: list[str] = []
|
|
added = set(added_files)
|
|
for name, lines in changed.items():
|
|
data = report["files"].get(name)
|
|
if data is None:
|
|
if name not in OMITTED_PATHS:
|
|
unmeasured_files.append(name)
|
|
continue
|
|
executed = set(data["executed_lines"])
|
|
missing = set(data["missing_lines"])
|
|
statements = executed | missing
|
|
for line in lines & statements:
|
|
item = (name, line)
|
|
executable.add(item)
|
|
if line in missing:
|
|
uncovered.add(item)
|
|
if name in added and statements and not executed:
|
|
zero_coverage_files.append(name)
|
|
return DiffCoverage(
|
|
frozenset(executable),
|
|
frozenset(uncovered),
|
|
tuple(sorted(zero_coverage_files)),
|
|
tuple(sorted(unmeasured_files)),
|
|
)
|
|
|
|
|
|
def _git(*args: str) -> str:
|
|
executable = shutil.which("git")
|
|
if executable is None:
|
|
raise OSError("git is required for coverage diff checks")
|
|
completed = subprocess.run([executable, *args], check=True, capture_output=True, text=True)
|
|
return completed.stdout
|
|
|
|
|
|
def load_baseline_from_ref(base_ref: str, path: Path) -> dict[str, Any] | None:
|
|
_git("rev-parse", "--verify", f"{base_ref}^{{commit}}")
|
|
try:
|
|
content = _git("show", f"{base_ref}:{path.as_posix()}")
|
|
except subprocess.CalledProcessError:
|
|
return None
|
|
return json.loads(content)
|
|
|
|
|
|
def git_diff_inputs(base_ref: str, head: str | None) -> tuple[dict[str, set[int]], set[str]]:
|
|
"""Resolve the merge base and collect Python changes for CI or the local worktree."""
|
|
merge_base = _git("merge-base", base_ref, "HEAD").strip()
|
|
end = head or ""
|
|
range_args = [merge_base, end] if end else [merge_base]
|
|
diff = _git("diff", "--unified=0", "--no-color", "--diff-filter=ACMR", *range_args, "--", *PRODUCTION_PATHSPECS)
|
|
names = _git("diff", "--name-only", "--diff-filter=A", *range_args, "--", *PRODUCTION_PATHSPECS)
|
|
return parse_changed_lines(diff), {name for name in names.splitlines() if name}
|
|
|
|
|
|
def check_diff(report: dict[str, Any], base_ref: str, head: str | None, threshold: float) -> bool:
|
|
changed, added = git_diff_inputs(base_ref, head)
|
|
result = calculate_diff_coverage(report, changed, added)
|
|
covered = len(result.executable) - len(result.uncovered)
|
|
print(f"Diff coverage: {result.percent:.2f}% ({covered}/{len(result.executable)} executable changed lines)")
|
|
if result.uncovered:
|
|
print("Uncovered changed lines:")
|
|
for name, line in sorted(result.uncovered):
|
|
print(f" {name}:{line}")
|
|
if result.zero_coverage_files:
|
|
print("New production files with 0% line coverage:")
|
|
for name in result.zero_coverage_files:
|
|
print(f" {name}")
|
|
if result.unmeasured_files:
|
|
print("Changed production files missing from the coverage report:")
|
|
for name in result.unmeasured_files:
|
|
print(f" {name}")
|
|
passed = result.percent + 1e-9 >= threshold and not result.zero_coverage_files and not result.unmeasured_files
|
|
if passed:
|
|
print(f"Diff coverage passed the {threshold:.2f}% threshold.")
|
|
else:
|
|
print(f"Diff coverage failed the {threshold:.2f}% threshold.")
|
|
return passed
|
|
|
|
|
|
def _parser() -> argparse.ArgumentParser:
|
|
parser = argparse.ArgumentParser(description=__doc__)
|
|
parser.add_argument("--coverage-json", type=Path, default=Path("coverage.json"))
|
|
subparsers = parser.add_subparsers(dest="command", required=True)
|
|
|
|
summary = subparsers.add_parser("summary")
|
|
summary.add_argument("--limit", type=int, default=10)
|
|
|
|
baseline = subparsers.add_parser("baseline")
|
|
baseline.add_argument("--output", type=Path, default=Path("coverage-baseline-candidate.json"))
|
|
baseline.add_argument("--commit", default="HEAD")
|
|
|
|
baseline_check = subparsers.add_parser("baseline-check")
|
|
baseline_check.add_argument("--baseline", type=Path, default=Path(".github/coverage-baseline.json"))
|
|
baseline_check.add_argument("--base-ref", required=True)
|
|
|
|
ratchet = subparsers.add_parser("ratchet")
|
|
ratchet.add_argument("--baseline", type=Path, default=Path(".github/coverage-baseline.json"))
|
|
ratchet.add_argument("--tolerance", type=float, default=DEFAULT_TOLERANCE)
|
|
ratchet.add_argument("--limit", type=int, default=10)
|
|
|
|
diff = subparsers.add_parser("diff")
|
|
diff.add_argument("--base-ref", required=True)
|
|
diff.add_argument("--head", default=None)
|
|
diff.add_argument("--threshold", type=float, default=DEFAULT_DIFF_THRESHOLD)
|
|
return parser
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
args = _parser().parse_args(argv)
|
|
try:
|
|
if args.command == "baseline-check":
|
|
with args.baseline.open(encoding="utf-8") as handle:
|
|
current_baseline = json.load(handle)
|
|
previous_baseline = load_baseline_from_ref(args.base_ref, args.baseline)
|
|
if previous_baseline is None:
|
|
print(f"No coverage baseline exists on {args.base_ref}; accepting the initial bootstrap baseline.")
|
|
return 0
|
|
return 0 if check_baseline_update(current_baseline, previous_baseline) else 1
|
|
|
|
report = load_coverage(args.coverage_json)
|
|
if args.command == "summary":
|
|
print_summary(report, args.limit)
|
|
return 0
|
|
if args.command == "baseline":
|
|
commit = _git("rev-parse", args.commit).strip()
|
|
write_baseline(report, args.output, commit)
|
|
return 0
|
|
if args.command == "ratchet":
|
|
with args.baseline.open(encoding="utf-8") as handle:
|
|
baseline = json.load(handle)
|
|
return 0 if check_ratchet(report, baseline, args.tolerance, args.limit) else 1
|
|
if args.command == "diff":
|
|
return 0 if check_diff(report, args.base_ref, args.head, args.threshold) else 1
|
|
except (OSError, ValueError, KeyError, json.JSONDecodeError, subprocess.CalledProcessError) as exc:
|
|
print(f"coverage gate error: {exc}", file=sys.stderr)
|
|
return 2
|
|
raise AssertionError(f"unhandled command: {args.command}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|