## Summary Nothing in this repo could see a signature change reach its callers. Ruff reads one file at a time, the import contracts read edges rather than calls, and the suite only covers lines it executes. The consequence was concrete: a required `sandbox_home` argument added to `BoxliteExecutor` reached `plugins-dist/design-engine`'s render path with CI green, and a reviewer -- not a gate -- found it. `ty` answers that question. This adds it as a pinned dev dependency, a `make lint-types` target and a CI step, with **110 of its 133 rules live**: nine switched off with their measured backlog recorded beside each, fourteen shipped disabled upstream and untouched. Getting there meant clearing 81 diagnostics. This PR folds what were three stacked PRs (#422 and #423, now closed); each rule's story is a commit of its own. ### 1. The gate, at 108 rules (`chore(*): gate call correctness...`) 21 diagnostics to clear, five of them real: | Where | What | |---|---| | `evolver/orchestrator/production.py` | `design_of` annotated as taking two parameters while its docstring, its only implementation and its one call site all use three | | `raven/cli/provider_commands.py` | `dict[str, callable]` -- the builtin function, not a type, so the annotation asserted nothing | | `raven/config/schema.py` | `Config` is a `BaseSettings` but declared `env_prefix` / `env_nested_delimiter` through pydantic's `ConfigDict`, which does not carry those keys | | `raven/spine/delivery.py` | `StreamDelta.conversation_id` is optional and was used as a `dict[str, str]` key; a delta without one stored a `None` entry the close path then read as "no open stream", losing that stream's done marker | | `research_flow/support/harness_text.py` | a predicate table typed its matchers as `object`, so nothing checked they are callable | The rest were unproven invariants and annotations wider than the code: three late-bound callables asserted where a comment used to carry the claim, one closure binding the callable it guarded instead of re-reading it, four `hasattr` probes that now check what the call actually needs, and a contextmanager yielding the value it just set. ### 2. `unresolved-import` (`fix(*): resolve the imports the type gate could not`) 21 diagnostics; three were stale module paths nothing had run into: - `HarnessNode.current_core_version()` read `raven.__core_version__`, a module that exists neither here nor in the wheel `scripts/build_core_wheel.py` produces. The read sat inside `try/except ImportError`, so the except branch was the only branch and **every node `store.py` saved recorded `core_version="unknown"`**. It reads `raven.__version__` now and returns `0.1.13`. - `raven/agent/loop/wiring.py` imported `SkillForgeRouterConfig` from `raven.config.schema`; it lives in `raven.config.raven`. Under `TYPE_CHECKING`, so the annotation simply meant nothing. - three scripts imported `raven.context_engine.base` and `raven.memory_engine.base`, neither of which exists. **Two failed at module import** -- running them was not possible. The other eighteen were the environment: the three agent products keep their plugin package beside them, so their roots are declared in `[tool.ty.environment]`; `lxml` ships a compiled `etree` with no type information, so `lxml-stubs` is a dev dependency (real types rather than five silenced lines); `uno` and `com.sun.star.*` are suppressed per line, since that worker runs inside LibreOffice's own interpreter. ### 3. `invalid-return-type` (`fix(*): make the declared return types true`) 39 diagnostics, and **one stale annotation held eight of them**: `run_turn` and the `_run_turn` behind it were declared `-> LoopOutcome` while all three of their returns build a `TurnOutcome` -- a different dataclass with no field in common. Nothing broke, because the spine worker reads what it actually gets and `LoopOutcome` never leaves `_run_agent_loop`. The annotation was the only thing wrong. The other 31: nine annotations narrower than the code (five `return tuple(<genexpr>)` typed as fixed-arity tuples, an inner `one()` still declaring the two-tuple it returned before a third element was added, `image_block`'s TypedDict), five declarations refusing a value the function returns (four tool `execute` overrides saying `-> str` while returning `ToolResult`), seven invariants that lived in a `# type: ignore` comment, and ten honest widenings. ### 4. What the gates then caught in this PR's own work - The qq factory guard passed locally only because an earlier test had left an event loop behind. Reproduced the CI state (`asyncio.set_event_loop(None)`), then made the test async so the loop is its own. - `invalid-return-type`, while still disabled, hid a regression introduced here: the qq factory returning the class instead of an instance. Found by diffing the rule histogram, which is why that rule is on now. - The diff-coverage gate named four lines in `raven/routing/cache.py` -- the asserts this work had just added, sitting inside a `try` whose `except Exception` would have swallowed the `AssertionError` and served the stale cache. `_do_refresh` returns its data now; the asserts, their noqa and the swallowing all go with it. `child_scope`, `QQChannel.start()` and `BenchmarkCache` had no tests at all and have them now. Every one was mutation-checked. ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [x] CI / tooling - [ ] Refactor - [ ] Other ## Verification - `make lint-types` -> All checks passed; 81 diagnostics cleared across the three rule sets - Gate proven live three times, once per rule: a missing argument, an import of a module that does not exist, and a function returning a str under `-> int` each turn it red, and removing the probe turns it green - `uv run ruff check` + `ruff format --check` over all six trees -> clean - `uv run lint-imports` -> 10 kept, 0 broken; `deptry raven` (all extras) -> no issues; `uv lock --check` -> resolved - `make check-source-language` -> exit 0; `commitlint` -> exit 0 - `make coverage` locally: **22050 passed, 45 skipped, 4 failed**; diff coverage 92.59% - All four failures are **pre-existing on `main`**, checked by a real checkout in the same environment where they fail identically: two `test_the_products_tool_face_*` launcher tests, `test_a_ceiling_as_large_as_the_window_still_leaves_room_for_history` (`reserved_tools = 6044 > 6_000`), and `test_a_missing_key_is_said_once_rather_than_per_query` (environment-dependent: it asserts `SERPER_API_KEY` is absent) - Rule counts measured with both `--extra dev` and `--all-extras` -- identical, so this gate's verdict does not depend on which optional packages are installed - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed ## Risk Behaviour changes are confined to the fixes listed above, and each replaces an error path with a correct one: nodes are stamped with the real raven version instead of `"unknown"`, two scripts that could not start now start, the stream bookkeeping no longer stores a `None` key, and a cache refresh returns what it fetched. Everything else is a declaration moved to match what the function already returned. `tests/` is out of the gate's scope (113 diagnostics under the same rules). Tests are executed, so a broken call there is already loud; this gate is for code CI does not run. Bringing them in is sized, separate work. Rollback: revert the whole PR to drop the gate and its fixes together, or delete the `[tool.ty]` block, the `lint-types` target and its CI step to keep the fixes and drop only the gate. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
170 lines
6.3 KiB
Python
170 lines
6.3 KiB
Python
"""Skill Hub e2e probe.
|
|
|
|
Exercises the full skill_hub client surface against a real Hub:
|
|
``search → get → install`` plus a SkillsSegmentBuilder dry-run that
|
|
hits the rewriter → router → pre-gate body hydrate → gate → post-gate
|
|
refs hydrate pipeline end-to-end.
|
|
|
|
Run from the project root:
|
|
|
|
.venv/bin/python scripts/skill_hub_e2e.py \\
|
|
--endpoint https://dev-skillhub.aws.evermind.ai \\
|
|
--query "generate pdf report"
|
|
|
|
Requires network reach to the Hub endpoint. In containers that route
|
|
through a Squid proxy refusing CONNECT to ``*.evermind.ai``, this
|
|
script will fail at the first ``search`` call with a network error —
|
|
run from a machine with direct internet or whitelist the host.
|
|
|
|
Skips LLM-driven stages (rewriter / gate) unless ``--provider <id>``
|
|
is supplied; without a provider, the segment builder runs in
|
|
no-rewriter / no-gate mode (router + hydrate only) so the script is
|
|
useful even when no LLM credentials are wired.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import asyncio
|
|
import sys
|
|
import textwrap
|
|
from pathlib import Path
|
|
|
|
# Allow `python scripts/skill_hub_e2e.py` without `-m`.
|
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
|
|
|
from raven.memory_engine.skill_forge import (
|
|
HubSkillSource,
|
|
SkillForgeRouter,
|
|
)
|
|
from raven.skill_hub import SkillHubClient
|
|
|
|
|
|
async def probe_client(endpoint: str, api_key: str | None, query: str, limit: int) -> None:
|
|
"""Stage 1 — direct SkillHubClient roundtrip (search → get → install)."""
|
|
print("\n=== Stage 1: SkillHubClient direct ===")
|
|
client = SkillHubClient(endpoint, api_key=api_key, timeout_s=10.0)
|
|
try:
|
|
print(f"GET /openapi/v1/skills?q={query!r}&limit={limit} ...")
|
|
items = await client.search(query, limit=limit)
|
|
print(f" → {len(items)} item(s)")
|
|
for it in items[:3]:
|
|
print(f" - {it.get('name')!r:40s} id={it.get('id')} q={it.get('quality_score')}")
|
|
|
|
if not items:
|
|
print(" (empty result — nothing more to probe)")
|
|
return
|
|
|
|
top = items[0]
|
|
print(f"\nGET /openapi/v1/skills/{top['id']} ...")
|
|
meta = await client.get(top["id"])
|
|
body = meta.get("skill_md", "") or ""
|
|
print(f" → slug={meta.get('slug')} version={meta.get('version')} body_chars={len(body)}")
|
|
if body:
|
|
preview = body.splitlines()[:3]
|
|
print(" body head:")
|
|
for line in preview:
|
|
print(f" {line[:80]}")
|
|
|
|
print(f"\ninstall({top['id']}, prefetched_meta=<from get>) ...")
|
|
installed = await client.install(top["id"], prefetched_meta=meta)
|
|
print(f" → dir={installed.get('dir')}")
|
|
print(f" scripts_dir={installed.get('scripts_dir')}")
|
|
bundle_dir = Path(installed["dir"]) if installed.get("dir") else None
|
|
if bundle_dir and bundle_dir.is_dir():
|
|
entries = sorted(p.name for p in bundle_dir.iterdir())
|
|
print(f" files: {entries[:10]}{'...' if len(entries) > 10 else ''}")
|
|
finally:
|
|
await client.aclose()
|
|
|
|
|
|
async def probe_hub_source(endpoint: str, api_key: str | None, query: str, k: int) -> None:
|
|
"""Stage 2 — HubSkillSource → RouterHit mapping (Tier 0 catalog)."""
|
|
print("\n=== Stage 2: HubSkillSource → RouterHit ===")
|
|
client = SkillHubClient(endpoint, api_key=api_key, timeout_s=10.0)
|
|
try:
|
|
src = HubSkillSource(client)
|
|
hits = await src.search(query, [], k=k)
|
|
print(f" → {len(hits)} RouterHit(s); body empty (catalog only):")
|
|
for h in hits[:5]:
|
|
assert h.content == "", "HubSkillSource must emit empty content"
|
|
print(f" {h.qualified_id} name={h.name!r} score={h.score:.3f}")
|
|
finally:
|
|
await client.aclose()
|
|
|
|
|
|
async def probe_segment(
|
|
endpoint: str,
|
|
api_key: str | None,
|
|
query: str,
|
|
k: int,
|
|
) -> None:
|
|
"""Stage 3 — SkillsSegmentBuilder body hydrate (no rewriter / gate)."""
|
|
print("\n=== Stage 3: SkillsSegmentBuilder hydrate (no LLM) ===")
|
|
from raven.context_engine.segments.skills import SkillsSegmentBuilder
|
|
from raven.contracts.assembled import TokenBudget
|
|
from raven.contracts.context import AssemblyContext
|
|
|
|
client = SkillHubClient(endpoint, api_key=api_key, timeout_s=10.0)
|
|
try:
|
|
hub_source = HubSkillSource(client)
|
|
router = SkillForgeRouter([hub_source])
|
|
builder = SkillsSegmentBuilder(
|
|
router,
|
|
skill_top_k=k,
|
|
hub_client=client,
|
|
)
|
|
ctx = AssemblyContext(
|
|
session_key="e2e",
|
|
current_message=query,
|
|
media=None,
|
|
channel=None,
|
|
chat_id=None,
|
|
session_messages=[],
|
|
budget=TokenBudget(
|
|
context_length=200_000,
|
|
reserved_output=8000,
|
|
reserved_tools=4000,
|
|
reserved_system=4000,
|
|
available_history=184_000,
|
|
),
|
|
)
|
|
seg = await builder.build(ctx)
|
|
ids = seg.meta.get("injected_skill_ids", [])
|
|
print(f" → injected_skill_ids: {ids}")
|
|
print(f" text length: {len(seg.text)} chars")
|
|
print(" text preview:")
|
|
for line in textwrap.indent(seg.text[:1200], " ").splitlines()[:25]:
|
|
print(line)
|
|
finally:
|
|
await client.aclose()
|
|
|
|
|
|
async def main() -> None:
|
|
p = argparse.ArgumentParser(description="Skill Hub e2e probe")
|
|
p.add_argument("--endpoint", required=True, help="Hub base URL")
|
|
p.add_argument("--api-key", default=None, help="Bearer token (optional)")
|
|
p.add_argument("--query", default="generate pdf report")
|
|
p.add_argument("--limit", type=int, default=5)
|
|
p.add_argument("--k", type=int, default=3, help="top-K for hub source / segment")
|
|
p.add_argument(
|
|
"--stage",
|
|
choices=["1", "2", "3", "all"],
|
|
default="all",
|
|
help="Which stage to run (1=client, 2=hub_source, 3=segment).",
|
|
)
|
|
args = p.parse_args()
|
|
|
|
if args.stage in {"1", "all"}:
|
|
await probe_client(args.endpoint, args.api_key, args.query, args.limit)
|
|
if args.stage in {"2", "all"}:
|
|
await probe_hub_source(args.endpoint, args.api_key, args.query, args.k)
|
|
if args.stage in {"3", "all"}:
|
|
await probe_segment(args.endpoint, args.api_key, args.query, args.k)
|
|
|
|
print("\n✓ all stages completed")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
asyncio.run(main())
|