Raven-1/scripts/skill_hub_e2e.py
zhanghui a018351f95
chore(*): gate call correctness with a pinned type checker (#416)
## Summary

Nothing in this repo could see a signature change reach its callers.
Ruff reads one file at a time, the import contracts read edges rather
than calls, and the suite only covers lines it executes. The consequence
was concrete: a required `sandbox_home` argument added to
`BoxliteExecutor` reached `plugins-dist/design-engine`'s render path
with CI green, and a reviewer -- not a gate -- found it.

`ty` answers that question. This adds it as a pinned dev dependency, a
`make lint-types` target and a CI step, with **110 of its 133 rules
live**: nine switched off with their measured backlog recorded beside
each, fourteen shipped disabled upstream and untouched.

Getting there meant clearing 81 diagnostics. This PR folds what were
three stacked PRs (#422 and #423, now closed); each rule's story is a
commit of its own.

### 1. The gate, at 108 rules (`chore(*): gate call correctness...`)

21 diagnostics to clear, five of them real:

| Where | What |
|---|---|
| `evolver/orchestrator/production.py` | `design_of` annotated as taking
two parameters while its docstring, its only implementation and its one
call site all use three |
| `raven/cli/provider_commands.py` | `dict[str, callable]` -- the
builtin function, not a type, so the annotation asserted nothing |
| `raven/config/schema.py` | `Config` is a `BaseSettings` but declared
`env_prefix` / `env_nested_delimiter` through pydantic's `ConfigDict`,
which does not carry those keys |
| `raven/spine/delivery.py` | `StreamDelta.conversation_id` is optional
and was used as a `dict[str, str]` key; a delta without one stored a
`None` entry the close path then read as "no open stream", losing that
stream's done marker |
| `research_flow/support/harness_text.py` | a predicate table typed its
matchers as `object`, so nothing checked they are callable |

The rest were unproven invariants and annotations wider than the code:
three late-bound callables asserted where a comment used to carry the
claim, one closure binding the callable it guarded instead of re-reading
it, four `hasattr` probes that now check what the call actually needs,
and a contextmanager yielding the value it just set.

### 2. `unresolved-import` (`fix(*): resolve the imports the type gate
could not`)

21 diagnostics; three were stale module paths nothing had run into:

- `HarnessNode.current_core_version()` read `raven.__core_version__`, a
module that exists neither here nor in the wheel
`scripts/build_core_wheel.py` produces. The read sat inside `try/except
ImportError`, so the except branch was the only branch and **every node
`store.py` saved recorded `core_version="unknown"`**. It reads
`raven.__version__` now and returns `0.1.13`.
- `raven/agent/loop/wiring.py` imported `SkillForgeRouterConfig` from
`raven.config.schema`; it lives in `raven.config.raven`. Under
`TYPE_CHECKING`, so the annotation simply meant nothing.
- three scripts imported `raven.context_engine.base` and
`raven.memory_engine.base`, neither of which exists. **Two failed at
module import** -- running them was not possible.

The other eighteen were the environment: the three agent products keep
their plugin package beside them, so their roots are declared in
`[tool.ty.environment]`; `lxml` ships a compiled `etree` with no type
information, so `lxml-stubs` is a dev dependency (real types rather than
five silenced lines); `uno` and `com.sun.star.*` are suppressed per
line, since that worker runs inside LibreOffice's own interpreter.

### 3. `invalid-return-type` (`fix(*): make the declared return types
true`)

39 diagnostics, and **one stale annotation held eight of them**:
`run_turn` and the `_run_turn` behind it were declared `-> LoopOutcome`
while all three of their returns build a `TurnOutcome` -- a different
dataclass with no field in common. Nothing broke, because the spine
worker reads what it actually gets and `LoopOutcome` never leaves
`_run_agent_loop`. The annotation was the only thing wrong.

The other 31: nine annotations narrower than the code (five `return
tuple(<genexpr>)` typed as fixed-arity tuples, an inner `one()` still
declaring the two-tuple it returned before a third element was added,
`image_block`'s TypedDict), five declarations refusing a value the
function returns (four tool `execute` overrides saying `-> str` while
returning `ToolResult`), seven invariants that lived in a `# type:
ignore` comment, and ten honest widenings.

### 4. What the gates then caught in this PR's own work

- The qq factory guard passed locally only because an earlier test had
left an event loop behind. Reproduced the CI state
(`asyncio.set_event_loop(None)`), then made the test async so the loop
is its own.
- `invalid-return-type`, while still disabled, hid a regression
introduced here: the qq factory returning the class instead of an
instance. Found by diffing the rule histogram, which is why that rule is
on now.
- The diff-coverage gate named four lines in `raven/routing/cache.py` --
the asserts this work had just added, sitting inside a `try` whose
`except Exception` would have swallowed the `AssertionError` and served
the stale cache. `_do_refresh` returns its data now; the asserts, their
noqa and the swallowing all go with it.

`child_scope`, `QQChannel.start()` and `BenchmarkCache` had no tests at
all and have them now. Every one was mutation-checked.

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [x] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- `make lint-types` -> All checks passed; 81 diagnostics cleared across
the three rule sets
- Gate proven live three times, once per rule: a missing argument, an
import of a module that does not exist, and a function returning a str
under `-> int` each turn it red, and removing the probe turns it green
- `uv run ruff check` + `ruff format --check` over all six trees ->
clean
- `uv run lint-imports` -> 10 kept, 0 broken; `deptry raven` (all
extras) -> no issues; `uv lock --check` -> resolved
- `make check-source-language` -> exit 0; `commitlint` -> exit 0
- `make coverage` locally: **22050 passed, 45 skipped, 4 failed**; diff
coverage 92.59%
- All four failures are **pre-existing on `main`**, checked by a real
checkout in the same environment where they fail identically: two
`test_the_products_tool_face_*` launcher tests,
`test_a_ceiling_as_large_as_the_window_still_leaves_room_for_history`
(`reserved_tools = 6044 > 6_000`), and
`test_a_missing_key_is_said_once_rather_than_per_query`
(environment-dependent: it asserts `SERPER_API_KEY` is absent)
- Rule counts measured with both `--extra dev` and `--all-extras` --
identical, so this gate's verdict does not depend on which optional
packages are installed

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

## Risk

Behaviour changes are confined to the fixes listed above, and each
replaces an error path with a correct one: nodes are stamped with the
real raven version instead of `"unknown"`, two scripts that could not
start now start, the stream bookkeeping no longer stores a `None` key,
and a cache refresh returns what it fetched. Everything else is a
declaration moved to match what the function already returned.

`tests/` is out of the gate's scope (113 diagnostics under the same
rules). Tests are executed, so a broken call there is already loud; this
gate is for code CI does not run. Bringing them in is sized, separate
work.

Rollback: revert the whole PR to drop the gate and its fixes together,
or delete the `[tool.ty]` block, the `lint-types` target and its CI step
to keep the fixes and drop only the gate.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
2026-09-16 07:33:14 -07:00

170 lines
6.3 KiB
Python

"""Skill Hub e2e probe.
Exercises the full skill_hub client surface against a real Hub:
``search → get → install`` plus a SkillsSegmentBuilder dry-run that
hits the rewriter → router → pre-gate body hydrate → gate → post-gate
refs hydrate pipeline end-to-end.
Run from the project root:
.venv/bin/python scripts/skill_hub_e2e.py \\
--endpoint https://dev-skillhub.aws.evermind.ai \\
--query "generate pdf report"
Requires network reach to the Hub endpoint. In containers that route
through a Squid proxy refusing CONNECT to ``*.evermind.ai``, this
script will fail at the first ``search`` call with a network error —
run from a machine with direct internet or whitelist the host.
Skips LLM-driven stages (rewriter / gate) unless ``--provider <id>``
is supplied; without a provider, the segment builder runs in
no-rewriter / no-gate mode (router + hydrate only) so the script is
useful even when no LLM credentials are wired.
"""
from __future__ import annotations
import argparse
import asyncio
import sys
import textwrap
from pathlib import Path
# Allow `python scripts/skill_hub_e2e.py` without `-m`.
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from raven.memory_engine.skill_forge import (
HubSkillSource,
SkillForgeRouter,
)
from raven.skill_hub import SkillHubClient
async def probe_client(endpoint: str, api_key: str | None, query: str, limit: int) -> None:
"""Stage 1 — direct SkillHubClient roundtrip (search → get → install)."""
print("\n=== Stage 1: SkillHubClient direct ===")
client = SkillHubClient(endpoint, api_key=api_key, timeout_s=10.0)
try:
print(f"GET /openapi/v1/skills?q={query!r}&limit={limit} ...")
items = await client.search(query, limit=limit)
print(f" → {len(items)} item(s)")
for it in items[:3]:
print(f" - {it.get('name')!r:40s} id={it.get('id')} q={it.get('quality_score')}")
if not items:
print(" (empty result — nothing more to probe)")
return
top = items[0]
print(f"\nGET /openapi/v1/skills/{top['id']} ...")
meta = await client.get(top["id"])
body = meta.get("skill_md", "") or ""
print(f" → slug={meta.get('slug')} version={meta.get('version')} body_chars={len(body)}")
if body:
preview = body.splitlines()[:3]
print(" body head:")
for line in preview:
print(f" {line[:80]}")
print(f"\ninstall({top['id']}, prefetched_meta=<from get>) ...")
installed = await client.install(top["id"], prefetched_meta=meta)
print(f" → dir={installed.get('dir')}")
print(f" scripts_dir={installed.get('scripts_dir')}")
bundle_dir = Path(installed["dir"]) if installed.get("dir") else None
if bundle_dir and bundle_dir.is_dir():
entries = sorted(p.name for p in bundle_dir.iterdir())
print(f" files: {entries[:10]}{'...' if len(entries) > 10 else ''}")
finally:
await client.aclose()
async def probe_hub_source(endpoint: str, api_key: str | None, query: str, k: int) -> None:
"""Stage 2 — HubSkillSource → RouterHit mapping (Tier 0 catalog)."""
print("\n=== Stage 2: HubSkillSource → RouterHit ===")
client = SkillHubClient(endpoint, api_key=api_key, timeout_s=10.0)
try:
src = HubSkillSource(client)
hits = await src.search(query, [], k=k)
print(f" → {len(hits)} RouterHit(s); body empty (catalog only):")
for h in hits[:5]:
assert h.content == "", "HubSkillSource must emit empty content"
print(f" {h.qualified_id} name={h.name!r} score={h.score:.3f}")
finally:
await client.aclose()
async def probe_segment(
endpoint: str,
api_key: str | None,
query: str,
k: int,
) -> None:
"""Stage 3 — SkillsSegmentBuilder body hydrate (no rewriter / gate)."""
print("\n=== Stage 3: SkillsSegmentBuilder hydrate (no LLM) ===")
from raven.context_engine.segments.skills import SkillsSegmentBuilder
from raven.contracts.assembled import TokenBudget
from raven.contracts.context import AssemblyContext
client = SkillHubClient(endpoint, api_key=api_key, timeout_s=10.0)
try:
hub_source = HubSkillSource(client)
router = SkillForgeRouter([hub_source])
builder = SkillsSegmentBuilder(
router,
skill_top_k=k,
hub_client=client,
)
ctx = AssemblyContext(
session_key="e2e",
current_message=query,
media=None,
channel=None,
chat_id=None,
session_messages=[],
budget=TokenBudget(
context_length=200_000,
reserved_output=8000,
reserved_tools=4000,
reserved_system=4000,
available_history=184_000,
),
)
seg = await builder.build(ctx)
ids = seg.meta.get("injected_skill_ids", [])
print(f" → injected_skill_ids: {ids}")
print(f" text length: {len(seg.text)} chars")
print(" text preview:")
for line in textwrap.indent(seg.text[:1200], " ").splitlines()[:25]:
print(line)
finally:
await client.aclose()
async def main() -> None:
p = argparse.ArgumentParser(description="Skill Hub e2e probe")
p.add_argument("--endpoint", required=True, help="Hub base URL")
p.add_argument("--api-key", default=None, help="Bearer token (optional)")
p.add_argument("--query", default="generate pdf report")
p.add_argument("--limit", type=int, default=5)
p.add_argument("--k", type=int, default=3, help="top-K for hub source / segment")
p.add_argument(
"--stage",
choices=["1", "2", "3", "all"],
default="all",
help="Which stage to run (1=client, 2=hub_source, 3=segment).",
)
args = p.parse_args()
if args.stage in {"1", "all"}:
await probe_client(args.endpoint, args.api_key, args.query, args.limit)
if args.stage in {"2", "all"}:
await probe_hub_source(args.endpoint, args.api_key, args.query, args.k)
if args.stage in {"3", "all"}:
await probe_segment(args.endpoint, args.api_key, args.query, args.k)
print("\n✓ all stages completed")
if __name__ == "__main__":
asyncio.run(main())