## Summary A read-only audit of all 298 test files found the suite in good shape -- exactly one genuinely fake test -- with the real debt sitting in file names and a scattering of assertions that passed for the wrong reason. This clears that, and fixes the bugs found while clearing it. **Three dead tests dealt with.** Two removed because neither could ever assert anything: - `test_chat_e2e.py` was a module-level skip over three `NotImplementedError` stubs. The wire-up it waited for has shipped (`tui_commands` instantiates `SubscriptionEmitter`) and the proposal doc it cited is gone. - `test_tui_exit_e2e.py` asserted that building the agent loop leaves lancedb's thread live. It does not: the loop is constructed but never started, and the memory plugin talks to everos over HTTP, so lancedb is not even imported. The assertion could not hold in any configuration. The third, `slashParity.test.ts`, shelled out to python for a `raven_cli.commands.COMMAND_REGISTRY` that no longer exists, and skipped all three of its cases when the import failed -- three skips on every run. The parity it wanted needs no python: `createSlashHandler.ts:45-51` routes a slash locally when `findSlashCommand` resolves it and otherwise hands it to `slash.exec`, a CLI subprocess a mutation cannot reach the live session from. It now asserts that predicate directly. `NATIVE_MUTATING_COMMANDS` and the three-way `classifyRoute` went with it: all six names in that set are also in the local registry, so the native branch could not change any outcome. **19 files renamed** off phase, ticket and bug codes (AGENTS.md section 5.1), including one reversed CLI name and two integration files moved onto a legal `<kind>`. Reference updates ride along: `.pre-commit-config.yaml` names one of these files in a `detect-private-key` exclude, and three docs pages point at them. **Assertions that passed for the wrong reason, tightened**: eight over-broad `pytest.raises(Exception)` narrowed to the type each site actually raises; an `except` clause that swallowed the `AssertionError` its own fakes use as a tripwire; a `hasattr` probe replaced by binding a broker and checking it stuck; and a recall assertion that read as a pass on an empty result set. **Dead weight removed**: ten unused locals, three dead code blocks, six `importorskip` guards on hard dependencies, two hardcoded `/tmp` paths, and ticket codes left in docstrings after the renames. **Two coverage gaps the deletions exposed are now closed**: - `tests/test_cli_exit.py` pins the hard-exit guard, which had no tests at all. - The TUI chat e2e gains a multi-turn case. It reads the persisted session rather than the screen, because a prompt is echoed into the transcript the moment it is typed -- screen-scraping for a planted word passes even when history is broken. **Three timing bugs fixed in the TUI e2e tests.** They keyed off the banner, which paints about seven seconds before the app accepts input, and submitted the moment text was typed. Dropped prompts made the pipeline look dead in one test, and made a negative assertion (`assert not leak_detected`) pass vacuously in another -- no prompt means no streaming means no leak. **Gaps in the two features that landed during review**, small enough to close here: the `session.py` wiring that fills `update_available` / `update_command` (removing it left the suite green), `_upgrade_command_works` (a fixture stubbed it to `True` everywhere), an `episodeSummary` case with an empty test body, and `clipToWidth` / `hasMeaningfulReasoning`, which had no coverage at all. The larger items from that audit are tracked in #229. **Upstream review attributions dropped from ui-tui comments.** 36 mentions of another project's review thread (#19835 as "Copilot round-N review"), plus #18994, #19194 and #14045 -- none resolve to anything here, and AGENTS.md section 1.1 rules out comments pointing at information only visible elsewhere. The rationale each carried is kept; four `describe()` titles lose a trailing `(#18994)`. Three went further and documented a python counterpart that does not exist: `platform.ts` credited key-spelling normalization to `raven_cli/voice.py` (no such module -- `raven/tui_rpc/methods/_stubs.py` answers `voice.toggle` with "voice not supported in Raven v0.1"), and `useConfigSync.ts` plus its test credited an `interrupt` default to `raven_cli/config.py` and `tui_gateway/server.py::_load_busy_input_mode` (`busy_input_mode` appears nowhere in python; neither module exists). **One flaky test fixed, caught by this PR's own CI run.** `test_stdio_no_executor_does_not_raise` set `command="true"` and let `connect_mcp_servers` really spawn it, so the result depended on how a live child process raced the MCP handshake -- it failed once on CI in the full-suite run while passing 30 out of 30 standalone runs locally. It also asserted only that whatever came back was not a `SandboxInitError`. The transport now raises instead of running (the idiom its sibling test already uses) and the test asserts the stdio branch was reached. Review follow-ups, folded in: the hard-exit test now skips rather than asserting no lancedb thread exists process-wide, `tests/conftest.py` no longer credits the pytest segfault to a lancedb thread the suite never starts (raven imports lancedb nowhere), and the status-bar patterns the two TUI e2e tests wait on live in one module instead of a hand-mirrored copy per file. Every new or changed assertion was checked by mutating the code under test and confirming the assertion fails. ## Type - [x] Others Test files, test configuration, and comments. The only non-test files touched are ui-tui comments and four `describe()` titles -- no behaviour change. ## Verification ``` uv run --all-extras pytest -q # 4654 passed, 25 deselected cd ui-tui && npx vitest run # 936 passed, no skips uv run --all-extras pytest tests/tui/autotest -m e2e -q # passes locally with tui-use installed and ui-tui built bash .claude/scripts/preflight_ci.sh # ALL PREFLIGHT CHECKS PASSED (including the TUI lane) ``` Mutation checks, each confirming the new assertion fails when the code under test is broken: the `session.py` nudge wiring removed; `clipToWidth` replaced with identity; `hasMeaningfulReasoning` forced to `true`; a never-sent prompt added to the multi-turn session assertion. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes No behaviour changes: the non-test diff is comments only. The renames touch git history for 19 test files; #208 is the only open PR affected and has been notified. Two known items are deliberately left alone: `test_dogfood_whitelist.py` is flaky when the whole file runs (tracked in #228, unchanged here), and the `lancedb_finalization_hazard` gate in `commands.py` never fires today -- its test docstring says so explicitly, so the coverage does not read as an argument for keeping it. ## Related Issues Closes #223 --------- Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
198 lines
6.9 KiB
Python
198 lines
6.9 KiB
Python
"""Unit tests for RoutineLearner.learn_with_decay (MS2.5)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from datetime import datetime, timedelta
|
|
|
|
from raven.proactive_engine.sentinel.predictor.routine_learner import (
|
|
RoutineLearner,
|
|
_decay_factor,
|
|
)
|
|
|
|
_NOW = datetime(2026, 5, 8, 12, 0)
|
|
|
|
|
|
def _history_line(ts: datetime, content: str) -> str:
|
|
return f"[{ts.strftime('%Y-%m-%d %H:%M')}] {content}"
|
|
|
|
|
|
def _history_md(*lines: str) -> str:
|
|
"""Join HISTORY.md-style entries with the same blank-line separator
|
|
MemoryStore.append_history uses."""
|
|
return "\n\n".join(lines)
|
|
|
|
|
|
def test_decay_factor_today_is_one():
|
|
assert _decay_factor(now=_NOW, ts=_NOW, half_life_days=14) == 1.0
|
|
|
|
|
|
def test_decay_factor_one_half_life_is_half():
|
|
ts = _NOW - timedelta(days=14)
|
|
factor = _decay_factor(now=_NOW, ts=ts, half_life_days=14)
|
|
assert abs(factor - 0.5) < 1e-9
|
|
|
|
|
|
def test_decay_factor_two_half_lives_is_quarter():
|
|
ts = _NOW - timedelta(days=28)
|
|
factor = _decay_factor(now=_NOW, ts=ts, half_life_days=14)
|
|
assert abs(factor - 0.25) < 1e-9
|
|
|
|
|
|
def test_decay_factor_future_clamps_to_one():
|
|
"""Defensive: parsing artifact shouldn't make weights blow up."""
|
|
ts = _NOW + timedelta(days=5)
|
|
assert _decay_factor(now=_NOW, ts=ts, half_life_days=14) == 1.0
|
|
|
|
|
|
def test_decay_factor_zero_half_life_is_one():
|
|
"""Defensive against divide-by-zero / config typo."""
|
|
ts = _NOW - timedelta(days=10)
|
|
assert _decay_factor(now=_NOW, ts=ts, half_life_days=0) == 1.0
|
|
|
|
|
|
def test_learn_with_decay_recent_routine_outweighs_stale_one():
|
|
"""Two routines with same occurrence_count: recent fresh > stale old."""
|
|
learner = RoutineLearner(
|
|
min_occurrences=3,
|
|
learning_window_days=60,
|
|
now_fn=lambda: _NOW,
|
|
)
|
|
|
|
# Build entries for two clearly-distinct bins:
|
|
fresh_entries = []
|
|
stale_entries = []
|
|
# 4 Tuesdays at 9am, recent (within last 28 days)
|
|
for week in range(4):
|
|
d = _NOW - timedelta(days=_NOW.weekday() + 7 * week - 1) # Tuesday
|
|
d = d.replace(hour=9, minute=0)
|
|
fresh_entries.append(_history_line(d, f"meeting standup week {week}"))
|
|
# 4 Fridays at 14:00, OLD (35-56 days ago)
|
|
for week in range(4):
|
|
d = _NOW - timedelta(days=35 + 7 * week)
|
|
d = d.replace(hour=14, minute=0)
|
|
stale_entries.append(_history_line(d, f"old report week {week}"))
|
|
|
|
md = _history_md(*(fresh_entries + stale_entries))
|
|
routines = learner.learn_with_decay(md)
|
|
|
|
# Both bins should produce a candidate
|
|
assert len(routines) >= 2
|
|
# Sort guarantees fresh-and-frequent first
|
|
fresh_r = next(r for r in routines if r.day_of_week == 1) # Tuesday
|
|
stale_r = next(r for r in routines if r.day_of_week == 4) # Friday
|
|
assert fresh_r.weight > stale_r.weight
|
|
# Returned order has fresh first
|
|
fresh_idx = routines.index(fresh_r)
|
|
stale_idx = routines.index(stale_r)
|
|
assert fresh_idx < stale_idx
|
|
|
|
|
|
def test_learn_with_decay_skips_below_min_occurrences():
|
|
learner = RoutineLearner(
|
|
min_occurrences=3,
|
|
learning_window_days=60,
|
|
now_fn=lambda: _NOW,
|
|
)
|
|
# Only 2 occurrences in same bin — below threshold
|
|
md = _history_md(
|
|
_history_line(_NOW.replace(hour=9), "meeting"),
|
|
_history_line(_NOW - timedelta(days=7, hours=3), "meeting"),
|
|
)
|
|
routines = learner.learn_with_decay(md)
|
|
assert routines == []
|
|
|
|
|
|
def test_learn_with_decay_handles_empty_history():
|
|
learner = RoutineLearner(now_fn=lambda: _NOW)
|
|
assert learner.learn_with_decay("") == []
|
|
assert learner.learn_with_decay("no parseable lines here") == []
|
|
|
|
|
|
def test_learn_with_decay_outside_learning_window_drops_entries():
|
|
learner = RoutineLearner(
|
|
min_occurrences=3,
|
|
learning_window_days=14, # short window
|
|
now_fn=lambda: _NOW,
|
|
)
|
|
# 4 entries 30 days ago — outside window
|
|
old = [_history_line(_NOW - timedelta(days=30 + i), f"old activity {i}") for i in range(4)]
|
|
routines = learner.learn_with_decay(_history_md(*old))
|
|
assert routines == []
|
|
|
|
|
|
def test_tfidf_keywords_downweights_common_terms():
|
|
"""Words appearing in many bins (high DF) get TF-IDF score 0; bin-
|
|
specific terms surface."""
|
|
learner = RoutineLearner(
|
|
min_occurrences=3,
|
|
learning_window_days=60,
|
|
now_fn=lambda: _NOW,
|
|
)
|
|
# Three different bins, each with 'work' and one unique term.
|
|
# Bin 1 (Mon 9): work + standup
|
|
# Bin 2 (Wed 14): work + review
|
|
# Bin 3 (Fri 16): work + planning
|
|
entries = []
|
|
for week in range(4):
|
|
# Monday 9am — standup
|
|
mon = _NOW - timedelta(days=_NOW.weekday() + 7 * week)
|
|
mon = mon.replace(hour=9, minute=0)
|
|
entries.append(_history_line(mon, "work standup meeting"))
|
|
# Wednesday 2pm — review
|
|
wed = _NOW - timedelta(days=(_NOW.weekday() - 2) % 7 + 7 * week)
|
|
wed = wed.replace(hour=14, minute=0)
|
|
entries.append(_history_line(wed, "work review session"))
|
|
# Friday 4pm — planning
|
|
fri = _NOW - timedelta(days=(_NOW.weekday() - 4) % 7 + 7 * week)
|
|
fri = fri.replace(hour=16, minute=0)
|
|
entries.append(_history_line(fri, "work planning sync"))
|
|
|
|
routines = learner.learn_with_decay(_history_md(*entries))
|
|
# Each bin's keywords should NOT be dominated by 'work' (which appears
|
|
# in all 3 bins → DF=3, log(3/3)=0 score). It should be bin-specific.
|
|
by_dow = {r.day_of_week: r for r in routines}
|
|
# 'work' should be dropped or at least not first in any bin's keywords
|
|
for r in by_dow.values():
|
|
if r.keywords:
|
|
assert r.keywords[0] != "work", f"TF-IDF should downweight common 'work' term, got {r.keywords}"
|
|
|
|
|
|
def test_learn_with_decay_sets_weight_field():
|
|
learner = RoutineLearner(
|
|
min_occurrences=3,
|
|
learning_window_days=60,
|
|
now_fn=lambda: _NOW,
|
|
)
|
|
# 3 fresh Tuesday 9am entries
|
|
entries = []
|
|
for week in range(3):
|
|
d = _NOW - timedelta(days=_NOW.weekday() + 7 * week - 1)
|
|
d = d.replace(hour=9, minute=0)
|
|
entries.append(_history_line(d, "morning meeting"))
|
|
routines = learner.learn_with_decay(_history_md(*entries))
|
|
assert len(routines) == 1
|
|
r = routines[0]
|
|
assert r.weight > 0
|
|
# 3 entries within ~21 days, half-life 14: roughly 1 + 0.5 + 0.25 = ~1.75
|
|
# (loose bound — just check it's plausibly in range)
|
|
assert 0.5 < r.weight < 3.0
|
|
|
|
|
|
def test_legacy_learn_still_works():
|
|
"""`learn` (without decay) shouldn't have regressed — existing
|
|
ContextAssembler paths still call it."""
|
|
learner = RoutineLearner(
|
|
min_occurrences=3,
|
|
learning_window_days=60,
|
|
now_fn=lambda: _NOW,
|
|
)
|
|
entries = []
|
|
for week in range(3):
|
|
d = _NOW - timedelta(days=_NOW.weekday() + 7 * week - 1)
|
|
d = d.replace(hour=9, minute=0)
|
|
entries.append(_history_line(d, "morning meeting"))
|
|
routines = learner.learn(_history_md(*entries))
|
|
assert len(routines) == 1
|
|
# weight defaults to 0.0 in legacy path (only set by learn_with_decay)
|
|
assert routines[0].weight == 0.0
|