Raven-1/tests/test_guards.py
Handsome-wzw 7b45527b5b fix(tools): an array sent as its own JSON is still an array
## Summary

Reported as "deliver_files is broken". It is not: **the tool never ran.**

Every `deliver_files` call on one install was refused before dispatch -- five in a row in a single session -- and nothing was ever delivered:

```
Error: Invalid parameters for tool 'deliver_files': files should be array
```

What the model sent was the array, spelled as a string:

```json
{"files": "[{\"path\": \"...poster.png\", \"title\": \"...\"}]", "message": "..."}
```

`cast_params` already forgives that spelling for every scalar type -- `"3"` for an integer, `"true"` for a boolean -- and did not for arrays or objects, which are exactly the types a model is most likely to stringify.

## Type

- [x] Fix
- [ ] Feature

## The edge of the forgiving is the design

When the value is a string, try `json.loads`, and take the result **only if it is the type the schema asked for**. A string that is not JSON, or is JSON of the wrong shape, is returned unchanged so the validator still names the real problem -- the point is to widen what is accepted, not to swallow a type error.

**Nothing here can raise.** Turning "your argument has the wrong type" into a traceback would be worse than the refusal it replaces, so the parse is wrapped and failure falls through.

**A `type: "string"` field is never parsed.** JSON is ordinary text in a message body or a file's content, and replacing it would hand the tool something it never asked for. Two independent things prevent it today (the string branch returns before the parse is reached, and the parse only wins on a type match), which is why the case pinning it is a tripwire rather than a load-bearing assertion -- said so in its own docstring rather than left to look like coverage.

## Not just deliver_files

Six built-in tools declare an array or object parameter and were all exposed:

| tool | parameter |
|---|---|
| `deliver_files` | `files` |
| `ask_user` | `questions` |
| `image_generate` | `images` |
| `message` | `media` |
| `create_playbook` | `skills` |
| `video_generate` | `params` |

The same failure was already seen from the other side: `acp_client/acp_agent.py` records `update_task_state` and `operations should be array` from a real dispatch.

## Verification

```
make lint-python    ruff check + ruff format --check, 1850 files
make lint-imports   9 contracts kept, 0 broken
pytest tests/       19,699 passed, 7 failed (all pre-existing, see below)
```

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots -- no user-facing string

**End to end, through the real tool**, with the exact payload that failed five times in production:

```
files sent as:            str -> [{"path": ".../poster.png", "title": "..."}]
returned:                 Delivered 1 file(s): poster.png (72 B).
still "should be array":  False
registered in the store:  True
```

Twelve cases, five mutations, each killing at least one:

| mutation | red |
|---|---|
| drop the whole branch (the old behaviour) | 3 |
| raise instead of falling through on a parse failure | 5 |
| accept the parse whatever type it yielded | 1 |
| do not re-cast the parsed value, so items are not cast | 1 |
| widen the tuple to include `string` | **survived -- unreachable** |

The last is worth stating rather than hiding: that mutation cannot change behaviour, because the `type == "string"` branch returns earlier and the type check would refuse the result anyway. I also tried moving the new block above that branch, alone and combined with widening the tuple, and neither reddens anything either. The invariant holds by construction; the case that documents it says so.

**Pre-existing failures, checked against the base.** 7 failed in the full run, all in `tests/test_security_network.py` (`test_a_redirect_to_a_dangerous_address_is_refused` and siblings). Running that file alone on this same tree gives 4 -- the count is order-dependent -- and the change touches only `raven/agent/tools/params.py` plus a new test file, neither of which `raven/security/network.py` imports. The failures are the box's resolver: it returns poisoned answers roughly half the time, which I measured separately (`en.wikipedia.org` came back as `2001::1` on 3 of 6 lookups). Not introduced here, and worth someone's attention on its own.

## Risk

- [x] Security impact considered -- widens what reaches a tool's own validation, not what bypasses it. A parsed value is still validated against the same schema, and a parse that yields the wrong type is refused exactly as before
- [x] Backward compatibility considered -- a call that worked before takes the identical path; the new branch is reached only by a `str` where an array or object is declared, which previously always failed
- [x] Rollback path is clear -- single-commit revert

## Related Issues

N/A

## Swept

`mr-review-patterns` pre-submit sweep, over `git diff origin/main...HEAD`.

- **1 (a write location moved)** -- nothing moved; a branch was added to an existing dispatch on `target_type`. Grepped every caller of `cast_params`: the registry is the only one, and it runs `validate_params` right after, which is the reader this affects.
- **2 (a value with several entry points)** -- the cast is the single funnel every tool argument passes through, which is why the fix belongs here and not in `deliver_files`.
- **3 (a guard)** -- checked both directions: what it wrongly lets through (JSON of the wrong type, non-JSON text -- both refused, with cases) and what it wrongly refuses (a real list, unaffected, with a case).
- **4 (tests)** -- five mutations above, one honestly reported as unreachable. Failing-ID sets diffed against the base rather than counts.
- **6 (stale text)** -- the module docstring enumerated what is coerced ("a string that should be an integer"); it now names arrays and objects too. Grepped the tracked tree for other statements of the rule: none.
- **9 (error handling)** -- the new path adds one way to end and it is caught: `json.loads` raises `ValueError` on bad text and `TypeError` on a non-string, and both fall through to the untouched value.
- **11** -- `git ls-files | grep -c node_modules` is 0; the commit-message gate was run, not eyeballed.

Not applicable, with reasons: **3b, 3c, 3d, 3e** (no tool schema produced, no state moved, one lane, no assertion lost its symbol); **4b, 4c, 4d** (no platform branch, no shell, no rendered number); **5, 5b** (no dependency; `json` is stdlib and already imported across this package); **6b, 6c** (no backup, no generated artifact); **7, 7b** (nothing on disk); **8** (no asset, no secret); **10** (no prompt text).

## Deliberately not fixed

The error text still ends with `[Analyze the error above and try a different approach.]`, and the model plainly did not -- it sent the same shape five times. A message naming the expected spelling (`files must be a JSON array, not a string containing one`) would give it something to act on. That is a change to the registry's error surface, affects every tool, and deserves its own argument.

## Moved from !594

Opened on a compliant source branch. AGENTS.md 2.1 asks for a 3-5 word snake_case description and the previous branch had seven; GitLab cannot change an existing merge request's source branch, so the change moves rather than the name.

Same commits, rebased onto `c2a9768f`. Both code findings from the review on !594 are already in them -- `RecursionError` caught with its regression case, and the implementation comment reduced to the two non-obvious reasons. Nothing else changed, so that review carries.
2026-09-09 15:42:36 +00:00

289 lines
9.7 KiB
Python

"""Code-layer guards.
Each guard is a pure function isolated from MemoryStore. These tests
pin the contract so prompt-side regressions or stricter rules don't
silently change behavior at 30-day scale.
"""
from __future__ import annotations
from raven.memory_engine.consolidate.consolidator import (
_PROCESS_TAGS,
_VALID_CONFIDENCE,
_drop_bullets_without_src,
_foresight_token_set,
_format_foresight_bullet,
_is_process_only_episode,
_is_semantic_duplicate_foresight,
_normalize_confidence,
)
# ---------- _is_process_only_episode --------------------------------
def test_process_only_question_tag_alone_rejected():
line = "[2026-05-12 14:00] which framework for new dashboard #question"
assert _is_process_only_episode(line) is True
def test_process_only_habit_tag_alone_rejected():
line = "[2026-05-12 07:30] morning routine recap #habit"
assert _is_process_only_episode(line) is True
def test_process_only_answer_tag_alone_rejected():
line = "[2026-05-12 09:00] explained the concept of CAP #answer"
assert _is_process_only_episode(line) is True
def test_question_with_content_tag_kept():
line = "[2026-05-12 14:00] which framework #question #project-dashboard"
assert _is_process_only_episode(line) is False
def test_pure_content_tag_kept():
line = "[2026-05-12 14:00] merged PR #142 #project-clawtrack #pr"
assert _is_process_only_episode(line) is False
def test_episode_without_tags_kept():
line = "[2026-05-12 14:00] free-form note without any tags"
assert _is_process_only_episode(line) is False
def test_unparseable_line_kept():
# No timestamp prefix — parse_episode_line returns None.
# Guard returns False so we don't drop unrelated freeform input.
assert _is_process_only_episode("just a stray string with #habit") is False
def test_empty_input_kept():
assert _is_process_only_episode("") is False
def test_uppercase_tag_not_recognized_as_tag():
# ``_TAG_RE`` only matches lowercase kebab-case slugs (system spec).
# ``#QUESTION`` thus produces zero parsed tags → guard treats the line
# as untagged free-form input and returns False (lets it through).
line = "[2026-05-12 14:00] uppercase tag test #QUESTION"
assert _is_process_only_episode(line) is False
def test_multiple_process_tags_only_still_rejected():
line = "[2026-05-12 14:00] mixed process tags #question #habit"
assert _is_process_only_episode(line) is True
# ---------- _normalize_confidence -----------------------------------
def test_confidence_low_valid():
assert _normalize_confidence("low") == "low"
def test_confidence_medium_valid():
assert _normalize_confidence("medium") == "medium"
def test_confidence_high_valid():
assert _normalize_confidence("high") == "high"
def test_confidence_case_insensitive():
assert _normalize_confidence("LOW") == "low"
assert _normalize_confidence("Medium") == "medium"
def test_confidence_strips_whitespace():
assert _normalize_confidence(" high ") == "high"
def test_confidence_invalid_strong_rendered_unknown():
assert _normalize_confidence("strong") == "?"
def test_confidence_invalid_likely_rendered_unknown():
assert _normalize_confidence("likely") == "?"
def test_confidence_empty_rendered_unknown():
assert _normalize_confidence("") == "?"
def test_format_foresight_bullet_normalizes_bad_confidence():
bullet = _format_foresight_bullet(
{
"prediction": "User will deploy on Friday",
"window": "1-3 days",
"confidence": "strong",
"src_ts": "2026-05-10 14:00",
},
generation_ts="2026-05-10 21:00",
)
assert "confidence: ?" in bullet
assert "strong" not in bullet
def test_format_foresight_bullet_keeps_valid_confidence():
bullet = _format_foresight_bullet(
{
"prediction": "User will deploy on Friday",
"window": "1-3 days",
"confidence": "medium",
"src_ts": "2026-05-10 14:00",
},
generation_ts="2026-05-10 21:00",
)
assert "confidence: medium" in bullet
# ---------- _foresight_token_set ------------------------------------
def test_token_set_lowercases_and_drops_short_words():
tokens = _foresight_token_set("User runs every Saturday morning")
# "user" is stopword; "runs" is 4 chars (included).
assert "runs" in tokens
assert "saturday" in tokens
assert "morning" in tokens
assert "user" not in tokens
def test_token_set_drops_framing_stopwords_and_s_stems():
tokens = _foresight_token_set("User will continue daily medication reminders")
assert "user" not in tokens
assert "will" not in tokens
assert "continue" not in tokens
assert "daily" in tokens
assert "medication" in tokens
# ``reminders`` gets s-stemmed to ``reminder`` for plural/singular
# collapse (see ``_stem_trailing_s``).
assert "reminder" in tokens
assert "reminders" not in tokens
def test_token_set_empty_for_pure_stopword_text():
tokens = _foresight_token_set("user will continue tomorrow")
assert tokens == frozenset()
# ---------- _is_semantic_duplicate_foresight ------------------------
def test_semantic_dup_reworded_saturday_run():
new = "User runs every Saturday morning (recurring habit pattern)"
existing = ["User runs every Saturday morning (recurring habit)"]
assert _is_semantic_duplicate_foresight(new, existing) is True
def test_semantic_dup_caregiver_medication_cluster():
# Real longrun pattern (caregiver-01, day 20 vs day 22): same claim
# reworded slightly. With s-stemming and Jaccard ≥ 0.6 the dedup
# collapses them.
new = "User will set daily medication reminders for mom (amlodipine, donepezil, metoprolol) at similar times"
existing = ["Daily medication reminders for mom (amlodipine, donepezil, metoprolol) - recurring care routine"]
assert _is_semantic_duplicate_foresight(new, existing) is True
def test_semantic_dup_misses_heavy_morphology():
# Documented limitation: ``remind`` vs ``reminders`` differ even
# after s-stem (remind has no trailing s; reminders → reminder).
# Without a real stemmer we accept this false negative; the data
# showed near-identical wording dominates the dup cluster anyway.
new = "User will remind mom about daily medications"
existing = ["Daily medication reminders for mom"]
assert _is_semantic_duplicate_foresight(new, existing) is False
def test_semantic_dup_unrelated_predictions_distinct():
new = "User will release clawtrack v1.0 after final testing"
existing = [
"User runs every Saturday morning (recurring habit)",
"User will attend NeurIPS workshop submission deadline next week",
]
assert _is_semantic_duplicate_foresight(new, existing) is False
def test_semantic_dup_empty_existing_returns_false():
assert _is_semantic_duplicate_foresight("anything", []) is False
def test_semantic_dup_empty_new_returns_false():
assert _is_semantic_duplicate_foresight("", ["something"]) is False
def test_semantic_dup_blocks_within_batch():
# Simulate caller appending duplicates one at a time to existing.
existing: list[str] = []
first = "User attends weekly boss meetings on Mondays"
second = "User attends weekly Monday boss meetings"
assert _is_semantic_duplicate_foresight(first, existing) is False
existing.append(first)
assert _is_semantic_duplicate_foresight(second, existing) is True
# ---------- _drop_bullets_without_src --------------------------------
def test_drop_keeps_bullet_with_src_link():
body = "- **Status**: v1.0 released [src: episodes.md @ 2026-05-15 19:30]"
cleaned, n_dropped = _drop_bullets_without_src(body)
assert cleaned == body
assert n_dropped == 0
def test_drop_removes_bullet_missing_src_link():
body = "- **Status**: v1.0 released [src: episodes.md @ 2026-05-15 19:30]\n- Senior developer working remotely"
cleaned, n_dropped = _drop_bullets_without_src(body)
assert "- Senior developer" not in cleaned
assert "v1.0 released" in cleaned
assert n_dropped == 1
def test_drop_preserves_non_bullet_lines():
body = "### clawtrack\n- **Status**: v1.0 released [src: episodes.md @ 2026-05-15 19:30]\n\nSome prose paragraph.\n"
cleaned, n_dropped = _drop_bullets_without_src(body)
assert "### clawtrack" in cleaned
assert "Some prose paragraph." in cleaned
assert n_dropped == 0
def test_drop_handles_indented_bullets():
body = " - **Status**: ok [src: episodes.md @ 2026-05-15 19:30]\n - missing-src bullet"
cleaned, n_dropped = _drop_bullets_without_src(body)
assert "missing-src bullet" not in cleaned
assert "ok" in cleaned
assert n_dropped == 1
def test_drop_empty_body_noop():
cleaned, n_dropped = _drop_bullets_without_src("")
assert cleaned == ""
assert n_dropped == 0
def test_drop_rejects_src_with_wrong_format():
# Wrong file name (not episodes.md), wrong bracket style, etc.
body = (
"- bullet a (src: episodes.md @ 2026-05-15 19:30)\n" # parens — wrong
"- bullet b [src: notes.md @ 2026-05-15 19:30]\n" # wrong file
"- bullet c [src: episodes.md @ 2026-05-15 19:30]\n" # correct
)
cleaned, n_dropped = _drop_bullets_without_src(body)
assert "bullet a" not in cleaned
assert "bullet b" not in cleaned
assert "bullet c" in cleaned
assert n_dropped == 2
# ---------- module-level constants snapshot --------------------------
def test_process_tags_constant():
# Stored without leading '#' since ``_TAG_RE`` strips it on parse.
assert _PROCESS_TAGS == frozenset({"question", "habit", "answer"})
def test_valid_confidence_constant():
assert _VALID_CONFIDENCE == frozenset({"low", "medium", "high"})