Raven-1/tests/test_trajectory_bugreport.py
江国庆/Forrest 2a139f8fbf
fix(*): show bug report values in full and give the wizard room (#952)
## Summary

Follow-ups to the web bug report wizard from #948, after trying it.

- **Decisions page lists the values in full, and only them.** The left
column showed each value masked (`544e***5100`) with its category,
count, decision and links under it. The reader decides on the value
itself, so the list now holds the values and nothing else, each in full
and wrapping rather than cut. `bugreport.status` carries a new `value`
field beside `masked_sample` for each item while the report waits on the
decisions; no other phase carries one. A confirmed item (a private key)
keeps no value of its own, since it is replaced whatever the reader
says, so it shows its replacement. The picked value's category, count,
decision and links, and a confirmed item's acknowledgement, move to the
right column's header; a redacted value is struck through in the list.
- **More room.** The wizard's dialog grows from min(640px, 94vw) x
min(560px, 88vh) to min(960px, 94vw) x min(720px, 90vh), close to the
settings dialog, and the decisions page's two columns fill it instead of
a fixed 360px.
- **Clipped fields.** The severity select and the file name and reporter
fields inherited the page-wide field rule's fixed 32px height and
monospace face, which cut off part of their text. The wizard's sheet now
sizes and sets them.
- **Completeness off the page.** The upload page no longer shows the
completeness verdict. It stays in both JSON files of the report
(`record.json` and the package's `bugreport.json`), for command-line and
web reports alike, and a new test pins that for both paths.

Why some web reports come out `unreplayable` is not addressed here; it
is tracked separately.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- `uv run pytest tests/ -q` (full suite): 57 failed / 31566 passed / 177
skipped. The 57 are the same set, one for one, as on the tree #948
merged, where each also fails on `main` (environment-specific on this
machine, pre-existing); none is new.
- `uv run pytest tests/test_rpc_*.py tests/test_trajectory_*.py
tests/test_cli_trajectory_commands.py -q`: 4423 passed.
- `uv run pytest -m integration
tests/integration/test_web_bugreport_e2e.py
tests/integration/test_web_traj_export_e2e.py
tests/integration/test_bug_report_upload_e2e.py -q`: 10 passed.
- `uv run ruff check`, `uv run ruff format --check`, `uv run ty check`
on the changed Python: pass.
- ui-web: `npm run gen:check` (239 methods), `npm run lint`, `npm run
type-check`, `npx vitest run` (246 files / 3869 passed), `npm run build`
and `python3 ui-web/build.py` (both boot snapshots match),
`check-page.mjs`, `check-css.mjs`, `check-class-namespace.mjs`: pass.
- ui-tui: `npm run lint:rpc`, `npm run lint:i18n`: pass.
- `make check-source-language check-large-files check-commits`: pass.
- Headless Chromium on the built page (offline fixtures), at 1440 and
390 px: the dialog is 960 x 720 at 1440 px and 367 x 684 at 390 px; at
both widths the select and both text fields are 38px tall in the body
face at 14px, the decisions list shows only the full values, the right
header shows the facts and the acknowledgement, and both columns stay
inside the dialog.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- `bugreport.status` now sends the values being decided in full while a
report is in the decisions phase. They already reached the page through
`bugreport.item`'s contexts; the status is not polled in that phase,
travels only between the local gateway and its page, and carries no
value in any other phase. A private key is never sent, only its
replacement.
- `BugReportItem` gains a required `value` field. The only client is
this page, regenerated in the same change.
- Rollback: revert the squash commit.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-10-15 12:06:34 +08:00

1912 lines
80 KiB
Python

"""Tests for bug report records, packaging, and export sanitization
(`raven.trajectory.bugreport`, `raven.trajectory.sanitize`, `store.member_traces`)."""
from __future__ import annotations
import json
import os
import re
import tarfile
from pathlib import Path
import pytest
from raven.trajectory import bugreport as breport
from raven.trajectory import review as treview
from raven.trajectory import sanitize as tsan
from raven.trajectory import store as tstore
from raven.trajectory.redact import KnownSecret, RedactionReport, ResidualFinding
@pytest.fixture
def state(tmp_path, monkeypatch):
state = tmp_path / "traces"
monkeypatch.setenv("RAVEN_TRACING_DIR", str(state))
return state
@pytest.fixture
def workspace(tmp_path):
ws = tmp_path / "ws"
ws.mkdir()
return ws
@pytest.fixture(autouse=True)
def _isolated_secrets(monkeypatch):
"""Keep the real config/env secrets of the test machine out of every run."""
monkeypatch.setattr(breport, "collect_known_secrets", lambda _p: ([], True))
def _write_log(path, spans):
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("".join(json.dumps(s) + "\n" for s in spans), encoding="utf-8")
def _span(trace_id, *, attempt_id=None, session_key=None, name="session.turn", attrs=None, span_id=None):
attributes = {"session.key": session_key}
if attempt_id is not None:
attributes["attempt.id"] = attempt_id
if attrs:
attributes.update(attrs)
return {
"schemaVersion": "audit.span.v1",
"traceId": trace_id,
"spanId": span_id or f"span-{trace_id}-{name}",
"name": name,
"startTime": "2026-08-20T10:00:00+00:00",
"endTime": "2026-08-20T10:00:01+00:00",
"attributes": attributes,
}
# Assembled at runtime: the detect-private-key pre-commit hook scans source
# bytes for the marker substring and cannot tell this fake fixture apart.
_PEM = "-----BEGIN PRIVATE " + "KEY-----\nMIIabcdef\n-----END PRIVATE " + "KEY-----"
_ENTROPY_TOKEN = "aB3xK9mQ7pL2vR8sT4wZ6yN1"
def _log_simple(state, attrs=None):
_write_log(state / "logs" / "audit-spans.log", [_span("trace-1", session_key="cli:a", attrs=attrs)])
def _prepared(state, workspace, attrs=None, **prepare_kw):
_log_simple(state, attrs)
return breport.prepare_trajectory("trace-1", workspace=workspace, state_dir=state, **prepare_kw)
def _full_flow(state, workspace, description="the agent replied wrongly", attrs=None, **fields):
prep = _prepared(state, workspace, attrs)
# keep-all by default: incidental findings (e.g. a long pytest tmp-dir
# segment) must not force every flow test to script the review screen.
fields.setdefault("decide", _decide_all())
prep = breport.freeze_export(prep, description=description, **fields)
return breport.confirm_and_package(prep, state_dir=state)
def _decide_all(suspected_action=treview.ACTION_KEPT):
"""A scripted decide callback: acknowledge confirmed items, one action for the rest."""
def _decide(items, _reasons):
return [
treview.ReviewDecision(
item.id,
treview.ACTION_ACKNOWLEDGED if item.kind == treview.KIND_CONFIRMED else suspected_action,
)
for item in items
]
return _decide
# ── path parser / sanitizer ────────────────────────────────────────────
@pytest.mark.parametrize(
"text",
[
"see /tmp/customer-dump.json here",
"/customer-data",
"macos /private/var/folders/ab/T/x.log tmp",
"mount /mnt/company/project/source.py",
"opt /opt/internal/model/config.yaml",
"unicode /\u5ba2\u6237\u8d44\u6599/\u9879\u76ee path",
'quoted "/Project Files/data.json" path',
"user dir /Users/alice/project/config.json",
"win C:\\Users\\alice\\f.txt path",
"unc \\\\srv\\share\\x path",
"file url file:///etc/passwd leaks",
'{"u":"https://api.x.com/v1","p":"/etc/raven/license.key"}',
"log https://x.com/v1,/var/db/raven/secrets.sqlite end",
"[docs](https://x.com/d)/opt/secret/key.pem",
"note (see below)/srv/raven/keys.json end",
"[ref]/opt/secret/key.pem",
'{"a":1}/opt/secret/key.pem',
"GET https://x.com/a?f=/var/folders/9z/T/tmp8ab/session_secrets.json",
"see https://x.com/a#/etc/shadow now",
],
ids=[
"tmp",
"single",
"private-var",
"mnt",
"opt",
"unicode",
"quoted-space",
"users",
"drive",
"unc",
"file-url",
"json-after-url",
"comma-after-url",
"after-markdown-link",
"after-parens",
"after-bracket",
"after-brace",
"in-url-query",
"in-url-fragment",
],
)
def test_parser_hits_absolute_paths(text):
sanitized = tsan.sanitize_text(text)
assert tsan.PATH_PLACEHOLDER in sanitized
assert not tsan.find_absolute_paths(sanitized)
@pytest.mark.parametrize(
"text",
[
"and/or 1/2 a/b words",
"https://host/path/to/x stays",
"see artifacts/001-trace-1-cli-a.json relative",
"[REDACTED:path]/name.json placeholder tail",
"see https://host:8080/api/v1 now",
"see https://host/a?x=1&y=2 now",
],
ids=["natural", "url", "bundle-relative", "placeholder-tail", "url-port", "url-query"],
)
def test_parser_leaves_non_paths_alone(text):
assert tsan.sanitize_text(text) == text
def test_parser_stops_at_the_requested_number_of_spans():
text = "see /a and /b and /c"
assert len(tsan.find_absolute_paths(text)) == 3
assert tsan.find_absolute_paths(text, limit=2) == tsan.find_absolute_paths(text)[:2]
assert tsan.find_absolute_paths(text, limit=0) == []
@pytest.mark.parametrize(
("value", "text_level_hits"),
[
("problem: \\n- step one \\n- step two", True),
("problem:\n- step one", False),
("matches \\d+ digits", True),
("\\d+ at start", True),
("a\\nb", False),
],
ids=["literal-newline", "real-newline", "regex", "leading-backslash", "unbounded-backslash"],
)
def test_json_scan_ignores_escape_artifacts(value, text_level_hits):
document = {"problem": {"steps": value}}
assert tsan.find_absolute_paths_in_json(document) == []
assert bool(tsan.find_absolute_paths(json.dumps(document))) is text_level_hits
assert tsan.find_absolute_paths_in_json(json.loads(json.dumps(document))) == []
@pytest.mark.parametrize(
"path",
["/etc/raven/license.key", "C:\\Users\\alice\\key.txt", "\\\\server\\share\\key.txt"],
ids=["posix", "drive", "unc"],
)
def test_json_scan_reports_the_field_path(path):
value = {
"problem": {"steps": f"see {path} here"},
"contents": [{"path": "bugreport.json"}, {"path": path}],
}
hits = tsan.find_absolute_paths_in_json(value)
assert ("problem.steps", path) in hits
assert ("contents[1].path", path) in hits
assert all(where != "contents[0].path" for where, _ in hits)
def test_json_scan_finds_a_known_root_glued_to_text():
value = "prefixC:\\Users\\alice\\file.txt"
roots = ["C:\\Users\\alice"]
assert tsan.find_absolute_paths(value) == []
assert tsan.find_absolute_paths_in_json({"problem": {"steps": value}}, roots) == [
("problem.steps", "C:\\Users\\alice")
]
def test_json_scan_checks_object_keys():
hits = tsan.find_absolute_paths_in_json({"problem": {"/etc/shadow": "x"}})
assert hits == [("problem./etc/shadow (key)", "/etc/shadow")]
def test_json_scan_walks_nesting_the_decoder_accepts():
"""The decoder admits nesting close to the recursion limit; the walk must not overflow on it."""
depth = 900
document = json.loads("[" * depth + '"/etc/shadow"' + "]" * depth)
hits = tsan.find_absolute_paths_in_json({"problem": {"steps": document}})
assert hits == [("problem.steps" + "[0]" * depth, "/etc/shadow")]
assert tsan.find_absolute_paths_in_json({"problem": {"steps": document}}, limit=0) == []
def test_json_scan_keeps_document_order():
value = {"a": {"k /x": "/one"}, "b": ["/two", {"c": "/three"}]}
assert [sample for _, sample in tsan.find_absolute_paths_in_json(value)] == ["/x", "/one", "/two", "/three"]
def test_a_path_glued_to_a_url_does_not_ride_out_in_an_export(tmp_path):
"""The export's own no-local-paths assertion cannot catch this one.
``scan_absolute_paths`` re-runs the parser ``sanitize_export_tree`` just
ran, so a path the parser cannot see is a path the assertion cannot see
either -- it reports CLEAN on a package that carries it. The assertion
below therefore reads the written file rather than the scanner's verdict.
"""
payload = '{"u":"https://api.x.com/v1","p":"/etc/raven/license.key"}'
(tmp_path / "log.json").write_text(json.dumps({"line": payload}), encoding="utf-8")
tsan.sanitize_export_tree(tmp_path)
written = (tmp_path / "log.json").read_text(encoding="utf-8")
assert "/etc/raven/license.key" not in written
assert tsan.PATH_PLACEHOLDER in written
def test_a_url_path_used_as_a_query_value_is_redacted_too():
"""The accepted cost of ending the URL run at "?".
A query value that is a legitimate URL path reads as an absolute path and
goes the same way as a real one. The alternative is exempting query values,
which cannot be combined with the export's clean claim:
``scan_absolute_paths`` is a hard stop, not a warning, so a package that
carried such a path would either fail to build or ship under a sentence
that is not true of it.
"""
assert (
tsan.sanitize_text("go https://x.com/a?redirect=/login now")
== "go https://x.com/a?redirect=[REDACTED:path] now"
)
# A single-page-app route in a fragment reads the same way. Pinned so that
# someone reading a report later, wondering where a redirect target went,
# finds the answer here rather than filing it as a rendering bug.
assert tsan.sanitize_text("open https://x.com/a#/spa/route/here now") == "open https://x.com/a#[REDACTED:path] now"
def test_a_parenthesised_url_segment_loses_its_tail():
"""The accepted cost of letting ")" open a path token.
A URL whose own path carries parentheses has the remainder after them read
as a local path and redacted. Balanced-bracket tracking inside the URL run
would preserve it, and is not worth the parser: this direction loses a URL
tail in a report, while the other loses a path out of one.
"""
assert tsan.sanitize_text("https://example.com/a_(b)/public/page") == "https://example.com/a_(b)[REDACTED:path]"
def test_known_roots_replaced_wherever_they_appear(tmp_path):
root = str(tmp_path)
text = f"glued{root}/deep/file.txt end"
sanitized = tsan.sanitize_text(text, [root])
assert root not in sanitized
assert tsan.PATH_PLACEHOLDER in sanitized
def test_sanitize_tree_rewrites_structured_paths(tmp_path):
tree = tmp_path / "redacted"
tree.mkdir()
(tree / "manifest.json").write_text(
json.dumps({"missing_artifacts": ["/abs/gone/report.png"], "session_key": "cli:a"}), encoding="utf-8"
)
span = _span("trace-1", attrs={"tool.artifact_path": "/abs/gone/report.png"})
(tree / "spans.jsonl").write_text(json.dumps(span) + "\n", encoding="utf-8")
tsan.sanitize_export_tree(tree)
manifest = json.loads((tree / "manifest.json").read_text(encoding="utf-8"))
assert manifest["missing_artifacts"] == ["[REDACTED:path]/report.png"]
rewritten = json.loads((tree / "spans.jsonl").read_text(encoding="utf-8"))
# null, not a placeholder path: replay treats a non-string reference as
# "missing at pack time", while a relative placeholder would make it raise.
assert rewritten["attributes"]["tool.artifact_path"] is None
assert tsan.scan_absolute_paths(tree) == []
def test_tree_digest_change_add_and_symlink(tmp_path):
tree = tmp_path / "t"
(tree / "sub").mkdir(parents=True)
(tree / "a.txt").write_text("one", encoding="utf-8")
(tree / "sub" / "b.txt").write_text("two", encoding="utf-8")
baseline = tsan.tree_digest(tree)
assert baseline == tsan.tree_digest(tree)
(tree / "a.txt").write_text("changed", encoding="utf-8")
assert tsan.tree_digest(tree) != baseline
(tree / "a.txt").write_text("one", encoding="utf-8")
assert tsan.tree_digest(tree) == baseline
(tree / "extra.txt").write_text("x", encoding="utf-8")
assert tsan.tree_digest(tree) != baseline
(tree / "extra.txt").unlink()
(tree / "link").symlink_to(tree / "a.txt")
with pytest.raises(ValueError):
tsan.tree_digest(tree)
# ── member trace resolution ────────────────────────────────────────────
def test_member_traces_covers_all_address_shapes(state, workspace):
_write_log(
state / "logs" / "audit-spans.log",
[
_span("trace-1", attempt_id="att-old", session_key="cli:a"),
_span("trace-2", attempt_id="att-old", session_key="cli:a"),
_span("trace-3", session_key="cli:a"),
_span("trace-4", session_key="cli:a"),
_span("trace-5", session_key="cli:a"),
],
)
aid = tstore.merge_attempts(["trace-4", "trace-5"], state_dir=state)
assert tstore.member_traces(aid, state) == ("trace-4", "trace-5")
assert tstore.member_traces("trace-4", state) == ("trace-4", "trace-5")
assert tstore.member_traces("att-old", state) == ("trace-1", "trace-2")
assert tstore.member_traces("trace-3", state) == ("trace-3",)
assert tstore.member_traces("nope", state) is None
# ── classification ─────────────────────────────────────────────────────
def _report(**kw):
defaults = dict(bundle_dir=None, redacted_dir=None)
defaults.update(kw)
return RedactionReport(**defaults)
def test_classify_private_key_hit_is_a_review_reason():
classification, reasons = breport.classify_redaction(_report(patterns={"private-key-block": 1}))
assert classification == "needs_review"
assert reasons == ["the original trajectory contained 1 private key block(s), replaced before export"]
classification, reasons = breport.classify_redaction(_report(), _report(patterns={"private-key-block": 2}))
assert classification == "needs_review"
assert reasons == ["the original trajectory contained 2 private key block(s), replaced before export"]
def test_classify_needs_review_reasons_merge_both_sides():
finding = ResidualFinding(category="high-entropy", sample="x", file="spans.jsonl")
classification, reasons = breport.classify_redaction(
_report(findings=[finding], config_loaded=False, skipped_binaries=["a.bin"]),
_report(findings=[finding]),
)
assert classification == "needs_review"
assert reasons == [
"residual scan flagged 2 suspicious token(s)",
"config could not be fully read — known-value redaction may be incomplete",
"1 non-UTF-8 file(s) excluded from the copy and not scanned",
]
def test_classify_clean_ignores_ordinary_pattern_hits():
classification, reasons = breport.classify_redaction(_report(patterns={"openai-style-key": 3}))
assert classification == "clean"
assert reasons == []
# ── record schema ──────────────────────────────────────────────────────
def _valid_payload(report_id="br-20260904-abcdef", status="failed"):
payload = {
"schema": breport.RECORD_SCHEMA,
"schema_version": 1,
"report_id": report_id,
"created_at": "2026-09-04T07:03:12Z",
"raven_version": "0.0.0",
"reporter": "",
"attempt": {
"attempt_id": "att-x",
"session_key": None,
"member_traces": ["trace-1"],
"merged_definition": False,
},
"problem": {"description": "it broke", "expected": "", "actual": "", "severity": "", "steps": ""},
"status": status,
"failure": {"reason": "boom", "retryable": True, "retry_count": 0},
"snapshot": {"source_digest": "sha256:" + "0" * 64, "export_digest": "sha256:" + "0" * 64, "kept": True},
"package": {"path": "", "sha256": "", "size_bytes": 0},
"completeness": {"status": "unknown", "reasons": []},
"redaction": {"classification": "clean", "reasons": [], "reviewed_by_user": True},
"upload": {"state": "", "issue_url": "", "receipt": ""},
"links": {"issue": "", "pr": "", "regression_case": ""},
}
if status == "local_ready":
payload["failure"] = {"reason": "", "retryable": False, "retry_count": 0}
payload["snapshot"]["kept"] = False
payload["package"] = {"path": "/somewhere/br.tar.gz", "sha256": "0" * 64, "size_bytes": 10}
elif status == "draft":
payload["failure"] = {"reason": "", "retryable": False, "retry_count": 0}
return payload
def test_record_roundtrip_is_atomic(tmp_path):
record_dir = tmp_path / "br-20260904-abcdef"
record_dir.mkdir()
breport.save_record(record_dir, _valid_payload())
assert breport.load_record(record_dir)["report_id"] == "br-20260904-abcdef"
assert not [p for p in record_dir.iterdir() if p.name != breport.RECORD_FILE]
def _damage(payload, spec):
if spec == "unknown-major":
payload["schema_version"] = 2
elif spec == "version-zero":
payload["schema_version"] = 0
elif spec == "version-bool":
payload["schema_version"] = True
elif spec == "foreign-schema":
payload["schema"] = "something_else"
elif spec == "missing-attempt":
del payload["attempt"]
elif spec == "bool-retry-count":
payload["failure"]["retry_count"] = True
elif spec == "bad-severity":
payload["problem"]["severity"] = "urgent"
elif spec == "bad-status":
payload["status"] = "queued"
elif spec == "bad-digest":
payload["snapshot"]["export_digest"] = "sha256:xyz"
elif spec == "traversal-id":
payload["report_id"] = "../../outside"
elif spec == "mismatched-id":
payload["report_id"] = "br-20260904-ffffff"
elif spec == "ready-empty-package":
payload["status"] = "local_ready"
payload["failure"] = {"reason": "", "retryable": False, "retry_count": 0}
payload["snapshot"]["kept"] = False
elif spec == "ready-kept-snapshot":
payload.update(_valid_payload(payload["report_id"], status="local_ready"))
payload["snapshot"]["kept"] = True
elif spec == "ready-bad-sha":
payload.update(_valid_payload(payload["report_id"], status="local_ready"))
payload["package"]["sha256"] = "zz"
elif spec == "ready-carries-failure":
payload.update(_valid_payload(payload["report_id"], status="local_ready"))
payload["failure"]["retryable"] = True
elif spec == "failed-empty-reason":
payload["failure"]["reason"] = ""
elif spec == "failed-retryable-unkept":
payload["snapshot"]["kept"] = False
elif spec == "draft-with-package":
payload.update(_valid_payload(payload["report_id"], status="draft"))
payload["package"]["path"] = "/somewhere/early.tar.gz"
elif spec == "bad-completeness":
payload["completeness"]["status"] = "anything"
elif spec == "missing-upload-fields":
payload["upload"] = {}
elif spec == "missing-links-fields":
payload["links"] = {}
elif spec == "traces-empty-string":
payload["attempt"]["member_traces"] = [""]
elif spec == "traces-duplicate":
payload["attempt"]["member_traces"] = ["trace-1", "trace-1"]
elif spec == "traces-unsorted":
payload["attempt"]["member_traces"] = ["trace-2", "trace-1"]
elif spec == "blocked-classification":
payload["redaction"]["classification"] = "blocked"
elif spec == "unreviewed":
payload["redaction"]["reviewed_by_user"] = False
elif spec == "bad-security-notices":
payload["redaction"]["security_notices"] = ["ok", 3]
elif spec == "decision-bad-action":
payload["redaction"]["user_decisions"] = [
{"id": "abc123abc123", "category": "high-entropy", "masked_sample": "x", "action": "shipped", "sources": []}
]
elif spec == "decision-missing-id":
payload["redaction"]["user_decisions"] = [
{"category": "high-entropy", "masked_sample": "x", "action": "kept", "sources": []}
]
elif spec == "decision-bad-masked-token":
payload["redaction"]["user_decisions"] = [
{
"id": "abc123abc123",
"category": "high-entropy",
"masked_sample": "x",
"masked_token": "",
"action": "kept",
"sources": [],
}
]
elif spec == "decision-bad-source-count":
payload["redaction"]["user_decisions"] = [
{
"id": "abc123abc123",
"category": "high-entropy",
"masked_sample": "x",
"action": "kept",
"sources": [{"source": "the span log", "count": -1}],
}
]
elif spec == "bad-upload-state":
payload["upload"]["state"] = "queued"
elif spec == "upload-error-type":
payload["upload"]["error"] = 1
elif spec == "upload-uploaded-no-receipt":
payload.update(_valid_payload(payload["report_id"], status="local_ready"))
payload["upload"] = {"state": "uploaded", "issue_url": "", "receipt": ""}
elif spec == "upload-failed-no-error":
payload.update(_valid_payload(payload["report_id"], status="local_ready"))
payload["upload"] = {"state": "failed", "issue_url": "", "receipt": ""}
elif spec == "upload-before-ready":
payload["upload"] = {"state": "uploaded", "issue_url": "", "receipt": "rc-000000000000"}
return payload
_DAMAGE_SPECS = [
"unknown-major",
"version-zero",
"version-bool",
"foreign-schema",
"missing-attempt",
"bool-retry-count",
"bad-severity",
"bad-status",
"bad-digest",
"traversal-id",
"mismatched-id",
"ready-empty-package",
"ready-kept-snapshot",
"ready-bad-sha",
"ready-carries-failure",
"failed-empty-reason",
"failed-retryable-unkept",
"draft-with-package",
"bad-completeness",
"missing-upload-fields",
"missing-links-fields",
"traces-empty-string",
"traces-duplicate",
"traces-unsorted",
"blocked-classification",
"unreviewed",
"bad-security-notices",
"decision-bad-action",
"decision-missing-id",
"decision-bad-masked-token",
"decision-bad-source-count",
"bad-upload-state",
"upload-error-type",
"upload-uploaded-no-receipt",
"upload-failed-no-error",
"upload-before-ready",
]
@pytest.mark.parametrize("status", ["draft", "failed", "local_ready"])
def test_every_legal_status_loads(tmp_path, status):
record_dir = tmp_path / "br-20260904-abcdef"
record_dir.mkdir()
breport.save_record(record_dir, _valid_payload(status=status))
assert breport.load_record(record_dir)["status"] == status
@pytest.mark.parametrize(
"upload",
[
{"state": "uploaded", "issue_url": "", "receipt": "rc-5f3c2a9d1e70", "uploaded_at": "2026-10-12T06:40:11Z"},
{"state": "failed", "issue_url": "", "receipt": "", "error": "connection refused"},
{"state": "", "issue_url": "", "receipt": ""},
],
)
def test_upload_fields_are_optional_and_validated(tmp_path, upload):
record_dir = tmp_path / "br-20260904-abcdef"
record_dir.mkdir()
payload = _valid_payload(status="local_ready")
payload["upload"] = upload
breport.save_record(record_dir, payload)
assert breport.load_record(record_dir)["upload"] == upload
@pytest.mark.parametrize(
"value,expected",
[
("br-20260904-abcdef", True),
("br-20260904-ABCDEF", False),
("br-20260904", False),
("br-2026090-abcdef", False),
(None, False),
(b"br-20260904-abcdef", False),
],
)
def test_is_report_id(value, expected):
assert breport.is_report_id(value) is expected
@pytest.mark.parametrize("spec", _DAMAGE_SPECS)
def test_record_validation_rejects_damage(tmp_path, spec):
record_dir = tmp_path / "br-20260904-abcdef"
record_dir.mkdir()
breport.save_record(record_dir, _damage(_valid_payload(), spec))
with pytest.raises(ValueError):
breport.load_record(record_dir)
def test_damaged_records_are_skipped_not_fatal(state, workspace):
_record_dir, _record = _full_flow(state, workspace)
root = breport.bugreports_root(state)
for i, spec in enumerate(_DAMAGE_SPECS):
broken = root / f"br-20260903-{i:06d}"
broken.mkdir()
breport.save_record(broken, _damage(_valid_payload(report_id=broken.name), spec))
(root / "br-20260902-aaaaaa").mkdir()
(root / "br-20260902-aaaaaa" / breport.RECORD_FILE).write_text("{broken", encoding="utf-8")
reports = breport.list_reports(state)
assert len(reports) == 1
def test_traversal_report_id_cannot_escape_on_retry(state):
root = breport.bugreports_root(state)
record_dir = root / "br-20260904-abcdef"
record_dir.mkdir(parents=True)
breport.save_record(record_dir, _damage(_valid_payload(), "traversal-id"))
with pytest.raises(ValueError):
breport.retry_packaging(record_dir)
outside = [p for p in root.parent.rglob("outside*")] + [p for p in root.parent.parent.glob("outside*")]
assert outside == []
def test_report_id_shape_and_uniqueness(tmp_path):
ids = {breport.new_report_id(tmp_path) for _ in range(32)}
assert len(ids) == 32
for report_id in ids:
assert re.fullmatch(r"br-\d{8}-[0-9a-f]{6}", report_id)
# ── the vertical flow ──────────────────────────────────────────────────
def test_full_flow_lands_local_ready(state, workspace):
record_dir, record = _full_flow(state, workspace)
assert record["status"] == "local_ready"
assert record["schema"] == breport.RECORD_SCHEMA
assert record["attempt"]["member_traces"] == ["trace-1"]
assert record["snapshot"]["kept"] is False
assert not (record_dir / "snapshot").exists()
assert not (breport.bugreports_root(state) / breport.STAGING_DIR / record["report_id"]).exists()
package = record_dir / f"{record['report_id']}.tar.gz"
assert str(package) == record["package"]["path"]
with tarfile.open(package) as tar:
names = set(tar.getnames())
rid = record["report_id"]
assert f"{rid}/bugreport.json" in names
assert f"{rid}/trajectory/trace-1.tar.gz" in names
meta = json.loads(tar.extractfile(f"{rid}/bugreport.json").read().decode("utf-8"))
assert meta["schema"] == breport.PACKAGE_SCHEMA
assert meta["redaction"]["classification"] == "clean"
assert meta["redaction"]["risk_accepted"] is False
assert "session_key" not in meta["attempt"]
assert meta["contents"][1]["path"] == "trajectory/trace-1.tar.gz"
def test_problem_token_redacted_but_clean(state, workspace):
secret = "sk-" + "a1b2c3d4" * 3
_record_dir, record = _full_flow(state, workspace, description=f"my key {secret} leaked")
assert record["status"] == "local_ready"
assert record["redaction"]["classification"] == "clean"
assert secret not in json.dumps(record)
assert "[REDACTED:pattern.openai-style-key]" in record["problem"]["description"]
def test_problem_entropy_needs_review_with_problem_file(state, workspace):
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description=f"weird token {_ENTROPY_TOKEN} appeared", decide=_decide_all())
assert prep.classification == "needs_review"
findings = prep.package_metadata["redaction"]["residual_findings"]
assert any(f["file"] == "problem.description" for f in findings)
prep.cleanup()
def test_problem_private_key_becomes_a_review_item(state, workspace):
prep = _prepared(state, workspace)
seen: list = []
def _decide(items, reasons):
seen.extend(items)
return _decide_all()(items, reasons)
prep = breport.freeze_export(prep, description=f"look: {_PEM}", decide=_decide)
assert prep.classification == "needs_review"
assert prep.export_dir.exists()
assert [item.kind for item in seen] == ["confirmed"]
assert prep.security_notices == ["the original trajectory contained 1 private key block(s), replaced before export"]
assert _PEM not in json.dumps(prep.package_metadata)
prep.cleanup()
def test_trajectory_private_key_preclassifies_as_review(state, workspace):
prep = _prepared(state, workspace, attrs={"llm.output": _PEM})
assert prep.classification == "needs_review"
assert prep.reasons == ["the original trajectory contained 1 private key block(s), replaced before export"]
prep.cleanup()
def test_description_paths_are_sanitized(state, workspace):
_record_dir, record = _full_flow(
state,
workspace,
description="crashed reading /Users/alice/project/config.json",
steps="run C:\\Users\\alice\\run.bat first",
)
text = json.dumps(record["problem"])
assert "/Users/alice" not in text and "C:\\\\Users" not in text and "C:\\Users" not in text
assert "[REDACTED:path]" in record["problem"]["description"]
assert "[REDACTED:path]" in record["problem"]["steps"]
def test_residual_sample_paths_are_sanitized(state, workspace):
prep = _prepared(state, workspace, attrs={"llm.output": f"token {_ENTROPY_TOKEN} at /tmp/leak/x.json"})
prep = breport.freeze_export(prep, description="it broke", decide=_decide_all())
findings = prep.package_metadata["redaction"]["residual_findings"]
assert findings, "the injected token must flag"
assert all("/tmp/leak" not in f["sample"] for f in findings)
prep.cleanup()
def test_missing_artifact_paths_never_reach_the_package(state, workspace, tmp_path):
gone = tmp_path / "artifacts" / "gone-evidence.png"
_record_dir, record = _full_flow(state, workspace, attrs={"tool.artifact_path": str(gone)})
assert record["status"] == "local_ready"
with tarfile.open(record["package"]["path"]) as tar:
rid = record["report_id"]
inner = tar.extractfile(f"{rid}/trajectory/trace-1.tar.gz").read()
inner_tar = tmp_path / "inner.tar.gz"
inner_tar.write_bytes(inner)
extract_dir = tmp_path / "inner"
with tarfile.open(inner_tar) as tar:
tar.extractall(extract_dir, filter="data")
manifest = json.loads((extract_dir / "trace-1" / "manifest.json").read_text(encoding="utf-8"))
assert manifest["missing_artifacts"] == ["[REDACTED:path]/gone-evidence.png"]
assert tsan.scan_absolute_paths(extract_dir) == []
def test_binary_artifact_flags_needs_review(state, workspace, tmp_path):
blob = tmp_path / "blob.bin"
blob.write_bytes(b"\xff\xfe\x00\x01")
prep = _prepared(state, workspace, attrs={"tool.artifact_path": str(blob)})
prep = breport.freeze_export(prep, description="binary attached")
assert prep.classification == "needs_review"
assert any("non-UTF-8" in reason for reason in prep.reasons)
prep.cleanup()
def test_config_read_failure_flags_needs_review(state, workspace, monkeypatch):
monkeypatch.setattr(breport, "collect_known_secrets", lambda _p: ([], False))
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description="it broke")
assert prep.classification == "needs_review"
assert any(reason.startswith("config could not be fully read") for reason in prep.reasons)
prep.cleanup()
def test_known_secret_redacted_in_problem_and_trajectory(state, workspace, monkeypatch):
monkeypatch.setattr(
breport, "collect_known_secrets", lambda _p: ([KnownSecret("config.api_key", "supersecretvalue")], True)
)
_record_dir, record = _full_flow(
state, workspace, description="failed with supersecretvalue", attrs={"llm.output": "sent supersecretvalue"}
)
assert "supersecretvalue" not in json.dumps(record)
assert record["redaction"]["classification"] == "clean"
def test_archive_headers_carry_no_local_identity(state, workspace, tmp_path):
"""Both tar layers must not leak this machine's uid/gid/user/group names."""
_record_dir, record = _full_flow(state, workspace)
rid = record["report_id"]
with tarfile.open(record["package"]["path"]) as tar:
for member in tar.getmembers():
assert (member.uid, member.gid, member.uname, member.gname) == (0, 0, "", "")
inner = tar.extractfile(f"{rid}/trajectory/trace-1.tar.gz").read()
inner_path = tmp_path / "inner.tar.gz"
inner_path.write_bytes(inner)
with tarfile.open(inner_path) as tar:
members = tar.getmembers()
assert members
for member in members:
assert (member.uid, member.gid, member.uname, member.gname) == (0, 0, "", "")
def test_backslashes_in_problem_fields_ship_verbatim(state, workspace):
fields = {
"description": "problem: \\n- step one \\n- step two",
"steps": "matches \\d+ digits",
"expected": "\\d+ at start",
}
_record_dir, record = _full_flow(state, workspace, **fields)
assert record["status"] == "local_ready"
with tarfile.open(record["package"]["path"]) as tar:
meta = json.loads(tar.extractfile(f"{record['report_id']}/bugreport.json").read().decode("utf-8"))
assert {name: meta["problem"][name] for name in fields} == fields
@pytest.mark.parametrize(
"path",
["/etc/raven/license.key", "C:\\Users\\alice\\key.txt", "\\\\server\\share\\key.txt"],
ids=["posix", "drive", "unc"],
)
def test_export_gate_reads_the_metadata_on_disk(state, workspace, monkeypatch, path):
"""The gate must stop a path that reached the written file, not the dict."""
real_gate = breport._assert_export_ready
def tampered_gate(prep, metadata):
target = prep.export_dir / "bugreport.json"
on_disk = json.loads(target.read_text(encoding="utf-8"))
on_disk["problem"]["steps"] = path
target.write_text(json.dumps(on_disk, ensure_ascii=False, indent=2), encoding="utf-8")
real_gate(prep, metadata)
monkeypatch.setattr(breport, "_assert_export_ready", tampered_gate)
prep = _prepared(state, workspace)
with pytest.raises(breport.ExportLeakError, match=r"bugreport\.json problem\.steps"):
breport.freeze_export(prep, description="it broke", steps="harmless", decide=_decide_all())
assert not (breport.bugreports_root(state) / prep.report_id).exists()
assert not (prep.staging_dir / "record.json").exists()
@pytest.mark.parametrize("damage", ["modify", "add", "symlink"], ids=["modified", "added-file", "symlink-injected"])
def test_first_packaging_verifies_frozen_export(state, workspace, damage):
"""A tree changed between freeze and confirm must never reach local_ready."""
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description="it broke")
export_dir = prep.export_dir
if damage == "modify":
target = export_dir / "bugreport.json"
target.write_text(target.read_text(encoding="utf-8") + " ", encoding="utf-8")
elif damage == "add":
(export_dir / "extra.txt").write_text("x", encoding="utf-8")
else:
(export_dir / "link").symlink_to(export_dir / "bugreport.json")
with pytest.raises(breport.PackagingError) as exc_info:
breport.confirm_and_package(prep, state_dir=state)
assert exc_info.value.retryable is False
record_dir = breport.bugreports_root(state) / prep.report_id
record = breport.load_record(record_dir)
assert record["status"] == "failed"
assert record["failure"]["retryable"] is False
assert record["failure"]["reason"] == breport.REASON_SNAPSHOT_CORRUPTED
assert not (record_dir / f"{prep.report_id}.tar.gz").exists()
# ── expected I/O failures are normalized ───────────────────────────────
def test_prepare_wraps_io_failure_and_cleans_staging(state, workspace, monkeypatch):
_log_simple(state)
def _boom(*_a, **_k):
raise OSError("disk full")
monkeypatch.setattr(breport, "collect_bundle", _boom)
with pytest.raises(breport.PreparationError):
breport.prepare_trajectory("trace-1", workspace=workspace, state_dir=state)
staging = breport.bugreports_root(state) / breport.STAGING_DIR
assert not any(staging.iterdir())
def test_freeze_wraps_tar_failure(state, workspace, monkeypatch):
prep = _prepared(state, workspace)
def _boom(*_a, **_k):
raise OSError("disk full")
monkeypatch.setattr(breport.tarfile, "open", _boom)
with pytest.raises(breport.PreparationError):
breport.freeze_export(prep, description="it broke")
prep.cleanup()
def test_confirm_wraps_record_landing_failure(state, workspace, monkeypatch):
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description="it broke")
def _boom(*_a, **_k):
raise OSError("read-only filesystem")
monkeypatch.setattr(breport.os, "replace", _boom)
with pytest.raises(breport.PreparationError):
breport.confirm_and_package(prep, state_dir=state)
monkeypatch.undo()
assert not (breport.bugreports_root(state) / prep.report_id).exists()
prep.cleanup()
assert not prep.staging_dir.exists()
# ── freezing, retry, and crash recovery ────────────────────────────────
def _fail_next_tar(monkeypatch):
real_open = tarfile.open
calls = {"n": 0}
def flaky(*args, **kwargs):
calls["n"] += 1
if calls["n"] == 1:
raise OSError("disk full")
return real_open(*args, **kwargs)
monkeypatch.setattr(breport.tarfile, "open", flaky)
def _failed_report(state, workspace, monkeypatch, **fields):
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description="it broke", **fields)
frozen = (prep.export_dir / "bugreport.json").read_bytes()
_fail_next_tar(monkeypatch)
with pytest.raises(breport.PackagingError):
breport.confirm_and_package(prep, state_dir=state)
record_dir = breport.bugreports_root(state) / prep.report_id
return record_dir, frozen
def test_retry_reuses_frozen_bytes_despite_config_change(state, workspace, monkeypatch):
record_dir, frozen = _failed_report(state, workspace, monkeypatch)
record = breport.load_record(record_dir)
assert record["status"] == "failed" and record["failure"]["retryable"] is True
assert (record_dir / "snapshot").exists() and record["snapshot"]["kept"] is True
monkeypatch.setattr(
breport, "collect_known_secrets", lambda _p: ([KnownSecret("config.other", "changedvalue")], False)
)
record = breport.retry_packaging(record_dir)
assert record["status"] == "local_ready"
assert record["failure"]["retry_count"] == 1
with tarfile.open(record["package"]["path"]) as tar:
shipped = tar.extractfile(f"{record['report_id']}/bugreport.json").read()
assert shipped == frozen
@pytest.mark.parametrize("damage", ["modify", "add", "symlink"], ids=["modified", "added-file", "symlink-injected"])
def test_retry_refuses_corrupted_snapshot(state, workspace, monkeypatch, damage):
record_dir, _frozen = _failed_report(state, workspace, monkeypatch)
export_dir = record_dir / "snapshot" / "export"
if damage == "modify":
target = export_dir / "bugreport.json"
target.write_text(target.read_text(encoding="utf-8") + " ", encoding="utf-8")
elif damage == "add":
(export_dir / "extra.txt").write_text("x", encoding="utf-8")
else:
(export_dir / "link").symlink_to(export_dir / "bugreport.json")
with pytest.raises(breport.PackagingError):
breport.retry_packaging(record_dir)
record = breport.load_record(record_dir)
assert record["status"] == "failed"
assert record["failure"]["retryable"] is False
assert record["failure"]["reason"] == breport.REASON_SNAPSHOT_CORRUPTED
def test_interrupted_draft_recovers_to_retryable_failed(state, workspace):
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description="it broke")
breport.save_record(prep.staging_dir, breport._new_record_payload(prep))
record_dir = breport.bugreports_root(state) / prep.report_id
os.replace(prep.staging_dir, record_dir)
((found_dir, record),) = breport.list_reports(state)
assert found_dir == record_dir
assert record["status"] == "failed"
assert record["failure"]["reason"] == breport.REASON_INTERRUPTED
assert record["failure"]["retryable"] is True
assert record["failure"]["retry_count"] == 0
record = breport.retry_packaging(record_dir)
assert record["status"] == "local_ready"
def test_interrupted_draft_without_snapshot_is_terminal(state, workspace):
import shutil
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description="it broke")
breport.save_record(prep.staging_dir, breport._new_record_payload(prep))
record_dir = breport.bugreports_root(state) / prep.report_id
os.replace(prep.staging_dir, record_dir)
shutil.rmtree(record_dir / "snapshot")
((_dir, record),) = breport.list_reports(state)
assert record["status"] == "failed"
assert record["failure"]["retryable"] is False
assert record["failure"]["reason"] == breport.REASON_INTERRUPTED_INCOMPLETE
with pytest.raises(breport.BugReportError):
breport.retry_packaging(record_dir)
# ── cardinality, association, cleanup ──────────────────────────────────
def test_two_reports_on_one_attempt_coexist(state, workspace):
_log_simple(state)
dirs = []
for description in ("first problem", "second problem"):
prep = breport.prepare_trajectory("trace-1", workspace=workspace, state_dir=state)
prep = breport.freeze_export(prep, description=description)
record_dir, record = breport.confirm_and_package(prep, state_dir=state)
assert record["status"] == "local_ready"
dirs.append(record_dir)
assert dirs[0] != dirs[1]
matches = breport.reports_for_attempt("trace-1", ("trace-1",), state)
assert len(matches) == 2
descriptions = {record["problem"]["description"] for _d, record in matches}
assert descriptions == {"first problem", "second problem"}
def test_legacy_multi_trace_attempt_freezes_both_members(state, workspace):
_write_log(
state / "logs" / "audit-spans.log",
[
_span("trace-1", attempt_id="att-old", session_key="cli:a"),
_span("trace-2", attempt_id="att-old", session_key="cli:a"),
],
)
prep = breport.prepare_trajectory("att-old", workspace=workspace, state_dir=state)
prep = breport.freeze_export(prep, description="legacy pair broke")
_record_dir, record = breport.confirm_and_package(prep, state_dir=state)
assert record["attempt"]["attempt_id"] == "att-old"
assert record["attempt"]["member_traces"] == ["trace-1", "trace-2"]
assert record["attempt"]["merged_definition"] is False
matches = breport.reports_for_attempt("att-old", ("trace-1", "trace-2"), state)
assert len(matches) == 1
def test_confirm_rejects_concurrent_member_change(state, workspace):
_write_log(
state / "logs" / "audit-spans.log",
[_span("trace-1", session_key="cli:a"), _span("trace-2", session_key="cli:a")],
)
prep = breport.prepare_trajectory("trace-1", workspace=workspace, state_dir=state)
prep = breport.freeze_export(prep, description="it broke")
tstore.merge_attempts(["trace-1", "trace-2"], state_dir=state)
with pytest.raises(breport.StaleAttemptError):
breport.confirm_and_package(prep, state_dir=state)
prep.cleanup()
assert breport.list_reports(state) == []
def test_prepare_rejects_expected_trace_mismatch(state, workspace):
_log_simple(state)
with pytest.raises(breport.StaleAttemptError):
breport.prepare_trajectory("trace-1", expected_traces=("trace-9",), workspace=workspace, state_dir=state)
assert not list((breport.bugreports_root(state) / breport.STAGING_DIR).iterdir())
def test_stale_staging_cleanup(state, workspace):
staging_root = breport.bugreports_root(state) / breport.STAGING_DIR
old = staging_root / "br-20260101-aaaaaa"
old.mkdir(parents=True)
os.utime(old, (0, 0))
fresh = staging_root / "br-20260904-bbbbbb"
fresh.mkdir()
removed = breport.cleanup_stale_staging(state)
assert removed == 1
assert not old.exists() and fresh.exists()
def test_unreadable_record_is_skipped(state, workspace):
_record_dir, _record = _full_flow(state, workspace)
broken = breport.bugreports_root(state) / "br-20260904-ffffff"
broken.mkdir()
(broken / breport.RECORD_FILE).write_text("{broken", encoding="utf-8")
assert len(breport.list_reports(state)) == 1
# ── environment summary, policy hook, deliverable probe ────────────────
def _config_file(tmp_path, payload):
path = tmp_path / "raven-config.json"
path.write_text(payload if isinstance(payload, str) else json.dumps(payload), encoding="utf-8")
return path
def test_environment_summary_lands_in_package(state, workspace, tmp_path):
config = _config_file(tmp_path, {"providers": {"alpha": {"apiKey": "x"}, "beta": {}}, "channels": {"cli": {}}})
_log_simple(state)
prep = breport.prepare_trajectory("trace-1", workspace=workspace, config_path=config, state_dir=state)
prep = breport.freeze_export(prep, description="it broke")
_record_dir, record = breport.confirm_and_package(prep, state_dir=state)
with tarfile.open(record["package"]["path"]) as tar:
meta = json.loads(tar.extractfile(f"{record['report_id']}/bugreport.json").read().decode("utf-8"))
env = meta["environment"]
assert env["providers"] == "alpha, beta"
assert env["channels"] == "cli"
for key in ("python", "os", "arch"):
assert env[key] and "unknown" not in env[key]
assert env["bundle_format_version"] == 1
assert env["record_schema_version"] == 1 and env["package_schema_version"] == 1
assert record["completeness"]["status"] in ("complete", "degraded", "unreplayable")
@pytest.mark.parametrize("source", ["attempt", "selection"])
def test_completeness_is_written_into_the_record_and_the_package(state, workspace, source):
"""The verdict and its reasons travel in both JSON files whichever way the
report was made: the command line's attempt or the web page's selection."""
if source == "attempt":
_log_simple(state)
prep = breport.prepare_trajectory("trace-1", workspace=workspace, state_dir=state)
else:
_selection_world(state, workspace)
prep = breport.prepare_selection(_plan(state, "trace-a"), workspace=workspace, state_dir=state)
prep = breport.freeze_export(prep, description="it broke", decide=_decide_all())
record_dir, record = breport.confirm_and_package(prep, state_dir=state)
on_disk = json.loads((record_dir / "record.json").read_text(encoding="utf-8"))["completeness"]
with tarfile.open(record["package"]["path"]) as tar:
packaged = json.loads(tar.extractfile(f"{record['report_id']}/{breport.PACKAGE_METADATA_FILE}").read())
assert on_disk == packaged["completeness"] == {"status": prep.completeness[0], "reasons": prep.completeness[1]}
assert on_disk["status"] in ("complete", "degraded", "unreplayable", "unknown")
assert isinstance(on_disk["reasons"], list)
def test_unreadable_config_degrades_environment_and_classification(state, workspace, tmp_path):
config = _config_file(tmp_path, "{broken")
_log_simple(state)
prep = breport.prepare_trajectory("trace-1", workspace=workspace, config_path=config, state_dir=state)
prep = breport.freeze_export(prep, description="it broke")
assert prep.environment["providers"] == "unknown (config unreadable)"
assert prep.classification == "needs_review"
assert any(reason.startswith("config could not be fully read") for reason in prep.reasons)
prep.cleanup()
def test_environment_values_go_through_redaction_pipeline(state, workspace, monkeypatch):
monkeypatch.setattr(
breport, "collect_known_secrets", lambda _p: ([KnownSecret("config.k", "supersecretvalue")], True)
)
monkeypatch.setattr(
breport,
"_collect_environment",
lambda _p: (
{
"python": "3.12.0",
"os": "Linux 5.4",
"arch": "x86_64",
"providers": "token supersecretvalue at /tmp/leak/env.log",
"channels": "",
},
True,
),
)
_record_dir, record = _full_flow(state, workspace)
with tarfile.open(record["package"]["path"]) as tar:
meta = json.loads(tar.extractfile(f"{record['report_id']}/bugreport.json").read().decode("utf-8"))
providers = meta["environment"]["providers"]
assert "supersecretvalue" not in providers and "/tmp/leak" not in providers
assert "[REDACTED:config.k]" in providers and "[REDACTED:path]" in providers
def test_policy_hook_upgrades_clean_to_needs_review(state, workspace, monkeypatch):
monkeypatch.setenv("RAVEN_BUGREPORT_REQUIRE_REVIEW", "1")
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description="it broke")
assert prep.classification == "needs_review"
assert prep.reasons == ["organization policy requires manual review"]
record_dir, record = breport.confirm_and_package(prep, state_dir=state)
assert record["redaction"]["reasons"] == ["organization policy requires manual review"]
with tarfile.open(record["package"]["path"]) as tar:
meta = json.loads(tar.extractfile(f"{record['report_id']}/bugreport.json").read().decode("utf-8"))
assert meta["redaction"]["risk_accepted"] is True
def test_policy_hook_reason_coexists_with_other_review_reasons(state, workspace, monkeypatch):
monkeypatch.setenv("RAVEN_BUGREPORT_REQUIRE_REVIEW", "true")
prep = _prepared(state, workspace, attrs={"llm.output": f"token {_ENTROPY_TOKEN}"})
prep = breport.freeze_export(prep, description="it broke", decide=_decide_all())
assert prep.classification == "needs_review"
assert any(r.startswith("residual scan flagged") for r in prep.reasons)
assert "organization policy requires manual review" in prep.reasons
prep.cleanup()
def test_policy_hook_reason_coexists_with_private_key_reason(state, workspace, monkeypatch):
monkeypatch.setenv("RAVEN_BUGREPORT_REQUIRE_REVIEW", "1")
prep = _prepared(state, workspace, attrs={"llm.output": _PEM})
prep = breport.freeze_export(prep, description="it broke", decide=_decide_all())
assert prep.classification == "needs_review"
assert any(r.startswith("the original trajectory contained") for r in prep.reasons)
assert "organization policy requires manual review" in prep.reasons
prep.cleanup()
def test_policy_hook_off_keeps_clean(state, workspace, monkeypatch):
monkeypatch.delenv("RAVEN_BUGREPORT_REQUIRE_REVIEW", raising=False)
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description="it broke")
assert prep.classification == "clean"
prep.cleanup()
# ── review flow through the pipeline ───────────────────────────────────
def test_freeze_without_decide_fails_when_items_exist(state, workspace):
prep = _prepared(state, workspace, attrs={"llm.output": f"token {_ENTROPY_TOKEN}"})
with pytest.raises(breport.BugReportError, match="decide"):
breport.freeze_export(prep, description="it broke")
assert not prep.export_dir.exists()
prep.cleanup()
def test_review_cancel_leaves_nothing(state, workspace):
prep = _prepared(state, workspace, attrs={"llm.output": f"token {_ENTROPY_TOKEN}"})
def _cancel(_items, _reasons):
raise treview.ReviewCancelledError("cancelled from the review screen")
with pytest.raises(treview.ReviewCancelledError):
breport.freeze_export(prep, description="it broke", decide=_cancel)
prep.cleanup()
assert not prep.staging_dir.exists()
root = breport.bugreports_root(state)
assert [p.name for p in root.iterdir() if p.name != breport.STAGING_DIR] == []
def test_private_key_flow_packages_with_notices(state, workspace):
record_dir, record = _full_flow(state, workspace, attrs={"llm.output": f"before {_PEM} after"})
assert record["status"] == "local_ready"
assert record["redaction"]["classification"] == "needs_review"
notices = ["the original trajectory contained 1 private key block(s), replaced before export"]
assert record["redaction"]["security_notices"] == notices
decisions = record["redaction"]["user_decisions"]
assert [entry["action"] for entry in decisions] == ["acknowledged"]
assert decisions[0]["category"] == "private-key-block"
assert decisions[0]["masked_token"] == "[REDACTED:pattern.private-key-block]"
with tarfile.open(record["package"]["path"]) as tar:
meta = json.loads(tar.extractfile(f"{record['report_id']}/bugreport.json").read().decode("utf-8"))
assert meta["redaction"]["security_notices"] == notices
assert meta["redaction"]["user_decisions"] == decisions
assert meta["redaction"]["risk_accepted"] is True
assert record_dir.name == record["report_id"]
def test_mixed_decisions_recorded_and_replaced_token_scrubbed(state, workspace, tmp_path):
"""Adjacent kept/redacted tokens: the kept finding's summary window covers
the redacted neighbor, so the package metadata must carry the filtered
spelling everywhere."""
keep = "qW3eR5tY7uI9oP1aS2dF4gH6"
drop = "zX8cV6bN4mL2kJ9hG7fD5sQ3"
def _decide(items, _reasons):
return [
treview.ReviewDecision(item.id, treview.ACTION_REDACTED if item.token == drop else treview.ACTION_KEPT)
for item in items
]
_record_dir, record = _full_flow(state, workspace, attrs={"llm.output": f"keep {keep} drop {drop}"}, decide=_decide)
assert record["status"] == "local_ready"
actions = sorted(entry["action"] for entry in record["redaction"]["user_decisions"])
assert actions == ["kept", "redacted"]
assert drop not in json.dumps(record)
rid = record["report_id"]
with tarfile.open(record["package"]["path"]) as tar:
meta_text = tar.extractfile(f"{rid}/bugreport.json").read().decode("utf-8")
inner = tar.extractfile(f"{rid}/trajectory/trace-1.tar.gz").read()
assert drop not in meta_text
assert "[REDACTED:user-confirmed]" in meta_text
inner_tar = tmp_path / "inner.tar.gz"
inner_tar.write_bytes(inner)
extract_dir = tmp_path / "inner"
with tarfile.open(inner_tar) as tar:
tar.extractall(extract_dir, filter="data")
for path in extract_dir.rglob("*"):
if path.is_file():
text = path.read_text(encoding="utf-8", errors="replace")
assert drop not in text, path
spans = (extract_dir / "trace-1" / "spans.jsonl").read_text(encoding="utf-8")
assert keep in spans and "[REDACTED:user-confirmed]" in spans
def test_redacted_value_in_artifact_filename_never_reaches_the_package(state, workspace, tmp_path):
"""Replacing a token rewrites its references,
but the artifact file itself would keep the name and ship as a tar member.
Member names and contents must both be clean after the rename."""
artifact = tmp_path / f"{_ENTROPY_TOKEN}.json"
artifact.write_text(json.dumps({"result": "clean"}), encoding="utf-8")
_record_dir, record = _full_flow(
state,
workspace,
attrs={"tool.output.artifact_path": str(artifact)},
decide=_decide_all(treview.ACTION_REDACTED),
)
assert record["status"] == "local_ready"
rid = record["report_id"]
with tarfile.open(record["package"]["path"]) as tar:
assert not any(_ENTROPY_TOKEN in name for name in tar.getnames())
inner = tar.extractfile(f"{rid}/trajectory/trace-1.tar.gz").read()
inner_tar = tmp_path / "inner.tar.gz"
inner_tar.write_bytes(inner)
with tarfile.open(inner_tar) as tar:
names = tar.getnames()
assert not any(_ENTROPY_TOKEN in name for name in names)
assert any(name.endswith("artifacts/[REDACTED:user-confirmed].json") for name in names)
extract_dir = tmp_path / "inner"
tar.extractall(extract_dir, filter="data")
spans = (extract_dir / "trace-1" / "spans.jsonl").read_text(encoding="utf-8")
assert _ENTROPY_TOKEN not in spans
assert "artifacts/[REDACTED:user-confirmed].json" in spans
def test_all_findings_replaced_still_needs_review(state, workspace):
prep = _prepared(state, workspace, attrs={"llm.output": f"token {_ENTROPY_TOKEN}"})
prep = breport.freeze_export(prep, description="it broke", decide=_decide_all(treview.ACTION_REDACTED))
assert prep.classification == "needs_review"
assert any("replaced at your direction" in reason for reason in prep.reasons)
assert prep.package_metadata["redaction"]["risk_accepted"] is True
assert prep.package_metadata["redaction"]["residual_findings"] == []
prep.cleanup()
def test_zero_item_policy_report_skips_decide_but_stays_review(state, workspace, monkeypatch):
monkeypatch.setenv("RAVEN_BUGREPORT_REQUIRE_REVIEW", "1")
calls: list = []
def _decide(items, _reasons):
calls.append(items)
return []
prep = _prepared(state, workspace)
prep = breport.freeze_export(prep, description="it broke", decide=_decide)
assert calls == []
assert prep.classification == "needs_review"
assert prep.package_metadata["redaction"]["risk_accepted"] is True
assert prep.package_metadata["redaction"]["user_decisions"] == []
prep.cleanup()
def test_completeness_probe_runs_on_the_post_decision_tree(state, workspace, monkeypatch):
seen: dict = {}
def _probe(redacted_dir, _report):
seen["spans"] = (redacted_dir / "spans.jsonl").read_text(encoding="utf-8")
return "complete", []
monkeypatch.setattr("raven.trajectory.completeness.evaluate_completeness", _probe)
prep = _prepared(state, workspace, attrs={"llm.output": f"token {_ENTROPY_TOKEN}"})
prep = breport.freeze_export(prep, description="it broke", decide=_decide_all(treview.ACTION_REDACTED))
assert _ENTROPY_TOKEN not in seen["spans"]
assert "[REDACTED:user-confirmed]" in seen["spans"]
prep.cleanup()
def test_completeness_reasons_inherit_user_redactions(state, workspace, monkeypatch):
def _probe(_redacted_dir, _report):
return "degraded", [f"probe saw {_ENTROPY_TOKEN}"]
monkeypatch.setattr("raven.trajectory.completeness.evaluate_completeness", _probe)
prep = _prepared(state, workspace, attrs={"llm.output": f"token {_ENTROPY_TOKEN}"})
prep = breport.freeze_export(prep, description="it broke", decide=_decide_all(treview.ACTION_REDACTED))
assert prep.completeness == ("degraded", ["probe saw [REDACTED:user-confirmed]"])
prep.cleanup()
def test_new_record_without_review_fields_still_validates(state, workspace):
"""The review fields are additive: a record stripped back to the old shape
(what pre-review writers produced) must still pass validation."""
record_dir, record = _full_flow(state, workspace, attrs={"llm.output": f"token {_ENTROPY_TOKEN}"})
payload = json.loads((record_dir / breport.RECORD_FILE).read_text(encoding="utf-8"))
assert payload["schema_version"] == 1
payload["redaction"].pop("security_notices")
payload["redaction"].pop("user_decisions")
breport._validate_record(payload, record_dir.name)
def test_completeness_evaluation_failure_degrades_to_unknown(state, workspace, monkeypatch):
def _boom(*_a, **_k):
raise OSError("probe environment failure")
monkeypatch.setattr("raven.trajectory.completeness.evaluate_completeness", _boom)
_record_dir, record = _full_flow(state, workspace)
assert record["completeness"]["status"] == "unknown"
assert record["completeness"]["reasons"] == ["completeness evaluation failed (OSError)"]
assert record["status"] == "local_ready"
def _artifact_file(tmp_path, name, payload):
path = tmp_path / name
path.write_text(json.dumps(payload), encoding="utf-8")
return path
def test_deliverable_probe_missing_llm_input_end_to_end(state, workspace, tmp_path):
import asyncio
from raven.trajectory.replay import load_recording, run_replay
turn_art = _artifact_file(tmp_path, "turn.json", {"content": "hi", "channel": "cli", "chat_id": "a"})
out_art = _artifact_file(tmp_path, "llm-out.json", {"content": "ok", "tool_calls": [], "usage": {}})
gone = tmp_path / "artifacts" / "llm-in-gone.json"
_write_log(
state / "logs" / "audit-spans.log",
[
_span("trace-1", session_key="cli:a", attrs={"turn.input.artifact_path": str(turn_art)}),
_span(
"trace-1",
session_key="cli:a",
name="llm.call",
span_id="llm-1",
attrs={"llm.input.artifact_path": str(gone), "llm.output.artifact_path": str(out_art)},
),
],
)
prep = breport.prepare_trajectory("trace-1", workspace=workspace, state_dir=state)
prep = breport.freeze_export(prep, description="it broke", decide=_decide_all())
record_dir, record = breport.confirm_and_package(prep, state_dir=state)
assert record["completeness"]["status"] == "degraded"
assert any("model call input" in r for r in record["completeness"]["reasons"])
with tarfile.open(record["package"]["path"]) as tar:
inner = tar.extractfile(f"{record['report_id']}/trajectory/trace-1.tar.gz").read()
inner_tar = tmp_path / "inner.tar.gz"
inner_tar.write_bytes(inner)
extract_dir = tmp_path / "inner"
with tarfile.open(inner_tar) as tar:
tar.extractall(extract_dir, filter="data")
delivered = extract_dir / "trace-1"
recording = load_recording(delivered)
assert recording.llm_calls[0].input is None
report = asyncio.run(run_replay(delivered, mode="warn"))
assert not [d for d in report.divergences if d.fatal]
def test_deliverable_probe_missing_tool_input_end_to_end(state, workspace, tmp_path):
import asyncio
from raven.trajectory.replay import load_recording, run_replay
turn_art = _artifact_file(tmp_path, "turn.json", {"content": "hi", "channel": "cli", "chat_id": "a"})
out1 = _artifact_file(
tmp_path, "llm-out-1.json", {"content": None, "tool_calls": [{"id": "t", "name": "x", "arguments": {}}]}
)
out2 = _artifact_file(tmp_path, "llm-out-2.json", {"content": "done", "tool_calls": []})
tool_out = _artifact_file(tmp_path, "tool-out.json", {"result": "done"})
gone = tmp_path / "artifacts" / "tool-in-gone.json"
_write_log(
state / "logs" / "audit-spans.log",
[
_span("trace-1", session_key="cli:a", attrs={"turn.input.artifact_path": str(turn_art)}),
_span(
"trace-1",
session_key="cli:a",
name="llm.call",
span_id="llm-1",
attrs={"llm.output.artifact_path": str(out1)},
),
_span(
"trace-1",
session_key="cli:a",
name="tool.call",
span_id="tool-1",
attrs={"tool.input.artifact_path": str(gone), "tool.output.artifact_path": str(tool_out)},
),
_span(
"trace-1",
session_key="cli:a",
name="llm.call",
span_id="llm-2",
attrs={"llm.output.artifact_path": str(out2)},
),
],
)
prep = breport.prepare_trajectory("trace-1", workspace=workspace, state_dir=state)
prep = breport.freeze_export(prep, description="it broke", decide=_decide_all())
_record_dir, record = breport.confirm_and_package(prep, state_dir=state)
assert record["completeness"]["status"] == "degraded"
assert any("tool call input" in r for r in record["completeness"]["reasons"])
with tarfile.open(record["package"]["path"]) as tar:
inner = tar.extractfile(f"{record['report_id']}/trajectory/trace-1.tar.gz").read()
inner_tar = tmp_path / "inner2.tar.gz"
inner_tar.write_bytes(inner)
extract_dir = tmp_path / "inner2"
with tarfile.open(inner_tar) as tar:
tar.extractall(extract_dir, filter="data")
delivered = extract_dir / "trace-1"
recording = load_recording(delivered)
assert recording.tool_calls[0].name is None
report = asyncio.run(run_replay(delivered, mode="warn"))
assert not [d for d in report.divergences if d.fatal]
def test_freeze_export_makes_no_network_calls(state, workspace, monkeypatch):
"""Counted with recording no-ops — the catalog warm swallows exceptions in
its daemon thread, so a raising stub would pass without proving anything."""
import raven.providers.rates as rates
calls: list[str] = []
monkeypatch.setattr(rates, "warm_catalog_in_background", lambda: calls.append("warm"))
monkeypatch.setattr(rates, "_fetch_openrouter_models", lambda **_k: calls.append("fetch") or {})
monkeypatch.setattr(rates, "_try_litellm_rates", lambda *_a, **_k: calls.append("litellm") or None)
_record_dir, record = _full_flow(state, workspace)
assert record["status"] == "local_ready"
assert calls == []
def test_merged_definition_attempt_full_flow(state, workspace):
_write_log(
state / "logs" / "audit-spans.log",
[_span("trace-1", session_key="cli:a"), _span("trace-2", session_key="cli:a")],
)
aid = tstore.merge_attempts(["trace-1", "trace-2"], state_dir=state)
prep = breport.prepare_trajectory(aid, workspace=workspace, state_dir=state)
prep = breport.freeze_export(prep, description="merged pair broke")
_record_dir, record = breport.confirm_and_package(prep, state_dir=state)
assert record["attempt"]["attempt_id"] == aid
assert record["attempt"]["member_traces"] == ["trace-1", "trace-2"]
assert record["attempt"]["merged_definition"] is True
import tempfile as _tempfile
with tarfile.open(record["package"]["path"]) as tar:
inner = tar.extractfile(f"{record['report_id']}/trajectory/{aid}.tar.gz").read()
scratch = Path(_tempfile.mkdtemp())
inner_path = scratch / "inner.tar.gz"
inner_path.write_bytes(inner)
with tarfile.open(inner_path) as tar:
tar.extractall(scratch, filter="data")
spans_text = (scratch / aid / "spans.jsonl").read_text(encoding="utf-8")
traces = {json.loads(line)["traceId"] for line in spans_text.splitlines() if line.strip()}
assert traces == {"trace-1", "trace-2"}
def test_freeze_rejects_whitespace_only_description(state, workspace):
prep = _prepared(state, workspace)
with pytest.raises(ValueError):
breport.freeze_export(prep, description=" ")
prep.cleanup()
def test_reports_order_by_creation_time_not_directory_name(tmp_path):
root = tmp_path / "traces" / "bugreports"
root.mkdir(parents=True)
# Same-day ids whose lexicographic order contradicts creation order.
older = _valid_payload("br-20260905-ffffff", status="local_ready")
older["created_at"] = "2026-09-05T08:00:00Z"
newer = _valid_payload("br-20260905-000000", status="local_ready")
newer["created_at"] = "2026-09-05T09:00:00Z"
for payload in (older, newer):
record_dir = root / payload["report_id"]
record_dir.mkdir()
breport.save_record(record_dir, payload)
listed = breport.list_reports(tmp_path / "traces")
assert [record["report_id"] for _dir, record in listed] == ["br-20260905-000000", "br-20260905-ffffff"]
# ── staging and finishing apart ────────────────────────────────────────
_INNER = "Hj5tR8uE3iO7pA1sD4fGk9lZ"
_OUTER = _INNER + "W6xC2v"
def test_stage_review_offers_the_items_freeze_export_would_ask_about(state, workspace, tmp_path):
asked: list[list[str]] = []
def record(items, reasons):
asked.append([item.id for item in items])
return _decide_all()(items, reasons)
prep = _prepared(state, workspace)
breport.freeze_export(prep, description=f"token {_ENTROPY_TOKEN}", decide=record)
prep.cleanup()
staged = _prepared(state, workspace)
stage = breport.stage_review(staged, description=f"token {_ENTROPY_TOKEN}")
assert [item.id for item in stage.items] == asked[0] and stage.items
assert stage.reasons == staged.reasons and stage.raw_fields["description"] == f"token {_ENTROPY_TOKEN}"
decisions = _decide_all(treview.ACTION_REDACTED)(stage.items, stage.reasons)
finished = breport.finish_export(staged, stage, decisions)
recorded = {entry["id"]: entry["action"] for entry in finished.package_metadata["redaction"]["user_decisions"]}
assert recorded == {decision.item_id: decision.action for decision in decisions}
assert _ENTROPY_TOKEN not in json.dumps(finished.package_metadata)
staged.cleanup()
def test_a_preparation_is_staged_once(state, workspace):
prep = _prepared(state, workspace)
breport.stage_review(prep, description="first")
with pytest.raises(breport.BugReportError, match="already been staged"):
breport.stage_review(prep, description="again")
prep.cleanup()
def test_finishing_refuses_decisions_for_a_report_without_items(state, workspace):
prep = _prepared(state, workspace)
stage = breport.stage_review(prep, description="plain words only")
assert stage.items == []
with pytest.raises(breport.BugReportError, match="without review items"):
breport.finish_export(prep, stage, [treview.ReviewDecision("x", treview.ACTION_KEPT)])
assert breport.finish_export(prep, stage, []).package_metadata is not None
prep.cleanup()
def test_conflicting_linked_decisions_freeze_nothing(state, workspace):
prep = _prepared(state, workspace)
stage = breport.stage_review(prep, description=f"{_INNER} then {_OUTER}")
by_token = {item.token: item for item in stage.items}
decisions = [
treview.ReviewDecision(by_token[_INNER].id, treview.ACTION_KEPT),
treview.ReviewDecision(by_token[_OUTER].id, treview.ACTION_REDACTED),
] + [
treview.ReviewDecision(item.id, treview.ACTION_KEPT)
for item in stage.items
if item.token not in (_INNER, _OUTER)
]
with pytest.raises(treview.ReviewConflictError):
breport.finish_export(prep, stage, decisions)
assert not prep.export_dir.exists()
prep.cleanup()
# ── a report built from selected turns ─────────────────────────────────
_SEL_SESSION = "cli:sel"
def _turn_span(trace_id, minute, *, dispatched_in=None, session_key=_SEL_SESSION):
attrs = {"turn.in_progress": False, "turn.input_preview": trace_id}
if dispatched_in:
attrs["trace.dispatched_in_trace_id"] = dispatched_in
span = _span(trace_id, session_key=session_key, attrs=attrs)
span["startTime"] = f"2026-08-20T10:{minute:02d}:00+00:00"
span["endTime"] = f"2026-08-20T10:{minute:02d}:30+00:00"
return span
def _selection_world(state, workspace):
from raven.session.manager import SessionManager
_write_log(
state / "logs" / "audit-spans.log",
[
_turn_span("trace-a", 1),
_span("trace-a", session_key=_SEL_SESSION, name="llm.call", span_id="span-a-llm"),
_turn_span("trace-sub", 2, dispatched_in="trace-a", session_key="subagent:x"),
_turn_span("trace-b", 3),
],
)
path = SessionManager(workspace).session_path(_SEL_SESSION)
path.parent.mkdir(parents=True, exist_ok=True)
rows = [
{"_type": "metadata", "key": _SEL_SESSION, "metadata": {"title": "t"}, "last_consolidated": 0},
{"role": "user", "content": "first", "timestamp": "2026-08-20T10:01:00", "trace_id": "trace-a"},
{"role": "assistant", "content": "answer one", "timestamp": "2026-08-20T10:01:20"},
{"role": "user", "content": "second", "timestamp": "2026-08-20T10:03:00", "trace_id": "trace-b"},
]
path.write_text("".join(json.dumps(r) + "\n" for r in rows), encoding="utf-8")
return path
def _plan(state, *trace_ids):
from raven.trajectory import selection as tsel
return tsel.plan_selection(_SEL_SESSION, list(trace_ids), answering=False, state_dir=state)
def test_the_synthetic_attempt_id_is_stable_for_a_set_of_turns():
assert re.fullmatch(r"sel-[0-9a-f]{12}", breport.selection_attempt_id(["trace-b", "trace-a"]))
assert breport.selection_attempt_id(["trace-b", "trace-a"]) == breport.selection_attempt_id(["trace-a", "trace-b"])
assert breport.selection_attempt_id(["trace-a"]) != breport.selection_attempt_id(["trace-a", "trace-b"])
def test_prepare_selection_bundles_the_turns_and_their_dispatched_traces(state, workspace):
_selection_world(state, workspace)
prep = breport.prepare_selection(_plan(state, "trace-a"), workspace=workspace, state_dir=state)
try:
assert prep.attempt_id == breport.selection_attempt_id(["trace-a"])
assert prep.member_traces == ("trace-a", "trace-sub")
assert prep.selected_traces == ("trace-a",)
assert prep.merged_definition is False and prep.session_key == _SEL_SESSION
bundle = prep.snapshot_dir / "bundle" / prep.attempt_id
rows = [json.loads(line) for line in (bundle / "session.jsonl").read_text(encoding="utf-8").splitlines()]
assert [r.get("content") for r in rows[1:]] == ["first", "answer one"]
assert prep.manifest["session_key"] == _SEL_SESSION and prep.manifest["session_included"] is True
assert sorted(p.name for p in state.iterdir()) == ["bugreports", "logs"], "nothing pinned or defined"
finally:
prep.cleanup()
def test_a_selection_bundle_has_the_same_manifest_keys_as_an_attempt_bundle(state, workspace):
_selection_world(state, workspace)
prep = breport.prepare_selection(_plan(state, "trace-a"), workspace=workspace, state_dir=state)
attempt = breport.prepare_trajectory("trace-b", workspace=workspace, state_dir=state)
try:
assert set(prep.manifest) == set(attempt.manifest)
finally:
prep.cleanup()
attempt.cleanup()
def test_a_selection_without_a_session_record_says_so(state, workspace):
_selection_world(state, workspace).unlink()
prep = breport.prepare_selection(_plan(state, "trace-a"), workspace=workspace, state_dir=state)
try:
assert prep.manifest["session_included"] is False
assert not (prep.snapshot_dir / "bundle" / prep.attempt_id / "session.jsonl").exists()
finally:
prep.cleanup()
def test_prepare_selection_takes_only_a_plan(state, workspace):
with pytest.raises(TypeError):
breport.prepare_selection(["trace-a"], workspace=workspace, state_dir=state) # type: ignore[arg-type]
def test_a_turn_purged_after_planning_is_stale_and_leaves_no_staging(state, workspace):
_selection_world(state, workspace)
plan = _plan(state, "trace-a")
_write_log(state / "logs" / "audit-spans.log", [_turn_span("trace-b", 3)])
with pytest.raises(breport.StaleAttemptError):
breport.prepare_selection(plan, workspace=workspace, state_dir=state)
assert list((breport.bugreports_root(state) / breport.STAGING_DIR).iterdir()) == []
def test_a_selection_report_lands_and_passes_the_receivers_check(state, workspace):
from raven.trajectory.package_check import inspect_package
_selection_world(state, workspace)
prep = breport.prepare_selection(_plan(state, "trace-a"), workspace=workspace, state_dir=state)
prep = breport.freeze_export(prep, description="the reply was wrong", decide=_decide_all())
record_dir, record = breport.confirm_and_package(prep, state_dir=state)
assert record["status"] == breport.STATUS_LOCAL_READY
assert breport.load_record(record_dir)["attempt"]["attempt_id"] == prep.attempt_id
summary = inspect_package(Path(record["package"]["path"]), expected_report_id=record["report_id"])
assert summary.metadata["attempt"]["member_traces"] == ["trace-a", "trace-sub"]
found = breport.reports_for_attempt("att-unrelated", ("trace-a",), state_dir=state)
assert [r["report_id"] for _d, r in found] == [record["report_id"]]
def test_a_selection_report_whose_turn_vanished_does_not_land(state, workspace):
_selection_world(state, workspace)
prep = breport.prepare_selection(_plan(state, "trace-a"), workspace=workspace, state_dir=state)
prep = breport.freeze_export(prep, description="the reply was wrong", decide=_decide_all())
_write_log(state / "logs" / "audit-spans.log", [_turn_span("trace-b", 3)])
with pytest.raises(breport.StaleAttemptError):
breport.confirm_and_package(prep, state_dir=state)
assert prep.staging_dir.is_dir() and not (breport.bugreports_root(state) / prep.report_id).exists()
prep.cleanup()
def _store(state):
return state / "logs" / "audit-artifacts"
def _put(state, rel, text):
path = _store(state) / rel
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
return path
def _bounded_world(state, workspace, tmp_path):
"""A selected turn whose spans point at every kind of out-of-bounds or non-regular file."""
from raven.tracing import artifact_v2
_selection_world(state, workspace)
outside = tmp_path / "outside-store"
outside.mkdir()
(outside / "secret.txt").write_text("OUTSIDE-BYTES-artifact", encoding="utf-8")
(outside / "blob.json").write_text('{"role": "user", "content": "OUTSIDE-BYTES-blob"}', encoding="utf-8")
good = {"role": "user", "content": "inside the store"}
good_sha = artifact_v2.message_sha1(good)
_put(state, f"_messages/{good_sha[:2]}/{good_sha}.json", artifact_v2.message_text(good))
linked_sha = "a" * 40
link = _store(state) / "_messages" / "aa" / f"{linked_sha}.json"
link.parent.mkdir(parents=True, exist_ok=True)
link.symlink_to(outside / "blob.json")
shard_sha = "b" * 40
(outside / "shard").mkdir()
(outside / "shard" / f"{shard_sha}.json").write_text('{"content": "OUTSIDE-BYTES-shard"}', encoding="utf-8")
(_store(state) / "_messages" / "bb").symlink_to(outside / "shard", target_is_directory=True)
dir_sha = "c" * 40
(_store(state) / "_messages" / "cc" / f"{dir_sha}.json").mkdir(parents=True)
fifo_sha = "d" * 40
(_store(state) / "_messages" / "dd").mkdir()
os.mkfifo(_store(state) / "_messages" / "dd" / f"{fifo_sha}.json")
shell = _put(
state,
"llm.input/2026-08-20/shell.json",
json.dumps(
{
"artifactFormat": artifact_v2.ARTIFACT_FORMAT,
"messages": [artifact_v2.make_ref(s) for s in (good_sha, linked_sha, shard_sha, dir_sha, fifo_sha)],
}
),
)
linked_artifact = _store(state) / "tool.output" / "2026-08-20" / "linked.txt"
linked_artifact.parent.mkdir(parents=True, exist_ok=True)
linked_artifact.symlink_to(outside / "secret.txt")
fifo_artifact = _store(state) / "tool.output" / "2026-08-20" / "fifo.txt"
os.mkfifo(fifo_artifact)
spans = [json.loads(line) for line in (state / "logs" / "audit-spans.log").read_text().splitlines()]
spans.append(
_span(
"trace-a",
session_key=_SEL_SESSION,
name="tool.call",
span_id="span-a-tool",
attrs={
"llm.input.artifact_path": str(shell),
"tool.outside.artifact_path": str(outside / "secret.txt"),
"tool.linked.artifact_path": str(linked_artifact),
"tool.fifo.artifact_path": str(fifo_artifact),
},
)
)
_write_log(state / "logs" / "audit-spans.log", spans)
return {
"good_sha": good_sha,
"bad_shas": {linked_sha, shard_sha, dir_sha, fifo_sha},
"bad_artifacts": {str(outside / "secret.txt"), str(linked_artifact), str(fifo_artifact)},
}
@pytest.mark.skipif(not hasattr(os, "mkfifo"), reason="needs FIFOs")
def test_a_selection_report_reads_only_regular_files_inside_the_store(state, workspace, tmp_path):
import time
world = _bounded_world(state, workspace, tmp_path)
started = time.monotonic()
prep = breport.prepare_selection(_plan(state, "trace-a"), workspace=workspace, state_dir=state)
try:
assert time.monotonic() - started < 5, "a FIFO must not hang the preparation"
bundle = prep.snapshot_dir / "bundle" / prep.attempt_id
packed = b"".join(p.read_bytes() for p in bundle.rglob("*") if p.is_file())
assert b"OUTSIDE-BYTES" not in packed
assert world["bad_artifacts"] <= set(prep.manifest["missing_artifacts"])
assert world["bad_shas"] <= set(prep.manifest["missing_messages"])
shell = next(p for p in (bundle / "artifacts").iterdir() if p.name.endswith("shell.json"))
messages = json.loads(shell.read_text(encoding="utf-8"))["messages"]
assert messages[0] == {"role": "user", "content": "inside the store"}, "a legal shell still resolves to v1"
from raven.tracing import artifact_v2
assert messages[1:] == [artifact_v2.placeholder(s) for s in ("a" * 40, "b" * 40, "c" * 40, "d" * 40)]
finally:
prep.cleanup()