## Summary The web page gains two things a conversation could only get from the terminal before: exporting the selected turns as a download, and reporting a problem through the bug report pipeline. The trajectory view is now on for everyone. - **Trajectory view without `--dev` (R0).** `raven`, `raven web`, `raven serve` and `raven gateway` no longer take `--dev`; the trajectory policy starts enabled. `raven tui --dev` (running the TUI from source) is unchanged. - **Turn selection.** From this version each turn's opening user message records its trace id (the Turn Trace Mark, never sent to the model). `raven/trajectory/selection.py` lists a session's main-line turns, closes a selection over the sub-agent and delegated traces it dispatched, collects their spans, and cuts the session record down to the selected turns; older records are matched conservatively and an unmatched turn stays in the trajectory only. - **Export.** The Trajectory Export package v1 (`schema: raven_trajectory_export`, `format_version: 1`) is the input contract of the coming `web-traj-import`; it is defined in `CONTEXT.md` and pinned by tests. It holds the raw spans, artifacts and v2 message blobs of the selected traces plus the cut session record, unredacted. Exports run as background jobs (at most two) and the browser downloads them from the same-origin `/trajectory/export/download` route with a one-time token. New RPCs: `trajectory.turns`, `trajectory.export`, `trajectory.export_status`, `trajectory.export_cancel`. - **Bug reports.** `freeze_export` is split into staging and finishing so a report can be prepared from selected turns; the CLI's behaviour is unchanged. A Report Session holds a web report in the gateway between page requests: at most two at once, all blocking I/O on worker threads, a cancel checked after every step (a report cancelled while packaging is never sent), a failed or uploaded report releases its slot and keeps its last status for ten minutes, and an idle one is cancelled after thirty minutes. Seven `bugreport.*` RPCs, the `bugreport-v1` capability (announced only while the receiver URL is valid), and error codes -32026 / -32027 / -32028. Every decision made on the web page, defaults and bulk buttons included, is remembered in the redaction history with `web` as its source, unlike the CLI's bulk flags. - **Web page.** A three-dot conversation menu in the header's corner (the desk launcher moves one slot in) opens the new `export` wizard. Export: pick turns, name the file under an "unredacted" warning, follow the job, download once automatically with a link to download again. Report a problem: pick turns, describe the problem, decide the uncertain values in two columns (hits highlighted at code points), upload, and get a receipt. Closing before the upload starts discards the report being prepared; once the upload starts the report is saved on the machine, a failure marked retryable offers a retry, and the saved report can be sent later from the command line by its id. Every answer is checked against the wizard's epoch and the job or report it was asked about, and a write step whose answer is lost is settled from the report's status instead of being sent again. Layouts were measured at 360 and 390 px in English and Chinese. - **Docs and tests.** The webui and trajectory-debugging pages (English and Chinese) describe both flows. Two integration tests drive them end to end: the export through the real download route, and the bug report against a real receiver process, including a stopped receiver, a retry, the redaction history's replay and a cancel at each step. ## Type - [ ] Fix - [x] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - The branch merges the current `main` (one merge commit, no conflicts); everything below ran on the merged tree. - `uv run pytest tests/ -q` (full suite): 57 failed / 31491 passed / 177 skipped. Every one of the 57 also fails on the current `main` on this machine (environment-specific, pre-existing); none is introduced by this branch. - `uv run pytest -m integration tests/integration/test_web_traj_export_e2e.py tests/integration/test_web_bugreport_e2e.py tests/integration/test_bug_report_upload_e2e.py -q`: 10 passed (three consecutive runs of the two new files). - `uv run ruff check`, `uv run ruff format --check`, `uv run ty check` on the changed Python: pass. - ui-web: `npm run gen:check` (239 methods), `npm run lint`, `npm run type-check`, `npx vitest run` (246 files / 3868 passed), `npm run build` and `python3 ui-web/build.py` (both boot snapshots match), `check-page.mjs`, `check-css.mjs`, `check-class-namespace.mjs`: pass. - ui-tui: `npm run lint:rpc`, `npm run lint:i18n`: pass. - `uv run --group docs mkdocs build -f docs-site/mkdocs.yml --strict`: pass. - `make check-source-language check-large-files check-commits`: pass. - Headless Chromium on the built page (offline fixtures): both flows walked to the download and to the receipt; the header corner measured in five layouts (desk panel open and closed, workspace panel open, shell mode, narrow split) and the wizard at 360 and 390 px in both languages. - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed ## Risk - A process started with the old `--dev` flag that runs `system.upgrade` relaunches itself with `--dev`, which the new version refuses; the user runs `raven web` once by hand. This was accepted as the cost of dropping the flag. - The trajectory view is on by default: each open page calls the light, in-memory `trajectory.state` every five seconds, and the index is built only when the view is opened. No switch to turn it off is added here; the policy keeps a replaceable source for a settings entry later. - The export is the raw trajectory and may contain secrets. The page says so in a warning; the download is served only on the same origin, behind the session guard, by an unguessable one-time token, and no path ever reaches the page. - Rollback: revert the squash commit. Nothing is migrated; the `trace_id` field already written into session records is ignored by older versions. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues N/A --------- Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
592 lines
24 KiB
Python
592 lines
24 KiB
Python
"""Tests for the trajectory bundle collector (`raven.trajectory.bundle`)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import shutil
|
|
|
|
import pytest
|
|
|
|
from raven.session.manager import SessionManager
|
|
from raven.trajectory import bundle as tbundle
|
|
from raven.trajectory import store as tstore
|
|
from raven.trajectory import verdict as tverdict
|
|
|
|
|
|
def _write_log(path, spans):
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
path.write_text("".join(json.dumps(s) + "\n" for s in spans), encoding="utf-8")
|
|
|
|
|
|
def _span(trace_id, *, attempt_id=None, session_key=None, name="session.turn", start=None, end=None, attrs=None):
|
|
"""Span in the current format; pass ``attempt_id`` for a legacy-format span."""
|
|
attributes = {"session.key": session_key}
|
|
if attempt_id is not None:
|
|
attributes["attempt.id"] = attempt_id
|
|
if attrs:
|
|
attributes.update(attrs)
|
|
return {
|
|
"schemaVersion": "audit.span.v1",
|
|
"traceId": trace_id,
|
|
"spanId": f"span-{trace_id}-{name}",
|
|
"name": name,
|
|
"startTime": start or "2026-08-20T10:00:00+00:00",
|
|
"endTime": end or "2026-08-20T10:00:01+00:00",
|
|
"attributes": attributes,
|
|
}
|
|
|
|
|
|
@pytest.fixture
|
|
def state(tmp_path):
|
|
return tmp_path / "traces"
|
|
|
|
|
|
@pytest.fixture
|
|
def workspace(tmp_path):
|
|
ws = tmp_path / "ws"
|
|
ws.mkdir()
|
|
return ws
|
|
|
|
|
|
def _make_artifact(state, name, content):
|
|
path = state / "logs" / "audit-artifacts" / "tool.output" / "2026-08-20" / name
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
path.write_text(content, encoding="utf-8")
|
|
return path
|
|
|
|
|
|
def _read_lines(path):
|
|
return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
|
|
|
|
|
|
def test_bundle_roundtrip(state, workspace):
|
|
artifact = _make_artifact(state, "output.json", '{"result": 42}')
|
|
manager = SessionManager(workspace)
|
|
session = manager.get_or_create("cli:chat1")
|
|
session.add_message("user", "hello")
|
|
manager.save(session)
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[
|
|
_span("trace-1", session_key="cli:chat1", start="2026-08-20T10:00:00+00:00"),
|
|
_span(
|
|
"trace-1",
|
|
session_key="cli:chat1",
|
|
name="tool.call",
|
|
end="2026-08-20T10:00:05+00:00",
|
|
attrs={"tool.output.artifact_path": str(artifact)},
|
|
),
|
|
],
|
|
)
|
|
tverdict.record_verdict("trace-1", "fail", source="user", why="wrong answer", state_dir=state)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", state_dir=state, workspace=workspace)
|
|
|
|
assert bundle_dir == (state / "bundles" / "trace-1").resolve()
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["format_version"] == tbundle.BUNDLE_FORMAT_VERSION
|
|
assert manifest["attempt_id"] == "trace-1"
|
|
assert manifest["session_key"] == "cli:chat1"
|
|
assert manifest["time_range"] == {"start": "2026-08-20T10:00:00+00:00", "end": "2026-08-20T10:00:05+00:00"}
|
|
assert manifest["span_count"] == 2
|
|
assert manifest["artifact_count"] == 1
|
|
assert manifest["verdict_count"] == 1
|
|
assert manifest["session_included"] is True
|
|
assert manifest["missing_artifacts"] == []
|
|
assert manifest["raven_version"]
|
|
|
|
spans = _read_lines(bundle_dir / "spans.jsonl")
|
|
assert [s["name"] for s in spans] == ["session.turn", "tool.call"]
|
|
verdicts = _read_lines(bundle_dir / "verdicts.jsonl")
|
|
assert verdicts[0]["status"] == "fail" and verdicts[0]["why"] == "wrong answer"
|
|
source_session = manager.session_path("cli:chat1")
|
|
assert (bundle_dir / "session.jsonl").read_text(encoding="utf-8") == source_session.read_text(encoding="utf-8")
|
|
|
|
|
|
def test_artifact_paths_rewritten_offline(state, workspace, tmp_path):
|
|
artifact = _make_artifact(state, "big-output.json", '{"tokens": [1, 2, 3]}')
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1", name="tool.call", attrs={"tool.output.artifact_path": str(artifact)})],
|
|
)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", out_dir=tmp_path / "out", state_dir=state, workspace=workspace)
|
|
|
|
span = _read_lines(bundle_dir / "spans.jsonl")[0]
|
|
rel = span["attributes"]["tool.output.artifact_path"]
|
|
assert rel == "artifacts/big-output.json"
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["rewritten_artifact_paths"] == 1
|
|
|
|
# The bundle must read on its own after the live store is gone.
|
|
moved = tmp_path / "elsewhere"
|
|
shutil.move(str(bundle_dir), str(moved))
|
|
shutil.rmtree(state)
|
|
assert (moved / rel).read_text(encoding="utf-8") == '{"tokens": [1, 2, 3]}'
|
|
|
|
|
|
def test_missing_artifact_not_fatal(state, workspace):
|
|
gone = str(state / "logs" / "audit-artifacts" / "purged.json")
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1", name="tool.call", attrs={"tool.output.artifact_path": gone})],
|
|
)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", state_dir=state, workspace=workspace)
|
|
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["missing_artifacts"] == [gone]
|
|
assert manifest["artifact_count"] == 0
|
|
span = _read_lines(bundle_dir / "spans.jsonl")[0]
|
|
assert span["attributes"]["tool.output.artifact_path"] == gone
|
|
|
|
|
|
def test_artifact_basename_collision_disambiguated(state, workspace):
|
|
a = _make_artifact(state, "output.json", "first")
|
|
b = state / "other" / "output.json"
|
|
b.parent.mkdir(parents=True)
|
|
b.write_text("second", encoding="utf-8")
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[
|
|
_span("trace-1", name="tool.call", attrs={"tool.output.artifact_path": str(a)}),
|
|
_span("trace-1", name="llm.call", attrs={"llm.output.artifact_path": str(b)}),
|
|
],
|
|
)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", state_dir=state, workspace=workspace)
|
|
|
|
spans = _read_lines(bundle_dir / "spans.jsonl")
|
|
rel_a = spans[0]["attributes"]["tool.output.artifact_path"]
|
|
rel_b = spans[1]["attributes"]["llm.output.artifact_path"]
|
|
assert rel_a != rel_b
|
|
assert (bundle_dir / rel_a).read_text(encoding="utf-8") == "first"
|
|
assert (bundle_dir / rel_b).read_text(encoding="utf-8") == "second"
|
|
|
|
|
|
def test_bundle_spans_rotation(state, workspace):
|
|
archive = state / "logs" / "archive"
|
|
_write_log(
|
|
archive / "2026-08-18" / "audit-spans-2026-08-18-010101.log",
|
|
[_span("trace-1", attempt_id="att-x", start="2026-08-18T09:00:00+00:00")],
|
|
)
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[
|
|
_span("trace-2", attempt_id="att-x", end="2026-08-20T11:00:00+00:00"),
|
|
_span("trace-9"),
|
|
],
|
|
)
|
|
|
|
bundle_dir = tbundle.collect_bundle("att-x", state_dir=state, workspace=workspace)
|
|
|
|
spans = _read_lines(bundle_dir / "spans.jsonl")
|
|
assert [s["traceId"] for s in spans] == ["trace-1", "trace-2"]
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["time_range"] == {"start": "2026-08-18T09:00:00+00:00", "end": "2026-08-20T11:00:00+00:00"}
|
|
|
|
|
|
def test_save_auto_pins(state, workspace):
|
|
_write_log(state / "logs" / "audit-spans.log", [_span("trace-1")])
|
|
tbundle.collect_bundle("trace-1", state_dir=state, workspace=workspace)
|
|
registry = tstore.pins(state)
|
|
assert registry["trace-1"]["reason"] == "bundled"
|
|
|
|
|
|
def test_missing_session_recorded(state, workspace):
|
|
_write_log(state / "logs" / "audit-spans.log", [_span("trace-1", session_key="cli:never-saved")])
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", state_dir=state, workspace=workspace)
|
|
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["session_included"] is False
|
|
assert not (bundle_dir / "session.jsonl").exists()
|
|
|
|
|
|
def test_unknown_id_raises(state, workspace):
|
|
_write_log(state / "logs" / "audit-spans.log", [_span("trace-1")])
|
|
with pytest.raises(LookupError):
|
|
tbundle.collect_bundle("no-such-id", state_dir=state, workspace=workspace)
|
|
with pytest.raises(ValueError):
|
|
tbundle.collect_bundle("", state_dir=state, workspace=workspace)
|
|
|
|
|
|
def test_trace_id_resolves_to_full_attempt(state, workspace):
|
|
"""One turn's trace id must pack the whole multi-turn attempt it belongs to."""
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[
|
|
_span("trace-1", attempt_id="att-x", session_key="cli:a"),
|
|
_span("trace-2", attempt_id="att-x", session_key="cli:a", name="tool.call"),
|
|
],
|
|
)
|
|
tverdict.record_verdict("att-x", "fail", source="user", state_dir=state)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", state_dir=state, workspace=workspace)
|
|
|
|
assert bundle_dir.name == "att-x"
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["attempt_id"] == "att-x"
|
|
assert manifest["span_count"] == 2
|
|
assert manifest["verdict_count"] == 1
|
|
assert [s["traceId"] for s in _read_lines(bundle_dir / "spans.jsonl")] == ["trace-1", "trace-2"]
|
|
registry = tstore.pins(state)
|
|
assert "att-x" in registry and "trace-1" not in registry
|
|
|
|
|
|
def test_unsafe_id_rejected(state, workspace, tmp_path):
|
|
"""An attempt id (user-mintable) must not escape the output root as a path."""
|
|
for evil in ("../escaped", str(tmp_path / "abs-escaped"), "a/b"):
|
|
_write_log(state / "logs" / "audit-spans.log", [_span("trace-1", attempt_id=evil)])
|
|
with pytest.raises(ValueError):
|
|
tbundle.collect_bundle(evil, state_dir=state, workspace=workspace)
|
|
assert not (tmp_path / "escaped").exists()
|
|
assert not (tmp_path / "abs-escaped").exists()
|
|
assert not (state / "escaped").exists()
|
|
|
|
|
|
def test_resave_drops_stale_artifacts(state, workspace):
|
|
"""A re-pack must not keep artifacts whose source was purged since last pack."""
|
|
artifact = _make_artifact(state, "secret.json", '{"api_key": "sk-live"}')
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1", name="tool.call", attrs={"tool.output.artifact_path": str(artifact)})],
|
|
)
|
|
first = tbundle.collect_bundle("trace-1", state_dir=state, workspace=workspace)
|
|
assert (first / "artifacts" / "secret.json").exists()
|
|
|
|
artifact.unlink()
|
|
second = tbundle.collect_bundle("trace-1", state_dir=state, workspace=workspace)
|
|
|
|
assert not (second / "artifacts" / "secret.json").exists()
|
|
manifest = json.loads((second / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["artifact_count"] == 0
|
|
assert manifest["missing_artifacts"] == [str(artifact)]
|
|
leftovers = [p.name for p in second.parent.iterdir() if p.name.startswith(".")]
|
|
assert leftovers == []
|
|
|
|
|
|
def test_default_workspace_reads_config(state, tmp_path, monkeypatch):
|
|
"""Without an explicit workspace, the configured agent workspace is used."""
|
|
import types
|
|
|
|
ws = tmp_path / "custom-ws"
|
|
ws.mkdir()
|
|
manager = SessionManager(ws)
|
|
session = manager.get_or_create("cli:chat1")
|
|
session.add_message("user", "hello")
|
|
manager.save(session)
|
|
monkeypatch.setattr(
|
|
"raven.config.loader.load_config",
|
|
lambda config_path=None: types.SimpleNamespace(workspace_path=ws),
|
|
)
|
|
_write_log(state / "logs" / "audit-spans.log", [_span("trace-1", session_key="cli:chat1")])
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", state_dir=state)
|
|
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["session_included"] is True
|
|
assert (bundle_dir / "session.jsonl").exists()
|
|
|
|
|
|
def test_end_to_end_with_real_tracer(tmp_path, monkeypatch):
|
|
"""A trajectory recorded by the live tracer bundles into an offline dir."""
|
|
from raven.tracing import spans as _spans
|
|
from raven.tracing import trace
|
|
|
|
state = tmp_path / "traces"
|
|
workspace = tmp_path / "ws"
|
|
workspace.mkdir()
|
|
monkeypatch.setenv("RAVEN_TRACING", "1")
|
|
monkeypatch.setenv("RAVEN_TRACING_DIR", str(state))
|
|
_spans._store = None
|
|
try:
|
|
# A single turn is addressed by its trace id (attempt id = trace id).
|
|
with trace.span("session.turn", session_key="cli:e2e") as root:
|
|
with trace.span("tool.call") as s:
|
|
s.artifact("tool.output", {"result": "ok", "items": list(range(50))})
|
|
aid = root.trace_id
|
|
|
|
bundle_dir = tbundle.collect_bundle(aid, state_dir=state, workspace=workspace)
|
|
|
|
spans = _read_lines(bundle_dir / "spans.jsonl")
|
|
tool_span = next(s for s in spans if s["name"] == "tool.call")
|
|
rel = tool_span["attributes"]["tool.output.artifact_path"]
|
|
assert rel.startswith("artifacts/")
|
|
payload = json.loads((bundle_dir / rel).read_text(encoding="utf-8"))
|
|
assert payload["result"] == "ok" and len(payload["items"]) == 50
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["attempt_id"] == aid
|
|
assert manifest["artifact_count"] == 1
|
|
assert tstore.pins(state)[aid]["reason"] == "bundled"
|
|
finally:
|
|
_spans._store = None
|
|
|
|
|
|
def _v2_artifact(state, messages, name="llm-input.json"):
|
|
"""A v2 shell on disk with its message blobs, as the writer leaves it."""
|
|
from raven.tracing import artifact_v2 as v2
|
|
|
|
artifacts = state / "logs" / "audit-artifacts"
|
|
refs = []
|
|
for message in messages:
|
|
sha1 = v2.message_sha1(message)
|
|
blob = v2.message_path(artifacts, sha1)
|
|
blob.parent.mkdir(parents=True, exist_ok=True)
|
|
blob.write_text(v2.message_text(message), encoding="utf-8")
|
|
refs.append(v2.make_ref(sha1))
|
|
shell = {
|
|
"artifactFormat": v2.ARTIFACT_FORMAT,
|
|
"model": "openrouter/x",
|
|
"systemPrompt": refs[0],
|
|
"prompt": refs[-1],
|
|
"messages": refs,
|
|
"tools": None,
|
|
}
|
|
path = artifacts / "llm.input" / "2026-08-20" / name
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
path.write_text(json.dumps(shell, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
return path
|
|
|
|
|
|
_V2_MESSAGES = [
|
|
{"role": "system", "content": "sys"},
|
|
{"role": "user", "content": "hi"},
|
|
]
|
|
|
|
|
|
def test_a_bundled_v2_artifact_is_written_out_resolved(state, workspace, tmp_path):
|
|
source = _v2_artifact(state, _V2_MESSAGES)
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1", name="llm.call", attrs={"llm.input.artifact_path": str(source)})],
|
|
)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", out_dir=tmp_path / "out", state_dir=state, workspace=workspace)
|
|
|
|
packed = json.loads((bundle_dir / "artifacts" / source.name).read_text(encoding="utf-8"))
|
|
assert "artifactFormat" not in packed
|
|
assert packed["messages"] == _V2_MESSAGES
|
|
assert packed["systemPrompt"] == "sys"
|
|
assert packed["prompt"] == "hi"
|
|
|
|
|
|
def test_a_bundled_v2_artifact_reads_after_the_live_store_is_gone(state, workspace, tmp_path):
|
|
"""The bundle's whole promise: it survives being copied to another machine."""
|
|
source = _v2_artifact(state, _V2_MESSAGES)
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1", name="llm.call", attrs={"llm.input.artifact_path": str(source)})],
|
|
)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", out_dir=tmp_path / "out", state_dir=state, workspace=workspace)
|
|
moved = tmp_path / "elsewhere"
|
|
shutil.move(str(bundle_dir), str(moved))
|
|
shutil.rmtree(state)
|
|
|
|
packed = json.loads((moved / "artifacts" / source.name).read_text(encoding="utf-8"))
|
|
assert packed["messages"] == _V2_MESSAGES
|
|
|
|
|
|
def test_a_missing_message_blob_becomes_a_placeholder_and_reaches_the_manifest(state, workspace, tmp_path):
|
|
from raven.tracing import artifact_v2 as v2
|
|
|
|
source = _v2_artifact(state, _V2_MESSAGES)
|
|
gone = v2.message_sha1(_V2_MESSAGES[1])
|
|
v2.message_path(state / "logs" / "audit-artifacts", gone).unlink()
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1", name="llm.call", attrs={"llm.input.artifact_path": str(source)})],
|
|
)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", out_dir=tmp_path / "out", state_dir=state, workspace=workspace)
|
|
|
|
packed = json.loads((bundle_dir / "artifacts" / source.name).read_text(encoding="utf-8"))
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert len(packed["messages"]) == 2, "position is preserved; a message is never dropped"
|
|
assert packed["messages"][0] == _V2_MESSAGES[0]
|
|
assert gone in packed["messages"][1]["content"]
|
|
assert manifest["missing_messages"] == [gone]
|
|
|
|
|
|
def test_a_v1_artifact_is_still_copied_byte_for_byte(state, workspace, tmp_path):
|
|
raw = '{"messages": [{"role": "user", "content": "hi"}]}'
|
|
source = _make_artifact(state, "v1.json", raw)
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1", name="tool.call", attrs={"tool.output.artifact_path": str(source)})],
|
|
)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", out_dir=tmp_path / "out", state_dir=state, workspace=workspace)
|
|
|
|
assert (bundle_dir / "artifacts" / "v1.json").read_text(encoding="utf-8") == raw
|
|
|
|
|
|
def test_bundle_tolerates_non_string_legacy_id(state, workspace):
|
|
"""save by trace id still bundles when the span carries a list attempt.id
|
|
(unvalidated history must degrade, not crash the collector)."""
|
|
span = _span("trace-1", session_key="cli:chat1")
|
|
span["attributes"]["attempt.id"] = ["not", "a", "string"]
|
|
_write_log(state / "logs" / "audit-spans.log", [span])
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", state_dir=state, workspace=workspace)
|
|
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["attempt_id"] == "trace-1"
|
|
assert manifest["span_count"] == 1
|
|
assert "trace-1" in tstore.pins(state)
|
|
|
|
|
|
def test_bundle_of_merged_attempt_survives_bad_neighbor_record(state):
|
|
"""A list-traceId record elsewhere in the log must not break bundling a
|
|
merged attempt (the member filter is a set lookup)."""
|
|
bad = {"schemaVersion": "audit.span.v1", "traceId": ["x"], "attributes": {}}
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1", session_key="cli:chat1"), bad, _span("trace-2", session_key="cli:chat1")],
|
|
)
|
|
aid = tstore.merge_attempts(["trace-1", "trace-2"], state_dir=state)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-1", state_dir=state)
|
|
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["attempt_id"] == aid
|
|
assert manifest["span_count"] == 2
|
|
|
|
|
|
def test_bundle_of_merged_attempt_tolerates_malformed_verdict_lines(state):
|
|
"""The alias-set verdict filter must skip a list attempt_id line instead
|
|
of failing the whole collection."""
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1", session_key="cli:chat1"), _span("trace-2", session_key="cli:chat1")],
|
|
)
|
|
tverdict.record_verdict("trace-1", "fail", source="user", state_dir=state)
|
|
aid = tstore.merge_attempts(["trace-1", "trace-2"], state_dir=state)
|
|
with (state / "verdicts.jsonl").open("a", encoding="utf-8") as f:
|
|
f.write(json.dumps({"attempt_id": ["bad"], "status": "fail", "source": "user", "ts": "now"}) + "\n")
|
|
|
|
bundle_dir = tbundle.collect_bundle(aid, state_dir=state)
|
|
|
|
verdicts = _read_lines(bundle_dir / "verdicts.jsonl")
|
|
assert {v["attempt_id"] for v in verdicts} == {"trace-1"}
|
|
|
|
|
|
def test_bundle_of_merged_attempt_collects_all_member_traces(state):
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[
|
|
_span("trace-1", session_key="cli:chat1", start="2026-08-20T10:00:00+00:00"),
|
|
_span("trace-2", session_key="cli:chat1", start="2026-08-20T11:00:00+00:00"),
|
|
],
|
|
)
|
|
tverdict.record_verdict("trace-1", "fail", source="user", state_dir=state)
|
|
aid = tstore.merge_attempts(["trace-1", "trace-2"], state_dir=state)
|
|
tverdict.record_verdict(aid, "pass", source="user", state_dir=state)
|
|
|
|
bundle_dir = tbundle.collect_bundle("trace-2", state_dir=state)
|
|
|
|
assert bundle_dir == (state / "bundles" / aid).resolve()
|
|
spans = _read_lines(bundle_dir / "spans.jsonl")
|
|
assert [s["traceId"] for s in spans] == ["trace-1", "trace-2"]
|
|
manifest = json.loads((bundle_dir / "manifest.json").read_text(encoding="utf-8"))
|
|
assert manifest["attempt_id"] == aid
|
|
assert manifest["span_count"] == 2
|
|
verdicts = _read_lines(bundle_dir / "verdicts.jsonl")
|
|
assert {v["attempt_id"] for v in verdicts} == {"trace-1", aid}
|
|
|
|
|
|
def test_bundle_of_definition_with_purged_members_raises_lookup_error(state):
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1"), _span("trace-2")],
|
|
)
|
|
aid = tstore.merge_attempts(["trace-1", "trace-2"], state_dir=state)
|
|
(state / "logs" / "audit-spans.log").unlink()
|
|
|
|
with pytest.raises(LookupError, match="no spans found for attempt"):
|
|
tbundle.collect_bundle(aid, state_dir=state)
|
|
|
|
|
|
def test_bundle_pin_falls_back_to_member_traces_when_attempt_vanishes(state, monkeypatch):
|
|
_write_log(
|
|
state / "logs" / "audit-spans.log",
|
|
[_span("trace-1"), _span("trace-2")],
|
|
)
|
|
aid = tstore.merge_attempts(["trace-1", "trace-2"], state_dir=state)
|
|
|
|
def vanished(id_, *, reason="", state_dir=None):
|
|
raise LookupError(f"no spans found for id {id_!r}")
|
|
|
|
monkeypatch.setattr(tbundle, "pin_attempt", vanished)
|
|
tbundle.collect_bundle(aid, state_dir=state)
|
|
|
|
registry = tstore.pins(state)
|
|
assert registry["trace-1"]["reason"] == "bundled"
|
|
assert registry["trace-2"]["reason"] == "bundled"
|
|
|
|
|
|
def test_write_bundle_tree_takes_the_session_key_it_is_given(state, tmp_path):
|
|
from raven.trajectory.bundle import write_bundle_tree
|
|
|
|
dest = tmp_path / "tree"
|
|
dest.mkdir()
|
|
seen: list[str | None] = []
|
|
|
|
def no_session(target, key):
|
|
seen.append(key)
|
|
return False
|
|
|
|
spans = [_span("trace-sub", session_key="subagent:x"), _span("trace-a", session_key="cli:main")]
|
|
manifest = write_bundle_tree(
|
|
dest,
|
|
spans,
|
|
attempt_id="sel-000000000000",
|
|
state_dir=state,
|
|
write_session=no_session,
|
|
verdicts=[],
|
|
session_key="cli:main",
|
|
)
|
|
assert manifest["session_key"] == "cli:main" and seen == ["cli:main"]
|
|
assert manifest["session_included"] is False and not (dest / "session.jsonl").exists()
|
|
derived = tmp_path / "derived"
|
|
derived.mkdir()
|
|
assert (
|
|
write_bundle_tree(
|
|
derived, spans, attempt_id="sel-000000000000", state_dir=state, write_session=no_session, verdicts=[]
|
|
)["session_key"]
|
|
== "subagent:x"
|
|
)
|
|
|
|
|
|
def test_write_bundle_tree_confined_to_the_store_leaves_outside_files_missing(state, tmp_path):
|
|
from raven.trajectory.bundle import write_bundle_tree
|
|
|
|
outside = tmp_path / "outside.txt"
|
|
outside.write_text("not from the store", encoding="utf-8")
|
|
inside = _make_artifact(state, "inside.json", '{"content": "from the store"}')
|
|
spans = [
|
|
_span(
|
|
"trace-a", session_key="cli:main", attrs={"a.artifact_path": str(outside), "b.artifact_path": str(inside)}
|
|
)
|
|
]
|
|
store = (state / "logs" / "audit-artifacts").resolve()
|
|
|
|
def none(_target, _key):
|
|
return False
|
|
|
|
confined = tmp_path / "confined"
|
|
confined.mkdir()
|
|
manifest = write_bundle_tree(
|
|
confined, spans, attempt_id="sel-000000000000", state_dir=state, write_session=none, verdicts=[], confine=store
|
|
)
|
|
assert manifest["missing_artifacts"] == [str(outside)]
|
|
assert sorted(p.name for p in (confined / "artifacts").iterdir()) == ["inside.json"]
|
|
|
|
open_tree = tmp_path / "open"
|
|
open_tree.mkdir()
|
|
manifest = write_bundle_tree(
|
|
open_tree, spans, attempt_id="sel-000000000000", state_dir=state, write_session=none, verdicts=[]
|
|
)
|
|
assert manifest["missing_artifacts"] == [], "without confine the attempt bundle keeps its old behavior"
|