Raven-1/tests/test_trajectory_export.py
江国庆/Forrest 6b938a4267
feat(*): export conversations and report problems from the web page (#948)
## Summary

The web page gains two things a conversation could only get from the
terminal before: exporting the selected turns as a download, and
reporting a problem through the bug report pipeline. The trajectory view
is now on for everyone.

- **Trajectory view without `--dev` (R0).** `raven`, `raven web`, `raven
serve` and `raven gateway` no longer take `--dev`; the trajectory policy
starts enabled. `raven tui --dev` (running the TUI from source) is
unchanged.
- **Turn selection.** From this version each turn's opening user message
records its trace id (the Turn Trace Mark, never sent to the model).
`raven/trajectory/selection.py` lists a session's main-line turns,
closes a selection over the sub-agent and delegated traces it
dispatched, collects their spans, and cuts the session record down to
the selected turns; older records are matched conservatively and an
unmatched turn stays in the trajectory only.
- **Export.** The Trajectory Export package v1 (`schema:
raven_trajectory_export`, `format_version: 1`) is the input contract of
the coming `web-traj-import`; it is defined in `CONTEXT.md` and pinned
by tests. It holds the raw spans, artifacts and v2 message blobs of the
selected traces plus the cut session record, unredacted. Exports run as
background jobs (at most two) and the browser downloads them from the
same-origin `/trajectory/export/download` route with a one-time token.
New RPCs: `trajectory.turns`, `trajectory.export`,
`trajectory.export_status`, `trajectory.export_cancel`.
- **Bug reports.** `freeze_export` is split into staging and finishing
so a report can be prepared from selected turns; the CLI's behaviour is
unchanged. A Report Session holds a web report in the gateway between
page requests: at most two at once, all blocking I/O on worker threads,
a cancel checked after every step (a report cancelled while packaging is
never sent), a failed or uploaded report releases its slot and keeps its
last status for ten minutes, and an idle one is cancelled after thirty
minutes. Seven `bugreport.*` RPCs, the `bugreport-v1` capability
(announced only while the receiver URL is valid), and error codes -32026
/ -32027 / -32028. Every decision made on the web page, defaults and
bulk buttons included, is remembered in the redaction history with `web`
as its source, unlike the CLI's bulk flags.
- **Web page.** A three-dot conversation menu in the header's corner
(the desk launcher moves one slot in) opens the new `export` wizard.
Export: pick turns, name the file under an "unredacted" warning, follow
the job, download once automatically with a link to download again.
Report a problem: pick turns, describe the problem, decide the uncertain
values in two columns (hits highlighted at code points), upload, and get
a receipt. Closing before the upload starts discards the report being
prepared; once the upload starts the report is saved on the machine, a
failure marked retryable offers a retry, and the saved report can be
sent later from the command line by its id. Every answer is checked
against the wizard's epoch and the job or report it was asked about, and
a write step whose answer is lost is settled from the report's status
instead of being sent again. Layouts were measured at 360 and 390 px in
English and Chinese.
- **Docs and tests.** The webui and trajectory-debugging pages (English
and Chinese) describe both flows. Two integration tests drive them end
to end: the export through the real download route, and the bug report
against a real receiver process, including a stopped receiver, a retry,
the redaction history's replay and a cancel at each step.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- The branch merges the current `main` (one merge commit, no conflicts);
everything below ran on the merged tree.
- `uv run pytest tests/ -q` (full suite): 57 failed / 31491 passed / 177
skipped. Every one of the 57 also fails on the current `main` on this
machine (environment-specific, pre-existing); none is introduced by this
branch.
- `uv run pytest -m integration
tests/integration/test_web_traj_export_e2e.py
tests/integration/test_web_bugreport_e2e.py
tests/integration/test_bug_report_upload_e2e.py -q`: 10 passed (three
consecutive runs of the two new files).
- `uv run ruff check`, `uv run ruff format --check`, `uv run ty check`
on the changed Python: pass.
- ui-web: `npm run gen:check` (239 methods), `npm run lint`, `npm run
type-check`, `npx vitest run` (246 files / 3868 passed), `npm run build`
and `python3 ui-web/build.py` (both boot snapshots match),
`check-page.mjs`, `check-css.mjs`, `check-class-namespace.mjs`: pass.
- ui-tui: `npm run lint:rpc`, `npm run lint:i18n`: pass.
- `uv run --group docs mkdocs build -f docs-site/mkdocs.yml --strict`:
pass.
- `make check-source-language check-large-files check-commits`: pass.
- Headless Chromium on the built page (offline fixtures): both flows
walked to the download and to the receipt; the header corner measured in
five layouts (desk panel open and closed, workspace panel open, shell
mode, narrow split) and the wizard at 360 and 390 px in both languages.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

- A process started with the old `--dev` flag that runs `system.upgrade`
relaunches itself with `--dev`, which the new version refuses; the user
runs `raven web` once by hand. This was accepted as the cost of dropping
the flag.
- The trajectory view is on by default: each open page calls the light,
in-memory `trajectory.state` every five seconds, and the index is built
only when the view is opened. No switch to turn it off is added here;
the policy keeps a replaceable source for a settings entry later.
- The export is the raw trajectory and may contain secrets. The page
says so in a warning; the download is served only on the same origin,
behind the session guard, by an unguessable one-time token, and no path
ever reaches the page.
- Rollback: revert the squash commit. Nothing is migrated; the
`trace_id` field already written into session records is ignored by
older versions.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-10-14 21:34:46 +08:00

459 lines
18 KiB
Python

"""The Trajectory Export package: its layout and manifest are the contract web-traj-import reads.
Every assertion on a constant, a manifest key set or a member rule below pins
the format at EXPORT_FORMAT_VERSION 1. Changing any of them is a format change:
bump EXPORT_FORMAT_VERSION and update the importer in the same change.
"""
from __future__ import annotations
import hashlib
import io
import json
import os
import tarfile
import time
from pathlib import Path
from typing import Any
import pytest
from raven.tracing import artifact_v2
from raven.trajectory import export as ex
from raven.trajectory import selection as sel
SESSION = "tui:s1"
EXPORT_ID = "ex-20261001-abc123"
@pytest.fixture
def state(tmp_path: Path, monkeypatch) -> Path:
root = tmp_path / "traces"
(root / "logs" / "audit-artifacts").mkdir(parents=True)
monkeypatch.setenv("RAVEN_TRACING_DIR", str(root))
return root
def _artifact(state: Path, kind: str, name: str, payload: Any) -> Path:
path = state / "logs" / "audit-artifacts" / kind / "2026-10-01" / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(payload if isinstance(payload, str) else json.dumps(payload), encoding="utf-8")
return path
def _blob(state: Path, message: dict[str, Any]) -> str:
sha1 = artifact_v2.message_sha1(message)
path = artifact_v2.message_path(state / "logs" / "audit-artifacts", sha1)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(artifact_v2.message_text(message), encoding="utf-8")
return sha1
def _span(trace: str, span: str, name: str, *, parent: str | None = None, minute: int = 1, **attrs: Any):
record: dict[str, Any] = {
"traceId": trace,
"spanId": span,
"name": name,
"startTime": f"2026-10-01T10:{minute:02d}:00+00:00",
"endTime": f"2026-10-01T10:{minute:02d}:30+00:00",
"attributes": {"session.key": SESSION, **attrs},
}
if parent:
record["parentSpanId"] = parent
return record
@pytest.fixture
def world(state: Path, tmp_path: Path) -> dict[str, Any]:
"""Two turns, a sub-agent of the first, a v2 shell with one blob missing, and a stray pointer."""
kept = _blob(state, {"role": "user", "content": "plan a trip"})
prompt = _blob(state, {"role": "system", "content": "you are raven"})
gone = artifact_v2.message_sha1({"role": "assistant", "content": "this blob was purged"})
shell = _artifact(
state,
"llm.input",
"a-llm.json",
{
"artifactFormat": artifact_v2.ARTIFACT_FORMAT,
"messages": [artifact_v2.make_ref(kept), artifact_v2.make_ref(gone)],
"systemPrompt": artifact_v2.make_ref(prompt),
},
)
turn_input = _artifact(state, "turn.input", "a-in.json", {"content": "plan a trip"})
secret = _artifact(state, "tool.output", "a-tool.txt", "api_key=sk-live-not-redacted-0123456789")
sub_out = _artifact(state, "turn.output", "sub-out.json", {"content": "found 3 hotels"})
outside = tmp_path / "elsewhere.json"
outside.write_text("{}", encoding="utf-8")
other_turn_input = _artifact(state, "turn.input", "b-in.json", {"content": "not selected"})
records = [
_span(
"trace-a",
"span-a",
"session.turn",
minute=1,
**{
"turn.input_preview": "plan a trip",
"turn.in_progress": False,
"turn.input.artifact_path": str(turn_input),
},
),
_span("trace-a", "span-a1", "llm.call", parent="span-a", **{"llm.input.artifact_path": str(shell)}),
_span(
"trace-a",
"span-a2",
"tool.call",
parent="span-a",
**{
"tool.output.artifact_path": str(secret),
"tool.input.artifact_path": str(outside),
},
),
_span(
"trace-sub",
"span-s",
"session.turn",
minute=2,
**{
"session.key": "subagent:x",
"trace.dispatched_in_trace_id": "trace-a",
"turn.output.artifact_path": str(sub_out),
},
),
_span(
"trace-b",
"span-b",
"session.turn",
minute=5,
**{
"turn.input_preview": "not selected",
"turn.input.artifact_path": str(other_turn_input),
},
),
]
log = state / "logs" / "audit-spans.log"
log.write_text("".join(json.dumps(r) + "\n" for r in records), encoding="utf-8")
session = tmp_path / "s1.jsonl"
rows = [
{"_type": "metadata", "key": SESSION, "metadata": {"title": "Trip \u00e9t\u00e9"}, "last_consolidated": 2},
{"role": "user", "content": "plan a trip", "timestamp": "2026-10-01T10:01:00", "trace_id": "trace-a"},
{"role": "assistant", "content": "sure", "timestamp": "2026-10-01T10:01:20"},
{"role": "user", "content": "not selected", "timestamp": "2026-10-01T10:05:00", "trace_id": "trace-b"},
]
session.write_text("".join(json.dumps(r) + "\n" for r in rows), encoding="utf-8")
return {
"shell": shell,
"secret": secret,
"turn_input": turn_input,
"sub_out": sub_out,
"outside": outside,
"kept": kept,
"prompt": prompt,
"gone": gone,
"session": session,
}
def _export(state: Path, world: dict[str, Any], tmp_path: Path, **kwargs: Any) -> tuple[Path, ex.ExportSummary]:
plan = sel.plan_selection(SESSION, ["trace-a"], answering=False)
selection = sel.collect_selection(plan)
cut = sel.cut_session_record(world["session"], plan)
out = tmp_path / "out" / "export.tar.gz"
out.parent.mkdir()
return out, ex.write_export(selection, cut, out, export_id=EXPORT_ID, **kwargs)
def _members(path: Path) -> dict[str, tarfile.TarInfo]:
with tarfile.open(path, "r:gz") as tar:
return {member.name: member for member in tar.getmembers()}
def _read(path: Path, name: str) -> bytes:
with tarfile.open(path, "r:gz") as tar:
handle = tar.extractfile(f"{EXPORT_ID}/{name}")
assert handle is not None
return handle.read()
# -- the contract ------------------------------------------------------------
def test_the_format_constants_are_pinned():
assert ex.EXPORT_SCHEMA == "raven_trajectory_export"
assert ex.EXPORT_FORMAT_VERSION == 1
assert (ex.MANIFEST_FILE, ex.SPANS_FILE, ex.SESSION_FILE, ex.ARTIFACTS_DIR, ex.MESSAGES_DIR) == (
"export.json",
"spans.jsonl",
"session.jsonl",
"artifacts",
"messages",
)
assert ex.MANIFEST_KEYS == (
"schema",
"format_version",
"export_id",
"created_at",
"raven_version",
"source",
"turns",
"traces",
"counts",
"missing_artifacts",
"missing_messages",
"transcript_unmatched",
)
assert ex.EXPORT_ID.fullmatch(ex.new_export_id())
def test_the_manifest_has_exactly_the_contract_keys(state, world, tmp_path):
out, _ = _export(state, world, tmp_path)
manifest = json.loads(_read(out, ex.MANIFEST_FILE))
assert tuple(manifest) == ex.MANIFEST_KEYS
assert set(manifest["source"]) == {"session_key", "title", "transcript_included"}
assert set(manifest["traces"]) == {"selected", "dispatched"}
assert set(manifest["counts"]) == {"spans", "artifacts", "messages"}
assert all(set(turn) == {"number", "trace_id", "start", "input_preview"} for turn in manifest["turns"])
assert manifest["schema"] == ex.EXPORT_SCHEMA and manifest["format_version"] == 1
assert manifest["export_id"] == EXPORT_ID
assert manifest["created_at"].endswith("+00:00")
def test_members_are_plain_files_under_the_export_id(state, world, tmp_path):
out, _ = _export(state, world, tmp_path)
members = _members(out)
for name, info in members.items():
assert info.isreg(), name
assert name.startswith(f"{EXPORT_ID}/") and ".." not in name.split("/") and not name.startswith("/")
assert (info.uid, info.gid, info.uname, info.gname, info.mode) == (0, 0, "", "", 0o644)
assert set(members) == {
f"{EXPORT_ID}/{name}"
for name in (
"export.json",
"spans.jsonl",
"session.jsonl",
"artifacts/turn.input/2026-10-01/a-in.json",
"artifacts/llm.input/2026-10-01/a-llm.json",
"artifacts/tool.output/2026-10-01/a-tool.txt",
"artifacts/turn.output/2026-10-01/sub-out.json",
f"messages/{world['kept'][:2]}/{world['kept']}.json",
f"messages/{world['prompt'][:2]}/{world['prompt']}.json",
)
}
# -- the content -------------------------------------------------------------
def test_artifacts_travel_byte_for_byte_unredacted_and_unresolved(state, world, tmp_path):
out, _ = _export(state, world, tmp_path)
assert _read(out, "artifacts/tool.output/2026-10-01/a-tool.txt") == world["secret"].read_bytes()
shell = json.loads(_read(out, "artifacts/llm.input/2026-10-01/a-llm.json"))
assert artifact_v2.is_v2(shell), "a v2 shell stays a shell"
blob = _read(out, f"messages/{world['kept'][:2]}/{world['kept']}.json")
assert hashlib.sha1(blob).hexdigest() == world["kept"]
def test_spans_are_rewritten_only_where_the_artifact_was_exported(state, world, tmp_path):
out, summary = _export(state, world, tmp_path)
spans = [json.loads(line) for line in _read(out, ex.SPANS_FILE).decode().splitlines()]
assert [(s["traceId"], s["spanId"]) for s in spans] == [
("trace-a", "span-a"),
("trace-a", "span-a1"),
("trace-a", "span-a2"),
("trace-sub", "span-s"),
]
tool = spans[2]["attributes"]
assert tool["tool.output.artifact_path"] == "artifacts/tool.output/2026-10-01/a-tool.txt"
assert tool["tool.input.artifact_path"] == str(world["outside"])
assert summary.missing_artifacts == (str(world["outside"]),)
def test_the_manifest_says_what_was_exported_and_what_is_missing(state, world, tmp_path):
out, summary = _export(state, world, tmp_path)
manifest = json.loads(_read(out, ex.MANIFEST_FILE))
assert manifest["source"] == {"session_key": SESSION, "title": "Trip \u00e9t\u00e9", "transcript_included": True}
assert manifest["turns"] == [
{"number": 1, "trace_id": "trace-a", "start": "2026-10-01T10:01:00+00:00", "input_preview": "plan a trip"}
]
assert manifest["traces"] == {"selected": ["trace-a"], "dispatched": ["trace-sub"]}
assert manifest["counts"] == {"spans": 4, "artifacts": 4, "messages": 2}
assert manifest["missing_artifacts"] == [str(world["outside"])]
assert manifest["missing_messages"] == [world["gone"]]
assert manifest["transcript_unmatched"] == []
assert summary.counts == manifest["counts"] and summary.size_bytes == out.stat().st_size
def test_the_session_record_holds_only_the_selected_turns(state, world, tmp_path):
out, _ = _export(state, world, tmp_path)
rows = [json.loads(line) for line in _read(out, ex.SESSION_FILE).decode().splitlines()]
assert rows[0]["_type"] == "metadata" and rows[0]["last_consolidated"] == 0
assert [r["content"] for r in rows[1:]] == ["plan a trip", "sure"]
def test_a_missing_session_record_is_left_out_and_said_so(state, world, tmp_path):
world["session"].unlink()
out, summary = _export(state, world, tmp_path)
assert f"{EXPORT_ID}/{ex.SESSION_FILE}" not in _members(out)
manifest = json.loads(_read(out, ex.MANIFEST_FILE))
assert manifest["source"]["transcript_included"] is False and manifest["source"]["title"] == ""
assert manifest["transcript_unmatched"] == ["trace-a"] and summary.transcript_unmatched == ("trace-a",)
def test_a_hard_linked_artifact_is_packed_as_a_plain_file(state, world, tmp_path):
blob = state / "logs" / "audit-artifacts" / "_blobs" / "ab" / "abcdef.txt"
blob.parent.mkdir(parents=True)
os.replace(world["secret"], blob)
os.link(blob, world["secret"])
out, _ = _export(state, world, tmp_path)
info = _members(out)[f"{EXPORT_ID}/artifacts/tool.output/2026-10-01/a-tool.txt"]
assert info.isreg() and not info.islnk()
assert not any("_blobs" in name for name in _members(out))
def test_a_file_that_vanishes_while_packing_is_reported_missing(state, world, tmp_path, monkeypatch):
real = ex._read
def flaky(path: Path, root: Path):
return None if path.name == "a-tool.txt" else real(path, root)
monkeypatch.setattr(ex, "_read", flaky)
out, summary = _export(state, world, tmp_path)
assert str(world["secret"]) in summary.missing_artifacts
spans = [json.loads(line) for line in _read(out, ex.SPANS_FILE).decode().splitlines()]
assert spans[2]["attributes"]["tool.output.artifact_path"] == str(world["secret"])
with tarfile.open(out, "r:gz") as tar:
tar.extractall(tmp_path / "unpacked", filter="data")
def test_every_pointer_to_a_file_that_fails_to_read_keeps_its_original_value(state, world, tmp_path, monkeypatch):
log = state / "logs" / "audit-spans.log"
extra = _span(
"trace-a", "span-a3", "tool.call", parent="span-a", **{"tool.result.artifact_path": str(world["secret"])}
)
log.write_text(log.read_text(encoding="utf-8") + json.dumps(extra) + "\n", encoding="utf-8")
real = ex._read
monkeypatch.setattr(ex, "_read", lambda path, root: None if path.name == "a-tool.txt" else real(path, root))
out, summary = _export(state, world, tmp_path)
members = _members(out)
assert f"{EXPORT_ID}/artifacts/tool.output/2026-10-01/a-tool.txt" not in members
spans = {s["spanId"]: s for s in map(json.loads, _read(out, ex.SPANS_FILE).decode().splitlines())}
assert spans["span-a2"]["attributes"]["tool.output.artifact_path"] == str(world["secret"])
assert spans["span-a3"]["attributes"]["tool.result.artifact_path"] == str(world["secret"])
assert summary.missing_artifacts.count(str(world["secret"])) == 1
manifest = json.loads(_read(out, ex.MANIFEST_FILE))
assert manifest["counts"]["artifacts"] == 3
assert str(world["secret"]) in manifest["missing_artifacts"]
def _blob_path(state: Path, sha1: str) -> Path:
return artifact_v2.message_path(state / "logs" / "audit-artifacts", sha1)
def test_a_blob_linked_out_of_the_store_is_not_carried(state, world, tmp_path):
outside = tmp_path / "outside-secret.json"
outside.write_text('{"role": "user", "content": "from outside the store"}', encoding="utf-8")
blob = _blob_path(state, world["kept"])
blob.unlink()
blob.symlink_to(outside)
out, summary = _export(state, world, tmp_path)
assert f"{EXPORT_ID}/messages/{world['kept'][:2]}/{world['kept']}.json" not in _members(out)
assert world["kept"] in summary.missing_messages
def test_a_blob_directory_linked_out_of_the_store_is_not_carried(state, world, tmp_path):
shard = _blob_path(state, world["prompt"]).parent
elsewhere = tmp_path / "elsewhere-shard"
elsewhere.mkdir()
for entry in shard.iterdir():
entry.rename(elsewhere / entry.name)
shard.rmdir()
shard.symlink_to(elsewhere, target_is_directory=True)
out, summary = _export(state, world, tmp_path)
assert world["prompt"] in summary.missing_messages
assert not any(name.endswith(f"{world['prompt']}.json") for name in _members(out))
def test_a_blob_that_is_a_directory_is_missing(state, world, tmp_path):
blob = _blob_path(state, world["kept"])
blob.unlink()
blob.mkdir()
_, summary = _export(state, world, tmp_path)
assert world["kept"] in summary.missing_messages
@pytest.mark.skipif(not hasattr(os, "mkfifo"), reason="needs FIFOs")
def test_a_fifo_never_blocks_the_export(state, world, tmp_path, monkeypatch):
"""A FIFO in the blob slot, and one swapped in after the artifact check:
both are refused by descriptor, so the export ends without a writer."""
blob = _blob_path(state, world["kept"])
blob.unlink()
os.mkfifo(blob)
world["secret"].unlink()
os.mkfifo(world["secret"])
real = ex.store_artifact
monkeypatch.setattr(
ex, "store_artifact", lambda value, root: Path(value) if value == str(world["secret"]) else real(value, root)
)
started = time.monotonic()
_, summary = _export(state, world, tmp_path)
assert time.monotonic() - started < 5
assert world["kept"] in summary.missing_messages
assert str(world["secret"]) in summary.missing_artifacts
def test_cancelling_leaves_nothing_behind(state, world, tmp_path):
calls = {"n": 0}
def stop_after_two() -> bool:
calls["n"] += 1
return calls["n"] >= 2
with pytest.raises(ex.ExportCancelledError):
_export(state, world, tmp_path, should_stop=stop_after_two)
assert list((tmp_path / "out").iterdir()) == []
def test_a_bad_export_id_is_refused(state, world, tmp_path):
plan = sel.plan_selection(SESSION, ["trace-a"], answering=False)
with pytest.raises(ValueError):
ex.write_export(
sel.collect_selection(plan),
sel.cut_session_record(world["session"], plan),
tmp_path / "x.tar.gz",
export_id="../escape",
)
# -- the download name -------------------------------------------------------
@pytest.mark.parametrize(
("raw", "clean"),
[
("trip.tar.gz", "trip.tar.gz"),
("trip", "trip.tar.gz"),
("../../etc/passwd", "passwd.tar.gz"),
("C:\\Users\\me\\trip.tar.gz", "trip.tar.gz"),
('a<b>c:d"e|f?g*h.tar.gz', "abcdefgh.tar.gz"),
("tab\there\n.tar.gz", "tabhere.tar.gz"),
("", "raven-export.tar.gz"),
(".tar.gz", "raven-export.tar.gz"),
(" ... ", "raven-export.tar.gz"),
("Trip \u00e9t\u00e9-20261001-1000", "Trip \u00e9t\u00e9-20261001-1000.tar.gz"),
],
)
def test_download_names_are_cleaned(raw, clean):
assert ex.clean_file_name(raw) == clean
def test_a_long_download_name_is_cut_before_the_suffix():
name = ex.clean_file_name("x" * 500 + ".tar.gz")
assert len(name) == 120 and name.endswith(".tar.gz")
def test_the_package_reads_back_with_the_standard_library(state, world, tmp_path):
out, _ = _export(state, world, tmp_path)
data = out.read_bytes()
with tarfile.open(fileobj=io.BytesIO(data), mode="r:gz") as tar:
tar.extractall(tmp_path / "unpacked", filter="data")
assert (tmp_path / "unpacked" / EXPORT_ID / ex.MANIFEST_FILE).is_file()