Raven-1/docs-site/docs/trajectory-debugging.md
userName20260323 8a1ebdc577
docs: organize bilingual documentation navigation (#635)
## Summary

Reorganize the Raven documentation landing page and expand the bilingual
documentation site.

- Add bilingual guidance for agent collaboration, ACP and A2A protocols,
DAG orchestration, channels, browser collaboration, knowledge bases,
Playbooks, proactive follow-ups, Oncall, permissions, tracing,
trajectory replay, and Evolver.
- Group daily usage under `Use Raven` and `Using Raven`.
- Group `Technical Reference` and `Developer Guide` under `Explore the
documentation` while preserving separate cards.

## Type

- [ ] Fix
- [ ] Feature
- [x] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- [x] Relevant tests pass locally
- [ ] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

Exact checks:

- `uv run --frozen --group docs mkdocs build -f docs-site/mkdocs.yml
--strict` - passed.
- `uv run --frozen --group docs pytest tests/test_docs_site.py
tests/test_living_docs.py
tests/integration/test_docs_site_controls_e2e.py -q -o addopts=''` - 77
passed.
- `uv run --frozen python scripts/check_commit_messages.py
origin/main..HEAD` - passed.
- `git diff --check origin/main...HEAD` - passed.
- `make check-large-files` - passed.

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

This is a documentation-only change. It does not change runtime
behavior, configuration semantics, or public APIs. Reverting the
documentation commits restores the previous site content and navigation.

## Related Issues

N/A

---------

Co-authored-by: userName20260323 <270284818+userName20260323@users.noreply.github.com>
Co-authored-by: Claude (claude) <noreply@anthropic.com>
2026-09-22 16:14:44 +08:00

7.0 KiB

Trajectory debugging and replay

Turn a real failure into inspectable evidence and, when useful, a deterministic regression test. Tracing records what ran; a trajectory groups the relevant attempt, model/tool artifacts, conversation context, and outcome labels. Use Tracing and Instrumentation API when adding instrumentation; this page covers the operator's replay workflow.

Capture the failing attempt

Tracing is enabled by default. RAVEN_TRACING or tracing.enabled can change that; missing historical artifacts cannot be reconstructed by turning tracing on after the failure. Use the same installation/trace store as the failing run, including any RAVEN_TRACING_DIR override.

raven trajectory list
raven trajectory save ATTEMPT_ID

Replace ATTEMPT_ID with a listed id. A trace id can resolve to its containing attempt, so saving it may collect more than one turn. For another agent's workspace, use save --workspace /path/to/agent/home to locate its session records.

save creates a self-contained bundle and pins the attempt against normal purging. It does not sanitize the bundle. The manifest reports missing artifacts/messages and whether session context was included. A bundle with missing evidence is not necessarily replayable.

Keep raw bundles local and outside git. They can contain source files, prompts, credentials, tool outputs, and private conversation history.

Produce a reviewed report

raven trajectory report ATTEMPT_ID --config /path/to/agent/config.json

Use the config the traced agent actually used to seed known-secret redaction. The command repacks current evidence, redacts a copy, and creates a local shareable tarball after its review/confirmation flow. The original raw bundle is retained. A successful automatic scan does not guarantee every private business fact was removed.

For a structured bug package:

raven trajectory report-bug ATTEMPT_ID \
  --description "The tool result was omitted from the next model request" \
  --expected "The next request includes the tool result" \
  --actual "The next request has no tool result"

This creates a local record/package; it does not open a GitHub issue or upload the report. Review flagged items and the package before sharing. --yes is not sensitive-data risk consent. Do not routinely bypass findings with --accept-risk or declare every token harmless.

Replay the harness without repeating actions

raven trajectory replay ATTEMPT_ID --strict
raven trajectory replay ATTEMPT_ID --warn --json --out /tmp/raven-replay-report.json

An id here resolves an already saved bundle; alternatively pass its directory. Replay supplies recorded model replies and recorded tool results to the live harness in a temporary workspace. It does not dispatch real tool code or re-evaluate the task with a fresh model. Replaying a recorded deployment does not deploy again through its recorded tool calls.

Mode/result Meaning
--strict Halt at the first request/recording divergence
--warn, CLI default Record differences and continue feeding by order
Exit 0 Reached the end; warn mode can still contain divergences
Exit 1 Bad target or invocation
Exit 2 Replay halted at a strict divergence or ran out of recording

Read the divergence's call kind/index, field, expected value, and actual value. “No divergence” means this harness reproduced the compared recording, not that the original task was correct. After a fix, a specific divergence can be exactly the evidence you want.

Make a regression case

In a source checkout, scaffold from a saved bundle or reviewed report:

raven trajectory regression init /path/to/bundle --name tool_result_preserved

The command minimizes and redacts a cassette, checks residual findings, and publishes a draft under tests/trajectories/tool_result_preserved/ only after its gates pass. It refuses an existing case. Complete both files before considering the result ready:

  • case.yaml: issue, owner, protected contract, and when re-recording is justified; review any residual findings with an explicit rationale.
  • expect.yaml: whether faithful reproduction or a particular first divergence is expected, plus assertions on the live request.

For a fixture where the first model request now contains the corrected user message, this illustrative expectation checks the change:

mode: strict
divergence:
  kind: llm
  index: 0
  field: messages[1]
checks:
  - call: llm
    index: 0
    message: 1
    op: contains
    value: expected corrected text

Derive the index, field, and text from your replay report; do not paste this expectation unchanged into an unrelated case. To expect faithful reproduction, omit divergence and assert the behavior that must remain stable.

raven trajectory regression validate tests/trajectories/tool_result_preserved
raven trajectory regression validate --all
uv run pytest tests/test_trajectory_regressions.py -q

Static validation checks metadata, cassette completeness, residual review, and size. Pytest performs replay assertions. Validation success is not a substitute for actually running the case. Read tests/trajectories/README.md before contributing a cassette; raw reports and report assets do not belong in the repository.

Know what replay does not test

The current comparison checks model/streaming path, message roles and non-system contents, tool calls, and offered tool names. It deliberately does not compare system-message content or complete tool-schema bodies.

Recorded media are not re-fed, tool display/abort/block metadata is not fully reconstructed, and parallel interleavings are consumed sequentially. A changed order appears as divergence rather than an intelligent rematch. Missing pre-attempt history may also cause an early difference.

Use targeted unit tests for system-prompt assembly and real integration tests for provider behavior, browser interactions, concurrency, and external side effects. Recorded replies cannot prove that a new model would choose the same actions. Evolver runs fresh benchmark evaluations; replay diagnoses harness behavior against a recording.

Troubleshooting and retention

Symptom Check
Attempt not listed Correct trace root, tracing configuration, and retained logs
Missing session/artifacts Agent workspace and bundle manifest completeness
Replay stops immediately First divergence, missing input, or pre-attempt history
Warn mode exits successfully despite a bug Inspect divergence list and write assertions
Regression validation fails after init Fill draft metadata/expectations and review residuals
Trace storage keeps growing Pinned corpus and retained bundles, not just log rotation

raven trajectory unpin ATTEMPT_ID removes retention protection; it does not delete exported copies or redact existing bundles. Treat evidence retention, sharing, and deletion as separate decisions.

Implementation: raven/trajectory/replay.py, regression.py, cassette.py, redact.py, and raven/cli/trajectory_commands.py.