Raven-1/tests/test_agents_research_process_appendix.py
userName20260323 8f42a1ec03
feat: ship research reports in a reader layout, measured and link-checked (#890)
## Summary

raven-research reports were written for a grader: fixed `## Answer` /
`## Findings` / `## Limitations` headings, HTTP codes beside links, and
a
chat reply that drifted from the file saved with `write_file`. This
branch
ships them in a reader layout and closes the gaps that product runs on
DeepSeek showed between the brief, the reply and the saved file.

Reader layout (`finalShape.reportLayout = "reader"`, on for
raven-research):

- The report opens with the answer as a blockquote, with no label or
bold
  lead-in. Anything the brief asks the report to open with (a retrieval
date, the kinds of sources) follows in the blockquote. Delivery facts
the
brief asks back (path, counts) go on its last line, which the chat reply
  keeps and the saved file leaves out.
- The body sits under its own `##` headings, using the brief's section
  names word for word, numbering included, when it gives them. The last
  section names what could not be verified (`## Limitations` in English;
  the reply's language otherwise, with no English word in that heading).
The shape gate reads the last `##` section as the limits when no word it
  knows names them, since no word list covers every language, and when
  only the first heading has such a word with no body above it (a body
  heading on export limits is body, not limits). The audit reads it the
  same way and counts a body written as prose under the answer.
- Prose cites through markdown links, with no fetch status beside them.
A
brief that asks for link checks gets one table before the limits,
listing
  every cited URL as fetched, with its status, marked blocked where the
  site blocked the fetch.
- The layout is derived from the deep clause, so the evidence rules stay
  one text. The shape gate, its reminder, rewrite prompt and interrupted
  fallback, and `scripts/research_report_audit.py` read it as the same
  three parts. A test pins the audit's limits matcher to the gate's.

Chat and file are one text: at turn end the delivered reply is written
over
the report file the turn wrote. Which file that is:

- A markdown file the turn wrote whole, or appended to or edited when
the
turn wrote it whole first or it still holds exactly what an earlier turn
  of the session left in it. The session record keeps the sha256 the
session left in each report file, checked at turn start before any tool
  runs. Any other file (a log the user keeps, a report the user changed
  between turns) holds someone else's text, so it is never the report. A
later turn that only edits a report nobody else touched still syncs it.
- Whose `##` headings are at least half the reply's. On the
reader-layout
  product runs a report draft shared 75-100% of its headings with the
reply, and a notes file saved beside the report shared none, so a notes
file or a source index is never overwritten, alone or beside the report.
- Of several such files, the one the reply names; when it names none or
  more than one, every file is left alone.

The file keeps its `#` title when the reply has none. It is also left
alone
when the reply is not a report, or when the reply is under 80% of the
file's length, so a summary reply never replaces the report. The path is
read from the first line of the tool's result only, matched against the
tool's whole fixed wording, and the file tools now
refuse a path that carries a line break or other control character: they
echo the path into their results, so such a path could forge a second
result line naming a file outside the workspace.

Length and counts, measured rather than estimated:

- Every markdown write or edit gets a `[report length: ...]` line on its
  tool result, carrying:
  - CJK characters and other words, with prose and tables apart;
  - the same counts per `##` section;
  - the number of distinct URLs cited, extracted the way the research
    trail extracts them.
- The report checks a brief's length against that line, and takes any
  count it states (sources included) from it.
- `write_file` now reports bytes. Its success line said "N bytes" and
  counted characters, so a Chinese report was sized at about a third of
  its real length.

Fetch status and the research trail:

- The research fetch tool reports the target's own status from the Jina
reader's header instead of the reader's 200, and marks a CAPTCHA wall as
  `blocked`.
- The trail counts such pages as refused by the site, not as read, per
  fetch: a page refused once and read on a retry counts as read. When a
  thin page is re-read at another address, each address keeps its own
  status: the stub its own, each fallback the one in its own header.
- The trail drops click-tracking parameters (`utm_*`, `srsltid`,
`gclid`,
  `fbclid`, `msclkid`) before matching a citation.
- Its `<details>` folds are now plain labelled lists, since a terminal
  shows the tags raw.
- `scripts/research_report_audit.py` reads the refused-page clause and
  note. It also reads the reviewer verdict when the trail names the
  reviewing model. That second break predates this branch: a model id
  carries `/` and `-`, so the verdict read as missing on every product
  trail.

Other:

- The research reviewer now runs on DeepSeek in thinking mode. That
  provider refuses a named `tool_choice` with HTTP 400, so every review
  failed open and each report shipped unreviewed. The gate retries with
  `tool_choice=auto`.
- `measuredGuidance` is off in the shipped config. Its numbers were
  measured on a short-answer benchmark and pull a long report toward a
  terse answer.
- The appendix twin's guard names the twin's lead over the frozen dr@3.5
  record instead of requiring the two to be equal: 4 added and 5 changed
  definitions, with `ResearchTrail` compared member by member so its
  untouched members stay pinned. The record itself is unchanged from
  `main`.
- The product label moves to the `-reader` suffix. Its digest was
  re-stamped several times on this branch only; `main` never shipped an
  intermediate digest.

Checked and deliberately not changed here:

- The main agent's own `web_fetch` (`raven/agent/tools/web.py`) also
  reads through Jina and still reports the reader's 200 for a target
  error. It is a separate module, not a twin of the research tool, so it
  needs its own change.
- `fetch_result_ok` is unchanged, so a page the site refused still
releases the fetch gate, counts toward the evidence floor and enters the
  conversation memo as a source. Changing it would move when the gate
  releases.
- The report sync writes the file directly, not through `write_file`, so
it
  emits no file-change event and a client's diff panel does not show it.
raven-research runs with no write approval, so no approval is bypassed.
- Every CAPTCHA flag in 36 local fetches that carried a reader warning
was
  on a page of 862 characters or fewer. Target errors appear on pages of
  up to 18k characters, and each one checked was a real error page. No
  false positive was found.

## Type

- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

Run after rebasing onto the current `origin/main`. The range-diff
against
the pre-rebase branch is `=` for all 15 commits.

- `uv run pytest tests/test_agents_research_*.py
tests/test_research_*.py tests/test_filesystem_write_modes.py
tests/test_agent_tools_web_fetch.py tests/test_history_trimmer.py
tests/test_context_engine*.py -q`: 1070 passed.
- Full suite before the rebase, `uv run pytest tests -q`: 27745 passed,
  118 skipped, 12 failed. None of the 12 were introduced by this branch:
  - 11 fail identically in clean worktrees of `origin/main` and of the
    branch head. They are environment-bound: proxy redaction, NAT64
    addresses, the node path.
  - `test_agents_ppt_launcher` passes in the clean branch worktree and
    failed only because of an uncommitted local edit to
    `agents/raven-ppt/subagent.json`.
- Mutation checks: each of these, reverted alone, turns its new tests
red:
  - the target-status override;
  - the refused-page split;
  - the ledger classification;
  - choosing the named report file (3 tests fail with `paths[-1]`);
  - the audit's refused clause;
  - the audit's ` via ` terminator.
- After the second review round:
`uv run pytest tests/test_agents_research_*.py tests/test_research_*.py
tests/test_filesystem_write_modes.py tests/test_filesystem_tools.py
tests/test_deliver_files_tool.py tests/test_rpc_files.py
tests/test_rpc_turn_send.py tests/test_tool_registry_execute.py
tests/test_tools_workdir.py tests/test_agent_tools_web_fetch.py -q`:
  1238 passed. Each fix, reverted alone, turns its new test red: the
  control-character refusal, the first-line read, the append rule, the
  last-section limits (gate and audit apart), the fallback verdicts, and
  the URL count, the per-fetch refusal, the draft-heading filter and the
session-wide edit rule. The twin guard fails on an undeclared definition
and on a
  stale lead entry.
- After the third review round, on the rebased head:
`uv run pytest tests/test_agents_research_*.py tests/test_research_*.py
tests/test_filesystem_write_modes.py tests/test_filesystem_tools.py
tests/test_deliver_files_tool.py tests/test_rpc_files.py
tests/test_rpc_turn_send.py tests/test_tool_registry_execute.py
tests/test_tools_workdir.py tests/test_agent_tools_web_fetch.py
tests/test_history_trimmer.py -q`:
  1278 passed. Mutants each turn the new tests red: the unanchored
unchanged-file match; the control-character guard narrowed to LF, or to
  `Cc` only; each of the four success lines reworded in the real tools;
  a twin-only change to an untouched `ResearchTrail` default.
- After the fourth review round, same command: 1288 passed. Mutants each
  turn the new tests red: held files taken without the digest check, the
  digest never recorded, held files ignored, the first-heading rule
  reverted in the gate and in the audit apart, the gate's body-above
  condition dropped, and the audit's prose-body count dropped.
- `make lint-python`, `make lint-imports`, `make lint-deps`,
  `make lint-types`: all pass.
- `scripts/check_commit_messages.py origin/main..HEAD`,
  `scripts/check_source_language.py origin/main...HEAD`,
  `make check-large-files`: all exit 0.
- Product runs through the TUI host, on reports about personal agents,
AI
  glasses, AI coding agents, office agents and edge AI:
  - The saved file equals the reply, less the delivery line, with its
    title kept.
  - The stated length and source count match the `[report length]` line.
  - The link-check table has one row per counted URL.
- A Gartner 403 and an HBS 405 behind a CAPTCHA came through with their
    real status.

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

## Risk

User-visible changes:

- raven-research replies and saved reports change form: an answer
blockquote, no fixed headings, no HTTP codes beside links, and a single
  link-check table.
- The report file a research turn wrote is overwritten with the
delivered
  reply at turn end, except in the cases above.
- `read_file`, `write_file`, `edit_file`, `list_dir`, `deliver_files`
and
the RPC file reads refuse a path containing a line break or other
control
  character (they share `resolve_path`).

Existing installs keep the contract they seeded into
`agent_memory/profile/agent.md`. The new layout reaches them only after
that file is deleted; until then the gate still accepts the old
three-heading replies. The changelog says so.

Not covered by this branch:

- The research subagent pins `contextWindowTokens` to 65536. On a host
  model with a 1M window, the budget note reaches 80-136% early and cuts
  research short. Left as is on purpose; it is a separate decision.
- Post-delivery behaviour of the host agent is unchanged. It may recount
  or edit the report after delivery.

Rollback: revert the squash commit. To keep the code but drop the
layout,
set `finalShape.reportLayout` to `"sections"` in
`agents/raven-research/config.json`.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

N/A

---------

Co-authored-by: userName20260323 <270284818+userName20260323@users.noreply.github.com>
Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
2026-10-12 15:41:52 +08:00

1404 lines
67 KiB
Python

"""dr@2.8 research trail: computed from the record, never asked of the model.
The appendix exists because the alternative - a contract clause asking the model
to describe its own search strategy - produces an unverifiable self-report. So
every test here is about the properties that make it *not* a self-report: it can
only say what the ledger says, it says nothing when the ledger is absent rather
than rendering zeroes, and it never reaches the model.
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
import pytest
REPO = Path(__file__).resolve().parent.parent
PLUGIN_DIR = REPO / "agents" / "raven-research" / "plugins" / "research-flow"
sys.path.insert(0, str(PLUGIN_DIR))
from research_flow.support.process_appendix import build_appendix, build_trail # noqa: E402
LEDGER = [
{"op": "search", "query": "arwa university founding", "replay": False, "zero_hit": False},
{"op": "search", "query": "Arwa University founding", "replay": True, "zero_hit": False},
{"op": "search", "query": "arwa charter 1974", "replay": False, "zero_hit": True},
{"op": "fetch", "url": "https://a.example/one", "chars": 12431, "ok": True},
{"op": "fetch", "url": "https://b.example/two", "chars": 0, "ok": False},
{"op": "verify", "outcome": "reject", "unsupported_claims": ["the 1974 charter date"]},
{"op": "verify", "outcome": "pass", "unsupported_claims": []},
]
def test_the_trail_reports_only_what_the_ledger_recorded():
t = build_trail(LEDGER, "see https://a.example/one")
assert t.searches == 3
assert t.replays == 1
assert t.zero_hit == 1
# Whitespace and case folded: the same query typed twice is one query, and a
# trail that called it two would overstate the breadth of the research.
assert t.distinct_queries == ["arwa university founding", "arwa charter 1974"]
assert t.counters()["pages_ok"] == 1 and t.counters()["pages_opened"] == 2
# Last verdict wins - the turn ended on a pass, and its open points went with it.
assert t.verify_outcome == "pass"
assert t.unsupported == []
def test_a_cited_url_that_was_never_opened_is_named_as_such():
"""The one line here that accuses the answer, and the only report-quality
number in this build that needs no judge and does not reward length."""
t = build_trail(LEDGER, "per https://a.example/one and https://never.example/x it holds")
assert t.cited_not_opened == ["https://never.example/x"]
assert t.counters()["citation_grounding_rate"] == 0.5
# No search row in LEDGER carries a ``urls`` list, so nothing was ever on
# screen: this is the fabricated-citation branch.
assert t.cited_never_surfaced == ["https://never.example/x"]
assert "appear nowhere in this run" in t.render()
assert "https://never.example/x" in t.render()
def test_the_warning_separates_a_fabricated_link_from_one_merely_left_unopened():
"""Both branches, and the number that must not move between them.
"Cited a link the search listed but never opened" and "cited a link that
appears nowhere in this run" were one sentence in one wording. The first
breaks the contract's read-before-you-cite rule and the reader can still go
read the link; the second is a citation with nothing behind it. Collapsing
them made the appendix's only accusation unactionable.
``cited_not_opened`` and ``citation_grounding_rate`` are asserted unchanged
across both branches on purpose: they have readings on disk, and the split
is additive by construction (``cited_never_surfaced`` is a strict subset).
"""
ledger = [
{
"op": "search",
"query": "q",
"replay": False,
"zero_hit": False,
"urls": ["https://a.example/one", "https://listed.example/two"],
},
{"op": "fetch", "url": "https://a.example/one", "chars": 900, "ok": True},
]
answer = "opened https://a.example/one , listed https://listed.example/two , invented https://ghost.example/three"
t = build_trail(ledger, answer)
assert t.cited_not_opened == ["https://listed.example/two", "https://ghost.example/three"]
assert t.cited_never_surfaced == ["https://ghost.example/three"]
assert t.counters()["citation_grounding_rate"] == round(1 - 2 / 3, 4)
assert t.counters()["cited_never_surfaced"] == 1
rendered = t.render()
assert "appear nowhere in this run" in rendered
assert "https://ghost.example/three" in rendered
assert "returned by a search but never opened" in rendered
assert "https://listed.example/two" in rendered
clean = build_trail(ledger, "only https://a.example/one")
assert clean.cited_not_opened == [] and clean.cited_never_surfaced == []
assert clean.counters()["citation_grounding_rate"] == 1.0
assert "\u26a0" not in clean.render()
def test_a_truncated_citation_of_a_listed_link_is_unopened_not_fabricated():
"""The same recovery the opened check makes, applied to the surfaced set.
A model citing a long listed path truncates it the way it truncates opened
ones (``_match_form``'s unique-prefix rule). Bare set membership called that
"appear nowhere in this run" - a fabrication charge - for a link the reader
can go and find in the search results. Unopened, yes; invented, no.
"""
listed = "https://listed.example/wiki/a-long-article-path/section-three"
ledger = [{"op": "search", "query": "q", "replay": False, "zero_hit": False, "urls": [listed]}]
t = build_trail(ledger, "see https://listed.example/wiki/a-long-article-path/sec and that is all")
assert t.cited_not_opened == ["https://listed.example/wiki/a-long-article-path/sec"]
assert t.cited_never_surfaced == []
rendered = t.render()
assert "returned by a search but never opened" in rendered
assert "appear nowhere in this run" not in rendered
def test_trailing_punctuation_does_not_manufacture_a_fabricated_citation():
"""A URL ending a sentence is the common case; treating the period as part of
it would report a correctly-read page as invented - a false alarm on the one
assertion here that calls the answer wrong."""
t = build_trail(LEDGER, "It is stated at https://a.example/one.")
assert t.cited_not_opened == []
assert t.counters()["citation_grounding_rate"] == 1.0
def test_canonicalisation_is_folded_but_a_deeper_path_is_still_fabricated():
"""The two directions this check can fail, in one test.
Folding too little raises a false alarm on a page that really was read; folding
too much - the first implementation matched on prefix - absolves any deep link
invented on top of a page that was read, which is the most plausible
fabrication there is and the one this check exists to catch.
"""
t = build_trail(
LEDGER,
"as shown at HTTP://A.example/one/ and in https://a.example/one/appendix-c",
)
# Asserted present first. Without this line the check below passes whenever the
# URL is not extracted at all - which is exactly what happened: the extractor was
# case-sensitive, ``HTTP://`` never became a citation, and "it was not flagged"
# was true for the wrong reason.
assert "HTTP://A.example/one/" in t.cited, "uppercase scheme was never extracted"
assert "https://a.example/one/appendix-c" in t.cited_not_opened, "deep link absolved"
assert "HTTP://A.example/one/" not in t.cited_not_opened, (
"scheme/case/trailing-slash difference reported as a fabricated citation"
)
def test_an_answer_citing_nothing_scores_null_not_perfect():
t = build_trail(LEDGER, "The founding year is 1974.")
c = t.counters()
assert c["urls_cited"] == 0
# Not 1.0. An answer with no sources is unmeasured, not perfectly grounded, and
# a rate of 1.0 here would put it at the top of any ranking built on this field.
assert c["citation_grounding_rate"] is None
def test_no_ledger_yields_no_appendix_and_says_why(tmp_path):
"""Missing data must not render as zero activity.
"0 searches, 0 pages read" printed under an answer that did plenty of both is
the shape of every missing-data-read-as-zero defect this project has logged.
"""
text, counters = build_appendix(None, "answer")
assert text == ""
assert counters == {"emitted": False, "reason": "ledger_not_configured"}
missing = tmp_path / "nope.jsonl"
text, counters = build_appendix(str(missing), "answer")
assert text == "" and counters["reason"] == "ledger_empty"
empty_research = tmp_path / "l.jsonl"
empty_research.write_text(json.dumps({"op": "verify", "outcome": "pass"}) + "\n")
text, counters = build_appendix(str(empty_research), "answer")
assert text == "" and counters["reason"] == "no_research_events"
def test_a_torn_last_line_does_not_lose_the_whole_trail(tmp_path):
p = tmp_path / "l.jsonl"
p.write_text("\n".join(json.dumps(r) for r in LEDGER) + '\n{"op": "sear')
text, counters = build_appendix(str(p), "https://a.example/one")
assert counters["emitted"] is True
assert counters["searches"] == 3
assert "Research trail" in text
def test_the_rendered_trail_leads_with_the_summary_and_folds_the_detail():
"""Readability is the product requirement: answer first, detail collapsible."""
text = build_trail(LEDGER, "https://a.example/one").render()
lines = [line for line in text.splitlines() if line.strip()]
assert lines[0] == "---"
assert lines[1].startswith("**Research trail**")
# "distinct" reworded: the folding rule is unchanged - ``distinct_queries``
# still counts two - only the noun stops claiming two different things were
# looked for.
assert "3 searches (2 unique query strings), 1 pages read" in lines[1]
assert "**Queries run**" in lines and "**Pages read**" in lines # no open points on a pass
assert "**Reviewer's open points**" not in text
assert "<details>" not in text
assert text.index("**Queries run**") > text.index("**Research trail**")
def test_the_appendix_is_gated_on_the_assembly_so_the_anchor_cannot_reach_it():
from research_flow.config import FlowConfig
on = FlowConfig(enabled=True)
assert on.final_shape.process_appendix is True
off = FlowConfig(enabled=True, final_shape={"process_appendix": False})
assert off.final_shape.process_appendix is False
def test_the_appendix_never_reaches_the_model():
"""Stated as a source-level assertion because it cannot be observed from output.
The seam appends to ``final_content`` after ``messages`` already holds the raw
answer, so the persisted trajectory and the next turn's history are unchanged.
If that ordering is ever inverted, the model starts reading our commentary on
its own work as if it had written it, and the ledger stops being an instrument
that cannot steer what it measures.
"""
import inspect
from research_flow import flow
src = inspect.getsource(flow.TurnFrame)
# Anchored on the render call, not on the first mention of the knob: dr@2.9 added an
# earlier reference (the turn-scoped ledger is opened at the top of the loop), and a
# "first occurrence" anchor silently retargets to whichever line mentions it soonest.
seam = src.index("appendix, trail = build_appendix")
tail = src[seam : seam + 1200]
assert 'final_content = final_content.rstrip() + "\\n" + appendix' in tail
assert "add_assistant_message" not in tail, (
"the appendix must not be persisted as a message - that is the feedback path"
)
def test_dead_end_fetches_are_counted_but_not_listed():
"""A research turn spends fetches finding the right URL; they are not "pages read".
Measured on the first two product runs: one answer's appendix reached 26% of the
output because every guessed path that returned a 200-with-nothing got its own
line. They still get counted - "23 pages read" over 9 real ones stops being an
audit and becomes decoration - they just do not each get a URL.
"""
rows = [
{"op": "search", "query": "q", "replay": False},
{"op": "fetch", "url": "https://good.example/real", "chars": 9000, "ok": True},
{"op": "fetch", "url": "https://guess.example/wrong", "chars": 199, "ok": True},
{"op": "fetch", "url": "https://guess.example/also-wrong", "chars": 203, "ok": True},
{"op": "fetch", "url": "https://dead.example/x", "chars": 0, "ok": False},
]
text = build_trail(rows, "https://good.example/real").render()
assert "https://good.example/real" in text
assert "guess.example" not in text, "a 199-char stub is not a page that was read"
assert "2 fetch(es) returned almost nothing" in text
assert "1 page(s) could not be retrieved" in text
# --------------------------------------------------------------------------- #
# dr@3.0: the check's scope once a conversation can have more than one turn #
# --------------------------------------------------------------------------- #
def test_a_page_an_earlier_turn_opened_is_not_a_fabricated_citation():
"""The defect this parameter exists to fix, reproduced from the run that found it.
On the first three-turn conversation, turn three answered a follow-up, cited two
URLs the research memo had listed - both opened in turn one - and the appendix
reported both as never opened. The numerator had become per-conversation the
moment the memo started handing later turns their own earlier sources, while the
denominator was still per-turn. The direction is fixed and the trigger is the
well-behaved case: an answer that credits where a fact came from.
"""
answer = "per https://a.example/one and https://earlier.example/p2 it holds"
without = build_trail(LEDGER, answer)
assert without.cited_not_opened == ["https://earlier.example/p2"]
t = build_trail(LEDGER, answer, ["https://earlier.example/p2"])
assert t.cited_not_opened == []
assert t.counters()["citation_grounding_rate"] == 1.0
assert t.counters()["opened_earlier"] == 1
assert "never opened" not in t.render()
def test_the_rendered_trail_still_lists_only_what_this_turn_opened():
"""Widening the check must not let a turn claim credit for earlier reading.
``pages`` drives the "N pages read" headline, and a turn that read one page while
the conversation had read five must still say one - otherwise every later turn of
a long conversation looks like a deeper piece of research than it was.
"""
t = build_trail(LEDGER, "https://earlier.example/p2", ["https://earlier.example/p2"])
assert len(t.pages) == len(build_trail(LEDGER, "").pages)
assert "https://earlier.example/p2" not in t.render()
def test_an_unopened_url_is_still_caught_when_earlier_sources_are_supplied():
"""The reverse direction: the parameter widens the denominator, it does not
disable the check. A fix that made the accusation unreachable would be worth
less than the false positive it replaced."""
t = build_trail(
LEDGER,
"see https://never.example/x",
["https://earlier.example/p2"],
)
assert t.cited_not_opened == ["https://never.example/x"]
assert t.counters()["citation_grounding_rate"] == 0.0
def test_earlier_sources_are_normalised_the_same_way_as_fetched_ones():
"""A memo stores whatever the reader handed back, so the two sides can differ in
canonical form. Comparing them raw would reinstate the false positive for exactly
the URLs a reader service rewrote."""
t = build_trail(LEDGER, "see https://earlier.example/p2/", ["http://earlier.example/p2"])
assert t.cited_not_opened == []
def test_the_scope_is_reported_so_two_runs_are_comparable():
"""``opened_earlier`` is 0 on every single-turn run, which is every measured arm -
so a non-zero value is the signal that the rate beside it was computed over a
conversation rather than a turn."""
assert build_trail(LEDGER, "x").counters()["opened_earlier"] == 0
assert build_trail(LEDGER, "x", []).counters()["opened_earlier"] == 0
# --------------------------------------------------------------------------- #
# dr@3.0: the extractor's stop set, and the ASCII behaviour it must not change #
# --------------------------------------------------------------------------- #
@pytest.mark.parametrize(
"answer,expected",
[
("\u4f9d\u636e https://a.example/one\uff08\u539f\u6587\uff1axxx\uff09", "https://a.example/one"),
("\u89c1 https://a.example/one\uff09\u3002\u4f46\u8be5\u9875", "https://a.example/one"),
("\u53c2\u8003 https://a.example/one\uff1b\u53e6\u89c1", "https://a.example/one"),
("\u6587\u6863 https://a.example/one\uff0c\u8bf4\u660e\u4e86\u8fd9\u4e00\u70b9", "https://a.example/one"),
("\u5f15\u81ea https://a.example/one`\uff09\uff1b\u4f50\u8bc1", "https://a.example/one"),
("**https://a.example/one**\uff0c\u89c1\u4e0a", "https://a.example/one"),
("\u8be6\u89c1 https://a.example/one\u548c\u5176\u5b83\u9875", "https://a.example/one"),
],
)
def test_a_url_followed_by_chinese_punctuation_is_not_a_fabricated_citation(answer, expected):
"""The defect an eight-turn Chinese conversation surfaced, one form per row.
The stop set was ASCII-only, and on Chinese output that is not a near-miss: nothing
ended the match, so the citation swallowed the rest of the clause and matched no
fetch record. A page the run really had opened was then named as invented - fixed
direction, and only on Chinese output, which is this product's whole surface. Four
of one eight-turn run's flagged citations were this and none were real.
"""
t = build_trail(LEDGER, answer)
assert t.cited == [expected]
assert t.cited_not_opened == []
assert t.counters()["citation_grounding_rate"] == 1.0
def test_the_ascii_forms_every_published_reading_was_measured_on_are_unchanged():
"""The regression risk that matters more than the fix.
This counter has been reported since dr@2.8. Widening the stop set must not move a
single ASCII extraction, or every historical value silently refers to a different
quantity than the one recomputed from the same trajectory.
"""
assert build_trail(LEDGER, "see https://a.example/one).").cited == ["https://a.example/one"]
assert build_trail(LEDGER, "(https://a.example/one)").cited == ["https://a.example/one"]
assert build_trail(LEDGER, "https://a.example/one, and more").cited == ["https://a.example/one"]
assert build_trail(LEDGER, "per https://a.example/one and https://never.example/x it holds").cited == [
"https://a.example/one",
"https://never.example/x",
]
# Query strings and fragments are part of the link, not sentence punctuation.
assert build_trail(LEDGER, "at https://a.example/one?q=1&b=2#frag.").cited == ["https://a.example/one?q=1&b=2#frag"]
def test_a_chinese_answer_can_still_be_caught_fabricating():
"""The reverse direction. A stop set wide enough to stop catching anything would be
worth less than the false positives it removed."""
t = build_trail(LEDGER, "\u4f9d\u636e https://never.example/x\uff08\u539f\u6587\uff1axxx\uff09")
assert t.cited_not_opened == ["https://never.example/x"]
assert t.counters()["citation_grounding_rate"] == 0.0
# ── decoded CJK paths, and the prose they drag in ──────────────────────
#
# A page is fetched percent-encoded and cited decoded. The extractor used to end
# a URL at the first ideograph, so every such citation was reported as never
# opened - and two different wiki pages truncated to the SAME prefix, multiplying
# the accusation. Observed on every Chinese-subject turn of the first product
# sessions that cited zh.wikipedia paths.
CJK_LEDGER = [
{"op": "search", "query": "\u674e\u5f66\u5b8f", "replay": False, "zero_hit": False},
{
"op": "fetch",
"url": "https://zh.wikipedia.org/wiki/%E6%9D%8E%E5%BD%A6%E5%AE%8F",
"chars": 1711,
"ok": True,
},
]
def test_a_decoded_cjk_citation_matches_its_percent_encoded_fetch():
t = build_trail(
CJK_LEDGER, "\u89c1 https://zh.wikipedia.org/wiki/\u674e\u5f66\u5b8f \uff1b\u53e6\u89c1\u7ef4\u57fa"
)
assert t.cited == ["https://zh.wikipedia.org/wiki/\u674e\u5f66\u5b8f"]
assert t.cited_not_opened == []
assert t.counters()["citation_grounding_rate"] == 1.0
def test_prose_glued_onto_a_cjk_path_resolves_to_the_fetched_page():
"""Keeping ideographs in the extractor swallows glued prose into the match;
whether the tail is path or prose is decidable only against the fetch record,
so the trimmed form wins exactly when a record exists for it."""
t = build_trail(
CJK_LEDGER,
"\u8be6\u89c1 https://zh.wikipedia.org/wiki/\u674e\u5f66\u5b8f\u8bcd\u6761\u4e2d\u7684\u8bf4\u660e",
)
assert t.cited == ["https://zh.wikipedia.org/wiki/\u674e\u5f66\u5b8f"]
assert t.cited_not_opened == []
def test_a_fabricated_cjk_path_is_still_caught():
"""Trimming never invents a match: a CJK path with no fetch record behind any
trailing-ideograph form stays an accusation, reported as written."""
t = build_trail(CJK_LEDGER, "\u89c1 https://zh.wikipedia.org/zh-hans/\u6587\u5fc3\u4e00\u8a00 \u3002")
assert t.cited_not_opened == ["https://zh.wikipedia.org/zh-hans/\u6587\u5fc3\u4e00\u8a00"]
assert t.counters()["citation_grounding_rate"] == 0.0
def test_memo_pages_nobody_cited_do_not_inflate_the_scope_count():
"""The live defect: a warning listed one link while the scope line said
"(2 of those...)" - the 2 was every memo page, cited or not, so the sentence
named a number with no relation to the links beside it."""
t = build_trail(
LEDGER,
"see https://never.example/x",
["https://memo.example/a", "https://memo.example/b"],
)
assert t.opened_earlier == 0
assert t.cited_not_opened == ["https://never.example/x"]
assert "earlier turn" not in t.render()
def test_the_scope_sentence_counts_cited_links_and_reads_coherently():
answer = "per https://earlier.example/p2 and https://never.example/x"
t = build_trail(LEDGER, answer, ["https://earlier.example/p2", "https://memo.example/unused"])
assert t.opened_earlier == 1
assert "1 of the cited links was opened on an earlier turn" in t.render()
# ── the salvage seam ships unreviewed, and the trail must say so ───────
def test_a_salvaged_answer_cannot_wear_a_reviewed_trail():
"""A salvaged answer never reaches the reviewer (observer order, ``dr.py``),
and its trail used to simply omit the reviewer segment - silence a reader
cannot tell apart from "not configured". The ledger row renders as an
explicit skip, and the counter makes the salvage share of an arm measurable
next to ``verify`` outcomes."""
rows = [r for r in LEDGER if r["op"] != "verify"] + [
{"op": "force_finalize", "event": "salvage", "seam": "iteration"}
]
t = build_trail(rows, "x")
assert t.salvaged is True
assert t.counters()["salvaged"] is True
assert "reviewer: skipped (salvaged answer)" in t.render()
def test_a_reviewed_turn_does_not_say_skipped():
t = build_trail(LEDGER, "x")
assert t.counters()["salvaged"] is False
assert "skipped" not in t.render()
def test_the_trail_names_the_reviewer_when_the_rows_do():
"""A mode may move the reviewer per session, so a verdict without its
reviewer cannot be compared with another's. Named off the last verify row,
counted for the exporter, and silent when the rows predate the field."""
rows = list(LEDGER) + [
{"op": "verify", "outcome": "rejected", "unsupported_claims": ["c1"], "model": "judge/cheap"},
{"op": "verify", "outcome": "pass", "unsupported_claims": [], "model": "judge/cheap"},
]
t = build_trail(rows, "x")
assert t.verify_model == "judge/cheap"
assert t.counters()["verify_model"] == "judge/cheap"
assert "reviewer: pass via judge/cheap" in t.render()
legacy = build_trail(list(LEDGER), "x")
assert legacy.verify_model is None and legacy.counters()["verify_model"] is None
assert " via " not in legacy.render()
def test_a_salvage_after_a_rejection_does_not_hide_behind_the_verdict():
"""The common salvage path runs THROUGH a verdict: reject -> revision ->
empty visible answer -> salvage. The verdict was about a draft that never
shipped, so rendered alone it dresses the salvage as a reviewed turn - the
exact trail this seam's visibility fix exists to prevent."""
rows = list(LEDGER) + [
{"op": "verify", "outcome": "rejected", "unsupported_claims": ["c1"]},
{"op": "force_finalize", "event": "salvage", "seam": "terminal"},
]
t = build_trail(rows, "x")
rendered = t.render()
assert "reviewer: rejected (1 open point)" in rendered
assert "shipped answer: salvaged (not reviewed)" in rendered
# ── dr@3.3: the grounding line is rendered in every direction ─────────
#
# Before dr@3.3 only the failing direction reached the page. A check that is
# visible when it fails and invisible when it passes trains its reader to read
# silence as "not checked", which is the same failure shape as a gate that only
# prints on red. All three states are asserted here so none can be dropped as
# "the obvious case".
def test_a_clean_answer_says_so_instead_of_saying_nothing():
t = build_trail(LEDGER, "see https://a.example/one")
out = t.render()
assert "All 1 cited link was opened" in out
assert "⚠️" not in out
def test_an_answer_that_cites_nothing_is_not_rendered_as_perfect():
"""Undefined, not 1.0 - the ``counters()`` rule, now also on the page.
The invariant is that a zero-citation answer never renders as verified, and
it is asserted as an invariant rather than as one sentence, because the
sentence now depends on *why* there were no citations.
"""
t = build_trail(LEDGER, "no links here at all")
out = t.render()
assert "\u2713" not in out
assert "Nothing above was verified." in out or "cites no link" in out
assert "All 0" not in out
assert t.counters()["citation_grounding_rate"] is None
def test_the_warning_carries_its_denominator():
"""The module rule is that the rate never appears without ``urls_cited``.
Rendering "2 links were never opened" without the total states a numerator
on its own, which is the same number saying two different things depending
on whether the answer cited two links or forty."""
t = build_trail(LEDGER, "see https://a.example/one and https://ghost.example/x")
out = t.render()
assert "of 2 link(s) cited" in out
assert t.counters()["urls_cited"] == 2
def test_the_conversation_scope_travels_with_the_fraction():
t = build_trail(LEDGER, "see https://a.example/one")
t.opened_earlier = 1
assert "opened on an earlier turn" in t.render()
def test_scope_note_is_absent_when_nothing_was_cited():
t = build_trail(LEDGER, "no links")
t.opened_earlier = 3
assert "earlier turn" not in t.render()
# ── citation-tail and truncated-citation recoveries (both set-aware) ──────
DATE_TAIL_LEDGER = [
{"op": "fetch", "url": "https://finance.sina.com.cn/roll/doc-abc.shtml", "chars": 900, "ok": True},
{"op": "fetch", "url": "https://news.qq.com/rain/a/20260109A024R500", "chars": 900, "ok": True},
]
def test_a_comma_date_citation_tail_resolves_to_the_fetched_page():
"""Observed live: the model cites ``(url,2026-01-09)`` and the extractor
swallows the comma and date into the URL. The tail comes off only because a
fetch record exists for the clipped form."""
t = build_trail(
DATE_TAIL_LEDGER,
"\u6765\u6e90\uff1ahttps://finance.sina.com.cn/roll/doc-abc.shtml,2026-01-09 \u62a5\u9053",
)
assert t.cited == ["https://finance.sina.com.cn/roll/doc-abc.shtml"]
assert t.cited_not_opened == []
def test_two_citations_glued_by_a_semicolon_recover_the_first():
"""The extractor hands back one string; clipping at the first separator
recovers the first citation. The second is lost to the glue, which
understates ``cited`` rather than accusing anything - the safe direction."""
t = build_trail(
DATE_TAIL_LEDGER,
"\u89c1 https://finance.sina.com.cn/roll/doc-abc.shtml,2026-01-09;"
"https://news.qq.com/rain/a/20260109A024R500,2026-01-09 \u4e24\u5904",
)
assert t.cited == ["https://finance.sina.com.cn/roll/doc-abc.shtml"]
assert t.cited_not_opened == []
def test_a_comma_tail_with_no_fetch_record_stays_an_accusation():
"""Reported AS WRITTEN, tail included - clipping is set-aware, so a miss
rewrites nothing, same as the fabricated-CJK-path precedent."""
t = build_trail(DATE_TAIL_LEDGER, "\u89c1 https://finance.sina.com.cn/roll/doc-xyz.shtml,2026-01-09 \u3002")
assert t.cited_not_opened == ["https://finance.sina.com.cn/roll/doc-xyz.shtml,2026-01-09"]
PREFIX_LEDGER = [
{
"op": "fetch",
"url": (
"https://www.idc.com/resource-center/blog/%E6%99%BA%E8%83%BD%E4%BD%93token"
"%E6%B6%88%E8%80%97%E5%B9%B4%E5%9D%87%E5%A2%9E%E8%B6%8530%E5%80%8D%E4%B8%AD%E5%9B%BD/"
),
"chars": 5000,
"ok": True,
},
]
def test_a_truncated_citation_that_prefixes_one_opened_page_matches_it():
"""Observed live: the model truncated a long ideograph path when citing. The
cited form is a strict prefix of exactly one opened page, so it matches -
the SHALLOWER direction, which invents nothing deeper than what was read."""
t = build_trail(
PREFIX_LEDGER,
"\u6765\u6e90\uff1ahttps://www.idc.com/resource-center/blog/"
"\u667a\u80fd\u4f53token\u6d88\u8017\u5e74\u5747\u589e\u8d8530\u500d \u4e00\u6587",
)
assert t.cited_not_opened == []
assert t.counters()["citation_grounding_rate"] == 1.0
def test_a_short_prefix_does_not_absolve_itself_against_a_deeper_page():
"""A string prefix is not a path prefix: ``/a`` must not match ``/about``.
The path floor keeps short-segment citations in accusation territory."""
rows = [{"op": "fetch", "url": "https://ex.example/about-the-project", "chars": 10, "ok": True}]
t = build_trail(rows, "\u89c1 https://ex.example/a \u9875\u9762")
assert t.cited_not_opened == ["https://ex.example/a"]
def test_an_ambiguous_prefix_stays_an_accusation():
"""Two opened pages share the cited prefix: the citation is ambiguous, and
the check accuses rather than guesses."""
rows = [
{"op": "fetch", "url": "https://ex.example/reports/2026-industry-alpha", "chars": 10, "ok": True},
{"op": "fetch", "url": "https://ex.example/reports/2026-industry-beta", "chars": 10, "ok": True},
]
t = build_trail(rows, "\u89c1 https://ex.example/reports/2026-industry \u62a5\u544a")
assert t.cited_not_opened == ["https://ex.example/reports/2026-industry"]
# --------------------------------------------------------------------------- #
# Where the rendered trail is parked #
# --------------------------------------------------------------------------- #
# ── why "no links were cited": the causes, split ──────────────────────
#
# The grounding check reporting that it did not apply is correct and easy to
# read as "passed". These give the line its cause: pages read but nothing
# cited, sources named without a scheme, or a run that really did nothing.
def test_the_generic_nothing_to_check_line_survives_when_no_page_was_read():
"""Separating the causes must not delete the case the line was written for:
a turn that opened no page and cited no link has an inapplicable check for
the ordinary reason, and accusing it of citing nothing after reading would
be a false alarm."""
searches_only = [r for r in LEDGER if r["op"] != "fetch"]
out = build_trail(searches_only, "no links here at all").render()
assert "nothing to check" in out
assert "\u26a0" not in out
assert build_trail(searches_only, "x").counters()["read_but_cited_nothing"] is False
def test_reading_pages_and_citing_nothing_is_called_out():
"""The state the grounding check cannot see into."""
c = build_trail(LEDGER, "no links here at all").counters()
assert c["read_but_cited_nothing"] is True
assert c["cited_schemeless"] == 0
assert c["citation_grounding_rate"] is None
assert "1 page(s) were read but the answer cites no link" in build_trail(LEDGER, "no links here at all").render()
def test_a_scheme_less_citation_is_disclosed_without_moving_the_rate():
"""A product run wrote all 43 of its references as bare hosts, so ``_URL_RE``
saw none of them and the appendix reported that there was nothing to check.
That is a generation-side formatting problem wearing the costume of an answer
that cited nothing. Widening ``_URL_RE`` was refused: it would move
``citation_grounding_rate``'s denominator under an unchanged name.
Real TLDs on purpose: ``.example`` is reserved for documentation, and adding
it to the production pattern to make a test pass would be fitting the code to
the fixture. The counter does not consult the ledger, so these hosts need not
appear in ``LEDGER``."""
t = build_trail(LEDGER, "evidence: kyutai.org/blog/arc and github.com/x/y (web_fetch #ab12)")
c = t.counters()
assert c["urls_cited"] == 0
assert c["citation_grounding_rate"] is None
assert c["cited_not_opened"] == 0 and c["cited_never_surfaced"] == 0
assert c["cited_schemeless"] == 2
assert "without a URL scheme" in t.render()
def test_a_scheme_ful_citation_is_never_counted_twice():
"""``arxiv.org/x`` inside ``https://arxiv.org/x`` is one citation; a counter
that double-counts the well-behaved case fires on every correct answer."""
t = build_trail(LEDGER, "see https://a.example/one")
assert t.counters()["urls_cited"] == 1
assert t.counters()["cited_schemeless"] == 0
assert t.counters()["read_but_cited_nothing"] is False
def test_the_scheme_less_pattern_does_not_fire_on_ordinary_prose():
"""A path separator is required precisely so version numbers, figure
references and ratios are not hosts."""
noise = "arXiv:2510.20535, Fig.2/3, v1.5/beta, 4x/8x, 1.0/2.0, user@host.com/x"
assert build_trail(LEDGER, noise).counters()["cited_schemeless"] == 0
def test_a_page_cited_both_with_and_without_its_scheme_is_one_reference():
"""``_norm`` drops the scheme, so a reference list writing the URL in full and
prose naming the same page bare fold to one key. The host needs a TLD
``_SCHEMELESS_RE`` actually lists; ``.example`` is not one."""
url = "https://kyutai.org/blog/arc-encoder/"
ledger = [*LEDGER, {"op": "fetch", "url": url, "chars": 9001, "ok": True}]
t = build_trail(ledger, f"see {url} and, as kyutai.org/blog/arc-encoder/ explains, it holds")
c = t.counters()
assert c["urls_cited"] == 1
assert c["cited_schemeless"] == 0, t.cited_schemeless
def test_research_seconds_spans_the_ledger_not_the_turn():
"""Derived from ``ts``: first research event to last, so it excludes the
final generation. That is a definition, not a defect, and pinning it stops
the gap from being read as one."""
rows = [
{"op": "search", "query": "q", "ts": 1000.0, "replay": False, "zero_hit": False},
{"op": "fetch", "url": "https://a.org/x", "chars": 9000, "ok": True, "ts": 1130.4},
]
t = build_trail(rows, "https://a.org/x")
assert t.counters()["research_seconds"] == 130
one = build_trail([rows[0]], "x")
assert one.counters()["research_seconds"] is None
no_ts = build_trail([{"op": "search", "query": "q"}, {"op": "search", "query": "r"}], "x")
assert no_ts.counters()["research_seconds"] is None
def test_a_sub_minute_research_span_is_not_rendered():
short = [
{"op": "search", "query": "q", "ts": 0.0},
{"op": "fetch", "url": "https://a.org/x", "chars": 9000, "ok": True, "ts": 40.0},
]
long_ = [
{"op": "search", "query": "q", "ts": 0.0},
{"op": "fetch", "url": "https://a.org/x", "chars": 9000, "ok": True, "ts": 1815.0},
]
assert "of research" not in build_trail(short, "https://a.org/x").render()
assert "30m of research" in build_trail(long_, "https://a.org/x").render()
def test_thin_pages_are_counted_not_only_rendered():
"""``ok`` is a transport verdict, not a content one: a 199-character stub
counts as ``pages_ok``. The split was rendered but never emitted, so every
fetch-productivity reading downstream could not tell a page from a wrapper."""
rows = [
{"op": "search", "query": "q", "replay": False, "zero_hit": False},
{"op": "fetch", "url": "https://a.example/real", "chars": 12431, "ok": True},
{"op": "fetch", "url": "https://a.example/stub", "chars": 199, "ok": True},
{"op": "fetch", "url": "https://a.example/dead", "chars": 0, "ok": False},
]
t = build_trail(rows, "https://a.example/real")
c = t.counters()
assert c["pages_ok"] == 2
assert c["thin_pages"] == 1
assert c["pages_failed"] == 1
assert "(1 returned almost nothing)" in t.render()
def test_a_page_the_site_refused_is_not_counted_as_read():
"""The reader answers 200 over a CAPTCHA wall; the trail said every page was read
while the report beside it listed two as blocked."""
rows = [
{"op": "search", "query": "q", "replay": False, "zero_hit": False},
{"op": "fetch", "url": "https://a.example/real", "chars": 12431, "ok": True, "status": 200},
{"op": "fetch", "url": "https://a.example/wall", "chars": 862, "ok": True, "status": 405, "blocked": "captcha"},
{"op": "fetch", "url": "https://a.example/gate", "chars": 491, "ok": True, "status": 200, "blocked": "captcha"},
{"op": "fetch", "url": "https://a.example/pixel", "chars": 38, "ok": True, "status": 200},
]
t = build_trail(rows, "[a](https://a.example/real) [w](https://a.example/wall) [g](https://a.example/gate)")
c = t.counters()
assert (c["pages_ok"], c["pages_refused"], c["thin_pages"], c["pages_failed"]) == (4, 2, 1, 0)
# Requested all the same, so citing one is not citing a page never opened.
assert c["cited_not_opened"] == 0
rendered = t.render()
# The stub note stays beside "pages read", whose count it belongs to.
assert "2 pages read (1 returned almost nothing), 2 refused by the site" in rendered
assert "- (2 page(s) the site refused: https://a.example/gate, https://a.example/wall)" in rendered
assert "- https://a.example/wall" not in rendered
def test_a_refusal_is_per_fetch_so_a_retry_that_reads_the_page_counts_as_read():
"""A page refused once and read on a retry was read; one refused twice is two refusals
of one URL, listed once."""
rows = [
{"op": "fetch", "url": "https://a.example/p", "chars": 300, "ok": True, "status": 403},
{"op": "fetch", "url": "https://a.example/p", "chars": 9000, "ok": True, "status": 200},
{"op": "fetch", "url": "https://b.example/q", "chars": 300, "ok": True, "status": 403},
{"op": "fetch", "url": "https://b.example/q", "chars": 300, "ok": True, "status": 403},
]
t = build_trail(rows, "[a](https://a.example/p)")
c = t.counters()
assert (c["pages_ok"], c["pages_refused"], c["thin_pages"]) == (4, 3, 0)
rendered = t.render()
assert "1 pages read, 3 refused by the site" in rendered
assert "- https://a.example/p (9,000 chars)" in rendered
assert "- (3 page(s) the site refused: https://a.example/p, https://b.example/q)" in rendered
def test_the_refused_listing_is_capped_like_the_others():
rows = [
{"op": "fetch", "url": f"https://a.example/{i:02d}", "chars": 300, "ok": True, "status": 403} for i in range(45)
]
line = next(ln for ln in build_trail(rows, "").render().splitlines() if "the site refused" in ln)
assert line.endswith("https://a.example/39, ... and 5 more)")
def test_a_truncated_fence_tag_is_still_a_fence_tag():
"""The ``{4,16}`` width. ``token_hex(4)`` mints exactly 8, so ``{8}`` looks
right and passes every other test here; a clipped quote is still a fence tag."""
t = build_trail(LEDGER, 'halves it ("quote" web_fetch #cd81)')
c = t.counters()
assert c["urls_cited"] == 0
assert c["cited_fence_tags"] == 1
assert "fence tag" in t.render()
# ── the twin is the fork's module, and a test says so ─────────────────
def _module_body(path: Path) -> str:
"""The module with every docstring and comment removed, re-printed from its AST.
Prose is where the two files legitimately differ (this repo avoids literal
CJK in code and names its own seams); code is where they must not.
"""
import ast
class _Strip(ast.NodeTransformer):
def generic_visit(self, node):
super().generic_visit(node)
body = getattr(node, "body", None)
if isinstance(body, list):
kept = [
b
for b in body
if not (
isinstance(b, ast.Expr) and isinstance(b.value, ast.Constant) and isinstance(b.value.value, str)
)
]
node.body = kept or [ast.Pass()]
return node
return ast.unparse(_Strip().visit(ast.parse(path.read_text(encoding="utf-8"))))
# The record is frozen at dr@3.5 and is never edited to make this green
# (tests/fixtures/vendored_fork/README.md); the twin's deliberate lead over it is
# named here, definition by definition. A difference outside these sets is drift,
# and an entry that has stopped differing is a stale allowance.
#: Definitions the twin has that the record does not; a class member is ``Class.member``.
TWIN_LEADS_ADDED: frozenset[str] = frozenset(
{"_TRACKING_KEYS", "_is_tracking", "_drop_tracking"} # click-tracking parameters dropped before matching
| {"ResearchTrail.refused"} # pages the site refused, counted apart
)
#: Definitions in both whose bodies the twin changed.
TWIN_LEADS_CHANGED: frozenset[str] = frozenset(
{"_norm", "build_trail"} # the tracking drop, and the refused pages
| {"ResearchTrail._page_split", "ResearchTrail.counters", "ResearchTrail.render"}
)
def _definitions(path: Path) -> dict[str, str]:
"""Each definition of :func:`_module_body`'s stripped module, by name.
A class is split into its header and one entry per member, so a lead inside a
class names the members it changed and the rest of the class stays pinned.
"""
import ast
def name_of(node: ast.stmt, i: int) -> str:
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
return node.name
if isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name):
return node.targets[0].id
if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name):
return node.target.id
return f"{i}:{type(node).__name__}"
out: dict[str, str] = {}
for i, node in enumerate(ast.parse(_module_body(path)).body):
name = name_of(node, i)
if isinstance(node, ast.ClassDef):
for j, member in enumerate(node.body):
out[f"{name}.{name_of(member, j)}"] = ast.unparse(member)
node.body = [ast.Pass()]
out[name] = ast.unparse(node)
return out
def test_the_appendix_twin_is_the_forks_module_body_plus_its_named_lead():
"""The third layer of the twin, after config values and class defaults.
A fix that lands on the fork's appendix and not here ships a launcher whose
integrity appendix says something the fork's no longer does - the fence-tag
disclosure did exactly that for one review round. Docstrings and comments
are stripped before comparing, so the two files may explain themselves in
their own words; the code has to be one, apart from the lead named above.
"""
fork = (
REPO
/ "tests"
/ "fixtures"
/ "vendored_fork"
/ "raven-research"
/ "Raven-X"
/ "raven"
/ "agent"
/ "process_appendix.py"
)
twin = PLUGIN_DIR / "research_flow" / "support" / "process_appendix.py"
record, ours = _definitions(fork), _definitions(twin)
assert set(ours) - set(record) == TWIN_LEADS_ADDED, set(ours) ^ set(record)
assert not (set(record) - set(ours)), "the twin dropped a definition the record has: " + repr(
set(record) - set(ours)
)
changed = {name for name in record if record[name] != ours[name]}
assert changed == TWIN_LEADS_CHANGED, "changed outside the named lead (or a stale entry): " + repr(
changed ^ TWIN_LEADS_CHANGED
)
# ── fence-tag citations, and the unreviewed banner ─────────────────────
#
# Ported from the fork's fix: one product run wrote 75 of its 87 citation
# handles as the security fence's nonce - ``(web_fetch #cd8187b0)`` - and the
# grounding check, seeing no URL, said there was nothing to check over an answer
# whose every reference pointed at a page the run had really opened. The trunk
# runtime wraps tool results in the same nonce-tagged fences, so the shape is
# reachable on this launcher too. The counter gives "no links were cited" its
# cause, and the banner keeps "the reviewer never saw this" from hiding in one
# word of the head line.
def test_a_fence_tag_citation_is_disclosed_without_moving_the_rate():
"""Same contract as every other disclosure here: the landed metric cannot move.
The tag is NOT resolved to a URL - the fence nonce never reaches the ledger,
and a disclosure counter that guessed would accuse or absolve on data it does
not have. Repeats of one tag are one reference."""
t = build_trail(
LEDGER,
'halves it ("quote" web_fetch #cd8187b0), again (web_fetch #cd8187b0), listed at (web_search #aa11bb22)',
)
c = t.counters()
assert c["urls_cited"] == 0
assert c["citation_grounding_rate"] is None
assert c["cited_not_opened"] == 0 and c["cited_never_surfaced"] == 0
assert c["cites_nothing"] is True
assert c["cited_fence_tags"] == 2
out = t.render()
assert "fence tag" in out
assert "Nothing above was verified." in out
assert "nothing to check" not in out
def test_the_fence_tag_pattern_needs_the_tool_name():
"""A bare ``#hex`` is an issue number, an anchor, a color. Only the tool-name
prefix marks the token as a quoted fence, and a counter that fired on GitHub
issue references would get ignored - which costs the whole mechanism."""
noise = "see issue #cd8187b0, color #ab12cd, and the web_fetch tool itself"
t = build_trail(LEDGER, noise)
assert t.counters()["cited_fence_tags"] == 0
assert "fence tag" not in t.render()
def test_fence_tags_beside_real_urls_are_noted_not_folded():
"""The mixed case: the check runs over the real URLs and prints its ✓, and
without the scope note that ✓ silently covers references it never saw."""
t = build_trail(LEDGER, "see https://a.example/one (web_fetch #cd8187b0)")
out = t.render()
assert t.counters()["urls_cited"] == 1
assert t.counters()["citation_grounding_rate"] == 1.0
assert t.counters()["cited_fence_tags"] == 1
assert "All 1 cited link was opened" in out
assert "could not be checked" in out
def test_a_web_search_tag_is_never_reported_as_a_web_fetch():
"""The rendered example quotes what the answer wrote, tool name included: an
integrity line that misstates the event it discloses is its own counterexample."""
t = build_trail(LEDGER, "per (web_search #aa11bb22) it holds")
out = t.render()
assert t.cited_fence_tags == ["web_search #aa11bb22"]
assert "web_search #aa11bb22" in out
assert "web_fetch" not in out
mixed = build_trail(LEDGER, "see https://a.example/one and (web_search #aa11bb22)")
assert "web_search #aa11bb22" in mixed.render()
assert "web_fetch" not in mixed.render()
def test_an_unreviewed_run_wears_its_banner_before_the_head():
"""``reviewer: budget_spent`` was one word in the head - a reader scanning
for a verdict reads right past it. The state bounds how much the whole
answer can be trusted, so it leads the block."""
rows = [r for r in LEDGER if r["op"] != "verify"] + [{"op": "verify", "outcome": "budget_spent", "reviewed": False}]
out = build_trail(rows, "see https://a.example/one").render()
assert "shipped unreviewed" in out
assert "revision budget" in out
assert out.index("shipped unreviewed") < out.index("**Research trail**")
def test_a_fail_open_review_banners_too():
rows = [r for r in LEDGER if r["op"] != "verify"] + [{"op": "verify", "outcome": "unavailable"}]
out = build_trail(rows, "x").render()
assert "shipped unreviewed" in out and "unavailable" in out
def test_a_salvaged_answer_banners_as_unreviewed():
"""Salvage is the third unreviewed-shipped state, and it can hide behind a
verdict on a draft that never shipped (reject -> revision -> salvage)."""
rows = list(LEDGER) + [{"op": "force_finalize", "event": "salvage", "seam": "terminal"}]
out = build_trail(rows, "x").render()
assert "shipped unreviewed" in out
assert "salvage synthesis" in out
def test_a_reviewed_or_unconfigured_run_has_no_banner():
"""``budget_spent_reviewed`` reviewed the shipped draft - that is its whole
point - and a None outcome is an arm with no reviewer configured, where a
banner on every run would cry wolf until ignored."""
assert "shipped unreviewed" not in build_trail(LEDGER, "x").render()
reviewed = [r for r in LEDGER if r["op"] != "verify"] + [
{"op": "verify", "outcome": "budget_spent_reviewed", "unsupported_claims": ["c1"]}
]
assert "shipped unreviewed" not in build_trail(reviewed, "x").render()
no_reviewer = [r for r in LEDGER if r["op"] != "verify"]
assert "shipped unreviewed" not in build_trail(no_reviewer, "x").render()
def test_the_rendered_trail_is_kept_where_a_host_can_read_it(tmp_path, monkeypatch):
"""The appendix rides on the reply; the trail has to survive somewhere else.
Riding on the outbound string alone is the whole of the appendix's
distribution neutrality - the persisted transcript never carries it, so no
model reads it this turn or as history next turn. That also means the only
machine-readable copy is the one parked beside the counters, and the turn's
ledger file is deleted moments later. Measured once on a live turn: 17 pages
read, 8 cited, and the record of the other 9 built and then dropped.
Kept OUT of ``process_appendix``: those are counters a reader parses as a
measurement payload, and a multi-kilobyte string does not belong beside them.
"""
import asyncio
from research_flow.config import FlowConfig
from research_flow.flow import ResearchFlowHook, ToolHandles
from research_flow.state import SessionStore
from research_flow.support import ledger as ledger_mod
from raven.contracts.loop_hooks import AgentHookContext
store = SessionStore(tmp_path)
monkeypatch.setattr(ledger_mod, "_ledger_dir", store.ledger_dir)
cfg = FlowConfig.from_slice({"enabled": True, "finalShape": {"processAppendix": True}})
hook = ResearchFlowHook(cfg=cfg, provider=None, tools=ToolHandles(), store=store)
async def turn():
ctx = AgentHookContext(
session_key="s",
iteration=1,
messages=[{"role": "user", "content": "q"}],
metadata={"mode": "", "mode_overlay": {"drFlow": {}}},
)
await hook.before_iteration(ctx)
path = ledger_mod.ledger_path()
Path(path).write_text("\n".join(json.dumps(r) for r in LEDGER) + "\n", encoding="utf-8")
return await hook.after_send(
AgentHookContext(
session_key="s",
outbound_content="the answer cites https://a.example/one",
metadata=ctx.metadata,
)
)
decision = asyncio.run(turn())
observers = store.load("s").observers
assert "Research trail" in (decision.modified_content or ""), "the appendix still rides on the reply"
assert "Research trail" in observers["research_trail"]
assert observers["research_trail"] in (decision.modified_content or "")
assert observers["process_appendix"]["emitted"] is True
assert "research_trail" not in observers["process_appendix"]
def test_turn_observers_reach_the_trunk_stamp_seam(tmp_path, monkeypatch):
"""[B-1] The trunk stamps ``ctx.metadata["observers"]`` onto the turn's last
substantive assistant message at persist -- the seam the fork's loop had
natively. The flow must feed it at turn end, or only the record store's
latest-turn copy survives and the per-turn history the fork kept is gone.
The send context shares the turn's metadata dict in production, so the
test hands it the same object the inbound phase saw.
"""
import asyncio
from research_flow.config import FlowConfig
from research_flow.flow import ResearchFlowHook, ToolHandles
from research_flow.state import SessionStore
from research_flow.support import ledger as ledger_mod
from raven.contracts.loop_hooks import AgentHookContext
store = SessionStore(tmp_path)
monkeypatch.setattr(ledger_mod, "_ledger_dir", store.ledger_dir)
cfg = FlowConfig.from_slice({"enabled": True, "finalShape": {"processAppendix": True}})
hook = ResearchFlowHook(cfg=cfg, provider=None, tools=ToolHandles(), store=store)
async def turn():
first = AgentHookContext(
session_key="s",
iteration=1,
messages=[{"role": "user", "content": "q"}],
metadata={"mode": "", "mode_overlay": {"drFlow": {}}},
)
await hook.before_iteration(first)
path = ledger_mod.ledger_path()
Path(path).write_text("\n".join(json.dumps(r) for r in LEDGER) + "\n", encoding="utf-8")
send_ctx = AgentHookContext(session_key="s", outbound_content="the answer", metadata=first.metadata)
await hook.after_send(send_ctx)
return send_ctx
send_ctx = asyncio.run(turn())
stamped = send_ctx.metadata.get("observers")
assert isinstance(stamped, dict) and stamped, "after_send must feed the stamp seam"
assert stamped == store.load("s").observers, "the stamp and the record read the same counters"
assert "Research trail" in stamped["research_trail"]
# ── two addresses that are one page ───────────────────────────────────
# ★ 20260906 (Framework, product feedback). Every test below is about the same
# accusation - "cited a link that appears nowhere in this run" - and about the
# two shapes of address the 2026-09-04 run made it wrong on. The negatives carry
# the weight: a fold that reaches one segment too far absolves the fabrication
# this check exists to catch, and it does so silently.
_ARXIV_PAPER = "https://arxiv.org/abs/2401.12345"
def test_a_paper_opened_at_its_abstract_and_cited_as_a_pdf_is_one_page():
"""Four prefixes and a version stamp serve one paper, and the run's own
thin-page recovery re-reads an abstract at its PDF - so the address the
ledger holds and the address the answer carries routinely differ in exactly
the parts that do not name the paper."""
ledger = [*LEDGER, {"op": "fetch", "url": _ARXIV_PAPER, "chars": 41002, "ok": True}]
t = build_trail(ledger, "as https://arxiv.org/pdf/2401.12345 reports, it holds")
assert t.cited_not_opened == [], t.cited_not_opened
assert t.counters()["citation_grounding_rate"] == 1.0
def test_the_paper_fold_stops_at_the_identifier():
"""A neighbouring id is a different paper, and the fold must leave it to be
accused. Without this the whole of arxiv.org would read as opened."""
ledger = [*LEDGER, {"op": "fetch", "url": _ARXIV_PAPER, "chars": 41002, "ok": True}]
t = build_trail(ledger, "as https://arxiv.org/pdf/2401.99999 reports, it holds")
assert t.cited_never_surfaced == ["https://arxiv.org/pdf/2401.99999"]
def test_an_arxiv_form_folds_before_the_never_surfaced_check_too():
"""The softer accusation and the heavier one are separately reachable, and
the fold has to reach the heavier one. A paper a search listed and nobody
opened is "listed but never opened" - not "appears nowhere in this run"."""
ledger = [*LEDGER, {"op": "search", "query": "compression", "urls": [_ARXIV_PAPER]}]
t = build_trail(ledger, "as https://arxiv.org/pdf/2401.12345.pdf reports, it holds")
assert t.cited_not_opened == ["https://arxiv.org/pdf/2401.12345.pdf"]
assert t.cited_never_surfaced == [], "a listed paper cited in another of its forms is not fabricated"
def test_a_repository_cited_in_the_owners_other_casing_is_the_same_repository():
"""The forge redirects to the owner's own casing, so both spellings are one
page and a run that opened either read it."""
ledger = [*LEDGER, {"op": "fetch", "url": "https://github.com/microsoft/LLMLingua", "chars": 8200, "ok": True}]
t = build_trail(ledger, "the code at https://github.com/Microsoft/llmlingua does it")
assert t.cited_not_opened == [], t.cited_not_opened
def test_the_repository_fold_stops_at_the_repository():
"""A file inside a repository is served case-sensitively. Folding the whole
path would let an invented ``RESULTS.md`` absolve itself against a real
``results.md``, which is a fabricated deep link passing silently - the one
failure mode this check may not have."""
opened = "https://github.com/o/r/blob/main/results.md"
ledger = [*LEDGER, {"op": "fetch", "url": opened, "chars": 8200, "ok": True}]
t = build_trail(ledger, "the numbers in https://github.com/o/r/blob/main/RESULTS.md say so")
assert t.cited_never_surfaced == ["https://github.com/o/r/blob/main/RESULTS.md"]
def test_a_hub_dataset_folds_at_the_pair_that_names_it():
"""Only a model sits at the bare owner/name pair; a dataset carries its kind
first, so the naming pair is one segment deeper and a fold measured from the
root would leave the owner's casing to accuse."""
opened = "https://huggingface.co/datasets/THUDM/LongBench"
ledger = [*LEDGER, {"op": "fetch", "url": opened, "chars": 6100, "ok": True}]
t = build_trail(ledger, "the card at https://huggingface.co/datasets/thudm/longbench lists them")
assert t.cited_not_opened == [], t.cited_not_opened
def test_a_query_value_is_held_out_of_the_fold():
"""``_norm`` keeps query strings because ``?id=2`` is a different document,
and the case fold must not quietly take that back: two configs of one dataset
are two pages, and citing the one nobody opened is still a citation to a page
nobody opened."""
opened = "https://huggingface.co/datasets/o/D?config=Wikitext"
ledger = [*LEDGER, {"op": "fetch", "url": opened, "chars": 6100, "ok": True}]
t = build_trail(ledger, "see https://huggingface.co/datasets/o/D?config=wikitext for the split")
assert t.cited_not_opened == ["https://huggingface.co/datasets/o/D?config=wikitext"]
def test_a_search_results_tracking_parameter_does_not_make_a_citation_unseen():
"""Google lists a shop page with ``?srsltid=...``; the model cites it without.
That is the page the search returned, so it is unopened, not never surfaced."""
listed = "https://www.shop.example/glasses/display/?srsltid=AU7gw4Wc0iu9"
ledger = [*LEDGER, {"op": "search", "query": "display glasses", "urls": [listed]}]
t = build_trail(ledger, "see https://www.shop.example/glasses/display/ for the price")
assert t.cited_not_opened == ["https://www.shop.example/glasses/display/"]
assert t.cited_never_surfaced == []
def test_dropping_tracking_parameters_keeps_the_ones_that_name_the_page():
opened = "https://d.example/item?id=2&utm_source=x&gclid=y#notes"
ledger = [*LEDGER, {"op": "fetch", "url": opened, "chars": 6100, "ok": True}]
assert build_trail(ledger, "see https://d.example/item?id=2#notes").cited_not_opened == []
t = build_trail(ledger, "see https://d.example/item?id=3#notes")
assert t.cited_never_surfaced == ["https://d.example/item?id=3#notes"]
def test_an_unlisted_host_keeps_every_character_of_its_path():
"""Neither fold is a general rule about URLs. Both are claims about what two
named services serve, and a host that made no such promise gets the module's
original normalisation - scheme, host case, one trailing slash."""
opened = "https://c.example/Corpus"
ledger = [*LEDGER, {"op": "fetch", "url": opened, "chars": 6100, "ok": True}]
t = build_trail(ledger, "see https://c.example/corpus for the split")
assert t.cited_never_surfaced == ["https://c.example/corpus"]
def test_the_fold_decides_whether_a_page_was_read_not_how_often_it_was_cited():
"""``urls_cited`` still counts the references the answer wrote, so a paper
cited at two of its addresses is two.
Deliberate, and the narrower of the two readings of "normalisation". The
counters here have published readings - the 2026-09-04 baseline records 39
cited against 10 unopened - and folding the denominator would move every one
of them for a reason that has nothing to do with the check this fold was
added for. What the fold settles is whether an address was read; how many
times the report pointed at it is the report's own doing.
"""
ledger = [*LEDGER, {"op": "fetch", "url": _ARXIV_PAPER, "chars": 41002, "ok": True}]
t = build_trail(ledger, f"see {_ARXIV_PAPER} and https://arxiv.org/pdf/2401.12345.pdf for it")
assert t.counters()["urls_cited"] == 2
# Both forms reach the same opened page, so neither is accused.
assert t.counters()["citation_grounding_rate"] == 1.0
# ── a version is not an address, and a namespace nests ────────────────
# Both from review. The fold's job is to let one document reach the check under any of
# its addresses; neither of these was an address of the fetched document.
def test_two_pinned_arxiv_versions_are_two_documents():
"""arXiv keeps every version retrievable, and a later one may carry corrections,
expanded content, a translation, or content that changed entirely. Folding the
version away let a citation to something the run never read pass the grounding
check, which is the one thing this check may not do."""
ledger = [*LEDGER, {"op": "fetch", "url": "https://arxiv.org/pdf/2401.12345v1.pdf", "chars": 41002, "ok": True}]
t = build_trail(ledger, "as https://arxiv.org/pdf/2401.12345v2.pdf reports, it holds")
assert t.cited_not_opened == ["https://arxiv.org/pdf/2401.12345v2.pdf"]
def test_a_bare_arxiv_address_and_a_pinned_one_stay_distinct():
"""Neither direction proves the run read what the answer cites.
A bare address serves the latest revision, and this ledger records the URL a fetch
REQUESTED rather than the revision that address resolved to. So opening `/abs/X`
while the paper stands at v3 says nothing about a citation to `v1`, and opening `v1`
does not make a bare citation - which sends the reader to v3 - the same document.
An earlier form of this fix matched them whenever at most one side pinned a version,
on the reasoning that an unversioned address names the paper rather than a revision
of it. That is true of the address and irrelevant to the check: what has to be
established is which bytes were read. Re-enabling the match needs the effective
revision in the ledger, not a rule about which form is more general.
"""
opened_bare = [*LEDGER, {"op": "fetch", "url": "https://arxiv.org/abs/2401.12345", "chars": 41002, "ok": True}]
assert build_trail(opened_bare, "see https://arxiv.org/pdf/2401.12345v1.pdf").cited_not_opened == [
"https://arxiv.org/pdf/2401.12345v1.pdf"
]
opened_pinned = [*LEDGER, {"op": "fetch", "url": "https://arxiv.org/pdf/2401.12345v2", "chars": 41002, "ok": True}]
assert build_trail(opened_pinned, "see https://arxiv.org/abs/2401.12345").cited_not_opened == [
"https://arxiv.org/abs/2401.12345"
]
def test_a_pinned_version_still_folds_across_the_path_prefixes():
"""The equivalence that is provable from the address alone survives, with the version
pinned on both sides just as with it absent on both."""
ledger = [*LEDGER, {"op": "fetch", "url": "https://arxiv.org/abs/2401.12345v2", "chars": 41002, "ok": True}]
assert build_trail(ledger, "see https://arxiv.org/pdf/2401.12345v2.pdf").cited_not_opened == []
assert build_trail(ledger, "see https://arxiv.org/html/2401.12345v2").cited_not_opened == []
def test_the_prefix_and_extension_equivalence_survives_the_version_rule():
"""The original fold still holds where no version is pinned on either side."""
ledger = [*LEDGER, {"op": "fetch", "url": "https://arxiv.org/abs/2401.12345", "chars": 41002, "ok": True}]
assert build_trail(ledger, "see https://arxiv.org/pdf/2401.12345.pdf").cited_not_opened == []
assert build_trail(ledger, "see https://arxiv.org/html/2401.12345").cited_not_opened == []
def test_a_nested_gitlab_namespace_folds_the_whole_project_path():
"""GitLab groups nest, so the project is not reliably two segments deep.
`gitlab-org/quality/testcases` is one project three deep, and a fixed depth of two
left `TestCases` cased and had the check accuse a page the run had opened."""
opened = "https://gitlab.com/gitlab-org/quality/testcases"
ledger = [*LEDGER, {"op": "fetch", "url": opened, "chars": 8200, "ok": True}]
t = build_trail(ledger, "the cases at https://gitlab.com/GitLab-Org/Quality/TestCases cover it")
assert t.cited_not_opened == [], t.cited_not_opened
def test_the_gitlab_fold_stops_at_the_projects_own_boundary():
"""GitLab publishes the boundary: everything before `/-/` is the project path and
everything after is served from inside it. So a file inside the project keeps its
case, for the same reason it does on the other forges - otherwise a fabricated deep
link absolves itself against a real one."""
opened = "https://gitlab.com/group/sub/project/-/blob/main/results.md"
ledger = [*LEDGER, {"op": "fetch", "url": opened, "chars": 8200, "ok": True}]
same_project = build_trail(ledger, "see https://gitlab.com/Group/Sub/Project/-/blob/main/results.md")
assert same_project.cited_not_opened == [], same_project.cited_not_opened
invented = build_trail(ledger, "see https://gitlab.com/group/sub/project/-/blob/main/RESULTS.md")
assert invented.cited_never_surfaced == ["https://gitlab.com/group/sub/project/-/blob/main/RESULTS.md"]
def test_the_prefix_rule_does_not_quietly_restore_the_version_match():
"""Removing the bare-versus-pinned exception is not enough on its own.
`_match_form`'s unique-prefix rule matches a cited form that is a strict prefix of
exactly one opened page, for a path the model truncated when citing. `abs/X` is a
strict prefix of `abs/Xv2` and nothing else about it looks unusual, so that rule
silently put the match back after the version rule took it away - found by a test
asserting the new behaviour, not by reading the diff.
"""
ledger = [*LEDGER, {"op": "fetch", "url": "https://arxiv.org/abs/2401.12345v2", "chars": 41002, "ok": True}]
t = build_trail(ledger, "see https://arxiv.org/abs/2401.12345 for it")
assert t.cited_not_opened == ["https://arxiv.org/abs/2401.12345"]
def test_a_genuinely_truncated_path_still_matches_its_one_opened_page():
"""And the guard is scoped to arXiv version suffixes, so the rule it sits inside
keeps working: a mid-segment truncation is still a truncation."""
rows = [{"op": "fetch", "url": "https://ex.example/reports/2026-industry-alpha", "chars": 10, "ok": True}]
t = build_trail(rows, "see https://ex.example/reports/2026-industry for it")
assert t.cited_not_opened == []