## Summary
raven-research reports were written for a grader: fixed `## Answer` /
`## Findings` / `## Limitations` headings, HTTP codes beside links, and
a
chat reply that drifted from the file saved with `write_file`. This
branch
ships them in a reader layout and closes the gaps that product runs on
DeepSeek showed between the brief, the reply and the saved file.
Reader layout (`finalShape.reportLayout = "reader"`, on for
raven-research):
- The report opens with the answer as a blockquote, with no label or
bold
lead-in. Anything the brief asks the report to open with (a retrieval
date, the kinds of sources) follows in the blockquote. Delivery facts
the
brief asks back (path, counts) go on its last line, which the chat reply
keeps and the saved file leaves out.
- The body sits under its own `##` headings, using the brief's section
names word for word, numbering included, when it gives them. The last
section names what could not be verified (`## Limitations` in English;
the reply's language otherwise, with no English word in that heading).
The shape gate reads the last `##` section as the limits when no word it
knows names them, since no word list covers every language, and when
only the first heading has such a word with no body above it (a body
heading on export limits is body, not limits). The audit reads it the
same way and counts a body written as prose under the answer.
- Prose cites through markdown links, with no fetch status beside them.
A
brief that asks for link checks gets one table before the limits,
listing
every cited URL as fetched, with its status, marked blocked where the
site blocked the fetch.
- The layout is derived from the deep clause, so the evidence rules stay
one text. The shape gate, its reminder, rewrite prompt and interrupted
fallback, and `scripts/research_report_audit.py` read it as the same
three parts. A test pins the audit's limits matcher to the gate's.
Chat and file are one text: at turn end the delivered reply is written
over
the report file the turn wrote. Which file that is:
- A markdown file the turn wrote whole, or appended to or edited when
the
turn wrote it whole first or it still holds exactly what an earlier turn
of the session left in it. The session record keeps the sha256 the
session left in each report file, checked at turn start before any tool
runs. Any other file (a log the user keeps, a report the user changed
between turns) holds someone else's text, so it is never the report. A
later turn that only edits a report nobody else touched still syncs it.
- Whose `##` headings are at least half the reply's. On the
reader-layout
product runs a report draft shared 75-100% of its headings with the
reply, and a notes file saved beside the report shared none, so a notes
file or a source index is never overwritten, alone or beside the report.
- Of several such files, the one the reply names; when it names none or
more than one, every file is left alone.
The file keeps its `#` title when the reply has none. It is also left
alone
when the reply is not a report, or when the reply is under 80% of the
file's length, so a summary reply never replaces the report. The path is
read from the first line of the tool's result only, matched against the
tool's whole fixed wording, and the file tools now
refuse a path that carries a line break or other control character: they
echo the path into their results, so such a path could forge a second
result line naming a file outside the workspace.
Length and counts, measured rather than estimated:
- Every markdown write or edit gets a `[report length: ...]` line on its
tool result, carrying:
- CJK characters and other words, with prose and tables apart;
- the same counts per `##` section;
- the number of distinct URLs cited, extracted the way the research
trail extracts them.
- The report checks a brief's length against that line, and takes any
count it states (sources included) from it.
- `write_file` now reports bytes. Its success line said "N bytes" and
counted characters, so a Chinese report was sized at about a third of
its real length.
Fetch status and the research trail:
- The research fetch tool reports the target's own status from the Jina
reader's header instead of the reader's 200, and marks a CAPTCHA wall as
`blocked`.
- The trail counts such pages as refused by the site, not as read, per
fetch: a page refused once and read on a retry counts as read. When a
thin page is re-read at another address, each address keeps its own
status: the stub its own, each fallback the one in its own header.
- The trail drops click-tracking parameters (`utm_*`, `srsltid`,
`gclid`,
`fbclid`, `msclkid`) before matching a citation.
- Its `<details>` folds are now plain labelled lists, since a terminal
shows the tags raw.
- `scripts/research_report_audit.py` reads the refused-page clause and
note. It also reads the reviewer verdict when the trail names the
reviewing model. That second break predates this branch: a model id
carries `/` and `-`, so the verdict read as missing on every product
trail.
Other:
- The research reviewer now runs on DeepSeek in thinking mode. That
provider refuses a named `tool_choice` with HTTP 400, so every review
failed open and each report shipped unreviewed. The gate retries with
`tool_choice=auto`.
- `measuredGuidance` is off in the shipped config. Its numbers were
measured on a short-answer benchmark and pull a long report toward a
terse answer.
- The appendix twin's guard names the twin's lead over the frozen dr@3.5
record instead of requiring the two to be equal: 4 added and 5 changed
definitions, with `ResearchTrail` compared member by member so its
untouched members stay pinned. The record itself is unchanged from
`main`.
- The product label moves to the `-reader` suffix. Its digest was
re-stamped several times on this branch only; `main` never shipped an
intermediate digest.
Checked and deliberately not changed here:
- The main agent's own `web_fetch` (`raven/agent/tools/web.py`) also
reads through Jina and still reports the reader's 200 for a target
error. It is a separate module, not a twin of the research tool, so it
needs its own change.
- `fetch_result_ok` is unchanged, so a page the site refused still
releases the fetch gate, counts toward the evidence floor and enters the
conversation memo as a source. Changing it would move when the gate
releases.
- The report sync writes the file directly, not through `write_file`, so
it
emits no file-change event and a client's diff panel does not show it.
raven-research runs with no write approval, so no approval is bypassed.
- Every CAPTCHA flag in 36 local fetches that carried a reader warning
was
on a page of 862 characters or fewer. Target errors appear on pages of
up to 18k characters, and each one checked was a real error page. No
false positive was found.
## Type
- [ ] Fix
- [x] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other
## Verification
Run after rebasing onto the current `origin/main`. The range-diff
against
the pre-rebase branch is `=` for all 15 commits.
- `uv run pytest tests/test_agents_research_*.py
tests/test_research_*.py tests/test_filesystem_write_modes.py
tests/test_agent_tools_web_fetch.py tests/test_history_trimmer.py
tests/test_context_engine*.py -q`: 1070 passed.
- Full suite before the rebase, `uv run pytest tests -q`: 27745 passed,
118 skipped, 12 failed. None of the 12 were introduced by this branch:
- 11 fail identically in clean worktrees of `origin/main` and of the
branch head. They are environment-bound: proxy redaction, NAT64
addresses, the node path.
- `test_agents_ppt_launcher` passes in the clean branch worktree and
failed only because of an uncommitted local edit to
`agents/raven-ppt/subagent.json`.
- Mutation checks: each of these, reverted alone, turns its new tests
red:
- the target-status override;
- the refused-page split;
- the ledger classification;
- choosing the named report file (3 tests fail with `paths[-1]`);
- the audit's refused clause;
- the audit's ` via ` terminator.
- After the second review round:
`uv run pytest tests/test_agents_research_*.py tests/test_research_*.py
tests/test_filesystem_write_modes.py tests/test_filesystem_tools.py
tests/test_deliver_files_tool.py tests/test_rpc_files.py
tests/test_rpc_turn_send.py tests/test_tool_registry_execute.py
tests/test_tools_workdir.py tests/test_agent_tools_web_fetch.py -q`:
1238 passed. Each fix, reverted alone, turns its new test red: the
control-character refusal, the first-line read, the append rule, the
last-section limits (gate and audit apart), the fallback verdicts, and
the URL count, the per-fetch refusal, the draft-heading filter and the
session-wide edit rule. The twin guard fails on an undeclared definition
and on a
stale lead entry.
- After the third review round, on the rebased head:
`uv run pytest tests/test_agents_research_*.py tests/test_research_*.py
tests/test_filesystem_write_modes.py tests/test_filesystem_tools.py
tests/test_deliver_files_tool.py tests/test_rpc_files.py
tests/test_rpc_turn_send.py tests/test_tool_registry_execute.py
tests/test_tools_workdir.py tests/test_agent_tools_web_fetch.py
tests/test_history_trimmer.py -q`:
1278 passed. Mutants each turn the new tests red: the unanchored
unchanged-file match; the control-character guard narrowed to LF, or to
`Cc` only; each of the four success lines reworded in the real tools;
a twin-only change to an untouched `ResearchTrail` default.
- After the fourth review round, same command: 1288 passed. Mutants each
turn the new tests red: held files taken without the digest check, the
digest never recorded, held files ignored, the first-heading rule
reverted in the gate and in the audit apart, the gate's body-above
condition dropped, and the audit's prose-body count dropped.
- `make lint-python`, `make lint-imports`, `make lint-deps`,
`make lint-types`: all pass.
- `scripts/check_commit_messages.py origin/main..HEAD`,
`scripts/check_source_language.py origin/main...HEAD`,
`make check-large-files`: all exit 0.
- Product runs through the TUI host, on reports about personal agents,
AI
glasses, AI coding agents, office agents and edge AI:
- The saved file equals the reply, less the delivery line, with its
title kept.
- The stated length and source count match the `[report length]` line.
- The link-check table has one row per counted URL.
- A Gartner 403 and an HBS 405 behind a CAPTCHA came through with their
real status.
- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed
## Risk
User-visible changes:
- raven-research replies and saved reports change form: an answer
blockquote, no fixed headings, no HTTP codes beside links, and a single
link-check table.
- The report file a research turn wrote is overwritten with the
delivered
reply at turn end, except in the cases above.
- `read_file`, `write_file`, `edit_file`, `list_dir`, `deliver_files`
and
the RPC file reads refuse a path containing a line break or other
control
character (they share `resolve_path`).
Existing installs keep the contract they seeded into
`agent_memory/profile/agent.md`. The new layout reaches them only after
that file is deleted; until then the gate still accepts the old
three-heading replies. The changelog says so.
Not covered by this branch:
- The research subagent pins `contextWindowTokens` to 65536. On a host
model with a 1M window, the budget note reaches 80-136% early and cuts
research short. Left as is on purpose; it is a separate decision.
- Post-delivery behaviour of the host agent is unchanged. It may recount
or edit the report after delivery.
Rollback: revert the squash commit. To keep the code but drop the
layout,
set `finalShape.reportLayout` to `"sections"` in
`agents/raven-research/config.json`.
- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes
## Related Issues
N/A
---------
Co-authored-by: userName20260323 <270284818+userName20260323@users.noreply.github.com>
Co-authored-by: Claude (claude-opus-5-5) <noreply@anthropic.com>
agents/
Agent definitions built on installed raven, serving over raven acp.
Five shipped agents today: raven-code, raven-design, raven-oncall,
raven-ppt, raven-research. A new agent starts as raven agents new <name>
-- the scaffold command that instantiates this whole shape into a fresh
folder; BUILDING.md in this directory is the from-zero guide.
Agents that have no model key of their own inherit the host's model
binding, provider configuration, routing and reasoning effort.
Inheritance uses the same credential check as the host runtime,
including stored OAuth sign-ins such as OpenAI Codex. The child reads
OAuth credentials from the host's home, which it shares because the
host passes its RAVEN_HOME when one is set and both otherwise
normally resolve ~/.raven; tokens stay in that credential store
rather than the rendered agent config. An agent is spawned with the
login shell's environment -- raven's own environment only when that
shell cannot be captured -- so a token-location override
(CHATGPT_TOKEN_DIR, CHATGPT_AUTH_FILE) reaches it when the login
profile exports it, when the agent's stored row carries it in env (a
stored row replaces the discovered one whole), or in that fallback. A
RAVEN_HOME reaches the agent the same ways: from the login profile
while the host has none set, and from the agent's stored row even when
it has. The agent then reads that home instead of the host's.
What one agent directory carries:
-
run.py-- a stdlib-only launcher: render the agent's config (secret slots, host LLM inheritance, mode catalogue), then execpython -m raven acp --config <rendered>on the installed raven. -
config.json-- the baseline profile. -
modes/*.json-- optional per-session overlays, surfaced asacp.modesfor a client's mode picker (research,raven-oncall, andcodeship one today). An agent that ships nomodes/renders noacpblock, and the raven it execs then falls back to its own three built-in tiers -- sosession/set_modeanswers with those rather than method-not-found. An overlay'sagents.defaults.reasoningEffortis lifted onto the mode entry as the trunk'sreasoningEffortknob; the rest of an overlay is the agent's own hooks' to read. -
subagent.json-- the roster row templateinstall.pyregisters throughraven.config.update_subagents(the same pinned surface the retired vendored installers used). The inverse is the uninstall story -- delete the roster row with the interpreter that serves raven:# Unquoted on purpose: the shebang may be an `env` line, which is two words. $(sed -n '1s/^#!//p' "$(command -v raven)") -c \ 'from raven.config.update_subagents import remove_third_party_subagent as rm; print(rm("raven-code"))'The folder itself can stay -- an unregistered folder is inert. A live raven holds the roster it read at startup; restart it afterwards. Two manifest fields shape how the row is reached rather than what it runs:
"hidden": truekeeps the row off the roster the dispatching model reads (and off the WebUI's sub-agent page) while a task routed to it still runs there;"routes": [{"to": "<name>"}, ...]makes this row's backend a routing entry (agent/subagent/backends/routing.py): every caller --spawn, a DAG node, a direct chat -- runs the row, and the entry picks the implementation onrun, reading the task as the model wrote it (authored_task), never the rendering with file contents inlined. A reused instance handle stays where its transport bound it; otherwise the host's own model picks between the targets' roster lines and this row. A fronting row is only as ready as its targets: a route to a folder that is missing, unready or switched off disables the row with the reason on it, so a half-installed agent does not run the missing half's work on the wrong implementation. A route entry may also carry"owes"and"note": what the work still owes when the entry keeps it on this row instead of sending it, and what to tell this row's own implementation when that happens, appended to the task under a[host]marker. Both are the row's words, because the gate that closes a route serves every row that declares one; a route that declares neither hands the task over exactly as it arrived. Prose past a sentence goes in"noteFile": "<name>.md"beside the manifest instead -- discovery reads that file intonote, so a requirement long enough to be worth writing stays reviewable as a diff; declaring both is refused. What puts a route under that gate at all is its own declaration, and the three parts are independent:"needs": ["image_generation", "image_search"]names what the target's own pipeline cannot work without,"needsFile": ".pptx"a file the user attached that the dispatch hands over (a direct chat's media, or an attachment of the turn that the spawn task names -- a path in the text alone is not one) for the route to open, and"minTier": "max"the lowest tier the route may open at. A route naming none is dispatched exactly as routes were before the gate existed -- never probed, never asked for a file, never tiered -- becauseroutesis a general facility and a row routing for reasons of its own must not inherit conditions it never asked for. The names inneedscome from a closed vocabulary (ROUTE_REQUIREMENTS), since the host is what answers them; one outside it is warned about and treated as met, so a manifest written for a later raven keeps its route on an older one. And the probe answers for the target's lane, reading the product folder's own.envfirst (PPT_SERPER_API_KEY,PPT_IMAGE_API_KEY) and falling back to the host sections the launcher would inherit -- the host's own credentials answer for the host loop, and a lane is free to be equipped differently. It follows the launcher one branch further than the explicit keys: with no image key anywhere and an OpenRouter endpoint, the key paying for the lane's words pays for its pictures, so a folder holding onlyPPT_API_KEYstill counts as able to draw -- and towards any other gateway it does not. All four fields are manifest facts: a stored row takes them from the folder on every merge.raven-pptships hidden behindraven-design's routes, so the model sees one design agent and decks still build on the deck engine, its own model and key; the two halves share one everos identity (raven-design), so what the user says over a deck is remembered for the next design turn. ItsPPT_API_KEYis configured throughraven subagents setupor the folder's.env, since the page no longer lists it. -
plugins/<id>/-- the agent's own harness as raven plugins: hooks on the loop's six phases, replacement tools under the built-in names, its own config slice underplugins.config["<id>"].run.pynames the directory throughplugins.dirs; nothing inraven/knows the plugin exists. An agent's harness may instead ship as a standalone distribution:raven-pptandraven-designcarry noplugins/directory -- their engines are theplugins-dist/ppt-engineandplugins-dist/design-enginewheels, found through theraven.pluginsentry-point group. Each engine underplugins-dist/belongs to its agent -- but an entry-point plugin withenabled_by_default = trueACTIVATES in every raven process in the environment; staying out of the host's and the sibling agents' turns is the engine's own duty, done by factories that decline when their config slice is absent (what the scaffold's engine templates do), or by the operator'splugins.disabledlist. Prompt assets ride with the plugin whose conduct they teach, underplugins/<id>/prompts/: the workspace guiderun.pyseeds once (oncall's TOOLS.md section, code's whole-file TOOLS.md) lives there.raven-codealso seeds its identity assoul.md(besiderun.py, the raven-research shape) and a coding conduct asagent.md(one variant per model family, the fork's split; a partition still carrying the other variant or older managed seed is refreshed, an operator's edit never is), and its code-flow hook contributes the working directory's own instruction files (AGENTS.md/CLAUDE.md/CONTEXT.md, the slice'sprojectFiles) to each turn's system message through the context assembler, without rewriting the query. The same plugin carries one more surface:code-flow/code_flow/tools/-- the product's own tool face, contributed by the same plugin: the fork's spelling (file_path/old_string, with the host's names accepted as aliases),glob,todo. Four of them ride the built-in names, which is how a product serves its own file tools without touchingraven/agent/tools/or the four other products that share it: plugin tools register last and a same-name registration replaces the built-in instance, so the model is only ever shown this one. Disabling the built-in name instead is not the same thing -- withholding is by name and would take the replacement with it. The face is a section of the flow slice (tools), with its own switch and arestrictToWorkspacethe launcher renders from the product'stoolsblock: a plugin factory cannot read that field, and a replacement built without it is fenceless whatever the product asked for. The ask tier is decided per hosting, not product-wide: the ACP hosting keeps trunk'sask(raven dispatching a sub-agent answers those prompts itself, and a person in an editor should still be asked), and the one-turn CLI hosting renderspermissions.mode: fullbecause that hosting has no channel to ask on -- with the default it refused every write and the model reported the task incomplete.
-
soul.md+ the contract the plugin renders intoagent.md-- optional: the agent's own identity, seeded into the workspace once, for an agent that replaces the host identity (research does; itscontext.dropSegmentskeeps the host's own identity segment out of the prompt). An agent whose vendored twin served the host-generic identity carries nosoul.mdand keeps the host identity segment -- oncall and design: each of those forks' SOUL.md is byte-identical to trunk's own template, its ACP path never seeded it into a workspace, so an added identity file would change the very prompt face the parity tests pin.raven-codeis the exception with a reason: its one-turn CLI hosting runsraven agent, whose workspace sync writes trunk's templatesoul.md-- a personal assistant with a personality -- in front of the coding conduct, while its ACP hosting (no sync) read none; seeding its ownsoul.mdfirst gives both hostings one identity and keeps the host identity segment.raven-pptalso carries nosoul.md, for a different reason: its fork's SOUL.md is its own, and it rides byte-for-byte at the engine wheel's prompts home (raven_ppt/prompts/), seeded into the pinned home by the engine plugin's hook at first turn.
Agent vocabulary:
- Rendered config -- what
run.pywrites and execs against: the baselineconfig.jsonafter secret slots, host LLM inheritance and the mode catalogue are applied. Rendered files live under the state root, never in the agent directory. - State root -- where the agent keeps its work (repos, instance
buckets, flow stores, rendered configs):
product_state_root()inraven/config/product_render.py, default<raven home>/workspace/subagent_sessions/<agent>, overridden by the agent's*_STATE_ROOTvariable. - ACP home -- the engine's own Agent home,
product_acp_home(): under the raven data directory, never inside the host's Agent home (the host hands its home out as a session working directory, and a raven engine refuses a working directory that contains its own home). The agent's*_ACP_HOMEvariable overrides it outright. - Tool-face pin -- each launcher test pins the exact tool face the
rendered config exposes (
VENDORED_TOOL_FACEintests/test_agents_<agent>_launcher.py) to its vendored twin's face. - Seed-once -- workspace assets (guides, identity) are written only when
absent (
seed_once()inraven/config/product_render.py), so a user's later edits survive relaunches.
Agent notes:
raven-designgives each session its own working directory under<workdir>/designs/<session-slug>-<hash>/, following the deck agent's per-session directory mechanism. The hash uses the full session key so channel names, normalized characters and truncated names remain distinct. Task State is still stored under the configuredtaskState.stateRoot, keyed by this session directory; relative file and render paths use the same directory. The reply ends with that directory's absolute path and the note that its paths resolve against it: a caller dispatched the run into the directory above, which is what its dispatch receipt names, so it would otherwise resolve the reply's relative paths one level too high and deliver nothing. A symlink used asdesigns/or as the session directory stops the turn before model or tool execution, preserving the original working directory's access boundary. The caller's working directory may itself be a symlink; its resolved location is the root. Resuming a session with the same working directory returns to its files and Task State. Setplugins.config["design-engine"].workdirPerSessiontofalseto keep the caller's directory directly. Existing files and Task State from the shared directory are left there: their owning session cannot be inferred safely.raven-ppt(the deck agent) needs the ten bundled templates theppt-engineplugin offers, and gets them from the clone: they are tracked underplugins-dist/ppt-engine/raven_ppt/assets/templates/, which the editable install serves and the ppt-engine wheel bundles. Nothing to fetch and no token to hold -- the registry they used to be fetched from is private, and requiring it left an outside install with an empty catalogue and a wheel carrying zero templates.plugins-dist/ppt-engine/templates.manifest.jsonstill pins every file by sha256;make verify-templateschecks the tracked copies against those pins,make fetch-templateslands a fresh cut over them (maintainers,GITLAB_TOKEN), and release.yml refuses a wheel that does not carry all ten.
Ground rules:
- The A side of the comparison is the retired vendored tree: byte snapshots
of its record live under
tests/fixtures/vendored_fork/, and the full trees remain in git history (the commit that removedsubagents/is the anchor). Nothing here may re-grow a dependency on that tree. - The runtime never imports this directory -- enforced by the import-linter contract "the runtime does not import the agents pilots".
- In the wheel as data, never as code:
hatch_build.pymaps the tracked files toraven/agentsfor the roster's file-level discovery, and the import-linter contract still keeps the runtime from importing any of it. - Acceptance for a shipped agent is transport-face equivalence against its vendored twin: transcript shape, timeout semantics, everos records field by field.