With compression.prepare_ahead, compress() re-planned the tail cut and threw the
completed background summary away whenever the fresh cut landed before the
summary's boundary (the no-LLM prune made tail rows cheaper, tool-group
alignment moved the cut), then wrote a second summary inline while the user
waited. That breaks the one-summary rule: once the background summary exists it
is used, and everything after its boundary is the tail.
The splice check no longer consults the fresh cut at all. A summary is still
discarded only when the messages it covers changed (rewind, shape change, other
session, moved head). The pass planned its own window with a full tail, so the
tail is never empty. The now-unreachable inside_tail reason leaves the
(unreleased) prepared-compaction metric contract.
Greptile on 815681917c3:
- a boundary landing after the pass published (and cleared its pending token) but before the
waiting compaction read the summary still read as a failed pass; take() now notices a discard
that happened during its wait and leaves the boundary's row as the only one;
- compaction_failed was lost when a pre-commit cancel restored the compressor's attempt telemetry
(and its prepared_summary flag) before the record was emitted; the spliced attempt id now lives
on the prepared-compaction state, outside the restored snapshot;
- the deadline-extension test starts its 2 s deadline only once the pass is running.
Re-review findings:
- a summary take() handed to compress() that the compaction then aborted or no-opped before
adopt() spliced it was counted nowhere; it is now counted once at the next touch as
discarded/compaction_aborted;
- a session boundary landing after the pass published but before the waiting compaction woke
read as a failed pass (a second discarded row); the wake now reads the entry and the cancel flag
under one lock, so the boundary's row is the only one.
Live runs (flag on and off) show the next compaction retries the configured summary model before
falling back again, so the comment no longer claims the fallback is sticky for the session.
Independent review of the telemetry found:
- a session boundary that woke a waiting compaction was counted twice (the boundary's discard
plus the woken wait's 'discarded_while_waiting'); the boundary's row is now the only one and the
dead reason code is dropped from the vocabulary;
- 'used' was recorded in take(), before a stale-attempt abort could still drop the summary; it is
now recorded by adopt(), the splice itself, with the wait carried on the entry;
- the cancel reason is set under the lock, so a pass finishing in that gap reports the real reason;
- skipped_busy counts once per refused window instead of every tool batch.
The live matrix on real processes found two ways a prepared-compaction pass was paid for and never
usable:
- `hermes chat -q` / `-z` runs end with their turn (the gateway evicts the agent right after the
reply), so a turn-end pass there was always cancelled ~0.5 s later. One-shot sessions now join
cron runs and delegated children in skipping turn-end passes.
- Releasing an agent without a session boundary (gateway cache eviction, a Desktop rewind,
managed-worker retire, close()) left a running pass billed to completion. release_clients() and
close() now discard it (shared-metrics reason `agent_released`).
Pass log lines now carry the session id, so concurrent sessions in one gateway are attributable.
Wire hermes.prepared_compaction.count into the prepared-compaction state
machine so a profile that opts in to shared metrics reports whether its
passes pay off, and if not, why:
- pass side (pass thread): pass_started / skipped_busy (process slot
cap) / pass_completed (summary published) / pass_failed (classified
as an inline attempt would be, terminal summary failure first) /
pass_cancelled (reason set by whoever cancelled: superseded,
session_boundary, wait_timeout, host_cancelled) / internal_error.
- compaction side: used (wait bucket, aux_fallback_main when the pass
fell back to the main model), discarded (closed miss code + wait),
auto_disabled (first memory-provider compaction), and compaction_failed
from the attempt telemetry when a spliced compaction did not commit.
Splice-miss reasons become a str subclass carrying a closed code, so the
log text is unchanged. The pass thread already copies contextvars, so its
rows land in the owning profile.
Review follow-up (Greptile on 41ac847b3ee): caller-supplied conversation_history
that is never persisted carries no message_uid, so the structure-only check
accepted a dropped user/assistant pair once the transcript regrew to the same
shape, and the old summary replaced messages it never summarized.
Each message identity now also carries a text digest. A message whose uid is
present on both sides is still matched by uid alone, so plugin text rewrites
of persisted messages keep the summary; a message without a uid on one side
is matched by text.
Maintainer ruling on the prepared-compaction design, plus the review findings
(ours and Greptile's) on #144684:
- One pass per compaction. The extension passes (a new summary call every ~8k
tokens of growth) and the "stale after one tail budget" check are gone: the
summary splices at its own boundary and every message after it, however
many, stays verbatim in the tail. No second summary is ever paid for.
- The splice check compares message structure, not text: role, tool call
ids/names, tool result ids, plus the durable message_uid where both sides
carry one. Plugins that rewrite text in place (redaction, pre-compaction
notes) no longer throw the summary away; a dropped/inserted message or a
rewind into the window (new message_uids) still misses. This also settles
the fingerprint gap on input_text/output_text parts: text is not compared.
- A pass that recovered on the main model after the summary model failed now
hands that over on a splice, as an inline call would: summary_model goes
sticky to the main model, and the "configured compression model failed"
warning reaches the user. The attempt telemetry carries the pass's aux call
(model, tokens, duration, fallback) and a new prepared_summary flag.
- The first compaction that needs memory-provider context stops further
passes for the agent; each would have been discarded and paid twice.
- Turn-end passes are skipped for cron runs and delegated children, whose
sessions end with the turn.
Round-two review of the cancel and fast-path commits:
- A cancelled pass's stream raises AuxiliaryExplicitCancellation, a
BaseException that _generate_summary and _run_pass did not catch, so
every superseded pass printed "Exception in thread compaction-prepare"
to stderr (seen in the live probe). _run_pass now catches it and logs
the cancel.
- take() snapshotted the candidate at its start; a pass that published
while compaction was checking that snapshot survived the compaction.
It never spliced (its fingerprint no longer matched), but the
prompt-size skip could then keep it and start no pass for a cycle.
take() now clears every candidate on exit.
- The prompt-size skip measured growth since the plan, so once the
prompt grew past the threshold while the window gap stayed small,
every tool batch re-planned again. Keeping a candidate after a full
check now re-arms the skip from that check.
Tests: one stub models the aux client ending a cancelled stream, the
local wait helper gives way to tests.waiting.wait_until, and the
skipped small-window-delta path gets its own test. Docs and the config
comment name the discard case.
While a completed candidate was still fresh, maybe_prepare re-planned the
whole compaction window and re-fingerprinted the transcript before every
tool batch and at every turn end, only to keep the candidate. Measured on
the turn thread: ~23 ms per call at 180k tokens, ~31 ms at 540k.
The candidate now remembers the provider-reported prompt size its pass
was planned at; while the prompt has grown by less than the extension
threshold since then, the hook returns at once. A shrunk prompt or a
larger growth falls through to the full plan, and compaction still runs
the authoritative fingerprint and gap check.
Passes are also skipped while the summary model is benched after two
fallback summaries in a row: inline compaction would not call it then
either (#63008), and each pass would be a paid call.
A pass that compaction or a session boundary superseded kept streaming
its summary to the provider's own ceiling: billed, then thrown away. The
same happened when a waiting compaction gave up at the host deadline or
on a host cancel, so the summary was paid for twice (once in the
background, once inline). Live probe: after the deadline cut a wait
short, the pass still completed 4 s later and was billed (2 aux calls).
Each pass now carries a cancel event, wired into the existing aux
hard-cancel path (aux_interrupt_protection(cancel_event=...)) so the
stream is closed. take() sets it whenever it consumes or supersedes the
pass; discard() sets it on reset, end and rebind. A compaction waiting
on a discarded pass wakes at once instead of waiting out the pass.
Also:
- the waiting compaction installs its progress hooks before the rest of
the setup; resolving the aux timeout loads config on first use
(~0.8 s), and progress streamed in that window was dropped;
- a miss now says why: deadline, host cancel, discarded, or the pass
failed (every miss used to read "did not finish in time");
- the config comment mentions the wait.
Probe after the fix: deadline miss -> the pass exits ~0.1 s later, 1
aux call billed; reset during the wait -> the wait returns at once.
Inline, a llama.cpp server reading a long summary prompt pushes the host's
compression deadline back (compression_deadline_scope installs a
prompt_reading_hook that extends the fence). A compaction waiting on a
prepared pass lost that: the pass thread had no reading hook, and the wait
read the deadline once. A slow-but-healthy local prefill could hit the
original ceiling and fall back to an inline summary that pays the same
prefill again.
The pass now installs a reading hook that forwards to the waiting
compaction's hook, and the wait re-reads the live host deadline on every
poll.
Rebasing #120603 onto main lost a line in the conflict resolution: main's
summary success path (86bf8886c2 / 4f134d4c37) resets the sustained-overload
budget, but the extracted _record_summary_success() did not. Two overload
aborts, then a healthy summary whose compaction is refused (would-grow), left
the strikes armed, so one isolated overload later escalated straight to the
destructive fallback. The helper now clears terminal failures with the shared
helper and restarts the overload budget, exactly as main does.
The overload tests move to tests/agent/test_context_compressor_overload.py
(unchanged) so the new case fits the test-file size cap.
When automatic compaction fires with no completed candidate but a pass is
still writing a summary whose window fits, compaction waited for nothing and
wrote the same summary again inline, so the user paid the full summary
latency and the background call was thrown away.
Compaction now waits for that pass. Each streamed chunk of the pass touches
the compacting thread's aux progress hook, so the host's idle watchdog treats
the wait like an inline summary stream. The wait ends at the host's stream
deadline (the compression aux timeout when none is installed) or when the
host cancels, and compaction then summarizes inline as before. A running pass
whose window no longer fits is not waited on. Annotations in the module and
its tests move to builtin generics.
_compress_window moves unchanged, with _plan_compaction_window and the
prepared-summary splice (_take_prepared_summary), into CompactionWindowMixin
in agent/context_compressor_window.py: the window that compress() and the
prepared-compaction pass both plan now lives in one place, and
context_compressor.py stays under its file-size ratchet. config_defaults
comments in the compression section are tightened to keep that file at its
cap. The pre-existing summary catch-all in _generate_summary gets a health
waiver: the ratchet re-flags it only because the success block above it was
extracted into _record_summary_success.
_emit_compression_summary and the once-per-profile Codex gpt-5.x autoraise
notice helpers move unchanged from agent/agent_init.py to
agent/agent_init_compression_notice.py, so the prepared-compaction flag can
land in agent_init without growing the file past its ratchet cap.
- A candidate only splices when the messages between its boundary and the
fresh tail cut fit in one more tail budget; a staler candidate misses and
compaction summarizes inline.
- The preparation band is 12% of threshold_tokens, not of the context
window, so a small absolute trigger on a large window no longer starts a
pass on every turn.
- bind_session_state() discards the candidate and any running pass when it
binds a different session id (/resume, /branch); same-session rebinds
keep it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With compression.prepare_ahead (default false), once the provider-reported
prompt size is within 12% of the context window below the compaction
trigger, the turn loop starts a background pass before a tool batch runs
and at turn end. The pass plans the same window compress() plans and runs
the same summariser on a copy of the compressor and of the window rows.
The pass itself never rewrites the live transcript, system prompt or
prompt cache; the transcript keeps growing normally meanwhile.
At the next automatic threshold compaction, compress() splices the
finished candidate when the head boundary is unchanged, its boundary is no
later than the fresh tail cut, and the transcript up to that boundary is
unchanged (ignoring what the no-LLM prune rewrites). Messages after the
candidate boundary stay outside the summarized region and go through the
existing tail handling. Everything else, including a pass that has not
finished, manual/focus compaction, overflow recovery and memory-provider
summary context, compacts inline as before. A running pass is never
waited on and loses the right to publish once compaction or a session
boundary overtakes it. A failed pass backs off further passes without
touching the live failure cooldown.
State is per compressor and in memory only. Plugin context engines, Codex
app-server, native Responses compaction, micro-compaction and
background-review forks are not prepared. With the flag off no state is
created and compress() takes its existing path.
Phases 1-2 of compress() (prune old tool results, drop blank echoes, pick
the head/middle/tail boundaries) move into _plan_compaction_window so a
second caller can plan exactly the same window. No behaviour change.
The 'summarised with your main model' warning keyed on the answering model's name, so a
fallback_chain entry serving the same model id on another provider earned it too. call_llm now
reports the label of the route that answered and the compressor checks for main-agent. The two
tests that only called the new private helper are folded into one label-precedence case; the
end-to-end compress() test still proves the bug on main.
With auxiliary.compression on a dead endpoint (connection refused, quota), call_llm's own
recovery ladder answers the summary with the main agent model. The compressor only saw a
successful response, so the "configured compression model failed" warning never fired and
attempt telemetry read method=llm_summary, fallback_used=false: the user never learns their
compression endpoint is down. The compressor's own main-model fallback (#116472) already
reports both; only the ladder's recovery was invisible.
The ladder now records the route that failed and why in the caller's route_info when a
fallback candidate answers, and the compressor treats that like its own fallback: telemetry
fallback_used=true (failure_class aux_model_fallback) for any fallback route, and the
existing user warning plus method=aux_fallback_main when the main model answered (the
warning names the main model, so a configured fallback_chain entry gets telemetry only).
Live, real AIAgent._compress_context with auxiliary.compression on a closed port and a
reachable main model: base compacts with no warning and fallback_used=false; head shows
"Configured compression model 'slow-summarizer' failed, so Hermes summarised with your main
model instead" once and fallback_used=true, method=aux_fallback_main on both compactions.
/stop (or a session boundary) hard-cancels a running compression: the owner unwinds at once,
but the summary is streamed on an isolated daemon whose chunk consumer (_ChatStreamAccumulator)
checked only the total ceiling and the host deadline, never the explicit cancel source the
Anthropic and Codex stream consumers already honour. A hard cancel leaves the fence deadline in
the future, so the provider stream kept generating, and billing, until that deadline (600 s by
default), and only the process exiting closed it sooner.
The accumulator now raises AuxiliaryExplicitCancellation on the next chunk once the host cancels;
the stream's finally closes the socket on the owning thread. Shared by the sync and async
aggregators.
Live, real AIAgent._compress_context against a server trickling a chunk every 2 s, interrupt
(hard_cancel=True) 6 s in, process kept alive 40 s: base keeps the provider thread and the
socket open the whole hold; head closes the socket within one chunk and the thread exits.
The tail-cut walk charged in-transcript handoff rows as protected-tail
candidates, but the handoff scan folds every row up to the newest handoff
into the summary window: none of them can stay in the tail. When the
handoff row alone pushed the transcript over the soft ceiling, the walk
stopped on it and the planned window held only the handoff, so compress()
returned the empty_post_handoff_window structural no-op at every trigger.
Live on main (gemini-2.5-flash, context_length 64000, prepare_ahead off):
a Desktop manual compress, then one automatic compaction (handoff now at
index 0), then four consecutive empty_post_handoff_window no-ops while the
prompt grew 47,085 -> 57,143 -> 66,716 tokens, past the 64k window.
The tail-cut walk now prices rows at or before the newest handoff at zero,
so the cut is decided by the rows that can really stay in the tail and the
window covers the old handoff plus the turns after it. Same scenario on
this branch: the second automatic compaction summarises 17 rows and the
next prompt is 18,156 tokens.
The turn-budget rewrite ran rewrite_pruned_rows without the turn-lease check every append runs in its
write transaction, so a process that had lost the turn could still swap a row to the stub under the
new owner. rewrite_pruned_rows takes an optional turn_lease_holder (the proactive prune stays
lease-less by design) and the turn path passes its own; a lost lease keeps the stored full body.
Every tool result is flushed to state.db as it lands
(_flush_session_db_after_tool_progress), then _finalize_tool_batch runs
enforce_turn_budget, which swaps the largest results for
<persisted-output> previews in memory only. The gateway, TUI resume and
`--resume` rebuild history from state.db, so the next turn reloads every
spilled body in full: the request grows by everything the budget removed
and its cached prefix diverges at the first spilled row.
Live on c2c5eef1492 (gui session, openrouter gemini-2.5-flash,
context_length 64000, prepare_ahead off, 6 parallel read_file of ~28.6k
chars): burst call in=41,908 with 4 stubs, then "Reply OK" went out at
in=78,255, past context_length. Preflight logged "rough estimate ~64,889
... not anchored on real usage" and deferred; that deferral is intended
(e9313f64580: provider evidence adjudicates even past-window estimates),
so the defect is the durable row, not the gate.
sync_rewritten_tool_rows carries the end-of-batch rewrite onto the rows
through the existing in-place writer (rewrite_pruned_rows): row-addressed
by _row_id, CAS-fenced on the stored row, the full body kept as an
archived searchable twin. A store without the writer or a row another
writer moved keeps today's row. Same live repro on head: in=41,649 then
in=41,685, no deferral line; CLI --resume 60,431 -> 27,153.
Round-2 review of the fold found three more hand-written copies of the
helper the previous commit made public: the classic CLI config watcher
(exists check + try/config_version/except OSError, every 5 s; it compares
against the `_config_sig` the TUI mixin now seeds with the helper), the
MoA preset cache stamp, and skill_utils' raw-config cache key. Same
results, one stat fewer per watcher tick. The version_token test now
also asserts that the token changes when an opaque version changes.
Add hermes_cli/config_backend.py (config-config design §4.1, D10/D11): a
ConfigBackend protocol, the FileBackend (today's behaviour), selection by
HERMES_CONFIG_BACKEND only (default file; 'remote' and unknown values fail
closed with ConfigBackendUnavailable), and path-level helpers
(read_config_doc, read_config_doc_readonly, config_version, config_exists,
write_config_key, write_config_document, supports_file_tooling).
Migrate every direct reader/stat/exists/bypass writer of a user config.yaml
to the helpers (config.py loaders + caches, config_effective, env_loader,
gateway, TUI, CLI watchers, cron, tools, profiles, logging, time). File
tooling (config edit, backups, xAI migration rewrite, raw dashboard editor)
checks supports_file_tooling() first; migrations skip write-back without it.
Add scripts/check_config_yaml_readers.py (CI lint next to the writers gate)
that rejects a direct read of a config path outside the backend.
Every CLI start imports all provider plugins (hermes_cli.config lists their env vars), and
fireworks/gmi/xai/router/kimi/opencode plus auth_constants, models, model_catalog,
auxiliary_client, gemini_native_adapter, web_server_messaging, slack, yuanbao and perplexity
built a `<product>/<release>` string at module level. In a source checkout the release comes
from the 7-call git provenance walk (`git tag --merged HEAD` alone is ~0.35 s in a large
clone), so `hermes sessions list` paid it for headers it never sends.
Provider profiles take a LazyHeaders mapping (resolved on first read by the client builders,
which already copy with dict()); core modules call version_info.hermes_user_agent() where the
request is built. Header values are unchanged.
get_scratch_dir(prune=True) ran the once-per-process idle scan and rmtree inline. Its first
caller in nearly every process is the boot hook (hermes_bootstrap -> export_scratch_tmp_env ->
apply_scratch_tmp_env), so `hermes`, the TUI backend, serve, gateway and every Hermes child sat
on the prune before doing anything: a 34 GB idle entry held one CLI start for 13 s.
The prune=True path now schedules the same pass on the scratch-prune daemon thread (the
background helper from the prompt-builder fix), so the boot hook, the post --profile re-export,
the child-env builders (apply_subprocess_home_env) and the system prompt all share one
non-blocking path; the prompt builder goes back to a plain get_scratch_dir(). The directory is
still created and permissioned before the call returns, so TMPDIR is valid at once. Stamp,
once-per-process flag, 24 h idle rule, reap and audit log are unchanged; the thread gets the
path the caller resolved.
A process that exits mid-prune kills the thread with the hourly stamp already touched: entries
it never reached go on the next hourly pass, and an entry cut inside rmtree waits 24 h (the
unlinks freshen its directory mtimes), the same as a Ctrl-C during the old inline prune.
Tests: the 3 prompt-builder tests become 2 invariants (boot hook + prompt never wait and prune
once on the resolved dir; the thread prunes the routed home's dir), both red on main.
The host-hints section only needs the scratch path. It called get_scratch_dir() with the default prune=True,
so the first prompt build in a process ran the once-per-process scratch prune synchronously. On macOS under
launchd and on Windows the boot hook returns early (the OS already sets TMPDIR/TEMP), so that first build
pays the whole prune, sometimes inside an automatic compaction boundary while a user turn waits.
Ask for the path only and run the same once-per-process prune on one daemon thread, against the path the
caller resolved (threads do not inherit the profile override contextvar). The prompt text is unchanged.
An empty find_voices object is the model's way to browse the library with the
defaults, but a truthiness check sent it down the speech path ('Text is
required', or audio instead of a search). Check for presence, in the handler and
the display formatter (Greptile P1 on #144633).
A managed worker (every Kanban card, --safe-mode and --ignore-user-config
chats) built AIAgent from the frozen model.base_url with no api_mode, and
resolved the route only to fill a missing key. The in-process turn takes
api_key, base_url and api_mode from route_runtime(), so a route whose
api_mode or endpoint only resolution knows failed only in the worker:
opencode-go Responses models fell back to chat_completions (400), and
Bedrock, which stores no base_url, landed in us-east-1 where eu./apac.
inference profiles do not exist.
worker_route() runs the same route_runtime() call with the bound launch
key (if any) and returns api_key, base_url and api_mode for the agent. A
bound launch key still wins; on a fallback entry nothing is carried, as
before (#139155). Resolution stays in the worker's own profile home: no
new secret path.
Refs #143547
Co-authored-by: Kazantcev Andrey <45011689+heckad@users.noreply.github.com>
Co-authored-by: Gekiere Aurélien <11493426+aurel282@users.noreply.github.com>
On Linux psutil's create_time() is the cached /proc/<pid>/stat start ticks
plus boot_time(), and psutil 7.2.2 re-reads /proc/stat btime on every call.
WSL2 re-anchors btime after host sleep, so the owner's read of a live
worker's birth drifted from the one the worker reported in its hello, and
the exact-equality fences (reservation, worker.adopt/persist, delegation
dispatch, the launcher signal and the Kanban owner receipt) refused it:
every Kanban card, cron run and subagent ended permission_denied.
worker_birth() reads the kernel start ticks since boot
(create_time(monotonic=True)) on Linux and keeps process_birth() elsewhere;
every managed-worker identity fence now uses it. The spawn ledger keeps
process_birth() so its persisted epoch births still match after an update.
An owner_admitted receipt written before this change reads as another
process: its reclaim pauses, the fail-safe direction.
Refs #143547
Co-authored-by: Sebastian Hänisch <5960303+TKCen@users.noreply.github.com>
Teknium: the ElevenLabs variant of text_to_speech is where voice search belongs, not a
separate tts_voices tool. The ElevenLabs schema now carries voice_id plus a find_voices
object (text optional there); other providers' schema is byte-identical to main.
Live search findings folded in: the library is the default source and sorts by a year of
usage (most-used first); use_case (e.g. characters_animation) and sort filters; library
voices with a custom credit rate show credit_rate; the description tells the model the
library matches words literally, so a role word that finds nothing ('dwarf') falls back to
describing the sound ('gravelly old' + filters).
With tts.provider: elevenlabs, text_to_speech gains an optional voice_id and a new
tts_voices tool searches the user's account voices (/v2/voices) or the public
Voice Library (/v1/shared-voices) with query/gender/age/accent/language filters and
an opaque next_page cursor. Both are gated on the ElevenLabs provider (dynamic
schema override + check_fn), so every other provider's surface is unchanged. A
per-call voice_id never rewrites config; a stale one sent under another provider is
ignored with a warning.
The popover was empty for any session without a resident agent and counted
pre-compression history. Both reads now use the compression tip, and an idle
session is estimated from its stored prompt, tools pin and messages.
Rows RT-72, RT-75.
Refs #143547
Co-authored-by: Asotora <andreyw441@gmail.com>
custom is the billing class every named endpoint reports, not a route. Launch
policy, stored rows, synced runtimes and /status now carry custom:<name>,
matched by base URL and model in the owning profile. Also fixes the model-switch
crash on a profile-following session and accepts --source desktop as gui.
A client asking for a gateway-only label (cron, kanban, bot_room) is refused as
"reserved for the gateway", and a Kanban worker's descendants no longer inherit
its kanban label, so `hermes chat -q` from a task shell starts normally.
Rows MP-19, MP-20, MP-23, MP-25, MP-28; CLI-43.
Refs #143547
_emit_diagnostic_wait goes through _emit_wait_notice, which touches
activity itself, so kills past the limit still refreshed the cron and
delegate inactivity watchdogs even though the explicit touch was held.
Gate the notice with the touch. The inline test now asserts that no
activity is reported at all once the hold starts, instead of filtering
on the kill's own description.
Review changes:
- Replace the HERMES_STREAM_STALE_ABANDON_KILLS env var with the module
constant _STALE_ABANDON_KILLS = 3 (contribution rubric: no new HERMES_*
env vars for non-secret behaviour) and drop the "0 or less" branch.
- Inline calls (cron turns, delegated children) have no worker to abandon.
Past the limit the monitor now stops touching activity from the stale
kill and the 30 s heartbeat, so the cron and delegate inactivity
watchdogs can end the job. A real chunk, or a new attempt, lifts the
hold. This follows the inline half of #141399 (@suhas-svg).
- run() reads the outcome once into locals instead of rebinding
self.result, which the abandoned worker could still write through.
_partial_stream_stub takes the error as an argument, so a stub after
visible text is classified from the abandon error.
- The abandon path makes the same bounded Relay join as /stop (#81521)
before the in-turn retry opens a new scope.
- Module docstring names the stale-kill follow-up and the shared
invisible-delta check.
Tests: drop the late-write and turn-level tests (they pinned the rebind
and main's existing TimeoutError retry); add the inline-liveness, Relay
join and visible-text stub tests; note in the control test that it relies
on conftest zeroing jittered_backoff.
Addresses review on #144201:
- HERMES_STREAM_STALE_ABANDON_KILLS=1 now abandons on the first kill instead of
waiting a second stale window (count compared directly with the limit; 0 or
less disables abandoning).
- An abandon after deltas that delivered no visible text (whitespace or
thinking only, no tool call in flight) is now treated as undelivered and
retried, not turned into an empty partial stub. The existing #112419 check is
moved into StreamingWaitMonitor._forget_invisible_deltas and shared by the
stream-error path and run().
When a stale stream's worker is parked somewhere the socket shutdown
cannot reach, the monitor re-killed the same attempt every stale window
forever: each attempt counts toward the stale streak once, the give-up
breaker only runs before a call, and the kills keep the activity
watchdogs fed, so only /stop ended the turn.
Track consecutive kills of the same dispatched attempt. After
HERMES_STREAM_STALE_ABANDON_KILLS (default 3) with the worker still
alive, leave the daemon worker behind, as the /stop path already does,
and end the call with a retryable TimeoutError. A late write from the
abandoned worker cannot replace that error. Inline mode (no worker
thread) is unchanged.
Two more ways a held turn (hold_stream=True transform_llm_output) left history
out of step with what the user got:
- When verify-on-stop or pre_verify sent the answer back on the last allowed
call, _append_interim_answer had already flushed it as a hidden row. The
finalizer reused it as the budget fallback and transformed it, but the row
was settled: _store_rewrite_on_tail skipped it and _close_transcript_tail
added nothing (the tail already held assistant text), so reopening the
session lost the reply. A held turn's sent-back answer now flushes with the
turn instead of at once (the user never saw it), and when the turn ends on
it _store_rewrite_on_tail publishes that row: visible, content = the
rewrite, api_content = the bytes the model wrote and already saw, so the
wire and the cached prefix stay unchanged. One row, so role alternation
holds.
- A codex_responses reply can carry commentary and the final answer in
codex_message_items. Publishing the held final unhid the whole row, and
project_history_commentary rebuilt display_commentary from those items on
reload. publish_held_final_row now stamps display_metadata
commentary_withheld on such a row; the projection then shows no
commentary for it, while the items stay for model replay.
Also: the plugin-reload hold test waited 0.5 s for a negative check; it now
signals that the read started and waits 2 s (tests/AGENTS.md).
Tests (red on 72fd611fef): a pre_verify fallback ending a held turn (history
shows the rewrite, the resumed request replays the original bytes), and a
held codex_responses final with a commentary item (FakeResponsesServer
Message gains an optional phase).
Review: https://github.com/NousResearch/hermes-agent/pull/144266#discussion_r4370905634
Review: https://github.com/NousResearch/hermes-agent/pull/144266#discussion_r4370905635
Review: https://github.com/NousResearch/hermes-agent/pull/144266#discussion_r4370905636
The hidden-row step covered only the tool-call row of run_tool_round.
Other assistant rows a held turn appends and flushes without showing them
were stored displayable, so reopening the session (TUI/Desktop
display_projection, dashboard, classic CLI resume) showed the original text
even when the final was rewritten:
- the attempted answer a late LSP verdict, verify-on-stop or pre_verify
sends back (_append_interim_answer), and the kanban stop-guard answer;
- length-continuation fragments of a reply that then recovered (the final
row holds the rewrite of the whole, so the raw pieces also duplicated it);
- the intent-ack / stall / degenerate-final continuation rows, codex
status=incomplete partials, and commentary on a call to an unknown tool
or with broken JSON arguments (turn_tool_validation).
Every one of them is built by AIAgent._build_assistant_message, so the
hiding now happens there: in a held turn a row with visible text comes
back typed display_kind="hidden" (withhold_held_row) before anything can
flush it, and finish_text_response publishes the one row the user gets
(publish_held_final_row: visible again and, after recovered fragments,
carrying the whole reply; api_content keeps the piece the model wrote, so
the wire is unchanged). display_kind never reaches the wire, so the
model's context and cached prefix stay byte-for-byte (the test replays the
resumed session and compares the request).
Same class, also fixed: text beside a housekeeping call reused as the
reply after an empty follow-up was flagged response_previewed/reused,
though a held turn never showed it; the classic CLI then printed nothing.
Tests (tests/agent/test_hold_stream_exits.py, red on 3a5b54d980): late
LSP verdict, recovered continuation, unknown-tool commentary, reused
commentary.
Review: https://github.com/NousResearch/hermes-agent/pull/144266#discussion_r4370678628
A held turn (hold_stream=True transform_llm_output) only ran the transform
on the paths that reach finish_text_response / finalize_turn. Other exits
returned model text the plugin never saw:
- A reply that hit finish_reason="length" on every continuation: the
ceiling exit (_continue_text -> _Trunc.end_turn) stitched the raw
fragments into the final and the stored row and returned without
finalize_turn. collapse_continuation_trail now runs the turn's held
transform snapshot on the stitched text before it appends the row, so
the user and the stored row get the rewrite (or the withheld notice if
the transform fails). The same seam covers a provider error that ends
the turn after a capped fragment (settle_delivered_partial prepends that
fragment to the error copy).
- The iteration-budget summary appends its own assistant row; finalize
transformed the returned text but the row kept the raw summary, so
history showed it. A recovery tail row that still holds the untouched
text now stores the rewrite before its first flush
(_store_rewrite_on_tail, the #44239 contract).
- Every remaining exit (non-retryable / exhausted provider errors,
interrupts, compression and overflow failures, invalid-response caps,
preflight timeouts, ...) returns through run_conversation, which now
calls settle_held_final: a held turn whose final_response no seam
transformed runs the snapshot there, so no path hands the user text the
plugin did not see first. The hold snapshot is taken before
build_turn_context, so an early exit never reads the previous turn's
hold.
Tests (tests/agent/test_hold_stream_exits.py, real AIAgent + SessionDB on
the loopback model, red on 3a5b54d980): four capped responses, a capped
fragment then a 400, the budget summary, and a 400 with a tagging plugin.
Review: https://github.com/NousResearch/hermes-agent/pull/144266#discussion_r4370678627
Independent review of the hold_stream feature found four paths that still
let the untransformed reply or commentary reach the user:
- History loads showed held commentary. The tool-call assistant row of a
held turn is now typed display_kind="hidden" before its first flush
(withhold_held_commentary), so TUI/Desktop display_projection, the
dashboard and classic CLI resume skip it. display_kind never reaches the
wire: the model's context and cached prefix are unchanged (test replays
the resumed session and compares the request bytes). The TUI projection
still labels the hidden row's tool cards from its calls. Codex held turns
type their earlier agentMessages hidden the same way.
- A plugin reload mid-turn could empty the hook registry before the final
transform. The held turn now snapshots its transform_llm_output callbacks
together with the hold decision under the discovery lock
(held_stream_transforms) and runs that snapshot at the final. If no
callback rewrote and one raised or timed out, the user gets a fixed
"reply withheld" notice instead of the original (fail closed).
- Classic CLI replay-gap recovery painted and spoke pre_transform_response.
A held turn's result (generic and codex) no longer carries the original
at all (public_pre_transform), so receipts, recovery and post-stream
suffix painting only ever see the rewrite.
- Codex error-after-text returned the raw agentMessage. The codex reply is
now transformed whenever it has text, also when the turn errored (and,
when held, when interrupted).
Same class, also fixed: the end-of-stream scrubber flush recorded a held
tail as streamed text (an interrupted turn could return it as an
untransformed partial final), and quiet-mode CLI printed held commentary
via _vprint.
The codex app-server bookkeeping passed turn.final_text (pre-transform)
to _sync_external_memory_for_turn, so a memory provider stored the text
a transform_llm_output plugin removed. Pass the reply the user got, as
the generic loop's finalizer does.
Review: https://github.com/NousResearch/hermes-agent/pull/144266#discussion_r4369545047
The codex app-server runtime returns before finalize_turn, so
transform_llm_output never ran there (pre-existing): a rewriting
plugin's change never reached the user or the stored assistant row,
and with hold_stream the user waited only to get the raw reply.
Apply the transform before the projected rows are persisted and return
response_transformed / pre_transform_response like the generic loop.
Review: https://github.com/NousResearch/hermes-agent/pull/144266#discussion_r4369515509
The tool-loop guardrail halt pushes its explanation straight to
stream_delta_callback, bypassing the hold gate, so with hold_stream a
/v1/chat/completions client saw the halt reply and then the rewrite.
A held turn now delivers it only as the final, after transform_llm_output.
Review: https://github.com/NousResearch/hermes-agent/pull/144266#discussion_r4369515507