Add hermes_cli/config_backend.py (config-config design §4.1, D10/D11): a
ConfigBackend protocol, the FileBackend (today's behaviour), selection by
HERMES_CONFIG_BACKEND only (default file; 'remote' and unknown values fail
closed with ConfigBackendUnavailable), and path-level helpers
(read_config_doc, read_config_doc_readonly, config_version, config_exists,
write_config_key, write_config_document, supports_file_tooling).
Migrate every direct reader/stat/exists/bypass writer of a user config.yaml
to the helpers (config.py loaders + caches, config_effective, env_loader,
gateway, TUI, CLI watchers, cron, tools, profiles, logging, time). File
tooling (config edit, backups, xAI migration rewrite, raw dashboard editor)
checks supports_file_tooling() first; migrations skip write-back without it.
Add scripts/check_config_yaml_readers.py (CI lint next to the writers gate)
that rejects a direct read of a config path outside the backend.
The rest of the suite uses an in-memory keyring. These two tests run only on the
macOS and Windows OS lanes (platforms("macos", "windows")): store/read/delete a
uniquely named dummy entry through the real keyring backend, and a keychain-mode
vault create -> migrate to file -> migrate back round trip. They skip where the
host has no OS keychain and clean their entries up in teardown; Linux hosts never
run them, so a developer's Secret Service is never touched.
tests-os.yml installs the keychain extra so keyring is importable on those lanes.
The local vault key sat in vault.key next to vault.json.enc, so any backup, synced
folder or copied ~/.hermes carried the ciphertext together with its key.
vault.key_storage: keychain (default stays `file`) puts a NEW vault's key in the OS
keychain via keyring (new lazy `keychain` extra, keyring==25.7.0), one entry per
profile, named by a non-secret vault.keyref. No reachable keychain (headless Linux,
extra not installed, plaintext/null keyring backends) falls back to the 0600 file.
`hermes vault key` shows where the key is; `hermes vault key migrate --to keychain|file`
writes the destination, proves it decrypts the vault, and only then removes the
source, so an interrupted move leaves a working vault and a rerun finishes it.
A vault whose key is missing (file or keychain entry) now raises a recovery message
and never mints a new key over existing ciphertext (before, a deleted vault.key was
silently replaced and the vault reported "key mismatch or corruption").
Opt-in rather than default: macOS can prompt for keychain access after Hermes'
Python changes, locked Linux keyrings prompt or are absent over SSH, cron while
logged out cannot read the key, and backups stop being self-contained.
The e2e spec starts the gateway through hermes webapp, runs a turn from a
real browser, keeps the tab signed in across a gateway restart, and checks
that sign-out cuts it off. The smoke script targets the Webapp's port.
Co-authored-by: Bear Huddleston <bear@bearhuddleston.dev>
Root and desktop engines.node move from '^22.22.0 || ^24.11.0 || >=26.0.0' to '>=26', and the
website's from '>=20.0' to '>=26.0' (PM already pins Node 26.12.1). The lockfile workspace engine
mirrors, the CONTRIBUTING prerequisite, the tests-js engine-policy table and the
bootstrap-installer release workflow's setup-node pin (24.11.1 -> 26.12.1, which engine-strict
would now reject) follow.
Every Windows x64 MSIX chat smoke since the Blacksmith migration (91fae5f8cb4)
failed: first on the GitHub-hosted guard, and since that guard was removed
(272adc7b6fa) with Playwright's "Process failed to launch!", on Electron 40 and
44 alike. The same builds' arm64 MSIX, on a GitHub-hosted runner, chats; the
last green x64 smoke (Oct 14) ran on windows-latest-32-core. Move the x64 leg
back there, as f1b290e7734 did for the install/update GUI legs.
The failure itself carried no evidence: when the app exits before its debugger
line, Playwright's Electron.launch leaves sibling stderr-watcher rejections
unhandled, Node dies on them, and the smoke's catch never writes its receipt or
Playwright's call log. The driver now logs those rejections (still failing the
run), so the receipt records the app's exit code and stderr, and the Windows
smoke saves Application-log errors from the run on failure.
Review follow-up: deletion now sits in a finally around artifact collection,
so a full disk that breaks the copy cannot keep a whole install on the drive,
and the leak check step runs with if: always() so failed runs still name any
leftover install.
Machine.teardown copied the transcripts and logs out, then left the whole
fake profile behind: a full install (checkout, venv with every extra, PM
tools, caches) plus serve.git, all on the runner's only drive. With
test_damaged_checkout.py the suite reached 13 files / 15 machines and the
last journey ran the disk dry (run 39853224877: 92 MB free, the three
standard-user cells failed; the webapp dispatch hit 0 MB and failed 8).
Teardown now removes the profile and serve.git after the artifacts copy
(long-path prefix, read-only git objects via utils.rmtree_readonly, retry
while a killed process lets go); anything it cannot delete is named in
teardown-leftovers.txt, and a job step fails when a machine install
outlives its journey.
Retargets the 44.4.5 bump to 44.9.3, the newest 44.x. Per #142533,
44.9.1+ carries the eMudhra root distrust, and 44.9.3 adds upstream
Chromium/V8 security backports. The aging-gate exclusion now names
44.9.3 and its release date.
Electron 42+ ships no install script, so the electron allowScripts entry
has nothing left to allow. Nothing depends on yauzl any more, so the
override #47792 added for Electron's old JS extractor goes too. The
Windows attach E2E launched the binary the postinstall used to unpack;
it now prepares the runtime explicitly, like the other desktop lanes.
Electron 42 dropped the postinstall that unpacked its binary into
node_modules/electron/dist. The binary now appears on the first
require('electron'), so parallel vitest workers race to unpack it
("Text file busy" in #128267's CI), and `hermes desktop --source` finds
no path.txt to launch.
electron-runtime.mjs runs Electron's own install.js once: after npm ci
for the desktop workspace, and in CI before the parallel checks. It first
checks the installed package against the desktop pin and the lockfile,
and does nothing when the runtime is already in place. On the current
40.10.2 pin, whose postinstall still unpacks the binary, it is a no-op.
Salvaged from #137648 (the runtime-preparation subset only).
Collect ordinary actions/cache job logs alongside the existing timing
snapshot and retain per-invocation restore/write receipts and archive bytes.
Compare each cache with main and show saved-minus-restored archive changes
in the review comment, Actions summary and complete HTML report.
Count successful writes only after their save receipt; transfer progress
supplies a size once, never a sum. Preserve unknown sizes, failed or missing
logs, lookup-only results and exact-hit save skips. Match content-keyed
writes to their restored paths, without inspecting cache contents or using
the GitHub inventory as a proxy for a runner provider's cache store.
Print a compact tools/uv/npm summary from setup-pm, including the selected
restore outcome, exact-hit output, cache path, ref scope and save policy.
Keep actual archive sizes and upload results in the native cache logs.
Do not scan cache trees or add logging wrappers to non-PM cache actions.
b1aa63b9181 folded the pinned-input archive step into the `validate`
("Pre-build setup") job, which kept its 5-minute cap from when it only ran
admission. The step re-downloads every pinned input from R2 to verify its
sha256 (several GB: Chromium, llama.cpp CUDA). That takes ~20 s normally
(v0.22.3 rc.5), but v0.22.4 rc.1 hit slow R2 reads: both Windows candidate
chains were cancelled at 5:09 mid-verify, and every downstream job failed
"Release blocked: validate=cancelled". The standalone archive job it replaced
had 60 minutes; give the merged job 30.
Every job that only assembles or promotes Docker Hub manifest lists with
`imagetools create` (docker.yml merge + release-publish-manifest,
sandbox-image.yml merge, the stable-release `complete` and publication
`reconcile` alias promotions) moved to Blacksmith in 91fae5f8cb4. Since then
the index commit to Docker Hub fails every time:
failed commit on ref "index-sha256:...": unexpected status from GET request
to https://registry-1.docker.io:443/v2/.../manifests/rc.4-v0.22.3: 404 Not Found
It is not the buildx version: rc.4 ran buildx v0.38.1 (after 6932521f258) and
failed identically to rc.1-rc.3 on v0.23.0, while v0.22.2 published on
ubuntu-latest with docker/setup-buildx-action. Restore exactly that setup
(runner, checkout, setup-buildx-action pinned to the pre-migration SHAs) for
these five jobs; builds and per-arch pushes stay on Blacksmith.
useblacksmith/setup-docker-builder accepts only 'latest' or vX.Y.Z for
buildx-version. cf4f3a73968 replaced the version with a commit SHA, which the
action rejects ("Invalid buildx-version ... Falling back to default v0.23.0"),
so every job 50bab3867de moved to a current buildx was back on v0.23.0, whose
`imagetools create` cannot publish a manifest list to Docker Hub (404 on the
index commit). Stable v0.22.3 rc.3 failed "Assemble versioned manifest and
digest receipt" 3/3 that way.
Pin v0.38.1 (the release the SHA's own comment names, with the CVE fix the
SHA pin was after). The action still verifies the downloaded binary.
install-e2e-run.yml runs every Linux leg; the migration set its default runner
to blacksmith-2vcpu-ubuntu-2404 (pre-migration: ubuntu-latest). Restored byte
for byte with the four sibling files.
The runner migration (91fae5f8cb) moved the release install/update matrix
onto Blacksmith runners, where the macOS legs cannot record the screen
(ffmpeg avfoundation finds no devices) and the Windows GUI legs never see
the Hermes-Setup.exe window. Every rc.2-v0.22.3 desktop leg failed before
it reached the code under test. The install-e2e workflows go back to their
pre-migration GitHub-hosted runners; everyday CI stays on Blacksmith.
Docker publishing is left on Blacksmith: main fixed its manifest-list 404
by pinning buildx (50bab3867de, cf4f3a73968).
The blacksmith.sh runner migration (91fae5f8cb) mechanically replaced
docker/setup-buildx-action with useblacksmith/setup-docker-builder.
Side effect: buildx dropped from v0.38.0 (setup-buildx-action installs
the latest release) to the Blacksmith action's default pin, v0.23.0 -
and v0.23.0's `imagetools create` cannot publish a manifest list to
Docker Hub. The index commit on the target tag dies with:
failed commit on ref "index-sha256:...": unexpected status from GET
request to .../manifests/rc.2-v0.22.3: 404 Not Found
Stable v0.22.3 rc.1 and rc.2 both failed deterministically in
"Create both immutable versioned manifest lists" (3/3 attempts each,
identical error); the same step succeeded with buildx v0.38.0 in the
v0.22.2 release. Per-arch pushes go through BuildKit and are
unaffected, so the build jobs keep the action default.
Pin `buildx-version: latest` on the setup-docker-builder steps of
every job that only does registry-side imagetools work: docker.yml
release-publish-manifest and merge, sandbox-image.yml merge, and both
sequencer alias-promotion jobs (promote_stable() in
scripts/releases/docker.py runs imagetools create for
vX.Y.Z/stable/latest). This restores the pre-migration toolchain for
exactly those jobs.
use upload-artifact v7's archive:false option for the single HTML report.
remove the ignored name input: the raw artifact is named after its file,
ci-timings-report.html. keep the linked artifact URL, retention, advisory
failure behavior and zipped review-status payload unchanged.
save-python-cache and save-node-cache now default to false on
pull_request events, like save-tools-cache. Their restore-only steps
read the exact keys the saving steps write on main, so a PR gets main's
uv and npm caches and leaves nothing PR-scoped behind.
Restore-only used to mean an explicit-save flow: run- and job-scoped
keys meant for save-pm-cache or a caller's actions/cache/save to write
later. Nothing calls save-pm-cache, and the last caller restoring from
the npm explicit-save key moved to its own cache, so the flow and its
outputs are deleted.
osx-sign spends 2.1-3.3 min per mac leg running one codesign call for each of
610-836 payload binaries (run 39214394169), and only 321 of 837 arm64 names
are distinct: most Python extensions sit three times in the payload (venv,
uv wheels, uv archive). None of it is cached, unlike Windows' Authenticode
signatures (payload-sign-cache.mjs).
createMacSigner now signs loose payload Mach-O itself before osx-sign runs,
through a cache modelled on the Windows one:
- key: the unsigned bytes and file name (codesign derives the identifier
from the name), under a policy hash of the identity, team, flags and
inherited entitlements; the per-job keychain path is not part of it;
- codesign's embedded signature carries a secure timestamp, so the signed
bytes are stored, and every restored file must pass codesign --verify
--strict with this team's Developer ID, a timestamp and the hardened
runtime before it replaces the unsigned one;
- each distinct input is signed once (bounded concurrency) and copied to
its duplicates; files inside nested bundles and the app executable are
left to osx-sign, which still seals every bundle and the app.
The darwin legs restore the cache before building and save it only after
the notarized-app verification passes, on release runs. The mac runners are
GitHub-hosted, so it lands in the repo's GitHub cache store; commit builds
hold a read-only cache token.
Inject the typed app-signing function into the cache integration tests instead of
mocking its module. Exercise cold and warm reuse and refuse outer sealing after
cache verification fails; preserve the production lazy import and native skips.
stable-store commits the Store submission held (targetPublishMode Manual)
instead of leaving an uncommitted draft, so certification runs before the
release gate and a run whose stable-publish fails still shows customers
nothing. The publication pass then PUTs the held submission with
Immediate and commits it again, so it goes live without a Partner Center
click.
Microsoft documents no call that releases a held submission and does not
say whether update accepts a committed one (409 is documented only as
"the current state of the app"). A 409 from the switch leaves the
submission held and prints the Publish now instruction; the release
stays green. The first real release shows which answer the API gives.
Reconcile receipt-bound Store work independently of channel advancement, including
already-current published releases, so an interruption cannot strand a held
submission. Record each upload's claim in certification notes and replace a held
Manual predecessor only after its exact remote abandon marker matches the claim
commit. Unknown owners and publishing submissions remain protected.
Every full-history checkout in the stable release fetches every branch, tag
and blob: 53-74 s per job on Blacksmith Linux (stable run 38491090442:
admit 67 s, candidate-manifest 74 s, complete 53 s, each stage-receipt and
pre-build ~65 s), and Blacksmith's git mirror is unavailable on Windows
runners. These jobs need the commit graph and annotated tag objects (claim
custody, release-line ancestry, tagger timestamps), never historical file
contents. A blob:none partial clone keeps all commits, trees and tags and
fetches blobs on demand (the checked-out tree, `git show <sha>:file`).
Measured locally against the repo: full fetch 58 s / 1.3 GB, blob:none 8 s
/ 155 MB, same 2,826 refs.
Assembling the universal MSIX bundle spent ~5-8 min (canary 39214394169) on
setup it re-did every run:
- electron-builder toolset (winCodeSign kit, ATS dlib, dotnet runtime): the
eb3 cache was never saved. The build legs saved
.cache/electron-builder, but preparation points ELECTRON_BUILDER_CACHE at
desktop-inputs/packager/builder, so every save hit "Path Validation
Error ... no cache is being saved" and every restore missed (eb3 has 0
entries). The win32-x64 leg now saves the real directory, and the darwin
legs drop the same dead save/restore.
- npm: setup-pm looked up a per-job rolling key that only an explicit save
writes, and this read-only job has none. The win32-x64 leg already holds
npm's content cache in desktop-inputs/npm/_cacache after npm ci; save it
next to the toolset under one key and point assembly's npm at it.
- checkout: full history plus every tag (~128 s) for every assembly. Only a
stable candidate reads git (claim-tag tagger time for the MSIX version,
claim custody in the receipt step). A canary's version is the stamp in its
tag name, so canary/commit/channel assembly checks out depth 1. The old
comment's "minutes since the last stable" canary scheme was replaced by
yy.mmdd.hh.mmss in fcfd001dc03.
The key carries the runner architecture and package-lock.json hash; assembly
restores that dependency set, then falls back within the same architecture.
Select the stable candidate full-history path with a truthy first arm: numeric zero
is falsy in Actions expressions. A SHA-only shallow fetch loses its annotated
claim ref, which the real MSIX version reader requires.
The setup-pm tools cache key hashes pm/**, this action and
setup_toolchain.py, so every push to a PR touching them saved a new
100-170 MB store scoped to that PR. Nothing else can restore a
PR-scoped entry: 169 of them held 19.4 GB of the repo's 26.8 GB
Blacksmith cache, while main had 60 entries.
save-tools-cache now defaults to false for pull_request events, so PRs
restore main's store and write nothing. Pushes and dispatches keep
saving, and callers that pass the input explicitly are unchanged.
The Store build used to be committed at green with targetPublishMode
Manual, and the publication pass could only print "click Publish now":
the submission API has no call that releases a held submission (update
and commit return 409 once it is committed). Every stable release needed a
person in Partner Center.
stable-store now uploads the verified bundle as an uncommitted draft
(msstore publish --noCommit) and stops. The publication pass, which runs
only when the release really goes out, reads the receipt-bound candidate
manifest, checks the pending draft carries that release's Store bundle,
sets targetPublishMode Immediate and commits it. Certification starts
then, and the build goes live when it passes.
- A draft from a newer green run is left for its own release; a committed
or live submission is a no-op, so reruns are safe.
- submit refuses to replace a committed submission still on its way live
(an earlier release); msstore publish would delete it.
- complete's Reconcile step now receives the MS_STORE_* credentials; it had
none, so the success path silently skipped the Store.
- A failed or unreadable Store step stays a warning: the release publishes.
stable-store needed stable-publish, so the Store submission (~6 min on a
Windows runner) only started after verify-published (~8.5 min) finished,
serially at the very end of the stable release. Nothing passes between
them: stable-store materializes and digest-checks its own Store candidate
from the admitted manifest, and the submission is held (Manual) until the
publication pass releases it. Run both off validate.
The tools-cache key hashes pm/**, but the prepare step runs
setup_toolchain.py, which imports pm and writes pm/__pycache__ before
the cache step evaluates hashFiles(). Those .pyc files embed the
checkout's source mtimes, so two jobs on the same commit compute
different keys and the exact-key lookup misses.
Exclude **/__pycache__/** from hashFiles() in both the save and the
restore-only key, and add a test that the two keys match and ignore
bytecode.
(cherry picked from commit 69d29f42187ef187da5465c26829f85d4c53bc1a)
New core spec apps/desktop/e2e/core/local-gateway-attach.spec.ts. It starts
`hermes gateway run` first (detached, windowless, as a login item or console
leaves it). Then it opens Desktop twice and checks each launch:
- it attaches through Electron main's real path: `hermes gateway ensure`,
then the ticket mint (the SID-checked named pipe through the runtime
helper on Windows, the private control socket elsewhere);
- the connection descriptor is local loopback;
- gateway_state.json and the process census name the pre-started pid and
only it;
- a turn completes against the scripted provider and passes the transcript
oracle (launch 2 resumes the session and adds a turn);
- Electron main exits on quit and the gateway it did not start keeps
running.
`hermes gateway stop` then ends it. HERMES_E2E_PACKAGED_EXE runs the same spec
against a packaged build.
.github/workflows/windows-desktop-attach-e2e.yml runs it on
blacksmith-32vcpu-windows-2025 (workflow_dispatch, or a push to
wine2e-desktop/**, which nothing else listens to): first on the dev tree,
then on `electron-builder --dir --win` output of the same commit.
Verified locally on Linux (Xvnc), both census modes (procfs and
HERMES_E2E_PROC_CENSUS=ps, the path Windows takes): 1 passed each.
Refs #134889 (W3)
Refs #140337
The classic-CLI tmux suites (tests/e2e/core/cli_tmux/*, plus the Vertex
/bg CLI test) skipped whenever tmux was missing. In the release/dispatch
e2e job that would turn the lane green over zero coverage if the runner
image ever dropped tmux.
_classic.require_tmux() replaces the bare skipif markers in all four
files: it skips on a developer box, and fails when CI is set and
HERMES_E2E_REQUIRE_TUI=1 (which the e2e job in tests.yml already sets).
The e2e job gains an "Install tmux" step: it installs tmux only if absent
(the hosted image ships it: dispatch run 38982129242 on 2026-10-16 ran
test_resize_scrollback.py 1 passed, no skip) and prints `tmux -V`.
tests/ci/test_e2e_tmux_required.py pins the gate (fail in the CI lane,
skip otherwise), that no tmux suite keeps a bare skip, and that the e2e
job verifies tmux. 6 red before, green after. Through run_tests.sh with
tmux off PATH: CI+HERMES_E2E_REQUIRE_TUI=1 -> 3 failed; CI only -> 3
skipped.
Review finding by @unsupportedpastels on #140469
Refs #134889 (T24)
Refs #140337
A PR that changes only plugin-catalog entry YAML (plus contributor email
maps) ran all 23k Python tests on Linux, macOS and Windows, ~25 min, and
unrelated core flakes kept blocking one-file catalog additions.
The classifier now has a `catalog` lane. A diff made only of catalog entry
data turns `python` off and `catalog` on; ci.yaml then runs
catalog-tests.yml (every test file that reads the catalog or the email
maps, discovered by scripts/ci/catalog_test_files.py), plus the contributor
check. The lane is required by all-checks-pass whenever detect selects it.
plugin-catalog/README.md keeps the full suite (a docs-page test mirrors
it), and any non-catalog file in the diff brings the suite back. The
structural and pinned-source admission gates in plugin-catalog-ci.yml are
unchanged.
Both arm64 matrix rows were named (arm64, 1/2), so the PR checks list showed
one row for two jobs. The checkout comment and the macOS slice comment still
described the GitHub-hosted setup this PR replaces.