Committing the Store draft at publication makes every certified build go
live by itself, with no way left to keep one back for a chosen release time.
Setting the repository variable MS_STORE_HOLD=true makes the publication
pass commit the draft Manual instead: it is certified and then waits for
Publish now in Partner Center, with a warning that says so. A rerun of a
held submission is a no-op and never flips it to Immediate. Unset keeps the
auto-publish default.
Also quote the real v0.22.7 refusal (state 'PreProcessing') and update the
stable-releases developer guide, which still described the held commit.
The green run committed the Store submission held (Manual) and the
publication pass tried to switch it to Immediate. Partner Center refuses
to update a committed submission (v0.22.7: 409 InvalidState, "Cannot
update the submission because it is in certification"), so every
release needed Publish now by hand.
stable-store now leaves an uncommitted draft set to Immediate, and the
publication pass commits it. Nothing reaches certification or customers
before the release is published, and the build goes live as soon as it
passes certification.
The darwin matrix is Apple Silicon only, so the step's if: was always true
and its comment pointed at the removed macos-intel lane. The macOS bundle
update docs now say the publish job waits for the arm64 build and its smoke.
Vitest stores each file's last duration under node_modules/.vite/vitest and
runs the longest files first. The JS job caches node_modules keyed on the
lockfile, so the run that missed the cache saved its history and every hit
since replayed it: the heaviest jsdom suites start cold together and push
light files (browser-hints, terminal-ownership) past their 15s timeout. Main
has been red that way since run 41717106556. Remove the history after the
restore so each run orders its files from scratch.
Forwarding PSModuleAnalysisCachePath only helps when something already warmed
the cache. Whether the ARM64 build-deps step did depended on how long its
PowerShell stayed up: PowerShell writes the analysis from a timer ~10s after it
runs (AnalysisCache.QueueSerialization), so a process that exits right after
its first cmdlet never saves it. On runs where build-deps finished fast the
cache stayed cold and every PowerShell child a test spawned paid ~22s again
(arm64 test step 206-259s instead of ~130-150s).
Warm it explicitly from a cmd step and stay up until the cache file is
rewritten: ~39s when cold, ~1s when warm. Forced-cold check: a child launched
afterwards started in 0.4s and the test steps ran in 137/146s.
A fresh python -m pytest <file> spends about half a second starting the
interpreter and importing pytest and the third-party libraries nearly every
test file loads; over ~7000 files that is most of an hour of CPU per run.
With HERMES_TEST_FORKSERVER=1 (Linux CI) the runner starts one
multiprocessing fork server that imports those once and forks a child per
file. Each child still starts from a clean state: its own session, the
file's environment, python -m pytest's sys.path and argv, combined output and
pytest's exit code. Product code and tests/conftest.py are never preloaded.
The packaged arm now always runs on Apple Silicon, but install-e2e still
forwards the arch a supplied manifest names. Reject anything but arm64 as the
job's first step, and make the driver require a host-native package so a
direct run cannot fall through to Rosetta either.
The rule rejects outbound UDP 443 for every process for the rest of the
job, not only stapler: notarytool and the build's downloads and
uploads fall back to TCP too. Rename the step to "Disable QUIC (UDP
443) for the rest of the job" and say so in its comment.
Remove the darwin-x64 build, receipt and smoke jobs from Desktop Bundled
Release and the candidates/transitions/packaged-x64 jobs from Stable Release,
with the release scripts that required them: the macos-intel lane, the
darwin-x64 job group, the x64 macOS target in candidate/channel manifests and
the two-leg macOS feed merge. The macOS updater feed is arm64-only.
Pre-lanes claims still read: the published ones all carry
includeMacosIntel:false; a claim with true is now refused.
pm-bundle loses its darwin-x64 (macos-15-intel) leg. The desktop bundle smoke
and the install-e2e packaged arm always run on Apple Silicon; darwin/x64 is no
longer an accepted smoke target.
Intel Macs are unsupported on main: the macos-intel release lane is off
by default (scripts/releases/lanes.py). Run the TCP workaround only on
the darwin-arm64 jobs. The darwin-x64 jobs share the step list, so they
now skip it.
The probe in run 41655949959 had already shown that arm64 needs the
workaround. Without the rule, 3 of 63 arm64 staples failed with Error
68 / -1005. With it, 60 of 60 worked.
Branch verification on Blacksmith (CI, PM Bundle, Install & Update E2E
route=all, desktop bundle re-smokes) showed every Windows x64 install/
update E2E leg (0/53, vs 40/53 on windows-latest-32-core) and the Windows
x64 MSIX smoke failing on runner-session issues: empty screen recordings,
installer/app windows never seen, and 'Access is denied' launching the
installed MSIX. Put those jobs back on windows-latest-32-core.
Also keep the jobs a branch run cannot exercise where they were: the
desktop-bundled-release darwin-arm64 build (default-branch dispatch only)
and the macOS packaged-bundle E2E (needs signed bundle manifests).
What stays on Blacksmith, all green on the branch: pm-bundle darwin-arm64,
the macOS arm64 desktop smoke, and the macOS install/update source and
Hermes-Setup.dmg legs.
Moves the remaining GitHub-hosted macOS (arm64) and Windows x64 jobs to
Blacksmith now that the runner-side issues behind f1b290e773 and
3ba2b0787e are fixed: pm-bundle and desktop bundled-release darwin-arm64
builds, desktop bundle smokes (macOS arm64, Windows x64 MSIX) and the
install/update E2E macOS and Windows legs (with Blacksmith checkout).
macOS Intel legs stay on macos-15-intel (Blacksmith has no Intel macOS
runners); Windows arm64 stays on windows-latest-32-arm-core.
stapler fetches the notarization ticket from api.apple-cloudkit.com
over HTTP/3, and on GitHub's hosted macOS runners those QUIC
connections drop. The staple exits 68 with NSURLErrorDomain -1005
about 7 s in (actions/runner-images#14924, reproduced by GitHub).
The 2026-11-03 arm64 canary failed this way nine seconds after Apple
had created the ticket record. In this workflow's history the error hit
0 of 119 accepted builds before 2026-10-19 and 8 of 28 since, on both
arches. It also failed the verify step's `stapler validate` once.
Before Build and package, load a rule that rejects outbound UDP 443
into a com.apple sub-anchor. The runner's default ruleset already
evaluates com.apple/*, so it stays as it is. stapler then falls back to
TCP for the notarize hook and for the verify step.
Probe run 41655949959 used three macos-15 and three macos-15-intel
machines. Each stapled a copy of Firefox.app 21 times without the rule,
then 20 times with it.
- Without the rule, 60/63 staples worked on arm64 and 32/63 on Intel.
Every failure was Error 68 with -1005, after about 7 s.
- With the rule, 120/120 worked.
- Every machine sent UDP 443 to Apple's 17.x addresses, and the rule
rejected 60-85 packets per job.
- The main ruleset was unchanged on all six machines.
The sync-fork.yml workflow (PR #2) cannot push to main under this
repo's rules — PR-only for main, plus workflows permission required
for pushes that bring in upstream workflow files the fork lacks.
Every weekly run fails at the push step regardless of the fork's
sync state.
Closing the fork/main gap with upstream was done via PR #3 (merge
commit). Going forward, `hermes update` pulls from upstream directly
for local upgrades; the fork's main does not need to mirror upstream.
Remove the workflow to stop paying CI for failed weekly runs.
- Replace broken `gh repo sync --source NousResearch/hermes-agent` with
`-b main`. `--source` expects a branch/remote name, not a repo; `-b main`
tells gh repo sync to sync the fork's main from its parent upstream.
- Drop unnecessary actions/checkout@v4 + git config steps — gh repo sync
operates server-side and does not need a local checkout.
- Change cadence from daily (0 12 * * *) to weekly (0 12 * * 1, Mon 12:00
UTC). Upstream releases ~2-3×/week; weekly catches every release within a
week without daily no-op runs.
- Add timeout-minutes: 30 so a stuck/failed sync is not silently ignored.
- Add set -euo pipefail + echo so sync failures surface in the Actions log.
The published June Hermes-Setup.dmg updater writes the update marker with its
own PID and runs the v2026.7.30 checkout's `hermes update`, whose update lock
only recognises its orchestrator via HERMES_UPDATE_HANDOFF_PID (which the June
binary predates). The pre-pull CLI refuses with "Another Hermes update is
already running" and both desktop-installer@latest in-app legs from that tag
fail deterministically. The refusal runs in released code, so the update
target cannot fix it.
Extend the known-failures mechanism to macOS:
- rules carry `platforms` (default windows; existing rules unchanged)
- classifyWorkRoot reads the macOS harness layout (shas.env, home/.hermes)
- macos-desktop-e2e.sh rotates bootstrap-installer.log before the update,
collects HERMES_HOME/logs on failure, and only a full rule match (commit,
method pair, exact assertion, both fresh log signatures) yields a receipt
- install-e2e-macos-run.yml uploads the receipt as install-e2e-known-<id>--<leg>
- KNOWN_FAILURES.md documents the case with verified failing jobs
"Validate pinned bundle transitions" ran `npm ci` on the runner's system Node
22 / npm 10, which the root engines (`node >=26`, since 4d5c6cfb0c0) refuse
with EBADENGINE. The job now sets up the locked toolchain via setup-pm like
every other job in the install E2E workflows. The stable release calls this
workflow, so the red would fail a release attempt.
A child started in its own session or group (start_new_session, os.setsid /
setpgid / setpgrp, windows_detach_popen_kwargs) does not die with a SIGKILLed
Hermes: it reparents to init and runs forever. That is how every family this PR
migrates leaked. With the families on spawn_owned, this keeps the class closed.
scripts/check_detached_spawns.py (AST) flags those shapes in the production trees
unless the line, or the line above it, says `detached-spawn: ok — <why>`. It runs
in scripts/check ("detached-spawns") and as a blocking lint.yml step. The baseline
is empty: every remaining site is on spawn_owned / own_process_groups, or carries
its reason. The reasons this commit adds are the census verdicts:
- must outlive the starter on purpose: restart watchers, /update and the updater
hand-offs, dashboard/desktop/service launches, cron and kanban workers
(restart-safe), the update-custody watchdog itself;
- awaited with a group kill on timeout: cron scripts, rg search, the Browser Use
CLI, update completion (holds the checkout lock fd);
- covered by its own parent guard: the compute host (compute_host._parent_guard_loop
flushes sessions; a supervisor KILL could cut that short);
- not a spawn: the wake-word `start_new_session` config key and event field.
tests/scripts/test_check_detached_spawns.py: each shape flagged, non-detached and
commented/string shapes not, the marker opts out on its line or the line above
and nowhere else, tests and the helper are out of scope, main() fails on a new
unmarked spawn, and the repository scan is empty. Teeth: appending an unmarked
start_new_session Popen to tools/file_operations_search.py turns
`scripts/check --only detached-spawns` red (rc 1); restoring it turns it green.
scripts/whatsapp-bridge/*.test.mjs (allowlist, delivery_status,
bridge.native, ...) ran in no workflow: the bridge is a standalone npm
package outside the root workspaces, so run-workspace-checks never sees it.
js-tests.yml now installs the bridge's own lockfile (npm ci
--ignore-scripts; two suites import baileys) and runs
`node --test scripts/whatsapp-bridge/*.test.mjs`. The classifier selects
the frontend lane for scripts/whatsapp-bridge/ and
plugins/platforms/whatsapp/, so a bridge-only or adapter-only PR runs it.
No E2E trigger.
Every docs deploy since 2026-10-28 20:08 UTC failed at Deploy to GitHub Pages ('Artifact could not
be deployed ... total size'): the retained-assets pool reached 11 GB and the site artifact 2.95 GB
compressed. The pool exists so open tabs and edge caches can still load the previous build's hashed
chunks; that horizon is hours, so a 2-day window keeps the protection and drops the bulk.
A catalog entry is data (a pinned sha, a description, a card) reviewed by the maintainer landing it.
Since #131726 /plugin-catalog/ sat in CODEOWNERS, so every entry and re-pin needed a second
hermes-agent-core approval or an admin bypass. Drop that line; the README's tier note now names
maintainer review as the gate.
Add hermes_cli/config_backend.py (config-config design §4.1, D10/D11): a
ConfigBackend protocol, the FileBackend (today's behaviour), selection by
HERMES_CONFIG_BACKEND only (default file; 'remote' and unknown values fail
closed with ConfigBackendUnavailable), and path-level helpers
(read_config_doc, read_config_doc_readonly, config_version, config_exists,
write_config_key, write_config_document, supports_file_tooling).
Migrate every direct reader/stat/exists/bypass writer of a user config.yaml
to the helpers (config.py loaders + caches, config_effective, env_loader,
gateway, TUI, CLI watchers, cron, tools, profiles, logging, time). File
tooling (config edit, backups, xAI migration rewrite, raw dashboard editor)
checks supports_file_tooling() first; migrations skip write-back without it.
Add scripts/check_config_yaml_readers.py (CI lint next to the writers gate)
that rejects a direct read of a config path outside the backend.
The rest of the suite uses an in-memory keyring. These two tests run only on the
macOS and Windows OS lanes (platforms("macos", "windows")): store/read/delete a
uniquely named dummy entry through the real keyring backend, and a keychain-mode
vault create -> migrate to file -> migrate back round trip. They skip where the
host has no OS keychain and clean their entries up in teardown; Linux hosts never
run them, so a developer's Secret Service is never touched.
tests-os.yml installs the keychain extra so keyring is importable on those lanes.
The local vault key sat in vault.key next to vault.json.enc, so any backup, synced
folder or copied ~/.hermes carried the ciphertext together with its key.
vault.key_storage: keychain (default stays `file`) puts a NEW vault's key in the OS
keychain via keyring (new lazy `keychain` extra, keyring==25.7.0), one entry per
profile, named by a non-secret vault.keyref. No reachable keychain (headless Linux,
extra not installed, plaintext/null keyring backends) falls back to the 0600 file.
`hermes vault key` shows where the key is; `hermes vault key migrate --to keychain|file`
writes the destination, proves it decrypts the vault, and only then removes the
source, so an interrupted move leaves a working vault and a rerun finishes it.
A vault whose key is missing (file or keychain entry) now raises a recovery message
and never mints a new key over existing ciphertext (before, a deleted vault.key was
silently replaced and the vault reported "key mismatch or corruption").
Opt-in rather than default: macOS can prompt for keychain access after Hermes'
Python changes, locked Linux keyrings prompt or are absent over SSH, cron while
logged out cannot read the key, and backups stop being self-contained.
The e2e spec starts the gateway through hermes webapp, runs a turn from a
real browser, keeps the tab signed in across a gateway restart, and checks
that sign-out cuts it off. The smoke script targets the Webapp's port.
Co-authored-by: Bear Huddleston <bear@bearhuddleston.dev>
Root and desktop engines.node move from '^22.22.0 || ^24.11.0 || >=26.0.0' to '>=26', and the
website's from '>=20.0' to '>=26.0' (PM already pins Node 26.12.1). The lockfile workspace engine
mirrors, the CONTRIBUTING prerequisite, the tests-js engine-policy table and the
bootstrap-installer release workflow's setup-node pin (24.11.1 -> 26.12.1, which engine-strict
would now reject) follow.
Every Windows x64 MSIX chat smoke since the Blacksmith migration (91fae5f8cb4)
failed: first on the GitHub-hosted guard, and since that guard was removed
(272adc7b6fa) with Playwright's "Process failed to launch!", on Electron 40 and
44 alike. The same builds' arm64 MSIX, on a GitHub-hosted runner, chats; the
last green x64 smoke (Oct 14) ran on windows-latest-32-core. Move the x64 leg
back there, as f1b290e7734 did for the install/update GUI legs.
The failure itself carried no evidence: when the app exits before its debugger
line, Playwright's Electron.launch leaves sibling stderr-watcher rejections
unhandled, Node dies on them, and the smoke's catch never writes its receipt or
Playwright's call log. The driver now logs those rejections (still failing the
run), so the receipt records the app's exit code and stderr, and the Windows
smoke saves Application-log errors from the run on failure.
Review follow-up: deletion now sits in a finally around artifact collection,
so a full disk that breaks the copy cannot keep a whole install on the drive,
and the leak check step runs with if: always() so failed runs still name any
leftover install.
Machine.teardown copied the transcripts and logs out, then left the whole
fake profile behind: a full install (checkout, venv with every extra, PM
tools, caches) plus serve.git, all on the runner's only drive. With
test_damaged_checkout.py the suite reached 13 files / 15 machines and the
last journey ran the disk dry (run 39853224877: 92 MB free, the three
standard-user cells failed; the webapp dispatch hit 0 MB and failed 8).
Teardown now removes the profile and serve.git after the artifacts copy
(long-path prefix, read-only git objects via utils.rmtree_readonly, retry
while a killed process lets go); anything it cannot delete is named in
teardown-leftovers.txt, and a job step fails when a machine install
outlives its journey.
Retargets the 44.4.5 bump to 44.9.3, the newest 44.x. Per #142533,
44.9.1+ carries the eMudhra root distrust, and 44.9.3 adds upstream
Chromium/V8 security backports. The aging-gate exclusion now names
44.9.3 and its release date.
Electron 42+ ships no install script, so the electron allowScripts entry
has nothing left to allow. Nothing depends on yauzl any more, so the
override #47792 added for Electron's old JS extractor goes too. The
Windows attach E2E launched the binary the postinstall used to unpack;
it now prepares the runtime explicitly, like the other desktop lanes.
Electron 42 dropped the postinstall that unpacked its binary into
node_modules/electron/dist. The binary now appears on the first
require('electron'), so parallel vitest workers race to unpack it
("Text file busy" in #128267's CI), and `hermes desktop --source` finds
no path.txt to launch.
electron-runtime.mjs runs Electron's own install.js once: after npm ci
for the desktop workspace, and in CI before the parallel checks. It first
checks the installed package against the desktop pin and the lockfile,
and does nothing when the runtime is already in place. On the current
40.10.2 pin, whose postinstall still unpacks the binary, it is a no-op.
Salvaged from #137648 (the runtime-preparation subset only).
Collect ordinary actions/cache job logs alongside the existing timing
snapshot and retain per-invocation restore/write receipts and archive bytes.
Compare each cache with main and show saved-minus-restored archive changes
in the review comment, Actions summary and complete HTML report.
Count successful writes only after their save receipt; transfer progress
supplies a size once, never a sum. Preserve unknown sizes, failed or missing
logs, lookup-only results and exact-hit save skips. Match content-keyed
writes to their restored paths, without inspecting cache contents or using
the GitHub inventory as a proxy for a runner provider's cache store.
Print a compact tools/uv/npm summary from setup-pm, including the selected
restore outcome, exact-hit output, cache path, ref scope and save policy.
Keep actual archive sizes and upload results in the native cache logs.
Do not scan cache trees or add logging wrappers to non-PM cache actions.
b1aa63b9181 folded the pinned-input archive step into the `validate`
("Pre-build setup") job, which kept its 5-minute cap from when it only ran
admission. The step re-downloads every pinned input from R2 to verify its
sha256 (several GB: Chromium, llama.cpp CUDA). That takes ~20 s normally
(v0.22.3 rc.5), but v0.22.4 rc.1 hit slow R2 reads: both Windows candidate
chains were cancelled at 5:09 mid-verify, and every downstream job failed
"Release blocked: validate=cancelled". The standalone archive job it replaced
had 60 minutes; give the merged job 30.
Every job that only assembles or promotes Docker Hub manifest lists with
`imagetools create` (docker.yml merge + release-publish-manifest,
sandbox-image.yml merge, the stable-release `complete` and publication
`reconcile` alias promotions) moved to Blacksmith in 91fae5f8cb4. Since then
the index commit to Docker Hub fails every time:
failed commit on ref "index-sha256:...": unexpected status from GET request
to https://registry-1.docker.io:443/v2/.../manifests/rc.4-v0.22.3: 404 Not Found
It is not the buildx version: rc.4 ran buildx v0.38.1 (after 6932521f258) and
failed identically to rc.1-rc.3 on v0.23.0, while v0.22.2 published on
ubuntu-latest with docker/setup-buildx-action. Restore exactly that setup
(runner, checkout, setup-buildx-action pinned to the pre-migration SHAs) for
these five jobs; builds and per-arch pushes stay on Blacksmith.