Documents a6aad08/83916f8 in BUILD-LOG-2026-10-23.md's "Round 2 of 3" section (SQLite column encryption + the two live bugs found closing it out), adds a RUNBOOK.md entry for RATION_LEDGER_ENCRYPTION_KEY now that it covers both the WAL and SQLite columns, and adds scripts/record-demo.sh — narrates docs/DEMO-SCRIPT.md's exact sequence with real pacing, built to be captured with `asciinema rec demo.cast -c scripts/record-demo.sh` for the hackathon submission's video walkthrough. Also ignores demo-video/, the gitignored output directory that script's rendered .mp4/.cast land in (never source, never committed).
18 KiB
Runbook
On-call style: symptom first, then a diagnosis path that names exact files, log lines, and commands in this repo — not generic advice. If a step below doesn't resolve it, the next escalation is always "read the cited source file," because every path here traces back to a specific function.
Every service logs structured JSON (log/slog in Go, tracing in
Rust) and, once pkg/telemetry's correlation-ID propagation is wired
into a given call path, every log line for one CLI command / webhook /
agent tick carries the same x-ration-correlation-id
(pkg/telemetry/correlation.go) — grep logs for that ID across
services' stdout to follow one request end to end before reaching for a
tracing backend.
"ledger-core won't start"
Symptom: ./services/ledger-core/target/release/ledger-core (built
by make build-ledger, invoked by scripts/run-demo.sh as
$LEDGER_BIN) exits immediately, or scripts/run-demo.sh reports it
never became ready to accept connections.
Diagnosis path, in order:
-
Check how far startup got.
main.rs'srun()logs, in order:"opening ledger (recovering WAL before accepting RPCs)"→Ledger::open→"WAL recovery complete"(withreplayed_events) →"LedgerCore gRPC server listening". Whichever log line is the last one printed tells you which step failed:- Never logs "opening ledger": the process didn't even get past
env/arg parsing — check that
services/ledger-core/target/release/ ledger-coreis the binary actually being run (not a stale build;make build-ledgerrebuilds it) and that stdout isn't being swallowed (scripts/run-demo.shredirects each service's stdout/stderr todata/logs/). - Logs "opening ledger" but never "WAL recovery complete": the
failure is inside
Ledger::open(services/ledger-core/src/ ledger.rs) — go to step 2. - Logs "WAL recovery complete" but never "LedgerCore gRPC server
listening": the failure is
resolve_bind_addrortonic:: transport::Server::builder()...serve(addr)— go to step 4.
- Never logs "opening ledger": the process didn't even get past
env/arg parsing — check that
-
A
RATION_LEDGER_WAL_PATHorRATION_LEDGER_DB_PATHdirectory problem.Ledger::opencreatesdb_path's parent directory (std::fs::create_dir_all(parent)) but does not createwal_path's parent directory —wal_ffi::Wal::openjust calls C'swal_open, which does a plainopen(path, O_RDWR|O_CREAT, 0644)(services/wal/wal.c) and fails withWAL_ERR_IOif that directory doesn't exist. In the default config both paths are under the same./data/dir (.env.example:RATION_LEDGER_WAL_PATH=./data/ ledger.wal,RATION_LEDGER_DB_PATH=./data/ledger.db), so this only bites if you've pointed the two paths at different directories and only created one of them. Fix:mkdir -p "$(dirname "$RATION_LEDGER_WAL_PATH")"before starting the process. (This is a real, currently-undocumented gap inLedger::open, not a configuration mistake on your part — worth a follow-up patch to makeLedger::opencreate both parents, tracked here rather than silently worked around.) -
A corrupt-beyond-recovery WAL.
wal_recover(services/wal/ wal.c) already treats a torn tail (partial header/payload, or a payload whose CRC doesn't match) as expected and non-fatal — it truncates and keeps going, so this is NOT your problem if the last shutdown was a crash. Whatreplay_wal_into_index(services/ledger-core/src/ledger.rs) surfaces as a real, fatal error is a well-formed, CRC-valid frame whose payload isn't valid JSON for the currentWalEventenum — logged bymain.rs'sErrpath as"ledger-core exited with error"with a message starting"corrupt WAL event payload: ...". This happens if the WAL file was hand-edited, or if a previous deploy wrote frames with an older/newerWalEventshape than the binary now running expects (a schema migration hazard:WalEvent's variants are never versioned on disk today). There is no automatic recovery from this — the fix is a manual decision (restore a backed-up WAL perinfra/terraform/modules/ledger-backup/, or, for a dev-only database, deleteRATION_LEDGER_WAL_PATH/RATION_LEDGER_DB_PATHand accept the data loss). Never routinely delete a WAL as a first step — it is the durability source of truth (ADR 0002). -
Port/address problem.
resolve_bind_addrinmain.rsresolvesLEDGER_CORE_ADDR(defaultlocalhost:7001,.env.example) via the system resolver and fails fast with"LEDGER_CORE_ADDR {addr} did not resolve to any address"if the host part doesn't resolve at all. If it resolves but the process still fails to bind, something else already owns the port —lsof -iTCP:7001 -sTCP:LISTEN(adjust the port to yourLEDGER_CORE_ADDR), most commonly a previousledger-coreprocessscripts/stop-demo.shdidn't clean up (check.ration-demo.pid). -
It never finished building.
make build-ledgerdepends onbuild-wal($(MAKE) -C services/wal, producinglibwal.a) beforecargo build --release— iflibwal.ais stale or missing,ledger-core'sbuild.rs(which compilesservices/walwithcc) is the thing to re-run:cd services/ledger-core && cargo clean && cargo build --releaseforces a fresh link.
"I want to enable encryption at rest" / "ledger-core won't decrypt its own data after a restart"
What RATION_LEDGER_ENCRYPTION_KEY actually encrypts: one base64-
encoded 32-byte AES-256 key (services/ledger-core/src/wal_crypto.rs),
used for two independent things once set:
- Every frame
ledger-coreappends to its own write-ahead log (crate::ledger'sappend_and_commit/replay_wal_into_index) — the durable source of truth. - The free-text columns in its SQLite index that aren't already
covered by
tenant_id/amounts/timestamps/enum filters:deposit.external_ref,ledger_entries.reference/memo, andagent_decisions.trigger/rationale/policy_snapshot_json(crate::deposit,crate::agent_decision, theledger_entriesfunctions incrate::ledger, all going throughwal_crypto:: encrypt_opt/decrypt_opt).tenant_id, ids, amounts, timestamps, and status/type enums are deliberately left unencrypted — every query in the crate filters or sorts on one of those, and AES-GCM ciphertext can't support that.
Enabling this is NOT a live migration. Set
RATION_LEDGER_ENCRYPTION_KEY only against a fresh
RATION_LEDGER_WAL_PATH/RATION_LEDGER_DB_PATH pair (or a pair that
was always written with this exact key). Flipping it on against an
existing plaintext WAL/db, or changing the key against an existing
encrypted one, makes Ledger::open_with_encryption fail recovery
loudly (wal_crypto::Error::Decrypt/Truncated,
LedgerError::Storage/LedgerError::Wal) rather than silently
misreading or corrupting data — that is the intended, safe failure
mode, not a bug to work around. To actually rotate the key: stop
ledger-core, start a brand-new WAL/db path under the new key, and
replay/migrate application-level (there is no in-place re-encrypt
tool for either the WAL or the SQLite columns as of this writing).
Generate a key: openssl rand -base64 32 (must decode to exactly 32
bytes — wal_crypto::Error::WrongKeyLength rejects anything else at
startup, immediately, not on first write).
Disabled by default (env var unset) — every other security control in
this project uses the same opt-in posture for anything that changes an
on-disk format. See infra/terraform/modules/service for this
project's existing Secrets Manager wiring pattern a real deployment
would use to provision this value instead of a bare env var.
"a tenant's reserve floor shows negative headroom"
Symptom: ration status --tenant <id> (or the TUI's tenants tab)
shows a headroom value below zero, or bridge.Service.RationBridgeFunding
logs/AgentDecision entries show headroom_minor_units: 0 on every call
even when you'd expect the tenant to have room.
This should be structurally impossible for ledger-core's own
check_reserve_floor (services/ledger-core/src/ledger.rs): it
computes headroom = balance.available_minor_units - t.reserve_floor_minor_units and returns it as-is, so a genuinely
negative number IS a real signal that available_minor_units has
dropped below reserve_floor_minor_units — which apply_entry is
supposed to prevent for every DEBIT/HOLD by rejecting the entry (as
ApplyOutcome::ReserveFloorViolated) before writing it. If you are
seeing this, work the diagnosis in this order:
- Confirm it's real, not a display artifact.
bridge.Service. RationBridgeFunding(services/card-issuance/internal/bridge/service.go) deliberately clamps negative headroom to zero before using it (if headroom < 0 { snap.ReserveFloorBreached = true; headroom = 0 }) and reports that state asreserve_floor_breachedin the recordedAgentDecision's policy snapshot JSON — queryledger-core'sListAgentDecisionsfor that tenant (or check the TUI's activity feed) and look at theheadroom_minor_units/reserve_floor_breachedfields directly rather than trusting a UI that might itself clamp or mis-render a negative value. - Call
ledger-coredirectly, bypassing gateway, to rule out a proxy bug.check_reserve_floor(tenant_id, 0)(via a raw gRPC call toLedgerCore, orledger-core's own test harness) returns the unclamped(allowed, headroom)pair straight from SQLite'sbalancestable — if gateway and ledger-core disagree, the bug is in gateway's proxying (internal/ledgerclient), not in the ledger. - If
ledger-coreitself reports negative headroom, suspect the WAL replay path, not a live race.apply_entry's in-memory check runs underLedger.inner's mutex for its entire decide-append-commit duration (append_and_commit), so a live concurrent-call race that produces a floor breach should be structurally impossible — the realistic cause is WAL replay reapplying events that already happened, whichinsert_ledger_entry_if_new's idempotency guard (uniqueidempotency_key) is supposed to prevent. Checkdocs/ formal/ledger.tlasection 6's safety property — if it's been re-run against a model that doesn't match the currentapply_entrycode (e.g. after a change toWalEventorapply_entryitself without updatingledger_model.tla), the spec no longer actually covers what's running in production; re-run TLC (docs/formal/ README.md) against the current logic as the first real check here, not as an afterthought. - Check for a tenant whose
reserve_floor_minor_unitswas set negative.create_tenantrejectsreserve_floor_minor_units < 0at creation time, but if that validation was ever bypassed (direct SQLite write, a WAL frame written by tooling other thanLedger::create_tenant), headroom math downstream has no further defense —SELECT reserve_floor_minor_units FROM tenants WHERE id = ?against the ledger-core SQLite index file (RATION_LEDGER_DB_PATH) directly, read-only, to confirm.
"the agent loop is granting $0 bridges"
Symptom: services/agent's polling loop
(internal/loop/runner.go) runs, calls RationBridgeFunding for a
tenant with a genuinely late deposit, but every grant comes back
GrantedMinorUnits: 0 / FullyGranted: false.
This is very often correct behavior, not a bug — bridge.Service. RationBridgeFunding (services/card-issuance/internal/bridge/service.go) has
exactly four reasons to decline, and it always says which one in the
Explanation string it returns and records via RecordAgentDecision
(declineExplanation's four branches). Read the explanation string
first; it names the exact cause:
-
"reserve floor breached"—headroom <= 0. See the previous runbook entry; this is the reserve-floor scenario surfacing through the bridge path specifically. -
"tenant already has an unrepaid bridge grant within the last 24h (velocity cap)"—velocityBlockedfound abridge:-prefixed HOLD entry (bridgeReferencePrefix) withinRATION_BRIDGE_VELOCITY_ WINDOW_HOURS(default 24h,.env.example) with no matching RELEASE. CheckListEntriesfor that tenant for an openbridge:HOLD; if it's actually been repaid, confirm the RELEASE entry'sReferencefield matches the HOLD'sReferenceexactly —velocityBlockedmatches on exact string equality (released[ref]), so any drift in how the reference string is constructed (e.g. a manual RELEASE that doesn't copy the original HOLD'sreference) will falsely keep a tenant velocity-blocked forever. -
"no late deposit on file for this tenant, nothing to bridge against"despite the deposit genuinely being overdue — FIXED 2026-10-23.findRelevantLateDeposit(services/card-issuance/internal/bridge/service.go) used to filter strictly on the deposit's storedstatusfield (d.GetStatus() != DepositStatus_DEPOSIT_STATUS_LATE), but nothing onledger-core's write path (deposit::insertsetsDepositStatus::Expected;deposit::mark_receivedonly ever writesDepositStatus::Received— seeservices/ledger-core/src/ deposit.rs) ever wroteDepositStatus::Lateto a row, so that filter could never match a single real deposit:RationBridgeFundingdeclined every genuinely late deposit with exactly this message, every time, against a real ledger-core. Masked by every unit test that exercised this path (bridge/service_test.go,grpcserver/server_test.go,agent/internal/loop/runner_test.go), which all hand-constructed fakeDeposit{Status: DEPOSIT_STATUS_LATE}fixtures directly — fixtures that also happened to satisfy the real, intended definition (pastexpected_at_unix, zeroreceived_at_unix), so the bug was invisible to the green test suite.findRelevantLateDepositnow computes lateness the same time-based wayledger-core's ownlist_latedoes —ReceivedAtUnix == 0 && ExpectedAtUnix < s.now()— instead of trusting thestatusfield at all. No proto/wire change needed; the existing test suite (including the fixtures above, which already satisfied this definition) passes unchanged. If you still see this decline for a deposit that is unambiguously overdue, checkexpected_at_unix/received_at_unixon the rawListDepositsresponse directly — that is now exactly what this code checks too. -
"computed grant of $X is below the $5 minimum-useful-grant floor"—granted < MinGrantMinorUnits(default 500 = $5.00,RATION_BRIDGE_MIN_GRANT_MINOR_UNITS). The explanation string itself reportsheadroom,bridge cap(60% of the expected late deposit,RATION_BRIDGE_MAX_FRACTION), and the expected deposit amount — if headroom or the bridge cap is unexpectedly tiny, go back to the tenant's balance/reserve-floor state (previous entry) or to whether the deposit'sexpected_minor_unitsis actually what you think it is (ListDeposits).
If the explanation string itself is missing or empty (not one of
the four above), that is a real bug, not expected behavior — check that
s.ledger.RecordAgentDecision isn't failing silently (recordDecision
returns its error up through RationBridgeFunding, so a failure here
should surface as an error from the RPC, not a silent $0) and that
internal/llm/tools.go's tool-call wrapper around RationBridgeFunding
is actually propagating the Explanation field back to whatever
surfaced the "$0" symptom (CLI output, TUI activity feed) rather than
dropping it.
Also check ration_bridge_grants_total{result="declined"} (pkg/metrics/ metrics.go, section 2's named metric) against {result="granted"} +
{result="partial"} over time — a sudden step-change to all-declined
across every tenant (not just one) points at a policy or config
regression (e.g. RATION_BRIDGE_MAX_FRACTION accidentally set to 0),
not a per-tenant data issue.
"audit-log's mirror might be silently falling behind ledger-core"
Symptom: no specific error — this is a preventive check, not a
reactive one. services/ledger-core/src/audit_mirror.rs's mirroring of
every ApplyEntry/RecordAgentDecision outcome into audit-log is
explicitly best-effort and asynchronous (never allowed to block or fail
the original RPC); a network blip, audit-log being down, or a
serialization bug there fails silently from the caller's perspective
— audit_mirror_spawn_append_failures_total (pkg/metrics,
docs/THREAT-MODEL.md's "Known gaps" #3) increments, but nothing
compares that against what actually landed. VerifyChain/verify-chain
(above) only proves the chain is internally consistent — it says
nothing about whether every record that should be there actually
is.
Check: run ration audit-check (services/cli/cmd/ration/ audit_check.go). For every tenant (or just --tenant <id>), it counts
ledger-core's own ListEntries/ListAgentDecisions records and
audit-log's own mirrored ListRecords count of each kind for that
same tenant, fully paginated on both sides, and reports any tenant/kind
pair whose counts diverge by more than --tolerance (default 2 — the
two counts are two independent, non-atomic RPC round-trips, so a
record landing on ledger-core between the two reads is expected
noise, not a real gap). Exits 0 when everything matches; exits 1 and
prints every divergent pair otherwise, so this is safe to run from a
monitoring cron job, not just by hand:
ration audit-check # every tenant
ration audit-check --tenant <tenant-id> # one tenant
ration audit-check --tolerance 0 # strict, no slack for in-flight writes
A real, sustained divergence (not a one-off blip that clears on the
next run) means the mirror has a real, ongoing gap — check
audit_mirror_spawn_append_failures_total first (a nonzero, growing
counter directly explains it), then whether AUDIT_LOG_ADDR is even
configured on the ledger-core instance in question (unset disables
mirroring entirely, by design — see AuditMirror::disabled), then
audit-log's own process health/logs for the window the gap opened in.