Adds a focused NEAR Hackathon 2026 entry built on Ration's existing,
real audit-log service rather than a Ration-on-NEAR rewrite: a minimal
NEAR smart contract (contracts-near/audit_anchor) storing/verifying
{tenant_id, chain_tip_hash, sequence, occurred_at_unix}, plus a
from-scratch Go NEAR transaction signer and CLI
(services/audit-log/internal/nearanchor, cmd/near-anchor) that reads
audit-log's real local chain tip and anchors it on-chain.
Verified live on NEAR Testnet (account ration-anchor-286ca315f1.testnet):
contract deployed and code_hash-matched, two successful on-chain
anchors, one deliberate replay correctly rejected by nearcore's own
sequence guard, and a real (non-mocked) Go integration test passing
against the live contract. Transaction encoding cross-checked
byte-for-byte against near-api-js. Zero regressions in existing Go/Rust
suites. Full detail and every tx hash in docs/near-hackathon/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
IssueCard's reserve-floor/ledger failure path calls suspendAfterFailure
to zero out the just-created Airwallex card's limit, but the call's
error was discarded with a bare `_, _ = ...`. Unlike AdjustCardLimit's
mirror (revertCardControls), which already logs loudly on failure,
this was the only cleanup path with zero visibility — and the only one
that matters, since IssueCard returns before ever adding the card to
s.cards, so there is no persisted record anywhere in this service for
a reconciliation pass to retry. A transient Airwallex outage at exactly
this moment left a real, live card at its full requested spend limit
with no ledger backing and no way for an operator to find out.
Reproduced live (repro_suspend_bug_test.go, kept as a permanent
regression test): forced SetCardControls to fail during the suspend
path, confirmed the live Airwallex-side card still carries its full
limit, and confirmed the failure is now logged with the card id so an
operator can suspend it by hand. Same logging discipline as
revertCardControls and persist() elsewhere in this file.
go test -race ./... clean across all services/card-issuance packages.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
idempotency_key is UNIQUE across the whole ledger_entries table, not
scoped to (tenant_id, idempotency_key). apply_entry_checked's replay
lookup returned the matched entry as Applied regardless of whether its
tenant_id matched the caller's requested tenant_id, pairing the OTHER
tenant's entry (reference/memo/amount) with THIS tenant's own
unaffected balance — reporting success for a write that never actually
happened, with no error anywhere.
Every real caller today derives idempotency_key from a CSPRNG/UUIDv4
id (idgen.New, Uuid::new_v4), so a genuine collision across tenants is
cryptographically negligible in practice — but the lookup itself never
enforced that, which matters for a platform whose hackathon track is
explicitly multi-tenant isolation. Add LedgerError::IdempotencyKeyTenantMismatch
and fail loudly instead of silently mixing tenants.
Verified: cargo build --release clean, cargo check --release --tests
clean (new test type-checks), cargo fmt --check clean on touched
files, cargo clippy --release exit 0, and a full cargo test --release
--test idempotency run (including the new
replaying_another_tenants_idempotency_key_is_rejected_not_silently_applied
regression test) completed with exit code 0 on this box despite heavy
concurrent sibling-agent load intermittently hanging repeat attempts
at process exec (unrelated amfid/syspolicyd contention, not a code
issue — confirmed via `sample` showing the test binary stuck at
dyld_start, and the same stall reproducing on the already-built,
previously-working main ledger-core binary too).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
handleAdjustCardLimit forwarded the model-supplied card_id straight to
gateway.AdjustCardLimit with no check that the card belongs to the
tenant this decision cycle is for. BuildToolSchemas's own doc comment
claims tenant_id is deliberately never a model parameter "so the model
can never redirect an action at another tenant" — true for
ration_bridge_funding/issue_card/notify_tenant (all pass tc.TenantID
explicitly), false for adjust_card_limit, whose only tenant binding is
an indirect card_id->tenant lookup that gateway's own tenantauth
interceptor explicitly skips for an operator credential (exactly what
this agent always authenticates as). A hallucinated or
prompt-injected card_id (e.g. via the tenant-controlled TenantName
field already rendered verbatim into the prompt) could silently
tighten or loosen ANY tenant's card limit.
Reproduced live against the real bare-metal stack (ledger-core, auth,
tenant-registry, card-issuance, gateway): minted the agent's real
operator JWT, called the real exported tool handler with tenant A's
TenantID but tenant B's real card_id — tenant B's card limit dropped
from 1,000.00 EUR to 0.00 EUR and flipped to RATIONED, with zero
errors anywhere in the chain. Fix: resolve the card via gateway.GetCard
first and decline (recording a real DECLINE audit entry) when its
tenant_id doesn't match tc.TenantID. Re-ran the identical repro
post-fix: cross-tenant attempt now rejected with the card untouched;
same-tenant adjustment still succeeds normally.
go build/vet/test clean for every go.work module; 0 regressions.
append_and_commit returned a plain Err whenever the SQLite apply/commit
step failed, even after inner.wal.append() had already fsynced the
event. That frame is then guaranteed to be replayed by the next
Ledger::open regardless, so a caller who saw "failed" and retried would
get a second, distinct record (every mutating RPC's id is
server-generated) once the first one also resurrected on restart - a
real duplicate-record risk for a financial ledger, most reachable via
exactly the kind of disk pressure this shared box has hit repeatedly.
Reproduced live against the real append_and_commit via a scratch
harness forcing the apply closure to fail: the "failed" tenant
materialized after a simulated restart. Fix aborts the process instead
of returning a soft error once the WAL append has already succeeded,
leaning on the already-tested replay-on-open path to reconcile state
before any other call is accepted. Verified the abort fires correctly
(SIGABRT, correct diagnostic) via an isolated cargo test invocation,
then removed the temporary repro/verification tests. Also verified live
against the real running release binary (standalone ledger-core,
tenant auth disabled): normal CreateTenant calls are unaffected.
cargo test --lib: 51/51 passed, 0 regressions. cargo build --release
clean. No new test files added.
6 of 7 services that mint a credential from auth (agent, card-issuance,
tenant-registry, webhook-ingestion, cli, tui) called authclient.Dial(addr)
with zero DialOptions, while loading and correctly applying mTLS creds to
every OTHER peer dial in the same binary. gateway was the only caller that
got this right (authMTLS/authDialCreds pattern).
Reproduced live: with AUTH_MTLS_ENABLED=true (generated via
infra/dev-ca/generate-dev-ca.sh, the exact production-hardening path
docs/SCALE-EXPANSION.md section 8 documents), auth's server correctly
requires and verifies a client cert (tls.RequireAndVerifyClientCert) — a
plaintext dial against it doesn't degrade gracefully, it fails the TLS
handshake outright ("error reading server preface: EOF"/connection reset).
Confirmed with the real pre-fix card-issuance binary: it crashes at startup
("mint operator JWT: ... error reading server preface: EOF") the moment
mTLS is actually turned on, for every one of the 6 affected services.
Also found and fixed a second instance of the same root cause while
checking sibling call sites: the CLI's apikey issue/revoke/list and tenant
create/list commands each dial auth directly (bypassing clients.go's
dialClients entirely) and never attached mTLS creds either; tenant
create/list's own tenant-registry dial had the identical gap. Added a
shared dialAuthClient() helper in clients.go so this bug class can't
quietly reappear a 6th time within the same CLI binary.
Verified live end-to-end against the real 8-service bare-metal stack with
mTLS fully enabled (dev-ca-generated certs for every service): auth,
card-issuance, tenant-registry, webhook-ingestion, gateway, and agent all
now mint/verify successfully over mTLS at startup; a full CLI cycle (seed,
status, apikey issue/list/revoke, tenant create/list, card issue,
simulate-deposit --late, agent auto-grant, audit-check, verify-chain) ran
clean end to end. Re-verified the default mTLS-disabled path (every prior
round's own verification baseline) is completely unaffected by re-running
the identical demo cycle with mTLS off.
go build/go vet clean across all 11 go.work members, gofmt clean on every
touched file, go test clean for every affected package (0 regressions).
tools.go's handleRationBridgeFunding tagged EVERY nil-error gateway
response as AGENT_ACTION_RATION_BRIDGE, even when GrantedMinorUnits was
0. bridge.Service.RationBridgeFunding has three documented paths that
return exactly that shape (below the $5 minimum-useful-grant floor —
including its own independent, ledger-entry-based velocity check
disagreeing with this agent's decision-history-based
hasOpenGrantWithinWindow — a ledger-core-rechecked reserve-floor
violation, or "deposit already received"). None of those is a Go
error, so every one of them was recorded as if money had actually
moved.
This both corrupted the audit trail (a "granted $0" decision labeled
as a bridge grant) and fed hasOpenGrantWithinWindow's wasActuallyGranted
check a false "actually executed" signal, since the snapshot map always
carries a granted_minor_units key even at 0 — permanently
velocity-capping a tenant that was never actually granted anything, the
same fund-stranding-via-misclassification failure mode round 6 fixed
via a missing deposit_id, reached here instead via a wrong action tag.
Fix: tag the decision AGENT_ACTION_DECLINE whenever
GrantedMinorUnits <= 0, regardless of error.
Reproduced live against the real running stack (real auth + ledger-core,
real RecordAgentDecision/ListAgentDecisions, a stubbed gateway forcing
the exact zero-grant/nil-error response) before fixing, and confirmed
fixed the same way afterward. New regression tests in runner_test.go
cover both the misclassification itself and its downstream
velocity-cap consequence; both fail on the pre-fix code and pass after.
AdjustCardLimit/CloseCard's cardLocks deliberately let two DIFFERENT
card ids mutate concurrently, but persist() rewrites the ENTIRE cards.json
from a full map snapshot every time it runs, with no ordering relative to
any other in-flight persist() call. If one call's snapshot (taken before a
concurrent call's own update landed) finished its disk write AFTER the
other's newer, correct write, the file ended up reverting the other
call's already-"successful" limit change the moment it hit disk — a
later restart would then reload the stale value while ledger-core's real
Held balance still reflects the larger, correct amount, permanently
stranding those funds the same way round 25's cardLocks gap did, just
via a lost disk write instead of a lost in-memory update.
Reproducing this also caught a second, closely related `-race`-detected
bug: persist()'s snapshot only copied the map's bare *Card pointers, so
saveCardsFile's protojson.Marshal (run after s.mu is released) could read
a card's fields while a concurrent AdjustCardLimit/CloseCard mutated that
exact same shared object in place.
Fix: persistMu now serializes persist()'s entire snapshot-then-write
critical section, and the snapshot deep-copies each card (cloneCard,
the same proto.Clone helper already used elsewhere) instead of aliasing
the live map's pointers.
New test TestPersist_ConcurrentDifferentCardsDoesNotLoseEitherUpdate uses
a beforeSaveForTest hook to force the exact interleaving (real wall-clock
timing can't reliably do it); confirmed via falsification it fails 20/20
without the fix (lost update + data race) and passes 50/50 with it.
Verified live against the real 8-service stack: 30 concurrent
adjust-limit calls across 2 cards, on-disk state matched in-memory/ledger
state exactly, survived a real card-issuance restart.
go build/go vet/go test -race clean across all 11 go.work modules.
Round 28 of today's audit pass (plans/airwallex-hackathon.md).
RationBridgeFunding reads a deposit as late, then several RPC round
trips later applies a HOLD referencing that deposit_id — with nothing
serializing that gap against webhook-ingestion's own, separate-process
handleTransactionCreated marking the SAME deposit received in between.
mark_deposit_received_with_entry only releases bridge holds that exist
at the moment it runs, so a hold created afterward is permanently
unreleasable. Round 25/26's per-tenant locks live in different OS
processes and cannot fix this.
Fixes it the same way apply_entry already re-validates the reserve
floor atomically: ApplyEntryRequest gains an optional
require_deposit_outstanding_id, checked under ledger-core's own single
mutex before any mutation (apply_entry_checked), returning
DepositAlreadyReceived instead of applying the hold. bridge.Service
now always sets this on its bridge-hold ApplyEntry call and declines
cleanly instead of stranding funds.
Reproduced the exact interleaving via a deterministic raceLedger test
(confirmed it fails without the fix, passes with it, under go test
-race). Live-reproduced the race window against the real 8-service
stack; happy path and full build/test matrix unaffected.
Same missing-lock/TOCTOU class round 25 fixed for card-issuance's
AdjustCardLimit/CloseCard (per-card lock) and RationBridgeFunding
(per-tenant lock), found on a third, previously-unaudited money-moving
path: handleTransactionCreated's ListDeposits -> match ->
MarkDepositReceived sequence had nothing serializing it per tenant.
Two genuinely concurrent webhook deliveries for the same tenant (e.g.
two outstanding deposits of the same amount settling close together —
Go's net/http serves each request on its own goroutine) could both
read the deposit list before either call landed, so both independently
picked the same "oldest not-yet-received" deposit. ledger-core's own
MarkDepositReceived is idempotent per deposit_id, so this never
double-credited — but the second transaction's real, already-arrived
money was then never matched to anything: its deposit stayed Expected
forever, permanently understating the tenant's available balance.
Reproduced live against the real bare-metal stack before fixing: two
concurrent signed webhook deliveries for two $750.00 deposits on the
same tenant both matched deposit bcdb1859 via amount_heuristic; the
sibling deposit d234a440 was never credited (Available stayed flat at
one deposit's worth instead of rising by both). Fixed with a per-tenant
sync.Mutex (webhookhandler.Handler.tenantLocks/lockTenant), the
identical lazily-created-never-removed pattern cardsvc.Service.lockCard
and bridge.Service.lockTenant already established, serializing the
entire check-then-act critical section per tenant_id. Re-ran the
identical live repro post-fix: the two transactions correctly matched
the two distinct deposits, and Available rose by the full $1,500.00
(both $750.00 deposits settled).
New regression test (TestHandleAirwallexWebhook_ConcurrentSameAmount-
DepositsBothSettle) uses a concurrency-aware fake ledger whose first
ListDeposits call sleeps after taking its snapshot, deterministically
widening the TOCTOU window instead of relying on real scheduling.
Confirmed it fails 20/20 on the pre-fix code and passes 20/20 with the
fix, both under `go test -race`. `go build`/`go vet`/`go test ./...`
clean across all 11 go.work members; `make build-go` clean.
bridge.Service.RationBridgeFunding's rule-3 velocity check
(velocityBlocked) reads recent ledger-core entries to decide whether a
tenant already has an unrepaid bridge grant, then — much later —
applies a new HOLD via ApplyEntry if not blocked. Nothing serialized
that read-then-write per tenant, so two concurrent calls for the SAME
tenant both read velocityBlocked=false before either HOLD landed, and
both got granted. ledger-core's own ApplyEntry is defense in depth for
the reserve-floor invariant only; it has no notion of "velocity cap" or
"bridge cap of this specific late deposit" — those exist purely in this
method's own check-then-act Go logic.
Reproduced live against the real running stack: seeded a tenant with a
single $10,000.00 late deposit (60% bridge cap = $6,000.00, headroom far
above that), fired two concurrent RationBridgeFunding calls for
$6,000.00 each sharing one gRPC connection — both returned
granted=$6,000.00/fully_granted=true, and the tenant's real Held balance
rose by the full $12,000.00 — 120% of the single late deposit this was
meant to cap at 60%, and two simultaneous outstanding grants where rule
3 allows at most one.
Fix: added lockTenant, a lazily-created per-tenant sync.Mutex
serializing RationBridgeFunding's entire critical section (read
velocity/headroom -> decide -> ApplyEntry -> RecordAgentDecision) for
one tenant, the same pattern card-issuance's own cardsvc.lockCard
already uses for AdjustCardLimit/CloseCard. New
TestRationBridgeFunding_ConcurrentCallsRespectVelocityCap (with its own
mutex-protected concurrencyFakeLedger that feeds applied entries back
into ListEntries, unlike the file's existing sequential-only fakeLedger)
fires two real concurrent grants and asserts total granted never exceeds
the single bridge cap — fails 20/20 runs on the pre-fix code (both
calls granted, double the cap) and passes reliably 20/20 with the fix,
clean under `go test -race`.
Verified live against the real fix: rebuilt only bin/card-issuance,
restarted that one process, re-ran the identical two-concurrent-call
repro against a fresh tenant/deposit — exactly one call granted the
full $6,000.00 cap, the other correctly declined on the velocity cap,
confirmed via `ration status` that Held rose by exactly $6,000.00, not
$12,000.00. go build/vet clean across all 11 go.work members; go test
clean for every package in all 11 modules (0 regressions). gofmt -l on
service.go matches an already-documented pre-existing import-ordering
diff (confirmed via git stash, not introduced here); service_test.go is
gofmt-clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
AdjustCardLimit and CloseCard each read a card's current
LimitMinorUnits, called Airwallex, called ledger-core's ApplyEntry for
the delta, then wrote the card's new absolute LimitMinorUnits back —
but the initial read-then-later-write was never serialized against a
second concurrent call for the SAME card id. ledger-core's own
apply_entry is atomic and additive (two concurrent HOLDs both land
correctly on the real held balance), so the divergence was entirely in
this service's own cached field: the final write is "last writer wins"
on an absolute value, not oldLimit+delta, so one call's increase could
silently vanish from card.LimitMinorUnits while ledger-core still held
it for real.
Reproduced live against the real running stack (mock Airwallex, real
ledger-core): issued a card with a $1,000.00 limit, fired two
concurrent AdjustCardLimit calls (+$1,000.00 and +$2,000.00) sharing
one gRPC connection. Both succeeded; ledger-core's real Held balance
correctly became $4,000.00, but card.LimitMinorUnits ended at
$2,000.00 (whichever call wrote last). Closing that card then released
only $2,000.00 (CloseCard's release amount is always
card.GetLimitMinorUnits()), leaving $2,000.00 permanently stuck in the
tenant's Held balance with zero live cards referencing it — unreachable
by any code path in this repo ever again.
Added lockCard: a lazily-created per-card-id sync.Mutex serializing
AdjustCardLimit/CloseCard's full critical section (read → Airwallex →
ledger ApplyEntry → write) for one card, while leaving every other
card free to proceed concurrently. New
TestAdjustCardLimit_ConcurrentIncreasesStayConsistentWithLedger fires
two real concurrent increases and asserts card.LimitMinorUnits always
equals the sum of every HOLD/RELEASE ledger-core actually applied —
fails (both as a logical mismatch and as a go test -race data race) on
the pre-fix code in 9/10 runs, passes reliably 20/20 with the fix.
Verified live: rebuilt only bin/card-issuance, restarted that one
process (card metadata reloaded from data/cards.json), re-ran the
identical two-concurrent-increase repro against a fresh card — second
call now correctly blocks until the first commits, computes its delta
against the real current limit, and the two calls' effects compose
correctly instead of diverging. go build/vet clean across all 11
go.work members; go test -race ./... clean for every card-issuance
package. gofmt -l on the touched files matches round 21's already-
documented pre-existing import-ordering diff on service.go (confirmed
via git stash, not introduced here); service_test.go is gofmt-clean.
ration status (the single most-run command in this repo, per round 13's
own framing)'s printTenantHeader printed tenant.Name completely raw, with
no sanitization at all. Same gap, same root cause, across the TUI: the
Tenants tab table, the Activity tab's per-row TenantName, and the card
form's tenant picker + adjust-limit infoLine all rendered a Tenant's own
Name raw too. services/tenant-registry's Create only validates name
non-emptiness, no charset check — round 16 already proved a raw ESC byte
survives into a persisted Tenant record via "tenant create --name".
Rounds 13/16/23 sanitized CardholderName, the CLI's own tenant-create
echo, and the Activity feed's Summary, but never Tenant.Name as read
back through these five separate call sites.
Reproduced live against the real bare-metal 8-service stack
(scripts/start-individually.sh): created a tenant with
--name $'Evil\x1b[31;1mINJECTED\x1b[0m Status Co', then byte-counted real
stdout — `ration status --tenant <id>` carried 2 raw ESC bytes pre-fix,
0 after. Drove the real bin/tui through a tmux pty: Tenants tab and the
card-issue form's tenant picker both rendered the literal bracket text
pre-fix verification showed was otherwise a live ANSI escape (confirmed
via `tmux capture-pane -e` byte inspection, distinguishing it from tmux/
lipgloss's own legitimate styling codes).
Fix: sanitizeControlChars(tenant.Name) in status.go's printTenantHeader;
format.SanitizedCell(t.GetName(), ...) in tui/view/tenants.go;
textsanitize.StripControlChars on ActivityRow.TenantName in
tui/view/activity.go; and on both the tenant picker's display name and
newAdjustLimitForm's tenantName param in tui/model/cardform.go — all via
the same helpers this repo's prior sanitization rounds already
established.
go build/go vet/go test clean for services/cli and services/tui (0
regressions), gofmt clean, full make build-go (10 binaries) clean.
services/gateway/internal/grpcserver/server.go's publish() calls embed
IssueCard's raw CardholderName and AdjustCardLimit's raw Reason
directly into the activity feed's Summary string, with no
sanitization anywhere upstream. Round 13 fixed this exact
vulnerability class for CLI status/card list and the TUI's Cards
tab/adjust-limit modal; round 16 fixed a sibling case on `tenant
create`'s own printout — neither ever looked at the gateway's
StreamActivity Summary field, a separate code path feeding the TUI's
Activity tab.
Reproduced live: issued a card via the real 8-service bare-metal
stack with --cardholder containing a raw ESC byte + ANSI color code,
watched the TUI's Activity tab through a real tmux pty — tmux
capture-pane -e showed a genuine, uninterpreted `^[[31mINJECTED^[[0m`
escape sequence sitting in the pane content, not literal bracket
text. Confirmed the same gap on AdjustCardLimit's --reason.
Fix: strip control characters (pkg/textsanitize.StripControlChars,
same helper decisions.go already uses) on Summary at the render
boundary in services/tui/internal/view/activity.go, covering both
call sites (and any future free-text field embedded in a published
activity Summary) with one change. Re-verified the identical live
repro for both IssueCard and AdjustCardLimit: both now render as
literal bracket text, matching the established lipgloss styling
codes elsewhere in the same pane which still show their real `^[`
prefix.
go build/go vet/go test ./... clean for services/tui (0 regressions),
gofmt -l clean. Full module-by-module build (pkg + every service +
sdk/go/ration) clean.
handleTransactionCreated's tier-1 match (payload.Reference ==
deposit.ExternalRef) never checked the transaction's own amount.
MarkDepositReceived has no "amount actually transferred" field at all
(proto/ledger.proto) — it always credits the matched deposit's full
pre-recorded expected amount. Reproduced live against the real running
stack: a genuine signed webhook carrying a $5,000.00 deposit's
reference but only $1.00 in amount_minor_units would have marked that
deposit fully received and credited the tenant $5,000.00 for a $1.00
transfer.
matchDepositByReference now also requires the transaction's amount to
equal the deposit's expected amount before trusting the reference
match; a reference match that fails the amount check falls through to
the existing amount-heuristic tier instead of being treated as exact.
Verified live: rebuilt only webhook-ingestion, sent a real HMAC-signed
webhook with the correct reference but wrong amount — rejected ("no
matching expected deposit found"), deposit stayed unreceived, balance
untouched. Then sent the same reference with the correct amount —
matched_by=reference, deposit credited normally. go build/vet/test
clean across all 11 go.work members.
A --currency that didn't match the tenant's own currency was only ever
caught by ledger-core's ApplyEntry CurrencyMismatch check — but a
zero-limit card never calls ApplyEntry at all, so `ration card issue
--currency EUR --limit 0` against a USD tenant succeeded and created a
real (mock) Airwallex card. The card then sat permanently stuck at a
zero limit: the very next `card adjust-limit` above zero failed with a
confusing low-level "currency mismatch: USD vs EUR" error from
ledger-core, with no way to ever fund it. Reproduced live against the
real 8-service stack before fixing.
Reject the mismatch upfront in IssueCard, before any Airwallex card is
created, with a clear actionable error. Added a regression test
covering the zero-limit path specifically (the one that previously
skipped the check entirely). go build/vet/test clean across every
workspace member; verified live against the rebuilt card-issuance
binary that the same repro now fails immediately instead of creating a
stuck card.
decodeJSONBody (IssueCard, AdjustCardLimit, RationBridgeFunding,
SimulateDeposit) read r.Body with a bare json.NewDecoder and no bound,
before any auth check ever ran. Reproduced live: one unauthenticated
300MB POST to /v1/cards drove gateway's RSS from ~20MB to ~1.3GB in
~5.6s. Capped at 1MB via http.MaxBytesReader (same limit and
justification services/webhook-ingestion's handler already uses for
the identical class of payload), mapped to a 413 instead of the
existing 400. Verified live post-fix: the same 300MB body now returns
413 in ~2.5ms with RSS flat at ~25MB; real IssueCard/AdjustCardLimit/
SimulateDeposit/RationBridgeFunding round-trips through the REST
facade with a real minted API key still succeed unchanged. go build/
go vet/go test clean for services/gateway, full make build (Rust+Go+
SDK) clean, 0 regressions.
GetDashboardSnapshot and StreamActivity (empty tenant_id, the documented
"all tenants" mode) carry no tenant_id/card_id, so internal/ratelimit's
gRPC-layer interceptor falls back to peer-address keying. But the REST
facade reaches gateway's own gRPC server over one single, long-lived
self-dial connection (cmd/gateway/main.go's selfConn) — so every REST
caller of GET /v1/dashboard/snapshot, GET /v1/dashboard/activity/stream,
and GET /v1/activity (no tenant_id) collapsed into ONE shared token
bucket, the opposite of "one noisy caller can't starve every other
caller" that package promises. Most exposed on the dashboard: a public,
zero-login, judge-facing page explicitly designed for multiple
simultaneous viewers.
Add a second, HTTP-layer limiter keyed by the real client's remote
address, applied only to those three call sites — every tenant/card-
scoped route keeps its existing, already-correct gRPC-level protection
untouched.
Verified live against the real bare-metal stack: a concurrent burst of
40 connections to /v1/dashboard/activity/stream now correctly splits
30 (burst) succeed / 10 rejected with 429, keyed per real client IP;
confirmed tenant-scoped /v1/activity calls are unaffected (10/10 200s).
go build/vet/test clean for services/gateway, 0 regressions.
Round 13 sanitized CardholderName across card.go/status.go/tui but never
audited every other free-text field's echo path. `ration tenant create`
printed the operator-supplied --name straight back to the terminal raw,
while `ration tenant list` (same field) already sanitized it — a real
terminal-escape-injection gap on tenant onboarding's own success message.
Reproduced live: creating a tenant with an ESC-laden name produced 2 raw
ESC bytes in stdout (byte-counted, not visually inspected). Fixed by
reusing the same sanitizeControlChars helper tenant list/apikey
list/card already use. Re-verified: 0 ESC bytes post-fix, list/create
both render the literal bracket text. go build/vet/test clean for
services/cli, full make build clean, 0 regressions.
Round 14 found but deliberately deferred this: both methods applied
their ledger HOLD/RELEASE before calling Airwallex, with no rollback if
the Airwallex call then failed. A failing retry (e.g. a stale/orphaned
Airwallex card id) would re-apply the same ledger mutation every time,
leaking reserve-floor headroom without bound.
Reordered both to match IssueCard's existing safe pattern: Airwallex
first, ledger mutation second. AdjustCardLimit now reverts the Airwallex
limit (new revertCardControls, mirrors suspendAfterFailure) if the
follow-up ledger call fails. CloseCard's Airwallex close is terminal (no
reopen endpoint exists) so there's no symmetric revert, but the reorder
alone makes its failure mode fail-safe instead of a leak: a failed
ledger release now just leaves the hold stuck in place for a retry to
clear, rather than double-releasing it.
Reproduced live on the pre-fix bare-metal stack: a restart-orphaned
card's AdjustCardLimit leaked +1,000.00 GBP of ledger HOLD per failing
retry (3 attempts -> 1,000 -> 2,000 -> 3,000 Held) despite every call
returning an error to the client. Rebuilt with this fix and reran the
identical repro: Held stayed exactly flat across repeated failures for
both AdjustCardLimit and CloseCard. Confirmed happy-path increase,
decrease, close, and idempotent re-close still behave correctly.
go build/vet/test clean for services/card-issuance, 0 regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cardsvc.Service's own `cards map[string]*cardissuancepb.Card` was the
only place this repo stores Card metadata (ledger-core's proto has no
Card message), and it was pure in-memory state with zero durability.
A card-issuance restart lost every issued card's id/cardholder/limit/
status while the ledger HOLD backing its limit lived on in ledger-core
forever, unreachable by GetCard/AdjustCardLimit/CloseCard ever again
(all three look the card up by id first). card_issuance.proto's own
service comment also claimed this service was "stateless... safe to
run as N replicas" — false given this in-memory map; corrected.
Adds a file-backed store (services/card-issuance/internal/cardsvc/
store.go): atomic write-then-fsync-then-rename JSON file, loaded on
startup via a new EnablePersistence(path) method (opt-in, so existing
unit tests that never call it are unaffected) and written after every
IssueCard/AdjustCardLimit/CloseCard. New RATION_CARD_ISSUANCE_DATA_DIR
env var (default ./data, same convention as RATION_AUDIT_LOG_DATA_DIR),
wired in main.go.
Verified live against the real bare-metal stack: issued a card,
confirmed data/cards.json held it, killed and restarted card-issuance,
and confirmed `ration card get`/`ration status` still found the card
(pre-fix this would have been NotFound). go build/vet/test clean for
card-issuance (0 regressions), full `make build` clean.
Does not make this service safe to run as N replicas (each replica
would still load/save its own independent copy of the file) — that
needs a real shared store, the same class of gap already named and
left open for services/auth's apikey.Store; documented, not attempted,
in this round.
CardholderName is operator-authored free text (ration card issue
--cardholder, and the TUI's own issue-card form) — the same class of
terminal-escape-injection risk already closed for Trigger/Rationale/
PolicySnapshotJson. card.go's printCard (card issue/get/adjust-limit/
close) already sanitized it, but status.go's printCards (used by both
`ration status` and `ration card list`) printed it raw, and the TUI's
Cards tab table (view/cards.go) and the adjust-limit modal's infoLine
(model/cardform.go) did too. A cardholder name containing an ANSI
escape sequence would inject into `ration status` output — the single
most commonly run command in this whole repo's demo flow — and into
the TUI dashboard a judge/operator is meant to watch live.
Fixed all three raw render sites to go through the same
sanitizeControlChars/format.SanitizedCell/textsanitize.StripControlChars
helpers already used everywhere else free text is displayed.
Verified live against the real bare-metal 8-service stack: issued a
card with a cardholder name containing a real ESC (0x1b) byte +
"[31;1mINJECTED[0m", confirmed `ration status` and `ration card list`
output now contain zero raw ESC bytes (byte-counted, not just visual),
and drove the real TUI through a tmux pty — both the Cards tab table
and the adjust-limit modal's header render the literal bracket text
with no live escape sequence. go build/go vet/go test clean for cli
and tui, gofmt clean, 0 regressions.
CARD_STATUS_SUSPENDED/CLOSED have existed in common.proto since the
original build, and the Airwallex client package even defined
CardStatusSuspended/CardStatusClosed constants, but nothing in this
repo ever transitioned a stored card into either state: card-issuance's
airwallex.Client interface only exposed IssueCard/GetCard/
SetCardControls (limit/controls only, never status), so a leaked
credential, an offboarded cardholder's card, or any card that should
stop spending forever had no real operator path to close — only
"adjust-limit" to a smaller (but still nonzero-possible, still active)
limit.
Adds card_issuance.proto/gateway.proto's CloseCard RPC, a real
airwallex.Client.CloseCard on both Mock and Live (POST
/api/v1/issuing/cards/{id}/close), cardsvc.Service.CloseCard (releases
the card's full remaining ledger HOLD back to the tenant's available
balance, idempotent on an already-closed card), the gateway fan-out,
and `ration card close --card <id>`. AdjustCardLimit now rejects a
closed card instead of silently reopening a zombie hold.
tenantauth/ratelimit needed zero changes: both resolve card_id via
reflection, so CloseCardRequest's card_id field is automatically
tenant-scoped and rate-limited the same way GetCard/AdjustCardLimit
already are.
Verified live against the real bare-metal 7-service stack
(scripts/run-demo.sh): issued a real card (Held 1,000.00 USD), closed
it (status CLOSED, Held back to 0.00, Available back to 7,500.00),
re-closed it (idempotent — Available unchanged, no double-release),
confirmed adjust-limit on the closed card is rejected, and confirmed
closing an unknown card_id surfaces the same NotFound-wrapped error
GetCard/AdjustCardLimit already do. verify-chain: intact (5 records).
`go build`/`go vet` clean across every Go module (gateway,
card-issuance, cli, tui, auth, tenant-registry, webhook-ingestion,
agent, audit-log, sdk/go/ration), full `make build` (Rust+Go+SDK)
clean, 0 regressions.
tenant-registry's own gRPC server (CreateTenant/GetTenant/ListTenants)
had no inbound auth check at all, unlike gateway's internal/tenantauth
and ledger-core's src/tenantauth.rs. Since it mints and attaches its
own pre-provisioned operator JWT to every ledger-core call it forwards
regardless of caller identity, ANY network caller to its gRPC port
(published 0.0.0.0:7011 in docker-compose.yml) could onboard unlimited
platform tenants or list every tenant's name/currency/reserve-floor
with no credential whatsoever — a real, newly-live gap since round 9
wired `ration tenant create/list` to this exact RPC surface.
Adds services/tenant-registry/internal/operatorauth (mirrors gateway's
tenantauth credential handling, simplified for the operator-only case),
gated by RATION_TENANT_REGISTRY_REQUIRE_OPERATOR_AUTH (true by
default). `ration tenant create`/`list` now mint and attach their own
operator credential, same pattern the CLI's other commands already use
for gateway/ledger-core.
Verified live against the real bare-metal stack: a throwaway gRPC
client with no credential gets Unauthenticated, a garbage bearer token
gets Unauthenticated, a valid but non-operator (tenant-scoped) API key
gets PermissionDenied, and the real `ration tenant create`/`list`
commands succeed end-to-end with their own minted operator credential.
go build/vet/test clean across tenant-registry and cli, 0 regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
services/tenant-registry implements real tenant CRUD (CreateTenant/
GetTenant/ListTenants, validated, delegating to ledger-core) and is
started by every demo/deploy path (run-demo.sh, start-individually.sh,
docker-compose.yml) alongside every other service — but nothing in
this repo ever dialed it. `ration seed` creates its 3 fixed demo
tenants by calling ledger-core's own CreateTenant RPC directly
(internal/demoseed), bypassing tenant-registry entirely, and no CLI/
TUI/gateway command ever called it either. An operator had no way to
onboard a new platform tenant into a running deployment without
editing demoseed.Tenants() in Go source and rebuilding.
Adds internal/tenantregistryclient (mirrors the CLI's existing
authclient Dial/Close shape) and `ration tenant create`/`ration tenant
list`, following the same tabwriter/moneyfmt conventions apikey.go and
status.go already established.
Verified live against the real 8-service bare-metal stack
(scripts/start-individually.sh): `ration tenant list` on an empty
store reports no tenants; `ration tenant create --name "Acme Platform
Payouts" --currency USD --reserve-floor-minor-units 500000` returns a
real tenant_id; it then appears in both `ration tenant list` and
`ration status --tenant <id>` (ledger-core's own independent read
path), proving tenant-registry persisted through to the same
authoritative store `ration seed` and every other command already
read from — confirmed again after running `ration seed` alongside it
(all 4 tenants listed together). Missing --name and a non-3-letter
--currency both fail fast before dialing anything. go build/go vet/go
test clean for services/cli, 0 regressions; full `make build-go`
clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PLAN-2026-10-23-DEEP.md's own "Remaining" note for the Cards tab was
still true: no in-TUI IssueCard/AdjustCardLimit form, CLI/API only.
Round 7 closed the CLI half of this gap; this closes the TUI half.
- New modal (services/tui/internal/model/cardform.go,
internal/view/cardform.go): press `n` on the Cards tab to issue a
card (tenant picker defaulting to the selected row's owner,
cardholder/limit/interval/currency fields), `a` to adjust an
existing card's limit (pre-filled from the selected row). Submits
via two new client.Clients methods (IssueCard/AdjustCardLimit) that
call the same gateway RPCs the CLI's `ration card` commands use.
On success the form closes and triggers an immediate refresh; on
failure (validation or a real RPC rejection) it stays open with the
error shown and prior input preserved, so the operator can fix and
retry without re-entering everything.
- Real bug found live, not by reading code: the field-typing handler
only ever appended msg.Runes[0], assuming one KeyMsg per keystroke.
bubbletea's input reader batches consecutive printable bytes read
in one syscall into a single KeyMsg carrying multiple runes — typing
(or pasting) a cardholder name arrived as one such batched message,
so every character after the first was silently dropped. Fixed to
append every rune in the batch; re-verified with multi-character
names/limits typing correctly.
Verified live against the real bare-metal 7-service stack (audit-log,
ledger-core, auth, tenant-registry, card-issuance, webhook-ingestion,
gateway — started individually with inter-service delays, the same
round-6/7-documented workaround for this shared box's sibling-agent
load), driven through a real pty via tmux send-keys (not code reading):
seeded 3 tenants, opened the issue-card form, picked a tenant, typed
cardholder "Jane Operator", limit 250000, interval daily, submitted —
card appeared in the Cards tab and in `ration status`'s independent
CLI output (card_7816bf..., ACTIVE, $2,500.00, daily). Opened
adjust-limit on that card, backspaced/retyped the limit to 500000,
submitted — `ration status` confirmed $5,000.00 and the correct new
Held balance. Checked both error paths live: empty cardholder is
rejected client-side before any RPC; a limit request exceeding the
tenant's real reserve-floor headroom returns the actual
card-issuance rejection into the still-open form. `go build`/`go vet`/
`go test ./...` clean for services/tui; full `make build` (Rust+Go+SDK)
clean.
Also added scripts/start-individually.sh, a reusable version of the
manual individual-service-startup workaround rounds 6/7's build logs
already documented ad hoc for this box's contention — same PID-file
format as run-demo.sh, so scripts/stop-demo.sh tears it down too
(verified: it did).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gateway.IssueCard/AdjustCardLimit/GetCard/ListCards have existed since the
original build and `ration status` already read ListCards, but no CLI
command ever called IssueCard or AdjustCardLimit — a judge or operator
running the documented demo flow had no way to actually issue a card or
change its limit from the terminal, only observe cards created as a side
effect of the agent's own RationBridgeFunding decisions. Adds `ration card
issue/get/list/adjust-limit`, following the same dialClients/tabwriter
conventions apikey.go and status.go already use.
Verified live against the real bare-metal 7-service stack (seed -> card
issue -> card get -> card list -> card adjust-limit raise -> status shows
the raised hold -> card adjust-limit lower -> status shows the released
hold), not just a build check.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
card-issuance's own RecordAgentDecision snapshot (recordDecision in
internal/bridge/service.go) never carried deposit_id, unlike the
agent's separate tools.go-recorded decision for the same event. The
agent's hasOpenGrantWithinWindow (services/agent/internal/loop/runner.go)
resolves every recent RATION_BRIDGE decision's deposit_id to check
whether its backing deposit has since been repaid (rule 3's "unless
fully repaid" exception) — when it can't resolve a deposit_id it fails
closed and blocks. Since card-issuance's own decision row could never
be resolved, ONE successful bridge grant, ever, permanently
velocity-capped the tenant for the rest of the decision-history
lookback window, even after that exact deposit was received and its
hold released.
Fixed by setting policySnapshot.DepositID from the matched late
deposit, same as the agent's own snapshot already does.
Verified live against the real bare-metal stack (ledger-core, auth,
tenant-registry, card-issuance, webhook-ingestion, gateway, agent):
granted and repaid (via a real signed webhook) two separate bridge
grants for one tenant, then confirmed a third late deposit still got a
real grant on its first tick. Reproduced the bug for real by rebuilding
card-issuance from the pre-fix commit: with that binary, deposit B was
declined with "velocity cap ... has not yet been repaid" on every
single tick after deposit A was repaid, and never granted at all.
go vet/go test clean for services/card-issuance and services/agent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`apikey issue`/`revoke` had no read side: an operator could mint or
revoke a key by key_id, but had no way to discover what key_ids exist
in the first place — only a key_id personally kept from its own
one-time `issue` output could ever be acted on. Adds auth.proto's
ListAPIKeys RPC, apikey.Store.List (sorted, tenant-filterable), the
server implementation, a CLI authclient wrapper, and `ration apikey
list [--tenant <id>]` — never returning the plaintext secret, which is
never stored anywhere after issuance.
Verified live against the real bare-metal 8-service stack
(scripts/run-demo.sh): issue -> list -> simulate-deposit --late ->
agent poll -> status (real recorded bridge-funding decision) ->
revoke -> list again (revoked: true) -> verify-chain (intact).
docker-compose rebuild was blocked by genuine host-disk exhaustion
from concurrent sibling workflows on this shared machine, not by
anything in this diff (pure proto/Go, zero compose/env changes).
Also corrects docs/THREAT-MODEL.md's "Known gaps" items 1, 6, and 7,
which still described tenant-auth, encryption-at-rest, and terminal-
injection as open/off-by-default despite rounds 2-4 actually closing
all three (RATION_GATEWAY_REQUIRE_TENANT_AUTH and
RATION_LEDGER_REQUIRE_TENANT_AUTH both default true, AES-256-GCM is
live on ledger-core's WAL/SQLite, pkg/textsanitize is wired into
cli/tui) — a judge reading the old text would see a security posture
strictly worse than what the code ships. Appends a short, honest
"Updates since initial submission" section to docs/SUBMISSION.md for
the same reason, without touching its original round-1 content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
services/auth already implemented the RevokeAPIKey RPC (proto/auth.proto,
backed by internal/apikey.Store.Revoke) since the section-1 split, but
nothing in this repo ever called it: `ration apikey issue` (round 3) could
mint a platform customer's sdk/go/ration credential, but there was no
operator-facing way to invalidate a leaked or rotated one. Added
RevokeAPIKey to the CLI's authclient wrapper and a `ration apikey revoke
--key-id <id>` command.
Verified live against the real bare-metal stack (scripts/run-demo.sh):
issued a real key via `ration apikey issue`, used it successfully through
the actual Go SDK (sdk/go/ration) to IssueCard against the containerless
gateway on localhost:7002, ran `ration apikey revoke --key-id <id>`, then
re-ran the identical SDK call — it now fails with
`Unauthenticated: tenantauth: invalid credential: revoked`. Also checked
idempotent re-revoke, an unknown key_id (NotFound), and the missing-flag
error path. `go build ./...` and `go vet ./...` clean across services/cli;
existing `go test ./...` for services/cli unaffected (no new test
infrastructure added, per standing rule).
sdk/go/ration had no way to attach a credential, so its own README/doc.go
"Typical use" example was rejected with codes.Unauthenticated against any
default-config gateway (RATION_GATEWAY_REQUIRE_TENANT_AUTH defaults to
true) — nothing in this repo even called auth.IssueAPIKey before this.
Adds ration.WithAPIKey/WithBearerToken dial options (mirroring every
internal services/*/internal/authclient package's own interceptor
pattern) and a new `ration apikey issue` CLI command that actually mints
one via auth.IssueAPIKey. Verified live against a real running
auth+gateway+ledger-core+tenant-registry+card-issuance+webhook-ingestion
stack: no credential -> Unauthenticated, bad credential -> Unauthenticated
"invalid credential: not found", real minted credential -> IssueCard
succeeds end to end.
Also found, live, by actually running `docker compose up --build` for
real (not by reading code): gateway, card-issuance, tenant-registry,
webhook-ingestion, and agent all crash-looped forever in a plain
`docker compose up --build` — every one of them mints an operator JWT
from auth at startup and os.Exit(1)s if that fails, and none of their
compose service blocks set the env var each one's own config actually
reads (AUTH_GRPC_ADDR for all but gateway, which uniquely reads
AUTH_ADDR). Only ledger-core had this fixed previously (round 2). This
meant the entire containerized stack was non-functional end to end, not
just ledger-core's corner of it. Fixed all five; re-verified by tearing
the whole stack down and bringing it back up from scratch, seeding
tenants via `docker compose run --rm cli seed`, minting a real API key
via `docker compose run --rm cli apikey issue`, and issuing a card via
the Go SDK against the real containerized gateway on localhost:7002.
Also fixed a separate real bug in scripts/run-demo.sh's .env loader,
found live the same way: `IFS='=' read -r key value` silently drops a
single trailing IFS character, truncating any .env value ending in
exactly one '=' by one character — which is *every* `openssl rand
-base64 32` key (the exact command this repo's own docs recommend for
RATION_LEDGER_ENCRYPTION_KEY/AUTH_JWT_SIGNING_KEY), since a 32-byte
input always base64-encodes with exactly one padding '='. Reproduced
live: a real generated key crashed ledger-core at startup with
InvalidBase64(InvalidPadding) on a plain scripts/run-demo.sh run.
Switched to parameter-expansion splitting on the first '=' only;
re-verified the full run-demo.sh -> seed -> status cycle with a real
generated key, confirmed ledger.wal/ledger.db are genuine ciphertext
under the correct, untruncated key.
cargo test/go test/go vet all still pass (pre-existing, unrelated:
sdk/rust/ration-client's own #[cfg(test)] FakeGateway fixture doesn't
implement a newer Gateway trait method and fails to compile under
`cargo test` — a test-fixture-only gap, left alone per standing no-new-
test-work policy; production cargo build is unaffected).
Finishes the one item round 1's build log explicitly scoped but didn't
land: column-level AES-256-GCM encryption for deposit.external_ref,
ledger_entries.reference/memo, and agent_decisions.trigger/rationale/
policy_snapshot_json, reusing the same RATION_LEDGER_ENCRYPTION_KEY
that already encrypts the WAL (wal_crypto.rs). tenant_id, amounts,
timestamps and enums stay plaintext, same as before — they're the
WHERE/ORDER BY columns every query filters or sorts on.
ledger_entries.reference is also the one column a real query filtered
on (bridge-hold-release matching by `bridge:<deposit_id>:*` LIKE +
exact-match prefix). Since AES-GCM ciphertext can't support that,
list_outstanding_bridge_holds_for_deposit now fetches by the
(unencrypted) tenant_id/entry_type pair and matches the decrypted
reference in Rust instead of in SQL — covered by a new regression test
run both with and without encryption enabled.
Also fixes a real bug found verifying this live against scripts/
run-demo.sh: RATION_LEDGER_ENCRYPTION_KEY set to an empty string (the
exact value .env.example documents, and the exact value run-demo.sh's
load_env_file falls back to when the var is never set by hand) made
wal_crypto::Key::from_env() attempt to parse a zero-byte key and crash
ledger-core at startup with WrongKeyLength(0) — meaning a totally clean
`scripts/run-demo.sh` run, the default/documented judge path, could
never start. from_env() now treats empty the same as unset, matching
main.rs's own auth_grpc_addr.is_empty() convention one function away.
Verified: cargo test 70/70 (51 lib incl. 7 new encryption tests + 19
integration), cargo clippy clean, cargo fmt clean. Live against the
real demo stack with RATION_LEDGER_ENCRYPTION_KEY set: ledger.wal and
ledger.db are genuine ciphertext on disk (grep/JSON-parse both fail),
`ration status`/`ration audit-check`/`verify-chain` all read back
correctly decrypted data, and a full seed -> simulate-deposit --late ->
agent poll -> partial bridge grant -> 24h velocity-cap decline cycle
ran with zero regressions. Live again with the var unset entirely
(the previously-broken path): ledger-core now starts and the same
cycle runs correctly in plaintext mode.
processLateDeposit unconditionally records a RATION_BRIDGE-tagged
"envelope pre-check" decision before ever attempting the LLM/tool call
that would actually move money — found live while verifying the
circuit breaker (internal/llm/breaker.go): when that first LLM attempt
failed outright (e.g. the provider being transiently unreachable,
before the breaker even trips), hasOpenGrantWithinWindow treated that
pre-check recording as "an open unrepaid grant" and velocity-capped
every later attempt against the same deposit for a full 24h — even
though no money had ever actually moved and the breaker's whole point
is to execute that same envelope directly once tripped.
hasOpenGrantWithinWindow now only counts RATION_BRIDGE decisions whose
snapshot actually includes granted_minor_units (new
wasActuallyGranted helper) — the key only llm/tools.go's
handleRationBridgeFunding sets, on a real executed grant, never the
pre-check recording. Updated the two existing velocity-cap test
fixtures to include it, since a real "open prior grant" always has it
too.
docs/THREAT-MODEL.md's "Known gaps" #6: the WAL and SQLite index are
fully plaintext on local disk today.
Add src/wal_crypto.rs: AES-256-GCM envelope encryption (pure Rust, no
OpenSSL linking) keyed from RATION_LEDGER_ENCRYPTION_KEY
(base64-encoded 32 bytes — a dev stand-in for a real KMS/Secrets
Manager key, per this gap's own wording). Wire it into
append_and_commit/replay_wal_into_index so every WalEvent is encrypted
before wal_ffi::Wal::append ever sees it, and decrypted in the recovery
callback before JSON deserialization. This is the correct boundary,
not a compromise: wal_ffi's own interface already treats a payload as
an opaque &[u8], so this requires zero changes to the C WAL's frame
format, CRC, or recovery/truncation logic (services/wal/wal.c,
docs/formal's TLA+ spec).
Disabled by default (same opt-in posture as every other control here
that changes an on-disk format); Ledger::open_with_encryption's doc
comment spells out that this is not a live migration path. Verified
live: the on-disk WAL file is genuine ciphertext (no plaintext tenant
names, not valid JSON), a full seed/deposit/bridge-funding demo cycle
runs correctly against an encrypted ledger, and two new integration
tests prove a real restart-and-recover round trip plus loud failure on
a wrong key.
STRATEGY-2026-10-23.md names the exact risk this closes: "Ration's
actual differentiators... are invisible in a UI-less terminal demo
unless explicitly called out on screen." A judge can now open a
shareable gateway URL instead of running the TUI locally.
Add gateway.proto's GetDashboardSnapshot RPC (aggregates ListTenants +
per-tenant GetBalance/CheckReserveFloor + ListAgentDecisions into one
round trip — the same ledger-core data services/tui already fetches
per-tenant, just server-side aggregated for a browser) and three new
httpapi routes: GET /v1/dashboard (a single self-contained HTML/CSS/JS
page, zero external dependencies so it renders on a network-isolated
judging box), GET /v1/dashboard/snapshot (polled every 3s for
per-tenant reserve-floor headroom gauges and recent agent-decision
rationale), and GET /v1/dashboard/activity/stream (a true
text/event-stream SSE framing of StreamActivity for the page's
EventSource, separate from the existing NDJSON /v1/activity).
The dashboard is deliberately credential-free (GetDashboardSnapshot
carries no tenant_id/card_id, which internal/tenantauth's own
documented behavior already passes through unscoped); its activity SSE
stream presents gateway's own operator-scoped credential
(Handler.SetDashboardOperatorToken) specifically because
internal/tenantauth's StreamServerInterceptor requires some credential
on every streaming call — scoped narrowly to this one new route so the
existing, still-credentialed /v1/activity route's security posture is
unchanged.
Verified live: snapshot JSON returns correct per-tenant
available/held/headroom; the SSE stream delivers a real simulated-
deposit event end-to-end with zero auth errors; the page itself serves
at 200 text/html.
infra/dev-ca/generate-dev-ca.sh's own header names this gap: 1-year
leaf certs with no rotation story short of regenerating (and
redistributing) the entire CA by hand.
Add scripts/rotate-dev-ca.sh plus a new client,
services/auth/cmd/issue-service-cert, for auth.proto's IssueServiceCert
RPC: point a running auth instance's AUTH_CA_CERT_FILE/AUTH_CA_KEY_FILE
at the same static dev CA every service's *_MTLS_CA_FILE already
trusts, and the rotation script generates a fresh keypair+CSR per
service and gets back a leaf signed against that identical root — no
new CA to redistribute, only the one service whose leaf just rotated
needs restarting to pick it up.
Verified live end-to-end: generated a CA, started auth against it,
rotated all 10 service identities, confirmed the new leaf's serial
changed, its key matches its cert, and `openssl verify -CAfile` passes
against the untouched original CA.
docs/AGENT-DESIGN.md states plainly: "there is no confirmed... no
circuit breaker: a slow/erroring LLM provider stalls that tick's
decision loop." Add llm.CircuitBreaker (new internal/llm/breaker.go):
trips after N consecutive failures within a rolling window
(RATION_AGENT_LLM_BREAKER_* env vars), short-circuiting every call with
ErrCircuitOpen until a cooldown elapses, then allows one half-open
trial before fully closing again.
internal/loop.Runner now treats ErrCircuitOpen as a signal to execute
the already-computed, deterministic hard-rule envelope directly (same
tool-handler code path the LLM itself would call, canned
reason/message instead of LLM-authored text) rather than skipping the
tenant's decision entirely — the breaker's only job is making sure a
stuck provider never blocks or delays the money movement the hard
rules already decided was safe, matching the existing "LLM decides
phrasing only" design.
Verified live: with no ANTHROPIC_API_KEY configured, the breaker trips
after 3 failures, then correctly executes real bridge-funding grants
(HOLD entries landing on ledger-core, held balance increasing) via the
fallback path with zero LLM calls, and the velocity cap correctly
blocks further grants afterward.
findRelevantLateDeposit filtered on a deposit's stored status field
being DEPOSIT_STATUS_LATE, but nothing on ledger-core's write path ever
writes that status to a row (deposit::insert sets Expected,
deposit::mark_received sets Received) — so against a real ledger-core,
RationBridgeFunding declined every genuinely late deposit with "no late
deposit on file for this tenant", unconditionally. Every unit test
exercising this path hand-constructed fixtures that also happened to
satisfy the real, time-based definition, masking the bug behind a green
suite (see docs/RUNBOOK.md's now-updated entry for the full trace).
Compute lateness the same way ledger-core's own list_late does
(ReceivedAtUnix == 0 && ExpectedAtUnix < now) instead of trusting
status. No proto/wire change; existing tests pass unchanged since their
fixtures already satisfied this definition too.
docs/THREAT-MODEL.md's "Known gaps" #3: no operational cross-check
existed between ledger-core's own entry/decision counts and audit-log's
best-effort mirror, even though audit_mirror.rs's mirroring can fail
silently (network blip, audit-log down) with nothing comparing the two
afterward.
`ration audit-check` dials audit-log directly (new Audit client on
clients.go) and, per tenant, fully paginates both ledger-core's
ListEntries/ListAgentDecisions and audit-log's ListRecords (filtered by
SourceKind), reporting any count divergence beyond --tolerance and
exiting nonzero. Verified live against the demo stack: 18/18 tenants
report OK with exact count matches.
Wire the diagnosis path into docs/RUNBOOK.md next to the existing
on-call-style entries.
docs/THREAT-MODEL.md states plainly: "The TUI does not currently
display rationale/agent decisions at all — only tenants, deposits,
cards, and the live activity stream." Add a 5th tab (new
view/decisions.go, wired into model.go/state.go's tab state) that
fetches every tenant's ListAgentDecisions each refresh, sorted newest
first across tenants, and renders WHEN/TENANT/TRIGGER/ACTION/
RATIONALE/POLICY SNAPSHOT.
Extract the CLI's existing sanitizeControlChars into pkg/textsanitize
and apply it (via the new format.SanitizedCell, sanitize-then-truncate)
to every LLM-authored field this new tab renders, closing the same
terminal-escape-sequence injection gap for this surface at the same
time it's built, not after.
docs/THREAT-MODEL.md names this "the biggest one" sub-gap of item 1:
cli/tui/agent (and gateway's own forwarding, card-issuance,
webhook-ingestion, tenant-registry) all dial ledger-core directly,
bypassing gateway's internal/tenantauth entirely, so any operator or
compromised caller could act as any tenant_id against the system's
actual source of truth regardless of what gateway enforces upstream.
Add services/ledger-core/src/tenantauth.rs (authorize_tenant /
authorize_operator, checking x-api-key / authorization: Bearer headers
against a new src/authclient.rs that dials Auth's VerifyAPIKey/
VerifyJWT directly) and call it from every LedgerCore RPC handler.
RATION_LEDGER_REQUIRE_TENANT_AUTH defaults to true; ledger-core refuses
to start with it on but no AUTH_GRPC_ADDR configured.
Every direct ledger-core caller now mints and presents its own
operator-scoped credential: cli/tui/agent (reusing their existing
gateway credential), and new internal/authclient packages for gateway,
card-issuance, webhook-ingestion and tenant-registry.
Verified end-to-end against the live demo stack (seed, simulate-deposit,
status, and the agent's poll/bridge-funding loop all succeed with the
new enforcement on; zero Unauthenticated/PermissionDenied anywhere in
any service's logs).
handleTransactionCreated previously matched an incoming Airwallex
transaction to a deposit purely by (tenant, amount, not-yet-received,
oldest) — its own doc comment called this "intentionally conservative"
and flagged a real misattribution risk once a tenant has two
outstanding deposits of the same amount.
Carry the depositor-facing reference through RecordDeposit's
external_ref and Airwallex's payload.Reference, and prefer an exact
reference match before falling back to the amount heuristic. Add the
matching.go helpers (matchDepositByReference,
matchDepositByAmountHeuristic) the handler already called but never
shipped.
Close THREAT-MODEL.md's top-named gap end-to-end: services/cli,
services/tui and services/agent each mint an operator-scoped
credential (new internal/authclient in all three, mirroring gateway's
existing authclient) at startup and attach it as x-api-key/
authorization metadata on every gateway call, so
RATION_GATEWAY_REQUIRE_TENANT_AUTH now defaults to true instead of
false.
Also close the two named sub-gaps this left open:
- GetCard/AdjustCardLimit carry only card_id, not tenant_id, so
tenantauth and ratelimit previously passed them through unscoped.
Add internal/gateway/cardtenant, a cached card_id -> tenant_id
resolver backed by card-issuance, and use it in both interceptors.
- StreamActivity accepted an empty tenant_id (meaning "all tenants")
from any caller, authenticated or not. Add
tenantauth.StreamServerInterceptor: a non-operator credential may
only subscribe to its own tenant_id; only an operator credential may
request the all-tenants view.
Add an is_operator flag to IssueAPIKey/MintJWT and thread it through
VerifyAPIKey/VerifyJWT's response, so a credential can authenticate as
a trusted cross-tenant caller (CLI/TUI operator dashboards, the
agent's polling loop) instead of being bound to one tenant_id. This is
the credential type gateway's tenantauth interceptor now exempts from
its per-tenant match.
Bridge-grant HOLDs were never released on repayment: every HOLD was
tagged only "bridge:<hold_id>" with no deposit linkage, and nothing
emitted a RELEASE when the backing deposit actually settled, so the
24h velocity cap could only ever be unblocked by waiting out the
window, never by real repayment (AGENT-DESIGN.md rule 3 was dead code).
Tag every bridge HOLD's reference as "bridge:<deposit_id>:<hold_id>",
and have mark_deposit_received_with_entry atomically find and RELEASE
every outstanding HOLD tied to the now-received deposit in the same
WAL frame as the credit, mirroring the releases into audit-log exactly
like the credit itself already was.
Closes docs/THREAT-MODEL.md Known-gaps #3's named TODO: ledger-core had
no Prometheus endpoint at all, so a sustained audit-log outage was
only a tracing::warn! log line, nothing alertable. Adds
ledger_core_audit_mirror_spawn_append_failures_total{kind}, served on
GET /metrics at LEDGER_CORE_METRICS_ADDR (default localhost:9099),
pre-registered at zero per kind so absence vs zero is distinguishable.
Wires the new endpoint into docker-compose.yml and the Prometheus
scrape config, which already named this exact gap as the trigger to
add the target. Fixes a render() visibility bug and a test assumption
that unobserved CounterVec label values still emit TYPE/HELP lines,
both left broken by an earlier interrupted pass on this same change.
handleTransactionCreated picked the oldest deposit matching the incoming
transaction's amount, but compared amount only — it never excluded
deposits that were already marked received. A tenant with two deposits
of the same amount (e.g. two $500 recurring deposits) where the older
one already settled would have a genuinely new webhook for the newer
deposit re-match the older, already-received one, since it was still
the "oldest" expected_at_unix among amount-matches. MarkDepositReceived
then no-ops against that already-Received deposit via ledger-core's own
idempotency check, so the newer deposit is left stuck in Expected
forever and the tenant's available balance never reflects money that
genuinely arrived — a silent, permanent undercount of real cash
position with no error surfaced anywhere.
Fix: exclude any deposit with ReceivedAtUnix != 0 from the candidate
set before comparing amounts, so only genuinely outstanding deposits
can ever be matched.
Verified locally (mock mode only, no Airwallex credentials touched):
added a throwaway test reproducing the exact scenario (older
already-received $500 deposit + newer still-expected $500 deposit, new
webhook arrives) against the unmodified code — it failed by marking the
old settled deposit received again. Applied the fix, re-ran — it now
correctly matches the newer outstanding deposit. Deleted the throwaway
test afterward; `go test ./...` for services/webhook-ingestion passes
unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MarkDepositReceived's ledger credit — the only path by which a deposit's
money becomes spendable headroom — was never mirrored into audit-log,
unlike ApplyEntry's credits and RecordAgentDecision. The claim that
"every money move is mirrored" (README, docs/SUBMISSION.md) didn't
actually hold for it.
Ledger::mark_deposit_received_with_entry now returns the credit
LedgerEntryRecord it produces (mark_deposit_received keeps its original
signature, delegating to the new method and dropping the entry, so
existing callers/tests are unchanged); grpc.rs's MarkDepositReceived
handler mirrors it into audit-log exactly like ApplyEntry's handler
already does.
Also wires audit-log into scripts/run-demo.sh (the bare-metal demo path
every judge following README's Quickstart/docs/DEMO-SCRIPT.md actually
runs) — it was only ever started via docker-compose, so the headline
"audit trail is the point" claim and the standalone verify-chain tool
were never exercised end to end by the documented demo flow. Verified
live: after a seed + simulate-deposit + agent-decision run, verify-chain
now reports real records (6, matching the 6 live Append RPCs audit-log's
own metrics counted) and correctly flags a single flipped byte as a
broken chain at the right sequence.
wal_recover() treated every CRC-mismatched frame as a crash-consistent
torn tail: it truncated the file from that frame to EOF unconditionally.
A real crash can only ever tear the very last frame (nothing runs after
a crash to write more good frames past a torn one), but non-crash
corruption (disk bit rot, a stray write) can hit a frame in the middle
of the log while fully valid, already-durable frames still follow it.
In that case the old code silently and permanently destroyed every
committed ledger entry after the corruption point.
wal_recover() now looks ahead past a CRC-mismatched frame: if the rest
of the file parses as a complete, valid frame chain, this cannot be a
crash-torn tail, so it returns WAL_ERR_CORRUPT_MID_LOG with the corrupt
frame's offset (wal_recovery_stats_t.corrupt_frame_offset) and leaves
the file completely untouched instead of truncating. Genuine
crash-torn tails (corruption with nothing valid after it) still
truncate exactly as before.
Threaded through services/ledger-core/src/wal_ffi.rs as
WalError::CorruptMidLog{offset}, which Ledger::open already propagates
as a hard boot failure via the existing WalError->LedgerError mapping
— ledger-core now refuses to boot on this condition rather than
silently losing data.
Tested at both layers: services/wal/tests/test_wal.c (C, via
wal_recover() directly) and services/ledger-core/src/wal_ffi.rs +
services/ledger-core/tests/recovery.rs (Rust, through the safe wrapper
and through Ledger::open end-to-end), each with a sibling test proving
the existing single-frame-at-the-true-tail case still truncates as
before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes docs/THREAT-MODEL.md's "A compromised audit-log" Repudiation
gap (item 9's precise TODO): VerifyChain was only exposed as
audit-log's own RPC, so a compromised audit-log process could simply
lie about its own integrity. internal/store.VerifyFile re-derives the
hash chain directly from the on-disk audit.log file, opened strictly
read-only (O_RDONLY, never O_RDWR), with no Store/gRPC involvement —
the same per-record checks replay()/VerifyChain() already perform, but
reporting every failure (corrupt record, broken chain, non-contiguous
sequence) as a result rather than requiring a live, trusted process to
ask.
services/audit-log/cmd/verify-chain wraps it as an operator-run CLI
(`verify-chain [path]`, defaulting to
$RATION_AUDIT_LOG_DATA_DIR/audit.log) that dials nothing and exits
nonzero on any break or unreadable file — safe to wire into a health
check or an out-of-band cron, independent of whether the audit-log
binary itself would lie about VerifyChain's RPC result.
Manually verified end-to-end against a real generated audit.log:
intact chain exits 0, a payload_json edit without hash recomputation
is detected and exits 1 with the exact broken sequence. RED->GREEN
with new store/cmd test suites; added to `make build-go`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>