Same missing-lock/TOCTOU class round 25 fixed for card-issuance's
AdjustCardLimit/CloseCard (per-card lock) and RationBridgeFunding
(per-tenant lock), found on a third, previously-unaudited money-moving
path: handleTransactionCreated's ListDeposits -> match ->
MarkDepositReceived sequence had nothing serializing it per tenant.
Two genuinely concurrent webhook deliveries for the same tenant (e.g.
two outstanding deposits of the same amount settling close together —
Go's net/http serves each request on its own goroutine) could both
read the deposit list before either call landed, so both independently
picked the same "oldest not-yet-received" deposit. ledger-core's own
MarkDepositReceived is idempotent per deposit_id, so this never
double-credited — but the second transaction's real, already-arrived
money was then never matched to anything: its deposit stayed Expected
forever, permanently understating the tenant's available balance.
Reproduced live against the real bare-metal stack before fixing: two
concurrent signed webhook deliveries for two $750.00 deposits on the
same tenant both matched deposit bcdb1859 via amount_heuristic; the
sibling deposit d234a440 was never credited (Available stayed flat at
one deposit's worth instead of rising by both). Fixed with a per-tenant
sync.Mutex (webhookhandler.Handler.tenantLocks/lockTenant), the
identical lazily-created-never-removed pattern cardsvc.Service.lockCard
and bridge.Service.lockTenant already established, serializing the
entire check-then-act critical section per tenant_id. Re-ran the
identical live repro post-fix: the two transactions correctly matched
the two distinct deposits, and Available rose by the full $1,500.00
(both $750.00 deposits settled).
New regression test (TestHandleAirwallexWebhook_ConcurrentSameAmount-
DepositsBothSettle) uses a concurrency-aware fake ledger whose first
ListDeposits call sleeps after taking its snapshot, deterministically
widening the TOCTOU window instead of relying on real scheduling.
Confirmed it fails 20/20 on the pre-fix code and passes 20/20 with the
fix, both under `go test -race`. `go build`/`go vet`/`go test ./...`
clean across all 11 go.work members; `make build-go` clean.
handleTransactionCreated's tier-1 match (payload.Reference ==
deposit.ExternalRef) never checked the transaction's own amount.
MarkDepositReceived has no "amount actually transferred" field at all
(proto/ledger.proto) — it always credits the matched deposit's full
pre-recorded expected amount. Reproduced live against the real running
stack: a genuine signed webhook carrying a $5,000.00 deposit's
reference but only $1.00 in amount_minor_units would have marked that
deposit fully received and credited the tenant $5,000.00 for a $1.00
transfer.
matchDepositByReference now also requires the transaction's amount to
equal the deposit's expected amount before trusting the reference
match; a reference match that fails the amount check falls through to
the existing amount-heuristic tier instead of being treated as exact.
Verified live: rebuilt only webhook-ingestion, sent a real HMAC-signed
webhook with the correct reference but wrong amount — rejected ("no
matching expected deposit found"), deposit stayed unreceived, balance
untouched. Then sent the same reference with the correct amount —
matched_by=reference, deposit credited normally. go build/vet/test
clean across all 11 go.work members.
handleTransactionCreated previously matched an incoming Airwallex
transaction to a deposit purely by (tenant, amount, not-yet-received,
oldest) — its own doc comment called this "intentionally conservative"
and flagged a real misattribution risk once a tenant has two
outstanding deposits of the same amount.
Carry the depositor-facing reference through RecordDeposit's
external_ref and Airwallex's payload.Reference, and prefer an exact
reference match before falling back to the amount heuristic. Add the
matching.go helpers (matchDepositByReference,
matchDepositByAmountHeuristic) the handler already called but never
shipped.
handleTransactionCreated picked the oldest deposit matching the incoming
transaction's amount, but compared amount only — it never excluded
deposits that were already marked received. A tenant with two deposits
of the same amount (e.g. two $500 recurring deposits) where the older
one already settled would have a genuinely new webhook for the newer
deposit re-match the older, already-received one, since it was still
the "oldest" expected_at_unix among amount-matches. MarkDepositReceived
then no-ops against that already-Received deposit via ledger-core's own
idempotency check, so the newer deposit is left stuck in Expected
forever and the tenant's available balance never reflects money that
genuinely arrived — a silent, permanent undercount of real cash
position with no error surfaced anywhere.
Fix: exclude any deposit with ReceivedAtUnix != 0 from the candidate
set before comparing amounts, so only genuinely outstanding deposits
can ever be matched.
Verified locally (mock mode only, no Airwallex credentials touched):
added a throwaway test reproducing the exact scenario (older
already-received $500 deposit + newer still-expected $500 deposit, new
webhook arrives) against the unmodified code — it failed by marking the
old settled deposit received again. Applied the fix, re-ran — it now
correctly matches the newer outstanding deposit. Deleted the throwaway
test afterward; `go test ./...` for services/webhook-ingestion passes
unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both were precise TODOs in docs/THREAT-MODEL.md, not speculative hardening:
- Elevation of privilege: VerifySignature now rejects any Airwallex
webhook delivery whose x-timestamp is more than 5 minutes from the
verifier's clock (ErrStaleTimestamp), closing an unbounded replay
window down to Airwallex's own documented delivery tolerance.
- Denial of service: webhook-ingestion gets its own internal/ratelimit
(token-bucket HTTP middleware, source-IP keyed, mirroring gateway's
gRPC-side design per docs/adr/0004) in front of the plain HTTP
webhook listener, tunable via WEBHOOK_INGESTION_RATELIMIT_* env vars.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 2 service decomposition per docs/SCALE-EXPANSION.md section 1: extracts
internal/{airwallex,bridge,card,tenant,httpserver} out of the monolithic
gateway into four independently deployable gRPC services (each own go.mod,
Dockerfile, and tests), with gateway now a thin BFF that fans out through new
cardissuanceclient/webhookeventsclient gRPC clients. gateway.proto's external
surface is unchanged — agent/tui/cli still talk to gateway exactly as before.
Verified: all 10 go.work modules build and `go test ./...` passes across
every one, including the new gateway client packages against the live split
services (via `make build && make test`).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>