wal_recover() treated every CRC-mismatched frame as a crash-consistent
torn tail: it truncated the file from that frame to EOF unconditionally.
A real crash can only ever tear the very last frame (nothing runs after
a crash to write more good frames past a torn one), but non-crash
corruption (disk bit rot, a stray write) can hit a frame in the middle
of the log while fully valid, already-durable frames still follow it.
In that case the old code silently and permanently destroyed every
committed ledger entry after the corruption point.
wal_recover() now looks ahead past a CRC-mismatched frame: if the rest
of the file parses as a complete, valid frame chain, this cannot be a
crash-torn tail, so it returns WAL_ERR_CORRUPT_MID_LOG with the corrupt
frame's offset (wal_recovery_stats_t.corrupt_frame_offset) and leaves
the file completely untouched instead of truncating. Genuine
crash-torn tails (corruption with nothing valid after it) still
truncate exactly as before.
Threaded through services/ledger-core/src/wal_ffi.rs as
WalError::CorruptMidLog{offset}, which Ledger::open already propagates
as a hard boot failure via the existing WalError->LedgerError mapping
— ledger-core now refuses to boot on this condition rather than
silently losing data.
Tested at both layers: services/wal/tests/test_wal.c (C, via
wal_recover() directly) and services/ledger-core/src/wal_ffi.rs +
services/ledger-core/tests/recovery.rs (Rust, through the safe wrapper
and through Ledger::open end-to-end), each with a sibling test proving
the existing single-frame-at-the-true-tail case still truncates as
before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 2 sections 2-6 and 8 of docs/SCALE-EXPANSION.md:
- pkg/telemetry + pkg/metrics: OpenTelemetry trace propagation and
Prometheus metrics across every service hop (CLI -> agent -> gateway ->
card-issuance/tenant-registry/auth -> ledger-core), tracing added to
ledger-core itself via services/ledger-core/src/telemetry.rs.
- infra/: otel-collector + Prometheus + Grafana docker-compose stack, a
Terraform ECS Fargate module + dev/staging/prod envs (plan-only, no
apply — real AWS spend is an owner action), and a dev-only mTLS CA
(infra/dev-ca) wired through pkg/mtls + services/ledger-core/src/mtls.rs.
- services/audit-log (from the prior commit) gets its async mirror writer
in services/ledger-core/src/audit_mirror.rs, and services/gateway gets
a per-tenant token-bucket rate limiter (internal/ratelimit).
- Test depth: a Rust proptest suite proving concurrent ApplyEntry never
breaches the reserve floor (tests/apply_entry_proptest.rs, passing), a Go
fuzz seed corpus for the policy envelope (rules_fuzz_test.go), a libFuzzer
harness for the C WAL frame parser (services/wal/fuzz, deliberately not
wired into `make test` per its own Makefile comment - needs a
libFuzzer-bundled clang), and a real chaos test
(scripts/chaos/kill-ledger-mid-write.sh) that SIGKILLs ledger-core
mid-write under concurrent load and verifies via ListEntries that
nothing was lost or double-applied.
- docs/formal/ledger.tla: a TLA+ spec of the ledger state machine (balance
>= reserve_floor safety, HOLD-eventually-resolves liveness) plus a TLC
model config - not run through TLC in this pass, that's a precise
documented TODO, not faked.
- docs/adr/0001-0004, docs/RUNBOOK.md, docs/SLO.md, docs/THREAT-MODEL.md.
Verified: `make build && make test` clean (C WAL, Rust incl. the new
proptest suite, all 10 Go modules) from a cold target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Six services, independently built then integrated:
- services/wal (C): crash-safe write-ahead log, CRC32-checksummed
frames, torn-write recovery via truncate-to-last-valid-frame.
- services/ledger-core (Rust): double-entry ledger over the C WAL via
FFI, reserve-floor enforcement BEFORE any WAL write, idempotent
ApplyEntry, full LedgerCore gRPC service.
- services/gateway (Go): Airwallex Issuing/Global-Accounts client
(mock mode default, live mode ready), the actual rationing math
(reserve floor, 60% bridge cap, 24h velocity cap, partial-over-decline).
- services/agent (Go): deterministic hard-rule policy envelope computed
BEFORE any LLM call; Claude only decides within that pre-approved
envelope; every decision audit-logged even when the LLM call fails.
- services/tui (Go, Bubble Tea) + services/cli (Go, cobra): the only
UI — no web frontend, per explicit direction.
Integration found and fixed three real bugs, not just compile errors:
- proto regen collision shadowing checked-in pkg/pb packages
- ledger-core/gateway address-variable mismatch (stack would never
actually connect on default config)
- the big one: MarkDepositReceived never credited the ledger, so every
tenant's balance was permanently stuck at 0 and bridge-funding could
only ever decline — the core agentic-rationing feature could never
produce a real grant. Fixed at the ledger level with idempotent
mark-and-credit in one WAL frame, covered by 3 new tests.
End-to-end verified live, reproducibly: seed -> simulate a late deposit
-> agent grants a partial bridge with a computed rationale -> second
grant correctly declines once the 24h velocity cap engages. Verified
independently by the orchestrator too: 'make build' and 'make test'
both clean (Rust: 29 tests; Go: all packages; C: WAL test suite).
Airwallex sandbox credentials still pending organizer approval —
tracked as an owner blocker in docs/AIRWALLEX-SETUP.md, not a code gap.