3 Commits

Author SHA1 Message Date
d8c8e4cc97 fix: WAL recovery no longer truncates valid frames after mid-log corruption
wal_recover() treated every CRC-mismatched frame as a crash-consistent
torn tail: it truncated the file from that frame to EOF unconditionally.
A real crash can only ever tear the very last frame (nothing runs after
a crash to write more good frames past a torn one), but non-crash
corruption (disk bit rot, a stray write) can hit a frame in the middle
of the log while fully valid, already-durable frames still follow it.
In that case the old code silently and permanently destroyed every
committed ledger entry after the corruption point.

wal_recover() now looks ahead past a CRC-mismatched frame: if the rest
of the file parses as a complete, valid frame chain, this cannot be a
crash-torn tail, so it returns WAL_ERR_CORRUPT_MID_LOG with the corrupt
frame's offset (wal_recovery_stats_t.corrupt_frame_offset) and leaves
the file completely untouched instead of truncating. Genuine
crash-torn tails (corruption with nothing valid after it) still
truncate exactly as before.

Threaded through services/ledger-core/src/wal_ffi.rs as
WalError::CorruptMidLog{offset}, which Ledger::open already propagates
as a hard boot failure via the existing WalError->LedgerError mapping
— ledger-core now refuses to boot on this condition rather than
silently losing data.

Tested at both layers: services/wal/tests/test_wal.c (C, via
wal_recover() directly) and services/ledger-core/src/wal_ffi.rs +
services/ledger-core/tests/recovery.rs (Rust, through the safe wrapper
and through Ledger::open end-to-end), each with a sibling test proving
the existing single-frame-at-the-true-tail case still truncates as
before.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-22 02:00:50 +03:00
13abc31645 feat(ration): observability, infra-as-code, security hardening, and test depth
Phase 2 sections 2-6 and 8 of docs/SCALE-EXPANSION.md:

- pkg/telemetry + pkg/metrics: OpenTelemetry trace propagation and
  Prometheus metrics across every service hop (CLI -> agent -> gateway ->
  card-issuance/tenant-registry/auth -> ledger-core), tracing added to
  ledger-core itself via services/ledger-core/src/telemetry.rs.
- infra/: otel-collector + Prometheus + Grafana docker-compose stack, a
  Terraform ECS Fargate module + dev/staging/prod envs (plan-only, no
  apply — real AWS spend is an owner action), and a dev-only mTLS CA
  (infra/dev-ca) wired through pkg/mtls + services/ledger-core/src/mtls.rs.
- services/audit-log (from the prior commit) gets its async mirror writer
  in services/ledger-core/src/audit_mirror.rs, and services/gateway gets
  a per-tenant token-bucket rate limiter (internal/ratelimit).
- Test depth: a Rust proptest suite proving concurrent ApplyEntry never
  breaches the reserve floor (tests/apply_entry_proptest.rs, passing), a Go
  fuzz seed corpus for the policy envelope (rules_fuzz_test.go), a libFuzzer
  harness for the C WAL frame parser (services/wal/fuzz, deliberately not
  wired into `make test` per its own Makefile comment - needs a
  libFuzzer-bundled clang), and a real chaos test
  (scripts/chaos/kill-ledger-mid-write.sh) that SIGKILLs ledger-core
  mid-write under concurrent load and verifies via ListEntries that
  nothing was lost or double-applied.
- docs/formal/ledger.tla: a TLA+ spec of the ledger state machine (balance
  >= reserve_floor safety, HOLD-eventually-resolves liveness) plus a TLC
  model config - not run through TLC in this pass, that's a precise
  documented TODO, not faked.
- docs/adr/0001-0004, docs/RUNBOOK.md, docs/SLO.md, docs/THREAT-MODEL.md.

Verified: `make build && make test` clean (C WAL, Rust incl. the new
proptest suite, all 10 Go modules) from a cold target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-19 12:25:58 +03:00
2b686594a3 feat: Ration phase 1 — full polyglot build for Airwallex Starter Kit 5
Six services, independently built then integrated:
- services/wal (C): crash-safe write-ahead log, CRC32-checksummed
  frames, torn-write recovery via truncate-to-last-valid-frame.
- services/ledger-core (Rust): double-entry ledger over the C WAL via
  FFI, reserve-floor enforcement BEFORE any WAL write, idempotent
  ApplyEntry, full LedgerCore gRPC service.
- services/gateway (Go): Airwallex Issuing/Global-Accounts client
  (mock mode default, live mode ready), the actual rationing math
  (reserve floor, 60% bridge cap, 24h velocity cap, partial-over-decline).
- services/agent (Go): deterministic hard-rule policy envelope computed
  BEFORE any LLM call; Claude only decides within that pre-approved
  envelope; every decision audit-logged even when the LLM call fails.
- services/tui (Go, Bubble Tea) + services/cli (Go, cobra): the only
  UI — no web frontend, per explicit direction.

Integration found and fixed three real bugs, not just compile errors:
- proto regen collision shadowing checked-in pkg/pb packages
- ledger-core/gateway address-variable mismatch (stack would never
  actually connect on default config)
- the big one: MarkDepositReceived never credited the ledger, so every
  tenant's balance was permanently stuck at 0 and bridge-funding could
  only ever decline — the core agentic-rationing feature could never
  produce a real grant. Fixed at the ledger level with idempotent
  mark-and-credit in one WAL frame, covered by 3 new tests.

End-to-end verified live, reproducibly: seed -> simulate a late deposit
-> agent grants a partial bridge with a computed rationale -> second
grant correctly declines once the 24h velocity cap engages. Verified
independently by the orchestrator too: 'make build' and 'make test'
both clean (Rust: 29 tests; Go: all packages; C: WAL test suite).

Airwallex sandbox credentials still pending organizer approval —
tracked as an owner blocker in docs/AIRWALLEX-SETUP.md, not a code gap.
2026-10-19 03:04:13 +03:00