infra/terraform/modules/service hardcoded deployment_minimum_healthy_percent=100
and deployment_maximum_percent=200 for every service, including ledger-core and
audit-log — both explicitly documented in their own env-file comments as
"single-writer, not horizontally scaled" and backed by a local WAL/hash-chain on
Fargate ephemeral storage with no replication between tasks. That rolling config
let ECS run the old task and a brand-new, empty-disk task concurrently for the
length of every deploy's health-check window, both reachable under the same
Cloud Map service-discovery name.
Reproduced live: started two real ledger-core processes (one seeded with a real
tenant+balance simulating the pre-deploy task, one against a fresh empty data
dir simulating the new Fargate task on fresh ephemeral storage) and called
GetBalance for the identical tenant_id against both — the "old" task returned
the real balance, the "new" task returned NotFound, with no way for a caller to
tell "tenant genuinely doesn't exist" apart from "this task's disk never saw it
because it's a brand new task from this exact deploy." Any write ECS routes to
the old task in that overlap window is lost the moment the old task is stopped.
Fix: added deployment_minimum_healthy_percent/deployment_maximum_percent as
module variables (default 100/200, preserving today's rolling behavior for
stateless services), and overrode them to 0/100 ("recreate" semantics — old task
fully stopped before the new one starts) for module.ledger_core and
module.audit_log in all three envs (dev/staging/prod).
Verified: terraform validate clean for the module and all three envs;
terraform plan reaches the expected provider-auth wall (this repo's own
documented verification ceiling — no AWS credentials available) with no
configuration errors before that point; terraform fmt -check -recursive clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes docs/THREAT-MODEL.md Known-gaps #3's named TODO: ledger-core had
no Prometheus endpoint at all, so a sustained audit-log outage was
only a tracing::warn! log line, nothing alertable. Adds
ledger_core_audit_mirror_spawn_append_failures_total{kind}, served on
GET /metrics at LEDGER_CORE_METRICS_ADDR (default localhost:9099),
pre-registered at zero per kind so absence vs zero is distinguishable.
Wires the new endpoint into docker-compose.yml and the Prometheus
scrape config, which already named this exact gap as the trigger to
add the target. Fixes a render() visibility bug and a test assumption
that unobserved CounterVec label values still emit TYPE/HELP lines,
both left broken by an earlier interrupted pass on this same change.
Phase 2 section 7 (the other half, alongside sdk/go/ration and
sdk/rust/ration-client already pushed in c5297e4/8215fb8): a hand-rolled
internal/httpapi on services/gateway translating JSON <-> the gRPC source
of truth (cards, deposits, bridge funding, activity), documented as a
convenience facade over gRPC, not a parallel implementation. Also folds in
the remaining phase-2 doc updates (ARCHITECTURE/RUNBOOK/THREAT-MODEL/
AIRWALLEX-SETUP/SUBMISSION, ADR 0003/0004) and docker-compose/Makefile/
.env.example wiring from the resumed phase-2 workflow's final agents.
Verified: go build/vet/test clean across all 10 services + both new SDKs,
`make build && make test` clean end to end (C WAL, Rust ledger-core + the
new Rust SDK, every Go module including sdk/go/ration).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 2 sections 2-6 and 8 of docs/SCALE-EXPANSION.md:
- pkg/telemetry + pkg/metrics: OpenTelemetry trace propagation and
Prometheus metrics across every service hop (CLI -> agent -> gateway ->
card-issuance/tenant-registry/auth -> ledger-core), tracing added to
ledger-core itself via services/ledger-core/src/telemetry.rs.
- infra/: otel-collector + Prometheus + Grafana docker-compose stack, a
Terraform ECS Fargate module + dev/staging/prod envs (plan-only, no
apply — real AWS spend is an owner action), and a dev-only mTLS CA
(infra/dev-ca) wired through pkg/mtls + services/ledger-core/src/mtls.rs.
- services/audit-log (from the prior commit) gets its async mirror writer
in services/ledger-core/src/audit_mirror.rs, and services/gateway gets
a per-tenant token-bucket rate limiter (internal/ratelimit).
- Test depth: a Rust proptest suite proving concurrent ApplyEntry never
breaches the reserve floor (tests/apply_entry_proptest.rs, passing), a Go
fuzz seed corpus for the policy envelope (rules_fuzz_test.go), a libFuzzer
harness for the C WAL frame parser (services/wal/fuzz, deliberately not
wired into `make test` per its own Makefile comment - needs a
libFuzzer-bundled clang), and a real chaos test
(scripts/chaos/kill-ledger-mid-write.sh) that SIGKILLs ledger-core
mid-write under concurrent load and verifies via ListEntries that
nothing was lost or double-applied.
- docs/formal/ledger.tla: a TLA+ spec of the ledger state machine (balance
>= reserve_floor safety, HOLD-eventually-resolves liveness) plus a TLC
model config - not run through TLC in this pass, that's a precise
documented TODO, not faked.
- docs/adr/0001-0004, docs/RUNBOOK.md, docs/SLO.md, docs/THREAT-MODEL.md.
Verified: `make build && make test` clean (C WAL, Rust incl. the new
proptest suite, all 10 Go modules) from a cold target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>