Closes docs/THREAT-MODEL.md Known-gaps #3's named TODO: ledger-core had
no Prometheus endpoint at all, so a sustained audit-log outage was
only a tracing::warn! log line, nothing alertable. Adds
ledger_core_audit_mirror_spawn_append_failures_total{kind}, served on
GET /metrics at LEDGER_CORE_METRICS_ADDR (default localhost:9099),
pre-registered at zero per kind so absence vs zero is distinguishable.
Wires the new endpoint into docker-compose.yml and the Prometheus
scrape config, which already named this exact gap as the trigger to
add the target. Fixes a render() visibility bug and a test assumption
that unobserved CounterVec label values still emit TYPE/HELP lines,
both left broken by an earlier interrupted pass on this same change.
Phase 2 sections 2-6 and 8 of docs/SCALE-EXPANSION.md:
- pkg/telemetry + pkg/metrics: OpenTelemetry trace propagation and
Prometheus metrics across every service hop (CLI -> agent -> gateway ->
card-issuance/tenant-registry/auth -> ledger-core), tracing added to
ledger-core itself via services/ledger-core/src/telemetry.rs.
- infra/: otel-collector + Prometheus + Grafana docker-compose stack, a
Terraform ECS Fargate module + dev/staging/prod envs (plan-only, no
apply — real AWS spend is an owner action), and a dev-only mTLS CA
(infra/dev-ca) wired through pkg/mtls + services/ledger-core/src/mtls.rs.
- services/audit-log (from the prior commit) gets its async mirror writer
in services/ledger-core/src/audit_mirror.rs, and services/gateway gets
a per-tenant token-bucket rate limiter (internal/ratelimit).
- Test depth: a Rust proptest suite proving concurrent ApplyEntry never
breaches the reserve floor (tests/apply_entry_proptest.rs, passing), a Go
fuzz seed corpus for the policy envelope (rules_fuzz_test.go), a libFuzzer
harness for the C WAL frame parser (services/wal/fuzz, deliberately not
wired into `make test` per its own Makefile comment - needs a
libFuzzer-bundled clang), and a real chaos test
(scripts/chaos/kill-ledger-mid-write.sh) that SIGKILLs ledger-core
mid-write under concurrent load and verifies via ListEntries that
nothing was lost or double-applied.
- docs/formal/ledger.tla: a TLA+ spec of the ledger state machine (balance
>= reserve_floor safety, HOLD-eventually-resolves liveness) plus a TLC
model config - not run through TLC in this pass, that's a precise
documented TODO, not faked.
- docs/adr/0001-0004, docs/RUNBOOK.md, docs/SLO.md, docs/THREAT-MODEL.md.
Verified: `make build && make test` clean (C WAL, Rust incl. the new
proptest suite, all 10 Go modules) from a cold target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>