sdk/go/ration had no way to attach a credential, so its own README/doc.go
"Typical use" example was rejected with codes.Unauthenticated against any
default-config gateway (RATION_GATEWAY_REQUIRE_TENANT_AUTH defaults to
true) — nothing in this repo even called auth.IssueAPIKey before this.
Adds ration.WithAPIKey/WithBearerToken dial options (mirroring every
internal services/*/internal/authclient package's own interceptor
pattern) and a new `ration apikey issue` CLI command that actually mints
one via auth.IssueAPIKey. Verified live against a real running
auth+gateway+ledger-core+tenant-registry+card-issuance+webhook-ingestion
stack: no credential -> Unauthenticated, bad credential -> Unauthenticated
"invalid credential: not found", real minted credential -> IssueCard
succeeds end to end.
Also found, live, by actually running `docker compose up --build` for
real (not by reading code): gateway, card-issuance, tenant-registry,
webhook-ingestion, and agent all crash-looped forever in a plain
`docker compose up --build` — every one of them mints an operator JWT
from auth at startup and os.Exit(1)s if that fails, and none of their
compose service blocks set the env var each one's own config actually
reads (AUTH_GRPC_ADDR for all but gateway, which uniquely reads
AUTH_ADDR). Only ledger-core had this fixed previously (round 2). This
meant the entire containerized stack was non-functional end to end, not
just ledger-core's corner of it. Fixed all five; re-verified by tearing
the whole stack down and bringing it back up from scratch, seeding
tenants via `docker compose run --rm cli seed`, minting a real API key
via `docker compose run --rm cli apikey issue`, and issuing a card via
the Go SDK against the real containerized gateway on localhost:7002.
Also fixed a separate real bug in scripts/run-demo.sh's .env loader,
found live the same way: `IFS='=' read -r key value` silently drops a
single trailing IFS character, truncating any .env value ending in
exactly one '=' by one character — which is *every* `openssl rand
-base64 32` key (the exact command this repo's own docs recommend for
RATION_LEDGER_ENCRYPTION_KEY/AUTH_JWT_SIGNING_KEY), since a 32-byte
input always base64-encodes with exactly one padding '='. Reproduced
live: a real generated key crashed ledger-core at startup with
InvalidBase64(InvalidPadding) on a plain scripts/run-demo.sh run.
Switched to parameter-expansion splitting on the first '=' only;
re-verified the full run-demo.sh -> seed -> status cycle with a real
generated key, confirmed ledger.wal/ledger.db are genuine ciphertext
under the correct, untruncated key.
cargo test/go test/go vet all still pass (pre-existing, unrelated:
sdk/rust/ration-client's own #[cfg(test)] FakeGateway fixture doesn't
implement a newer Gateway trait method and fails to compile under
`cargo test` — a test-fixture-only gap, left alone per standing no-new-
test-work policy; production cargo build is unaffected).
ledger-core's compose entry never set AUTH_GRPC_ADDR, so
src/main.rs's env_or("AUTH_GRPC_ADDR", "localhost:7010") default
resolved to the ledger-core container's own loopback, not the `auth`
service's. RATION_LEDGER_REQUIRE_TENANT_AUTH defaults to true and the
authclient channel is lazy (src/authclient.rs), so the container would
start fine but every real tenant-scoped RPC would then fail its
credential check against an address nothing listens on there — a
docker-compose-up--build deployment of this repo was silently broken
for anything beyond a health check.
scripts/run-demo.sh never hit this because every service there runs on
the same host, where localhost:7010 really is auth's bind address;
only the compose network, where each service is its own namespace,
needed the explicit service-name override every other direct
ledger-core caller already has for LEDGER_CORE_ADDR. Found while
preparing this repo for a real containerized deployment target.
Closes docs/THREAT-MODEL.md Known-gaps #3's named TODO: ledger-core had
no Prometheus endpoint at all, so a sustained audit-log outage was
only a tracing::warn! log line, nothing alertable. Adds
ledger_core_audit_mirror_spawn_append_failures_total{kind}, served on
GET /metrics at LEDGER_CORE_METRICS_ADDR (default localhost:9099),
pre-registered at zero per kind so absence vs zero is distinguishable.
Wires the new endpoint into docker-compose.yml and the Prometheus
scrape config, which already named this exact gap as the trigger to
add the target. Fixes a render() visibility bug and a test assumption
that unobserved CounterVec label values still emit TYPE/HELP lines,
both left broken by an earlier interrupted pass on this same change.
Phase 2 section 7 (the other half, alongside sdk/go/ration and
sdk/rust/ration-client already pushed in c5297e4/8215fb8): a hand-rolled
internal/httpapi on services/gateway translating JSON <-> the gRPC source
of truth (cards, deposits, bridge funding, activity), documented as a
convenience facade over gRPC, not a parallel implementation. Also folds in
the remaining phase-2 doc updates (ARCHITECTURE/RUNBOOK/THREAT-MODEL/
AIRWALLEX-SETUP/SUBMISSION, ADR 0003/0004) and docker-compose/Makefile/
.env.example wiring from the resumed phase-2 workflow's final agents.
Verified: go build/vet/test clean across all 10 services + both new SDKs,
`make build && make test` clean end to end (C WAL, Rust ledger-core + the
new Rust SDK, every Go module including sdk/go/ration).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 2 sections 2-6 and 8 of docs/SCALE-EXPANSION.md:
- pkg/telemetry + pkg/metrics: OpenTelemetry trace propagation and
Prometheus metrics across every service hop (CLI -> agent -> gateway ->
card-issuance/tenant-registry/auth -> ledger-core), tracing added to
ledger-core itself via services/ledger-core/src/telemetry.rs.
- infra/: otel-collector + Prometheus + Grafana docker-compose stack, a
Terraform ECS Fargate module + dev/staging/prod envs (plan-only, no
apply — real AWS spend is an owner action), and a dev-only mTLS CA
(infra/dev-ca) wired through pkg/mtls + services/ledger-core/src/mtls.rs.
- services/audit-log (from the prior commit) gets its async mirror writer
in services/ledger-core/src/audit_mirror.rs, and services/gateway gets
a per-tenant token-bucket rate limiter (internal/ratelimit).
- Test depth: a Rust proptest suite proving concurrent ApplyEntry never
breaches the reserve floor (tests/apply_entry_proptest.rs, passing), a Go
fuzz seed corpus for the policy envelope (rules_fuzz_test.go), a libFuzzer
harness for the C WAL frame parser (services/wal/fuzz, deliberately not
wired into `make test` per its own Makefile comment - needs a
libFuzzer-bundled clang), and a real chaos test
(scripts/chaos/kill-ledger-mid-write.sh) that SIGKILLs ledger-core
mid-write under concurrent load and verifies via ListEntries that
nothing was lost or double-applied.
- docs/formal/ledger.tla: a TLA+ spec of the ledger state machine (balance
>= reserve_floor safety, HOLD-eventually-resolves liveness) plus a TLC
model config - not run through TLC in this pass, that's a precise
documented TODO, not faked.
- docs/adr/0001-0004, docs/RUNBOOK.md, docs/SLO.md, docs/THREAT-MODEL.md.
Verified: `make build && make test` clean (C WAL, Rust incl. the new
proptest suite, all 10 Go modules) from a cold target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>