Closes docs/THREAT-MODEL.md Known-gaps #3's named TODO: ledger-core had no Prometheus endpoint at all, so a sustained audit-log outage was only a tracing::warn! log line, nothing alertable. Adds ledger_core_audit_mirror_spawn_append_failures_total{kind}, served on GET /metrics at LEDGER_CORE_METRICS_ADDR (default localhost:9099), pre-registered at zero per kind so absence vs zero is distinguishable. Wires the new endpoint into docker-compose.yml and the Prometheus scrape config, which already named this exact gap as the trigger to add the target. Fixes a render() visibility bug and a test assumption that unobserved CounterVec label values still emit TYPE/HELP lines, both left broken by an earlier interrupted pass on this same change.
11 KiB
Service Level Objectives
Targets per service, and exactly how each would be measured from the
Prometheus metrics pkg/metrics (Go) exposes today and the metrics
Rust ledger-core does not yet expose (see the explicit gap called out
under ledger-core below — this is a real TODO, not glossed over).
Every Go service that wires pkg/metrics exposes its own /metrics
(Metrics.Server(addr) in pkg/metrics/server.go) on a dedicated
metrics address distinct from its gRPC/HTTP ports (e.g. gateway:
GATEWAY_METRICS_ADDR, default localhost:9091,
services/gateway/internal/config/config.go) — scrape that, not the
service port. Every metric carries a service const-label (set at
metrics.New(serviceName)), so one Prometheus instance scraping every
service's /metrics can slice any query by service=.
How to read this doc
For each service: a latency target (p50/p99 for the RPC(s) that matter
most), an availability target, and the exact PromQL this would run
against once Prometheus is actually scraping every service's
/metrics endpoint (per infra/otel-collector/Grafana, section 2 —
wiring the scrape config itself is infra work, not covered here).
Targets are deliberately conservative for a system at this project's
real scale (a hackathon-grown ledger, not a payments unicorn's current
traffic) — they are meant to catch real regressions, not to be
impressive numbers.
ledger-core (Rust)
Why it matters most: every balance-moving RPC in the whole system
ultimately calls LedgerCore.ApplyEntry
(services/ledger-core/src/grpc.rs → ledger.rs::apply_entry), which
holds Ledger.inner's mutex for the full decide→WAL-append-fsync→SQLite-
commit duration (append_and_commit). Its tail latency is a lower bound
on every other service's tail latency for anything that moves money.
| RPC | p50 target | p99 target | Availability |
|---|---|---|---|
ApplyEntry |
10ms | 50ms | 99.9% (error rate < 0.1%, excluding FAILED_PRECONDITION/failed_precondition reserve-floor declines, which are a correct business outcome, not an error) |
GetBalance / CheckReserveFloor (read paths) |
5ms | 20ms | 99.9% |
CreateTenant / RecordDeposit / MarkDepositReceived |
15ms | 75ms | 99.9% |
The ApplyEntry target is set by the dominant fixed cost in its path:
one fsync() per append (wal_append in services/wal/wal.c,
deliberately "durability > throughput" per ARCHITECTURE.md) plus one
SQLite transaction commit. 10ms/50ms assumes a local SSD WAL file (not
network-attached storage); the Terraform ledger-backup module notes
ledger-core itself "stays on local/EBS disk for the WAL" for exactly
this reason — moving the WAL to network storage would blow this budget.
Tracing landed, metrics did not (real, scoped gap remains):
ledger-core now has tracing-opentelemetry wired
(src/telemetry.rs): every RPC gets a server span via
tonic-tracing-opentelemetry's OtelGrpcLayer (joined to the caller's
W3C traceparent when the Go services' pkg/telemetry client
interceptors sent one), exported over OTLP/gRPC to
infra/otel-collector, plus structured JSON logs via
tracing_subscriber's fmt().json() layer — the full "Rust
ledger-core: tracing + tracing-opentelemetry, same trace
propagation via gRPC metadata" half of docs/SCALE-EXPANSION.md
section 2. So a trace started by a CLI/TUI command or an agent tick now
extends all the way into ledger-core's own ApplyEntry span, visible
in Grafana/Tempo/Jaeger by trace_id.
What's still missing is the full pkg/metrics-equivalent half:
ledger-core now has one Prometheus counter and one /metrics HTTP
listener (src/metrics.rs, bound in main.rs at
LEDGER_CORE_METRICS_ADDR, scraped per
infra/prometheus/prometheus.yml's ledger-core job) — but that
counter is scoped narrowly to docs/THREAT-MODEL.md's "Known gaps" #3
(audit-mirror spawn_append failures), not this table's per-RPC
latency/availability targets. There is still no
ledger_core_grpc_server_requests_total /
..._request_duration_seconds pair (the Go services' pkg/metrics
equivalent) for ApplyEntry or any other RPC, so no PromQL query can
check this table's targets directly from scraped metrics yet (traces
give per-request timing, but not the aggregated histograms/counters
this table's targets are phrased against); the
fallback diagnostics are RUNBOOK.md's "ledger-core won't start" /
"negative headroom" entries, a trace-based p50/p99 read against
whatever backend infra/otel-collector exports spans to, and
wall-clock timing from outside the process (e.g. gateway's own
ledgerclient call latency, which IS visible — see below). TODO,
precise blocker: add a Rust Prometheus client (the prometheus
crate, matching pkg/metrics's prometheus/client_golang choice on
the Go side) emitting the Go-side-equivalent
ledger_core_grpc_server_requests_total{method,code} /
ledger_core_grpc_server_request_duration_seconds{method}, record them
from a small wrapper around TracedService (or a second tower layer
composed the same way, per src/telemetry.rs's doc comment on why the
per-service wrap is required under this crate's pinned tonic 0.12), and
bind a /metrics HTTP listener (e.g. via hyper or axum, both
already in the dependency tree transitively) next to the gRPC server in
main.rs — this is additive instrumentation code, deliberately not
done in this pass alongside the tracing wiring so each lands as its own
reviewable unit.
Indirect measurement available today: gateway's
internal/ledgerclient calls are themselves instrumented (gateway's own
ration_grpc_server_request_duration_seconds covers gateway's own
RPCs, not its outbound calls to ledger-core — so even this indirect
path currently only bounds total request time from the caller's
RPC-to-RPC latency, via gateway's bridge/card service latency minus
known gateway-side work; it is not a substitute for direct ledger-core
instrumentation and should not be treated as one in an incident).
services/gateway (Go) — monolith today; BFF after the split (ADR 0004)
| RPC | p50 target | p99 target | Availability |
|---|---|---|---|
IssueCard / AdjustCardLimit |
150ms | 500ms | 99.5% (dominated by the Airwallex HTTP call in live mode; AIRWALLEX_MODE=mock should comfortably beat this) |
RationBridgeFunding |
30ms | 120ms | 99.9% (no external HTTP call in this path — it's GetTenant+GetBalance+ListDeposits+ListEntries+ApplyEntry against ledger-core, per bridge.Service.RationBridgeFunding) |
StreamActivity |
n/a (long-lived stream) | — | 99.5% connection uptime; target < 2s from event publish (internal/activity/stream.go's fan-out) to client delivery |
Measurement (available today): ration_grpc_server_requests_total{service="gateway",method,code}
and ration_grpc_server_request_duration_seconds{service="gateway",method}
(pkg/metrics/metrics.go, wired via UnaryServerInterceptor/
StreamServerInterceptor in pkg/metrics/interceptors.go, registered on
gateway's gRPC server in cmd/gateway/main.go). Example PromQL for the
RationBridgeFunding p99:
histogram_quantile(0.99,
sum(rate(ration_grpc_server_request_duration_seconds_bucket{
service="gateway", method="/ration.gateway.v1.Gateway/RationBridgeFunding"
}[5m])) by (le)
)
Availability (error budget) for the same RPC:
1 - (
sum(rate(ration_grpc_server_requests_total{
service="gateway", method="/ration.gateway.v1.Gateway/RationBridgeFunding", code!="OK"
}[5m]))
/
sum(rate(ration_grpc_server_requests_total{
service="gateway", method="/ration.gateway.v1.Gateway/RationBridgeFunding"
}[5m]))
)
Business-outcome metrics, not pure latency/availability, but part of
this service's health story: ration_bridge_grants_total{service="gateway",result="granted"|"partial"|"declined"}
and ration_reserve_floor_headroom_minor_units{service="gateway",tenant_id}
(both in pkg/metrics/metrics.go, recorded by bridge.Service). A step
change in the ratio of declined to granted+partial across every
tenant simultaneously is a policy/config regression, not a latency/
availability breach — see RUNBOOK.md's "agent loop is granting $0
bridges" entry, which this metric feeds directly.
After the service split (ADR 0004), these same targets apply to the
thin BFF's fan-out calls; services/card-issuance,
services/tenant-registry, and services/webhook-ingestion each get
their own copy of this table once they exist as independently
deployable services with their own /metrics — not written here
speculatively, since their exact RPC surfaces (card_issuance.proto
etc.) don't exist in proto/ yet.
services/agent (Go)
Not gRPC-served (it's a polling loop, internal/loop/runner.go), so its
SLO is framed as the loop's own cadence and downstream-call health
rather than an RPC table:
| Dimension | Target |
|---|---|
Poll cycle latency (one full pass over ListLateDeposits + per-tenant decision pipeline) |
p99 < RATION_AGENT_POLL_INTERVAL_SECONDS (default 10s per cmd/agent/main.go's defaultPollInterval) — a cycle that takes longer than the interval it's supposed to run on is itself the alert condition, not a separate threshold |
LLM round-trip (internal/llm/provider.go's Anthropic Messages API call) |
p99 < 8s per tool-calling turn |
| Availability (successful poll cycles, excluding cycles with zero late deposits to act on) | 99% |
Measurement: the agent process does not currently export its own
/metrics (no metrics.New("agent") call exists in
services/agent/cmd/agent/main.go today, unlike gateway) — its downstream
calls land in gateway's and ledger-core's RPC metrics (as the
caller), so agent-side health today is read from structured JSON logs
(log/slog, one line per decision cycle) grep'd by correlation ID, not
PromQL. TODO: wire metrics.New("agent") + a /metrics listener in
cmd/agent/main.go, mirroring gateway's pattern exactly — scoped,
small, not done in this pass for the same reason as the ledger-core gap
above (instrumentation code, not documentation).
services/wal (C, linked into ledger-core, not independently
deployed)
Not a service with its own SLO (it has no network surface — it's a
linked library), but its append latency is the hard floor the
ledger-core ApplyEntry target above is built on top of:
| Dimension | Target |
|---|---|
wal_append (header write + payload write + fsync()) |
p99 < 5ms on local SSD |
Measurement: services/wal/bench/bench_wal.c is the throughput
micro-benchmark that would catch a regression here (run via make -C services/wal targets) — this is a build-time/CI benchmark, not a
Prometheus metric, since the WAL has no process of its own to scrape.
If ledger-core's ApplyEntry p99 (once instrumented, per the gap
above) regresses without any change to ledger.rs's logic, bench_wal.c
is the first place to check whether the regression is actually here.