ration/docs/SLO.md
caelum0x f74a5ee73b feat(ledger-core): add Prometheus counter for audit-mirror failures
Closes docs/THREAT-MODEL.md Known-gaps #3's named TODO: ledger-core had
no Prometheus endpoint at all, so a sustained audit-log outage was
only a tracing::warn! log line, nothing alertable. Adds
ledger_core_audit_mirror_spawn_append_failures_total{kind}, served on
GET /metrics at LEDGER_CORE_METRICS_ADDR (default localhost:9099),
pre-registered at zero per kind so absence vs zero is distinguishable.
Wires the new endpoint into docker-compose.yml and the Prometheus
scrape config, which already named this exact gap as the trigger to
add the target. Fixes a render() visibility bug and a test assumption
that unobserved CounterVec label values still emit TYPE/HELP lines,
both left broken by an earlier interrupted pass on this same change.
2026-10-23 07:35:50 +03:00

11 KiB

Service Level Objectives

Targets per service, and exactly how each would be measured from the Prometheus metrics pkg/metrics (Go) exposes today and the metrics Rust ledger-core does not yet expose (see the explicit gap called out under ledger-core below — this is a real TODO, not glossed over).

Every Go service that wires pkg/metrics exposes its own /metrics (Metrics.Server(addr) in pkg/metrics/server.go) on a dedicated metrics address distinct from its gRPC/HTTP ports (e.g. gateway: GATEWAY_METRICS_ADDR, default localhost:9091, services/gateway/internal/config/config.go) — scrape that, not the service port. Every metric carries a service const-label (set at metrics.New(serviceName)), so one Prometheus instance scraping every service's /metrics can slice any query by service=.

How to read this doc

For each service: a latency target (p50/p99 for the RPC(s) that matter most), an availability target, and the exact PromQL this would run against once Prometheus is actually scraping every service's /metrics endpoint (per infra/otel-collector/Grafana, section 2 — wiring the scrape config itself is infra work, not covered here). Targets are deliberately conservative for a system at this project's real scale (a hackathon-grown ledger, not a payments unicorn's current traffic) — they are meant to catch real regressions, not to be impressive numbers.


ledger-core (Rust)

Why it matters most: every balance-moving RPC in the whole system ultimately calls LedgerCore.ApplyEntry (services/ledger-core/src/grpc.rs → ledger.rs::apply_entry), which holds Ledger.inner's mutex for the full decide→WAL-append-fsync→SQLite- commit duration (append_and_commit). Its tail latency is a lower bound on every other service's tail latency for anything that moves money.

RPC p50 target p99 target Availability
ApplyEntry 10ms 50ms 99.9% (error rate < 0.1%, excluding FAILED_PRECONDITION/failed_precondition reserve-floor declines, which are a correct business outcome, not an error)
GetBalance / CheckReserveFloor (read paths) 5ms 20ms 99.9%
CreateTenant / RecordDeposit / MarkDepositReceived 15ms 75ms 99.9%

The ApplyEntry target is set by the dominant fixed cost in its path: one fsync() per append (wal_append in services/wal/wal.c, deliberately "durability > throughput" per ARCHITECTURE.md) plus one SQLite transaction commit. 10ms/50ms assumes a local SSD WAL file (not network-attached storage); the Terraform ledger-backup module notes ledger-core itself "stays on local/EBS disk for the WAL" for exactly this reason — moving the WAL to network storage would blow this budget.

Tracing landed, metrics did not (real, scoped gap remains): ledger-core now has tracing-opentelemetry wired (src/telemetry.rs): every RPC gets a server span via tonic-tracing-opentelemetry's OtelGrpcLayer (joined to the caller's W3C traceparent when the Go services' pkg/telemetry client interceptors sent one), exported over OTLP/gRPC to infra/otel-collector, plus structured JSON logs via tracing_subscriber's fmt().json() layer — the full "Rust ledger-core: tracing + tracing-opentelemetry, same trace propagation via gRPC metadata" half of docs/SCALE-EXPANSION.md section 2. So a trace started by a CLI/TUI command or an agent tick now extends all the way into ledger-core's own ApplyEntry span, visible in Grafana/Tempo/Jaeger by trace_id.

What's still missing is the full pkg/metrics-equivalent half: ledger-core now has one Prometheus counter and one /metrics HTTP listener (src/metrics.rs, bound in main.rs at LEDGER_CORE_METRICS_ADDR, scraped per infra/prometheus/prometheus.yml's ledger-core job) — but that counter is scoped narrowly to docs/THREAT-MODEL.md's "Known gaps" #3 (audit-mirror spawn_append failures), not this table's per-RPC latency/availability targets. There is still no ledger_core_grpc_server_requests_total / ..._request_duration_seconds pair (the Go services' pkg/metrics equivalent) for ApplyEntry or any other RPC, so no PromQL query can check this table's targets directly from scraped metrics yet (traces give per-request timing, but not the aggregated histograms/counters this table's targets are phrased against); the fallback diagnostics are RUNBOOK.md's "ledger-core won't start" / "negative headroom" entries, a trace-based p50/p99 read against whatever backend infra/otel-collector exports spans to, and wall-clock timing from outside the process (e.g. gateway's own ledgerclient call latency, which IS visible — see below). TODO, precise blocker: add a Rust Prometheus client (the prometheus crate, matching pkg/metrics's prometheus/client_golang choice on the Go side) emitting the Go-side-equivalent ledger_core_grpc_server_requests_total{method,code} / ledger_core_grpc_server_request_duration_seconds{method}, record them from a small wrapper around TracedService (or a second tower layer composed the same way, per src/telemetry.rs's doc comment on why the per-service wrap is required under this crate's pinned tonic 0.12), and bind a /metrics HTTP listener (e.g. via hyper or axum, both already in the dependency tree transitively) next to the gRPC server in main.rs — this is additive instrumentation code, deliberately not done in this pass alongside the tracing wiring so each lands as its own reviewable unit.

Indirect measurement available today: gateway's internal/ledgerclient calls are themselves instrumented (gateway's own ration_grpc_server_request_duration_seconds covers gateway's own RPCs, not its outbound calls to ledger-core — so even this indirect path currently only bounds total request time from the caller's RPC-to-RPC latency, via gateway's bridge/card service latency minus known gateway-side work; it is not a substitute for direct ledger-core instrumentation and should not be treated as one in an incident).


services/gateway (Go) — monolith today; BFF after the split (ADR 0004)

RPC p50 target p99 target Availability
IssueCard / AdjustCardLimit 150ms 500ms 99.5% (dominated by the Airwallex HTTP call in live mode; AIRWALLEX_MODE=mock should comfortably beat this)
RationBridgeFunding 30ms 120ms 99.9% (no external HTTP call in this path — it's GetTenant+GetBalance+ListDeposits+ListEntries+ApplyEntry against ledger-core, per bridge.Service.RationBridgeFunding)
StreamActivity n/a (long-lived stream) — 99.5% connection uptime; target < 2s from event publish (internal/activity/stream.go's fan-out) to client delivery

Measurement (available today): ration_grpc_server_requests_total{service="gateway",method,code} and ration_grpc_server_request_duration_seconds{service="gateway",method} (pkg/metrics/metrics.go, wired via UnaryServerInterceptor/ StreamServerInterceptor in pkg/metrics/interceptors.go, registered on gateway's gRPC server in cmd/gateway/main.go). Example PromQL for the RationBridgeFunding p99:

histogram_quantile(0.99,
  sum(rate(ration_grpc_server_request_duration_seconds_bucket{
    service="gateway", method="/ration.gateway.v1.Gateway/RationBridgeFunding"
  }[5m])) by (le)
)

Availability (error budget) for the same RPC:

1 - (
  sum(rate(ration_grpc_server_requests_total{
    service="gateway", method="/ration.gateway.v1.Gateway/RationBridgeFunding", code!="OK"
  }[5m]))
  /
  sum(rate(ration_grpc_server_requests_total{
    service="gateway", method="/ration.gateway.v1.Gateway/RationBridgeFunding"
  }[5m]))
)

Business-outcome metrics, not pure latency/availability, but part of this service's health story: ration_bridge_grants_total{service="gateway",result="granted"|"partial"|"declined"} and ration_reserve_floor_headroom_minor_units{service="gateway",tenant_id} (both in pkg/metrics/metrics.go, recorded by bridge.Service). A step change in the ratio of declined to granted+partial across every tenant simultaneously is a policy/config regression, not a latency/ availability breach — see RUNBOOK.md's "agent loop is granting $0 bridges" entry, which this metric feeds directly.

After the service split (ADR 0004), these same targets apply to the thin BFF's fan-out calls; services/card-issuance, services/tenant-registry, and services/webhook-ingestion each get their own copy of this table once they exist as independently deployable services with their own /metrics — not written here speculatively, since their exact RPC surfaces (card_issuance.proto etc.) don't exist in proto/ yet.


services/agent (Go)

Not gRPC-served (it's a polling loop, internal/loop/runner.go), so its SLO is framed as the loop's own cadence and downstream-call health rather than an RPC table:

Dimension Target
Poll cycle latency (one full pass over ListLateDeposits + per-tenant decision pipeline) p99 < RATION_AGENT_POLL_INTERVAL_SECONDS (default 10s per cmd/agent/main.go's defaultPollInterval) — a cycle that takes longer than the interval it's supposed to run on is itself the alert condition, not a separate threshold
LLM round-trip (internal/llm/provider.go's Anthropic Messages API call) p99 < 8s per tool-calling turn
Availability (successful poll cycles, excluding cycles with zero late deposits to act on) 99%

Measurement: the agent process does not currently export its own /metrics (no metrics.New("agent") call exists in services/agent/cmd/agent/main.go today, unlike gateway) — its downstream calls land in gateway's and ledger-core's RPC metrics (as the caller), so agent-side health today is read from structured JSON logs (log/slog, one line per decision cycle) grep'd by correlation ID, not PromQL. TODO: wire metrics.New("agent") + a /metrics listener in cmd/agent/main.go, mirroring gateway's pattern exactly — scoped, small, not done in this pass for the same reason as the ledger-core gap above (instrumentation code, not documentation).


services/wal (C, linked into ledger-core, not independently

deployed)

Not a service with its own SLO (it has no network surface — it's a linked library), but its append latency is the hard floor the ledger-core ApplyEntry target above is built on top of:

Dimension Target
wal_append (header write + payload write + fsync()) p99 < 5ms on local SSD

Measurement: services/wal/bench/bench_wal.c is the throughput micro-benchmark that would catch a regression here (run via make -C services/wal targets) — this is a build-time/CI benchmark, not a Prometheus metric, since the WAL has no process of its own to scrape. If ledger-core's ApplyEntry p99 (once instrumented, per the gap above) regresses without any change to ledger.rs's logic, bench_wal.c is the first place to check whether the regression is actually here.