Closes docs/THREAT-MODEL.md Known-gaps #3's named TODO: ledger-core had no Prometheus endpoint at all, so a sustained audit-log outage was only a tracing::warn! log line, nothing alertable. Adds ledger_core_audit_mirror_spawn_append_failures_total{kind}, served on GET /metrics at LEDGER_CORE_METRICS_ADDR (default localhost:9099), pre-registered at zero per kind so absence vs zero is distinguishable. Wires the new endpoint into docker-compose.yml and the Prometheus scrape config, which already named this exact gap as the trigger to add the target. Fixes a render() visibility bug and a test assumption that unobserved CounterVec label values still emit TYPE/HELP lines, both left broken by an earlier interrupted pass on this same change.
211 lines
11 KiB
Markdown
211 lines
11 KiB
Markdown
# Service Level Objectives
|
|
|
|
Targets per service, and exactly how each would be measured from the
|
|
Prometheus metrics `pkg/metrics` (Go) exposes today and the metrics
|
|
Rust `ledger-core` does not yet expose (see the explicit gap called out
|
|
under ledger-core below — this is a real TODO, not glossed over).
|
|
|
|
Every Go service that wires `pkg/metrics` exposes its own `/metrics`
|
|
(`Metrics.Server(addr)` in `pkg/metrics/server.go`) on a dedicated
|
|
metrics address distinct from its gRPC/HTTP ports (e.g. gateway:
|
|
`GATEWAY_METRICS_ADDR`, default `localhost:9091`,
|
|
`services/gateway/internal/config/config.go`) — scrape that, not the
|
|
service port. Every metric carries a `service` const-label (set at
|
|
`metrics.New(serviceName)`), so one Prometheus instance scraping every
|
|
service's `/metrics` can slice any query by `service=`.
|
|
|
|
## How to read this doc
|
|
|
|
For each service: a latency target (p50/p99 for the RPC(s) that matter
|
|
most), an availability target, and the exact PromQL this would run
|
|
against once Prometheus is actually scraping every service's
|
|
`/metrics` endpoint (per `infra/otel-collector`/Grafana, section 2 —
|
|
wiring the scrape config itself is infra work, not covered here).
|
|
Targets are deliberately conservative for a system at this project's
|
|
real scale (a hackathon-grown ledger, not a payments unicorn's current
|
|
traffic) — they are meant to catch real regressions, not to be
|
|
impressive numbers.
|
|
|
|
---
|
|
|
|
## `ledger-core` (Rust)
|
|
|
|
**Why it matters most:** every balance-moving RPC in the whole system
|
|
ultimately calls `LedgerCore.ApplyEntry`
|
|
(`services/ledger-core/src/grpc.rs` → `ledger.rs::apply_entry`), which
|
|
holds `Ledger.inner`'s mutex for the full decide→WAL-append-fsync→SQLite-
|
|
commit duration (`append_and_commit`). Its tail latency is a lower bound
|
|
on every other service's tail latency for anything that moves money.
|
|
|
|
| RPC | p50 target | p99 target | Availability |
|
|
|---|---|---|---|
|
|
| `ApplyEntry` | 10ms | 50ms | 99.9% (error rate < 0.1%, excluding `FAILED_PRECONDITION`/`failed_precondition` reserve-floor declines, which are a correct business outcome, not an error) |
|
|
| `GetBalance` / `CheckReserveFloor` (read paths) | 5ms | 20ms | 99.9% |
|
|
| `CreateTenant` / `RecordDeposit` / `MarkDepositReceived` | 15ms | 75ms | 99.9% |
|
|
|
|
The `ApplyEntry` target is set by the dominant fixed cost in its path:
|
|
one `fsync()` per append (`wal_append` in `services/wal/wal.c`,
|
|
deliberately "durability > throughput" per `ARCHITECTURE.md`) plus one
|
|
SQLite transaction commit. 10ms/50ms assumes a local SSD WAL file (not
|
|
network-attached storage); the Terraform `ledger-backup` module notes
|
|
ledger-core itself "stays on local/EBS disk for the WAL" for exactly
|
|
this reason — moving the WAL to network storage would blow this budget.
|
|
|
|
**Tracing landed, metrics did not (real, scoped gap remains):**
|
|
`ledger-core` now has `tracing-opentelemetry` wired
|
|
(`src/telemetry.rs`): every RPC gets a server span via
|
|
`tonic-tracing-opentelemetry`'s `OtelGrpcLayer` (joined to the caller's
|
|
W3C `traceparent` when the Go services' `pkg/telemetry` client
|
|
interceptors sent one), exported over OTLP/gRPC to
|
|
`infra/otel-collector`, plus structured JSON logs via
|
|
`tracing_subscriber`'s `fmt().json()` layer — the full "Rust
|
|
`ledger-core`: `tracing` + `tracing-opentelemetry`, same trace
|
|
propagation via gRPC metadata" half of `docs/SCALE-EXPANSION.md`
|
|
section 2. So a trace started by a CLI/TUI command or an agent tick now
|
|
extends all the way into `ledger-core`'s own `ApplyEntry` span, visible
|
|
in Grafana/Tempo/Jaeger by `trace_id`.
|
|
|
|
What's still missing is the **full `pkg/metrics`-equivalent** half:
|
|
`ledger-core` now has one Prometheus counter and one `/metrics` HTTP
|
|
listener (`src/metrics.rs`, bound in `main.rs` at
|
|
`LEDGER_CORE_METRICS_ADDR`, scraped per
|
|
`infra/prometheus/prometheus.yml`'s `ledger-core` job) — but that
|
|
counter is scoped narrowly to `docs/THREAT-MODEL.md`'s "Known gaps" #3
|
|
(audit-mirror `spawn_append` failures), not this table's per-RPC
|
|
latency/availability targets. There is still no
|
|
`ledger_core_grpc_server_requests_total` /
|
|
`..._request_duration_seconds` pair (the Go services' `pkg/metrics`
|
|
equivalent) for `ApplyEntry` or any other RPC, so no PromQL query can
|
|
check this table's targets directly from scraped metrics yet (traces
|
|
give per-request timing, but not the aggregated histograms/counters
|
|
this table's targets are phrased against); the
|
|
fallback diagnostics are `RUNBOOK.md`'s "ledger-core won't start" /
|
|
"negative headroom" entries, a trace-based p50/p99 read against
|
|
whatever backend `infra/otel-collector` exports spans to, and
|
|
wall-clock timing from outside the process (e.g. gateway's own
|
|
`ledgerclient` call latency, which IS visible — see below). **TODO,
|
|
precise blocker:** add a Rust Prometheus client (the `prometheus`
|
|
crate, matching `pkg/metrics`'s `prometheus/client_golang` choice on
|
|
the Go side) emitting the Go-side-equivalent
|
|
`ledger_core_grpc_server_requests_total{method,code}` /
|
|
`ledger_core_grpc_server_request_duration_seconds{method}`, record them
|
|
from a small wrapper around `TracedService` (or a second tower layer
|
|
composed the same way, per `src/telemetry.rs`'s doc comment on why the
|
|
per-service wrap is required under this crate's pinned tonic 0.12), and
|
|
bind a `/metrics` HTTP listener (e.g. via `hyper` or `axum`, both
|
|
already in the dependency tree transitively) next to the gRPC server in
|
|
`main.rs` — this is additive instrumentation code, deliberately not
|
|
done in this pass alongside the tracing wiring so each lands as its own
|
|
reviewable unit.
|
|
|
|
**Indirect measurement available today:** gateway's
|
|
`internal/ledgerclient` calls are themselves instrumented (gateway's own
|
|
`ration_grpc_server_request_duration_seconds` covers gateway's *own*
|
|
RPCs, not its outbound calls to ledger-core — so even this indirect
|
|
path currently only bounds total request time from the *caller's*
|
|
RPC-to-RPC latency, via gateway's bridge/card service latency minus
|
|
known gateway-side work; it is not a substitute for direct ledger-core
|
|
instrumentation and should not be treated as one in an incident).
|
|
|
|
---
|
|
|
|
## `services/gateway` (Go) — monolith today; BFF after the split (ADR 0004)
|
|
|
|
| RPC | p50 target | p99 target | Availability |
|
|
|---|---|---|---|
|
|
| `IssueCard` / `AdjustCardLimit` | 150ms | 500ms | 99.5% (dominated by the Airwallex HTTP call in live mode; `AIRWALLEX_MODE=mock` should comfortably beat this) |
|
|
| `RationBridgeFunding` | 30ms | 120ms | 99.9% (no external HTTP call in this path — it's `GetTenant`+`GetBalance`+`ListDeposits`+`ListEntries`+`ApplyEntry` against ledger-core, per `bridge.Service.RationBridgeFunding`) |
|
|
| `StreamActivity` | n/a (long-lived stream) | — | 99.5% connection uptime; target < 2s from event publish (`internal/activity/stream.go`'s fan-out) to client delivery |
|
|
|
|
**Measurement (available today):** `ration_grpc_server_requests_total{service="gateway",method,code}`
|
|
and `ration_grpc_server_request_duration_seconds{service="gateway",method}`
|
|
(`pkg/metrics/metrics.go`, wired via `UnaryServerInterceptor`/
|
|
`StreamServerInterceptor` in `pkg/metrics/interceptors.go`, registered on
|
|
gateway's gRPC server in `cmd/gateway/main.go`). Example PromQL for the
|
|
`RationBridgeFunding` p99:
|
|
|
|
```promql
|
|
histogram_quantile(0.99,
|
|
sum(rate(ration_grpc_server_request_duration_seconds_bucket{
|
|
service="gateway", method="/ration.gateway.v1.Gateway/RationBridgeFunding"
|
|
}[5m])) by (le)
|
|
)
|
|
```
|
|
|
|
Availability (error budget) for the same RPC:
|
|
|
|
```promql
|
|
1 - (
|
|
sum(rate(ration_grpc_server_requests_total{
|
|
service="gateway", method="/ration.gateway.v1.Gateway/RationBridgeFunding", code!="OK"
|
|
}[5m]))
|
|
/
|
|
sum(rate(ration_grpc_server_requests_total{
|
|
service="gateway", method="/ration.gateway.v1.Gateway/RationBridgeFunding"
|
|
}[5m]))
|
|
)
|
|
```
|
|
|
|
**Business-outcome metrics, not pure latency/availability, but part of
|
|
this service's health story:** `ration_bridge_grants_total{service="gateway",result="granted"|"partial"|"declined"}`
|
|
and `ration_reserve_floor_headroom_minor_units{service="gateway",tenant_id}`
|
|
(both in `pkg/metrics/metrics.go`, recorded by `bridge.Service`). A step
|
|
change in the ratio of `declined` to `granted`+`partial` across *every*
|
|
tenant simultaneously is a policy/config regression, not a latency/
|
|
availability breach — see `RUNBOOK.md`'s "agent loop is granting $0
|
|
bridges" entry, which this metric feeds directly.
|
|
|
|
After the service split (ADR 0004), these same targets apply to the
|
|
thin BFF's fan-out calls; `services/card-issuance`,
|
|
`services/tenant-registry`, and `services/webhook-ingestion` each get
|
|
their own copy of this table once they exist as independently
|
|
deployable services with their own `/metrics` — not written here
|
|
speculatively, since their exact RPC surfaces (`card_issuance.proto`
|
|
etc.) don't exist in `proto/` yet.
|
|
|
|
---
|
|
|
|
## `services/agent` (Go)
|
|
|
|
Not gRPC-served (it's a polling loop, `internal/loop/runner.go`), so its
|
|
SLO is framed as the loop's own cadence and downstream-call health
|
|
rather than an RPC table:
|
|
|
|
| Dimension | Target |
|
|
|---|---|
|
|
| Poll cycle latency (one full pass over `ListLateDeposits` + per-tenant decision pipeline) | p99 < `RATION_AGENT_POLL_INTERVAL_SECONDS` (default 10s per `cmd/agent/main.go`'s `defaultPollInterval`) — a cycle that takes longer than the interval it's supposed to run on is itself the alert condition, not a separate threshold |
|
|
| LLM round-trip (`internal/llm/provider.go`'s Anthropic Messages API call) | p99 < 8s per tool-calling turn |
|
|
| Availability (successful poll cycles, excluding cycles with zero late deposits to act on) | 99% |
|
|
|
|
**Measurement:** the agent process does not currently export its own
|
|
`/metrics` (no `metrics.New("agent")` call exists in
|
|
`services/agent/cmd/agent/main.go` today, unlike gateway) — its downstream
|
|
calls land in *gateway's* and *ledger-core's* RPC metrics (as the
|
|
caller), so agent-side health today is read from structured JSON logs
|
|
(`log/slog`, one line per decision cycle) grep'd by correlation ID, not
|
|
PromQL. **TODO:** wire `metrics.New("agent")` + a `/metrics` listener in
|
|
`cmd/agent/main.go`, mirroring gateway's pattern exactly — scoped,
|
|
small, not done in this pass for the same reason as the ledger-core gap
|
|
above (instrumentation code, not documentation).
|
|
|
|
---
|
|
|
|
## `services/wal` (C, linked into `ledger-core`, not independently
|
|
deployed)
|
|
|
|
Not a service with its own SLO (it has no network surface — it's a
|
|
linked library), but its append latency is the hard floor the
|
|
`ledger-core` `ApplyEntry` target above is built on top of:
|
|
|
|
| Dimension | Target |
|
|
|---|---|
|
|
| `wal_append` (header write + payload write + `fsync()`) | p99 < 5ms on local SSD |
|
|
|
|
**Measurement:** `services/wal/bench/bench_wal.c` is the throughput
|
|
micro-benchmark that would catch a regression here (run via `make -C
|
|
services/wal` targets) — this is a build-time/CI benchmark, not a
|
|
Prometheus metric, since the WAL has no process of its own to scrape.
|
|
If `ledger-core`'s `ApplyEntry` p99 (once instrumented, per the gap
|
|
above) regresses without any change to `ledger.rs`'s logic, `bench_wal.c`
|
|
is the first place to check whether the regression is actually here.
|