5.5 KiB
LawSynth operations runbook
Response procedures for the alerts defined in alerts.yaml.
Each section is linked from the alert's runbook annotation. Alerts map to the
SLOs and failure modes in the production architecture (section 23).
General principle: stabilize first, root-cause second. Protect accepted-run durability and artifact integrity over raw throughput.
Quick references:
# Compose (single-node reference):
docker compose -f deploy/compose/production/compose.yaml ps
docker compose -f deploy/compose/production/compose.yaml logs -f <service>
# Health:
curl -fsS https://$LAWSYNTH_DOMAIN/v1/health
api-unavailable
Alert: LawSynthApiProbeDown — the external /v1/health probe fails.
- Confirm scope: is it the probe/network or the service?
curl -v https://$LAWSYNTH_DOMAIN/v1/healthand from inside the networkdocker compose exec gateway wget -qO- http://api:8080/v1/health. - If the proxy is down, check certificate/ACME issues in the
proxylogs. - If the API is down, check
apilogs for crash loops and its dependencies (Postgres, object store, NATS) — a hard dependency failure surfaces here. - Restart the affected service; scale API replicas if it is resource-bound.
- Verify recovery: probe returns 200 and error ratio falls.
api-error-ratio
Alerts: LawSynthApiHighErrorRatioFast / ...Slow.
- Break down
lawsynth_api_requests_totalby route and status to find the failing endpoint. - Correlate with a recent deploy (
LAWSYNTH_VERSION), a dependency alert, or a saturation alert. - If a deploy regressed, roll back to the previous pinned tag.
- If a dependency is degraded, follow its section below.
- Fast-burn is budget-critical: mitigate within minutes even if root cause is still open.
api-latency
Alert: LawSynthApiLatencyHigh (p95 > 1s).
- Check Postgres saturation and slow-query logs
(
log_min_duration_statement=1000is enabled). - Check object-store latency for artifact reads.
- Look for GC/CPU throttling on the API containers; raise CPU limits or replicas.
run-persistence
Alert: LawSynthAcceptedRunPersistenceLoss — SLO-critical (99.99%).
- This means the API acknowledged a run but the outbox/transaction did not commit. Inspect API logs for transaction/outbox errors around the spike.
- Verify Postgres write health and disk space.
- Reconcile: the scheduler is reconstructable from DB state — confirm no accepted run is missing an outbox row; requeue if a durable record exists.
- If data was lost, restore from the most recent backup set and replay.
- Open an incident; this alert must never be silenced without root cause.
queue-delay
Alert: LawSynthQueueDelayHigh (p95 wait > 5m).
- Check worker replica count and
lawsynth_job_lease_age_seconds. - Scale workers:
docker compose up -d --scale worker=N. - Confirm the scheduler is running (exactly one) and draining the outbox.
- Inspect for a poison job repeatedly failing and blocking a pool.
stale-lease
Alert: LawSynthStaleJobLease.
- A worker likely died holding a lease. Confirm with
workerlogs / restarts. - The scheduler returns the job to schedulable state after lease expiry — verify this happens; if not, check scheduler recovery logic.
- Checkpoint-compatible jobs resume; others restart explicitly (expected).
cancellation
Alert: LawSynthCancelAckSlow (p95 > 2s).
- Cancellation is cooperative plus hard timeout. Check whether workers are blocked in a long native call ignoring the cancel signal.
- Verify the hard-timeout path fires and releases resources.
artifact-checksum
Alert: LawSynthArtifactChecksumFailure — SLO-critical (100% detection).
- Detection working as designed: an artifact is corrupt or tampered.
- Quarantine the object; do NOT serve it. Identify referencing runs/worlds.
- Restore the object from the latest verified backup; re-verify on read.
- If corruption is on the object-store volume, run a storage integrity check and consider replacing the disk.
event-ordering
Alert: LawSynthEventOrderingViolation.
- Per-run-attempt event sequence must be monotonic. Check for duplicate producers or a NATS redelivery not handled idempotently.
- Confirm consumers order by
sequence, not arrival time. - Inspect the run's
run_eventsfor the offending attempt.
backup-rpo
Alert: LawSynthBackupRpoBreached (no success > 20m; RPO 15m).
- Check the backup schedule (cron/systemd timer) and
backup.shlogs. - Common causes: disk full at
BACKUP_ROOT, DB unreachable, object mirror failing. Fix and run./backup.shmanually. - Confirm a fresh set with
sha256sum -c checksums.sha256.
db-saturation
Alert: LawSynthPostgresConnectionsSaturated (> 90%).
- Identify connection sources; check for leaks or an over-provisioned pool.
- Raise
POSTGRES_MAX_CONNECTIONSor lower per-service pool sizes. - Watch for latency knock-on effects (see api-latency).
object-capacity
Alert: LawSynthObjectStoreCapacityLow (< 10% free).
- Uploads will soon fail. Expand the object-store volume.
- Run artifact garbage collection (mark live references, wait the safety window, sweep) to reclaim unreferenced objects.
scientific-warnings
Alert: LawSynthUnstableCandidateRateHigh (informational).
- Not an outage. Elevated unstable-candidate or unsupported-extrapolation warnings suggest dataset quality or method-setting issues.
- Review recent datasets/plans; feed back to the science team. No paging.