ration/docs/SUBMISSION.md
caelum0x 25a118a131 docs(submission): judge-facing SUBMISSION.md was stuck describing round 5; add rounds 6-37
The "Updates since initial submission" section stopped at round 5
(2026-10-29) while the codebase is 32 rounds past that — agent tool
tenant-pinning, mTLS actually enforced, concurrency correctness on every
money-moving path, WAL/FFI crash-safety hardening, and a cross-tenant
ledger idempotency fix all landed since, with zero mention in the one
doc judges are most likely to read end to end.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-05 18:50:16 +03:00

12 KiB

Ration — hackathon submission

Track: Airwallex Agentic Banking Hackathon — Starter Kit 5, Platform Spend Controller (Multi-Tenant Platforms track).

Problem (Starter Kit 5)

A multi-tenant platform (marketplace, payroll, gig/freight exchange — see the 3 demo tenants) issues cards to its own customers against money the platform expects to receive on a schedule (e.g. settlement payouts). When an expected deposit is late, the customer side still wants to spend, but the platform can't just let spend run unconstrained against money that hasn't landed — that risks breaching the platform's own reserve floor (its own capital, never touchable) if the deposit never shows up at all.

Starter Kit 5 asks for a spend controller that makes this tradeoff safely and explainably: let the tenant bridge a safe fraction of what's coming, never risk the platform's floor, and make every decision auditable enough that a human (or a judge) can see exactly why a given amount was granted or declined.

Approach

Full design in docs/ARCHITECTURE.md (service layout, money model, crash-safety) and docs/AGENT-DESIGN.md (the rationing policy itself). Summary:

Six services, four languages, one invariant that's never delegated to an LLM:

  • services/wal (C) — a real crash-safe write-ahead log: length-prefixed, CRC32-checksummed frames, fsync() after every append, torn-tail recovery on restart. This is the durability floor everything else sits on.
  • services/ledger-core (Rust) — the authoritative ledger. Owns tenants, deposits, balances, and the one function that's allowed to move money (apply_entry): idempotent on an idempotency key, and for DEBIT/HOLD, checks the reserve floor before writing to the WAL. Links libwal.a via FFI, with exactly one unsafe module (src/wal_ffi.rs) wrapping it safely.
  • services/gateway (Go) — owns the Airwallex integration (mock or live behind one interface, see below) and re-validates every balance-moving request against the same reserve-floor/bridge-cap rules again, independently of the agent, before proxying to ledger-core. Defense in depth: the agent is never trusted as the only gate.
  • services/agent (Go) — the actual agentic loop. Every poll cycle: list currently-late deposits, compute a deterministic hard-rule envelope per tenant (reserve floor, 60% bridge cap, 24h velocity cap — see AGENT-DESIGN.md), record that envelope as an auditable decision before calling any LLM, then hand the LLM only the already-safe envelope to decide phrasing / spend-control tightening / escalation — never whether to breach a hard rule. Talks to the Anthropic Messages API directly over net/http (no SDK dependency), with a real tool-calling loop (ration_bridge_funding, issue_card, adjust_card_limit, notify_tenant).
  • services/tui (Go, Bubble Tea) — live dashboard: tenants, deposits, cards, and an activity feed fed by gateway.proto's StreamActivity.
  • services/cli (Go, cobra) — the judge-facing demo driver: ration seed (creates + funds 3 demo tenants), ration simulate-deposit --late (forces the scenario), ration status (shows the resulting balance, cards, and agent decisions).

The audit trail is the point. Every hard-rule computation and every LLM tool call is written via RecordAgentDecision to ledger-core before any money moves, with a policy_snapshot_json capturing the exact inputs (headroom, deposit amount, velocity state) that produced the decision — see docs/DEMO-SCRIPT.md for what that looks like for a real run, including the case where the LLM call itself fails (missing ANTHROPIC_API_KEY): the hard-rule decision is still computed, still recorded, and the LLM failure is logged loudly, never swallowed.

Scale-out posture is written down, not hand-waved: service decomposition, observability, infra-as-code, formal verification of the reserve-floor invariant, and fuzzing/chaos testing are scoped in docs/SCALE-EXPANSION.md as the next phase past this submission.

What's mocked vs. real

See docs/AIRWALLEX-SETUP.md for the live-credential status in detail. Short version:

  • Real: the WAL, the ledger (double-entry-equivalent balance tracking, reserve-floor enforcement, idempotency, crash recovery — all covered by tests that actually kill the process mid-write and assert nothing was lost or double-applied), the policy math, the agent loop's tool-calling against the Anthropic API, the gRPC wiring between every service, the TUI, the CLI, and defense-in-depth re-validation at the gateway layer.
  • Mocked: AIRWALLEX_MODE=mock (the default) runs against services/card-issuance/internal/airwallex/mock.go, an in-memory fake that mirrors the real Issuing API, Global Accounts, and webhook payload shapes closely enough that flipping to AIRWALLEX_MODE=live with real sandbox credentials is a one-line env change, not a rewrite — same code path, same interface (airwallex.Client), different implementation. Sandbox platform access is a per-participant approval from the organizers, requested but not yet granted as of this submission (see AIRWALLEX-SETUP.md for the exact blocker and status).

How to run it

make build
scripts/run-demo.sh      # starts ledger-core + gateway (mock) + agent, seeds + funds 3 tenants
./bin/tui                # second terminal — live activity feed
./bin/ration simulate-deposit --tenant <id> --amount 400000 --late
# wait ~10s for the agent's next poll
./bin/ration status --tenant <id>   # see the recorded decision + rationale
scripts/stop-demo.sh

Full walkthrough with exact expected output: docs/DEMO-SCRIPT.md.

Verification performed for this submission

  • make build — clean build of the C WAL, Rust ledger-core (release), and all 4 Go binaries.
  • go build ./..., go vet ./..., go test ./... — clean across every Go module in the workspace (pkg, agent, cli, gateway, tui).
  • cargo build --release && cargo test --release — 29 passing tests (unit + FFI + idempotency + crash-recovery + reserve-floor integration suites), 0 failed.
  • make -C services/wal test — the C WAL's own append/recovery/corruption test binary, all passing.
  • A real end-to-end run: built everything, ran scripts/run-demo.sh, seeded + funded 3 tenants, forced a late deposit via ration simulate-deposit --late, and confirmed ration status recorded a real AGENT_ACTION_RATION_BRIDGE decision with a computed (not templated) rationale, followed by a correctly-triggered AGENT_ACTION_DECLINE on the next poll once the 24h velocity cap kicked in — both without ANTHROPIC_API_KEY set, proving the hard-rule envelope and its audit trail run independently of (and before) the LLM call.

Two real bugs were found and fixed during that end-to-end run (not just compile-time issues):

  1. services/ledger-core's bind address env var didn't match what every other service dials (RATION_LEDGER_ADDR vs. the LEDGER_CORE_ADDR every other service, and .env.example, agree on) — the whole stack would never actually connect together with default config.
  2. MarkDepositReceived updated a deposit's status but never credited the tenant's ledger balance — meaning no tenant could ever accumulate headroom, so the bridge-funding policy could only ever decline, regardless of how a late deposit was sized. Fixed at the ledger-core level (atomic WAL event covering both the status update and the credit, so a crash between the two is impossible by construction), and covered by new tests (tests/deposit_received_credits_balance.rs). ration seed was also extended to fund each demo tenant with settled opening capital through this same path, so the judge-facing demo starts with real headroom instead of a guaranteed decline.

Updates since initial submission (rounds 2-5, through 2026-10-29)

This section stays short on purpose — full detail per round is in BUILD-LOG-2026-10-23.md and plans/airwallex-hackathon.md (both outside this repo's judge-facing scope for the latter) — but the "What's mocked vs. real" section above predates all of it and would otherwise undersell real work:

  • Encryption at rest: ledger-core's WAL and its SQLite index's free-text columns (deposit references, entry memos, agent decision rationale/trigger/policy-snapshot) are AES-256-GCM-encrypted under RATION_LEDGER_ENCRYPTION_KEY — verified live by confirming the on-disk files are genuine ciphertext, not just that encryption code exists.
  • Tenant-scoped authorization end to end: RATION_GATEWAY_REQUIRE_ TENANT_AUTH and RATION_LEDGER_REQUIRE_TENANT_AUTH both default to true now (not opt-in false) — cli/tui/agent each mint an operator credential at startup and attach it to every call, gateway's GetCard/AdjustCardLimit resolve card_id → tenant via internal/cardtenant, StreamActivity has its own stream interceptor, and ledger-core's own gRPC surface enforces the same check directly — closing the "cli/tui/agent bypass gateway entirely" gap docs/THREAT-MODEL.md used to name as the biggest one.
  • API-key lifecycle, now complete: ration apikey issue mints a credential for sdk/go/ration callers, ration apikey revoke invalidates one immediately (verified live: a revoked key's next IssueCard call gets Unauthenticated: ... revoked), and ration apikey list (this round) is the missing read side — an operator can see every issued key's tenant, name, creation time, and revocation state without needing to have personally kept the key_id from when it was minted. No RPC in this family ever returns a plaintext secret after issue's one-time response.
  • docker compose up --build actually brought up the full 12-service stack for the first time — five services previously crash-looped on missing AUTH_GRPC_ADDR/AUTH_ADDR env wiring, now fixed.
  • A real ~68s demo video exists locally (demo-video/, gitignored — uploading it and the HackerEarth submission click-through are the two owner-only steps left).

Updates since previous update (rounds 6-37, through 2026-11-05)

Also kept short — full per-round detail in BUILD-LOG-2026-10-23.md and plans/airwallex-hackathon.md. 32 more rounds landed after the section above, most of them real correctness/security gaps found by re-reading this exact codebase adversarially, not new features:

  • The agent's money-moving tools are now tenant-pinned: the LLM's adjust_card_limit tool previously let a model redirect a limit change to any tenant's card (round 34); the other three agent tools were re-audited and confirmed to already hardcode the calling tenant. The hard-rule envelope was never bypassable, but this closed the one tool surface where a model's own tool-call arguments, not just its reasoning, could move someone else's money.
  • mTLS was silently a no-op in production mode: every internal service's dial to auth skipped client certs entirely (round 30) — fixed so the production-hardening path documented in docs/THREAT-MODEL.md actually does what it claims.
  • Concurrency correctness on every money-moving path: per-card and per-tenant mutexes now serialize AdjustCardLimit/CloseCard (round 25a), RationBridgeFunding (round 25b), and deposit-webhook matching (round 26) — each reproduced live first (two concurrent calls on the same card/tenant permanently stranding real funds or double-matching a deposit) before the fix.
  • WAL/FFI crash-safety boundary hardened: an apply failure after a durable WAL append could misreport success as failure, risking a resurrected entry on restart — now process::abort()s instead of silently misclassifying (round 32), followed by a 298k-execution fuzz pass (round 33) that found no further issue.
  • Ledger idempotency closed a cross-tenant replay gap: the idempotency_key lookup in apply_entry_checked ignored tenant_id, so a key collision across tenants could silently no-op a second tenant's real entry as if it had already succeeded (round 36).
  • This round (37): suspendAfterFailure — IssueCard's cleanup path when a just-issued Airwallex card can't be backed by a ledger hold — discarded its own SetCardControls error, leaving a real, live Airwallex card at full limit with zero operator-visible signal (unlike its sibling revertCardControls, which already logged loudly). Fixed to log loudly with the card id; kept the repro as a permanent regression test.