From 05f6460e5f94478b412ee5448dcfcd12e7a6a662 Mon Sep 17 00:00:00 2001 From: caelum0x Date: Mon, 24 Aug 2026 20:52:24 +0300 Subject: [PATCH] docs: enterprise architecture + graph-engineering + scale for rit/rithub/ranguard --- ARCHITECTURE.md | 384 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 384 insertions(+) create mode 100644 ARCHITECTURE.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..153684f --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,384 @@ +# rit — Enterprise Architecture + +> **Scope of this document.** The *target* enterprise shape for `rit`: the full +> Git implementation as a multi-crate Rust workspace, the phased build order that +> maps the [`PLAN.md`](./PLAN.md) deferred scope into crates, and the SDK/interop +> surface other Railyard apps (chiefly [`rithub`](../rithub)) build on. This is a +> **planning artifact** — it does not change app code. Today `rit` is a single +> crate (`src/{object,odb,oid,refs,repo,blob,tree,commit,tag,signature,headers,error}.rs`, +> 35 tests incl. real-git interop against `git` 2.50). The target below decomposes +> that core and lands the deferred scope (packfiles/delta, index/worktree, wire +> protocol v2, merge/rebase, SHA-256) as discrete crates behind a hard +> compatibility gate. + +--- + +## 1. Design invariants (carried from PLAN.md) + +These are non-negotiable and shape every crate boundary: + +- **Library-first.** Every capability lives in a library crate; the CLI is a thin + driver. No behavior is CLI-only. +- **Real-git-compat gate.** No compatibility claim ships without a named test. + Object ids are byte-identical to `git hash-object`; `git fsck` accepts what we + write; every parser has a differential test against the reference `git` CLI. + **The forge (`rithub`) is not built until the `rit` core passes interop, + crash-recovery, parser-fuzzing, and concurrent-ref gates** (PLAN "Initial + customer and promise" + "Release sequence"). +- **Adversarial input.** Repository data is hostile. Every parser is fail-closed, + bounded (the `MAX_OBJECT_SIZE` cap and `ZlibDecoder::take` guard in today's + `odb.rs` are the template), and fuzzed. +- **Shared crates cannot depend on the forge, a UI, or a specific database** + (PLAN "Rust workspace architecture"). Storage APIs expose bounded streams, not + "assume it fits in memory." +- **Hash-agnostic core.** The object model is generic over the hash so the + SHA-1 → SHA-256 path is a type parameter, not a rewrite. +- **Immutability.** Objects are content-addressed and never mutated in place; + writes are atomic (temp file → `fsync` → rename, as `Odb::write` already does). + +--- + +## 2. Target workspace tree + +The standalone `rit` package becomes a Cargo **workspace** (it keeps +`[workspace]` isolation from the desktop GPUI/Metal build, exactly like `mint` +and `agent-runtime` are excluded at the repo root). Crates use the `rit-` prefix; +the PLAN's `git-*` names are noted as aliases. + +```text +apps/rit/ + Cargo.toml # [workspace] members = crates/* + rust-toolchain.toml # pinned MSRV (PLAN pre-code gate item 1) + README.md + PLAN.md # product + release plan (source of truth) + ARCHITECTURE.md # this document + deny.toml # cargo-deny: license/advisory gate + docs/ + COMPATIBILITY.md # the frozen compat matrix (Stage 0 deliverable) + THREAT-MODEL.md # adversarial-input threat model (Stage 0) + RECOVERY.md # corruption/crash-recovery rules (Stage 0) + fixtures/ # licensed real-git repos for differential tests + crates/ + rit-hash/ # [git-hash] hash abstraction: Sha1, Sha256, + # sha1dc collision detection, ObjectId, + # hex, prefix. Everything else is generic + # over this so SHA-256 is a type param. + rit-object/ # [git-object] object model: ObjectType, loose + # framing " \0", canonical + # (de)serialization for blob/tree/commit/tag, + # validation. (today's object/blob/tree/ + # commit/tag/signature/headers.rs) + rit-odb/ # [git-odb] object database: loose store (today's + # odb.rs) + packed store, quarantine dirs, + # alternates, streaming reads, fsck-grade + # verification. Facade over rit-pack + loose. + rit-pack/ # [git-pack] packfile + .idx parsing, ref/ofs + # delta resolution, thin-pack fixups, pack + # writing, reachability bitmaps, delta chain + # bounds. (PLAN deferred: packfiles/delta) + rit-refs/ # [git-refs] refs, symbolic refs, packed-refs + # (today's refs.rs) + reflogs, atomic + # compare-and-swap transactions, lock files, + # lock recovery, ref races. + rit-index/ # [git-index] index (staging) formats v2/3/4 + + # extensions (TREE, UNTR, sparse). (PLAN + # deferred: index/worktree) + rit-worktree/ # [git-worktree] checkout, status, add, restore, + # .gitignore rules, filesystem-safety adapters + # (path traversal, case-fold, symlink hazards). + rit-revision/ # [git-revision] rev syntax (HEAD~3, a..b, ^), + # graph walks, reachability, merge-base — the + # bounded walk in today's Repository::log, + # generalized. + rit-diff/ # [git-diff] Myers/histogram text diff, binary + # diff, rename/copy detection, hunk model. + rit-merge/ # [git-merge] merge bases, three-way merge, + # recursive/ort strategy, conflict model, the + # sequencer for rebase/cherry-pick/revert. + rit-protocol/ # [git-protocol] smart protocol v2, pkt-line, + # capability negotiation, upload-pack (fetch/ + # clone) + receive-pack (push), quarantine of + # received objects. Transport-agnostic. + rit-transport/ # HTTP + SSH transport adapters for rit-protocol + # (client side): git-smart-HTTP, ssh exec. + rit-fsck/ # object/connectivity/reachability verification, + # corruption reporting — the read-only "is this + # repo sound?" pass used by Stage 1 + CI. + rit-repo/ # [porcelain] the Repository facade (today's + # repo.rs) tying odb+refs+index+worktree into + # init/add/commit/log/branch/checkout/merge. + rit-porcelain/ # high-level command semantics matching git's + # UX (status output, log formatting) — what the + # CLI and SDK both call. + rit-cli/ # [git-cli] the `rit` binary: thin clap driver + # over rit-porcelain (today's main.rs, + # hash-object/cat-file/ls-tree/log → grows to + # init/add/commit/branch/clone/fetch/push). + rit-sdk/ # the stable, semver'd public API surface other + # apps depend on (rithub, tooling). Re-exports + # a curated subset; hides crate churn. + rit-testkit/ # differential harness: shell out to real `git`, + # assert byte-equality (today's tests/interop.rs + # promoted to a reusable crate), fixture repos, + # property-test generators. + rit-fuzz/ # cargo-fuzz targets: every untrusted parser + # (loose, tree, commit, pack, idx, pkt-line, + # index, packed-refs) + protocol state machines. +``` + +### Crate dependency graph (acyclic, leaf → root) + +```text +rit-hash + └─ rit-object + ├─ rit-pack ──┐ + ├─ rit-odb ◄──┘ (odb is the facade over loose + pack) + ├─ rit-refs + ├─ rit-index + ├─ rit-diff + ├─ rit-revision (← odb, refs) + ├─ rit-merge (← diff, revision, odb) + ├─ rit-worktree (← odb, index, refs) + ├─ rit-fsck (← odb, refs, revision) + └─ rit-protocol (← odb, refs, pack) + └─ rit-transport (HTTP/SSH) +rit-repo (← odb, refs, index, worktree, revision, merge, protocol) +rit-porcelain (← rit-repo) +rit-cli (← rit-porcelain) rit-sdk (← curated re-exports) +rit-testkit / rit-fuzz (dev-only, depend across the graph) +``` + +**Boundary rules enforced by CI:** `rit-hash`/`rit-object` depend on nothing +Railyard-specific; nothing below `rit-repo` knows about a CLI; no crate depends on +a database or on `rithub`. `cargo-deny` guards the license/advisory surface (PLAN +"Verification strategy"). + +--- + +## 3. Phased build order (maps PLAN Release Sequence → crates) + +Each stage is gated: it does not start until the prior stage's named tests pass. + +| Stage | PLAN name | Crates delivered / extended | Gate to exit | +|------|-----------|------------------------------|--------------| +| **0** | Specification & harness | `rit-testkit`, `rit-fuzz`, `docs/{COMPATIBILITY,THREAT-MODEL,RECOVERY}.md`, `fixtures/` | Compat matrix + threat model frozen; differential harness green on the existing 4 commands | +| **1** | Read-only core | Split today's crate into `rit-hash` `rit-object` `rit-odb` `rit-refs` `rit-revision` `rit-repo` `rit-porcelain` `rit-cli`; add `rit-pack` (read), `rit-fsck` | `cat-file`/`ls-tree`/`log`/`fsck` on hostile repos never write outside the repo or use unbounded resources; pack **reads** interop-verified | +| **2** | Local authoring | `rit-index`, `rit-worktree`, `rit-diff`; extend `rit-repo`/`rit-cli` with init/status/add/restore/commit/branch/tag/reflog | Crash tests pass for every state-changing command; index round-trips vs real git | +| **3** | Exchange | `rit-protocol` (upload-pack + receive-pack), `rit-transport` (HTTP+SSH), `rit-pack` (write/thin), quarantine in `rit-odb` | clone/fetch/push interop with reference git client **and** server; atomic refs + quarantine + interrupted-transfer recovery proven | +| **4** | History editing | `rit-merge` (three-way, sequencer), rebase/cherry-pick/revert/conflict continue-abort in `rit-repo`/`rit-cli` | Merge/rebase match documented git behavior; user work preserved across induced failures | +| **5→6** | Forge / public | *No new `rit` crates* — this is where [`rithub`](../rithub) consumes `rit-sdk`. `rit`'s SHA-256 repository format (`rit-hash` second hash) lands here as an explicit, separately-gated matrix entry. | Security review; SHA-256 interop matrix; capacity + restore drills (owned by rithub) | + +**SHA-256** is intentionally *not* a late rewrite: `rit-hash` is generic from +Stage 1, so SHA-256 is a new hash implementation + a compat-matrix row, gated at +Stage 5/6 per PLAN ("explicit path to Git's SHA-256 repository format"). + +--- + +## 4. SDK / interop surface + +`rit-sdk` is the **only** crate external consumers (rithub, internal tooling) are +allowed to depend on. It is semver-stable and hides the churn of the inner crates. + +**Rust API contract (the shape rithub builds on):** + +```text +rit_sdk::Repository open / discover / init (today's Repository, widened) + .odb() -> Odb content-addressed object read/write, streaming + .refs() -> RefStore atomic CAS ref transactions, reflog, packed-refs + .index()/.worktree() staging + checkout (Stage 2+) + .revwalk(spec) bounded graph traversal / merge-base (rit-revision) + .diff(a, b)/.merge(...) (Stage 4) +rit_sdk::server upload_pack / receive_pack handlers over an + AsyncRead/AsyncWrite pair — the seam a forge mounts + behind smart-HTTP and SSH (Stage 3) +rit_sdk::ObjectId / Object / Signature canonical types, hash-generic +``` + +**Wire/tooling interop (beyond the Rust API):** + +- **Git wire protocol v2** — `rit-protocol` speaks the real thing, so *any* stock + `git` client can clone/fetch/push against a server embedding `rit-sdk::server`. + This is the interop guarantee that lets rithub host real repos for real users. +- **On-disk format** — the `.git` directory `rit` reads/writes is byte-compatible, + so a rithub-hosted repo is inspectable by stock `git` and vice-versa. +- **`rit` CLI** — a scriptable boundary for CI hooks and ops tooling. + +--- + +## 5. Composability with the Railyard platform + +`rit` is the **core dependency of `rithub`**: rithub adds no object storage, refs, +packfiles, protocol, or PR truth of its own — it path-depends on `rit-sdk` and +mounts `rit-sdk::server` behind its HTTP/SSH front doors (see +[`../rithub/ARCHITECTURE.md`](../rithub/ARCHITECTURE.md) §"How rithub depends on +rit"). + +Honest fit with the rest of the rail: + +- **`rit` is a developer/systems tool, not part of the AITP-402 payment rail.** + It has no `Mint` dependency and settles no money. That separation is deliberate + and correct — coupling repository correctness to a ledger would violate the PLAN + security contract. +- **Provenance touch-points (opt-in, additive):** `rit`'s content-addressed + object ids and signed-commit/tag verification (`rit-object` signatures) are a + natural anchor for provenance. A commit id can be referenced by a **Mint** + hash-linked receipt or a **context-passport** version record so "this artifact + was produced at commit ``" is verifiable — but that binding lives in the + *consumer* (Passport/Mint), never inside `rit`. `rit` stays a pure Git core. +- **AITimeMachine** may attach checkpoints to commit ids; again, `rit` exposes the + id, the consumer owns the mapping (PLAN "AITimeMachine … does not implement or + replace Git"). + +--- + +## 6. Graph engineering layer + +Git *is* a knowledge graph: the object database is a content-addressed DAG, and +refs are named roots over it. Modeling `rit` as a graph (per +`~/.claude/skills/graph-engineering/references/modeling.md`) makes the ontology +explicit and gives `rit-fsck`, `rit-revision`, and `rithub`'s code-intelligence a +shared vocabulary. + +### 6.1 Ontology (the object DAG) + +**Competency questions** (the schema's spec + test suite): +1. Which commits are reachable from ref `R`? +2. What is the merge-base of commits `A` and `B`? +3. Which blobs/trees does commit `C` introduce vs. its parents? +4. Who authored, and who committed, `C`, and when? +5. Which tag objects point (transitively) at commit `C`? +6. Is object `O` reachable from any ref (i.e. safe from GC)? + +**Entities** (canonical name = lowercase 40/64-hex object id; refs by full name): + +| Entity | Definition | Backing type | +|--------|-----------|--------------| +| `Blob` | file contents (a leaf) | `rit-object::Blob` | +| `Tree` | directory listing | `rit-object::Tree` | +| `Commit` | a history node | `rit-object::Commit` | +| `Tag` | annotated tag | `rit-object::Tag` | +| `Ref` | named mutable pointer (branch/tag/HEAD) | `rit-refs::RefTarget` | +| `Signature` | *attribute-bearing* author/committer identity (name, email, time, tz) | `rit-object::Signature` | + +**Relations** (precise verbs, `domain → range`, with time + provenance): + +| Relation | Domain → Range | Provenance / time carried | +|----------|----------------|---------------------------| +| `HAS_PARENT` | Commit → Commit | ordered (first-parent); defines history edges | +| `HAS_TREE` | Commit → Tree | root tree of the snapshot | +| `CONTAINS` | Tree → {Tree, Blob} | entry `name` + `mode` as edge attrs | +| `POINTS_AT` | Ref → {Commit, Tag} | via reflog: who moved it, old→new oid, time | +| `TAGS` | Tag → {Commit, Tree, Blob, Tag} | tagger `Signature` | +| `AUTHORED` / `COMMITTED` | Signature → Commit | **time + tz are the edge's timestamp** | + +Provenance is intrinsic here — it is not retrofitted metadata but the object bytes +themselves: author/committer signatures and reflog `who/old/new/when` lines *are* +the edge properties, and every id is a cryptographic content hash, so the graph is +tamper-evident by construction. This is exactly the "decide how every fact carries +time + provenance at modeling time" rule, satisfied for free by Git's design. + +`rit-revision` is the query engine over this graph (reachability, merge-base, +`a..b`), and `rit-fsck` is the consistency checker (every edge's range object must +exist and hash-verify — connectivity). The bounded visited-set walk already in +`Repository::log` is the template: no unbounded graph walks on adversarial input. + +### 6.2 Task graphs (fetch / push / merge) + +Exchange and history-editing operations are naturally task graphs (per +`task-graphs.md`), and modeling them that way is what lets `rit-protocol` and +`rit-odb` parallelize safely without corrupting data. + +**Fetch/clone — the diamond.** Negotiation produces a set of wanted objects that +never read each other's bytes → the "delete fake edges" win: object/pack transfer +fans out in parallel, then a mandatory **separate verify node** (quarantine) gates +the merge into the live odb: + +```text + ┌─ transfer pack slice 1 ─┐ +negotiate ────┼─ transfer pack slice 2 ─┼─→ verify (fsck + connectivity, ─→ install +(wants/haves) └─ transfer pack slice 3 ─┘ in QUARANTINE odb) refs (CAS) +``` + +The verify node is non-negotiable (a peer's pack is adversarial input): delta +chains resolved, every object hash-checked, connectivity proven — *before* any ref +moves. This is quarantine, straight from PLAN's security contract. + +**Push — diamond + human/policy gate.** Same parallel receive → quarantine verify, +but the ref-update edge is *irreversible on a shared repo*, so it routes through an +explicit gate: + +```text +receive-pack ─→ quarantine verify ─→ [POLICY/HUMAN GATE: protected-branch, + non-fast-forward, signed-commit, hooks] + ─→ atomic CAS ref transaction (owned merge) +``` + +`rit` exposes the gate as a hook/policy seam; **rithub** supplies the human side +(branch protection, required reviews — see its task-graph section). One owner of +the ref merge (the atomic CAS transaction in `rit-refs`) — never two writers, +satisfying the "one writer per artifact" guardrail. + +**Merge — bounded diamond.** merge-base (parallel ancestry walks from each tip) → +three-way content merge per path (parallelizable, independent files) → verify (no +conflict markers left / conflicts surfaced) → single sequencer owns the resulting +commit. The sequencer in `rit-merge` is the single merge-owner the stop rule +demands; every rebase/cherry-pick loop carries a max-round cap. + +--- + +## 7. Enterprise scale estimate + +**The current `rit` is ~14 source files. That is the loose-object read slice, NOT +a complete Git.** A complete Git implementation (packfiles + delta, index, +worktree, wire protocol v2, diff, merge/rebase, SHA-256) is *hundreds* of source +files and six figures of Rust. For calibration: libgit2 is ~100K LOC of C and +gitoxide (the mature Rust Git) spans ~30 crates and well over 200K lines. The +estimate below is **sizing, not empty scaffolding** — it exists so no one mistakes +today's slice for a finished product. + +| Crate | ~Files | ~LOC | Notes | +|-------|-------:|-----:|-------| +| rit-hash | 3 | 800 | SHA-1, SHA-256, sha1dc, generic id | +| rit-object | 12 | 4,000 | *(today's object/blob/tree/commit/tag lives here)* | +| rit-odb | 8 | 3,500 | *(today's odb.rs)* + packed facade, quarantine, alternates | +| rit-pack | 20 | 12,000 | packfile/.idx, delta resolution, bitmaps, pack writing | +| rit-refs | 10 | 4,500 | *(today's refs.rs)* + reflog, CAS transactions, locks | +| rit-index | 12 | 6,000 | index v2/3/4 + extensions | +| rit-worktree | 15 | 8,000 | checkout, status, ignore, fs-safety adapters | +| rit-revision | 10 | 5,000 | rev syntax, graph walks, merge-base | +| rit-diff | 12 | 7,000 | Myers/histogram, rename/copy detection | +| rit-merge | 14 | 9,000 | three-way, ort strategy, sequencer | +| rit-protocol | 18 | 11,000 | v2, pkt-line, upload-pack, receive-pack | +| rit-transport | 10 | 5,000 | HTTP + SSH adapters | +| rit-fsck | 6 | 2,500 | object/connectivity verification | +| rit-repo | 12 | 6,000 | *(today's repo.rs)* widened to full porcelain | +| rit-porcelain | 20 | 9,000 | command semantics (status/log/diff formatting) | +| rit-cli | 25 | 6,000 | *(today's main.rs)* → one module per subcommand | +| rit-sdk | 4 | 1,200 | stable public surface | +| rit-testkit | 10 | 4,000 | *(today's tests/interop.rs promoted)* + fixtures | +| rit-fuzz | 12 | 2,000 | cargo-fuzz targets per parser | +| **Total (src)** | **~233** | **~106,500** | +tests/fixtures → **~300–400 files, ~150K LOC** | + +**What "done" means per phase:** Stage 1 (read-only core) ≈ 60–80 files. Stage 2 +(local authoring) ≈ +80 files. Stage 3 (exchange) ≈ +70 files. Stage 4 (history +editing) ≈ +50 files. The **current 14-file slice is ~5% of a complete `rit`** and +corresponds to a *portion* of Stage 1. Declaring `rit` "done" at the loose-object +slice would be declaring Git done at `cat-file`. + +--- + +## 8. Key decisions + +1. **Workspace, not one crate** — decompose along PLAN's crate lines so the + deferred scope (pack/index/worktree/protocol/merge) lands as isolated, + independently-testable, independently-fuzzed units. +2. **`rit-hash` generic from day one** — SHA-256 becomes a hash impl + a matrix + row, never a rewrite. +3. **`rit-sdk` as the single stable seam** — rithub and tooling depend only on + the SDK, insulating them from inner-crate churn. +4. **Compat gate is a release blocker, not a wish** — `rit-testkit` + + `rit-fuzz` + fixtures are Stage 0, and the forge cannot begin until the core + gate is green. +5. **No payment-rail coupling** — `rit` stays a systems tool; provenance ties are + additive and owned by consumers (Mint/Passport), preserving the security + contract.