rit/PLAN.md

223 lines
9.8 KiB
Markdown
Raw Permalink Normal View History

# Rust Git Forge Product and Architecture Plan
Last revised: 23 August 2026
Tracking issue: `videogen-zyo`
## Decision record
This is a production open-source systems product, not GitGram and not a
hackathon implementation. The neutral working name is **Rust Git Forge** until
brand, crate, and package-name checks are complete.
The product has two layers:
1. A library-first Rust implementation of Git repositories and the Git wire
protocol, compatible with existing Git repositories and clients.
2. A self-hostable collaboration forge built on that implementation, including
repositories, organizations, permissions, issues, pull requests, review,
protected branches, webhooks, releases, and audit logs.
GitGram remains a mostly TypeScript social product. It turns repositories into
Instagram-like accounts with posts, reels, stories, media publishing,
followers, and feeds. GitGram does not own object storage, refs, packfiles, the
Git protocol, pull-request truth, or repository permissions. It integrates with
GitHub and, later, Rust Git Forge through public authenticated APIs and webhooks.
AITimeMachine captures and restores AI coding workflows. It may attach its
checkpoints to Git commits, but it does not implement or replace Git.
## Initial customer and promise
The initial user is an engineer or small team that needs a trustworthy local
Git implementation with readable internals and strong compatibility evidence.
The first production promise is narrower than a complete GitHub replacement:
users can inspect, create, verify, exchange, and recover standard Git
repositories without corrupting data.
The forge becomes a supported product only after the core passes
interoperability, crash-recovery, parser-fuzzing, and concurrent-ref tests.
## Compatibility scope
The core implementation must cover:
- loose blobs, trees, commits, annotated tags, object verification, and object
traversal;
- SHA-1 repository compatibility with collision detection, plus an explicit
path to Git's SHA-256 repository format;
- refs, symbolic refs, packed refs, reflogs, atomic compare-and-swap updates,
and lock recovery;
- index formats, worktrees, ignore rules, status, add, restore, and commit;
- revision parsing, graph walks, merge bases, diffs, rename heuristics, and
conflict representation;
- pack indexes, packfiles, delta resolution, reachability, negotiation, and
quarantine of received objects;
- smart protocol v2 over HTTP and SSH for clone, fetch, and push;
- three-way merge, rebase, cherry-pick, revert, and deterministic conflict
handling.
Submodules, sparse checkout, partial clone, large-file storage, replace refs,
notes, and advanced maintenance enter the compatibility matrix explicitly. None
is silently claimed before its test suite passes.
## Rust workspace architecture
```text
crates/
git-object object IDs, parsing, validation, canonical serialization
git-odb loose and packed object databases, quarantine, alternates
git-refs refs, reflogs, transactions, locks
git-index index formats and extensions
git-worktree checkout, status, ignore rules, filesystem safety
git-revision revision syntax, graph traversal, reachability
git-diff text and binary diff, rename detection
git-merge merge bases, strategies, conflicts, sequencer
git-pack pack/index parsing, deltas, negotiation inputs
git-protocol protocol v2, upload-pack, receive-pack transports
git-cli compatibility-oriented command-line interface
forge-domain users, organizations, repositories, issues, reviews
forge-store metadata persistence and repository coordination
forge-http browser/API and smart-HTTP server
forge-ssh SSH command gateway and key authorization
forge-jobs durable maintenance, hooks, webhooks, notifications
```
Shared crates cannot depend on the forge UI or a specific database. Parsing and
storage APIs expose bounded streams rather than assuming objects or packs fit in
memory. Platform-specific filesystem behavior stays behind explicit adapters
and platform tests.
## Forge architecture
The first forge is a single-node, self-hosted service:
- repositories and packfiles on a durable local filesystem;
- PostgreSQL for users, organizations, permissions, issues, reviews, jobs, and
immutable audit events;
- an SSH service for Git transport and an HTTP service for API, browser, and
smart HTTP;
- a transactional job outbox for webhooks, email, repository maintenance, and
indexing;
- filesystem snapshots plus PostgreSQL backups with a tested coordinated
restore procedure.
SQLite may support desktop or single-user mode, but it does not imply safe
multi-node repository coordination. Multi-node storage, geo replication,
packages, runners, Actions-compatible CI, and large-file storage require later
designs with separate threat and cost reviews.
Cloudflare Workers are not the Git execution or storage plane. Large pack
streams, SSH, repository filesystem semantics, and long-running maintenance
need a local process or conventional server. Cloudflare may serve static project
documentation and, after measurement, proxy ordinary browser/API traffic. R2 is
only a future cold-object or backup option under the shared account budget.
## Security contract
Repository data is adversarial input. The product must defend against:
- malformed object, index, pack, bitmap, commit-graph, and protocol data;
- path traversal, case-folding collisions, NTFS/HFS hazards, unsafe symlinks,
and checkout outside the worktree;
- decompression bombs, delta chains, integer overflow, unbounded graph walks,
resource exhaustion, and archive bombs;
- SHA-1 collision attacks and object-ID confusion between hash formats;
- ref races, lock theft, non-fast-forward policy bypass, hook injection, and
partially received pushes;
- cross-tenant authorization errors, leaked deploy keys or tokens, forged
webhooks, and unsafe HTML or Markdown rendering.
Incoming pushes remain quarantined until object verification, connectivity,
authorization, policy, and size checks pass. Ref changes use atomic
transactions. The forge supports scoped tokens, SSH key fingerprints,
phishing-resistant operator MFA, protected branches, signed commit and tag
verification, secret redaction, security logs, and auditable administrative
actions.
## Verification strategy
Production claims require retained evidence from:
- differential tests that run equivalent operations through Rust Git Forge and
the reference Git CLI, then compare repository state and command semantics;
- interoperability matrices across supported Git versions, operating systems,
filesystem behaviors, HTTP servers, and SSH clients;
- property tests for serialization, graph traversal, deltas, refs, and index
round trips;
- coverage-guided fuzzing for every untrusted parser and protocol state machine;
- Git project fixtures where licensing and harness boundaries permit;
- fault injection around writes, fsync, locks, process death, partial network
reads, interrupted clone/fetch/push, and coordinated restore;
- concurrency tests for ref updates, simultaneous pushes, garbage collection,
and read/write isolation;
- independent security review before exposing public repository hosting.
Every compatibility claim is tied to a named test. A failing repository fixture
remains a release blocker or a documented unsupported capability.
## Release sequence
### Stage 0: specification and harness
Publish the compatibility matrix, threat model, repository fixtures,
differential harness, fuzz targets, performance budgets, and
corruption-recovery rules.
### Stage 1: read-only repository core
Ship object inspection, `cat-file`, `ls-tree`, log traversal, refs, pack reads,
and `fsck`. Opening hostile repositories cannot write outside the repository or
consume unbounded resources.
### Stage 2: local authoring
Ship init, status, add, diff, commit, branch, checkout/restore, tag, reflog, and
garbage-safe object writes. Pass crash tests for every state-changing command.
### Stage 3: exchange
Ship protocol v2 clone, fetch, and push over HTTP and SSH. Prove quarantine,
atomic refs, authentication, authorization, interrupted transfer recovery, and
interop with the reference Git client and server.
### Stage 4: history editing
Ship merge, rebase, cherry-pick, revert, conflict continuation/abort, and
recovery. Preserve user work during failures and match documented Git behavior.
### Stage 5: single-node forge
Ship accounts, organizations, repository permissions, issues, pull requests,
review, branch protection, webhooks, releases, operator tools, backups, restore,
export, deletion, abuse handling, and upgrade/rollback procedures.
### Stage 6: public production
Onboard outside repositories only after security review, capacity tests,
restore drills, support procedures, privacy/terms, observability, incident
response, and an upgrade from the previous release all pass. Multi-node and CI
features remain separate approvals.
## Non-goals for the first public release
- replacing every Git command at once;
- claiming GitHub parity;
- running arbitrary CI jobs;
- hosting unbounded public binary releases or package registries;
- coupling repository correctness to GitGram feeds;
- storing primary repositories in browser storage, R2, or a Worker runtime;
- inventing a new incompatible wire protocol before Git interoperability works.
## Pre-code gate
Before implementation:
1. approve the project name, license, governance, supported platforms, and MSRV;
2. freeze the Stage 1 compatibility matrix and threat model;
3. select reference Git versions and reusable fixtures after license review;
4. decide whether existing Rust Git crates are comparison subjects,
dependencies, or clean-room references;
5. define benchmark repositories and memory, CPU, disk, and network budgets;
6. approve the single-node data and backup contract without using Cloudflare as
the repository execution plane.