This repository has been archived on 2026-11-17. You can view files and clone it, but cannot push or open issues or pull requests.
gitgram-ai/docs/ARCHITECTURE.md
caelum0x 49b183818c Unify gitgram + aitimemachine into the monorepo (de-submodule) and merge vendored engines
Convert gitgram/ and aitimemachine/ from embedded git repos (which showed as
opaque submodule gitlinks on GitHub) into fully-tracked first-party content.

AITM now merges its 7 vendored repos as first-party engines:
- memory: beads (Go task DAG) + context7 (docs SDK), unified into saved_contexts
- git: vendored C git (bundled binary) + gitnexus (in-process code intelligence)
- agentic: a2a (@a2a-js/sdk) + pal (AITM-managed Python subprocess)

Includes the full session's product build: Gitgram web/mobile/backend features,
AITM website/CLI/MCP, the unified memory layer, engine build tooling
(scripts/build-engines.sh, Dockerfile, CI), docs (README, ENGINES.md, ARCHITECTURE.md),
and extensive test suites.

Build artifacts, node_modules, external/, and gemini-cli are gitignored.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 14:14:09 +03:00

236 lines
13 KiB
Markdown

# Architecture
This document goes deeper than the front-door [README](../README.md): it covers the
end-to-end data flow, the unified memory layer, the git engine and checkpoint
mirroring, the agentic communication path (a2a registry + pal subprocess), and how
each MCP tool maps to an engine.
Two products live in one repo:
- **Gitgram** — a repository-first social network (the "new GitHub"). Every repo is a
social profile with feeds, posts, and follows.
- **AITimeMachine (AITM)** — an AI-native version-control substrate (the "new Git")
with a unified memory layer, real git underneath, code intelligence, and agent
orchestration.
Gitgram is a product *on top of* AITM. AITM is the substrate that makes AI agents
making dozens of PRs per day deterministic and recoverable.
---
## 1. Layered Topology
```
Clients
┌─────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────────────┐
│ Gitgram web │ │ Gitgram │ │ aitm CLI │ │ MCP clients │
│ (Next.js) │ │ mobile (Expo)│ │ │ │ (Claude, IDEs) │
└──────┬──────┘ └──────┬───────┘ └────┬─────┘ └────────┬─────────┘
│ GraphQL/tRPC/WS │ │ HTTP / MCP │
└────────┬────────┘ └────────┬────────┘
▼ ▼
┌────────────────────┐ ┌────────────────────────┐
│ Gitgram backend │◀──bridge──▶│ AITM api-server │
│ Express 5 / Prisma│ │ Express / Drizzle │
└─────────┬──────────┘ └───────────┬────────────┘
▼ ▼
┌───────────────────┐ ┌──────────────────────┐
│ Postgres + Redis │ │ engineManager.ts │
└───────────────────┘ └──────────┬───────────┘
▼
┌──────── 7 first-party engines ────────┐
│ memory : beads · context7 │
│ git : git (C) · gitnexus │
│ agentic: a2a · pal │
│ (+ gitnexus-shared support lib) │
└────────────────────────────────────────┘
```
- **Gitgram backend** (`gitgram/backend`) is an Express 5 app exposing GraphQL
(`@apollo/server`), tRPC (`@trpc/server`), and Socket.IO WebSockets over Prisma +
Postgres, with Redis for caching/queues. Controllers, jobs, models, and services
are split by domain under `src/`.
- **AITM api-server** (`aitimemachine/api-server`) is the HTTP surface over the
engines, using Drizzle ORM (Neon / node-postgres dual driver). `engineManager.ts`
resolves bundled engine paths (`bin/bd`, `bin/git`, `engines/pal`) with first-party
defaults — no external URLs.
- The two products talk through a **bridge**: `services/gitgramBridgeService.ts` plus
`routes/gitgram.ts` and `routes/gitgramWebhooks.ts`, so Gitgram repo events can
drive AITM memory/checkpoints and AITM can surface intelligence back to Gitgram.
---
## 2. Data Flow
### Gitgram request path
1. Web (`Next.js`) or mobile (`Expo`/`react-native`) issues a GraphQL/tRPC call or
opens a Socket.IO connection.
2. Express middleware authenticates and authorizes the request.
3. A domain controller/service reads or writes Postgres through Prisma; hot paths and
pub/sub use Redis.
4. WebSocket events (`socket.io`) push feed updates to connected clients.
5. Repo-level events relevant to version control are forwarded to AITM via the bridge.
### AITM request path
1. A client (`aitm` CLI, an MCP client, or the Gitgram bridge) hits the api-server or
speaks MCP over stdio/HTTP.
2. The route handler validates input, then calls a service.
3. The service resolves the relevant engine through `engineManager.ts` and either
calls it in-process (context7, gitnexus, a2a) or spawns/shells out to a bundled
binary or subprocess (beads, git, pal).
4. Side effects (tasks, docs, checkpoints) write through to the unified memory layer.
5. `GET /v1/health/providers` reports each engine's availability.
---
## 3. The Unified Memory Layer
AITM's defining feature is a single searchable memory that fuses three sources into
the `saved_contexts` table with vector embeddings.
```
beads tasks ──► source='task' ─┐
context7 docs ─► source='library' ─┼─► memoryService.ts ─► saved_contexts
conversations ─► source='memory' ─┘ (embeddings via embeddings.ts)
│
▼
GET /v1/contexts/search/*
```
- **`services/memoryService.ts`** ingests beads tasks and context7 docs, tagging each
row with `saved_contexts.source` (added in migration
`0005_add_saved_context_source.sql`).
- **`services/embeddings.ts`** produces the vectors so semantic search spans all three
sources at once.
- A search for "how does retention work" can therefore return a beads task, a
context7 doc snippet, and a prior conversation memory in one ranked result set.
- When a request carries no user/project, write-through targets
`AITM_MEMORY_USER_ID` / `AITM_MEMORY_PROJECT_ID`; if those are unset, write-through
is skipped (reads still work).
This is what lets agents accumulate durable, queryable context across sessions instead
of re-deriving state every run.
---
## 4. The Git Engine + Checkpoint Mirroring
AITM uses the **real upstream Git** (vendored C source in `aitimemachine/git/`), not a
reimplementation. The Rust `gitcore` crate present in the tree is a non-functional stub
and is **not** used.
```
aitm checkpoint / before_action
│
▼
git-engine.ts ──► bin/git (or system git fallback)
│
├─► AITM session/checkpoint store (Drizzle)
│
└─► AITM_GIT_MIRROR=on ─► real git commit ◄── deterministic restore
```
- **`src/core/git-engine.ts`** shells out to `bin/git` (built by
`build-engines.sh`), falling back to the system `git` if the bundled binary is
absent. `AITM_GIT_BIN` overrides the path; `aitm git …` is a passthrough.
- **Checkpoint mirroring**: every AITM checkpoint is mirrored to a real git commit
when `AITM_GIT_MIRROR` is enabled (the default). This gives deterministic,
inspectable restore points backed by ordinary git objects — restores are exact, not
approximate.
- **`gitnexus`** (with `gitnexus-shared`) layers a code-intelligence graph over the
repo using Tree-sitter parsing and LadybugDB. It runs in-process via
`gitnexus/src/index.ts` → `LocalBackend.callTool`; `services/gitNexusService.ts`
calls it directly. This powers impact analysis, call/route tracing, and structural
queries.
---
## 5. Agentic Communication (a2a registry + pal subprocess)
The agentic layer is how AITM coordinates multiple agents and runs their code.
```
┌──────────── a2a (in-process) ────────────┐ ┌──── pal (subprocess) ────┐
│ @a2a-js/sdk │ │ pal-mcp-server (Python) │
│ DefaultRequestHandler │ │ bundled venv (engines/ │
│ InMemoryTaskStore ◄── registry │ │ pal), MCP over stdio │
│ A2AClient │ │ via palEngine.ts │
└───────────────────────────────────────────┘ └──────────────────────────┘
agent ↔ agent messaging/tasks code execution / multi-model
```
- **a2a (agent-to-agent protocol)** is consolidated onto `@a2a-js/sdk`. The
hand-rolled `a2aService` is replaced; the agent **registry is backed by the SDK's
`InMemoryTaskStore`**, with `DefaultRequestHandler` serving requests and `A2AClient`
for outbound calls. This standardizes how agents discover each other and exchange
tasks/messages. The `AITM_A2A_AGENTS` external seam was removed.
- **pal (Python agent execution)** runs as an **AITM-managed subprocess**.
`api-server/src/engines/palEngine.ts` spawns the bundled venv
(`engines/pal`, override via `PAL_VENV_DIR` / `PAL_PYTHON`) and speaks MCP over
stdio. It handles sandboxed code execution, notebooks, package installs, and
multi-model orchestration. `pal_*` MCP tools route through it.
Together: a2a decides *who does what*, pal *runs the code*, and the memory layer
*remembers the outcome*.
---
## 6. MCP Tools → Engines
The MCP server (`aitimemachine/mcp`) registers tools in groups under
`src/tools/register-*.ts`. Each group maps to one engine or to AITM core.
| Tool group / prefix | Backing engine | Representative tools |
|---------------------|----------------|----------------------|
| `register-core-tools` / `register-session-tools` | AITM core (git engine + memory) | `before_action`, `after_action`, `create_checkpoint`, `restore_session`, `list_sessions` |
| `register-beads-tools` | **beads** (memory) | task/issue DAG create, list, link |
| `register-context7-tools` | **context7** (memory) | `context7` library-doc lookup |
| `register-gitnexus-tools` / `register-analysis-tools` | **gitnexus** (git) | `gn_query`, `gn_impact`, `gn_trace`, `gn_route_map`, `gn_explain`, `gn_cypher`, `gn_pdg_query`, `gn_api_impact`, `gn_rename`, `gn_check` |
| `register-a2a-tools` | **a2a** (agentic) | agent register, send-task, list-agents |
| `register-pal-tools` | **pal** (agentic) | `pal_execute`, `pal_analyze_code`, `pal_create_session`, `pal_install_package`, `pal_execute_notebook`, `pal_convert_notebook` |
| `register-file-tools` / `register-workspace-tools` / `register-worktree-tools` | AITM core (git engine) | file scan, workspace, worktree ops |
| `register-github-tools` / `register-integration-tools` / `register-storage-tools` / `register-claude-tools` | AITM core / integrations | GitHub, storage, and Claude-side glue |
Resolution flow for any tool call:
```
MCP client → mcp/src/tools/register-<group>.ts → AITM service
→ engineManager.ts resolves engine
→ in-process call (context7/gitnexus/a2a) OR spawn (beads/git/pal)
→ optional memory write-through (memoryService.ts)
```
Core safety tools (`before_action` / `after_action` / `create_checkpoint`) wrap any
agent action in a checkpoint, so the git engine's mirroring guarantees the action is
recoverable — the foundation for letting agents make many PRs per day safely.
---
## 7. Health, Path Resolution, and First-Party Guarantees
- **`services/engineManager.ts`** is the single place that resolves engine locations,
always preferring bundled, first-party defaults (`bin/bd`, `bin/git`,
`engines/pal`).
- **`GET /v1/health/providers`** reports each engine independently:
gitnexus → `gitNexusService.isAvailable()`; context7 → `CONTEXT7_API_KEY` presence;
pal → `palEngine.health()`; beads → `bin/bd --version`; git → bundled-or-system.
- Removed external seams: `GITNEXUS_URL`, `CONTEXT7_URL`, `PAL_URL`, `PAL_MCP_URL`,
`AITM_A2A_AGENTS`. Engines are part of the product, not assumed services.
See [aitimemachine/ENGINES.md](../aitimemachine/ENGINES.md) for the per-engine build
wiring and the full environment-variable table, and
[environment-matrix.md](environment-matrix.md) for per-environment values.
---
## 8. Build & Migration Order (why it matters)
The TS engine libraries (`gitnexus-shared`, `context7` SDK, `a2a`, `gitnexus`) must be
built **before** `api-server`/`mcp` can resolve their imports —
`build-engines.sh` enforces this order, then compiles the native engines
(`beads`, `git`) and provisions the `pal` venv, then links workspace deps. Database
migrations run independently per product: Prisma for Gitgram, Drizzle for AITM, both
driven by `scripts/dev.sh`.