Convert gitgram/ and aitimemachine/ from embedded git repos (which showed as opaque submodule gitlinks on GitHub) into fully-tracked first-party content. AITM now merges its 7 vendored repos as first-party engines: - memory: beads (Go task DAG) + context7 (docs SDK), unified into saved_contexts - git: vendored C git (bundled binary) + gitnexus (in-process code intelligence) - agentic: a2a (@a2a-js/sdk) + pal (AITM-managed Python subprocess) Includes the full session's product build: Gitgram web/mobile/backend features, AITM website/CLI/MCP, the unified memory layer, engine build tooling (scripts/build-engines.sh, Dockerfile, CI), docs (README, ENGINES.md, ARCHITECTURE.md), and extensive test suites. Build artifacts, node_modules, external/, and gemini-cli are gitignored. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
22 KiB
AITimeMachine Data Storage Architecture
This document explains how AITimeMachine stores code snapshots, chat history, and related data from AI coding tools.
Canonical reference: ../../NORTH_STAR_PRD_ARCHITECTURE_V1.md.
Overview
AITimeMachine uses a backend-canonical storage architecture:
- Canonical Backend Storage (Postgres + object storage; source of truth)
- Project-Level Local Cache (
.aitimemachine/folder in each project) - Global Local Cache (
~/.aitimemachine-server/for fast lookup/offline buffering)
┌─────────────────────────────────────────────────────────────────────────┐
│ AITimeMachine Storage Architecture │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────┐ ┌──────────────────────────────────────┐ │
│ │ AI Coding Tools │ │ Global Daemon │ │
│ │ ┌───────────────┐ │ │ ┌──────────────────────────────┐ │ │
│ │ │ Claude Code │──┼────▶│ │ Auto-detect projects from: │ │ │
│ │ │ Cursor │ │ │ │ • Claude, Cursor, Aider │ │ │
│ │ │ Aider │ │ │ │ • Windsurf, Copilot, Zed │ │ │
│ │ │ Windsurf │ │ │ │ • Active terminals │ │ │
│ │ │ VS Code │ │ │ │ • Recent git repos │ │ │
│ │ │ Copilot │ │ │ └──────────────────────────────┘ │ │
│ │ └───────────────┘ │ │ │ │ │
│ └─────────────────────┘ │ ▼ │ │
│ │ ┌──────────────────────────────┐ │ │
│ │ │ Periodic Capture (5 min) │ │ │
│ │ │ • Incremental file scanning │ │ │
│ │ │ • Content-addressable store │ │ │
│ │ │ • Delta compression │ │ │
│ │ └──────────────────────────────┘ │ │
│ └──────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ Project-Level Storage │ │
│ │ .aitimemachine/ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │
│ │ │ store/ │ │ timepoints/ │ │ content/ │ │ events. │ │ │
│ │ │ (blobs by │ │ (JSON │ │ (messages, │ │ jsonl │ │ │
│ │ │ hash) │ │ snapshots) │ │ code, diffs)│ │ │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ Cloud Storage (Neon) │ │
│ │ postgres://...neon.tech │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │
│ │ │ projects │ │ timepoints │ │ messages │ │ blobs │ │ │
│ │ │ (registry) │ │ (snapshots) │ │ (full-text │ │ (content) │ │ │
│ │ │ │ │ │ │ search) │ │ │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │
│ │ Features: SQL queries, cross-device sync, Neon branching │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
1. Project-Level Storage (.aitimemachine/)
Each project has a .aitimemachine/ directory containing:
1.1 Content Store (store/)
Git-like content-addressable storage for all file content.
.aitimemachine/
└── store/
├── 7a/
│ └── 7a343253936975ff77526dbbc56ff539 # SHA-256 hash (first 32 chars)
├── 85/
│ └── 85a953d052d9cf2504d1adafc463c349
└── ...
Key Features:
- SHA-256 hashing (truncated to 32 chars)
- Deduplication: Same content stored once regardless of filename
- Two-level directory: First 2 chars of hash = subdirectory (prevents filesystem limitations)
Implementation (src/core/content-store.ts):
put(content: string | Buffer): string {
const buffer = typeof content === 'string' ? Buffer.from(content, 'utf-8') : content;
const hash = crypto.createHash('sha256').update(buffer).digest('hex').slice(0, 32);
const filePath = this.getPath(hash); // e.g., store/7a/7a343...
if (!fs.existsSync(filePath)) {
fs.writeFileSync(filePath, buffer);
}
return hash;
}
1.2 TimePoints (timepoints/)
Complete project snapshots stored as JSON files.
.aitimemachine/
└── timepoints/
├── tp_1703123456789_abc123.json
├── tp_1703123567890_def456.json
└── ...
TimePoint Structure:
interface TimePoint {
id: string; // e.g., "tp_1703123456789_abc123"
timestamp: Date; // When captured
description: string; // User or auto-generated description
trigger: {
type: 'manual' | 'auto' | 'ai_action' | 'restore';
tool?: string; // "claude", "cursor", etc.
action?: string; // "create_file", "edit_file", etc.
};
linkedCheckpointId?: string; // Links to conversation checkpoint
conversation: {
sessionId: string;
messageCount: number;
lastMessagePreview: string;
};
files: FileEntry[]; // All files at this point
changes: {
added: string[]; // New files since last timepoint
modified: string[]; // Changed files
deleted: string[]; // Removed files
};
stats: {
totalFiles: number;
totalSize: number;
};
}
interface FileEntry {
path: string; // Relative path
hash: string; // Reference to content in store/
size: number;
mode: number;
mtime: number;
}
1.3 Event Store (events.jsonl)
Append-only log of all events (JSONL format for streaming writes).
.aitimemachine/
└── events.jsonl
Event Types:
type EventType =
| 'user_message' // User prompt to AI
| 'assistant_message' // AI response
| 'tool_call' // Tool invocation (create_file, edit, etc.)
| 'tool_result' // Tool execution result
| 'checkpoint_created' // Explicit checkpoint
| 'code_change' // File modification event
| 'file_created'
| 'file_modified'
| 'file_deleted'
| 'tool_error'
| 'terminal_command'
| 'error_occurred';
interface Event {
id: string;
timestamp: Date;
sessionId: string;
type: EventType;
data: any; // Type-specific payload
}
Example Events:
{"id":"event_1703123456_abcd","timestamp":"2024-12-21T10:30:00.000Z","sessionId":"sess_123","type":"user_message","data":{"content":"Create a REST API","fullSize":20}}
{"id":"event_1703123457_efgh","timestamp":"2024-12-21T10:30:01.000Z","sessionId":"sess_123","type":"tool_call","data":{"tool":"create_file","args":{"path":"src/api.ts"}}}
{"id":"event_1703123458_ijkl","timestamp":"2024-12-21T10:30:02.000Z","sessionId":"sess_123","type":"file_created","data":{"filePath":"src/api.ts","newSize":1024,"diffRef":"content/diffs/diff_1703123458.patch"}}
1.4 Content Storage (content/)
Large content stored separately from event log (referenced by pointer).
.aitimemachine/
└── content/
├── messages/ # Long AI messages
│ └── msg_1703123456_abc.txt
├── code/ # Code snapshots
│ └── code_1703123457_def.ts
├── diffs/ # Generated diffs
│ └── diff_1703123458_ghi.patch
└── outputs/ # Tool outputs
└── out_1703123459_jkl.txt
Why Separate Storage?
- Event log stays small and fast to parse
- Large content only loaded on demand
- Easy garbage collection
1.5 Pack Files (packs/)
Delta-compressed archives for space efficiency.
.aitimemachine/
└── packs/
├── pack_1703123456.pack
└── pack_1703123456.idx
Pack Format:
- Similar to Git pack files
- Delta compression between similar objects
- Index file for O(1) hash lookup
2. Global Server Storage (~/.aitimemachine-server/)
Centralized storage for cross-project management.
2.1 Sessions (sessions/)
Complete project snapshots with full metadata.
interface Session {
id: string;
projectPath: string;
projectName: string;
createdAt: Date;
updatedAt: Date;
trigger: {
type: 'auto' | 'manual' | 'ai_action' | 'restore';
tool?: string;
action?: string;
description: string;
};
files: SessionFile[];
diffs: SessionDiff[];
conversation: SessionMessage[];
stats: {
totalFiles: number;
totalSize: number;
addedFiles: number;
modifiedFiles: number;
deletedFiles: number;
messageCount: number;
};
git?: GitState;
environment?: EnvironmentState;
}
2.2 Shared Blobs (blobs/)
Deduplicated content shared across all projects.
2.3 Projects Index (projects.json)
Project registry with metadata.
interface Project {
id: string;
path: string;
name: string;
createdAt: Date;
lastSessionAt: Date;
sessionCount: number;
totalSize: number;
}
2.4 Conversations (conversations/)
Conversation logs per session.
3. Conversation Logging
The ConversationLogger class orchestrates conversation capture:
class ConversationLogger {
private eventStore: EventStore;
private contentStorage: ContentStorage;
private sessionId: string;
// Log user message
logUserMessage(content: string): void {
const contentRef = content.length > 500
? this.contentStorage.storeMessage(content)
: undefined;
this.eventStore.append({
sessionId: this.sessionId,
type: 'user_message',
data: {
content: content.slice(0, 500), // Preview
contentRef, // Full content reference
fullSize: content.length
}
});
}
// Log AI response
logAssistantMessage(content: string, thinking?: string): void {
// Similar pattern - store large content separately
}
// Log tool calls and results
logToolCall(tool: string, args: any): string;
logToolResult(callId: string, result: any): void;
// Log code changes with automatic diff generation
logCodeChange(filePath: string, oldContent: string | null, newContent: string | null): void {
const changeType = !oldContent ? 'file_created'
: !newContent ? 'file_deleted'
: 'file_modified';
// Store full content
const oldContentRef = oldContent ? this.contentStorage.storeCode(oldContent) : undefined;
const newContentRef = newContent ? this.contentStorage.storeCode(newContent) : undefined;
// Generate and store diff
if (oldContent && newContent) {
const generatedDiff = diff.createPatch(filePath, oldContent, newContent);
const diffRef = this.contentStorage.storeDiff(generatedDiff);
}
this.eventStore.append({
sessionId: this.sessionId,
type: changeType,
data: {
filePath,
oldContentRef,
newContentRef,
diffRef,
oldSize: oldContent?.length || 0,
newSize: newContent?.length || 0,
oldSnippet: oldContent?.slice(0, 500),
newSnippet: newContent?.slice(0, 500),
}
});
}
}
4. AI Tool Detection & Capture
The GlobalDaemon automatically discovers and captures from multiple AI tools:
4.1 Supported Tools
| Tool | Detection Method | Data Source |
|---|---|---|
| Claude Code | ~/.claude/projects/ |
Encoded project paths |
| Cursor | ~/Library/.../Cursor/workspaceStorage/ |
workspace.json files |
| Aider | ~/.aider/ + .aider.chat.history.md |
Chat history files |
| Windsurf | ~/.windsurf/ or ~/.codeium/ |
Workspace storage |
| VS Code | ~/Library/.../Code/workspaceStorage/ |
workspace.json files |
| Copilot | (via VS Code) | Same as VS Code |
| Zed | ~/Library/.../Zed/workspaces/ |
JSON workspace files |
4.2 Active Detection
// Scan active terminals using lsof
scanActiveTerminals(): void {
const output = execSync(
`lsof -c node -c python -c ruby -c cargo -c go 2>/dev/null | grep -E "cwd|txt"`
);
// Find project roots (package.json, Cargo.toml, etc.)
}
// Scan git repos with recent activity
scanRecentGitRepos(): void {
// Check ~/Projects, ~/Developer, ~/dev, ~/code, ~/src
// Look for .git directories with HEAD modified in last 24h
}
4.3 Capture Flow
┌─────────────┐ ┌───────────────┐ ┌────────────────┐
│ AI Tool │────▶│ Global Daemon │────▶│ FastSession- │
│ Activity │ │ (detects) │ │ Capture │
└─────────────┘ └───────────────┘ └────────────────┘
│
┌─────────────────────────────┘
▼
┌───────────────────────────────────────────────────┐
│ 1. Incremental file scan (mtime-based) │
│ 2. Hash only changed files │
│ 3. Store new content to blobs │
│ 4. Generate diffs from previous session │
│ 5. Save session with conversation context │
│ 6. Optional: Database & container state capture │
└───────────────────────────────────────────────────┘
5. Storage Optimization
5.1 Incremental Scanning
// Only hash files that changed (by mtime)
for (const filePath of files) {
const prev = previousFiles.get(relativePath);
if (prev && prev.mtime === info.mtime && prev.size === info.size) {
// File unchanged - reuse hash without reading content
hash = prev.hash;
} else {
// File changed - read and store
const content = this.fileScanner.readFile(filePath);
hash = this.contentStore.put(content);
}
}
5.2 Content Deduplication
Same file content across different files or timepoints is stored once:
File A: "hello world" → hash: abc123
File B: "hello world" → hash: abc123 (same blob reused)
5.3 Delta Compression
Pack files use delta encoding for similar files:
class DeltaCompressor {
compress(source: Buffer, target: Buffer): Buffer {
// Find common prefix, sliding window matching, etc.
return encodeDelta(instructions);
}
}
5.4 Garbage Collection
tm gc # Clean unreferenced blobs, pack loose objects
6. Data Access Patterns
6.1 Retrieve File at TimePoint
engine.getFileAt(timepointId: string, filePath: string): string | null {
const tp = this.timePointManager.load(timepointId);
const fileEntry = tp.files.find(f => f.path === filePath);
return this.contentStore.getString(fileEntry.hash);
}
6.2 Get Conversation Context
conversationLogger.getContextForCheckpoint(checkpointId: string): ConversationContext {
const events = this.eventStore.getEventsUntil(checkpointId);
return {
messages: events.filter(e => e.type === 'user_message' || e.type === 'assistant_message'),
toolCalls: events.filter(e => e.type === 'tool_call'),
codeChanges: events.filter(e => ['file_created', 'file_modified', 'file_deleted'].includes(e.type)),
checkpoints: events.filter(e => e.type === 'checkpoint_created')
};
}
6.3 Timeline Query
// Get timepoints in range
timePointManager.getInRange(start: Date, end: Date): TimePoint[];
// Get timepoints by trigger
timePointManager.getByTrigger('ai_action'): TimePoint[];
7. File Paths Summary
| Path | Purpose |
|---|---|
.aitimemachine/store/ |
Content-addressable blob storage |
.aitimemachine/timepoints/ |
JSON snapshot files |
.aitimemachine/events.jsonl |
Append-only event log |
.aitimemachine/content/ |
Large content (messages, code, diffs) |
.aitimemachine/packs/ |
Delta-compressed archives |
.aitimemachine/config.json |
Project configuration |
~/.aitimemachine-server/sessions/ |
Global session storage |
~/.aitimemachine-server/blobs/ |
Shared blob storage |
~/.aitimemachine-server/projects.json |
Project registry |
8. Related Components
- MCP Server (
src/mcp-daemon.ts) - Exposes storage via MCP tools - CLI (
src/cli-v2.ts) - Command-line access to storage - API Server (
src/api-server.ts) - REST API for storage - Extension (
extensions/vscode/) - VS Code integration - Website (
website/) - Web UI for browsing history
9. Canonical Backend + Local Cache
Backend storage is required and is the source of truth for checkpoints, messages, contexts, diffs, and governance records.
Local storage (.aitimemachine/ and ~/.aitimemachine-server/) is a cache/offline buffer optimized for performance and resilience.
9.1 Canonical Schema
Use the canonical schema file:
../../CANONICAL_SQL_SCHEMA_V1.sql
Schema notes:
- Social + time machine + governance tables are unified.
checkpoints/messages/tool_calls/contextsare linked for deterministic restore-with-context.- Consent, retention, audit, and export controls are first-class entities.
9.2 Sync Behavior
- Capture writes to local cache first for low latency.
- Sync service pushes to backend and reconciles by canonical checkpoint lineage.
- Restore operations always resolve from canonical backend state, then materialize locally.
9.3 Legacy Cloud Commands
Legacy tm cloud ... and /api/cloud/... interfaces are compatibility paths only.
New implementation should target canonical time, actions, context, and governance APIs defined in:
../../NORTH_STAR_PRD_ARCHITECTURE_V1.md
See Also
- GIT_INTERNALS_INTEGRATION.md - Git-like algorithms
- GIT_INTEGRATION_COMPLETE.md - Enhanced features
- API.md - API documentation