This repository has been archived on 2026-11-17. You can view files and clone it, but cannot push or open issues or pull requests.
gitgram-ai/aitimemachine/docs/DATA_STORAGE_ARCHITECTURE.md
caelum0x 49b183818c Unify gitgram + aitimemachine into the monorepo (de-submodule) and merge vendored engines
Convert gitgram/ and aitimemachine/ from embedded git repos (which showed as
opaque submodule gitlinks on GitHub) into fully-tracked first-party content.

AITM now merges its 7 vendored repos as first-party engines:
- memory: beads (Go task DAG) + context7 (docs SDK), unified into saved_contexts
- git: vendored C git (bundled binary) + gitnexus (in-process code intelligence)
- agentic: a2a (@a2a-js/sdk) + pal (AITM-managed Python subprocess)

Includes the full session's product build: Gitgram web/mobile/backend features,
AITM website/CLI/MCP, the unified memory layer, engine build tooling
(scripts/build-engines.sh, Dockerfile, CI), docs (README, ENGINES.md, ARCHITECTURE.md),
and extensive test suites.

Build artifacts, node_modules, external/, and gemini-cli are gitignored.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 14:14:09 +03:00

22 KiB

AITimeMachine Data Storage Architecture

This document explains how AITimeMachine stores code snapshots, chat history, and related data from AI coding tools. Canonical reference: ../../NORTH_STAR_PRD_ARCHITECTURE_V1.md.

Overview

AITimeMachine uses a backend-canonical storage architecture:

  1. Canonical Backend Storage (Postgres + object storage; source of truth)
  2. Project-Level Local Cache (.aitimemachine/ folder in each project)
  3. Global Local Cache (~/.aitimemachine-server/ for fast lookup/offline buffering)
┌─────────────────────────────────────────────────────────────────────────┐
│                    AITimeMachine Storage Architecture                    │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ┌─────────────────────┐     ┌──────────────────────────────────────┐  │
│  │   AI Coding Tools   │     │          Global Daemon                │  │
│  │  ┌───────────────┐  │     │  ┌──────────────────────────────┐    │  │
│  │  │ Claude Code   │──┼────▶│  │ Auto-detect projects from:   │    │  │
│  │  │ Cursor        │  │     │  │ • Claude, Cursor, Aider      │    │  │
│  │  │ Aider         │  │     │  │ • Windsurf, Copilot, Zed     │    │  │
│  │  │ Windsurf      │  │     │  │ • Active terminals           │    │  │
│  │  │ VS Code       │  │     │  │ • Recent git repos           │    │  │
│  │  │ Copilot       │  │     │  └──────────────────────────────┘    │  │
│  │  └───────────────┘  │     │                │                      │  │
│  └─────────────────────┘     │                ▼                      │  │
│                              │  ┌──────────────────────────────┐    │  │
│                              │  │ Periodic Capture (5 min)     │    │  │
│                              │  │ • Incremental file scanning   │    │  │
│                              │  │ • Content-addressable store   │    │  │
│                              │  │ • Delta compression           │    │  │
│                              │  └──────────────────────────────┘    │  │
│                              └──────────────────────────────────────┘  │
│                                            │                           │
│                                            ▼                           │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │                   Project-Level Storage                           │  │
│  │                   .aitimemachine/                                 │  │
│  │  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐  │  │
│  │  │   store/    │ │ timepoints/ │ │   content/  │ │  events.   │  │  │
│  │  │ (blobs by   │ │  (JSON      │ │ (messages,  │ │   jsonl    │  │  │
│  │  │  hash)      │ │  snapshots) │ │ code, diffs)│ │            │  │  │
│  │  └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘  │  │
│  └──────────────────────────────────────────────────────────────────┘  │
│                                            │                           │
│                                            ▼                           │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │                   Cloud Storage (Neon)                            │  │
│  │                   postgres://...neon.tech                         │  │
│  │  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐  │  │
│  │  │  projects   │ │ timepoints  │ │  messages   │ │   blobs    │  │  │
│  │  │  (registry) │ │ (snapshots) │ │ (full-text  │ │  (content) │  │  │
│  │  │             │ │             │ │  search)    │ │            │  │  │
│  │  └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘  │  │
│  │  Features: SQL queries, cross-device sync, Neon branching        │  │
│  └──────────────────────────────────────────────────────────────────┘  │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

1. Project-Level Storage (.aitimemachine/)

Each project has a .aitimemachine/ directory containing:

1.1 Content Store (store/)

Git-like content-addressable storage for all file content.

.aitimemachine/
└── store/
    ├── 7a/
    │   └── 7a343253936975ff77526dbbc56ff539  # SHA-256 hash (first 32 chars)
    ├── 85/
    │   └── 85a953d052d9cf2504d1adafc463c349
    └── ...

Key Features:

  • SHA-256 hashing (truncated to 32 chars)
  • Deduplication: Same content stored once regardless of filename
  • Two-level directory: First 2 chars of hash = subdirectory (prevents filesystem limitations)

Implementation (src/core/content-store.ts):

put(content: string | Buffer): string {
  const buffer = typeof content === 'string' ? Buffer.from(content, 'utf-8') : content;
  const hash = crypto.createHash('sha256').update(buffer).digest('hex').slice(0, 32);
  const filePath = this.getPath(hash);  // e.g., store/7a/7a343...
  if (!fs.existsSync(filePath)) {
    fs.writeFileSync(filePath, buffer);
  }
  return hash;
}

1.2 TimePoints (timepoints/)

Complete project snapshots stored as JSON files.

.aitimemachine/
└── timepoints/
    ├── tp_1703123456789_abc123.json
    ├── tp_1703123567890_def456.json
    └── ...

TimePoint Structure:

interface TimePoint {
  id: string;                    // e.g., "tp_1703123456789_abc123"
  timestamp: Date;               // When captured
  description: string;           // User or auto-generated description
  
  trigger: {
    type: 'manual' | 'auto' | 'ai_action' | 'restore';
    tool?: string;               // "claude", "cursor", etc.
    action?: string;             // "create_file", "edit_file", etc.
  };
  
  linkedCheckpointId?: string;   // Links to conversation checkpoint
  
  conversation: {
    sessionId: string;
    messageCount: number;
    lastMessagePreview: string;
  };
  
  files: FileEntry[];            // All files at this point
  
  changes: {
    added: string[];             // New files since last timepoint
    modified: string[];          // Changed files
    deleted: string[];           // Removed files
  };
  
  stats: {
    totalFiles: number;
    totalSize: number;
  };
}

interface FileEntry {
  path: string;                  // Relative path
  hash: string;                  // Reference to content in store/
  size: number;
  mode: number;
  mtime: number;
}

1.3 Event Store (events.jsonl)

Append-only log of all events (JSONL format for streaming writes).

.aitimemachine/
└── events.jsonl

Event Types:

type EventType =
  | 'user_message'           // User prompt to AI
  | 'assistant_message'      // AI response
  | 'tool_call'              // Tool invocation (create_file, edit, etc.)
  | 'tool_result'            // Tool execution result
  | 'checkpoint_created'     // Explicit checkpoint
  | 'code_change'            // File modification event
  | 'file_created'
  | 'file_modified'
  | 'file_deleted'
  | 'tool_error'
  | 'terminal_command'
  | 'error_occurred';

interface Event {
  id: string;
  timestamp: Date;
  sessionId: string;
  type: EventType;
  data: any;                 // Type-specific payload
}

Example Events:

{"id":"event_1703123456_abcd","timestamp":"2024-12-21T10:30:00.000Z","sessionId":"sess_123","type":"user_message","data":{"content":"Create a REST API","fullSize":20}}
{"id":"event_1703123457_efgh","timestamp":"2024-12-21T10:30:01.000Z","sessionId":"sess_123","type":"tool_call","data":{"tool":"create_file","args":{"path":"src/api.ts"}}}
{"id":"event_1703123458_ijkl","timestamp":"2024-12-21T10:30:02.000Z","sessionId":"sess_123","type":"file_created","data":{"filePath":"src/api.ts","newSize":1024,"diffRef":"content/diffs/diff_1703123458.patch"}}

1.4 Content Storage (content/)

Large content stored separately from event log (referenced by pointer).

.aitimemachine/
└── content/
    ├── messages/              # Long AI messages
    │   └── msg_1703123456_abc.txt
    ├── code/                  # Code snapshots
    │   └── code_1703123457_def.ts
    ├── diffs/                 # Generated diffs
    │   └── diff_1703123458_ghi.patch
    └── outputs/               # Tool outputs
        └── out_1703123459_jkl.txt

Why Separate Storage?

  • Event log stays small and fast to parse
  • Large content only loaded on demand
  • Easy garbage collection

1.5 Pack Files (packs/)

Delta-compressed archives for space efficiency.

.aitimemachine/
└── packs/
    ├── pack_1703123456.pack
    └── pack_1703123456.idx

Pack Format:

  • Similar to Git pack files
  • Delta compression between similar objects
  • Index file for O(1) hash lookup

2. Global Server Storage (~/.aitimemachine-server/)

Centralized storage for cross-project management.

2.1 Sessions (sessions/)

Complete project snapshots with full metadata.

interface Session {
  id: string;
  projectPath: string;
  projectName: string;
  createdAt: Date;
  updatedAt: Date;
  
  trigger: {
    type: 'auto' | 'manual' | 'ai_action' | 'restore';
    tool?: string;
    action?: string;
    description: string;
  };
  
  files: SessionFile[];
  diffs: SessionDiff[];
  conversation: SessionMessage[];
  
  stats: {
    totalFiles: number;
    totalSize: number;
    addedFiles: number;
    modifiedFiles: number;
    deletedFiles: number;
    messageCount: number;
  };
  
  git?: GitState;
  environment?: EnvironmentState;
}

2.2 Shared Blobs (blobs/)

Deduplicated content shared across all projects.

2.3 Projects Index (projects.json)

Project registry with metadata.

interface Project {
  id: string;
  path: string;
  name: string;
  createdAt: Date;
  lastSessionAt: Date;
  sessionCount: number;
  totalSize: number;
}

2.4 Conversations (conversations/)

Conversation logs per session.


3. Conversation Logging

The ConversationLogger class orchestrates conversation capture:

class ConversationLogger {
  private eventStore: EventStore;
  private contentStorage: ContentStorage;
  private sessionId: string;

  // Log user message
  logUserMessage(content: string): void {
    const contentRef = content.length > 500 
      ? this.contentStorage.storeMessage(content) 
      : undefined;
    
    this.eventStore.append({
      sessionId: this.sessionId,
      type: 'user_message',
      data: { 
        content: content.slice(0, 500),  // Preview
        contentRef,                       // Full content reference
        fullSize: content.length
      }
    });
  }

  // Log AI response
  logAssistantMessage(content: string, thinking?: string): void {
    // Similar pattern - store large content separately
  }

  // Log tool calls and results
  logToolCall(tool: string, args: any): string;
  logToolResult(callId: string, result: any): void;

  // Log code changes with automatic diff generation
  logCodeChange(filePath: string, oldContent: string | null, newContent: string | null): void {
    const changeType = !oldContent ? 'file_created' 
                     : !newContent ? 'file_deleted' 
                     : 'file_modified';
    
    // Store full content
    const oldContentRef = oldContent ? this.contentStorage.storeCode(oldContent) : undefined;
    const newContentRef = newContent ? this.contentStorage.storeCode(newContent) : undefined;
    
    // Generate and store diff
    if (oldContent && newContent) {
      const generatedDiff = diff.createPatch(filePath, oldContent, newContent);
      const diffRef = this.contentStorage.storeDiff(generatedDiff);
    }
    
    this.eventStore.append({
      sessionId: this.sessionId,
      type: changeType,
      data: {
        filePath,
        oldContentRef,
        newContentRef,
        diffRef,
        oldSize: oldContent?.length || 0,
        newSize: newContent?.length || 0,
        oldSnippet: oldContent?.slice(0, 500),
        newSnippet: newContent?.slice(0, 500),
      }
    });
  }
}

4. AI Tool Detection & Capture

The GlobalDaemon automatically discovers and captures from multiple AI tools:

4.1 Supported Tools

Tool Detection Method Data Source
Claude Code ~/.claude/projects/ Encoded project paths
Cursor ~/Library/.../Cursor/workspaceStorage/ workspace.json files
Aider ~/.aider/ + .aider.chat.history.md Chat history files
Windsurf ~/.windsurf/ or ~/.codeium/ Workspace storage
VS Code ~/Library/.../Code/workspaceStorage/ workspace.json files
Copilot (via VS Code) Same as VS Code
Zed ~/Library/.../Zed/workspaces/ JSON workspace files

4.2 Active Detection

// Scan active terminals using lsof
scanActiveTerminals(): void {
  const output = execSync(
    `lsof -c node -c python -c ruby -c cargo -c go 2>/dev/null | grep -E "cwd|txt"`
  );
  // Find project roots (package.json, Cargo.toml, etc.)
}

// Scan git repos with recent activity
scanRecentGitRepos(): void {
  // Check ~/Projects, ~/Developer, ~/dev, ~/code, ~/src
  // Look for .git directories with HEAD modified in last 24h
}

4.3 Capture Flow

┌─────────────┐     ┌───────────────┐     ┌────────────────┐
│ AI Tool     │────▶│ Global Daemon │────▶│ FastSession-   │
│ Activity    │     │ (detects)     │     │ Capture        │
└─────────────┘     └───────────────┘     └────────────────┘
                                                  │
                    ┌─────────────────────────────┘
                    ▼
    ┌───────────────────────────────────────────────────┐
    │ 1. Incremental file scan (mtime-based)            │
    │ 2. Hash only changed files                        │
    │ 3. Store new content to blobs                     │
    │ 4. Generate diffs from previous session           │
    │ 5. Save session with conversation context         │
    │ 6. Optional: Database & container state capture   │
    └───────────────────────────────────────────────────┘

5. Storage Optimization

5.1 Incremental Scanning

// Only hash files that changed (by mtime)
for (const filePath of files) {
  const prev = previousFiles.get(relativePath);
  
  if (prev && prev.mtime === info.mtime && prev.size === info.size) {
    // File unchanged - reuse hash without reading content
    hash = prev.hash;
  } else {
    // File changed - read and store
    const content = this.fileScanner.readFile(filePath);
    hash = this.contentStore.put(content);
  }
}

5.2 Content Deduplication

Same file content across different files or timepoints is stored once:

File A: "hello world"  → hash: abc123
File B: "hello world"  → hash: abc123 (same blob reused)

5.3 Delta Compression

Pack files use delta encoding for similar files:

class DeltaCompressor {
  compress(source: Buffer, target: Buffer): Buffer {
    // Find common prefix, sliding window matching, etc.
    return encodeDelta(instructions);
  }
}

5.4 Garbage Collection

tm gc  # Clean unreferenced blobs, pack loose objects

6. Data Access Patterns

6.1 Retrieve File at TimePoint

engine.getFileAt(timepointId: string, filePath: string): string | null {
  const tp = this.timePointManager.load(timepointId);
  const fileEntry = tp.files.find(f => f.path === filePath);
  return this.contentStore.getString(fileEntry.hash);
}

6.2 Get Conversation Context

conversationLogger.getContextForCheckpoint(checkpointId: string): ConversationContext {
  const events = this.eventStore.getEventsUntil(checkpointId);
  return {
    messages: events.filter(e => e.type === 'user_message' || e.type === 'assistant_message'),
    toolCalls: events.filter(e => e.type === 'tool_call'),
    codeChanges: events.filter(e => ['file_created', 'file_modified', 'file_deleted'].includes(e.type)),
    checkpoints: events.filter(e => e.type === 'checkpoint_created')
  };
}

6.3 Timeline Query

// Get timepoints in range
timePointManager.getInRange(start: Date, end: Date): TimePoint[];

// Get timepoints by trigger
timePointManager.getByTrigger('ai_action'): TimePoint[];

7. File Paths Summary

Path Purpose
.aitimemachine/store/ Content-addressable blob storage
.aitimemachine/timepoints/ JSON snapshot files
.aitimemachine/events.jsonl Append-only event log
.aitimemachine/content/ Large content (messages, code, diffs)
.aitimemachine/packs/ Delta-compressed archives
.aitimemachine/config.json Project configuration
~/.aitimemachine-server/sessions/ Global session storage
~/.aitimemachine-server/blobs/ Shared blob storage
~/.aitimemachine-server/projects.json Project registry

  • MCP Server (src/mcp-daemon.ts) - Exposes storage via MCP tools
  • CLI (src/cli-v2.ts) - Command-line access to storage
  • API Server (src/api-server.ts) - REST API for storage
  • Extension (extensions/vscode/) - VS Code integration
  • Website (website/) - Web UI for browsing history

9. Canonical Backend + Local Cache

Backend storage is required and is the source of truth for checkpoints, messages, contexts, diffs, and governance records. Local storage (.aitimemachine/ and ~/.aitimemachine-server/) is a cache/offline buffer optimized for performance and resilience.

9.1 Canonical Schema

Use the canonical schema file:

  • ../../CANONICAL_SQL_SCHEMA_V1.sql

Schema notes:

  • Social + time machine + governance tables are unified.
  • checkpoints/messages/tool_calls/contexts are linked for deterministic restore-with-context.
  • Consent, retention, audit, and export controls are first-class entities.

9.2 Sync Behavior

  • Capture writes to local cache first for low latency.
  • Sync service pushes to backend and reconciles by canonical checkpoint lineage.
  • Restore operations always resolve from canonical backend state, then materialize locally.

9.3 Legacy Cloud Commands

Legacy tm cloud ... and /api/cloud/... interfaces are compatibility paths only. New implementation should target canonical time, actions, context, and governance APIs defined in:

  • ../../NORTH_STAR_PRD_ARCHITECTURE_V1.md

See Also