# Manual Launch Runbook Last updated: 2026-05-12 This file is the launch source of truth for Gitgram and AITM. - Primary topology: one operator-managed host. - Primary process manager: `pm2`. - AITM `systemd` unit: fallback only. - `docker-compose.prod.yml`, `deploy/DEPLOYMENT.md`, and GitHub deploy workflows are not the canonical launch path. ## Canonical Topology | Service | Product | Runtime | Control Plane | Port | Health Gate | |---------|---------|---------|---------------|------|-------------| | `gitgram-api` | Gitgram | Node / Express | `pm2` | `4000` | `GET /health`, `GET /health/ready` | | `gitgram-web` | Gitgram | Node / Next.js | `pm2` | `3000` | `GET /` returns `<500` | | `aitm-api` | AITM | Node / Express | `pm2` primary, `systemd` fallback | `3001` | `GET /health`, `GET /health/ready` | | `postgres` | Shared infra | PostgreSQL 16 | operator-managed | `5432` | `pg_isready` | | `redis` | Shared infra | Redis 7 | operator-managed | `6379` | `redis-cli ping` | Optional companion service: | Service | Product | Runtime | Port | Notes | |---------|---------|---------|------|------| | `aitm-website` | AITM | Node / Next.js | `3002` | Self-host only if you want the hosted dashboard on the same machine. Not part of the launch-critical path in this repo. | ## Repo Paths | Artifact | Path | |---------|------| | Gitgram PM2 config | `gitgram/ecosystem.config.js` | | AITM PM2 config | `aitimemachine/deploy/ecosystem.config.cjs` | | AITM `systemd` fallback unit | `aitimemachine/deploy/aitm-api.service` | | Gitgram backend env template | `.env.gitgram.example` | | Gitgram web env template | `.env.gitgram-web.example` | | AITM API env template | `.env.aitm-api.example` | | Optional AITM website env template | `.env.aitm-website.example` | | Gitgram backend schema | `gitgram/backend/prisma/schema.prisma` | | AITM API schema | `aitimemachine/api-server/src/db/schema.ts` | ## Preflight Run these before any staging-like dry run or production launch: ```bash node -v npm -v pm2 -v ``` Use Node 20.x for build and runtime consistency with the repo workflows. Copy and fill the operator env files at repo root. Preferred layout: ```bash cp .env.gitgram.example .env.gitgram.production cp .env.gitgram-web.example .env.gitgram-web.production cp .env.aitm-api.example .env.aitm-api.production cp .env.gitgram.example .env.gitgram.staging cp .env.gitgram-web.example .env.gitgram-web.staging cp .env.aitm-api.example .env.aitm-api.staging # Optional: cp .env.aitm-website.example .env.aitm-website ``` Fallback layout still works if you want one shared set of root files: ```bash cp .env.gitgram.example .env.gitgram cp .env.gitgram-web.example .env.gitgram-web cp .env.aitm-api.example .env.aitm-api ``` Validate launch-critical env contracts: ```bash bash scripts/validate-prod-secrets.sh staging core bash scripts/validate-prod-secrets.sh production core ``` One-command readiness gate: ```bash npm run verify:launch ``` This is the deployable core profile: database, app secrets, routing URLs, storage mode, and PM2/deploy artifacts. Optional deeper gate: ```bash npm run verify:launch:full ``` This adds third-party integrations that are not required just to boot the services: - GitHub OAuth - Polar billing - Resend - Sentry Manual deploy helper: ```bash bash scripts/deploy-launch.sh staging ``` To require the full integrations gate before deploy: ```bash bash scripts/deploy-launch.sh staging --verify-profile full ``` Secret entry reference: - `docs/launch-secrets-checklist.md` Notes: - The validator accepts both unprefixed vars and `GITGRAM_*` / `AITM_*` aliases. - It prefers `.env.gitgram.`, `.env.gitgram-web.`, `.env.aitm-api.`, and `.env.aitm-website.` when those files exist, and falls back to the root files otherwise. - `verify:launch` uses the `core` profile by default so you can deploy the stack before every external provider account is wired. - `verify:launch:full` and `validate-prod-secrets.sh full` enforce the full launch contract, including third-party integrations. - `verify:launch` also checks PM2 availability, required deploy artifacts, operator env file presence, and AITM package publishability. - `deploy-launch.sh` is the scripted PM2/manual deploy flow. It is the automation companion to this runbook, not a Docker deploy. ## Backup Gate Take backups before any production migration. Gitgram: ```bash ( set -a source .env.gitgram set +a cd gitgram ./scripts/backup-database.sh ) ``` AITM: ```bash ( set -a source .env.aitm-api set +a mkdir -p /var/backups/aitm pg_dump "$DATABASE_URL" | gzip > "/var/backups/aitm/aitm_$(date +%Y%m%d_%H%M%S).sql.gz" ) ``` Redis snapshot: ```bash redis-cli -u "$(grep '^REDIS_URL=' .env.gitgram | cut -d= -f2-)" BGSAVE ``` ## AITM Package Release Publish order matters: 1. `@gitgram/timemachine` 2. `@gitgram/timemachine-mcp` ### Manual Publish Checklist Core package: ```bash cd aitimemachine npm ci npm run build npm test bash scripts/verify-npm-packages.sh npm whoami npm publish --access public ``` Post-publish verification: ```bash npm view @gitgram/timemachine version npx -y @gitgram/timemachine --version tmpdir="$(mktemp -d)" ( cd "$tmpdir" npx -y @gitgram/timemachine init npx -y @gitgram/timemachine checkpoint "launch-smoke" ) rm -rf "$tmpdir" ``` MCP package: ```bash cd aitimemachine/mcp npm ci npm run build npm publish --access public ``` Post-publish verification: ```bash npm view @gitgram/timemachine-mcp version npx -y @gitgram/timemachine-mcp --help ``` ### Package Rollback Policy Use manual deprecation plus a version bump. Do not rely on unpublish. ```bash npm deprecate @gitgram/timemachine@ "broken release; use >=" npm deprecate @gitgram/timemachine-mcp@ "broken release; use >=" ``` Then publish a new patch or minor release: ```bash cd aitimemachine npm version patch --no-git-tag-version cd mcp npm version patch --no-git-tag-version ``` Automation status: - `aitimemachine/.github/workflows/ci.yml` has a tag-driven npm publish step. - It currently publishes only `@gitgram/timemachine`. - The MCP publish remains manual until the workflow is brought into line with the manual launch process. - The GitHub secret name used there is `NPM_TOKEN`, passed to npm as `NODE_AUTH_TOKEN`. ## Gitgram Deploy ### Backend Build + Migrate ```bash cd gitgram/backend npm ci --legacy-peer-deps npx prisma generate npx prisma migrate deploy npm run build ``` ### Backend Start / Restart Initial start: ```bash ( set -a source ../../.env.gitgram set +a pm2 start ../ecosystem.config.js --only gitgram-api ) pm2 save ``` Redeploy: ```bash ( set -a source ../../.env.gitgram set +a pm2 restart gitgram-api --update-env ) pm2 save ``` Scripted equivalent: ```bash bash scripts/deploy-launch.sh staging --gitgram-only ``` ### Backend Health Gate ```bash curl -sf http://127.0.0.1:4000/health | jq . curl -sf http://127.0.0.1:4000/health/ready | jq . ``` ## Gitgram Web Deploy ### Build `NEXT_PUBLIC_*` values are build-time inputs. Rebuild whenever they change. ```bash ( set -a source .env.gitgram-web set +a cd gitgram/web npm ci --legacy-peer-deps npm run build ) ``` ### Start / Restart Initial start: ```bash ( set -a source .env.gitgram-web set +a cd gitgram pm2 start ecosystem.config.js --only gitgram-web ) pm2 save ``` Redeploy: ```bash ( set -a source .env.gitgram-web set +a pm2 restart gitgram-web --update-env ) pm2 save ``` Scripted equivalent: ```bash bash scripts/deploy-launch.sh staging --gitgram-only ``` ### Web Health Gate ```bash curl -sf -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3000 ``` ## AITM API Deploy ### Build + Migrate ```bash cd aitimemachine/api-server npm ci npm run db:migrate npm run build ``` ### Start / Restart With PM2 Initial start: ```bash ( set -a source ../../.env.aitm-api set +a pm2 start ../deploy/ecosystem.config.cjs --only aitm-api ) pm2 save ``` Redeploy: ```bash ( set -a source ../../.env.aitm-api set +a pm2 restart aitm-api --update-env ) pm2 save ``` Scripted equivalent: ```bash bash scripts/deploy-launch.sh staging --aitm-only ``` ### `systemd` Fallback Use only if PM2 is unavailable or intentionally replaced. ```bash sudo cp aitimemachine/deploy/aitm-api.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable aitm-api sudo systemctl restart aitm-api ``` The checked-in unit assumes the repo lives at `/opt/aitimemachine/api-server`. Adjust `WorkingDirectory`, `ExecStart`, and `EnvironmentFile` if your server layout differs. ### API Health Gate ```bash curl -sf http://127.0.0.1:3001/health | jq . curl -sf http://127.0.0.1:3001/health/ready | jq . ``` If `ENABLE_WORKERS=true`, also check worker status: ```bash ( set -a source .env.aitm-api set +a curl -sf -H "X-Internal-Secret: ${INTERNAL_SECRET}" \ http://127.0.0.1:3001/v1/workers/status | jq . ) ``` ## Optional AITM Website This repo includes `aitimemachine/website`, but it is not part of the launch-critical path defined here. If you self-host it on the same machine: ```bash ( set -a source .env.aitm-website set +a cd aitimemachine/website npm ci PORT=3002 npm run build PORT=3002 npm start ) ``` ## Launch-Day Order 1. Freeze merges and confirm the intended commit or tag. 2. Run `bash scripts/validate-prod-secrets.sh production`. 3. Take Gitgram and AITM database backups. 4. Publish `@gitgram/timemachine`. 5. Verify install + smoke. 6. Publish `@gitgram/timemachine-mcp`. 7. Verify install + smoke. 8. Deploy Gitgram backend. 9. Wait for Gitgram backend health gates. 10. Deploy Gitgram web. 11. Wait for Gitgram web health gate. 12. Deploy AITM API. 13. Wait for AITM API health gates. 14. Run smoke checks. 15. Monitor for at least 30 minutes before declaring success. ## Smoke Checks ### Gitgram Required: - `curl -sf http://127.0.0.1:4000/health` - `curl -sf http://127.0.0.1:4000/health/ready` - `curl -sf -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3000` Manual: - Load the web app. - Confirm it can reach the API. - Exercise one login path. - Create one basic post. ### AITM Required: - `curl -sf http://127.0.0.1:3001/health` - `curl -sf http://127.0.0.1:3001/health/ready` - `npx -y @gitgram/timemachine --version` - `npx -y @gitgram/timemachine-mcp --help` Manual: - Create a temp project. - Run `init`. - Create a checkpoint. - If workers are enabled, verify `/v1/workers/status`. Public-domain smoke: - `npm run verify:public-launch` ## Rollback ### npm Packages 1. Deprecate the bad version. 2. Bump the version. 3. Rebuild and republish. 4. Re-run the install smokes. ### Gitgram Backend / Web Rollback is manual-first. Use the previous known-good git tag, commit, or release directory already present on the server. Example with an in-place git checkout: ```bash git checkout cd gitgram/backend npm ci --legacy-peer-deps npx prisma generate npm run build cd ../web npm ci --legacy-peer-deps npm run build pm2 restart gitgram-api --update-env pm2 restart gitgram-web --update-env ``` ### AITM API ```bash git checkout cd aitimemachine/api-server npm ci npm run build pm2 restart aitm-api --update-env ``` ### Database Rollback Policy - Gitgram / Prisma: no automatic `down` migrations in this launch contract. - AITM / Drizzle: no automatic `down` migrations in this launch contract. - Production rollback policy is restore from backup or provider point-in-time restore unless you have already written and reviewed a reverse SQL plan. If a migration fails after partial apply: 1. Stop the app deploy. 2. Do not keep rolling forward blindly. 3. Restore from the backup taken immediately before the migration, or use provider PITR. ## Monitoring Window Watch for at least 30 minutes after launch: ```bash pm2 ls pm2 logs gitgram-api --lines 200 pm2 logs gitgram-web --lines 100 pm2 logs aitm-api --lines 200 ``` Recommended external checks: - Sentry error volume - Polar webhook delivery success - PostgreSQL connection count - Redis availability - repeated `curl` probes against `/health` and `/health/ready`