Compare commits

..

No commits in common. "main" and "v0.1.0" have entirely different histories.
main ... v0.1.0

11 changed files with 125 additions and 366 deletions

21
LICENSE
View File

@ -1,21 +0,0 @@
MIT License
Copyright (c) 2026 caelum0x
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

162
README.md
View File

@ -1,105 +1,36 @@
# scribase-migrate
Move a Supabase, Neon, Clerk or Resend setup onto [Scribase](https://scribase.com/migrate?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate)
with one command. Scribase is Postgres, auth, storage and email on one account instead of four.
## 10-second quickstart
One command to move a team onto [Scribase](https://scribase.com) — the one
account for Postgres, auth, storage and email — from Supabase, Neon, Clerk
and Resend.
```
npx scribase login
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git neon \
--to <org>/<project>/try1 --create-project --kind preview \
--source-url "$DATABASE_URL" --migrate-auth-users
```
That creates an empty preview environment and moves your schema, data and users into it. Your
source database is only read, never changed. No account yet?
[It is free to start](https://console.scribase.com/signup?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate).
Add `--dry-run` to see the plan without applying anything. The `neon` source works for any plain
Postgres and needs `pg_dump` on your machine. Node 18+.
## What you get
Real output, 2026-11-17: a Postgres 16 database with 2 users, 2 profiles and 3 notes, moved into
a brand-new Scribase preview environment. Project creation included, the whole run took 35
seconds.
```
exporting auth.users rows
auth.users: 2 rows, applied=true
dumping public schema
public schema: 0 lint findings, applied=true
dumping public data
public data: applied=true
Full result also written to run1.json
Scribase: Postgres, auth, storage and email on one account. Migration guide: https://scribase.com/migrate?utm_source=cli&utm_medium=cli-output&utm_campaign=scribase-migrate
```
From the JSON result of the same run (`--verify-row-counts public.profiles --verify-row-counts public.notes`):
```json
"rowCounts": [
{ "table": "public.profiles", "sourceRows": 2, "targetRows": 2, "match": true },
{ "table": "public.notes", "sourceRows": 3, "targetRows": 3, "match": true }
],
"rlsDecisions": [
{ "schema": "public", "table": "notes", "kind": "join", "detail": "profile_id -> public.profiles.id -> id = auth.uid()" },
{ "schema": "public", "table": "profiles", "kind": "direct", "detail": "id = auth.uid()" }
]
```
A migrated user then signed in on the new environment with their old password (`200` with an
access token; a wrong password got `400 Invalid login credentials`). Running the same command a
second time applied nothing and wrote no new files.
## The four sources
```
scribase-migrate supabase --to <org>/<project> --create-project \
npx scribase-migrate supabase --to <org>/<project> --create-project \
--source-ref <ref> --source-url <connection-string> --source-service-key <key>
scribase-migrate neon --to <org>/<project> --source-url <connection-string> --migrate-auth-users
npx scribase-migrate neon --to <org>/<project> \
--source-url <connection-string> --migrate-auth-users
scribase-migrate clerk --to <org>/<project> --secret-key <sk_...>
npx scribase-migrate clerk --to <org>/<project> --secret-key <sk_...>
scribase-migrate resend --api-key <re_...> --out resend-export.json
npx scribase-migrate resend --api-key <re_...> --out resend-export.json
```
Prefix each with `npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git` (Bun:
`bunx -p git+https://rithub.dev/caelum0x/scribase-migrate.git scribase-migrate`), or clone the
repo and run `node bin/scribase-migrate.mjs`. `--help` prints every flag.
Run `npx scribase-migrate --help` for the full flag reference.
| Source | Postgres schema + data | Auth users / passwords | Storage | Notes |
It needs a Scribase access token the same way `scribase` itself does — run
`npx scribase login` once, or export `SCRIBASE_ACCESS_TOKEN`.
## What each source actually moves
| Source | Postgres schema+data | Auth users / passwords | Storage | Notes |
|---|---|---|---|---|
| **Supabase** | yes | yes (hash preserved) | yes | one API call, Scribase's own `POST /v1/imports` |
| **Neon** | yes | yes, if the source has its own `auth.users` table | no | Neon has no storage or auth of its own; branches land in `branch_<name>` schemas |
| **Clerk** | no | yes, with a **new** one-time password (Clerk never exposes a portable hash) | no | every migrated user is flagged `password_reset_required` |
| **Resend** | no | no | no | exports sending domains (DNS records) and broadcasts to a JSON bundle. No Scribase account needed. Scribase has no import endpoint for either yet, so this is import-ready, not auto-applied |
| **Supabase** | yes | yes (hash preserved) | yes | one real API call, Scribase's own `POST /v1/imports` |
| **Neon** | yes | yes, if the source has its own `auth.users` table | — | Neon has no storage/auth of its own; branches land in `branch_<name>` schemas |
| **Clerk** | — | yes, but a **new** one-time password (Clerk never exposes a portable hash) | — | every migrated user is flagged `password_reset_required` |
| **Resend** | — | — | — | exports sending domains (DNS records) + broadcasts as "templates" to a JSON bundle; Scribase has no write endpoint for either yet, so this is import-ready, not auto-applied |
Everything except `resend` needs a Scribase access token: `npx scribase login` once, or export
`SCRIBASE_ACCESS_TOKEN`.
## What is verified and what is not
**Verified live** against a real Scribase organization, twice (first build, then the 0.1.1
re-run above): the `neon` source end to end, including `--create-project`, generated RLS passing
Scribase's migration lint with 0 blocking findings, row-count parity over HTTP, a real password
login, `--dry-run`, and an idempotent second run. The `resend` source reaches the real Resend API
(a fake key gets Resend's own `400 API key is invalid`). Artifacts:
[test-fixtures/verification-run](https://rithub.dev/caelum0x/scribase-migrate/src/branch/main/test-fixtures/verification-run).
**Not verified live**, because each needs a provider account this build did not have:
- `supabase`: the request shape was confirmed against the real API with a dry run (a fake project
ref is correctly rejected with `400 invalid_project_ref`), but the full one-call import needs a
real, network-reachable Supabase project.
- `clerk` and a full `resend` export: checked against a local mock of each provider's documented
API shape, not a real account.
## Why it works this way
## Why the design looks like this (real findings from building it)
This was built by composing Scribase's own already-shipped APIs — the
`scribase` npm SDK (`edge-repos/scribase-cli`) and the GoTrue admin API every
@ -172,11 +103,42 @@ probes the source's major version and prefers an exact-match Homebrew keg
(`postgresql@<major>`) over whatever `pg_dump` is on `PATH`, warning when
it has to fall back.
**Why a preview environment starts empty.** A `snapshot` or `sanitized` environment copies
production data, so the control plane refuses one in a project that has no production
environment yet (`source_environment_missing`). `--create-project --kind preview` therefore
creates a `schema_only` environment. A deleted environment's name cannot be reused (`409`), so
pick a new name each time you try this out.
## Real verification (not a test file — an actual live run)
This was verified against a real local Supabase-shaped Postgres and a
real, live Scribase project, not mocked:
1. A real Postgres 16 Docker container seeded with a GoTrue-shaped
`auth.users` table (two users, one password hashed with real `bcryptjs`),
`public.profiles` (owner = `auth.users.id`) and `public.notes`
(owner one hop through `profiles`) — `test-fixtures/seed-source.sql`.
2. `npx scribase-migrate neon --to ereboslab/scribase-migrate-verify-internal/production --source-url ... --migrate-auth-users --schema public` ran for real against a live Scribase org (`ereboslab`), through the real `scribase` SDK with a real `scribase login` token.
3. The schema step generated RLS automatically (`notes` via the
profiles→auth.uid() join rule, `profiles` directly) and passed Scribase's
real migration lint with **0 blocking findings**.
4. Row counts verified with a real HTTP call (`client.database.query`,
not a local connection) against the live environment:
`public.profiles` 2/2, `public.notes` 3/3, both `match: true`.
5. **Login verified for real**: `POST /auth/v1/token?grant_type=password`
against the live environment's real `api_url` — the correct password
returns `200` with a real `access_token`; a wrong password returns
`400 Invalid login credentials`. The password hash moved byte-for-byte
and still verifies.
Full before/after artifacts: `test-fixtures/verification-run/`.
**Not verified live** (both require a provider account/test credentials
this environment doesn't have — logged honestly rather than faked):
the `supabase` source's real one-call `POST /v1/imports` (its own request
shape was confirmed against the real API with a dry-run, which correctly
rejected a fake project ref with `400 invalid_project_ref`, but needs a
real reachable Supabase project — Scribase's control plane connects
*to* the source, so it must be a real, network-reachable Postgres, not a
local Docker container, to run the full one-call migration this source
wraps); the `clerk` and `resend` sources (logic-verified against a local
mock of each provider's documented API shape — pagination, the GoTrue
admin-create request body, the domains/broadcasts export — but not run
against a real Clerk or Resend account).
## Repo layout
@ -194,12 +156,8 @@ pick a new name each time you try this out.
- `docs/migrate-page.md` — the scribase.com/migrate docs-page snippet
- `test-fixtures/` — the real local-verification seed + its output
## Docs
## scribase.com/migrate
[docs/migrate-page.md](https://rithub.dev/caelum0x/scribase-migrate/src/branch/main/docs/migrate-page.md)
is the source of the migration guide on
[scribase.com/migrate](https://scribase.com/migrate?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate).
## License
MIT
`docs/migrate-page.md` is a ready-to-paste docs page for
`scribase.com/migrate` (the `scribase` checkout's owner adds it to
`apps/docs` and deploys — not done by this repo).

View File

@ -6,7 +6,7 @@
*/
import { parseArgs } from 'node:util';
import { writeFile, mkdir, readFile } from 'node:fs/promises';
import { writeFile, mkdir } from 'node:fs/promises';
import { dirname } from 'node:path';
import { scribaseClient } from '../lib/client.mjs';
@ -16,13 +16,6 @@ import { migrateNeon, verifyRowCounts } from '../lib/sources/neon.mjs';
import { exportClerkUsers, createGoTrueUsers } from '../lib/sources/clerk.mjs';
import { exportResend } from '../lib/sources/resend.mjs';
const CTA_URL = 'https://scribase.com/migrate?utm_source=cli&utm_medium=cli-output&utm_campaign=scribase-migrate';
const CTA_LINE = `Scribase: Postgres, auth, storage and email on one account. Migration guide: ${CTA_URL}`;
const SIGNUP_URL = 'https://console.scribase.com/signup?utm_source=cli&utm_medium=cli-output&utm_campaign=scribase-migrate';
const REGION_DEFAULT = 'us-east';
const KIND_DEFAULT = 'production';
const SOURCES = ['supabase', 'neon', 'clerk', 'resend', 'project-info'];
const USAGE = `scribase-migrate <source> [flags]
Sources:
@ -33,7 +26,6 @@ Sources:
Shared destination flags:
--to <org>/<project>/<environment> Scribase destination (default environment: production)
Not needed for resend, or for clerk with both --target-* flags
--create-project Create the project+environment first if missing
--region <region> Region for --create-project (default us-east)
--kind <preview|staging|production> Environment kind for --create-project (default production;
@ -47,7 +39,7 @@ supabase flags:
--verify-login-email <email> [--verify-login-password <password>]
--no-storage
neon flags (also works for any plain Postgres; needs pg_dump installed):
neon flags:
--source-url <postgres://...>
--schema <name> (repeatable; default public)
--migrate-auth-users Also copy an auth.users-shaped table's rows
@ -58,18 +50,12 @@ clerk flags:
--secret-key <sk_...>
--target-api-url <url> --target-service-key <key> (from \`scribase-migrate project-info\`)
resend flags (no Scribase account needed; pair with --out <file>):
resend flags:
--api-key <re_...>
Also: scribase-migrate project-info --to <org>/<project>/<environment>
prints the destination's api_url + keys (never the secrets to stdout
unless --reveal-secrets), the inputs clerk needs.
Needs a Scribase access token for everything except resend: run
\`npx scribase login\` once, or export SCRIBASE_ACCESS_TOKEN.
No account yet? It is free to start: ${SIGNUP_URL}
${CTA_LINE}
unless --reveal-secrets), the inputs clerk/resend need.
`;
function fail(message) {
@ -77,25 +63,6 @@ function fail(message) {
process.exit(1);
}
async function version() {
try {
return JSON.parse(await readFile(new URL('../package.json', import.meta.url), 'utf8')).version;
} catch {
return 'unknown';
}
}
/** The SDK client, or a plain exit that tells a brand-new user where to get an account. */
function clientOrExit() {
try {
return scribaseClient().client;
} catch (error) {
process.stderr.write(`scribase-migrate: ${error.message}\n`);
process.stderr.write(`No Scribase account yet? It is free to start: ${SIGNUP_URL}\n`);
process.exit(1);
}
}
function parseTo(value) {
if (!value) fail('--to <org>/<project>/<environment> is required');
const parts = value.split('/');
@ -109,23 +76,22 @@ async function writeResult(opts, result) {
await mkdir(dirname(opts.out), { recursive: true }).catch(() => {});
await writeFile(opts.out, JSON.stringify(result, null, 2));
}
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
if (opts.json) return;
// stdout stays pure JSON for pipes; the human-facing lines go to stderr.
if (opts.out) process.stderr.write(`\nFull result also written to ${opts.out}\n`);
process.stderr.write(`\n${CTA_LINE}\n`);
if (opts.json) {
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
} else {
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
if (opts.out) process.stderr.write(`\nFull result also written to ${opts.out}\n`);
}
}
async function resolveDestination(client, dest, values) {
if (values['create-project']) {
const region = values.region ?? REGION_DEFAULT;
const kind = values.kind ?? KIND_DEFAULT;
if (!['preview', 'staging', 'production'].includes(kind)) fail(`--kind "${kind}" must be preview, staging or production`);
// snapshot/sanitized copy production data, so the control plane refuses
// them for a project with no production environment yet (verified live:
// `source_environment_missing`). A try-it-out preview starts empty.
const dataMode = kind === 'production' ? 'snapshot' : 'schema_only';
return ensureProject(client, { ...dest, region, kind, dataMode });
async function resolveDestination(client, destArg, rawArgs) {
const dest = parseTo(destArg);
if (rawArgs.includes('--create-project')) {
const region = rawArgs[rawArgs.indexOf('--region') + 1] ?? 'us-east';
const kindIndex = rawArgs.indexOf('--kind');
const kind = kindIndex >= 0 ? rawArgs[kindIndex + 1] : 'production';
const project = await ensureProject(client, { ...dest, region, kind, dataMode: 'snapshot' });
return project;
}
const environment = await client.environments.get(dest);
const connection = await client.environments.connection(dest);
@ -135,17 +101,10 @@ async function resolveDestination(client, dest, values) {
async function main() {
const [source, ...rest] = process.argv.slice(2);
if (!source || source === '--help' || source === '-h' || rest.includes('--help') || rest.includes('-h')) {
if (!source || source === '--help' || source === '-h') {
process.stdout.write(USAGE);
process.exit(source ? 0 : 1);
}
if (source === '--version' || source === '-v') {
process.stdout.write(`${await version()}\n`);
return;
}
if (!SOURCES.includes(source)) {
fail(`unknown source "${source}". Use one of: ${SOURCES.join(', ')}. Run scribase-migrate --help.`);
}
const { values } = parseArgs({
args: rest,
@ -153,7 +112,6 @@ async function main() {
to: { type: 'string' },
'create-project': { type: 'boolean' },
region: { type: 'string' },
kind: { type: 'string' },
'dry-run': { type: 'boolean' },
out: { type: 'string' },
json: { type: 'boolean' },
@ -177,37 +135,9 @@ async function main() {
strict: false,
});
// Resend is a pure export: it needs neither a destination nor a Scribase
// account, so it must not ask for either.
if (source === 'resend') {
if (!values['api-key']) fail('resend needs --api-key');
const result = await exportResend({
apiKey: values['api-key'],
onLog: (line) => process.stderr.write(` ${line}\n`),
});
await writeResult(values, result);
return;
}
// Clerk with both --target-* flags talks to the environment directly.
if (source === 'clerk' && values['target-api-url'] && values['target-service-key']) {
if (!values['secret-key']) fail('clerk needs --secret-key');
await runClerk(values, values['target-api-url'], values['target-service-key']);
return;
}
// Check a source's own required flags before asking for a token, so the
// first error a new user sees is about the command they typed.
if (source === 'supabase' && (!values['source-ref'] || !values['source-url'] || !values['source-service-key'])) {
fail('supabase needs --source-ref, --source-url and --source-service-key');
}
if (source === 'neon' && !values['source-url']) fail('neon needs --source-url');
if (source === 'clerk' && !values['secret-key']) fail('clerk needs --secret-key');
const target = parseTo(values.to);
if (source === 'project-info') {
const client = clientOrExit();
const dest = await resolveDestination(client, target, values);
const { client } = scribaseClient();
const dest = await resolveDestination(client, values.to, rest);
const out = {
organizationId: dest.organizationId,
projectId: dest.projectId,
@ -222,8 +152,8 @@ async function main() {
return;
}
const client = clientOrExit();
const dest = await resolveDestination(client, target, values);
const { client } = scribaseClient();
const dest = await resolveDestination(client, values.to, rest);
const destination = {
organizationId: dest.organizationId,
projectId: dest.projectId,
@ -231,6 +161,9 @@ async function main() {
};
if (source === 'supabase') {
if (!values['source-ref'] || !values['source-url'] || !values['source-service-key']) {
fail('supabase needs --source-ref, --source-url and --source-service-key');
}
const result = await migrateSupabase(client, {
sourceProjectRef: values['source-ref'],
sourceConnectionString: values['source-url'],
@ -248,6 +181,7 @@ async function main() {
}
if (source === 'neon') {
if (!values['source-url']) fail('neon needs --source-url');
const schemas = values.schema ?? [];
const branches = (values.branch ?? []).map((pair) => {
const at = pair.indexOf('=');
@ -271,38 +205,38 @@ async function main() {
}
if (source === 'clerk') {
if (!values['secret-key']) fail('clerk needs --secret-key');
const apiUrl = values['target-api-url'] ?? dest.apiKeys.api_url;
const serviceRoleKey = values['target-service-key'] ?? dest.apiKeys.service_role_key;
if (!apiUrl || !serviceRoleKey) fail('could not resolve the destination api_url/service_role_key. Pass --target-api-url and --target-service-key');
await runClerk(values, apiUrl, serviceRoleKey);
}
}
async function runClerk(values, apiUrl, serviceRoleKey) {
const onLog = (line) => process.stderr.write(` ${line}\n`);
const users = await exportClerkUsers({ secretKey: values['secret-key'], onLog });
if (values['dry-run']) {
await writeResult(values, { exported: users.length, dryRun: true, wouldCreate: users.map((u) => u.email ?? u.phone ?? u.clerkId) });
if (!apiUrl || !serviceRoleKey) fail('could not resolve the destination api_url/service_role_key — pass --target-api-url/--target-service-key');
const users = await exportClerkUsers({
secretKey: values['secret-key'],
onLog: (line) => process.stderr.write(` ${line}\n`),
});
const result = await createGoTrueUsers({
apiUrl,
serviceRoleKey,
users,
onLog: (line) => process.stderr.write(` ${line}\n`),
});
await writeResult(values, { exported: users.length, ...result });
return;
}
const result = await createGoTrueUsers({ apiUrl, serviceRoleKey, users, onLog });
await writeResult(values, { exported: users.length, ...result });
}
/** The control plane's own detail line, plus the one fix that applies, for errors a new user hits first. */
function explain(error) {
const detail = error?.problem?.detail;
const lines = [detail ? `${error.message} (${detail})` : error.message];
if (error?.code === 'migration_conflict') {
lines.push(
'This environment already has migrations that are not in ./supabase/migrations here. ' +
'Run from the folder that holds them (your app repo), or pick a new environment in --to.',
);
if (source === 'resend') {
if (!values['api-key']) fail('resend needs --api-key');
const result = await exportResend({
apiKey: values['api-key'],
onLog: (line) => process.stderr.write(` ${line}\n`),
});
await writeResult(values, result);
return;
}
return lines;
fail(`unknown source "${source}" — one of: supabase, neon, clerk, resend, project-info`);
}
main().catch((error) => {
for (const line of explain(error)) process.stderr.write(`scribase-migrate: ${line}\n`);
process.stderr.write(`scribase-migrate: ${error.message}\n`);
process.exitCode = 1;
});

View File

@ -13,7 +13,7 @@ Scribase — no manual `pg_dump`, no hand-copying users, no re-typing DNS
records.
```bash
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git <source> --to <org>/<project> [flags]
npx scribase-migrate <source> --to <org>/<project> [flags]
```
Run `npx scribase login` once first. `scribase-migrate` finds the same
@ -22,7 +22,7 @@ token `scribase` itself does.
## From Supabase
```bash
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git supabase \
npx scribase-migrate supabase \
--to acme/storefront --create-project \
--source-ref abcdefghijklmnop \
--source-url "postgres://postgres:...@db.abcdefghijklmnop.supabase.co:5432/postgres" \
@ -39,7 +39,7 @@ to see the plan first; add `--no-storage` to skip buckets.
## From Neon
```bash
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git neon \
npx scribase-migrate neon \
--to acme/storefront \
--source-url "postgres://...@ep-xxx.region.aws.neon.tech/neondb" \
--migrate-auth-users \
@ -59,7 +59,7 @@ This writes into `supabase/migrations/` — the same folder
## From Clerk
```bash
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git clerk --to acme/storefront --secret-key sk_live_...
npx scribase-migrate clerk --to acme/storefront --secret-key sk_live_...
```
Creates each Clerk user on Scribase with a one-time password (Clerk never
@ -71,7 +71,7 @@ needs that email.
## From Resend
```bash
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git resend --api-key re_... --out resend-export.json
npx scribase-migrate resend --api-key re_... --out resend-export.json
```
Exports every sending domain (with its DNS records) and broadcast as a

View File

@ -39,14 +39,10 @@ export async function readMigrationFiles(dir) {
return files;
}
export function slugify(name) {
return name.replace(/[^a-z0-9]+/gi, '_').replace(/^_+|_+$/g, '').toLowerCase();
}
/** Writes one step, skipping (not overwriting) a version that already has a file. */
export async function writeMigrationFile(dir, { version, name, sql }) {
await mkdir(dir, { recursive: true });
const slug = slugify(name);
const slug = name.replace(/[^a-z0-9]+/gi, '_').replace(/^_+|_+$/g, '').toLowerCase();
const filename = `${version}_${slug}.sql`;
const path = join(dir, filename);
const existing = await readMigrationFiles(dir);

View File

@ -58,18 +58,7 @@ function spawnCapture(cmd, args) {
child.stderr.on('data', (c) => {
stderr += c.toString();
});
child.on('error', (error) => {
if (error.code === 'ENOENT') {
reject(
new Error(
`"${cmd}" was not found. This source needs the Postgres client tools: ` +
'install them (macOS: brew install libpq or postgresql; Debian/Ubuntu: apt install postgresql-client) and re-run.',
),
);
return;
}
reject(error);
});
child.on('error', reject);
child.on('close', (code) => resolve({ stdout: Buffer.concat(chunks).toString('utf8'), stderr, code }));
});
}
@ -79,10 +68,6 @@ function cleanDumpSql(text, { dropCreateSchema } = {}) {
const lines = text.split('\n').filter((line) => {
if (line.startsWith('\\restrict') || line.startsWith('\\unrestrict')) return false;
if (line.startsWith('SET ') || line.startsWith('SELECT pg_catalog.set_config')) return false;
// Version banners change with every minor release of the server or of
// pg_dump; left in, an otherwise identical re-run produces different
// bytes and can no longer be recognised as the step already applied.
if (line.startsWith('-- Dumped from database version') || line.startsWith('-- Dumped by pg_dump version')) return false;
if (dropCreateSchema) {
const schema = dropCreateSchema;
if (line.trim() === `CREATE SCHEMA ${schema};`) return false;

View File

@ -29,14 +29,6 @@ export async function ensureProject(client, opts) {
environment = null;
}
// A deleted environment lingers as a tombstone and its name cannot be
// reused (verified live: re-creating one answers 409 conflict).
if (environment?.phase === 'deleted') {
throw new Error(
`environment "${environmentId}" in ${organizationId}/${projectId} was deleted and its name cannot be reused. Pick a different environment name in --to.`,
);
}
if (!environment) {
const accepted = await client.environments.create({
organization_id: organizationId,

View File

@ -15,11 +15,8 @@
import pg from 'pg';
import { dumpSchemaSql, dumpDataSql, dumpAuthUsersSql, countRowsDirect } from '../postgres.mjs';
import { introspectSchema, generateRlsStatements } from '../rls.mjs';
import { readMigrationFiles, writeMigrationFile, nextVersion, submitMigrationsDir, slugify, MIGRATIONS_DIR } from '../migrationFiles.mjs';
import { readMigrationFiles, writeMigrationFile, nextVersion, submitMigrationsDir, MIGRATIONS_DIR } from '../migrationFiles.mjs';
import { join } from 'node:path';
import { cp, mkdtemp, rm } from 'node:fs/promises';
import { existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
async function hasAuthUsersTable(sourceUrl) {
const client = new pg.Client({ connectionString: sourceUrl });
@ -34,32 +31,13 @@ async function hasAuthUsersTable(sourceUrl) {
}
}
/**
* Adds one step file, unless the folder already holds a step with the same
* name and byte-identical SQL. Without that check a second run against the
* same source wrote the schema step again under a new version, and the
* control plane then tried to create every table twice.
*/
async function addStep(dir, name, sql) {
if (!sql || !sql.trim()) return null;
const existing = await readMigrationFiles(dir);
const same = existing.find((f) => f.name === slugify(name) && f.up === sql);
if (same) return { path: join(dir, `${same.version}_${same.name}.sql`), written: false };
const version = nextVersion(existing, existing.length);
return writeMigrationFile(dir, { version, name, sql });
}
/**
* A dry run must leave the app's real `supabase/migrations/` folder
* untouched: a leftover planned step would be picked up, stale, by the next
* real run. The plan is built in a throwaway copy instead.
*/
async function dryRunCopy(realDir) {
const scratch = await mkdtemp(join(tmpdir(), 'scribase-migrate-dry-run-'));
if (existsSync(realDir)) await cp(realDir, scratch, { recursive: true });
return scratch;
}
/**
* @param {import('scribase').ScribaseClient} client
* @param {{
@ -78,25 +56,8 @@ export async function migrateNeon(client, opts) {
const apply = opts.apply ?? true;
const schemas = opts.schemas?.length ? opts.schemas : ['public'];
const ref = opts.destination;
const realDir = opts.migrationsDir ?? join(process.cwd(), MIGRATIONS_DIR);
const dir = apply ? realDir : await dryRunCopy(realDir);
const results = {
migrationsDir: realDir,
dryRun: !apply,
...(apply ? {} : { note: 'dry run: nothing applied, nothing written to the migrations folder' }),
authUsers: null,
perSchema: [],
branches: [],
rlsDecisions: [],
};
try {
return await runSteps(client, opts, { onLog, apply, schemas, ref, dir, results });
} finally {
if (!apply) await rm(dir, { recursive: true, force: true });
}
}
async function runSteps(client, opts, { onLog, apply, schemas, ref, dir, results }) {
const dir = opts.migrationsDir ?? join(process.cwd(), MIGRATIONS_DIR);
const results = { migrationsDir: dir, authUsers: null, perSchema: [], branches: [], rlsDecisions: [] };
if (opts.migrateAuthUsers) {
if (await hasAuthUsersTable(opts.sourceUrl)) {

4
package-lock.json generated
View File

@ -1,12 +1,12 @@
{
"name": "scribase-migrate",
"version": "0.1.1",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "scribase-migrate",
"version": "0.1.1",
"version": "0.1.0",
"license": "MIT",
"dependencies": {
"bcryptjs": "^2.4.3",

View File

@ -1,6 +1,6 @@
{
"name": "scribase-migrate",
"version": "0.1.1",
"version": "0.1.0",
"description": "One-command migration from Supabase, Neon, Clerk and Resend onto Scribase (Postgres + auth + storage + email, one account).",
"license": "MIT",
"type": "module",
@ -11,8 +11,7 @@
"bin",
"lib",
"docs",
"README.md",
"LICENSE"
"README.md"
],
"engines": {
"node": ">=18"

View File

@ -1,45 +0,0 @@
# v0.1.1 re-verification (2026-11-17)
Source: throwaway Postgres 16.16 (`initdb` in `/tmp`), seeded with `test-fixtures/seed-source.sql`.
Destination: a real Scribase preview environment, `ereboslab/scribase-migrate-wave8c/try1`,
created by the tool itself and deleted afterwards.
```
$ scribase-migrate neon --to ereboslab/scribase-migrate-wave8c/try1 --create-project --kind preview \
--source-url postgres://postgres@127.0.0.1:55439/src --migrate-auth-users \
--verify-row-counts public.profiles --verify-row-counts public.notes --out run1.json
exporting auth.users rows
auth.users: 2 rows, applied=true
dumping public schema
public schema: 0 lint findings, applied=true
dumping public data
public data: applied=true
```
35 seconds wall clock including project and environment creation. Row counts: `public.profiles`
2/2, `public.notes` 3/3, both `match: true`.
Second identical run: no new migration files, `applied=false` for all three steps, counts still
match. After changing one source password hash and running a third time: exactly one new
`import_auth_users` step, applied; `POST /auth/v1/token?grant_type=password` on the destination
then returned `200` with an access token for the new password and `400 Invalid login credentials`
for a wrong one.
`--dry-run` against `ereboslab/scribase-migrate-verify-internal/production` returned a 3-step plan
with 0 lint findings and left `supabase/migrations/` untouched.
Bugs found by running it the way the README said, all fixed in 0.1.1:
- `resend` demanded `--to` and a Scribase token although it only exports a JSON file.
- `--create-project` without `--region` read the region from the first argument on the command
line (it sent `--to` or the source name as the region).
- `--create-project --kind preview` failed on a new project with `source_environment_missing`
(it asked for a production snapshot that does not exist yet).
- `clerk --dry-run` ignored the flag and created users.
- `neon --dry-run` wrote planned steps into the real `supabase/migrations/` folder.
- A second `neon` run wrote the schema step again under a new version, so the destination was
asked to create every table twice. `pg_dump` version banners also made identical dumps differ.
- `<source> --help` and `--version` answered "--to is required".
- A missing `pg_dump` surfaced as `spawn pg_dump ENOENT`.
- With no token, nothing told a new user where to get an account.
- The README told people to run `npx scribase-migrate`, which is not on npm.