Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9c406ac3a0 | |||
| d7bbaf483b |
21
LICENSE
Normal file
21
LICENSE
Normal file
@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 caelum0x
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
162
README.md
162
README.md
@ -1,36 +1,105 @@
|
||||
# scribase-migrate
|
||||
|
||||
One command to move a team onto [Scribase](https://scribase.com) — the one
|
||||
account for Postgres, auth, storage and email — from Supabase, Neon, Clerk
|
||||
and Resend.
|
||||
Move a Supabase, Neon, Clerk or Resend setup onto [Scribase](https://scribase.com/migrate?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate)
|
||||
with one command. Scribase is Postgres, auth, storage and email on one account instead of four.
|
||||
|
||||
## 10-second quickstart
|
||||
|
||||
```
|
||||
npx scribase-migrate supabase --to <org>/<project> --create-project \
|
||||
npx scribase login
|
||||
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git neon \
|
||||
--to <org>/<project>/try1 --create-project --kind preview \
|
||||
--source-url "$DATABASE_URL" --migrate-auth-users
|
||||
```
|
||||
|
||||
That creates an empty preview environment and moves your schema, data and users into it. Your
|
||||
source database is only read, never changed. No account yet?
|
||||
[It is free to start](https://console.scribase.com/signup?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate).
|
||||
|
||||
Add `--dry-run` to see the plan without applying anything. The `neon` source works for any plain
|
||||
Postgres and needs `pg_dump` on your machine. Node 18+.
|
||||
|
||||
## What you get
|
||||
|
||||
Real output, 2026-11-17: a Postgres 16 database with 2 users, 2 profiles and 3 notes, moved into
|
||||
a brand-new Scribase preview environment. Project creation included, the whole run took 35
|
||||
seconds.
|
||||
|
||||
```
|
||||
exporting auth.users rows
|
||||
auth.users: 2 rows, applied=true
|
||||
dumping public schema
|
||||
public schema: 0 lint findings, applied=true
|
||||
dumping public data
|
||||
public data: applied=true
|
||||
|
||||
Full result also written to run1.json
|
||||
|
||||
Scribase: Postgres, auth, storage and email on one account. Migration guide: https://scribase.com/migrate?utm_source=cli&utm_medium=cli-output&utm_campaign=scribase-migrate
|
||||
```
|
||||
|
||||
From the JSON result of the same run (`--verify-row-counts public.profiles --verify-row-counts public.notes`):
|
||||
|
||||
```json
|
||||
"rowCounts": [
|
||||
{ "table": "public.profiles", "sourceRows": 2, "targetRows": 2, "match": true },
|
||||
{ "table": "public.notes", "sourceRows": 3, "targetRows": 3, "match": true }
|
||||
],
|
||||
"rlsDecisions": [
|
||||
{ "schema": "public", "table": "notes", "kind": "join", "detail": "profile_id -> public.profiles.id -> id = auth.uid()" },
|
||||
{ "schema": "public", "table": "profiles", "kind": "direct", "detail": "id = auth.uid()" }
|
||||
]
|
||||
```
|
||||
|
||||
A migrated user then signed in on the new environment with their old password (`200` with an
|
||||
access token; a wrong password got `400 Invalid login credentials`). Running the same command a
|
||||
second time applied nothing and wrote no new files.
|
||||
|
||||
## The four sources
|
||||
|
||||
```
|
||||
scribase-migrate supabase --to <org>/<project> --create-project \
|
||||
--source-ref <ref> --source-url <connection-string> --source-service-key <key>
|
||||
|
||||
npx scribase-migrate neon --to <org>/<project> \
|
||||
--source-url <connection-string> --migrate-auth-users
|
||||
scribase-migrate neon --to <org>/<project> --source-url <connection-string> --migrate-auth-users
|
||||
|
||||
npx scribase-migrate clerk --to <org>/<project> --secret-key <sk_...>
|
||||
scribase-migrate clerk --to <org>/<project> --secret-key <sk_...>
|
||||
|
||||
npx scribase-migrate resend --api-key <re_...> --out resend-export.json
|
||||
scribase-migrate resend --api-key <re_...> --out resend-export.json
|
||||
```
|
||||
|
||||
Run `npx scribase-migrate --help` for the full flag reference.
|
||||
Prefix each with `npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git` (Bun:
|
||||
`bunx -p git+https://rithub.dev/caelum0x/scribase-migrate.git scribase-migrate`), or clone the
|
||||
repo and run `node bin/scribase-migrate.mjs`. `--help` prints every flag.
|
||||
|
||||
It needs a Scribase access token the same way `scribase` itself does — run
|
||||
`npx scribase login` once, or export `SCRIBASE_ACCESS_TOKEN`.
|
||||
|
||||
## What each source actually moves
|
||||
|
||||
| Source | Postgres schema+data | Auth users / passwords | Storage | Notes |
|
||||
| Source | Postgres schema + data | Auth users / passwords | Storage | Notes |
|
||||
|---|---|---|---|---|
|
||||
| **Supabase** | yes | yes (hash preserved) | yes | one real API call, Scribase's own `POST /v1/imports` |
|
||||
| **Neon** | yes | yes, if the source has its own `auth.users` table | — | Neon has no storage/auth of its own; branches land in `branch_<name>` schemas |
|
||||
| **Clerk** | — | yes, but a **new** one-time password (Clerk never exposes a portable hash) | — | every migrated user is flagged `password_reset_required` |
|
||||
| **Resend** | — | — | — | exports sending domains (DNS records) + broadcasts as "templates" to a JSON bundle; Scribase has no write endpoint for either yet, so this is import-ready, not auto-applied |
|
||||
| **Supabase** | yes | yes (hash preserved) | yes | one API call, Scribase's own `POST /v1/imports` |
|
||||
| **Neon** | yes | yes, if the source has its own `auth.users` table | no | Neon has no storage or auth of its own; branches land in `branch_<name>` schemas |
|
||||
| **Clerk** | no | yes, with a **new** one-time password (Clerk never exposes a portable hash) | no | every migrated user is flagged `password_reset_required` |
|
||||
| **Resend** | no | no | no | exports sending domains (DNS records) and broadcasts to a JSON bundle. No Scribase account needed. Scribase has no import endpoint for either yet, so this is import-ready, not auto-applied |
|
||||
|
||||
## Why the design looks like this (real findings from building it)
|
||||
Everything except `resend` needs a Scribase access token: `npx scribase login` once, or export
|
||||
`SCRIBASE_ACCESS_TOKEN`.
|
||||
|
||||
## What is verified and what is not
|
||||
|
||||
**Verified live** against a real Scribase organization, twice (first build, then the 0.1.1
|
||||
re-run above): the `neon` source end to end, including `--create-project`, generated RLS passing
|
||||
Scribase's migration lint with 0 blocking findings, row-count parity over HTTP, a real password
|
||||
login, `--dry-run`, and an idempotent second run. The `resend` source reaches the real Resend API
|
||||
(a fake key gets Resend's own `400 API key is invalid`). Artifacts:
|
||||
[test-fixtures/verification-run](https://rithub.dev/caelum0x/scribase-migrate/src/branch/main/test-fixtures/verification-run).
|
||||
|
||||
**Not verified live**, because each needs a provider account this build did not have:
|
||||
|
||||
- `supabase`: the request shape was confirmed against the real API with a dry run (a fake project
|
||||
ref is correctly rejected with `400 invalid_project_ref`), but the full one-call import needs a
|
||||
real, network-reachable Supabase project.
|
||||
- `clerk` and a full `resend` export: checked against a local mock of each provider's documented
|
||||
API shape, not a real account.
|
||||
|
||||
## Why it works this way
|
||||
|
||||
This was built by composing Scribase's own already-shipped APIs — the
|
||||
`scribase` npm SDK (`edge-repos/scribase-cli`) and the GoTrue admin API every
|
||||
@ -103,42 +172,11 @@ probes the source's major version and prefers an exact-match Homebrew keg
|
||||
(`postgresql@<major>`) over whatever `pg_dump` is on `PATH`, warning when
|
||||
it has to fall back.
|
||||
|
||||
## Real verification (not a test file — an actual live run)
|
||||
|
||||
This was verified against a real local Supabase-shaped Postgres and a
|
||||
real, live Scribase project, not mocked:
|
||||
|
||||
1. A real Postgres 16 Docker container seeded with a GoTrue-shaped
|
||||
`auth.users` table (two users, one password hashed with real `bcryptjs`),
|
||||
`public.profiles` (owner = `auth.users.id`) and `public.notes`
|
||||
(owner one hop through `profiles`) — `test-fixtures/seed-source.sql`.
|
||||
2. `npx scribase-migrate neon --to ereboslab/scribase-migrate-verify-internal/production --source-url ... --migrate-auth-users --schema public` ran for real against a live Scribase org (`ereboslab`), through the real `scribase` SDK with a real `scribase login` token.
|
||||
3. The schema step generated RLS automatically (`notes` via the
|
||||
profiles→auth.uid() join rule, `profiles` directly) and passed Scribase's
|
||||
real migration lint with **0 blocking findings**.
|
||||
4. Row counts verified with a real HTTP call (`client.database.query`,
|
||||
not a local connection) against the live environment:
|
||||
`public.profiles` 2/2, `public.notes` 3/3, both `match: true`.
|
||||
5. **Login verified for real**: `POST /auth/v1/token?grant_type=password`
|
||||
against the live environment's real `api_url` — the correct password
|
||||
returns `200` with a real `access_token`; a wrong password returns
|
||||
`400 Invalid login credentials`. The password hash moved byte-for-byte
|
||||
and still verifies.
|
||||
|
||||
Full before/after artifacts: `test-fixtures/verification-run/`.
|
||||
|
||||
**Not verified live** (both require a provider account/test credentials
|
||||
this environment doesn't have — logged honestly rather than faked):
|
||||
the `supabase` source's real one-call `POST /v1/imports` (its own request
|
||||
shape was confirmed against the real API with a dry-run, which correctly
|
||||
rejected a fake project ref with `400 invalid_project_ref`, but needs a
|
||||
real reachable Supabase project — Scribase's control plane connects
|
||||
*to* the source, so it must be a real, network-reachable Postgres, not a
|
||||
local Docker container, to run the full one-call migration this source
|
||||
wraps); the `clerk` and `resend` sources (logic-verified against a local
|
||||
mock of each provider's documented API shape — pagination, the GoTrue
|
||||
admin-create request body, the domains/broadcasts export — but not run
|
||||
against a real Clerk or Resend account).
|
||||
**Why a preview environment starts empty.** A `snapshot` or `sanitized` environment copies
|
||||
production data, so the control plane refuses one in a project that has no production
|
||||
environment yet (`source_environment_missing`). `--create-project --kind preview` therefore
|
||||
creates a `schema_only` environment. A deleted environment's name cannot be reused (`409`), so
|
||||
pick a new name each time you try this out.
|
||||
|
||||
## Repo layout
|
||||
|
||||
@ -156,8 +194,12 @@ against a real Clerk or Resend account).
|
||||
- `docs/migrate-page.md` — the scribase.com/migrate docs-page snippet
|
||||
- `test-fixtures/` — the real local-verification seed + its output
|
||||
|
||||
## scribase.com/migrate
|
||||
## Docs
|
||||
|
||||
`docs/migrate-page.md` is a ready-to-paste docs page for
|
||||
`scribase.com/migrate` (the `scribase` checkout's owner adds it to
|
||||
`apps/docs` and deploys — not done by this repo).
|
||||
[docs/migrate-page.md](https://rithub.dev/caelum0x/scribase-migrate/src/branch/main/docs/migrate-page.md)
|
||||
is the source of the migration guide on
|
||||
[scribase.com/migrate](https://scribase.com/migrate?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate).
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFile, mkdir } from 'node:fs/promises';
|
||||
import { writeFile, mkdir, readFile } from 'node:fs/promises';
|
||||
import { dirname } from 'node:path';
|
||||
|
||||
import { scribaseClient } from '../lib/client.mjs';
|
||||
@ -16,6 +16,13 @@ import { migrateNeon, verifyRowCounts } from '../lib/sources/neon.mjs';
|
||||
import { exportClerkUsers, createGoTrueUsers } from '../lib/sources/clerk.mjs';
|
||||
import { exportResend } from '../lib/sources/resend.mjs';
|
||||
|
||||
const CTA_URL = 'https://scribase.com/migrate?utm_source=cli&utm_medium=cli-output&utm_campaign=scribase-migrate';
|
||||
const CTA_LINE = `Scribase: Postgres, auth, storage and email on one account. Migration guide: ${CTA_URL}`;
|
||||
const SIGNUP_URL = 'https://console.scribase.com/signup?utm_source=cli&utm_medium=cli-output&utm_campaign=scribase-migrate';
|
||||
const REGION_DEFAULT = 'us-east';
|
||||
const KIND_DEFAULT = 'production';
|
||||
const SOURCES = ['supabase', 'neon', 'clerk', 'resend', 'project-info'];
|
||||
|
||||
const USAGE = `scribase-migrate <source> [flags]
|
||||
|
||||
Sources:
|
||||
@ -26,6 +33,7 @@ Sources:
|
||||
|
||||
Shared destination flags:
|
||||
--to <org>/<project>/<environment> Scribase destination (default environment: production)
|
||||
Not needed for resend, or for clerk with both --target-* flags
|
||||
--create-project Create the project+environment first if missing
|
||||
--region <region> Region for --create-project (default us-east)
|
||||
--kind <preview|staging|production> Environment kind for --create-project (default production;
|
||||
@ -39,7 +47,7 @@ supabase flags:
|
||||
--verify-login-email <email> [--verify-login-password <password>]
|
||||
--no-storage
|
||||
|
||||
neon flags:
|
||||
neon flags (also works for any plain Postgres; needs pg_dump installed):
|
||||
--source-url <postgres://...>
|
||||
--schema <name> (repeatable; default public)
|
||||
--migrate-auth-users Also copy an auth.users-shaped table's rows
|
||||
@ -50,12 +58,18 @@ clerk flags:
|
||||
--secret-key <sk_...>
|
||||
--target-api-url <url> --target-service-key <key> (from \`scribase-migrate project-info\`)
|
||||
|
||||
resend flags:
|
||||
resend flags (no Scribase account needed; pair with --out <file>):
|
||||
--api-key <re_...>
|
||||
|
||||
Also: scribase-migrate project-info --to <org>/<project>/<environment>
|
||||
prints the destination's api_url + keys (never the secrets to stdout
|
||||
unless --reveal-secrets), the inputs clerk/resend need.
|
||||
unless --reveal-secrets), the inputs clerk needs.
|
||||
|
||||
Needs a Scribase access token for everything except resend: run
|
||||
\`npx scribase login\` once, or export SCRIBASE_ACCESS_TOKEN.
|
||||
No account yet? It is free to start: ${SIGNUP_URL}
|
||||
|
||||
${CTA_LINE}
|
||||
`;
|
||||
|
||||
function fail(message) {
|
||||
@ -63,6 +77,25 @@ function fail(message) {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function version() {
|
||||
try {
|
||||
return JSON.parse(await readFile(new URL('../package.json', import.meta.url), 'utf8')).version;
|
||||
} catch {
|
||||
return 'unknown';
|
||||
}
|
||||
}
|
||||
|
||||
/** The SDK client, or a plain exit that tells a brand-new user where to get an account. */
|
||||
function clientOrExit() {
|
||||
try {
|
||||
return scribaseClient().client;
|
||||
} catch (error) {
|
||||
process.stderr.write(`scribase-migrate: ${error.message}\n`);
|
||||
process.stderr.write(`No Scribase account yet? It is free to start: ${SIGNUP_URL}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
function parseTo(value) {
|
||||
if (!value) fail('--to <org>/<project>/<environment> is required');
|
||||
const parts = value.split('/');
|
||||
@ -76,22 +109,23 @@ async function writeResult(opts, result) {
|
||||
await mkdir(dirname(opts.out), { recursive: true }).catch(() => {});
|
||||
await writeFile(opts.out, JSON.stringify(result, null, 2));
|
||||
}
|
||||
if (opts.json) {
|
||||
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
|
||||
} else {
|
||||
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
|
||||
if (opts.out) process.stderr.write(`\nFull result also written to ${opts.out}\n`);
|
||||
}
|
||||
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
|
||||
if (opts.json) return;
|
||||
// stdout stays pure JSON for pipes; the human-facing lines go to stderr.
|
||||
if (opts.out) process.stderr.write(`\nFull result also written to ${opts.out}\n`);
|
||||
process.stderr.write(`\n${CTA_LINE}\n`);
|
||||
}
|
||||
|
||||
async function resolveDestination(client, destArg, rawArgs) {
|
||||
const dest = parseTo(destArg);
|
||||
if (rawArgs.includes('--create-project')) {
|
||||
const region = rawArgs[rawArgs.indexOf('--region') + 1] ?? 'us-east';
|
||||
const kindIndex = rawArgs.indexOf('--kind');
|
||||
const kind = kindIndex >= 0 ? rawArgs[kindIndex + 1] : 'production';
|
||||
const project = await ensureProject(client, { ...dest, region, kind, dataMode: 'snapshot' });
|
||||
return project;
|
||||
async function resolveDestination(client, dest, values) {
|
||||
if (values['create-project']) {
|
||||
const region = values.region ?? REGION_DEFAULT;
|
||||
const kind = values.kind ?? KIND_DEFAULT;
|
||||
if (!['preview', 'staging', 'production'].includes(kind)) fail(`--kind "${kind}" must be preview, staging or production`);
|
||||
// snapshot/sanitized copy production data, so the control plane refuses
|
||||
// them for a project with no production environment yet (verified live:
|
||||
// `source_environment_missing`). A try-it-out preview starts empty.
|
||||
const dataMode = kind === 'production' ? 'snapshot' : 'schema_only';
|
||||
return ensureProject(client, { ...dest, region, kind, dataMode });
|
||||
}
|
||||
const environment = await client.environments.get(dest);
|
||||
const connection = await client.environments.connection(dest);
|
||||
@ -101,10 +135,17 @@ async function resolveDestination(client, destArg, rawArgs) {
|
||||
|
||||
async function main() {
|
||||
const [source, ...rest] = process.argv.slice(2);
|
||||
if (!source || source === '--help' || source === '-h') {
|
||||
if (!source || source === '--help' || source === '-h' || rest.includes('--help') || rest.includes('-h')) {
|
||||
process.stdout.write(USAGE);
|
||||
process.exit(source ? 0 : 1);
|
||||
}
|
||||
if (source === '--version' || source === '-v') {
|
||||
process.stdout.write(`${await version()}\n`);
|
||||
return;
|
||||
}
|
||||
if (!SOURCES.includes(source)) {
|
||||
fail(`unknown source "${source}". Use one of: ${SOURCES.join(', ')}. Run scribase-migrate --help.`);
|
||||
}
|
||||
|
||||
const { values } = parseArgs({
|
||||
args: rest,
|
||||
@ -112,6 +153,7 @@ async function main() {
|
||||
to: { type: 'string' },
|
||||
'create-project': { type: 'boolean' },
|
||||
region: { type: 'string' },
|
||||
kind: { type: 'string' },
|
||||
'dry-run': { type: 'boolean' },
|
||||
out: { type: 'string' },
|
||||
json: { type: 'boolean' },
|
||||
@ -135,9 +177,37 @@ async function main() {
|
||||
strict: false,
|
||||
});
|
||||
|
||||
// Resend is a pure export: it needs neither a destination nor a Scribase
|
||||
// account, so it must not ask for either.
|
||||
if (source === 'resend') {
|
||||
if (!values['api-key']) fail('resend needs --api-key');
|
||||
const result = await exportResend({
|
||||
apiKey: values['api-key'],
|
||||
onLog: (line) => process.stderr.write(` ${line}\n`),
|
||||
});
|
||||
await writeResult(values, result);
|
||||
return;
|
||||
}
|
||||
|
||||
// Clerk with both --target-* flags talks to the environment directly.
|
||||
if (source === 'clerk' && values['target-api-url'] && values['target-service-key']) {
|
||||
if (!values['secret-key']) fail('clerk needs --secret-key');
|
||||
await runClerk(values, values['target-api-url'], values['target-service-key']);
|
||||
return;
|
||||
}
|
||||
|
||||
// Check a source's own required flags before asking for a token, so the
|
||||
// first error a new user sees is about the command they typed.
|
||||
if (source === 'supabase' && (!values['source-ref'] || !values['source-url'] || !values['source-service-key'])) {
|
||||
fail('supabase needs --source-ref, --source-url and --source-service-key');
|
||||
}
|
||||
if (source === 'neon' && !values['source-url']) fail('neon needs --source-url');
|
||||
if (source === 'clerk' && !values['secret-key']) fail('clerk needs --secret-key');
|
||||
const target = parseTo(values.to);
|
||||
|
||||
if (source === 'project-info') {
|
||||
const { client } = scribaseClient();
|
||||
const dest = await resolveDestination(client, values.to, rest);
|
||||
const client = clientOrExit();
|
||||
const dest = await resolveDestination(client, target, values);
|
||||
const out = {
|
||||
organizationId: dest.organizationId,
|
||||
projectId: dest.projectId,
|
||||
@ -152,8 +222,8 @@ async function main() {
|
||||
return;
|
||||
}
|
||||
|
||||
const { client } = scribaseClient();
|
||||
const dest = await resolveDestination(client, values.to, rest);
|
||||
const client = clientOrExit();
|
||||
const dest = await resolveDestination(client, target, values);
|
||||
const destination = {
|
||||
organizationId: dest.organizationId,
|
||||
projectId: dest.projectId,
|
||||
@ -161,9 +231,6 @@ async function main() {
|
||||
};
|
||||
|
||||
if (source === 'supabase') {
|
||||
if (!values['source-ref'] || !values['source-url'] || !values['source-service-key']) {
|
||||
fail('supabase needs --source-ref, --source-url and --source-service-key');
|
||||
}
|
||||
const result = await migrateSupabase(client, {
|
||||
sourceProjectRef: values['source-ref'],
|
||||
sourceConnectionString: values['source-url'],
|
||||
@ -181,7 +248,6 @@ async function main() {
|
||||
}
|
||||
|
||||
if (source === 'neon') {
|
||||
if (!values['source-url']) fail('neon needs --source-url');
|
||||
const schemas = values.schema ?? [];
|
||||
const branches = (values.branch ?? []).map((pair) => {
|
||||
const at = pair.indexOf('=');
|
||||
@ -205,38 +271,38 @@ async function main() {
|
||||
}
|
||||
|
||||
if (source === 'clerk') {
|
||||
if (!values['secret-key']) fail('clerk needs --secret-key');
|
||||
const apiUrl = values['target-api-url'] ?? dest.apiKeys.api_url;
|
||||
const serviceRoleKey = values['target-service-key'] ?? dest.apiKeys.service_role_key;
|
||||
if (!apiUrl || !serviceRoleKey) fail('could not resolve the destination api_url/service_role_key — pass --target-api-url/--target-service-key');
|
||||
const users = await exportClerkUsers({
|
||||
secretKey: values['secret-key'],
|
||||
onLog: (line) => process.stderr.write(` ${line}\n`),
|
||||
});
|
||||
const result = await createGoTrueUsers({
|
||||
apiUrl,
|
||||
serviceRoleKey,
|
||||
users,
|
||||
onLog: (line) => process.stderr.write(` ${line}\n`),
|
||||
});
|
||||
await writeResult(values, { exported: users.length, ...result });
|
||||
if (!apiUrl || !serviceRoleKey) fail('could not resolve the destination api_url/service_role_key. Pass --target-api-url and --target-service-key');
|
||||
await runClerk(values, apiUrl, serviceRoleKey);
|
||||
}
|
||||
}
|
||||
|
||||
async function runClerk(values, apiUrl, serviceRoleKey) {
|
||||
const onLog = (line) => process.stderr.write(` ${line}\n`);
|
||||
const users = await exportClerkUsers({ secretKey: values['secret-key'], onLog });
|
||||
if (values['dry-run']) {
|
||||
await writeResult(values, { exported: users.length, dryRun: true, wouldCreate: users.map((u) => u.email ?? u.phone ?? u.clerkId) });
|
||||
return;
|
||||
}
|
||||
const result = await createGoTrueUsers({ apiUrl, serviceRoleKey, users, onLog });
|
||||
await writeResult(values, { exported: users.length, ...result });
|
||||
}
|
||||
|
||||
if (source === 'resend') {
|
||||
if (!values['api-key']) fail('resend needs --api-key');
|
||||
const result = await exportResend({
|
||||
apiKey: values['api-key'],
|
||||
onLog: (line) => process.stderr.write(` ${line}\n`),
|
||||
});
|
||||
await writeResult(values, result);
|
||||
return;
|
||||
/** The control plane's own detail line, plus the one fix that applies, for errors a new user hits first. */
|
||||
function explain(error) {
|
||||
const detail = error?.problem?.detail;
|
||||
const lines = [detail ? `${error.message} (${detail})` : error.message];
|
||||
if (error?.code === 'migration_conflict') {
|
||||
lines.push(
|
||||
'This environment already has migrations that are not in ./supabase/migrations here. ' +
|
||||
'Run from the folder that holds them (your app repo), or pick a new environment in --to.',
|
||||
);
|
||||
}
|
||||
|
||||
fail(`unknown source "${source}" — one of: supabase, neon, clerk, resend, project-info`);
|
||||
return lines;
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`scribase-migrate: ${error.message}\n`);
|
||||
for (const line of explain(error)) process.stderr.write(`scribase-migrate: ${line}\n`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
|
||||
@ -13,7 +13,7 @@ Scribase — no manual `pg_dump`, no hand-copying users, no re-typing DNS
|
||||
records.
|
||||
|
||||
```bash
|
||||
npx scribase-migrate <source> --to <org>/<project> [flags]
|
||||
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git <source> --to <org>/<project> [flags]
|
||||
```
|
||||
|
||||
Run `npx scribase login` once first. `scribase-migrate` finds the same
|
||||
@ -22,7 +22,7 @@ token `scribase` itself does.
|
||||
## From Supabase
|
||||
|
||||
```bash
|
||||
npx scribase-migrate supabase \
|
||||
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git supabase \
|
||||
--to acme/storefront --create-project \
|
||||
--source-ref abcdefghijklmnop \
|
||||
--source-url "postgres://postgres:...@db.abcdefghijklmnop.supabase.co:5432/postgres" \
|
||||
@ -39,7 +39,7 @@ to see the plan first; add `--no-storage` to skip buckets.
|
||||
## From Neon
|
||||
|
||||
```bash
|
||||
npx scribase-migrate neon \
|
||||
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git neon \
|
||||
--to acme/storefront \
|
||||
--source-url "postgres://...@ep-xxx.region.aws.neon.tech/neondb" \
|
||||
--migrate-auth-users \
|
||||
@ -59,7 +59,7 @@ This writes into `supabase/migrations/` — the same folder
|
||||
## From Clerk
|
||||
|
||||
```bash
|
||||
npx scribase-migrate clerk --to acme/storefront --secret-key sk_live_...
|
||||
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git clerk --to acme/storefront --secret-key sk_live_...
|
||||
```
|
||||
|
||||
Creates each Clerk user on Scribase with a one-time password (Clerk never
|
||||
@ -71,7 +71,7 @@ needs that email.
|
||||
## From Resend
|
||||
|
||||
```bash
|
||||
npx scribase-migrate resend --api-key re_... --out resend-export.json
|
||||
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git resend --api-key re_... --out resend-export.json
|
||||
```
|
||||
|
||||
Exports every sending domain (with its DNS records) and broadcast as a
|
||||
|
||||
@ -39,10 +39,14 @@ export async function readMigrationFiles(dir) {
|
||||
return files;
|
||||
}
|
||||
|
||||
export function slugify(name) {
|
||||
return name.replace(/[^a-z0-9]+/gi, '_').replace(/^_+|_+$/g, '').toLowerCase();
|
||||
}
|
||||
|
||||
/** Writes one step, skipping (not overwriting) a version that already has a file. */
|
||||
export async function writeMigrationFile(dir, { version, name, sql }) {
|
||||
await mkdir(dir, { recursive: true });
|
||||
const slug = name.replace(/[^a-z0-9]+/gi, '_').replace(/^_+|_+$/g, '').toLowerCase();
|
||||
const slug = slugify(name);
|
||||
const filename = `${version}_${slug}.sql`;
|
||||
const path = join(dir, filename);
|
||||
const existing = await readMigrationFiles(dir);
|
||||
|
||||
@ -58,7 +58,18 @@ function spawnCapture(cmd, args) {
|
||||
child.stderr.on('data', (c) => {
|
||||
stderr += c.toString();
|
||||
});
|
||||
child.on('error', reject);
|
||||
child.on('error', (error) => {
|
||||
if (error.code === 'ENOENT') {
|
||||
reject(
|
||||
new Error(
|
||||
`"${cmd}" was not found. This source needs the Postgres client tools: ` +
|
||||
'install them (macOS: brew install libpq or postgresql; Debian/Ubuntu: apt install postgresql-client) and re-run.',
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
reject(error);
|
||||
});
|
||||
child.on('close', (code) => resolve({ stdout: Buffer.concat(chunks).toString('utf8'), stderr, code }));
|
||||
});
|
||||
}
|
||||
@ -68,6 +79,10 @@ function cleanDumpSql(text, { dropCreateSchema } = {}) {
|
||||
const lines = text.split('\n').filter((line) => {
|
||||
if (line.startsWith('\\restrict') || line.startsWith('\\unrestrict')) return false;
|
||||
if (line.startsWith('SET ') || line.startsWith('SELECT pg_catalog.set_config')) return false;
|
||||
// Version banners change with every minor release of the server or of
|
||||
// pg_dump; left in, an otherwise identical re-run produces different
|
||||
// bytes and can no longer be recognised as the step already applied.
|
||||
if (line.startsWith('-- Dumped from database version') || line.startsWith('-- Dumped by pg_dump version')) return false;
|
||||
if (dropCreateSchema) {
|
||||
const schema = dropCreateSchema;
|
||||
if (line.trim() === `CREATE SCHEMA ${schema};`) return false;
|
||||
|
||||
@ -29,6 +29,14 @@ export async function ensureProject(client, opts) {
|
||||
environment = null;
|
||||
}
|
||||
|
||||
// A deleted environment lingers as a tombstone and its name cannot be
|
||||
// reused (verified live: re-creating one answers 409 conflict).
|
||||
if (environment?.phase === 'deleted') {
|
||||
throw new Error(
|
||||
`environment "${environmentId}" in ${organizationId}/${projectId} was deleted and its name cannot be reused. Pick a different environment name in --to.`,
|
||||
);
|
||||
}
|
||||
|
||||
if (!environment) {
|
||||
const accepted = await client.environments.create({
|
||||
organization_id: organizationId,
|
||||
|
||||
@ -15,8 +15,11 @@
|
||||
import pg from 'pg';
|
||||
import { dumpSchemaSql, dumpDataSql, dumpAuthUsersSql, countRowsDirect } from '../postgres.mjs';
|
||||
import { introspectSchema, generateRlsStatements } from '../rls.mjs';
|
||||
import { readMigrationFiles, writeMigrationFile, nextVersion, submitMigrationsDir, MIGRATIONS_DIR } from '../migrationFiles.mjs';
|
||||
import { readMigrationFiles, writeMigrationFile, nextVersion, submitMigrationsDir, slugify, MIGRATIONS_DIR } from '../migrationFiles.mjs';
|
||||
import { join } from 'node:path';
|
||||
import { cp, mkdtemp, rm } from 'node:fs/promises';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
|
||||
async function hasAuthUsersTable(sourceUrl) {
|
||||
const client = new pg.Client({ connectionString: sourceUrl });
|
||||
@ -31,13 +34,32 @@ async function hasAuthUsersTable(sourceUrl) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Adds one step file, unless the folder already holds a step with the same
|
||||
* name and byte-identical SQL. Without that check a second run against the
|
||||
* same source wrote the schema step again under a new version, and the
|
||||
* control plane then tried to create every table twice.
|
||||
*/
|
||||
async function addStep(dir, name, sql) {
|
||||
if (!sql || !sql.trim()) return null;
|
||||
const existing = await readMigrationFiles(dir);
|
||||
const same = existing.find((f) => f.name === slugify(name) && f.up === sql);
|
||||
if (same) return { path: join(dir, `${same.version}_${same.name}.sql`), written: false };
|
||||
const version = nextVersion(existing, existing.length);
|
||||
return writeMigrationFile(dir, { version, name, sql });
|
||||
}
|
||||
|
||||
/**
|
||||
* A dry run must leave the app's real `supabase/migrations/` folder
|
||||
* untouched: a leftover planned step would be picked up, stale, by the next
|
||||
* real run. The plan is built in a throwaway copy instead.
|
||||
*/
|
||||
async function dryRunCopy(realDir) {
|
||||
const scratch = await mkdtemp(join(tmpdir(), 'scribase-migrate-dry-run-'));
|
||||
if (existsSync(realDir)) await cp(realDir, scratch, { recursive: true });
|
||||
return scratch;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import('scribase').ScribaseClient} client
|
||||
* @param {{
|
||||
@ -56,8 +78,25 @@ export async function migrateNeon(client, opts) {
|
||||
const apply = opts.apply ?? true;
|
||||
const schemas = opts.schemas?.length ? opts.schemas : ['public'];
|
||||
const ref = opts.destination;
|
||||
const dir = opts.migrationsDir ?? join(process.cwd(), MIGRATIONS_DIR);
|
||||
const results = { migrationsDir: dir, authUsers: null, perSchema: [], branches: [], rlsDecisions: [] };
|
||||
const realDir = opts.migrationsDir ?? join(process.cwd(), MIGRATIONS_DIR);
|
||||
const dir = apply ? realDir : await dryRunCopy(realDir);
|
||||
const results = {
|
||||
migrationsDir: realDir,
|
||||
dryRun: !apply,
|
||||
...(apply ? {} : { note: 'dry run: nothing applied, nothing written to the migrations folder' }),
|
||||
authUsers: null,
|
||||
perSchema: [],
|
||||
branches: [],
|
||||
rlsDecisions: [],
|
||||
};
|
||||
try {
|
||||
return await runSteps(client, opts, { onLog, apply, schemas, ref, dir, results });
|
||||
} finally {
|
||||
if (!apply) await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function runSteps(client, opts, { onLog, apply, schemas, ref, dir, results }) {
|
||||
|
||||
if (opts.migrateAuthUsers) {
|
||||
if (await hasAuthUsersTable(opts.sourceUrl)) {
|
||||
|
||||
4
package-lock.json
generated
4
package-lock.json
generated
@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "scribase-migrate",
|
||||
"version": "0.1.0",
|
||||
"version": "0.1.1",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "scribase-migrate",
|
||||
"version": "0.1.0",
|
||||
"version": "0.1.1",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"bcryptjs": "^2.4.3",
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "scribase-migrate",
|
||||
"version": "0.1.0",
|
||||
"version": "0.1.1",
|
||||
"description": "One-command migration from Supabase, Neon, Clerk and Resend onto Scribase (Postgres + auth + storage + email, one account).",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
@ -11,7 +11,8 @@
|
||||
"bin",
|
||||
"lib",
|
||||
"docs",
|
||||
"README.md"
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
|
||||
45
test-fixtures/verification-run/2026-11-17-v0.1.1-rerun.md
Normal file
45
test-fixtures/verification-run/2026-11-17-v0.1.1-rerun.md
Normal file
@ -0,0 +1,45 @@
|
||||
# v0.1.1 re-verification (2026-11-17)
|
||||
|
||||
Source: throwaway Postgres 16.16 (`initdb` in `/tmp`), seeded with `test-fixtures/seed-source.sql`.
|
||||
Destination: a real Scribase preview environment, `ereboslab/scribase-migrate-wave8c/try1`,
|
||||
created by the tool itself and deleted afterwards.
|
||||
|
||||
```
|
||||
$ scribase-migrate neon --to ereboslab/scribase-migrate-wave8c/try1 --create-project --kind preview \
|
||||
--source-url postgres://postgres@127.0.0.1:55439/src --migrate-auth-users \
|
||||
--verify-row-counts public.profiles --verify-row-counts public.notes --out run1.json
|
||||
exporting auth.users rows
|
||||
auth.users: 2 rows, applied=true
|
||||
dumping public schema
|
||||
public schema: 0 lint findings, applied=true
|
||||
dumping public data
|
||||
public data: applied=true
|
||||
```
|
||||
|
||||
35 seconds wall clock including project and environment creation. Row counts: `public.profiles`
|
||||
2/2, `public.notes` 3/3, both `match: true`.
|
||||
|
||||
Second identical run: no new migration files, `applied=false` for all three steps, counts still
|
||||
match. After changing one source password hash and running a third time: exactly one new
|
||||
`import_auth_users` step, applied; `POST /auth/v1/token?grant_type=password` on the destination
|
||||
then returned `200` with an access token for the new password and `400 Invalid login credentials`
|
||||
for a wrong one.
|
||||
|
||||
`--dry-run` against `ereboslab/scribase-migrate-verify-internal/production` returned a 3-step plan
|
||||
with 0 lint findings and left `supabase/migrations/` untouched.
|
||||
|
||||
Bugs found by running it the way the README said, all fixed in 0.1.1:
|
||||
|
||||
- `resend` demanded `--to` and a Scribase token although it only exports a JSON file.
|
||||
- `--create-project` without `--region` read the region from the first argument on the command
|
||||
line (it sent `--to` or the source name as the region).
|
||||
- `--create-project --kind preview` failed on a new project with `source_environment_missing`
|
||||
(it asked for a production snapshot that does not exist yet).
|
||||
- `clerk --dry-run` ignored the flag and created users.
|
||||
- `neon --dry-run` wrote planned steps into the real `supabase/migrations/` folder.
|
||||
- A second `neon` run wrote the schema step again under a new version, so the destination was
|
||||
asked to create every table twice. `pg_dump` version banners also made identical dumps differ.
|
||||
- `<source> --help` and `--version` answered "--to is required".
|
||||
- A missing `pg_dump` surfaced as `spawn pg_dump ENOENT`.
|
||||
- With no token, nothing told a new user where to get an account.
|
||||
- The README told people to run `npx scribase-migrate`, which is not on npm.
|
||||
Loading…
x
Reference in New Issue
Block a user