2026-11-17 20:19:20 +03:00

206 lines
11 KiB
Markdown

# scribase-migrate
Move a Supabase, Neon, Clerk or Resend setup onto [Scribase](https://scribase.com/migrate?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate)
with one command. Scribase is Postgres, auth, storage and email on one account instead of four.
## 10-second quickstart
```
npx scribase login
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git neon \
--to <org>/<project>/try1 --create-project --kind preview \
--source-url "$DATABASE_URL" --migrate-auth-users
```
That creates an empty preview environment and moves your schema, data and users into it. Your
source database is only read, never changed. No account yet?
[It is free to start](https://console.scribase.com/signup?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate).
Add `--dry-run` to see the plan without applying anything. The `neon` source works for any plain
Postgres and needs `pg_dump` on your machine. Node 18+.
## What you get
Real output, 2026-11-17: a Postgres 16 database with 2 users, 2 profiles and 3 notes, moved into
a brand-new Scribase preview environment. Project creation included, the whole run took 35
seconds.
```
exporting auth.users rows
auth.users: 2 rows, applied=true
dumping public schema
public schema: 0 lint findings, applied=true
dumping public data
public data: applied=true
Full result also written to run1.json
Scribase: Postgres, auth, storage and email on one account. Migration guide: https://scribase.com/migrate?utm_source=cli&utm_medium=cli-output&utm_campaign=scribase-migrate
```
From the JSON result of the same run (`--verify-row-counts public.profiles --verify-row-counts public.notes`):
```json
"rowCounts": [
{ "table": "public.profiles", "sourceRows": 2, "targetRows": 2, "match": true },
{ "table": "public.notes", "sourceRows": 3, "targetRows": 3, "match": true }
],
"rlsDecisions": [
{ "schema": "public", "table": "notes", "kind": "join", "detail": "profile_id -> public.profiles.id -> id = auth.uid()" },
{ "schema": "public", "table": "profiles", "kind": "direct", "detail": "id = auth.uid()" }
]
```
A migrated user then signed in on the new environment with their old password (`200` with an
access token; a wrong password got `400 Invalid login credentials`). Running the same command a
second time applied nothing and wrote no new files.
## The four sources
```
scribase-migrate supabase --to <org>/<project> --create-project \
--source-ref <ref> --source-url <connection-string> --source-service-key <key>
scribase-migrate neon --to <org>/<project> --source-url <connection-string> --migrate-auth-users
scribase-migrate clerk --to <org>/<project> --secret-key <sk_...>
scribase-migrate resend --api-key <re_...> --out resend-export.json
```
Prefix each with `npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git` (Bun:
`bunx -p git+https://rithub.dev/caelum0x/scribase-migrate.git scribase-migrate`), or clone the
repo and run `node bin/scribase-migrate.mjs`. `--help` prints every flag.
| Source | Postgres schema + data | Auth users / passwords | Storage | Notes |
|---|---|---|---|---|
| **Supabase** | yes | yes (hash preserved) | yes | one API call, Scribase's own `POST /v1/imports` |
| **Neon** | yes | yes, if the source has its own `auth.users` table | no | Neon has no storage or auth of its own; branches land in `branch_<name>` schemas |
| **Clerk** | no | yes, with a **new** one-time password (Clerk never exposes a portable hash) | no | every migrated user is flagged `password_reset_required` |
| **Resend** | no | no | no | exports sending domains (DNS records) and broadcasts to a JSON bundle. No Scribase account needed. Scribase has no import endpoint for either yet, so this is import-ready, not auto-applied |
Everything except `resend` needs a Scribase access token: `npx scribase login` once, or export
`SCRIBASE_ACCESS_TOKEN`.
## What is verified and what is not
**Verified live** against a real Scribase organization, twice (first build, then the 0.1.1
re-run above): the `neon` source end to end, including `--create-project`, generated RLS passing
Scribase's migration lint with 0 blocking findings, row-count parity over HTTP, a real password
login, `--dry-run`, and an idempotent second run. The `resend` source reaches the real Resend API
(a fake key gets Resend's own `400 API key is invalid`). Artifacts:
[test-fixtures/verification-run](https://rithub.dev/caelum0x/scribase-migrate/src/branch/main/test-fixtures/verification-run).
**Not verified live**, because each needs a provider account this build did not have:
- `supabase`: the request shape was confirmed against the real API with a dry run (a fake project
ref is correctly rejected with `400 invalid_project_ref`), but the full one-call import needs a
real, network-reachable Supabase project.
- `clerk` and a full `resend` export: checked against a local mock of each provider's documented
API shape, not a real account.
## Why it works this way
This was built by composing Scribase's own already-shipped APIs — the
`scribase` npm SDK (`edge-repos/scribase-cli`) and the GoTrue admin API every
self-hosted Supabase migration tool already uses — not by reinventing a
dump format. Three real constraints, found by actually running this against
a live Scribase project while building it, shaped the final design:
**Why not `pg_restore` straight into Scribase.** `environments.connection()`
returns a real Postgres URI, but its hostname is on Scribase's internal
network only (the SDK already documents it `SERVER-SIDE ONLY`) — it does
not resolve from an external machine. Unlike Supabase or Neon, Scribase has
no public Postgres TCP endpoint today. The Neon/generic-Postgres source
instead builds plain SQL (`pg_dump --schema-only` / `--data-only --inserts`)
and sends it through `POST .../migrations` — the same HTTPS path
`scribase migrate` itself uses, and the only one that actually reaches a
Scribase environment from outside its network.
**Why every run resends the whole `supabase/migrations/` folder.** That
endpoint is not additive. Verified live: submitting one new migration
without the version(s) already recorded answers `409 migration_conflict`
("the database records migration X but it is absent from the set on
disk"), and re-submitting an already-applied version with even one
different byte (we hit this with a stray trailing newline) answers a
different `409` ("an already-applied migration must never be edited",
with the two checksums). So `scribase-migrate neon` writes into
`supabase/migrations/` using the **exact same naming convention**
`npx scribase migrate` reads, and always reads the whole folder back and
resends it. One practical upshot: commit that folder to the app's repo,
the same as you would for `scribase migrate` on its own — it is the one
source of truth the next run (or a teammate's) needs.
**Why RLS is generated, never skipped or blanket-opened.** Scribase's
migration lint `BLOCK`s any new `public` table with neither RLS enabled nor
a policy — verified live, and a `pg_dump --schema-only` of a typical
Supabase/Neon app has neither (Neon has no RLS concept at all; many
Supabase apps relied on the service key and never wrote policies). Writing
a blanket `using (true)` policy to satisfy the lint would quietly make
every row readable by every signed-in user — worse than the block it
papers over. `lib/rls.mjs` instead infers an owner column (a direct foreign
key to `auth.users`, or one hop through another table that has one) and
writes a real owner policy; a table it cannot prove an owner for gets RLS
enabled with a `using (false)` policy — locked to the service-role key
(which bypasses RLS) until someone writes a real policy, never silently
public.
**Why Clerk users get a new password, not a migrated hash.** Clerk's
Backend API has no endpoint that returns a portable password hash on any
plan — this is a platform limit, not a corner this tool cut. Each migrated
user is created via the destination environment's GoTrue admin API
(`POST /auth/v1/admin/users`, the same endpoint any self-hosted Supabase
migration script uses) with a random one-time password and
`app_metadata.password_reset_required: true`, and is returned in
`needsPasswordReset` so the caller can send a real password-reset email.
**A sharp edge if you hand-write `auth.users` rows yourself**: GoTrue's Go
driver scans several `varchar` columns (`confirmation_token`,
`recovery_token`, `email_change*`, `phone_change*`,
`reauthentication_token`) as a plain `string`, not nullable — leaving them
at Postgres's own `NULL` default makes every later login 500 with
`converting NULL to string is unsupported`. `dumpAuthUsersSql` forces them
to `''`, which is what GoTrue itself would have written for an account
with no pending confirmation/recovery flow.
**A sharp edge in `pg_dump` itself**: `pg_dump`'s "can dump an older
server" guarantee does not cover every statement it emits. A 17.x
`pg_dump` against a 16.x source produced
`ALTER TABLE ... ALTER COLUMN ... SET (pg_attver=…)` — a PG17-only
column-storage attribute Postgres 16 rejects outright. `pgDumpBinaryFor()`
probes the source's major version and prefers an exact-match Homebrew keg
(`postgresql@<major>`) over whatever `pg_dump` is on `PATH`, warning when
it has to fall back.
**Why a preview environment starts empty.** A `snapshot` or `sanitized` environment copies
production data, so the control plane refuses one in a project that has no production
environment yet (`source_environment_missing`). `--create-project --kind preview` therefore
creates a `schema_only` environment. A deleted environment's name cannot be reused (`409`), so
pick a new name each time you try this out.
## Repo layout
- `bin/scribase-migrate.mjs` — the CLI
- `lib/client.mjs`, `lib/auth.mjs` — the `scribase` SDK + the same token
lookup `scribase` itself uses (keychain / `SCRIBASE_ACCESS_TOKEN` / login)
- `lib/project.mjs` — idempotent `--create-project`
- `lib/postgres.mjs` — `pg_dump` wrapping, version matching, the
GoTrue-safe `auth.users` export
- `lib/rls.mjs` — the RLS-inference heuristic
- `lib/migrationFiles.mjs` — the `supabase/migrations/` read-everything,
resend-everything helper
- `lib/sources/{supabase,neon,clerk,resend}.mjs` — one module per source
- `lib/login.mjs` — the real password-login proof
- `docs/migrate-page.md` — the scribase.com/migrate docs-page snippet
- `test-fixtures/` — the real local-verification seed + its output
## Docs
[docs/migrate-page.md](https://rithub.dev/caelum0x/scribase-migrate/src/branch/main/docs/migrate-page.md)
is the source of the migration guide on
[scribase.com/migrate](https://scribase.com/migrate?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate).
## License
MIT