206 lines
11 KiB
Markdown
206 lines
11 KiB
Markdown
# scribase-migrate
|
|
|
|
Move a Supabase, Neon, Clerk or Resend setup onto [Scribase](https://scribase.com/migrate?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate)
|
|
with one command. Scribase is Postgres, auth, storage and email on one account instead of four.
|
|
|
|
## 10-second quickstart
|
|
|
|
```
|
|
npx scribase login
|
|
npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git neon \
|
|
--to <org>/<project>/try1 --create-project --kind preview \
|
|
--source-url "$DATABASE_URL" --migrate-auth-users
|
|
```
|
|
|
|
That creates an empty preview environment and moves your schema, data and users into it. Your
|
|
source database is only read, never changed. No account yet?
|
|
[It is free to start](https://console.scribase.com/signup?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate).
|
|
|
|
Add `--dry-run` to see the plan without applying anything. The `neon` source works for any plain
|
|
Postgres and needs `pg_dump` on your machine. Node 18+.
|
|
|
|
## What you get
|
|
|
|
Real output, 2026-11-17: a Postgres 16 database with 2 users, 2 profiles and 3 notes, moved into
|
|
a brand-new Scribase preview environment. Project creation included, the whole run took 35
|
|
seconds.
|
|
|
|
```
|
|
exporting auth.users rows
|
|
auth.users: 2 rows, applied=true
|
|
dumping public schema
|
|
public schema: 0 lint findings, applied=true
|
|
dumping public data
|
|
public data: applied=true
|
|
|
|
Full result also written to run1.json
|
|
|
|
Scribase: Postgres, auth, storage and email on one account. Migration guide: https://scribase.com/migrate?utm_source=cli&utm_medium=cli-output&utm_campaign=scribase-migrate
|
|
```
|
|
|
|
From the JSON result of the same run (`--verify-row-counts public.profiles --verify-row-counts public.notes`):
|
|
|
|
```json
|
|
"rowCounts": [
|
|
{ "table": "public.profiles", "sourceRows": 2, "targetRows": 2, "match": true },
|
|
{ "table": "public.notes", "sourceRows": 3, "targetRows": 3, "match": true }
|
|
],
|
|
"rlsDecisions": [
|
|
{ "schema": "public", "table": "notes", "kind": "join", "detail": "profile_id -> public.profiles.id -> id = auth.uid()" },
|
|
{ "schema": "public", "table": "profiles", "kind": "direct", "detail": "id = auth.uid()" }
|
|
]
|
|
```
|
|
|
|
A migrated user then signed in on the new environment with their old password (`200` with an
|
|
access token; a wrong password got `400 Invalid login credentials`). Running the same command a
|
|
second time applied nothing and wrote no new files.
|
|
|
|
## The four sources
|
|
|
|
```
|
|
scribase-migrate supabase --to <org>/<project> --create-project \
|
|
--source-ref <ref> --source-url <connection-string> --source-service-key <key>
|
|
|
|
scribase-migrate neon --to <org>/<project> --source-url <connection-string> --migrate-auth-users
|
|
|
|
scribase-migrate clerk --to <org>/<project> --secret-key <sk_...>
|
|
|
|
scribase-migrate resend --api-key <re_...> --out resend-export.json
|
|
```
|
|
|
|
Prefix each with `npx --yes git+https://rithub.dev/caelum0x/scribase-migrate.git` (Bun:
|
|
`bunx -p git+https://rithub.dev/caelum0x/scribase-migrate.git scribase-migrate`), or clone the
|
|
repo and run `node bin/scribase-migrate.mjs`. `--help` prints every flag.
|
|
|
|
| Source | Postgres schema + data | Auth users / passwords | Storage | Notes |
|
|
|---|---|---|---|---|
|
|
| **Supabase** | yes | yes (hash preserved) | yes | one API call, Scribase's own `POST /v1/imports` |
|
|
| **Neon** | yes | yes, if the source has its own `auth.users` table | no | Neon has no storage or auth of its own; branches land in `branch_<name>` schemas |
|
|
| **Clerk** | no | yes, with a **new** one-time password (Clerk never exposes a portable hash) | no | every migrated user is flagged `password_reset_required` |
|
|
| **Resend** | no | no | no | exports sending domains (DNS records) and broadcasts to a JSON bundle. No Scribase account needed. Scribase has no import endpoint for either yet, so this is import-ready, not auto-applied |
|
|
|
|
Everything except `resend` needs a Scribase access token: `npx scribase login` once, or export
|
|
`SCRIBASE_ACCESS_TOKEN`.
|
|
|
|
## What is verified and what is not
|
|
|
|
**Verified live** against a real Scribase organization, twice (first build, then the 0.1.1
|
|
re-run above): the `neon` source end to end, including `--create-project`, generated RLS passing
|
|
Scribase's migration lint with 0 blocking findings, row-count parity over HTTP, a real password
|
|
login, `--dry-run`, and an idempotent second run. The `resend` source reaches the real Resend API
|
|
(a fake key gets Resend's own `400 API key is invalid`). Artifacts:
|
|
[test-fixtures/verification-run](https://rithub.dev/caelum0x/scribase-migrate/src/branch/main/test-fixtures/verification-run).
|
|
|
|
**Not verified live**, because each needs a provider account this build did not have:
|
|
|
|
- `supabase`: the request shape was confirmed against the real API with a dry run (a fake project
|
|
ref is correctly rejected with `400 invalid_project_ref`), but the full one-call import needs a
|
|
real, network-reachable Supabase project.
|
|
- `clerk` and a full `resend` export: checked against a local mock of each provider's documented
|
|
API shape, not a real account.
|
|
|
|
## Why it works this way
|
|
|
|
This was built by composing Scribase's own already-shipped APIs — the
|
|
`scribase` npm SDK (`edge-repos/scribase-cli`) and the GoTrue admin API every
|
|
self-hosted Supabase migration tool already uses — not by reinventing a
|
|
dump format. Three real constraints, found by actually running this against
|
|
a live Scribase project while building it, shaped the final design:
|
|
|
|
**Why not `pg_restore` straight into Scribase.** `environments.connection()`
|
|
returns a real Postgres URI, but its hostname is on Scribase's internal
|
|
network only (the SDK already documents it `SERVER-SIDE ONLY`) — it does
|
|
not resolve from an external machine. Unlike Supabase or Neon, Scribase has
|
|
no public Postgres TCP endpoint today. The Neon/generic-Postgres source
|
|
instead builds plain SQL (`pg_dump --schema-only` / `--data-only --inserts`)
|
|
and sends it through `POST .../migrations` — the same HTTPS path
|
|
`scribase migrate` itself uses, and the only one that actually reaches a
|
|
Scribase environment from outside its network.
|
|
|
|
**Why every run resends the whole `supabase/migrations/` folder.** That
|
|
endpoint is not additive. Verified live: submitting one new migration
|
|
without the version(s) already recorded answers `409 migration_conflict`
|
|
("the database records migration X but it is absent from the set on
|
|
disk"), and re-submitting an already-applied version with even one
|
|
different byte (we hit this with a stray trailing newline) answers a
|
|
different `409` ("an already-applied migration must never be edited",
|
|
with the two checksums). So `scribase-migrate neon` writes into
|
|
`supabase/migrations/` using the **exact same naming convention**
|
|
`npx scribase migrate` reads, and always reads the whole folder back and
|
|
resends it. One practical upshot: commit that folder to the app's repo,
|
|
the same as you would for `scribase migrate` on its own — it is the one
|
|
source of truth the next run (or a teammate's) needs.
|
|
|
|
**Why RLS is generated, never skipped or blanket-opened.** Scribase's
|
|
migration lint `BLOCK`s any new `public` table with neither RLS enabled nor
|
|
a policy — verified live, and a `pg_dump --schema-only` of a typical
|
|
Supabase/Neon app has neither (Neon has no RLS concept at all; many
|
|
Supabase apps relied on the service key and never wrote policies). Writing
|
|
a blanket `using (true)` policy to satisfy the lint would quietly make
|
|
every row readable by every signed-in user — worse than the block it
|
|
papers over. `lib/rls.mjs` instead infers an owner column (a direct foreign
|
|
key to `auth.users`, or one hop through another table that has one) and
|
|
writes a real owner policy; a table it cannot prove an owner for gets RLS
|
|
enabled with a `using (false)` policy — locked to the service-role key
|
|
(which bypasses RLS) until someone writes a real policy, never silently
|
|
public.
|
|
|
|
**Why Clerk users get a new password, not a migrated hash.** Clerk's
|
|
Backend API has no endpoint that returns a portable password hash on any
|
|
plan — this is a platform limit, not a corner this tool cut. Each migrated
|
|
user is created via the destination environment's GoTrue admin API
|
|
(`POST /auth/v1/admin/users`, the same endpoint any self-hosted Supabase
|
|
migration script uses) with a random one-time password and
|
|
`app_metadata.password_reset_required: true`, and is returned in
|
|
`needsPasswordReset` so the caller can send a real password-reset email.
|
|
|
|
**A sharp edge if you hand-write `auth.users` rows yourself**: GoTrue's Go
|
|
driver scans several `varchar` columns (`confirmation_token`,
|
|
`recovery_token`, `email_change*`, `phone_change*`,
|
|
`reauthentication_token`) as a plain `string`, not nullable — leaving them
|
|
at Postgres's own `NULL` default makes every later login 500 with
|
|
`converting NULL to string is unsupported`. `dumpAuthUsersSql` forces them
|
|
to `''`, which is what GoTrue itself would have written for an account
|
|
with no pending confirmation/recovery flow.
|
|
|
|
**A sharp edge in `pg_dump` itself**: `pg_dump`'s "can dump an older
|
|
server" guarantee does not cover every statement it emits. A 17.x
|
|
`pg_dump` against a 16.x source produced
|
|
`ALTER TABLE ... ALTER COLUMN ... SET (pg_attver=…)` — a PG17-only
|
|
column-storage attribute Postgres 16 rejects outright. `pgDumpBinaryFor()`
|
|
probes the source's major version and prefers an exact-match Homebrew keg
|
|
(`postgresql@<major>`) over whatever `pg_dump` is on `PATH`, warning when
|
|
it has to fall back.
|
|
|
|
**Why a preview environment starts empty.** A `snapshot` or `sanitized` environment copies
|
|
production data, so the control plane refuses one in a project that has no production
|
|
environment yet (`source_environment_missing`). `--create-project --kind preview` therefore
|
|
creates a `schema_only` environment. A deleted environment's name cannot be reused (`409`), so
|
|
pick a new name each time you try this out.
|
|
|
|
## Repo layout
|
|
|
|
- `bin/scribase-migrate.mjs` — the CLI
|
|
- `lib/client.mjs`, `lib/auth.mjs` — the `scribase` SDK + the same token
|
|
lookup `scribase` itself uses (keychain / `SCRIBASE_ACCESS_TOKEN` / login)
|
|
- `lib/project.mjs` — idempotent `--create-project`
|
|
- `lib/postgres.mjs` — `pg_dump` wrapping, version matching, the
|
|
GoTrue-safe `auth.users` export
|
|
- `lib/rls.mjs` — the RLS-inference heuristic
|
|
- `lib/migrationFiles.mjs` — the `supabase/migrations/` read-everything,
|
|
resend-everything helper
|
|
- `lib/sources/{supabase,neon,clerk,resend}.mjs` — one module per source
|
|
- `lib/login.mjs` — the real password-login proof
|
|
- `docs/migrate-page.md` — the scribase.com/migrate docs-page snippet
|
|
- `test-fixtures/` — the real local-verification seed + its output
|
|
|
|
## Docs
|
|
|
|
[docs/migrate-page.md](https://rithub.dev/caelum0x/scribase-migrate/src/branch/main/docs/migrate-page.md)
|
|
is the source of the migration guide on
|
|
[scribase.com/migrate](https://scribase.com/migrate?utm_source=rithub&utm_medium=readme&utm_campaign=scribase-migrate).
|
|
|
|
## License
|
|
|
|
MIT
|