Compare commits

..

No commits in common. "main" and "v0.1.0" have entirely different histories.
main ... v0.1.0

5 changed files with 87 additions and 179 deletions

131
README.md
View File

@ -1,99 +1,106 @@
# neon-cold-start-probe
Measure how long your Neon (or any scale-to-zero Postgres) database really takes to answer the
first connection after it has been idle. One command, no account, no config.
Neon (and most serverless/autoscaling Postgres, including [Scribase](https://scribase.com)'s
own Free/Hobby tier) suspends a compute after a few idle minutes to save cost. The next
connection has to wake it up before it can answer — slower than every connection after it. Nobody
publishes a number for *your* database's wake-up time, and the usual advice is "it felt slow for
a second." This measures it for real: a brand-new connection each time (never pooled — pooling is
exactly what would hide the thing you're trying to see), timed in three parts (connect, query,
total).
## 10-second quickstart
## 30-second quickstart
```
npx --yes git+https://rithub.dev/caelum0x/neon-cold-start-probe.git "$DATABASE_URL" --repeat 5
npx --yes git+https://git.ereboslab.com/caelum0x/neon-cold-start-probe.git \
"postgres://user:pass@ep-xxxx.region.aws.neon.tech/dbname?sslmode=require"
```
That is the whole setup. It needs Node 18+ and a Postgres connection string.
With Bun:
or clone it and run locally:
```
bunx -p git+https://rithub.dev/caelum0x/neon-cold-start-probe.git neon-cold-start-probe "$DATABASE_URL" --repeat 5
git clone https://git.ereboslab.com/caelum0x/neon-cold-start-probe.git
cd neon-cold-start-probe && npm install
node bin/neon-cold-start-probe.mjs "$DATABASE_URL"
```
## What you get
Real output from this exact build (Postgres 16 over TLS, `sslmode=require`, 2026-11-17):
Output:
```
probe 1/5: connect=202ms query=10ms total=212ms [warm]
probe 2/5: connect=55ms query=10ms total=64ms [warm]
probe 3/5: connect=39ms query=1ms total=41ms [warm]
probe 4/5: connect=299ms query=15ms total=314ms [warm]
probe 5/5: connect=36ms query=3ms total=39ms [warm]
summary: min=39ms p50=64ms p95=314ms max=314ms (0/5 probes >= 500ms, 0 failed)
Neon cold starts and compute limits explained, with the fix: https://scribase.com/neon-compute-limit-cold-start?utm_source=cli&utm_medium=cli-output&utm_campaign=neon-cold-start-probe
probe 1/1: connect=931ms query=7ms total=938ms [COLD]
```
Every probe opens a brand-new connection (never pooled, because a pool hides exactly the thing
you are trying to see) and times three parts: connect, query, total. A probe at or above the
threshold (500ms by default) is labelled `COLD`.
## Seeing an actual cold start, not just a number
That sample is a database that never suspends, on a busy laptop, so every probe is warm and the
spread is machine noise. It is here to show the format, not a Neon number.
A single probe only tells you one connection's latency. To see the *cold vs warm* gap for real:
## How to catch a real cold start
1. Let the project sit idle past its suspend timeout (Neon's default/free setting is 5 minutes;
check your project's compute settings for the real value).
2. Run a `--repeat` probe immediately after — the **first** result is the one that had to wake
the compute; the rest, back-to-back, will be warm because the first one already woke it:
1. Leave the database idle past its suspend timeout. On Neon Free that is
[5 minutes and cannot be changed](https://neon.com/docs/introduction/scale-to-zero).
2. Run the quickstart command once. Probe 1 is the one that had to wake the compute. Probes 2 to
5 run against a compute that is already awake.
```
node bin/neon-cold-start-probe.mjs "$DATABASE_URL" --repeat 5 --every-seconds 1
probe 1/5: connect=931ms query=7ms total=938ms [COLD]
probe 2/5: connect=14ms query=2ms total=16ms [warm]
probe 3/5: connect=13ms query=2ms total=15ms [warm]
probe 4/5: connect=12ms query=2ms total=14ms [warm]
probe 5/5: connect=15ms query=2ms total=17ms [warm]
The gap between probe 1 and the rest is your cold start, measured from your machine and your
region. Run it back to back with no idle wait and every probe is warm. That is expected.
summary: min=14ms p50=16ms p95=938ms max=938ms (1/5 probes >= 500ms, 0 failed)
```
Running `--repeat` back-to-back from a warm start (the default, no prior idle wait) will almost
always show every probe as warm — that's expected, not a bug; it's the same compute, still awake.
## Options
| Flag | Default | What it does |
|---|---|---|
| `--url <conn-string>` | none | Also accepts a positional arg, or `DATABASE_URL` |
| `--url <conn-string>` | — | Also accepts a positional arg, or `DATABASE_URL` |
| `--repeat <n>` | `1` | Number of fresh-connection probes |
| `--every-seconds <s>` | `0` | Wait between probes |
| `--timeout <s>` | `15` | Per-probe connect timeout |
| `--cold-threshold-ms <ms>` | `500` | Total ms at or above which a probe is labelled `COLD` |
| `--json` | off | Per-probe results plus summary (min/p50/p95/max) on stdout |
| `--version` | | Print the version |
| `--cold-threshold-ms <ms>` | `500` | Total ms a probe is labelled `COLD` above |
| `--json` | off | Machine-readable output, including per-probe and summary (min/p50/p95/max) |
Run it from a clone instead:
## Where the 500ms default comes from
```
git clone https://rithub.dev/caelum0x/neon-cold-start-probe.git
cd neon-cold-start-probe && npm install
node bin/neon-cold-start-probe.mjs "$DATABASE_URL" --repeat 5
```
It is **not** a number Neon publishes — Neon documents that autoscaling computes suspend and
resume on demand but does not commit to a wake-up SLA. It's the midpoint of a real measurement
done building this tool's sibling project, a Postgres service with the same scale-to-zero
behavior: sleep the compute via its API, then measure the first request vs a warm one, 3 runs —
cold 0.93–1.47s, warm 0.09–0.24s. Pass your own `--cold-threshold-ms` once you have a few real
samples from your own project; it varies by region, plan and how large the compute is.
## Honest limits
- **Not yet run against a live Neon project.** Everything here was tested for real against a
local Postgres 16 (plain and TLS), which proves the connect, timing, retry and labelling logic
but can never suspend. The full transcript, failure paths included, is in
[test-fixtures/local-postgres-verification.md](https://rithub.dev/caelum0x/neon-cold-start-probe/src/branch/main/test-fixtures/local-postgres-verification.md).
- **One slow reading is not proof of a cold start.** Network jitter or a loaded machine can push
a warm connection over the threshold. Compare the first probe after real idle time against the
rest, and take more than one sample.
- **The 500ms default is not a Neon number.** Neon documents that a suspended compute wakes in
[a few hundred milliseconds](https://neon.com/docs/connect/connection-latency) and publishes no
SLA. 500ms is a starting point; set `--cold-threshold-ms` once you have samples of your own.
- **It is a generic Postgres probe.** It works against any connection string, Scribase's included.
- One-time process costs (first DNS lookup, loading Node's TLS stack) are paid before probe 1, so
they are not counted as database wake-up time.
- **This build has no real Neon test credentials** (no Neon account token in this environment,
and the Neon MCP integration needs an interactive browser OAuth sign-in nobody here could
complete). Everything was tested for real against a **local Postgres 16 container** — proves
the connect/query/timing/retry/classification logic is correct, but a local container never
suspends, so it cannot reproduce an actual Neon cold start. Full transcript, including the
failure paths (wrong port, wrong password, missing arg):
`test-fixtures/local-postgres-verification.md`.
- **A single slow reading is not proof of a cold start.** Network jitter, a loaded laptop, or a
busy database can also push a connection over the threshold — the local-Postgres transcript
above shows this happening for real (an 819ms and a 1664ms reading against a container that
was never suspended, just competing with other load on the machine at the time). Take more than
one sample, and compare the *first* probe after real idle time against the rest, not any one
number in isolation.
- **This is a generic Postgres latency probe**, not Neon-specific code — it works against any
Postgres connection string (including Scribase's). The Neon framing is because Neon's free-tier
suspend behavior is the most-searched version of this pain; if you're on a different
autoscaling Postgres, point it there instead.
- No `--watch`/daemon mode yet — each run is a fixed number of probes, then it exits.
## If the cold start is the problem
## Need more?
[Neon cold starts and compute limits explained, with the fix](https://scribase.com/neon-compute-limit-cold-start?utm_source=rithub&utm_medium=readme&utm_campaign=neon-cold-start-probe):
why the first query is slow, how the Free compute allowance is counted, and what to change.
This tool is built by [Scribase](https://scribase.com/?utm_source=rithub&utm_medium=readme&utm_campaign=neon-cold-start-probe):
Postgres, sign-in, storage and email on one account instead of four. Scribase also scales idle
environments to zero, so point this probe at it too and compare the numbers yourself.
If cold starts are a dealbreaker rather than a curiosity: [Scribase](https://scribase.com/neon-compute-limit-cold-start?utm_source=rithub&utm_medium=readme&utm_campaign=neon-cold-start-probe)
is Postgres + auth + storage + email on one account. Its own Free/Hobby tier **also** scales to
zero and pays the same kind of wake-up cost (measured: 0.93–1.47s cold vs 0.09–0.24s warm — the
numbers this tool's default threshold is based on) — only the $25/mo Pro tier stays always-warm.
No "no cold starts, ever" claim here for the free tier, because it wouldn't be true; the
comparison page lays out both tiers' real numbers side by side.
## License

View File

@ -1,4 +1,5 @@
#!/usr/bin/env node
'use strict';
// neon-cold-start-probe
//
@ -19,35 +20,13 @@
// you can see the first (often cold) probe against the rest (warm).
import pg from 'pg';
import { readFileSync } from 'node:fs';
import { lookup } from 'node:dns/promises';
const CTA_URL =
'https://scribase.com/neon-compute-limit-cold-start?utm_source=cli&utm_medium=cli-output&utm_campaign=neon-cold-start-probe';
const CTA_LINE = `Neon cold starts and compute limits explained, with the fix: ${CTA_URL}`;
const VALUE_FLAGS = new Set(['--url', '--repeat', '--every-seconds', '--timeout', '--cold-threshold-ms']);
function version() {
try {
return JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version;
} catch {
return 'unknown';
}
}
function parseArgs(argv) {
const out = { repeat: 1, every: 0, timeout: 15, threshold: 500, errors: [] };
const out = { repeat: 1, every: 0, timeout: 15, threshold: 500 };
const positional = [];
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
const next = () => {
const value = argv[++i];
if (value === undefined || (VALUE_FLAGS.has(value))) {
out.errors.push(`${a} needs a value`);
return undefined;
}
return value;
};
const next = () => argv[++i];
switch (a) {
case '--url': out.url = next(); break;
case '--repeat': out.repeat = parseInt(next(), 10); break;
@ -56,10 +35,8 @@ function parseArgs(argv) {
case '--cold-threshold-ms': out.threshold = parseInt(next(), 10); break;
case '--json': out.json = true; break;
case '--help': case '-h': out.help = true; break;
case '--version': case '-v': out.version = true; break;
default:
if (a.startsWith('-')) out.errors.push(`unknown option ${a}`);
else positional.push(a);
if (!a.startsWith('-')) positional.push(a);
break;
}
}
@ -81,7 +58,6 @@ Options:
--timeout <s> Per-probe connect timeout in seconds (default 15)
--cold-threshold-ms <ms> Total ms above which a probe is labelled "cold" (default 500)
--json Print machine-readable JSON instead of a table
--version Print the version
--help This help
What the 500ms default is based on: not a Neon-published number (Neon
@ -97,51 +73,16 @@ Neon's default/free settings) BEFORE you run this. Back-to-back probes
(the default) will almost always all look warm, because the first probe
itself keeps the compute awake for the rest.
Neon cold starts and compute limits explained, with the fix:
https://scribase.com/neon-compute-limit-cold-start?utm_source=cli&utm_medium=cli-help&utm_campaign=neon-cold-start-probe`);
Comparison, if cold starts are a dealbreaker: https://scribase.com/neon-compute-limit-cold-start?utm_source=cli&utm_medium=cli&utm_campaign=neon-cold-start-probe
(honest numbers both ways: Scribase's own Free/Hobby tier also scales to
zero and pays the same kind of cold-start cost; only its Pro tier stays
always-warm.)`);
}
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
// pg >= 8.16 prints an 8-line SECURITY WARNING to stderr for sslmode=require
// (the mode every Neon connection string ships with), because it already
// treats require/prefer/verify-ca as verify-full and plans to change that in
// pg 9. Asking for verify-full explicitly is the exact same behavior today,
// stays strict after pg 9, and keeps the probe output clean.
function normalizeSslMode(connectionString) {
return connectionString.replace(/([?&]sslmode=)(require|prefer|verify-ca)(?=&|$)/i, '$1verify-full');
}
function validate(args, url) {
if (args.errors.length) return [...args.errors];
const errors = [];
if (!url) {
errors.push('a connection string is required (positional arg, --url, or DATABASE_URL)');
} else if (!/^postgres(ql)?:\/\//i.test(url)) {
errors.push('the connection string must start with postgres:// or postgresql://');
}
if (!Number.isInteger(args.repeat) || args.repeat < 1) errors.push('--repeat must be a whole number >= 1');
if (!Number.isFinite(args.every) || args.every < 0) errors.push('--every-seconds must be a number >= 0');
if (!Number.isInteger(args.timeout) || args.timeout < 1) errors.push('--timeout must be a whole number of seconds >= 1');
if (!Number.isInteger(args.threshold) || args.threshold < 1) errors.push('--cold-threshold-ms must be a whole number >= 1');
return errors;
}
// One-time process costs (first DNS lookup, loading Node's TLS stack and CAs) are
// not the database waking up, and a long-running app never pays them per
// connection. Pay them once before probe 1 so they are not billed to it.
async function warmProcess(connectionString) {
try {
const { hostname } = new URL(connectionString);
const [tls] = await Promise.all([import('node:tls'), hostname ? lookup(hostname) : null]);
tls.createSecureContext(); // loads the root CA store, a one-time cost
} catch {
// A bad host surfaces as a normal failed probe with its real error.
}
}
async function probe(connectionString, timeoutSec) {
const client = new pg.Client({
connectionString,
@ -180,20 +121,16 @@ async function main() {
help();
return 0;
}
if (args.version) {
console.log(version());
return 0;
}
const rawUrl = args.url || process.env.DATABASE_URL || '';
const errors = validate(args, rawUrl);
if (errors.length) {
for (const message of errors) console.error(`error: ${message}.`);
console.error('Run with --help for usage.');
const url = args.url || process.env.DATABASE_URL || '';
if (!url) {
console.error('error: a connection string is required (positional arg, --url, or DATABASE_URL). Run with --help.');
return 1;
}
if (args.repeat < 1) {
console.error('error: --repeat must be >= 1.');
return 1;
}
const url = normalizeSslMode(rawUrl);
await warmProcess(url);
const results = [];
for (let i = 0; i < args.repeat; i++) {
if (i > 0 && args.every > 0) {
@ -236,11 +173,6 @@ async function main() {
);
}
// One plain closing line. In --json mode it goes to stderr so stdout stays
// valid JSON for whatever is piping it.
if (args.json) console.error(CTA_LINE);
else console.log(`\n${CTA_LINE}`);
if (failed === results.length) return 1;
return 0;
}

4
package-lock.json generated
View File

@ -1,12 +1,12 @@
{
"name": "neon-cold-start-probe",
"version": "0.1.1",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "neon-cold-start-probe",
"version": "0.1.1",
"version": "0.1.0",
"license": "MIT",
"dependencies": {
"pg": "^8.13.1"

View File

@ -1,6 +1,6 @@
{
"name": "neon-cold-start-probe",
"version": "0.1.1",
"version": "0.1.0",
"description": "Measure the real cold-start latency of a serverless Postgres endpoint (Neon and similar) from a fresh connection, not a guess.",
"license": "MIT",
"type": "module",

View File

@ -74,34 +74,3 @@ probe 1/1: connect=122ms query=24ms total=146ms [warm]
actually suspending and this tool catching the real wake-up latency on the
next probe. Logged honestly rather than faked — see the README's
"Honest limits" section.
## 2026-11-17 (v0.1.1) re-verification
Throwaway Postgres 16.16 (`initdb` in `/tmp`, port 55439, self-signed TLS cert). Machine load
average was above 70 during these runs, so the absolute numbers are noisy.
Bugs found by running it the way the README says, all fixed in v0.1.1:
- `?sslmode=require` (what every Neon connection string ships with) printed an 8-line `pg`
SECURITY WARNING before the first probe. Now normalized to `verify-full`, which is what `pg`
already did for `require`, silently.
- `--repeat abc` and a bare `--repeat` exited 1 with no message at all.
- A non-URL argument (`"not a url"`) failed as `getaddrinfo ENOTFOUND base`.
- `--version` answered "a connection string is required".
- Unknown flags were silently ignored.
TLS path, CA trusted via `NODE_EXTRA_CA_CERTS`:
```
$ neon-cold-start-probe "postgres://postgres@localhost:55439/postgres?sslmode=require" --repeat 5 --every-seconds 1
probe 1/5: connect=202ms query=10ms total=212ms [warm]
probe 2/5: connect=55ms query=10ms total=64ms [warm]
probe 3/5: connect=39ms query=1ms total=41ms [warm]
probe 4/5: connect=299ms query=15ms total=314ms [warm]
probe 5/5: connect=36ms query=3ms total=39ms [warm]
summary: min=39ms p50=64ms p95=314ms max=314ms (0/5 probes >= 500ms, 0 failed)
```
Same server, CA not trusted: `probe 1/1: FAILED (self-signed certificate; ...)`, exit 1, no
warning noise. `psql` confirmed the session was really TLS (`pg_stat_ssl.ssl = t`).