System Design: Turning PostgreSQL Into a Reliable Email Queue (No Redis, No Kafka, No New Infra) | by Mikayel Dadayan | Jul, 2026
Your cron job should schedule emails, not send them.
When your application has around 8,000 users, and you need to send a weekly digest, the simplest approach is often to write a cron job that queries all users, loops through them, and sends emails through your provider. That can work at first, but as your user base grows, the approach quickly starts to break down. It becomes harder to scale, more difficult to recover from failures, and less reliable overall. At that point, it’s worth rethinking the implementation and moving to a queue-based design that can handle delivery in a more controlled and dependable way.
Then the user base grows. 80,000 users. 400,000. 1.2 million. The same cron job now runs for eleven hours. The first customer gets their email at 6:01 AM, and the last one gets it the next day at 5 PM. Sometimes the Monday run is still going when the Tuesday run kicks off, and now two processes are stepping on each other. Someone deploys mid-run and half the users get nothing, with no record of who got what.
So the cron stops sending mail entirely. Its only job is to write down who needs an email, and then it exits, in seconds, not hours. The actual sending gets done by a handful of worker processes running in parallel, each one pulling a chunk of work off a table. Postgres — the database you’re already running and backing up and trusting for everything else — is what keeps those workers from stepping on each other. Nobody sends the same email twice, and if a worker gets killed mid-batch, nothing is lost.
Why the naive cron model breaks
The pattern almost everyone starts with looks like this:
async function sendWeeklyDigest() {
const { rows: users } = await db.query(
`SELECT id, email FROM users WHERE digest_enabled = true`
);
for (const user of users) {
await emailProvider.send({
to: user.email,
template: 'weekly-digest',
});
}
}
This is fine at the small scale and pathological at the large scale, for reasons that compound:
It’s sequential. If a send takes 300ms round-trip to your email provider, one million emails take ~83 hours. Even at 100ms, you’re at 27+ hours. The math is unforgiving.
It’s a single long-lived process. A process that runs for hours is exposed to everything: deploys, node restarts, OOM kills, and network blips. And when it dies at email 412,006, you have no idea where it died. Restart it, and you either resend to 412,006 people or skip everyone.
It has no memory. All states live in a loop variable. There’s no answer to “did user 8812 get the email?” other than grepping provider logs.
Runs overlap. When execution time exceeds the cron interval, run N+1 starts while run N is still alive. Now you have two processes iterating the same user list. So, some users get the digest twice.
Fairness collapses. The gap between the first and last recipient grows linearly with your user base. For a marketing email that’s annoying. For “your payment failed, act within 24 hours,” it’s a real problem.
None of these are bugs in the code. They’re consequences of the architecture: one process owns the entire job, holds all the state in memory, and has to finish or fail as a unit.
What actually makes it faster is not the DB
One thing needs to be said explicitly, because it’s the most common misconception when people see this design:
Writing rows into a Postgres table does not make email sending faster. Parallelism does.
If your only problem is throughput, you can solve it inside the cron process today:
const users = await getUsers();
const CONCURRENCY = 50;
for (let i = 0; i < users.length; i += CONCURRENCY) {
const chunk = users.slice(i, i + CONCURRENCY);
await Promise.allSettled(chunk.map(u => emailProvider.send(u)));
}
Fifty concurrent sends instead of one. That alone could turn 27 hours into 30–40 minutes. No schema changes, no workers, nothing.
So why bother with the database at all, if a Promise.allSettled fixes the speed problem in five lines? Because speed was never the only thing wrong with the original cron job. The in-process version is faster, sure, but it’s the same fragile process underneath. It still lives and dies as one unit, it still has no record of who actually got the email, and two overlapping runs will still double-send. At the end, nothing survives a crash, and nothing can be queried mid-flight to see how it’s going.
The database-backed version is chosen for durability and observability, not raw throughput. That constraint is what’s forcing the architecture, and speed just happens to come along for free once the work is spread across workers. Keep that straight, because it changes how you should judge everything that follows.
Why not just use a real queue?
Let’s be upfront: the tools built for this problem are more scalable and more advanced than anything in this article. pg-boss, BullMQ on Redis, SQS, Kafka: each of them, in the right context, is a better queue than a hand-rolled Postgres table. The question is never “is Kafka more capable than my table,” because it obviously is. The question is what each option costs you, right now, given where your system already is.
pg-boss is the closest neighbor to this design. It’s literally this pattern (Postgres + SKIP LOCKED) packaged as a library, with retries, backoff, scheduling, and archiving already built. If third-party job libraries are allowed in your codebase, it’s the strongest alternative here. In our case, they weren’t, and we needed a solution fast: no dependency review, no new framework owning a critical path, just SQL and code we fully control. That constraint, not any weakness of pg-boss, is what put it out of reach.
SQS (+ Lambda or ECS workers) is a genuinely great fit if you’re already deep in AWS. If your services run there, IAM is set up, and the team lives in that ecosystem, SQS gives you a managed, effectively infinitely scalable queue with better delivery characteristics than a Postgres table will ever have. Take it. But if you’re not already on that infrastructure, adopting it just for email means wiring IAM roles, dead-letter queues, VPC access back to your database, and a local dev story for all of it. That’s a project, not a fix.
Redis + BullMQ is fast and pleasant, when Redis is already part of your architecture. If it isn’t, you’re introducing a whole new stateful service for one feature: provisioning, persistence configuration (get it wrong, and a restart eats your queue), memory monitoring, failover, and one more thing on the on-call rotation. For a system with no Redis anywhere, that’s a real extra complication for a problem Postgres could already carry.
Kafka, for this problem, is even more overcomplicated. It’s built for high-throughput event streaming, replayable logs, and many independent consumer groups: a different category of problem. Standing up a Kafka cluster to fan out a nightly email batch is using a freight train to deliver a pizza. If you already run Kafka for event streaming, sure, an email topic is cheap. Standing it up for this isn’t.
On top of the per-tool math, some environments add blanket constraints: fixed approved-component lists in banks, healthcare, and on-prem deployments; compliance reviews where every new service is a new attack surface, and recipient email addresses in a Redis instance become a data-handling question someone has to answer.
So the argument is narrower than “queues bad”: when the advanced options are unavailable or disproportionate, and you already run, back up, monitor, and trust PostgreSQL, the database can carry a work queue surprisingly far. This isn’t a hack, either; FOR UPDATE SKIP LOCKED was added to Postgres 9.5 largely for this pattern.
The architecture
The core shift is a separation of responsibilities:
Cron / scheduler. Runs on schedule, creates a campaign record, bulk-inserts one recipient row per user, and exits. Total runtime: seconds, even for millions of users. It never talks to the email provider.
email_campaigns. One row per logical send (“Weekly digest 2026-07-06”). Holds metadata and overall status.
email_recipients. The work queue. One row per (campaign, user). Each row carries its own status, attempt count, and lock timestamp. This table is the entire coordination mechanism.
Workers. Ordinary long-running Node processes, run 1 or run 20, on one machine or several. Each worker loops: claim a batch of pending rows, send, record the outcome, repeat. Workers don’t know about each other. Postgres row locks handle coordination.
Email provider. SES, SendGrid, Postmark, whatever. Unchanged.
The good part is that workers are stateless and interchangeable. Killing one loses nothing. Adding one requires no reconfiguration. The database is the single source of truth for what’s done, what’s in flight, and what failed.
How the Database schema is configured
UNIQUE (campaign_id, user_id) is the duplicate guardrail.
If the cron job accidentally runs twice — because of a redeploy, a manual trigger, or some other operational mistake — the second insert for the same user in the same campaign simply conflicts and does nothing. That one constraint removes an entire class of “we emailed everyone twice” failures before they can happen.
The recipient email is snapshotted into the row on purpose.
Users change email addresses. If workers had to join back to the users table at send time, the address used could drift from the one that existed when the campaign was queued. Storing the email directly in email_recipients makes each row self-contained: workers can send without extra lookups, and an audit months later still shows the exact address that campaign targeted.
The pending index is partial because workers only care about pending rows.
Claim queries should not get slower just because the table has accumulated millions of historical sent rows. Indexing only status = 'pending' keeps the hot path small and focused, so workers can continue finding the next batch quickly even when the table itself has grown huge.
The status column acts as a visible state machine.
A row starts as pending, moves to sending, and eventually becomes either sent or failed. On retry, it can move from sending back to pending. That lifecycle is simple, explicit, and queryable by anyone with SQL access. It also makes operational visibility almost free: a progress dashboard is just a GROUP BY status.
The cron job now is just a scheduler
The cron job no longer loops through users and sends emails itself. It creates the campaign, inserts the recipient rows, and leaves delivery to the worker pool.
import { pool } from './db.js';export async function scheduleWeeklyDigest() {
const client = await pool.connect();
try {
await client.query('BEGIN');
const { rows: [campaign] } = await client.query(
`INSERT INTO email_campaigns (name, template_key, status, scheduled_at)
VALUES ($1, $2, 'processing', now())
RETURNING id`,
[`weekly-digest-${new Date().toISOString().slice(0, 10)}`, 'weekly_digest']
);
// One INSERT ... SELECT. No loop, no per-user round trips.
// 2M users -> one statement, a few seconds.
const { rowCount } = await client.query(
`INSERT INTO email_recipients (campaign_id, user_id, email)
SELECT $1, u.id, u.email
FROM users u
WHERE u.digest_enabled = true
AND u.email_verified = true
ON CONFLICT (campaign_id, user_id) DO NOTHING`,
[campaign.id]
);
await client.query('COMMIT');
console.log(`Campaign ${campaign.id}: enqueued ${rowCount} recipients`);
} catch (err) {
await client.query('ROLLBACK');
throw err;
} finally {
client.release();
}
}
Note what this does not do: it doesn’t touch the email provider, doesn’t loop over users in application code, and doesn’t stay alive. It converts “send emails” into data, and data is durable.
The workers
Here is a complete worker. It runs forever, claiming and processing batches until there’s nothing left, then idles:
import { pool } from './db.js';
import { sendEmail } from './email-provider.js';const BATCH_SIZE = 50;
const MAX_ATTEMPTS = 3;
const IDLE_MS = 5_000;
async function claimBatch() {
const { rows } = await pool.query(
`WITH claimed AS (
SELECT id
FROM email_recipients
WHERE status = 'pending'
ORDER BY id
LIMIT $1
FOR UPDATE SKIP LOCKED
)
UPDATE email_recipients r
SET status = 'sending',
locked_at = now(),
attempts = r.attempts + 1,
updated_at = now()
FROM claimed
WHERE r.id = claimed.id
RETURNING r.id, r.campaign_id, r.user_id, r.email, r.attempts`,
[BATCH_SIZE]
);
return rows;
}
async function markSent(id) {
await pool.query(
`UPDATE email_recipients
SET status = 'sent', sent_at = now(), locked_at = NULL, updated_at = now()
WHERE id = $1`,
[id]
);
}
async function markFailedOrRetry(id, attempts, err) {
if (attempts >= MAX_ATTEMPTS) {
await pool.query(
`UPDATE email_recipients
SET status = 'failed', failed_reason = $2, locked_at = NULL, updated_at = now()
WHERE id = $1`,
[id, String(err).slice(0, 500)]
);
} else {
await pool.query(
`UPDATE email_recipients
SET status = 'pending', locked_at = NULL, updated_at = now()
WHERE id = $1`,
[id]
);
}
}
async function processRow(row) {
try {
await sendEmail({ to: row.email, campaignId: row.campaign_id });
await markSent(row.id);
} catch (err) {
await markFailedOrRetry(row.id, row.attempts, err);
}
}
async function run() {
console.log(`worker ${process.pid} started`);
while (true) {
const batch = await claimBatch();
if (batch.length === 0) {
await new Promise(r => setTimeout(r, IDLE_MS));
continue;
}
await Promise.allSettled(batch.map(processRow));
}
}
run().catch(err => {
console.error('worker crashed:', err);
process.exit(1);
});
Workers can be scaled however you want — run node worker.js five times, under systemd, in separate containers, or across multiple machines. The rest of the system doesn’t need to know how many workers exist.
The important design choices are:
- Claim jobs atomically. A single SQL statement selects pending rows, locks them, and updates them to
sendingbefore the transaction commits. That removes the gap where a row could be chosen by one worker but still appear pending to another. - Mark as
sendingbefore sending the email. This avoids the worse failure mode. If a worker sent the email first and crashed before updating the row, the job would still look pending and could be sent again. With mark-first, a crash leaves the row stuck insending, which can later be safely reclaimed and retried. In practice, a delayed email is usually preferable to a duplicate one. - Increment
attemptsat claim time. That way, even if the worker crashes mid-send, the attempt is still counted. It also prevents “poisoned” rows—jobs that consistently crash a worker—from being retried forever without limit.
How multiple workers claim rows without collisions
This is what makes multiple workers safe.
With plain FOR UPDATE, if Worker 1 locks rows 1–50, every other worker trying to claim pending rows will wait behind it. So even with 10 workers, they end up moving one at a time.
SKIP LOCKED fixes that. If a worker hits rows already locked by someone else, it skips them and keeps going.
So instead of everyone fighting over rows 1–50:
- Worker 1 takes
1–50 - Worker 2 takes
51–100 - Worker 3 takes
101–150
That’s what lets multiple workers claim jobs from the same table in parallel without stepping on each other.
You don’t need custom coordination logic for this. Postgres is doing the coordination for you. If a worker crashes, its transaction rolls back, its locks disappear, and those rows become claimable again automatically.
Each worker doesn’t get “the true first 50 pending rows.” It gets the first 50 rows nobody else has locked yet. For a queue, that’s exactly what you want.
The only caveat is that SKIP LOCKED skips any locked rows, not just ones locked by other workers. So if some admin script or backfill holds locks on pending rows for too long, workers may temporarily see less work than actually exists.
What happens when a worker crashes
Row locks only protect the claim step itself. Once a worker commits its claim, the rows are permanently marked as sending and the locks are gone.
So if a worker crashes after claiming a batch but before finishing the sends, those rows don’t automatically recover. They just sit in sending with no active worker attached to them.
That’s what locked_at is for. If a row has been in sending for far longer than a normal batch should take, you can treat it as abandoned and move it back to pending.
UPDATE email_recipients
SET status = 'pending',
locked_at = NULL,
updated_at = now()
WHERE status = 'sending'
AND locked_at < now() - interval '10 minutes';
Pick the timeout with intent: it must be comfortably longer than any legitimate batch could take (batch size × worst-case send latency, with margin).
If a reclaimed row keeps crashing workers, attempts prevents it from looping forever. Each time it’s claimed again, the counter goes up. Once it hits MAX_ATTEMPTS, the row is marked failed with a reason and drops out of the queue. That’s your poison-pill protection.
The bigger win is operational: a worker crash stops being an incident. Stuck rows get reclaimed automatically, other workers keep draining the queue, and you can restart the dead process whenever convenient.
Overlapping campaigns
Overlapping campaigns stop being a problem in this model.
Campaign A and campaign B create different rows, because they have different campaign_id values. That means they never conflict, even if both target the same user.
SELECT r.id
FROM email_recipients r
JOIN email_campaigns c ON c.id = r.campaign_id
WHERE r.status = 'pending' AND c.status = 'processing'
ORDER BY c.priority DESC, r.id
LIMIT $1
FOR UPDATE OF r SKIP LOCKED
Workers also don’t care which campaign a row belongs to. They just keep claiming pending rows. So if Tuesday’s campaign starts while Monday’s is still running, the worker pool simply processes both at the same time.
That’s a real queueing feature, priorities, added with one column and one ORDER BY. Try getting that change through a message-broker deployment as quickly.
How fast this actually gets
A sequential cron job sending 1 million emails can take days. A small worker pool cuts that down to minutes, because email delivery is mostly waiting on the provider, not doing CPU-heavy work.
In practice, the first limit you’ll usually hit isn’t Postgres — it’s your email provider’s rate limit. The database is mostly doing small status updates as rows move through pending, sending, and sent or failed, which is relatively cheap compared to the actual network calls.
So the main tuning knobs are worker count and batch size. Start with something simple, like 4 workers with batches of 50, measure it, and adjust from there. For this kind of workload, the provider is usually the bottleneck long before the database is.
If you want, I can do one last pass over all the rewritten sections together so they read in one consistent Medium/article voice instead of section-by-section patches.
Why not just chunk users in cron?
If speed is genuinely the whole requirement, chunking users into arrays inside cron (the exact code shown earlier) works fine, and it’s a lot less code than everything above. For a few tens of thousands of recipients, in a system where a botched run just gets re-run, that’s probably the right call. It would be dishonest to pretend otherwise.
Conclusion
The fix for the eleven-hour cron job was not making the loop faster. It was changing the model.
Cron stopped being the process that sends emails and became the process that schedules them: create the campaign, insert the recipient rows, and exit. From there, a pool of workers picks up those rows and sends in parallel. That parallelism is what turns delivery from an hours-long sequential job into something that can finish in minutes.
PostgreSQL is what makes that safe. The queue lives in ordinary tables, every email has a visible state, failed work can be retried, and FOR UPDATE SKIP LOCKED lets multiple workers claim different batches without stepping on each other.
So the optimization here was bigger than “make cron faster.” It was moving email delivery out of cron, turning PostgreSQL into the queue, and letting worker parallelism do the heavy lifting.
