Backend & Data••7 min read

Background Jobs and Queues: Doing Work Outside the Request

Why slow work doesn't belong in a web request, how job queues fix it, and the hard parts nobody warns you about — retries, idempotency, dead letters, and the at-least-once delivery trap.

Backend & Data: Background Jobs and Queues: Doing Work Outside the Request
KA
Krishna Agarwal

Why not just do it in the request?

A user signs up. You need to: create their account, send a welcome email, provision a workspace, sync them to your CRM, and generate a starter report. If you do all that inside the signup request, the user stares at a spinner for eight seconds while your email provider and CRM take their time — and if the CRM is down, the whole signup fails even though the account was created fine.

The fix: do the essential thing synchronously (create the account), and push the rest onto a queue to happen in the background. The user gets an instant response; the slow, failure-prone work happens out of band where it can retry without anyone waiting.

What belongs in the background

The rule of thumb: if the user doesn’t need the result to continue, and it’s slow or can fail independently, background it.

Good candidates:

  • Sending emails and notifications
  • Image/video processing, thumbnail generation
  • Calling third-party APIs (payment webhooks, CRM sync)
  • Generating reports, exports, PDFs
  • Anything on a schedule (nightly cleanup, digests)

Keep in the request:

  • Creating the record the user is about to see
  • Validation and authorization
  • Anything whose result the response depends on

The basic shape

A queue has three parts: something enqueues jobs, a queue holds them, and workers pull and process them.

// In the web request — enqueue and return immediately
app.post('/signup', async (req, res) => {
  const user = await createUser(req.body);   // essential, synchronous
  await queue.add('welcome-email', { userId: user.id }); // deferred
  await queue.add('crm-sync', { userId: user.id });
  res.json({ success: true });               // user gets instant response
});
// In a separate worker process — processes jobs
worker.process('welcome-email', async (job) => {
  const { userId } = job.data;
  const user = await getUser(userId);
  await sendEmail(user.email, welcomeTemplate(user));
});

The web server and the worker are separate processes — often separate machines. That’s the point: a flood of image-processing jobs can’t slow down your web requests, and you can scale workers independently of web servers.

BullMQ (Redis-backed) is the common choice in Node; Python has Celery, Ruby has Sidekiq. They differ in detail but share the concepts below.

The hard part: things fail

In a request, if something fails you return a 500 and move on. In a queue, jobs will fail — the email provider times out, the API rate-limits you, the worker crashes mid-job — and the whole value proposition is handling that gracefully. This is where naive queue usage falls apart.

Retries with backoff

Transient failures (a timeout, a brief outage) should just be retried:

await queue.add('crm-sync', { userId }, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000 }, // 1s, 2s, 4s, 8s, 16s
});

Exponential backoff is essential — retrying a struggling service immediately, repeatedly, just kicks it while it’s down (and may be why it’s down). Spacing retries out gives it room to recover. Without backoff, a brief upstream blip turns into a self-inflicted denial-of-service as your workers hammer it.

The idempotency trap (the big one)

Here’s the thing nobody warns you about early enough: most queues guarantee at-least-once delivery, not exactly-once. A job can run more than once — the worker finishes the work, then crashes before marking the job done, so the queue redelivers it. If that job charges a credit card or sends an email, running twice means double-charging or double-emailing.

The fix is idempotency: design jobs so running them twice has the same effect as running once.

// ❌ Runs twice → user charged twice
worker.process('charge', async (job) => {
  await stripe.charges.create({ amount, customer });
});

// ✅ Idempotent — the second run is a no-op
worker.process('charge', async (job) => {
  const { orderId, amount, customer } = job.data;

  // Has this order already been charged?
  const existing = await db.query(
    'SELECT id FROM charges WHERE order_id = $1', [orderId]
  );
  if (existing.rows.length) return; // already done, safely skip

  // Stripe's own idempotency key — belt and suspenders
  await stripe.charges.create(
    { amount, customer },
    { idempotencyKey: `order-${orderId}` }
  );
  await db.query('INSERT INTO charges (order_id) VALUES ($1)', [orderId]);
});

The pattern: check whether the effect already happened before causing it, keyed by something stable (the order ID). Many APIs (Stripe, and others) also accept an idempotency key so they dedupe on their side. Use both. Assume every job can run twice, and this stops being scary.

Dead letter queues

Some jobs fail every retry — bad data, a permanently-gone resource, a bug. You don’t want them retrying forever, and you don’t want them silently vanishing. They go to a dead letter queue: a holding pen for jobs that exhausted their retries.

worker.on('failed', (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    logger.error({ jobId: job.id, data: job.data, err }, 'job dead-lettered');
    // moved to failed set; alert a human to investigate
  }
});

Monitor the dead letter queue. A growing pile there is a signal something is systematically broken — a changed API, a data problem — and it’s often the earliest warning you’ll get.

Scheduled and recurring jobs

Queues also handle “do this later” and “do this every night”:

// Delayed — run once, in an hour
await queue.add('trial-reminder', { userId }, { delay: 60 * 60 * 1000 });

// Recurring — every day at 2am (cron syntax)
await queue.add('nightly-cleanup', {}, {
  repeat: { pattern: '0 2 * * *' },
});

This replaces scattering setTimeout calls (which vanish when the process restarts) and hand-rolled cron scripts (which have no retries or visibility). The queue persists the schedule and survives restarts.

Ordering: usually not guaranteed

A common wrong assumption: that jobs run in the order you enqueued them. With multiple workers pulling concurrently, job B can finish before job A. If order matters (apply these account changes in sequence), you need a queue feature that supports it — BullMQ’s FIFO queues, or grouping by a key so related jobs run on one worker in order.

Better still, design so order doesn’t matter where you can. Order-dependent background work is fragile; independent jobs scale freely across workers.

Observability for queues

You can’t see queue problems the way you see a failing request — no user is watching. Track:

  • Queue depth — jobs waiting. A steadily growing backlog means workers can’t keep up; scale them or find the slow job.
  • Processing time — jobs getting slower is an early warning.
  • Failure/dead-letter rate — spikes mean something broke.
  • Worker health — a crashed worker means jobs pile up unprocessed, silently.

A backed-up queue is invisible until someone notices their email arrived three hours late. Dashboards and alerts on queue depth catch it first.

When you don’t need a queue

Queues add real operational weight: another process to run and monitor, Redis (or similar) to maintain, and the idempotency/retry complexity above. Don’t add one reflexively.

You might not need one if:

  • The work is genuinely fast and reliable (a quick local DB write)
  • You have very low volume — a slightly slower request is fine
  • Your platform offers a simpler primitive — serverless functions, platform-native background tasks, or a managed queue that hides the operational burden

For a small app, “just do it in the request” or a lightweight scheduled function often beats standing up a full queue system. Reach for a queue when you have real slow/unreliable work, real volume, or a real need to retry independently of the user — not before.

The summary

Queues let your app stay responsive by moving slow, failure-prone work out of the request path — and let that work retry on its own without anyone waiting. The concepts are simple; the discipline is in the failure handling. Retry with backoff, make every job idempotent because it will sometimes run twice, dead-letter what won’t succeed, and watch your queue depth. Get those right and background processing becomes one of the most reliable parts of your system instead of a source of mysterious double-charges.