Retry & Failed Jobs

Every job attempt can fail — a throw inside handle, an unknown job name in the worker, a downstream timeout. attempts and backoff control retrying; once every attempt is spent, the job is failed and stays in Redis (for a week, up to the newest 5,000, unless removeOnFail says otherwise) so it can be inspected and retried later.

Configuring retries

import { defineJob } from "@warlock.js/queue";

export const syncInventory = defineJob({
  name: "inventory.sync",
  attempts: 5,
  backoff: { type: "exponential", delay: 2000 }, // 2s, 4s, 8s, 16s
  async handle() {
    await supplierApi.pullInventory();
  },
});

backoff is a plain number (fixed delay in ms) or { type: "fixed" | "exponential", delay }. "exponential" waits delay * 2^(attempt - 1) before each retry. Attempts and backoff can also be set app-wide in queue.defaultJobOptions, and overridden per dispatch — see Defining jobs for the precedence order.

An unrecoverable failure — no handler registered for the job’s name — skips retries entirely and fails on the first attempt.

Timeouts

Give a job a timeout so one stuck attempt cannot hold a worker forever:

export const renderReport = defineJob({
  name: "reports.render",
  attempts: 3,
  backoff: 5000,
  timeout: "2m", // milliseconds, or "500ms" | "30s" | "2m" | "1h" | "1d"
  async handle(payload: { reportId: string }, ctx) {
    // Pass the signal on so the work really stops when the time is up.
    const response = await fetch(reportUrl(payload.reportId), { signal: ctx.signal });
    return response.json();
  },
});

When an attempt runs past the limit it fails with a JobTimeoutError (Job "reports.render" timed out after 120000ms.), which is the failed reason you see in failedJobs() and the dashboard. It is an ordinary failed attempt, so attempts and backoff apply and the job retries. Set failOnTimeout: true to fail the job immediately instead, with no further attempts — use it when running again would only time out again.

timeout and failOnTimeout can be set in defineJob, per dispatch, and app-wide in queue.defaultJobOptions, with the usual precedence: dispatch options > defineJob > queue.defaultJobOptions. A timeout that is zero, negative or not a valid duration throws instead of being ignored.

The limit works the same on the BullMQ, local and memory drivers. BullMQ has no job timeout of its own, so Warlock enforces it in the worker.

Caution

JavaScript cannot kill a running promise. On timeout the attempt is marked failed and ctx.signal aborts, but a handler that ignores the signal keeps running in the background until it finishes by itself, possibly while the retry starts. Pass ctx.signal to fetch, database clients and other cancellable calls (or check ctx.signal?.throwIfAborted() between steps), and keep side effects idempotent.

Listing failed jobs

import { failedJobs } from "@warlock.js/queue";

const failed = await failedJobs({ queue: "default", start: 0, end: 49 }); // newest first, default end is 99

for (const job of failed) {
  console.log(job.name, job.failedReason, job.attemptsMade);
}

Each FailedJob carries id, name, queue, payload, attemptsMade, failedReason, stacktrace, failedAt, and its own retry().

Retrying

import { retryFailedJob } from "@warlock.js/queue";

await failed[0]?.retry(); // move that job back to waiting

await retryFailedJob("invoice:43"); // by id
// throws FailedJobNotFoundError if no FAILED job has that id, in that queue

A retried job goes back to waiting with its attempt count reset, so it runs through the full retry budget again if it fails.

Keeping the failed list bounded

Failed jobs are kept for 7 days, at most the newest 5,000 per queue, by default (removeOnFail: { age: "7d", count: 5000 }), so failedJobs() has something to show without growing forever. Set removeOnFail on defineJob (or defaultJobOptions) to change it, or false to keep every failure:

defineJob({
  name: "webhooks.deliver",
  removeOnFail: 200, // keep only the newest 200
  async handle(payload) {
    /* ... */
  },
});

removeOnFail: true removes a failed job immediately instead — use that only when you never intend to inspect or retry failures for that job.