Skip to content

Health checks & graceful shutdown

A deploy should never drop a request, and a load balancer should never route to an instance that isn’t ready. Warlock ships both halves built in: two endpoints that report whether the instance should receive traffic, and a shutdown sequence that drains in-flight requests instead of cutting them off. No /health controller to hand-roll, no terminus-style glue.

The HTTP connector registers them directly on the Fastify server during boot — before route scanning — so they exist by the time the server listens, and they’re immune to HMR.

PathProbe200 when503 when
/healthlivenessthe process is upshutdown has begun
/readyreadinessbooted and not shutting down and every check passesbefore boot, during shutdown, or any failing check

The distinction matters for orchestrators:

  • Liveness answers “should you RESTART me?” — so it ignores dependency checks. A flaky database doesn’t mean the process is dead; restarting wouldn’t help.
  • Readiness answers “should you ROUTE to me?” — so it gates on boot completion, shutdown state, and your registered checks.
GET /health → 200 {"status":"ok"}
GET /ready → 200 {"status":"ok","checks":{"db":true}}
→ 503 {"status":"error","checks":{"db":false}}

In production /ready answers {"status":"ok"} / {"status":"error"} only: the probe is public, and the check names would tell anyone which backing services exist and which are down. Set http.health.details to choose (since 5.53).

Both endpoints are on by default. Tune them in src/config/http.ts:

src/config/http.ts
import type { HttpConfigurations } from "@warlock.js/core";
const httpConfigurations: HttpConfigurations = {
health: {
enabled: true, // default; set false to remove both endpoints
path: "/health", // liveness path
readinessPath: "/ready", // readiness path
rateLimit: false, // exempt probes from global client quotas (default)
details: false, // list checks in the /ready body (default: false in production, true elsewhere)
checkTimeout: 2000, // ms before a check counts as failed (default; 0 waits forever)
cacheTtl: 1000, // ms one readiness result is reused (default; 0 reruns every probe)
},
};
export default httpConfigurations;

Built-in liveness and readiness routes bypass http.rateLimit by default, including renamed probe paths. Application traffic cannot consume their quota, and probe traffic does not consume the application’s quota. Their normal liveness/readiness status still applies.

Set http.health.rateLimit: true to opt both probes into the global policy and shared client bucket. This requires the rate-limit plugin to be enabled and http.rateLimit.global to remain enabled; exhausted quotas can then return 429 for probes too. Disabling global limits leaves opted-in probes unlimited.

Static files follow the same rule: the public folder, router.directory() mounts, router.file() routes and the web package’s built assets are exempt by default, and http.rateLimit.staticFiles: true counts them. Pages and API routes always share the global bucket. See Rate limiting → Static files.

Out of the box, readiness is isBooted && !isShuttingDown. Layer on dependency checks — a database ping, a cache round-trip, a downstream health call — and /ready only reports ready when they all pass:

import { health } from "@warlock.js/core";
health.addCheck("db", async () => database.isConnected());
health.addCheck("cache", async () => cache.ping());
// later, if you need to drop one:
health.removeCheck("db");

A check returns boolean | Promise<boolean>. A thrown error counts as a failed check — it’s reported in the checks map and the 503, not logged (probes poll constantly; a failure is a normal signal, not an error event).

Probes stay cheap and bounded however often /ready is hit: concurrent probes share one run of the checks, a finished result is reused for http.health.cacheTtl ms (default 1000), and a check that does not answer within http.health.checkTimeout ms (default 2000) counts as failed, so a hanging dependency cannot hold the probe. Adding or removing a check drops the reused result.

A natural place to register a check is inside Application.onceBooted(...), or from a custom connector’s start().

When the process receives SIGINT/SIGTERM, Warlock tears down in two stages — app onShutdown hooks first (while connectors are still up), then connectors in reverse priority order. The HTTP connector’s teardown drains rather than drops:

flowchart TD
    sig["SIGTERM"]
    flag["Application.isShuttingDown = true<br/><i>/ready now returns 503</i>"]
    lb["Load balancer stops routing<br/>new traffic to this instance"]
    drain["Fastify stops accepting new requests,<br/>in-flight requests finish<br/><i>bounded by gracefulShutdown.timeout</i>"]
    close["Connectors close in reverse → process exits"]

    sig --> flag
    flag --> lb
    flag --> drain
    drain --> close
  1. Application.isShuttingDown flips true immediately, so /ready returns 503 and the load balancer drains this instance.
  2. Fastify stops accepting new connections (answering 503 while closing) and lets in-flight requests finish.
  3. Draining is bounded by a timeout so one stuck request can’t hang the deploy; after it, the server force-closes and logs a warning.
src/config/http.ts
const httpConfigurations: HttpConfigurations = {
gracefulShutdown: {
timeout: 10_000, // ms to wait for in-flight drain (default 10s)
forceCloseConnections: "idle", // close idle keep-alives, let active finish (default)
},
};

forceCloseConnections: "idle" (default) closes idle keep-alive connections and lets active requests finish; true force-closes everything immediately; false waits for every connection.

Under "idle", a request that is still running when shutdown begins answers with Connection: close, and its keep-alive connection is closed as soon as the response finishes, so a browser’s kept-alive connection does not hold the drain open until the timeout. Responses that had already sent their headers (streams, SSE) are closed when they end; a stream or SSE response that starts after shutdown began also answers with Connection: close. Active responses are never cut short.

  • /health is registered on Fastify, not the app router. That keeps it immune to HMR and route scanning — but if your app also defines a /health route, they collide. Rename via http.health.path.
  • Readiness needs a finished boot. Before Application.isBooted — e.g. while late-phase connectors are still starting — /ready is 503 by design. That’s the point: don’t route traffic to a half-booted instance.
  • A hanging onShutdown hook delays the drain. App hooks run before connector teardown and are only bounded by your process manager’s kill timeout. Keep them fast; the HTTP drain itself is bounded by gracefulShutdown.timeout.
  • Maintenance mode is a different 503. The maintenance middleware allowlists /health so probes pass during maintenance, but it’s operator-toggled downtime, not readiness.