Skip to content

Attach middleware

Middleware in @warlock.js/ai wraps an agent run at three granularities — execute, trip, tool. One middleware is one object. The package ships three built-ins and a small helper surface for composing your own.

import { ai } from "@warlock.js/ai";
import { OpenAISDK } from "@warlock.js/ai-openai";
import { cache } from "@warlock.js/cache";
const openai = new OpenAISDK({ apiKey: process.env.OPENAI_API_KEY! });
ai.config({ defaultStore: cache.driver("redis", { client: redisClient }) });
const myAgent = ai.agent({
model: openai.model({ name: "gpt-4o" }),
middleware: [
ai.middleware.semanticCache({
embedder: openai.embedder({ name: "text-embedding-3-small" }),
threshold: 0.95,
}),
ai.middleware.budget({ maxTokens: 50_000 }),
ai.middleware.guardrail({
inputCheck: async (text) =>
text.match(/\bSSN\b/) ? { ok: false, reason: "pii" } : { ok: true },
}),
],
});

Canonical order: [cache, budget, guardrail, observability]. See “Ordering invariants” below for why.

Cumulative token / USD cap across all trips of one execution.

ai.middleware.budget({
maxTokens: 50_000,
maxCostUSD: 0.5,
// ModelPricing (USD per 1M, cache rates included); the legacy
// { inputPer1K, outputPer1K } shape is still accepted.
pricing: { "gpt-4o": { input: 2.5, output: 10, cachedInput: 1.25 } },
onExceeded: "abort", // or "warn"
reserve: { outputTokens: 4_096 }, // optional: refuse a trip before it is sent
});

Breach → BudgetExceededError on result.error. Inspect error.unit ("tokens" | "usd"), error.limit, error.actual. Warn mode logs and continues — useful for measuring before enforcing.

With reserve, each trip’s worst case (an input-token estimate plus outputTokens) is reserved before the call: a trip that would cross a cap is never sent and fails with context.reason: "reservation"; after the call the reservation is replaced by the actual usage, and a failed call releases it. With scoped, the reservation is taken atomically in the shared ledger. See Budgets and SLOs.

When you set maxCostUSD, the middleware resolves pricing in this order:

  1. The explicit budget({ pricing }) entry for the running model.
  2. The model’s resolved ModelPricing, whether it came from sdk.model({ pricing }), SDK-level pricing, or the adapter’s built-in list prices.

An explicit budget entry takes precedence over model/SDK pricing. If neither source resolves a price, the run fails before its first model call with BudgetExceededError; the error names the unpriced model and tells you to configure pricing or opt out. This prevents a dollar cap from silently becoming unenforced.

Use onUnpriced: "allow" only when accepting that risk is intentional:

ai.middleware.budget({
maxCostUSD: 0.5,
onUnpriced: "allow", // permit a model with no resolved ModelPricing
});

Pre / post content checks.

ai.middleware.guardrail({
inputCheck: async (text, ctx) =>
text.includes("forbidden") ? { ok: false, reason: "policy-1" } : { ok: true },
outputCheck: async (text) =>
text.length > 10_000 ? { ok: false, reason: "too-long" } : { ok: true },
name: "pii-guardrail",
});

Rejection → GuardrailViolationError with phase: "input" | "output" and the configured reason. Output checks fire BEFORE tool dispatch — a rejected response means the tools it requested are never invoked.

outputCheck runs on every trip (including tool follow-ups and repair attempts). inputCheck runs once per distinct user text per run: a later trip whose text already passed skips it.

In a streaming run (agent.stream(), ai.serve), outputCheck holds each trip’s text until it passes, then streams the text as one delta; a rejected trip streams no text, and reasoning deltas are not streamed. Set stream: "unguarded" to stream deltas before the check runs (the check then only changes the result). A custom middleware that checks or rewrites response.content in trip.after gets the same hold by setting trip: { bufferStream: true, after(ctx, response) { … } }.

Two-tier cache — exact-match key first, vector similarity second. Delegates to any vector-capable CacheDriver.

ai.middleware.semanticCache({
embedder: openai.embedder({ name: "text-embedding-3-small" }),
// store optional — falls back to ai.config({ defaultStore })
store: cache.driver("pg", {
client: pgPool,
vector: { dimensions: 1536, index: "hnsw" },
}),
threshold: 0.95,
ttlMs: 60 * 60 * 1000,
namespace: "support-faq",
// scope: "session" (default) — see "Session-scoped by default" below
// cacheToolRuns: false (default) — answers built from tool results are not stored
});

Driver requirements. The store must support similar() — pg (with vector config), redis (with RediSearch), or memory drivers for dev. Without similarity → CacheUnsupportedError at first vector op.

How it works.

  • Exact-match — SHA-256 over the whole message list. store.get(hash) returns an instant hit; the stored prompt hash, scope and context are re-checked before it is served.
  • Vector-match — embeds the last user message, calls store.similar(vector, { topK, threshold }). The driver uses its native ANN index; when other scopes or conversations fill the first page of candidates, the page is widened so this caller’s own entry is still found.
  • Same context only. Every entry records a fingerprint of the agent, its system prompt and the conversation before the question (plus any image or file attached to it). Both paths require it to match, so two agents in one session, or the same “yes, go ahead” at two points of a conversation, never share an answer.
  • Hits return a synthetic ModelResponse with usage: { input: 0, output: 0, total: 0 }.
  • Writes happen at trip.after on miss.
  • Trip-zero only. Only first-trip responses get cached. Tool-using loops never cache tool-call responses — would infinite-loop.
  • Tool runs are not stored by default. When the first turn calls tools, the final answer is built from tool results (a balance, a stock level) that can change or belong to one caller, so it is not cached. Set cacheToolRuns: true to store it under the first turn’s prompt.
  • Never use memory drivers in production — they do a linear scan per query.

A semanticCache is normally built once at app boot and shared by every end user, and a hit is returned as the model’s answer with no LLM call in between — so without isolation, one caller’s merely-similar prompt can be served another caller’s cached answer, personal context included. SemanticCacheOptions.scope defaults to "session": every entry is keyed off the run’s AgentExecuteOptions.sessionId (as "session:<id>"), and a lookup re-checks that key as exact equality on the stored entry before serving it — the cache key alone never authorizes a read.

// Opt back into one shared pool — e.g. a genuinely public FAQ bot where no
// response carries a caller's private context:
ai.middleware.semanticCache({ embedder, threshold: 0.95, scope: "shared" });
// Derive your own isolation boundary, e.g. per tenant:
ai.middleware.semanticCache({ embedder, threshold: 0.95, scope: (ctx) => tenantIdFrom(ctx) });
  • "session" (default) — isolated per sessionId. A run made without a sessionId skips the cache: it neither reads nor writes (a one-time warning names the fix). Pass sessionId on agent.execute() (a supervisor / orchestrator forwards its own automatically).
  • "shared" — one pool for every caller, regardless of session. Use it for answers that may be served to anyone.
  • a resolver function — (context) => key | undefined; undefined (or "") skips the cache for that run. Return a constant to pool such runs on purpose.

Entries written by earlier releases carry no context fingerprint and are no longer served; the cache refills as questions come in.

When a burst of users sends the same prompt at the same moment (a popular question after an email campaign, a retried webhook), every run would make its own provider call. dedupe() lets the first one go to the model and hands its response to the identical requests that arrive while it is in flight:

const myAgent = ai.agent({
model,
middleware: [ai.middleware.semanticCache({ embedder, threshold: 0.95 }), ai.middleware.dedupe()],
});
  • “Identical” means the same agent, model, messages and modelOptions. Pass dedupe({ key: (ctx) => ... }) to choose your own key, or return undefined to leave a call alone.
  • The waiting runs get the response with zero usage, so budgets count one call.
  • Nothing is kept once the call ends; for reuse over time use semanticCache. List dedupe() after it, so a cache hit never starts a flight.
  • If the first call fails, each waiting run makes its own call.
  • The in-flight map is local to the process: two servers still make two calls.
  • Streaming runs (agent.stream()) are not deduplicated: every stream must deliver its own chunks as they arrive.

One object, any subset of three hook maps:

import type { AgentMiddleware } from "@warlock.js/ai";
const latencyLogger: AgentMiddleware = {
name: "latency-logger",
execute: {
before(ctx) {
ctx.state.set("latency.start", performance.now());
},
after(ctx, result) {
const start = ctx.state.get("latency.start") as number;
console.log(`agent ${ctx.agent.name} finished in ${performance.now() - start}ms`);
},
},
trip: {
before(ctx) {
ctx.state.set(`latency.trip.${ctx.tripIndex}.start`, performance.now());
},
after(ctx) {
const start = ctx.state.get(`latency.trip.${ctx.tripIndex}.start`) as number;
console.log(` trip ${ctx.tripIndex}: ${performance.now() - start}ms`);
},
},
};
  • Never close over mutable state. Use ctx.state — fresh per execute() call.
  • Abort with a typed AIError subclass. Never throw new Error(...).
  • Short-circuit by returning from before. Return the level’s result type — the pipeline skips the real work and outer after hooks still run on your synthetic value.
  • onError is opt-in recovery. Return a value to recover; return void to let the error propagate.
  • log: false suppresses framework debug emission for that middleware (the middleware itself still runs).

Ordering invariants — read before shipping

Section titled “Ordering invariants — read before shipping”
  1. Cache MUST be outermost when guardrails are present. Guardrail trip.after throws to reject bad output, but after hooks run bottom-up. If guardrail is outside the cache, rejection fires AFTER the cache has written the bad response. Canonical order [cache, budget, guardrail] keeps rejected outputs out of the cache.
  2. Budget before guardrails. Guardrails may call classifiers with their own token costs.
  3. Observability last. It should see the final decision every other middleware made.

The order you pass to middleware: [...] IS the order they nest. First in the array = outermost.

Flatten multiple sources into one ordered array. No sorting, no dedup.

ai.agent({
model,
middleware: ai.middleware.compose(standardStack, toolRules, auditMiddleware),
});

ai.middleware.forTool(name | names, middleware)

Section titled “ai.middleware.forTool(name | names, middleware)”

Scope tool.* hooks to specific tool names. execute and trip hooks pass through unscoped.

const scoped = ai.middleware.forTool(["paid_api", "expensive_db"], toolRateLimit({ maxCalls: 5 }));
  • tool.onError is almost-never-useful. ToolContract.invoke() never throws — errors are captured into result.error. tool.onError only fires when another middleware’s tool.before / tool.after itself throws. For “the tool itself failed”, branch on result.error in a tool.after.
  • Middleware does NOT observe unregistered tool calls. When the model asks for a tool the agent wasn’t configured with, the pipeline is bypassed and a failed ToolCall is recorded directly.
  • name must be unique across an agent’s middleware array.
  • Middleware state does NOT cross agent.execute() boundaries. One execute → one fresh ctx.state.

Workflow + middleware — what works today

Section titled “Workflow + middleware — what works today”
  • Inside a workflow step with agent: myAgent — the agent’s own middleware fires normally.
  • workflow.asTool() called from an agent — the calling agent’s tool-level middleware wraps the workflow.
  • Step-level / workflow-level / supervisor-level middleware does NOT exist yet.