Attach middleware
Middleware in @warlock.js/ai wraps an agent run at three granularities — execute, trip, tool. One middleware is one object. The package ships three built-ins and a small helper surface for composing your own.
Install order at a glance
Section titled “Install order at a glance”import { ai } from "@warlock.js/ai";import { OpenAISDK } from "@warlock.js/ai-openai";import { cache } from "@warlock.js/cache";
const openai = new OpenAISDK({ apiKey: process.env.OPENAI_API_KEY! });
ai.config({ defaultStore: cache.driver("redis", { client: redisClient }) });
const myAgent = ai.agent({ model: openai.model({ name: "gpt-4o" }), middleware: [ ai.middleware.semanticCache({ embedder: openai.embedder({ name: "text-embedding-3-small" }), threshold: 0.95, }), ai.middleware.budget({ maxTokens: 50_000 }), ai.middleware.guardrail({ inputCheck: async (text) => text.match(/\bSSN\b/) ? { ok: false, reason: "pii" } : { ok: true }, }), ],});Canonical order: [cache, budget, guardrail, observability]. See “Ordering invariants” below for why.
ai.middleware.budget
Section titled “ai.middleware.budget”Cumulative token / USD cap across all trips of one execution.
ai.middleware.budget({ maxTokens: 50_000, maxCostUSD: 0.5, // ModelPricing (USD per 1M, cache rates included); the legacy // { inputPer1K, outputPer1K } shape is still accepted. pricing: { "gpt-4o": { input: 2.5, output: 10, cachedInput: 1.25 } }, onExceeded: "abort", // or "warn" reserve: { outputTokens: 4_096 }, // optional: refuse a trip before it is sent});Breach → BudgetExceededError on result.error. Inspect error.unit ("tokens" | "usd"), error.limit, error.actual. Warn mode logs and continues — useful for measuring before enforcing.
With reserve, each trip’s worst case (an input-token estimate plus outputTokens) is reserved before the call: a trip that would cross a cap is never sent and fails with context.reason: "reservation"; after the call the reservation is replaced by the actual usage, and a failed call releases it. With scoped, the reservation is taken atomically in the shared ledger. See Budgets and SLOs.
USD pricing is fail-closed
Section titled “USD pricing is fail-closed”When you set maxCostUSD, the middleware resolves pricing in this order:
- The explicit
budget({ pricing })entry for the running model. - The model’s resolved
ModelPricing, whether it came fromsdk.model({ pricing }), SDK-level pricing, or the adapter’s built-in list prices.
An explicit budget entry takes precedence over model/SDK pricing. If neither source resolves a price, the run fails before its first model call with BudgetExceededError; the error names the unpriced model and tells you to configure pricing or opt out. This prevents a dollar cap from silently becoming unenforced.
Use onUnpriced: "allow" only when accepting that risk is intentional:
ai.middleware.budget({ maxCostUSD: 0.5, onUnpriced: "allow", // permit a model with no resolved ModelPricing});ai.middleware.guardrail
Section titled “ai.middleware.guardrail”Pre / post content checks.
ai.middleware.guardrail({ inputCheck: async (text, ctx) => text.includes("forbidden") ? { ok: false, reason: "policy-1" } : { ok: true }, outputCheck: async (text) => text.length > 10_000 ? { ok: false, reason: "too-long" } : { ok: true }, name: "pii-guardrail",});Rejection → GuardrailViolationError with phase: "input" | "output" and the configured reason. Output checks fire BEFORE tool dispatch — a rejected response means the tools it requested are never invoked.
outputCheck runs on every trip (including tool follow-ups and repair attempts). inputCheck runs once per distinct user text per run: a later trip whose text already passed skips it.
In a streaming run (agent.stream(), ai.serve), outputCheck holds each trip’s text until it passes, then streams the text as one delta; a rejected trip streams no text, and reasoning deltas are not streamed. Set stream: "unguarded" to stream deltas before the check runs (the check then only changes the result). A custom middleware that checks or rewrites response.content in trip.after gets the same hold by setting trip: { bufferStream: true, after(ctx, response) { … } }.
ai.middleware.semanticCache
Section titled “ai.middleware.semanticCache”Two-tier cache — exact-match key first, vector similarity second. Delegates to any vector-capable CacheDriver.
ai.middleware.semanticCache({ embedder: openai.embedder({ name: "text-embedding-3-small" }), // store optional — falls back to ai.config({ defaultStore }) store: cache.driver("pg", { client: pgPool, vector: { dimensions: 1536, index: "hnsw" }, }), threshold: 0.95, ttlMs: 60 * 60 * 1000, namespace: "support-faq", // scope: "session" (default) — see "Session-scoped by default" below // cacheToolRuns: false (default) — answers built from tool results are not stored});Driver requirements. The store must support similar() — pg (with vector config), redis (with RediSearch), or memory drivers for dev. Without similarity → CacheUnsupportedError at first vector op.
How it works.
- Exact-match — SHA-256 over the whole message list.
store.get(hash)returns an instant hit; the stored prompt hash, scope and context are re-checked before it is served. - Vector-match — embeds the last user message, calls
store.similar(vector, { topK, threshold }). The driver uses its native ANN index; when other scopes or conversations fill the first page of candidates, the page is widened so this caller’s own entry is still found. - Same context only. Every entry records a fingerprint of the agent, its system prompt and the conversation before the question (plus any image or file attached to it). Both paths require it to match, so two agents in one session, or the same “yes, go ahead” at two points of a conversation, never share an answer.
- Hits return a synthetic
ModelResponsewithusage: { input: 0, output: 0, total: 0 }. - Writes happen at
trip.afteron miss. - Trip-zero only. Only first-trip responses get cached. Tool-using loops never cache tool-call responses — would infinite-loop.
- Tool runs are not stored by default. When the first turn calls tools, the final answer is built from tool results (a balance, a stock level) that can change or belong to one caller, so it is not cached. Set
cacheToolRuns: trueto store it under the first turn’s prompt. - Never use memory drivers in production — they do a linear scan per query.
Session-scoped by default
Section titled “Session-scoped by default”A semanticCache is normally built once at app boot and shared by every end user, and a hit is returned as the model’s answer with no LLM call in between — so without isolation, one caller’s merely-similar prompt can be served another caller’s cached answer, personal context included. SemanticCacheOptions.scope defaults to "session": every entry is keyed off the run’s AgentExecuteOptions.sessionId (as "session:<id>"), and a lookup re-checks that key as exact equality on the stored entry before serving it — the cache key alone never authorizes a read.
// Opt back into one shared pool — e.g. a genuinely public FAQ bot where no// response carries a caller's private context:ai.middleware.semanticCache({ embedder, threshold: 0.95, scope: "shared" });
// Derive your own isolation boundary, e.g. per tenant:ai.middleware.semanticCache({ embedder, threshold: 0.95, scope: (ctx) => tenantIdFrom(ctx) });"session"(default) — isolated persessionId. A run made without asessionIdskips the cache: it neither reads nor writes (a one-time warning names the fix). PasssessionIdonagent.execute()(a supervisor / orchestrator forwards its own automatically)."shared"— one pool for every caller, regardless of session. Use it for answers that may be served to anyone.- a resolver function —
(context) => key | undefined;undefined(or"") skips the cache for that run. Return a constant to pool such runs on purpose.
Entries written by earlier releases carry no context fingerprint and are no longer served; the cache refills as questions come in.
ai.middleware.dedupe
Section titled “ai.middleware.dedupe”When a burst of users sends the same prompt at the same moment (a popular question after an email campaign, a retried webhook), every run would make its own provider call. dedupe() lets the first one go to the model and hands its response to the identical requests that arrive while it is in flight:
const myAgent = ai.agent({ model, middleware: [ai.middleware.semanticCache({ embedder, threshold: 0.95 }), ai.middleware.dedupe()],});- “Identical” means the same agent, model, messages and
modelOptions. Passdedupe({ key: (ctx) => ... })to choose your own key, or returnundefinedto leave a call alone. - The waiting runs get the response with zero usage, so budgets count one call.
- Nothing is kept once the call ends; for reuse over time use
semanticCache. Listdedupe()after it, so a cache hit never starts a flight. - If the first call fails, each waiting run makes its own call.
- The in-flight map is local to the process: two servers still make two calls.
- Streaming runs (
agent.stream()) are not deduplicated: every stream must deliver its own chunks as they arrive.
Writing your own middleware
Section titled “Writing your own middleware”One object, any subset of three hook maps:
import type { AgentMiddleware } from "@warlock.js/ai";
const latencyLogger: AgentMiddleware = { name: "latency-logger", execute: { before(ctx) { ctx.state.set("latency.start", performance.now()); }, after(ctx, result) { const start = ctx.state.get("latency.start") as number; console.log(`agent ${ctx.agent.name} finished in ${performance.now() - start}ms`); }, }, trip: { before(ctx) { ctx.state.set(`latency.trip.${ctx.tripIndex}.start`, performance.now()); }, after(ctx) { const start = ctx.state.get(`latency.trip.${ctx.tripIndex}.start`) as number; console.log(` trip ${ctx.tripIndex}: ${performance.now() - start}ms`); }, },};- Never close over mutable state. Use
ctx.state— fresh perexecute()call. - Abort with a typed
AIErrorsubclass. Neverthrow new Error(...). - Short-circuit by returning from
before. Return the level’s result type — the pipeline skips the real work and outerafterhooks still run on your synthetic value. onErroris opt-in recovery. Return a value to recover; returnvoidto let the error propagate.log: falsesuppresses framework debug emission for that middleware (the middleware itself still runs).
Ordering invariants — read before shipping
Section titled “Ordering invariants — read before shipping”- Cache MUST be outermost when guardrails are present. Guardrail
trip.afterthrows to reject bad output, butafterhooks run bottom-up. If guardrail is outside the cache, rejection fires AFTER the cache has written the bad response. Canonical order[cache, budget, guardrail]keeps rejected outputs out of the cache. - Budget before guardrails. Guardrails may call classifiers with their own token costs.
- Observability last. It should see the final decision every other middleware made.
The order you pass to middleware: [...] IS the order they nest. First in the array = outermost.
Helpers
Section titled “Helpers”ai.middleware.compose(...sources)
Section titled “ai.middleware.compose(...sources)”Flatten multiple sources into one ordered array. No sorting, no dedup.
ai.agent({ model, middleware: ai.middleware.compose(standardStack, toolRules, auditMiddleware),});ai.middleware.forTool(name | names, middleware)
Section titled “ai.middleware.forTool(name | names, middleware)”Scope tool.* hooks to specific tool names. execute and trip hooks pass through unscoped.
const scoped = ai.middleware.forTool(["paid_api", "expensive_db"], toolRateLimit({ maxCalls: 5 }));Caveats
Section titled “Caveats”tool.onErroris almost-never-useful.ToolContract.invoke()never throws — errors are captured intoresult.error.tool.onErroronly fires when another middleware’stool.before/tool.afteritself throws. For “the tool itself failed”, branch onresult.errorin atool.after.- Middleware does NOT observe unregistered tool calls. When the model asks for a tool the agent wasn’t configured with, the pipeline is bypassed and a failed
ToolCallis recorded directly. namemust be unique across an agent’s middleware array.- Middleware state does NOT cross
agent.execute()boundaries. One execute → one freshctx.state.
Workflow + middleware — what works today
Section titled “Workflow + middleware — what works today”- Inside a workflow step with
agent: myAgent— the agent’s own middleware fires normally. workflow.asTool()called from an agent — the calling agent’stool-level middleware wraps the workflow.- Step-level / workflow-level / supervisor-level middleware does NOT exist yet.
Related
Section titled “Related”- Run agent — agent lifecycle the middleware wraps.
- Persist AI data —
defaultStorefor semantic cache. - Handle errors —
BudgetExceededError/GuardrailViolationError.