Caching and Request Deduplication in High-Volume Agent Web Pipelines

A multi-step agent sends its system prompt with every single API call it makes, not once at the start of a task but at step one, step five, step twelve, and step thirty. None of this is sloppy engineering. It has made the same mistake thirty times, and the invoice reflects every one of them. This is the structural duplication tax: a cost baked into the architecture of agentic systems rather than a bug sitting in any one of them. To the authors' knowledge, this is the first study to quantify these cost benefits and compare caching strategies head to head for agentic workloads, which says something about how recently the field has started treating this as a problem worth measuring.

Three caching layers for three kinds of repetition

An agent pipeline has three distinct kinds of repetition, and each one needs its own mechanism to catch it. Identical prompts call for exact-match caching. Semantically equivalent queries, phrased differently but asking the same thing, call for semantic caching. Repeated token prefixes, the long stable stretches of a system prompt that cross call after call, call for provider-side prompt or KV caching. No single layer covers all three, and that is the premise the rest of this piece builds on.

Layer 1, exact-match caching, works by hashing the incoming query, typically with SHA-256, against a normalized version of the full request payload, including the model name, the messages, and the generation parameters. This happens before the request ever reaches the model, which makes it the fastest of the three layers whenever it hits.

Layer 2, semantic caching, stores past LLM responses alongside vector embeddings of the queries that produced them. Bifrost, an open-source, gateway-native implementation of this pattern, supports Weaviate, Redis (including Valkey), Qdrant, and Pinecone as vector store backends, and lets teams set TTL, similarity threshold, and cache type per request through headers like x-bf-cache-ttl, x-bf-cache-threshold, and x-bf-cache-type, which matters for pipelines serving mixed traffic. Bifrost's own tuning guidance recommends starting the similarity threshold at 0.92 for customer-facing applications and 0.88 for internal tooling, on the reasoning that customers tolerate less semantic drift in an answer than an internal team running its own workflows. Cache keys in Bifrost are namespaced by model and provider by default, so a response generated under one model configuration never leaks into a cache lookup for another.

Provider-side prompt caching in agentic tasks

Provider-side caching pays off consistently, but only when the cached prefix stays stable from one call to the next, and the way most agent pipelines are built works directly against that stability. Lumer et al. tested three strategies against each other: caching the full context, caching the system prompt alone, and caching everything except dynamic tool results. The finding that stands out is that controlling which blocks of the prompt get cached produces more consistent benefit than simply caching the whole context, which can make latency worse.

Three structural rules fall out of that comparison. Dynamic content belongs at the end of the system prompt, not the beginning or the middle, so the cacheable prefix in front of it stays fixed. Dynamic tool results should be excluded from the cached block.

The clearest illustration of what happens when those rules get ignored, and what happens when they're followed, comes from ProjectDiscovery's Neo security agent. Neo runs 20 to 40-plus LLM steps on a single task, and it carries a 20,000-token system prompt the whole way through. The cache hit rate sat at 7 percent. The fix was structural: move the dynamic working memory out of the system prompt and into a user message placed at the end of the prompt instead. Provider-side caching does what it says it will do, but the failure sits in how the prompt gets structured around it. Anthropic exposes this through explicit cache markers placed in the prompt, but OpenAI matches prefixes automatically and needs no marking from the developer, and either approach can hit the 74 to 84 percent range Neo reached once you move the dynamic content out of the way.

Provider changes affecting dynamic tool calling in cached pipelines since September 2026

Dynamic tool discovery has been the single biggest reason provider-side caching underperforms in modern agentic systems. Protocols like MCP let the set of available tools shift depending on which servers are connected or what the runtime context looks like at a given moment, and any change to the tool definitions sent with a request invalidates the cached prefix built around the old set. Between September 22 and 24, 2026, both OpenAI and Anthropic made changes aimed at this specific problem. Neither change makes dynamic tool sets a non-issue. What they do is raise the ceiling on how much of a tool-heavy pipeline can stay cached across steps, which is a meaningful gain for anyone building on MCP-style architectures, but it does not remove the underlying tension between flexible tooling and stable prefixes.

Semantic caching and intent-level key design

Embedding similarity alone breaks down for agentic workloads in a specific and dangerous way: two queries can sit close together in embedding space while requiring completely different sequences of tool calls to answer correctly. "Check email from Alice" and "send email to Alice" will score high on cosine similarity, but one reads a mailbox and the other sends a message, and a semantic cache keyed purely on similarity will happily serve the wrong plan for one of them. That is not a near-miss; it's a wrong answer delivered with confidence.

W5H2, a structured intent decomposition framework, decomposes a query into action-target cache keys. Basu describes a five-tier cascade built on this idea: it moves from fingerprint matching to BERT to SetFit to a cheap LLM and finally to a full agent call only if nothing upstream resolves the request, so it handles most interactions locally at zero marginal cost and models a 97.5 percent reduction in API cost under the paper's assumed traffic distribution. Key design is the whole problem, and no amount of embedding sophistication substitutes for getting the key right.

Interactions between the three layers and the new failure modes naive combination creates

Running exact-match, semantic, and provider-side caching together cuts the duplication tax by a wide margin, but stacking the three layers introduces problems that none of them has on its own. Order matters here. Exact-match caching should run first, since a hash lookup costs almost nothing and requires no embedding call. Semantic matching should run second, catching the paraphrases and near-duplicates the hash missed. Provider-side KV caching comes last, as the fallback for whatever actually reaches the model. Running these out of order wastes compute generating embeddings for requests a simple hash lookup would have caught for free.

The sharper risk is silent invalidation, which appears differently at each layer: at the semantic layer, a cache hit at 0.91 cosine similarity can return the answer to a neighboring paraphrase that is, in fact, factually different from what the current query needs. Neither failure appears in a hit-rate dashboard. Both surface in a post-mortem, after a user has already acted on a wrong answer.

Statefulness compounds the risk further. A cache key built from the current query alone, with no reference to where the agent is in a multi-step task, produces a higher hit rate but will return a cached response that was correct earlier in the conversation and wrong now, because the same literal query can mean something different depending on what came before it. Keying the cache on the last several turns of conversation, not just the current message, lowers the hit rate but prevents that kind of wrong-context hit, and it has become the standard approach for stateful agentic pipelines for that reason.

Multi-tenant deployments add a third failure mode on top of the first two: leakage. Semantic caches carry no such guarantee and need explicit per-tenant key scoping at the vector store level to keep one tenant's cached data from surfacing in another tenant's results.

Request deduplication as a distinct mechanism that caching alone cannot replace

Caching solves repetition across time: a question asked yesterday gets answered today without another trip to the model. Request deduplication solves a different problem: repetition across concurrent requests arriving at the same moment, and none of them can benefit from a cache entry that doesn't exist yet, because nothing has populated it. Picture fifty users asking the same question in the same second, right as the one cached answer for it expires. Every one of those fifty requests sees a cache miss, and without deduplication logic, all fifty proceed to the model independently, so each pays full inference cost for an answer the other forty-nine are about to receive anyway. This pattern of overwhelming simultaneous requests strikes at exactly the moment a cache is least able to help: the cold start or the expiry event.

Deduplication solves it by collapsing those concurrent in-flight requests into a single upstream call. Production-grade agentic pipelines treat this as its own stage sitting between ingestion and serving, not as a property the cache is somehow expected to provide on its own. Caching and deduplication solve adjacent but separate problems, and a pipeline that only builds one will still bleed cost at scale.

The reproducibility risk of aggressive caching in pipelines where consistency and auditability matter

Prefix caching introduces a cost most teams don't expect: it makes LLM serving non-reproducible. For agents operating in compliance, legal, medical, or financial contexts, where consistency and auditability are the whole point, it is the strongest argument against caching aggressively at the provider-KV layer.

The practical consequence runs deeper than it first appears: whether a given request hits the cache or misses it is not a neutral routing decision, because a hit and a miss can produce genuinely different outputs for the same input. Logging every response with its cache hit or miss status and a cache ID helps here, and Bifrost does this through a cache_debug object attached to every response, carrying hit type, cache ID, and, on semantic hits, the similarity score that triggered the match. The more durable fix is selective: disable provider-side KV caching on the specific agent paths where output consistency is a compliance requirement, while keeping exact-match and semantic caching active everywhere else. You give up Layer 3's savings only where reproducibility actually matters, so you don't need to abandon caching across the whole pipeline.

Emerging approaches that rethink what the unit of caching should be

The three-layer model described above caches responses and token prefixes, so it treats the output of a call, or the text leading into it, as the thing worth storing. A newer line of research asks whether that's even the right unit to cache in the first place: it proposes instead that structured plans and intermediate representations transfer better across paraphrased inputs, and they resist the key-consistency failures that trip up embedding-based semantic caching.

Agentic Plan Caching, from Zhang et al. at NeurIPS 2025, caches structured plan templates pulled from completed agent execution logs rather than raw responses, then uses keyword extraction to match new requests against those templates and retrieve the closest one. SemanticALLI (Chillara et al., arXiv:2601.16286, 2026) takes a related but distinct approach, splitting LLM pipelines into an Analytic Intent Resolution stage and a Visualization Synthesis stage, each with its own caching mechanism, and caches structured intermediate representations at logical checkpoints within the pipeline, including its inputs and outputs. Leyline's KV cache directives (arXiv:2606.01065, 2026) push the idea further still, proposing that agents get given active, controllable eviction signals rather than sitting on top of a passive, policy-agnostic cache that evicts in forward-only order regardless of what the agent actually needs kept. None of this displaces the three-layer model as the right current approach for a production pipeline. It does mean the question of what, exactly, deserves to be cached is still an open one, and the answer the field settles on in a year may look different from the response-and-prefix model in wide use today.

A decision framework for deploying the three layers in a production agent pipeline

Running all three caching layers plus deduplication in every pipeline, regardless of what that pipeline actually does, is not a sound default. The right combination depends on the shape of the query distribution, how stateful the agent is, how much latency the use case tolerates, and how strict the auditability requirements are.

A pipeline with high semantic overlap in its queries, like customer support, FAQ handling, or an internal knowledge base, gets the largest marginal return from semantic caching at Layer 2, with exact-match caching at Layer 1 mopping up the literal duplicates at almost no cost. A pipeline built around a large, stable system prompt reused across many steps, the pattern typical of research agents and coding assistants, gets the most value from Layer 3, provided the structural fix illustrated by ProjectDiscovery is in place first: dynamic content moved out of the system prompt and into a user message, so the cached prefix actually stays stable. A pipeline relying on dynamic tool sets through MCP or a similar protocol should apply the September 2026 provider changes, OpenAI's tool masking and Anthropic's mid-conversation tool appending, before assuming Layer 3 is a lost cause for tool-heavy work. And if a pipeline faces high concurrent request volume, it needs explicit deduplication logic sitting ahead of the cache, because caching by itself does nothing to stop a thundering herd at a cold start or an expiry event.

What ties all of these decisions together is observability. A caching layer that can't be inspected, that gives no visibility into whether a given response came from a hash match, a semantic match, or a fresh model call, is a liability once it's running at scale, not a saving.