Prompt Caches Leak Across Tenants

A customer’s security reviewer sends one question before they sign: can another customer on your shared server learn what we send you? Pricing that question means looking at the prefix cache, and the answer is not a flat no.

Here is the setup behind it. You run vLLM on one GPU and serve Llama 3.1 8B to two customers through the same endpoint. In security terms each customer is a tenant: someone whose data must stay separate from everyone else’s on shared hardware. Tenant A’s app starts every request with the same 2,000-token system prompt. One day tenant B sends those same 2,000 tokens, and B’s reply starts sooner than a fresh 2,000-token prompt should allow.

B has just learned something about A. Someone on this server sent exactly those tokens, recently enough that they are still cached. Nothing in the response contains A’s text, because B already had the tokens and the server only confirmed them. That confirmation is the leak, and it comes from the prefix cache, which current vLLM turns on by default.

What the prefix cache keeps #

Before a model writes its first token, it reads the whole prompt. For every token, in every layer, it computes a key and a value, which later tokens look back at. That reading step is called prefill, and the stored keys and values are the KV cache.

Two prompts that begin with the same tokens produce identical keys and values for that beginning. A prefix cache keeps them from earlier requests and reuses them instead of recomputing.

vLLM stores the cache in fixed-size blocks. It names each block by hashing that block’s tokens together with the name of the block before it, so a block’s name stands for the whole prompt up to that point. Every block a request finds is prefill it can skip. Only full blocks are cached: with 16-token blocks, a 2,010-token prompt caches 125 blocks and recomputes the last 10 tokens every time.

The saving is easy to size. Say your GPU prefills 10,000 tokens a second. A 2,000-token system prompt then costs 200 ms before the first token appears, and a hit skips most of that. Keeping the copy isn’t free: with a 16-bit KV cache, Llama 3.1 8B needs 128 KiB per token, so a cached 2,000-token prefix holds 250 MiB of GPU memory. An 8-bit cache halves both.

Why a faster reply is information #

The cache has to find A’s blocks when B sends the same tokens, so the lookup is keyed on the tokens themselves, and it answers whoever asks. For B, a short time to first token means “these exact tokens were sent here, and are still cached”. A long one means nobody sent them, or they were evicted. vLLM sets no expiry time; a block stays until its memory is needed, so on a quiet server a hit can reveal a prompt sent hours ago.

Each probe tests a guess. The cache never hands out anyone’s text; it says yes or no to text the asker brings. Plenty of secrets are guessable:

• A system prompt. A competitor who suspects you built on a leaked template can check whether it runs here, byte for byte.
• A document. Whoever holds a copy of a contract learns whether it is being processed, and roughly when.
• A template plus one detail. A fixed instruction followed by a customer’s name turns a guess about that person into a yes or no.

The answers add up. A guess that hits can be extended: B keeps the matched part and tries what comes next, so a run of probes can rebuild text B never had. Song et al. rebuilt prompts token by token this way on SGLang, and Wu et al. (NDSS 2025) recovered most of other users’ prompts with no prior knowledge.

B’s own probe writes its guess into the cache, so the same guess sent twice measures B’s copy. That slows an attacker without stopping one. Song et al. flooded the server with unrelated requests to evict the cache between readings, and ten readings per guess caught 99% of hits with 0.3% false alarms. A system prompt that rides on every request is back in the cache soon after. On vLLM each step must guess a whole 16-token block, which slows the search but still exposes a template with one short secret. vLLM added cache_salt in 0.9.0 to fix a report of exactly this timing channel.

What the usage counters show #

Hosted APIs add a second signal to the clock. They bill cached input tokens at a discount, so the usage report says how many came from cache:

A counter has no jitter, so it is cleaner than a clock. It is also coarse: hosted APIs count up to a breakpoint, GPT-5.5 and earlier round that down to 128-token steps, and on vLLM a hit covers whole blocks. The count says a section matched, not the token where a guess went wrong.

Whose entries a counter can see matters more. OpenAI says caches “are not shared across organizations”. Anthropic says caches are isolated between organizations, and per workspace on the Claude API, Claude Platform on AWS and Microsoft Foundry; on Amazon Bedrock and Google Cloud its isolation is per organization only. So when you call either API with your own key, a hit means your own organization, or on Anthropic your own workspace, sent those tokens.

A relay changes who your organization is. A gateway that forwards every customer through its own provider key puts all of them in one organization, and one cache. OpenRouter’s docs say requests on your own provider key (BYOK) go through your provider account; the rest, and by default BYOK failures, run on its shared capacity. In 2026, CacheProbe saw cross-account cache hits through OpenRouter’s default mode and none with BYOK, and KeyPooling found that none of five open-source gateways bound customers to their own upstream key by default.

What the 2024 audit found #

Those guarantees weren’t always written down. In 2024, Chenchen Gu and colleagues at Stanford tested 17 LLM API providers for prompt caching using timing alone, and published the results at ICML 2025. They detected caching at 8 of them, and at 7 of those the cache was shared globally, across users and organizations. At least five providers made changes after the October 2024 disclosure, such as turning off global sharing.

Read that result precisely. The auditors detected that a cache was shared, over many trials of prompts they had sent themselves, and they could not run a practical prompt extraction against those APIs; the extraction results above come from self-hosted frameworks. The clock proved the channel exists, with no counters needed. Isolation was a policy each provider chose, and on a server you run, the choice is yours.

Your own users are tenants too #

Per-organization isolation protects you from other companies. It does nothing to protect your users from each other. If your product sends every user’s requests through one API key, all of them share your organization’s cache.

The rule is about position, not about which field the text arrived in. The leak exists wherever one user’s input can land at the same place in the prompt as another user’s text, inside the cached span. That span is wider than the fixed part. OpenAI says it caches “the model’s full rendered context”, conversation history included, and on GPT-5.6 and later its implicit mode, the out-of-the-box setting, places a breakpoint at the end of the latest eligible message. Anthropic’s automatic caching, which you turn on with one top-level cache_control field, caches everything up to the last cacheable block and moves that point forward as a conversation grows. vLLM caches every full block.

So a shared system prompt followed by the user’s message is not safe by construction. Your app puts the same system prompt in front of everyone, so user B can send that system prompt plus a document B suspects A uploaded, and read the counter. Typed messages and uploaded files both qualify.

One thing doesn’t: a slot the attacker cannot write into. If your app pulls a record from your database and pastes it into the prompt, B’s request gets B’s record, and B can’t make that slot hold A’s. That holds only while B can’t edit its record or leave the slot empty; either one puts B’s guess back at A’s position.

The lever depends on the API. On OpenAI’s GPT-5.6 and later, set prompt_cache_options.mode to explicit and mark breakpoints only after shared content, so per-user text is never written. A per-user prompt_cache_key adds separate cache accounting, which OpenAI says helps prevent “cache-hit probing across users”. On earlier models the docs present the key as a routing hint, not a boundary. On Anthropic, keep cache_control breakpoints on shared content and leave automatic caching off where prompts carry per-user content. With a few large customers, give each one its own workspace; that separates caches on the Claude API, Claude Platform on AWS and Microsoft Foundry, not on Bedrock or Google Cloud.

Salting the cache on your own vLLM server #

Self-hosted vLLM has no organizations. Every request shares one cache unless you split it, and the split is a request field called cache_salt:

{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "messages": [{"role": "system", "content": "You are the support assistant ..."}],
  "cache_salt": "<a 43-character secret for this tenant>"
}

vLLM folds the salt into the hash of the first block. Each block’s name includes the name of the block before it, so a different salt changes every name from the first onwards. By design, requests with different salts can’t hit each other’s entries, even for identical tokens. Two 2026 bugs broke that: tool-call follow-ups on the Responses endpoint for gpt-oss models dropped the salt until 0.30.0, and until 0.31.0 a salt equal to a LoRA adapter’s name shared that adapter’s first block.

Here is a simplified version of that keying you can run. It isn’t vLLM’s code, and it chains the hashes the same way:

import hashlib

def block_keys(token_ids, salt="", block_size=16):
    """Simplified vLLM-style keys: one hash per full block, chained, salt in block 0."""
    keys, parent = [], salt.encode()
    for start in range(0, len(token_ids) - block_size + 1, block_size):
        block = str(token_ids[start:start + block_size]).encode()
        parent = hashlib.sha256(parent + block).digest()
        keys.append(parent.hex()[:8])
    return keys

prompt = list(range(2_010))                # stand-in for a 2,010-token system prompt
a = block_keys(prompt, salt="tenant-a-secret")
b = block_keys(prompt, salt="tenant-b-secret")
print(len(a), "full blocks cached,", len(prompt) - 16 * len(a), "tokens left over")
print("tenant A:", a[:3])
print("tenant B:", b[:3])
print("blocks A and B can share:", len(set(a) & set(b)))
print("same tenant, same prompt:", block_keys(prompt, "tenant-a-secret") == a)

125 full blocks cached, 10 tokens left over
tenant A: ['c1666a74', '908c02e7', '63a05071']
tenant B: ['7df7733a', '514c48cf', 'a7505713']
blocks A and B can share: 0
same tenant, same prompt: True

Set the salt in your gateway from the authenticated tenant, and drop any cache_salt a client sends. A client that can set its own salt can also leave it out and land back in the unsalted, shared pool. The gateway is the real control.

vLLM’s docs still ask for a salt that is random, kept from third parties, and long enough to be unguessable. Their example is 256 bits, written as 43 base64 characters. Secrecy covers what the gateway doesn’t: someone who reaches the vLLM port directly, or a salt that slips through a path nobody sanitised. You don’t need a table of secrets to get one per tenant, because you can derive it:

import base64, hashlib, hmac

def salt_for(tenant_id, server_key):   # server_key: 32 random bytes, kept secret
    mac = hmac.new(server_key, tenant_id.encode(), hashlib.sha256).digest()
    return base64.urlsafe_b64encode(mac).decode().rstrip("=")   # 43 characters

That is stable across restarts, unguessable without the server key, and stored nowhere. Keep the URL-safe alphabet: standard base64 can produce “/”, and since 0.30.0 vLLM rejects any salt containing it.

The salt costs something only when tenants really do send identical tokens. Take 20 tenants sharing your product’s standard 2,000-token system prompt. Unsalted, the server keeps one copy, 250 MiB. Salted, it keeps up to 20 copies, about 4.9 GiB, and each tenant pays the 200 ms prefill once. If every tenant has its own system prompt anyway, salting costs nothing.

A salt defines a trust group, and you choose how wide each group is:

If you add a second cache tier, such as host memory, local disk or a store shared between replicas, check that it keys on the salted block hash rather than the raw tokens. Otherwise the tenant falls out of the key one level down.

Three checks on a shared server #

1. On vLLM, run 0.31.0 or later and check enable_prefix_caching, on by default. Then set a per-tenant cache_salt in the gateway and strip any salt a client sends.
2. List every place a user’s own text sits inside the cached span: pasted messages, uploaded documents, retrieved snippets. On OpenAI’s GPT-5.6 and later, use explicit breakpoints on shared content and a per-user prompt_cache_key. On Anthropic, keep breakpoints on shared content, and use a workspace per customer where your platform isolates workspaces and you have few enough customers. Behind a relay, use your own provider key and turn off its fallback to shared capacity.
3. If you show usage numbers to your customers, check that a cached-token count can only reflect that customer’s own requests.