cache-aware LLM routing

compute once. reuse safely. pay less.

LLM APIs waste money when they recompute the same context over and over. Agent tool definitions, system prompts, codebases, policies, and long conversation prefixes are often identical between requests. surp.ivc.lol now uses a two-layer cache engine to avoid that waste while preserving live marketplace pricing.

layer 1 — cache-aware sticky routing

Provider-side prefix caching stores the model's computed state for the beginning of a prompt. Reusing a cached prefix cuts input cost by -75% to -90% and can drop time-to-first-token dramatically compared to processing fresh input.

But normal cheapest-price routing can destroy those savings: request one goes to model A, request two goes to model B, so model A's warm cache is useless. Our router now keeps a recently selected model when it remains within 30% of the current cheapest eligible model. If the price gap grows beyond that, it switches back to the live cheapest option.

request 1 → cheapest model A → provider writes prefix cache
request 2 → A is still within tolerance → reuse A → cache read
request 3 → model B becomes 40% cheaper → switch to B

This balances two things that normally conflict: marketplace arbitrage and cache locality.

layer 2 — exact response caching

Some requests are completely repeatable: temperature zero, no tools, one response, and no streaming. For those requests, surp.ivc.lol stores the completed JSON response under a SHA-256 fingerprint of the full request.

When the exact request arrives again within the cache window:

• The model is not called again.
• The response returns immediately.
• The request costs $0.001 by default instead of the normal 1¢ floor.
• The response carries X-Surp-Cache: HIT and cache metadata.

privacy and safety boundaries

We deliberately do not cache every request.

Raw prompts are never persisted in the exact cache. The database contains only a SHA-256 request fingerprint and the completed response. Responses expire automatically and the cache is size-bounded.

how to get more cache hits

1. Keep tool definitions and system instructions stable.
2. Put static content first and changing user input last.
3. Do not inject timestamps or random IDs into the reusable prefix.
4. Use temperature: 0 for deterministic tasks that may repeat.
5. Use non-streaming mode when you want exact-response caching.

observe it yourself

Every response reports its cache path:

X-Surp-Cache: HIT | MISS | BYPASS
X-Surp-Cache-Type: exact-response
X-Surp-Routing: sticky-within-tolerance | live-cheapest

The public status page shows exact-cache hit rate, tokens not recomputed, live cached answers, and sticky-route reuse. The same metrics are available as JSON at GET /api/stats.

why this is different

Most gateways optimize either price or caching. Pure cheapest-price routing jumps between models and loses warm prefixes. Pure session pinning ignores cheaper market offers. Our approach treats cached computation as an economic asset: keep it while its savings beat the price gap, then move when the market moves enough to justify losing it.

the cache flywheel — rewards for creating shared value

Cache hits create measurable economic value: the network avoids an upstream model call and keeps the difference. Instead of capturing all of that value, surp runs an experimental off-chain reward ledger called SRP:

• Cache writer: the payer who funds a deterministic MISS earns 1 SRP per token cached.
• Cache author: when another agent reuses that entry, the original writer earns 2 SRP per token saved.
• Cache reader: the agent that cooperatively reuses the cache earns 0.5 SRP per token saved.
• Revenue backing: 50% of estimated gateway markup — and 50% of cache-hit revenue — is earmarked to a rebate pool.

more usage → more shared cache → lower network costs
     ↑                              ↓
SRP claim on revenue ← more rebates ← more margin

SRP's estimated redemption value is rebate pool ÷ outstanding SRP. As protocol revenue enters the pool, existing SRP gains claim value. Heavy users who create useful cache entries can push their effective marginal cost toward zero over time.

anti-farming guardrails

• Rewards are deduplicated per cache entry, role, and payer for one hour.
• Tool calls, streaming, creative sampling, and nondeterministic requests never enter the exact cache.
• Raw prompts are never stored — only SHA-256 fingerprints and completed responses.
• Cache readers still pay the discounted $0.001 micropayment, preventing free probing.
• Most author rewards only appear when somebody actually reuses the entry — write spam has little value.

Live pool backing, SRP outstanding, holders, and implied value per SRP are visible on the status page. An agent can query its own balance with GET /api/rewards?payer=0x....

BYO guardrails — user-controlled routing

You don't have to trust the default routing. Every request can carry its own constraints without changing the combo:

Example: cheapest coder, but never above $0.50/1M and never from the cache:

curl -X POST https://surp.ivc.lol/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{"model":"surp/best-coding",
       "max_price_per_1m": 0.50,
       "surp_bypass_cache": true,
       "messages":[{"role":"user","content":"write a binary search"}],
       "max_tokens":100}'

Example: cheapest chat, but only from a specific provider:

curl -X POST https://surp.ivc.lol/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{"model":"surp/best-chat",
       "provider": "deepseek",
       "messages":[{"role":"user","content":"hi"}],
       "max_tokens":50}'

These guardrails compose. A request can pin a provider, cap the price, and bypass the cache at the same time. The surp/strict/ prefix layers on top of any combo, including custom ones (surp/strict/my/your-slug).

related: x402 LLM API · cheapest LLM API · x402 gateway · API docs · live cache metrics