cache-affinity auction

why cached inference is an ad network, not an orderbook

The Surplus Intelligence marketplace uses an orderbook: providers list prices per million tokens, buyers pay the cheapest listing. That's the right model for unused capacity — GPU hours are a commodity. But it's the wrong model for cached inference, and here's why.

the ad-network mapping

Cached inference behaves exactly like programmatic advertising. The mapping is precise:

why prompts are cookies, not secrets

The critical insight: the prompt prefix is a disclosed signal, not a secret. The gateway hashes the prefix (system prompt + first chunk of user message) into a 16-character fingerprint. This is exactly like a cookie ID in ad tech:

• Deterministic: same prompt → same hash (enables affinity tracking)
• Private: the hash cannot be reversed to recover the prompt (no plaintext leakage)
• Disclosed: the gateway publishes the hash in the bid request so providers can check their cache
• Stable: only the first 512 chars of user content are hashed, so tail variations don't break affinity

Providers who have prefix X warm in their KV cache should bid lower to win the fill — and they want to, because their marginal cost is near-zero. The orderbook prevents this price discovery from happening.

the verification layer (post-bid honesty)

Providers would lie about cache state ("yes I have 80% cached" when they have 0%). But we can verify: a cache hit is 5-10x faster than fresh compute for the same token count. surp already measures wall-clock latency per model via the health board. So we infer true cache state from the latency/token ratio, and penalize providers whose bids implied cache but whose latency proved fresh compute.

This is the ad-network "post-bid verification" layer — the same mechanism DSPs use to detect cookie fraud. A provider that discounts its bid (claiming cache) but serves with fresh-compute latency loses its affinity score, and future bids won't discount. The system is self-correcting.

live cache-affinity stats

methodology: Prefix-hash affinity: SHA-256 of normalized prefix (cookie-like). Cache state inferred from latency/token ratio (post-bid verification). Providers with high affinity discount below list price (Vickrey-style).

how the auction works (proposed)

1. Request arrives: gateway hashes the prompt prefix → 16-char fingerprint.
2. Bid request broadcast: gateway publishes "prefix X, Y output tokens, SLA Z" to eligible providers.
3. Provider self-assessment: each provider checks its KV cache for prefix X. If 80% is cached, marginal cost is ~20% of list → bid 40% below list.
4. Vickrey auction: lowest bid wins, pays second-lowest bid. This incentivizes truthful bidding (game-theoretic optimum).
5. Fill + measurement: winner serves the request; gateway measures wall-clock latency.
6. Post-bid verification: if latency implies cache hit but the bid was low, affinity score increases (honest). If latency implies fresh compute but bid was low, affinity score decreases (dishonest).
7. Feedback loop: future bids for that prefix use the updated affinity score to propose discounts.

what we built (gateway-side)

The full auction requires provider-side cooperation (they must check KV cache and bid). SI's static orderbook doesn't support that yet. But surp has built the gateway-side infrastructure that makes the auction possible:

• Prefix hashing: cache_affinity.prefix_hash() — SHA-256 of normalized prefix, 16 chars, no plaintext leakage.
• Sample recording: every non-streaming paid request records (prefix, model, tokens, latency) to the affinity DB.
• Cache-state inference: latency/token ratio determines if a fill was cached (< 200ms/1K tokens = cache hit).
• Affinity scoring: per (prefix, model) pair: hit rate over a 24h rolling window.
• Proposed bids: cache_affinity.proposed_bid() discounts list price up to 50% for models with high affinity — the DSP bid based on match rate.
• Dishonesty detection: if latency doesn't support a cache claim, no discount is applied. The bid stays at list.
• Public stats: live at /api/stats under cache_affinity.

what we still need (provider-side)

The missing piece is provider cooperation. To run a real auction, Surplus Intelligence providers would need to:

1. Accept bid requests: receive prefix hashes and respond with bids, not just static listings.
2. Check KV cache: inspect their own prefix cache for the hashed prefix and report match rate.
3. Submit truthful bids: bid below list price proportional to cached fraction.
4. Accept Vickrey settlement: be paid the second-lowest bid, not their own.

This is a protocol change, not a code change. surp's gateway is ready to participate the moment SI exposes a bid-request endpoint. Until then, the affinity data we collect is the public good that proves the mechanism works.

why this matters

Orderbook pricing for cached inference overcharges buyers and under-rewards cache creators. A buyer asking the same prefix twice pays list price both times, even though the second fill cost the provider ~5x less. An auction would let the cached provider bid lower, win the fill, and pass savings to the buyer — while still earning more than the commodity margin. The exact-response cache and SRP reward ledger are surp's current approximation of this; the affinity auction is the full vision.

related: cache-aware routing · provider health board · reward proposal · free models · API docs