▸ verified llm performance usage
new: genuinely free AI models are live — try surp/free + see live budgets · token-gating prototype · vote on SRP

verified llm performance

real output TPS, TTFT, and throughput-per-dollar — measured, not claimed.

Every number on this page comes from a real streaming request sent through surp to the Surplus Intelligence marketplace. No vendor spec sheets, no extrapolated benchmarks — only observed generation throughput. This is the data the Surplus dashboard doesn't show.

why this matters: A model listing at $0.01/M might sound cheap, but if it generates at 20 tokens/second, a 1000-token response takes 50 seconds. A model at $0.06/M generating at 100 TPS delivers the same response in 10 seconds. Throughput-per-dollar is the metric that actually determines your cost-per-task. We measure it directly.

throughput leaderboard

Models ranked by throughput-value score (p50 output TPS × tokens-per-dollar). Green ≥ 80 TPS, amber 40-79, red < 40.

#modelp50 TPSp95 TPSrange p50 TTFTprice /1Mthroughput/$runs
No benchmarks recorded yet. Run python3 benchmark_runner.py to generate verified data.

what "output TPS" means

Output TPS = generated completion tokens ÷ generation seconds (time from first token to last token). This is the metric LLM buyers care about: how fast the model produces text.

This is different from request RPS (requests per second), which measures how many separate requests the gateway handles. The health board tracks RPS; this page tracks output TPS.

methodology

  • Real requests: each run sends a streaming POST /v1/chat/completions through surp with max_tokens=400.
  • TTFT: time from request submission to first output token (time-to-first-token).
  • Generation time: wall time − TTFT (time spent generating, excluding queue/TTFT).
  • Output TPS: completion_tokens / generation_seconds.
  • Throughput-value score: p50_output_tps × (1,000,000 / price_per_1m) — higher is better.
  • p50/p95: percentile across all successful runs in the window.
  • Token counting: prefers usage.completion_tokens from the API; falls back to len(text)/4 estimate if usage is absent.
  • No cherry-picking: all runs (including slow ones) are recorded. Failed runs count against the failure rate.

try it yourself

The benchmark runner is open source. Clone the repo and run:

python3 benchmark_runner.py --model deepseek-v4-flash-0731 --runs 10 --api-key sk-sur-...

You'll get the same kind of numbers we publish here. Or just make a streaming request and time it:

curl -N -X POST https://surp.ivc.lol/v1/chat/completions \
  -H "Authorization: Bearer sk-sur-..." \
  -H "Content-Type: application/json" \
  -d '{"model":"surp/direct/deepseek-v4-flash-0731",
       "messages":[{"role":"user","content":"Write 200 words about HTTP/2"}],
       "max_tokens":400,
       "stream":true}'

related: provider health board (RPS, latency) · free models · cache-affinity auction · benchmark API