Understanding the SemiAnalysis Inference Curves

SemiAnalysis's InferenceX benchmark plots inference providers on a throughput-versus-interactivity plane and asks: which providers sit on the Pareto frontier? This post takes a different angle — instead of measuring providers, we derive the shape of the frontier from hardware physics, then build model curves that closely approximate what InferenceX observes.

Model setup

Throughout this post we model DeepSeek R1 [DeepSeek-AI, 2025] as a concrete instantiation of a frontier mixture-of-experts (MoE) decoder. DeepSeek R1 is one of the models SemiAnalysis benchmarks directly in InferenceX v2, making it a natural reference point.

Quantity	Symbol	Value
Total parameters	$P_\text{total}$	671 × 10⁹
Active parameters / token	$P_\text{active}$	37 × 10⁹
Layers	$L$	61
Hidden dimension	$d_\text{model}$	7,168
KV cache (MLA latent)	$d_c + d_r$	512 + 64 per layer
Weights (FP8)	$P_\text{total} \cdot b_w$	671 GB

We compare three realistic multi-GPU serving configurations that can host the full 671 GB model in FP8. Hardware specs are from official vendor datasheets; the TP scaling analysis follows Pope et al., 2023.

Config	HBM BW	Peak compute (FP8)	HBM total	KV headroom
8 × H200 SXM (TP8)	38.4 TB/s	31,664 TFLOPS	1,128 GB	457 GB
4 × MI300X (TP4)	21.2 TB/s	10,456 TFLOPS	768 GB	97 GB
8 × B200 (TP8)	64.0 TB/s	72,000 TFLOPS	1,536 GB	865 GB

Tensor-parallel bandwidth is treated as additive (each GPU reads its own weight shard in parallel). MoE routing uses only active parameters ($P_\text{active} = 37\text{B}$) for compute and bandwidth bounds; the full $P_\text{total} = 671\text{B}$ determines HBM capacity. All-reduce communication overhead is explicitly modeled in §7: NVLink TP=8 adds ~1.2 ms per decode step at B = 1 (latency-bound, dominated by 61 × 2 = 122 kernel launches at 10 µs each). See Patterson & Hennessy for a full treatment of parallel memory hierarchies.

The roofline model for decode

The roofline model [Williams et al., 2009] characterises compute performance as the minimum of two bounds: peak arithmetic throughput and the product of memory bandwidth with arithmetic intensity.

LLM inference has two structurally different phases. Prefill processes the prompt as a batched matrix multiplication — it is compute-bound and runs close to peak FLOP/s. Decode generates one token per step; we focus on decode because it dominates end-to-end serving cost and determines both throughput and TTFT.

Arithmetic intensity during decode

At decode batch size $B$ (concurrent requests), one step requires:

$$\text{FLOPs} = 2 \cdot B \cdot P_\text{active}$$

The factor of two counts one multiply and one add per active parameter per token (FMAs). For MoE models only the active expert weights are loaded per step; the remaining experts sit idle in HBM. The active-parameter weight bytes streamed from HBM per step are:

$$\text{Bytes}_\text{weights} = P_\text{active} \cdot b_w$$

where $b_w$ is bytes per weight (2 for FP16, 1 for FP8/INT8). The arithmetic intensity is therefore:

$$I(B) = \frac{2B \cdot P_\text{active}}{P_\text{active} \cdot b_w} = \frac{2B}{b_w}$$

This is the key insight: arithmetic intensity scales linearly with batch size. At $B = 1$ with FP8 ($b_w = 1$), $I = 2$ FLOP/byte — still far below the ridge point of any modern GPU. The hardware ridge point is:

$$I^* = \frac{\text{peak compute}}{\text{HBM bandwidth}}$$

Config	Ridge $I^*$	Ridge batch $B^*$	$I(B{=}1)$	Utilization at $B{=}1$
8 × H200 (FP8)	824 FLOP/B	412	2.0	0.24%
4 × MI300X (FP8)	493 FLOP/B	247	2.0	0.41%
8 × B200 (FP8)	1,125 FLOP/B	563	2.0	0.18%

Below the ridge batch $B^*$ the chip is memory-bandwidth bound: throughput scales linearly with $B$. Above $B^*$ it is compute bound: throughput saturates. Ignoring KV-cache overhead for the moment, decode throughput is:

$$T(B) = \frac{B}{t_\text{step}(B)}, \quad t_\text{step}(B) = \max\!\left(\frac{P_\text{active} \cdot b_w}{\text{BW}},\; \frac{2 P_\text{active} B}{\text{compute}}\right)$$

which gives the two-regime roofline shape:

$$T(B) = \begin{cases} \dfrac{B \cdot \text{BW}}{P_\text{active} \cdot b_w} & B \le B^* \quad\text{(memory bound)} \\[10pt] \dfrac{\text{compute}}{2 P_\text{active}} & B > B^* \quad\text{(compute bound)} \end{cases}$$

The KV-cache tax

So far we've only counted weight reads. Each decode step also reads the full key-value cache for every in-flight request. For a context of $S$ tokens per request, the KV bytes read per step are:

$$\text{Bytes}_\text{KV}(B, S) = B \cdot S \cdot \underbrace{L \cdot (d_c + d_r) \cdot b_{kv}}_{\kappa}$$

DeepSeek R1 uses Multi-head Latent Attention (MLA, DeepSeek-AI, 2024), which compresses the KV cache to a low-rank latent vector $d_c = 512$ plus a decoupled RoPE key $d_r = 64$ per layer — far smaller than the standard GQA formula ($L \cdot 2 \cdot H_{kv} \cdot d_h \cdot b_{kv}$). With $L = 61$ layers and $b_{kv}$ bytes per element:

$$\kappa = 61 \times (512 + 64) \times b_{kv} = 61 \times 576 \times b_{kv}$$

For FP8 KV ($b_{kv} = 1$): $\kappa = 35{,}136$ bytes/token ≈ 34 KB/token — roughly 9× smaller than GQA Llama 70B FP16 (320 KB/token). The total bytes per decode step is now:

$$\text{Bytes}_\text{total}(B, S) = P_\text{active} \cdot b_w + B \cdot S \cdot \kappa$$

so the memory-bound step time and throughput become:

$$t_\text{step}(B, S) = \max\!\left(\frac{P_\text{active} b_w + B S \kappa}{\text{BW}},\; \frac{2 P_\text{active} B}{\text{compute}}\right)$$ $$T(B, S) = \frac{B}{t_\text{step}(B, S)}$$

Note what happens as $B \to \infty$ in the memory-bound regime: the $BS\kappa$ term dominates the numerator, so throughput saturates at:

$$T_\infty(S) = \lim_{B \to \infty} T(B, S) = \frac{\text{BW}}{S \cdot \kappa}$$

This is the KV-cache throughput ceiling — independent of weight precision or batch size. For 8×H200 FP8 KV at $S = 8{,}192$:

$$T_\infty(8192) = \frac{38.4 \times 10^{12}}{8192 \times 35{,}136} \approx 133{,}000\ \text{tok/s}$$

MLA's small $\kappa$ pushes this ceiling dramatically higher than GQA models — the KV-cache bottleneck is much less binding for DeepSeek R1 at comparable contexts.

Modeling TTFT and inter-token latency

TTFT has two additive components: the time to process the prompt (prefill latency) and the time spent waiting in queue before the prefill can begin.

Prefill latency

Prefill with a prompt of $S_\text{in}$ tokens is compute-bound: the attention computation involves $O(S_\text{in}^2)$ FLOPs but the weight-multiply term dominates for typical prompt lengths:

$$t_\text{prefill}(S_\text{in}) = \frac{2 P_\text{active} \cdot S_\text{in} + 4 L H_\text{full} d_h S_\text{in}^2}{\text{compute}}$$

where $H_\text{full} = 128$ is the full query-head count for DeepSeek R1. The quadratic attention term grows with $S_\text{in}^2$; it contributes under 5% below ~3K tokens on 8×H200 but dominates at 8K+ (the "8k1k" InferenceX benchmark condition).

For $S_\text{in} = 1{,}024$ tokens on 8×H200:

$$t_\text{prefill} = \frac{2 \times 37 \times 10^9 \times 1024}{31.664 \times 10^{15}} \approx 2.4\ \text{ms}$$

Queuing delay

When the system is serving $B$ concurrent decode requests, new arrivals must wait for a scheduling slot. Under continuous (iteration-level) batching [Yu et al., 2022], this is well-described by an M/G/1 queue [Kleinrock, 1975]: Poisson arrivals and variable service time governed by output length. The mean waiting time in queue (Pollaczek–Khinchine formula) is:

$$W_q(\rho) = \frac{(1 + \text{CV}^2)}{2} \cdot \frac{\rho \cdot t_\text{step}}{1 - \rho}$$

where $\rho = B / B_\text{max}$ is the system utilisation, and $B_\text{max}$ is the maximum batch that fits in the available KV-cache headroom:

$$B_\text{max}(S) = \left\lfloor \frac{\text{HBM}_\text{free}}{S \cdot \kappa} \right\rfloor$$

Putting it together:

$$\boxed{\text{TTFT}(B, S_\text{in}, S) = t_\text{prefill}(S_\text{in}) + \frac{1 + \text{CV}^2}{2} \cdot \frac{t_\text{step}(B, S)}{1 - B/B_\text{max}(S)}}$$

where $\text{CV}$ is the coefficient of variation of output lengths ($\text{CV} = 0$ gives the M/D/1 formula; $\text{CV} = 1$ doubles the queuing penalty, recovering M/M/1). The $1/(1-\rho)$ factor is the queuing blowup: at $\rho = 0$ the residual wait is $(1 + \text{CV}^2)/2 \cdot t_\text{step}$ (expected remaining time in the current step); as $\rho \to 1$, the queue grows without bound. In practice, providers target $\rho \lesssim 0.85$ to keep TTFT well-behaved. The Output length CV slider in the interactive controls applies this correction live.

Inter-token latency

Once a request is being actively served, the time between successive output tokens is the inter-token latency (ITL), sometimes called time-between-tokens (TBT). Unlike TTFT, ITL has no queuing component — every decode step produces exactly one token per slot:

$$\boxed{\text{ITL}(B, S) = t_\text{step}(B, S) = \max\!\left(\frac{P_\text{active} b_w + B S \kappa}{\text{BW}},\; \frac{2 P_\text{active} B}{\text{compute}}\right) + t_\text{comm}(B)}$$

where $t_\text{comm}(B)$ is collective communication overhead (all-reduce for TP, point-to-point for PP) as derived in §7; it is zero for single-GPU deployments. A key algebraic identity links ITL and throughput at fixed batch size $B$:

$$\text{ITL}(B) \times T(B) = t_\text{step}(B) \times \frac{B}{t_\text{step}(B)} = B$$

This is a rectangular hyperbola: at fixed $B$, halving ITL requires halving $T$. The only way to achieve both lower ITL and higher throughput simultaneously is to improve hardware — higher bandwidth shifts the entire hyperbola downward. The minimum ITL is set by the single-request weight-read time:

$$\text{ITL}_\text{min} = \frac{P_\text{active} \cdot b_w}{\text{BW}}$$

For 8×H200 FP8: $37\,\text{GB} / 38.4\,\text{TB/s} \approx 0.96\,\text{ms}$. Active-parameter MoE gives a very low weight-read floor; the NVLink TP=8 collective overhead (~1.2 ms) is comparable in magnitude at $B = 1$, so reducing TP degree is not an option given the 671 GB model size. Every additional concurrent request adds $S \cdot \kappa / \text{BW}$ to the step time; for long contexts this term dominates:

$$\text{ITL}(B, S) \xrightarrow{BS\kappa \gg Pb_w} \frac{B S \kappa}{\text{BW}}$$

SemiAnalysis tracks TTFT and ITL separately because they respond differently to system load. TTFT explodes near saturation (the queuing blowup) while ITL grows only linearly with $B$ in the memory-bound regime. A provider can have excellent TTFT at low load and terrible ITL at high load if the system is scheduling large decode batches.

The throughput–TTFT curve

We now have a parametric curve in the $(T, \text{TTFT})$ plane with $B$ as the free parameter. As $B$ increases from 1 to $B_\text{max}$:

• $T(B) = B / t_\text{step}(B, S)$ increases — more requests share each weight read.
• $\text{TTFT}(B)$ increases — higher utilisation means longer queuing delay.

The resulting curve traces the efficiency frontier: any real provider must sit on or above this line. Sitting above it means achieving a given throughput at worse TTFT than the hardware physics requires — indicating suboptimal scheduling, kernel inefficiency, or fragmented KV memory.

Eliminating $B$ gives the implicit frontier equation. In the memory-bound regime (ignoring KV overhead for clarity):

$$T = \frac{B \cdot \text{BW}}{Pb_w}, \quad \text{TTFT} = t_\text{prefill} + \frac{Pb_w}{2 \cdot \text{BW} \cdot (1 - T/T_\text{max})}$$

which is a hyperbola in $(T, \text{TTFT})$ space with $T_\text{max}$ as the vertical asymptote. This explains the characteristic shape of every InferenceX chart: the frontier is not a line, it curves sharply upward as throughput approaches the system maximum.

Interactive operating point explorer

Selecting a batch size moves the amber dot along the efficiency frontier, updating the four readouts below in real time. Click Animate to sweep through all sustainable batch sizes — watch how throughput and TTFT trade off as utilization climbs from near-zero to the queue-saturation knee.

How hardware shifts the frontier

Three hardware parameters move the curve:

Memory bandwidth

Higher bandwidth shifts the curve rightward: more throughput at every batch size. It also reduces the base queuing delay $t_\text{step}/2$, shifting the curve downward. The 4×MI300X's 21.2 TB/s offers 55% of 8×H200's 38.4 TB/s, with a much smaller KV headroom (97 GB vs 457 GB after fitting the 671 GB model). However, the MI300X's 768 GB of HBM still hosts the full model in FP8, allowing a larger $B_\text{max}$ and therefore a longer rightward extension of the frontier.

HBM capacity

$B_\text{max}$ is proportional to the KV headroom (HBM minus weight size). Doubling headroom doubles the maximum sustainable batch, extending the frontier to the right before the queuing blowup. The 8×B200 curve extends farthest — not only because of higher bandwidth, but because its 865 GB of KV headroom supports far more concurrent requests at $S = 1024$ than the 97 GB available on 4×MI300X.

Quantization

Going from FP8 to FP16 doubles $b_w$, halving the arithmetic intensity $I(B) = 2B/b_w$ and doubling the weight bytes read per step — so FP8 serves as the natural baseline for DeepSeek R1. FP8 KV ($b_{kv} = 1$) keeps $\kappa = 35{,}136$ bytes/token; switching to FP16 KV doubles it to 70,272 bytes/token, halving $B_\text{max}$ and compressing the usable frontier. The ridge batch $B^* = b_w \cdot \text{compute} / (2 \cdot \text{BW})$ shifts with $b_w$, so the two-regime roofline shape is preserved but scaled.

Distributed parallelism and collective costs

Real serving deployments run across multiple GPUs or multiple nodes. Two distinct parallelism strategies are in common use, and they differ fundamentally in how communication cost scales with batch size.

Scale-up: tensor parallelism over NVLink

Scale-up means increasing resources within a single NVLink domain — adding more GPUs to the same job by splitting the model column-wise across $k$ GPUs (tensor parallelism, TP) [Pope et al., 2023]. Each GPU holds $P/k$ parameters and streams them from its own HBM; the aggregate bandwidth and capacity both grow linearly with $k$. After each attention and FFN projection, a ring all-reduce synchronises the partial sums.

For a ring all-reduce of message size $M$ across $k$ GPUs on an interconnect with unidirectional bandwidth $\text{BW}_\text{IC}$ and per-collective kernel launch latency $\alpha$:

$$t_\text{AR}(B, k) = L \cdot 2 \cdot \left(\alpha + \frac{k-1}{k} \cdot \frac{B \cdot d_\text{model} \cdot b_w}{\text{BW}_\text{IC}}\right)$$

For DeepSeek R1 ($d_\text{model} = 7168$) on NVLink ($\text{BW}_\text{IC} = 450\,\text{GB/s}$, $\alpha = 10\,\mu\text{s}$), with $k=8$ and $B=1$:

$$t_\text{AR}(1, 8) = 61 \times 2 \times \left(10\,\mu\text{s} + \frac{7}{8} \cdot \frac{7\,\text{KB}}{450\,\text{GB/s}}\right) \approx 1.2\,\text{ms}$$

The base step time at $B=1$ (active-weight read) is $37\,\text{GB} / 38.4\,\text{TB/s} \approx 0.96\,\text{ms}$, so the collective overhead is comparable to the compute in the single-token regime — a significant tax, but unavoidable since 671 GB of weights require distributing across the full 8-GPU pod. Because $\alpha$ is the dominant term (the 7 KB message transfers in ≪1 µs on NVLink), the overhead stays near 1.2 ms across all batch sizes.

Scale-out: pipeline parallelism over InfiniBand

Scale-out means distributing across multiple nodes connected by a datacenter fabric. The typical mechanism for LLM serving is pipeline parallelism (PP), where the $L$ layers are split into $m$ stages — each node handles $L/m$ layers. Between consecutive decode steps, the activations from one stage must be sent to the next via a point-to-point message:

$$t_\text{PP}(B, m) = (m - 1) \cdot \left(\alpha_\text{P2P} + \frac{B \cdot d_\text{model} \cdot b_w}{\text{BW}_\text{P2P}}\right)$$

On InfiniBand HDR200 ($\text{BW}_\text{P2P} = 25\,\text{GB/s}$), this term grows linearly with $B$ and becomes significant at large batches. At $B = 256$ with $m = 4$ (FP8 activations, $d_\text{model} = 7168$):

$$t_\text{PP}(256, 4) = 3 \times \left(5\,\mu\text{s} + \frac{256 \times 7\,\text{KB}}{25\,\text{GB/s}}\right) \approx 0.22\,\text{ms}$$

This adds 0.22 ms to a base step of ~1.2 ms — an 18% penalty. For larger models with FP16 activations or higher batch sizes the penalty grows proportionally. This explains why InfiniBand-based scale-out is reserved for models too large for a single NVLink domain, and why serving architects strongly prefer scale-up for latency-critical workloads.

The total step time with both collective sources is:

$$\boxed{t_\text{step}^{\text{dist}}(B, S) = t_\text{step}(B, S) + t_\text{AR}(B, k) + t_\text{PP}(B, m)}$$

The TP degree, PP degree, and interconnect type selectors in the shared controls apply these corrections live to every figure. Figure 8 isolates the collective overhead for four representative configurations.

Interconnect	BW (unidirectional)	Latency floor $\alpha$	Use case
NVLink 4.0 (H100)	450 GB/s	10 µs	Scale-up (TP within node)
InfiniBand HDR200	25 GB/s	5 µs	Scale-out (PP across nodes)
RoCE v2 (100 GbE)	~10 GB/s	10 µs	Scale-out (budget deployments)

Bandwidth figures are from vendor datasheets: NVIDIA NVLink 4.0 specification (900 GB/s bidirectional = 450 GB/s per direction) and InfiniBand Trade Association HDR200 specification. Kernel launch latency $\alpha$ values are representative of measured NCCL collective overheads; see Pope et al., 2023 §3.2 for measured latency floors.

Reading the SemiAnalysis charts

With the full model in hand, we can read every InferenceX chart as a diagnostic. The benchmark runs standardised traffic traces against real providers and records median TTFT, ITL, and aggregate output throughput. Each provider's position tells a story.

Position on the curve → hardware and batch configuration. A provider at the lower-left is running small effective batches $B$ — each request gets fast service because the decode step time $t_\text{step}$ is low, but throughput is low because few requests share each weight read. This is the interactive regime: low TTFT, low ITL, low cost-efficiency. A provider at the upper-right is running large batches: many requests share each weight read (higher throughput), but the queuing delay and per-step time are both larger (higher TTFT and ITL). This is the batch regime: high throughput, lower per-token cost, coarser latency. The curve between these extremes is the hardware's efficiency frontier — moving along it is a scheduling choice, not a capability gap.

Distance above the frontier → efficiency gap. Our model curves represent the theoretical optimum for given hardware and the M/G/1 queuing model. Real providers sit above these curves for reasons quantified in the Limitations section: prefill-decode interference, KV-cache fragmentation [Kwon et al., 2023], kernel overhead, and networking latency. The vertical gap between a provider's measured TTFT and the model curve at the same throughput is the operational headroom recoverable through better scheduling and kernels.

Kink in the frontier → compute-bound transition. The curve bends at $B^*$ where the system transitions from memory-bound to compute-bound. For DeepSeek R1 on 8×H200 FP8, $B^* \approx 412$ (from the ridge table above) — rarely reached in practice because KV memory runs out first at typical context lengths. When you see the frontier flatten horizontally in an InferenceX chart, that provider has hit the compute-bound regime.

The asymptote → max sustainable throughput. The near-vertical portion of each curve ($\rho \to 1$) is where TTFT explodes. No provider should operate there in production. The knee of the curve — roughly where TTFT is 2–3× its minimum — is the practical operating point that balances cost efficiency and user experience.

How closely does this model match InferenceX?

Our curves are a theoretical lower bound — what a perfect scheduler on ideal hardware would achieve. Real InferenceX measurements include additional overhead the model does not account for:

• Network round-trip time. SemiAnalysis runs tests from fixed infrastructure; the API round-trip adds 5–25 ms to every TTFT measurement depending on geography. This shifts the entire measured curve upward by a constant, making low-utilization TTFT appear worse than the hardware physics would predict.
• Kernel efficiency. Real CUDA kernels operate at 85–95% of the theoretical peak. Step times are therefore 5–15% higher than our model, shifting the frontier rightward.
• Scheduling overhead. Continuous batching and chunked prefill introduce per-step overhead of 0.5–2 ms. This is most visible at small batch sizes where it is a large fraction of $t_\text{step}$.

In practice, the model predicts:

• ITL floor within 10–20% — set by hardware bandwidth, the most predictable quantity.
• Throughput ceiling $T_\infty(S)$ within 10–15% — set by bandwidth and KV-cache architecture.
• TTFT at low load ($\rho < 0.3$) within 20–30% for optimized providers, once network RTT is subtracted.
• TTFT near saturation ($\rho > 0.8$) within a factor of 2 — queuing variance and burst patterns dominate here. The M/G/1 correction (CV slider) helps, but real workload distributions vary.

A provider whose measured TTFT lies within 1.5× of the model curve at median load is operating close to hardware limits. A provider consistently 3× above the curve at moderate throughput has significant operational headroom — from scheduling, fragmentation, or under-utilised bandwidth.

Other SemiAnalysis metrics

InferenceX reports a richer set of quantities than just TTFT and throughput. Each maps to a distinct part of the model.

Metric	Definition	Primary driver
TTFT	Time from request submission to first output token	Prefill compute + M/G/1 queuing delay $(\rho, \text{CV})$
ITL / TBT	Time between successive output tokens	HBM bandwidth, batch size $B$, context length $S$
TPOT	Time per output token from user's perspective	Equals ITL under steady decode; diverges with speculative decoding
End-to-end latency	$\text{TTFT} + N_\text{out} \times \text{ITL}$	ITL-dominated for responses longer than ~50 tokens
P99 TTFT	99th-percentile TTFT across requests	Tail of the queuing distribution; blows up faster than the mean near $\rho = 1$
Output throughput	Aggregate tokens/s across all requests	$T = B / t_\text{step}$; the x-axis of every InferenceX chart
Cost per million tokens	Hardware rental cost ÷ (3600 s/hr × throughput / 10⁶)	Directly inversely proportional to throughput at fixed hardware cost
Token efficiency	Throughput / TDP (tok/s/W)	HBM BW-per-watt; DRAM access dominates power (~60–100 pJ/bit)

P99 TTFT is the most diagnostic metric for production readiness. The M/G/1 tail grows faster than the mean as $\rho \to 1$: a provider at $\rho = 0.9$ may have median TTFT of 150 ms but P99 above 2 s. The gap between P50 and P99 is a direct readout of how close to saturation the system is running. SemiAnalysis reports both; providers with tight P50–P99 gaps are operating with headroom.

End-to-end latency matters more than TTFT alone for most users. A 500-token response at ITL = 30 ms takes 15 s to complete regardless of a 50 ms TTFT. This is why optimising for lower ITL (via more bandwidth or lower batch size) often matters more than optimising TTFT, and why providers focused purely on TTFT can still deliver a poor user experience at long output lengths:

$$\text{E2E}(B, S_\text{in}, S, N_\text{out}) = \text{TTFT}(B, S_\text{in}, S) + N_\text{out} \times \text{ITL}(B, S)$$

Implications for hardware designers

The model is not just descriptive — it prescribes what silicon optimized for LLM serving should look like, and the answer differs significantly from training hardware.

Bandwidth, not compute, is the primary serving knob

The roofline table above shows compute utilization of 0.34% at $B = 1$ on H100. Adding more TFLOP/s leaves that number essentially unchanged; adding more HBM bandwidth directly reduces $t_\text{step}$ and therefore both ITL and the TTFT floor. A chip with 2× the bandwidth at the same compute peak halves ITL at low batch sizes — far more user-visible than a 2× compute increase.

HBM capacity is throughput capital

$B_\text{max} \propto (\text{HBM} - P_\text{total} b_w) / (S \kappa)$. Doubling HBM headroom doubles the sustainable batch and doubles cost-efficient throughput — it extends the frontier rightward without moving it vertically. For DeepSeek R1, the model consumes 671 GB of HBM, leaving only 97 GB of KV headroom on 4×MI300X versus 457 GB on 8×H200. The 8×B200 (865 GB headroom) is the most throughput-capable configuration at long contexts.

Disaggregated serving favors different silicon for each phase

Prefill is compute-bound; decode is bandwidth-bound. A chip that maximises both simultaneously optimises neither. Recent disaggregated serving proposals (Splitwise [Patel et al., 2023], DistServe/TetriInfer [Zhong et al., 2024]) route prefill to compute-dense nodes and decode to bandwidth-dense nodes. A decode-optimised chip might carry lower TFLOP/s but more DRAM bandwidth and capacity — closer to a high-bandwidth memory controller than to a training GPU.

The memory wall is structural

The $I(B{=}1) = 1\,\text{FLOP/byte}$ vs $I^* = 295$ gap is not a design mistake — it reflects DRAM physics. Off-chip I/O costs ~60 pJ/bit; on-chip FMAs cost ~1 pJ [Hennessy & Patterson, 2017, Appendix D]. Closing the gap requires bringing computation closer to memory (processing-in-memory, CXL pooled HBM) or shrinking $\kappa$ architecturally — Multi-head Latent Attention (MLA, used in DeepSeek V3 [DeepSeek-AI, 2024]) compresses the KV cache to a single low-rank vector per layer, cutting $\kappa$ by roughly $d_h$-fold relative to standard GQA.

Tensor parallelism has diminishing returns for serving

Splitting a model across $k$ GPUs reduces weight bytes per GPU by $k$ but introduces all-reduce overhead on every forward pass. On NVLink (450 GB/s per direction on H200 SXM), the ring all-reduce at $d_\text{model} = 7168$ and TP=8 adds ~1.2 ms per decode step at $B = 1$ — comparable to the 0.96 ms active-weight read floor. This overhead is latency-bound (61 × 2 = 122 kernel launches at 10 µs each dominate, not the transfer), so it stays near 1.2 ms across all batch sizes. For DeepSeek R1, TP=8 is necessary since 671 GB cannot fit on fewer H200 GPUs; the overhead is a fixed cost of hosting this model class.

SRAM accelerators: extreme bandwidth for small models

HBM bandwidth is limited by the number of off-chip I/O pins — roughly 5,000 differential pairs per stack on HBM3. On-chip SRAM has no such constraint: bandwidth scales with bit-cells per die. The d-Matrix Corsair chip carries 2 GB of on-chip SRAM at 150 TB/s — about 20× the bandwidth of an H100 SXM5 (3.35 TB/s per die). This defines a different region of the roofline entirely.

Define BW density as bandwidth per unit of capacity:

$$\text{BW density} = \frac{W_\text{BW}}{C} \quad [\text{TB/s per GB}]$$

Memory technology	Bandwidth	Capacity (typical)	BW density
H100 HBM3	3.35 TB/s	80 GB	42 GB/s/GB
MI300X HBM3	5.30 TB/s	192 GB	28 GB/s/GB
Corsair SRAM (d-Matrix)	150 TB/s	2 GB	75,000 GB/s/GB

The ITL floor for a model that fits entirely in a given memory is simply $P b_w / W_\text{BW}$. For a 1.6B-parameter INT8 draft model (1.6 GB):

$$\text{ITL}_\text{floor,GPU} = \frac{1.6\,\text{GB}}{3.35\,\text{TB/s}} \approx 0.48\,\text{ms}$$ $$\text{ITL}_\text{floor,SRAM} = \frac{1.6\,\text{GB}}{150\,\text{TB/s}} \approx 0.011\,\text{ms}$$

The 43× gap in per-token latency for the draft model is what makes SRAM chips compelling for speculative decoding [Chen et al., 2023]: a small draft model generates $k$ candidate tokens, then the large target model verifies them in a single forward pass (equivalent to a prefill of $k$ tokens). The effective ITL under speculative decoding is:

$$\boxed{\text{ITL}_\text{eff}(k, \alpha) = \frac{k \cdot \delta + t_\text{verify}(k, B, S)}{\mathbb{E}[\text{accept}]}, \quad \mathbb{E}[\text{accept}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}}$$

where $\delta$ is the draft-model step time, $t_\text{verify}$ is one forward pass of the target model treating the $k$ candidate tokens as a prefill batch, and $\alpha$ is the per-token acceptance rate. For $k = 5$ and $\alpha = 0.8$: $\mathbb{E}[\text{accept}] = 3.69$ tokens.

Computing at $B = 1$ (single interactive user), $S_\text{ctx} = 1024$, DeepSeek R1 as target on 8×H200 FP8 ($k = 5$, $\alpha = 0.8$, $\mathbb{E}[\text{accept}] = 3.69$):

Configuration	Draft step $\delta$	Verify $t_v$	Effective ITL	vs baseline
No spec decode, 8×H200	—	—	2.2 ms	1×
GPU draft + GPU verify	0.48 ms	0.012 ms	0.65 ms	3.4×
SRAM draft + GPU verify	0.011 ms	0.012 ms	0.018 ms	122×

For DeepSeek R1 on 8×H200, the verification step is tiny (0.012 ms — a compute-bound prefill of just $k=5$ tokens at 31.7 PFLOPS). With GPU drafting, the 5 draft steps (5 × 0.48 ms = 2.4 ms) dominate the numerator, limiting the speedup to ~3.4×. With SRAM drafting the draft cost falls to 0.055 ms, dropping effective ITL 122× below the un-assisted baseline. At larger batch sizes $B$, the verification step grows with $t_\text{step}(B, S)$ and the gap narrows — SRAM is most valuable in the interactive regime where $t_\text{verify}$ stays short.

From a hardware design perspective, SRAM and HBM are complementary, not competing: SRAM handles bandwidth-critical small models; HBM handles capacity-critical large models. The optimal serving stack is therefore heterogeneous — draft on SRAM, verify on HBM-GPU — which is exactly the Corsair deployment model.

Limitations of this model

The curves derived here are instructive but rest on several simplifying assumptions that real systems violate to varying degrees.

Variable output lengths. The TTFT formula in §4 defaults to CV = 0, which assumes all requests generate the same number of output tokens. Real workloads violate this: service time varies with output length $N_\text{out}$, which the model cannot predict in advance. The M/G/1 Pollaczek–Khinchine mean value formula [Kleinrock, 1975] captures this via the squared coefficient of variation $\text{CV}^2$ of service times:

$$W_q^{M/G/1}(\rho) = \frac{1 + \text{CV}^2}{2} \cdot \frac{\rho \cdot t_\text{step}}{1 - \rho}$$

At $\text{CV} = 0$ (deterministic) this reduces to M/D/1; at $\text{CV} = 1$ (exponential service, i.e. M/M/1) the queuing penalty doubles. Interactive chat workloads empirically show $\text{CV} \approx 0.8$–$1.2$ depending on task mix [Yu et al., 2022]. The Output length CV slider in the shared controls applies this correction live: every TTFT curve already incorporates the M/G/1 factor $(1 + \text{CV}^2)$. Set it to 1.0 and watch the frontier shift upward — the gap is the region your SLA must account for beyond the CV = 0 baseline.

Prefill-decode interference. We treat prefill and decode as perfectly isolated. In practice, a 10K-token prefill takes ~13 ms on 8×H200 and stalls all in-flight decode slots, spiking ITL for existing requests. Chunked prefill [Sarathi-Serve] bounds this spike by interleaving sub-100-token prefill chunks with decode steps — but the chunking overhead means our $t_\text{prefill}$ formula underestimates effective prefill latency by ~15%.

Constant context length. We model a single average $S_\text{ctx}$. Real workloads are heavy-tailed: 10% of requests may have $4\times$ the average context, disproportionately increasing KV-cache pressure and reducing $B_\text{max}$ below the prediction, shifting the effective operating point left on the frontier.

KV-cache utilization. Our $B_\text{max}$ formula assumes perfectly contiguous allocation. PagedAttention [Kwon et al., 2023] reduces internal fragmentation to ~4% (their §4.2), but under high request heterogeneity effective KV-cache utilization is typically 85–90% of the theoretical maximum due to external fragmentation and reservation overhead. The effective capacity is:

$$B_\text{max}^{\text{eff}}(S) = \left\lfloor \eta \cdot \frac{\text{HBM}_\text{free}}{S \cdot \kappa} \right\rfloor, \quad \eta \in [0.85,\,1.0]$$

The KV utilization slider (default 87%) applies $\eta$ directly to $B_\text{max}$ in all figures. Reducing it shrinks the frontier rightward extent — fewer concurrent requests can be sustained, and the queuing blowup arrives sooner.

Speculative decoding changes the operating point substantially. The SRAM analysis in §10 gives the full model: effective ITL can drop by 25–400× depending on draft hardware and batch size. The main caveat is that the acceptance rate $\alpha$ is workload-dependent and difficult to predict from hardware specs alone.

Dense transformers only. MoE architectures (Mixtral, DeepSeek V3) activate only a fraction of parameters per token. Active parameter count sets FLOPs, but total parameter count still determines memory bytes, decoupling $I(B)$ from $2B/b_w$ in a way that generally favors larger batch sizes relative to dense models.

Pipeline bubble overhead. The PP model in §7 accounts for P2P transfer latency between stages but not pipeline bubbles — the idle time in stage $i$ while stage $i+1$ processes its micro-batch. Real pipeline-parallel deployments incur 20–40% effective throughput reduction from bubble overhead [Narayanan et al., 2021], which our model does not capture separately from the communication cost.

Quantifying the gap: ideal vs. realistic frontier

Figure 9 overlays three curves on the same axes to make the combined effect of the M/G/1 and KV-utilization corrections tangible. The ideal curve (CV = 0, util = 100%) is the pure physics limit — what a perfectly scheduled, zero-fragmentation system could achieve. The current curve reflects your slider settings. The realistic curve (CV = 1, util = 87%) approximates a production interactive workload. The vertical gap between ideal and realistic at any throughput is the TTFT headroom that real providers must absorb through better scheduling and tighter KV management.

References

1. Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An insightful visual performance model for floating-point programs and multicore architectures. Communications of the ACM, 52(4), 65–76.
2. Hennessy, J. L., & Patterson, D. A. (2017). Computer Architecture: A Quantitative Approach (6th ed.). Morgan Kaufmann. — Chapter 4 (Data-Level Parallelism) and Appendix D (Memory Hierarchy Design) cover the roofline methodology and parallel memory systems in depth.
3. Kleinrock, L. (1975). Queueing Systems, Volume 1: Theory. Wiley-Interscience. — The canonical reference for M/G/1 queuing theory and the Pollaczek–Khinchine formula used in the TTFT derivation.
4. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2023). Efficiently scaling transformer inference. Proceedings of MLSys 2023. — Derives partitioning strategies and gives the roofline analysis for large model inference under tensor parallelism.
5. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. Proceedings of SOSP 2023. — Introduces paged KV-cache management; the $B_\text{max}$ analysis here assumes an ideal (unfragmented) allocator equivalent to their design.
6. Yu, G., Kim, J., Jeong, H., Jeong, J., Kim, B., Park, S., Kwon, H., & Jeong, J. (2022). Orca: A distributed serving system for transformer-based generative models. Proceedings of OSDI 2022. — Introduced continuous (iteration-level) batching, the scheduling model underlying the M/G/1 queuing analysis in §4. Also the source for empirical CV ≈ 0.8–1.2 for interactive chat workloads.
7. Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., & Ramjee, R. (2023). SARATHI: Efficient LLM inference by piggybacking decodes with chunked prefills. arXiv:2308.16369. — Chunked prefill decouples TTFT from ITL; see Section 3 for the prefill-decode interference analysis referenced in the ITL discussion.
8. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., & Jumper, J. (2023). Accelerating large language model decoding with speculative sampling. arXiv:2302.01318. — Introduces speculative sampling; the acceptance-rate formula $\mathbb{E}[\text{accept}] = (1-\alpha^{k+1})/(1-\alpha)$ follows from their analysis.
9. Gimlet Labs. (2025). Low-latency speculative decoding with d-Matrix Corsair. gimletlabs.ai/blog/low-latency-spec-decode-corsair. — Demonstrates heterogeneous SRAM+GPU speculative decoding achieving 2–10× ITL improvement for interactive workloads. Hardware specs (150 TB/s, 2 GB SRAM) used in the SRAM analysis above.
10. DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948. — The primary model studied in this post. 671B total parameters, ~37B active per token, 61 layers, $d_\text{model} = 7168$, MLA KV compression with $d_c = 512$, $d_r = 64$.
11. Llama 3 model architecture: Dubey, A., et al. (2024). The Llama 3 herd of models. arXiv:2407.21783. Cited for architecture comparison (GQA KV cache formula); DeepSeek R1 is the primary model in this post.
12. DeepSeek-AI. (2024). DeepSeek-V3 technical report. arXiv:2412.19437. — Introduces Multi-head Latent Attention (MLA), which compresses the KV cache to a low-rank latent vector per layer, reducing KV memory by roughly $d_h$-fold relative to GQA.
13. Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2023). Splitwise: Efficient generative LLM inference using phase splitting. arXiv:2311.18677. — Proposes disaggregated prefill/decode serving, routing each phase to silicon optimized for compute vs. bandwidth respectively.
14. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., & Zhang, H. (2024). DistServe: Disaggregating prefill and decoding for goodput-optimized LLM serving. arXiv:2401.09670. — Contemporaneous disaggregated serving work (also marketed as TetriInfer); the goodput analysis motivates compute-dense prefill nodes and bandwidth-dense decode nodes.
15. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vyas, D., Rajbhandari, S., Puri, R., Zhang, M., Shoeybi, M., & Catanzaro, B. (2021). Efficient large-scale language model training on GPU clusters using Megatron-LM. Proceedings of SC 2021. — Measures pipeline bubble overhead of 20–30% for large PP degrees; the 20–40% range cited in the limitations section is from their Table 3.