# Post Hot Chips 2026: a review of custom chips

By Hanif Leoputera · 2026-09-08 · https://hanifleo.com/writing/post-hot-chips-2026/

> A tour of custom AI chips, and why the interesting comparison is speed per user against throughput, not FLOPs.

**Thesis:** I think most custom AI chips are bets on where a model's bytes live, and on which customer pays for speed, more than on compute.

In April 2025, when Meta announced it would serve Llama through Cerebras, running in Cerebras's own data centers, I said then that it made sense for Meta and nobody else ([Cerebras](https://www.cerebras.ai/press-release/meta-collaborates-with-cerebras-to-drive-fast-inference-for-developers-in-new-llama-api)). Meta cared about latency enough to pay for an unusual machine. I didn't think the other big buyers would bother.

Less than a year later, OpenAI signed for 750 megawatts of Cerebras capacity, a deal Cerebras's own filing values at more than $20 billion ([S-1/A](https://www.sec.gov/Archives/edgar/data/2021728/000162828026033143/cerebras-sx1a2.htm)). Cerebras went public at $185 a share ([IPOScoop](https://www.iposcoop.com/the-ipo-buzz-cerebras-cbrs-prices-ipo-at-185-25-above-range-2/)). I was wrong about Cerebras, and I wanted to understand why.

The short answer is that I was looking at the wrong numbers. Chips get compared on FLOPs. For serving a language model, two other things matter more: where the model's bytes live, and whether the customer is paying for speed or for volume.

## bandwidth, not FLOPs

Generating text has two phases. Prefill reads your whole prompt at once, which is a lot of arithmetic on data already loaded, so it is limited by compute. Decode produces one token at a time, and every new token needs every weight in the model read from memory again. Decode is limited by memory bandwidth.

That gives a rule of thumb you can do in your head. Tokens per second for one user is roughly memory bandwidth divided by the bytes read per token. A 70-billion-parameter model at 8 bits is 70 GB per token, so a chip that reads 8 TB/s gives that user a little over a hundred tokens per second, however many FLOPs it has:

*Figure 1 (interactive, see https://hanifleo.com/writing/post-hot-chips-2026/): One user is limited by bandwidth. Only large batches reach the compute ceiling, and each user gets slower as they do.*

The bend in that line is the trade-off the rest of this essay is about. Serve one user and the chip spends most of its time waiting on memory. Serve hundreds at once and you use the compute you paid for, but every user gets fewer tokens per second. So I think of a chip as having a curve rather than one speed, and the batch size slides you along it.

The bend has a simple cause. In each decode step the chip reads the weights once for the whole batch, but it reads each user's cache of their own conversation separately. While memory is the limit:

```text
time per step  ≈  (weight bytes read  +  batch × cache bytes per user)  ÷  bandwidth
speed per user =  1 ÷ time per step
throughput     =  batch ÷ time per step
```

At small batches the weights dominate, so another user is almost free. As the batch grows the caches take over, and every extra user slows everyone down. Long contexts make each cache larger, which is why the bend comes sooner for a coding agent with a 100,000-token prompt than for a short chat.

The 70B example is also simpler than what is actually served. Frontier open models are mixtures of experts. DeepSeek-V3 has 671 billion parameters but uses 37 billion for each token ([DeepSeek](https://arxiv.org/abs/2412.19437)), and gpt-oss-120b uses 5.1 billion of its 117 billion ([OpenAI](https://openai.com/index/introducing-gpt-oss/)). One user's token reads only the active experts, so the small number sets per-user speed, while the machine must still hold the large one. Batching also helps less than it does for a dense model. Different users route to different experts, so as the batch grows each step touches more of the model, and the weight term creeps from the active size toward the total.

## goodput

That is why careful benchmarks don't stop at peak throughput. They fix a speed every user must get, and ask how much throughput survives above it. The DistServe authors call this goodput ([DistServe](https://haoailab.com/blogs/distserve/)). A system that serves ten requests a second, of which only three meet the latency target, has a goodput of three.

Drag the line and watch the ranking change:

*Figure 2 (interactive, see https://hanifleo.com/writing/post-hot-chips-2026/): Each chip is a curve. Set the speed every user must get, and see which chip delivers the most above it.*

The ranking depends on where you draw the line. A chatbot that streams text at reading speed can batch heavily, and the cheapest chip per token wins. A coding agent that has to finish a long edit before you lose interest needs far more tokens per second per user, and some chips simply can't get there at any batch size.

## reading an inference number

There are a few less obvious things I've learned to check when someone shows me an inference number. The first is whether the number is an average. Users feel the slowest requests, so the useful figure is the 99th percentile, and averages can hide stalls of several seconds ([Etalon](https://arxiv.org/abs/2407.07000)). The second is whether long prompts were in the mix. A big prefill can stall everyone else's generation unless the scheduler breaks it into chunks ([Sarathi-Serve](https://arxiv.org/abs/2403.02310)).

The third is memory again. Every active conversation keeps a cache of its context, and that cache often fills the chip's memory before compute runs out, which ends the curve early. The last is speculative decoding, where a small model drafts tokens for the big one to check. It makes a lightly loaded system faster and a heavily loaded one slower. A benchmark that shows only one load level tells you very little.

## where the bytes live

Once I thought about chips this way, the landscape started to look less like a race and more like a map. I think each custom chip is a bet on one corner of it:

*Figure 3 (interactive, see https://hanifleo.com/writing/post-hot-chips-2026/): Each chip is a bet on where the weights live. The diagonals show how many times a second it can read its own memory.*

At the top left are the SRAM chips. Groq, whose technology Nvidia licensed in December 2025 in a deal reported at about $20 billion ([Groq](https://groq.com/newsroom/groq-and-nvidia-enter-non-exclusive-inference-technology-licensing-agreement-to-accelerate-ai-inference-at-global-scale), [CNBC](https://www.cnbc.com/2025/12/24/nvidia-buying-ai-chip-startup-groq-for-about-20-billion-biggest-deal.html)), keeps every weight in on-chip SRAM. Cerebras does the same on a single wafer the size of a dinner plate, with 44 GB of SRAM. These chips can read their entire memory tens or hundreds of thousands of times a second.

d-Matrix takes the idea one step further. Instead of moving weights to a separate compute unit, it puts small multipliers right next to the memory cells, so the weights barely move at all:

[![d-Matrix slide titled Energy Efficient DIMC Architecture: a diagram of a dual-bank SRAM array where each weight cell has a multiplier beside it and adder trees run between columns, with bullets: logic-rule SRAM in TSMC 6nm, dual bit-serial 4-bit-parallel MAC array, 64x64 matrix ops in MXINT8, 64x128 in MXINT4, fully digital design.](https://hanifleo.com/writing/post-hot-chips-2026/dmatrix-dimc-hotchips2025.jpg)](https://hc2025.hotchips.org/assets/program/conference/day2/24_dMatrix_Bhoja_final.pdf)

*Figure 4: d-Matrix's in-memory compute array: each weight (W) sits next to its own multiplier, and adder trees sum the columns inside the memory. Slide 13 of Sudeep Bhoja's Hot Chips 2025 talk.*

The reason this matters is energy. By d-Matrix's figures, reading a bit from on-chip SRAM costs about 50 femtojoules, while reading it from HBM costs about 2.4 picojoules, roughly fifty times more ([Hot Chips 2025](https://hc2025.hotchips.org/assets/program/conference/day2/24_dMatrix_Bhoja_final.pdf)). In inference, most of the energy seems to go into moving data rather than arithmetic.

At this year's Hot Chips, d-Matrix showed where it is going next, and it is a quiet admission of the catch below. Its Raptor design stacks DRAM directly under the compute die instead of relying on SRAM alone, aiming for 32 GB per card at about 100 TB/s and roughly a third of a picojoule per bit, all still projections ([ServeTheHome](https://www.servethehome.com/d-matrix-raptor-3d-dram-accelerator-for-generative-inference-at-hot-chips-2026/)).

The catch with SRAM is capacity. It is fast because it is small and sits next to the compute, which means you need many chips to hold one model:

*Figure 5 (interactive, see https://hanifleo.com/writing/post-hot-chips-2026/): Weights only. The SRAM chips need racks to hold what one GPU holds.*

A 70B model at 8 bits needs on the order of 140 Groq 3 chips, two Cerebras wafers, or one Rubin GPU. That was my original objection: 44 GB is not much memory for a frontier model, so you end up spreading it across a lot of silicon.

Mixtures of experts sharpen both sides of that objection. At 8 bits, DeepSeek-V3 needs about 671 GB just to sit in memory, which is more than 1,300 Groq 3 chips or 16 Cerebras wafers. But each token reads only about 37 GB of it, so a machine that can hold the whole model serves it very fast.

What I missed is where these chips sit on the goodput curve. As far as I can tell, they are the only ones that reach the far right, thousands of tokens per second for a single user, and some customers will pay a lot for that. For gpt-oss-120b, Cerebras was the most expensive provider Artificial Analysis listed when I checked, at a blended $0.39 per million tokens against $0.04–0.07 on GPU clouds, and also the fastest ([Artificial Analysis](https://artificialanalysis.ai/models/gpt-oss-120b/providers)). People pay that premium. Specialized chips seem to win where someone pays for speed, not where they're trying to save money.

## power

The other number that has quietly taken over is power. New data centers are limited by how many megawatts they can get, not by how many chips they can buy, so the real metric becomes tokens per second per megawatt. This is where a familiar trade-off in chip design shows up.

A chip's dynamic power is roughly its switched capacitance times voltage squared times clock frequency, and a higher clock needs a higher voltage. Near the top of the range, voltage rises roughly in step with frequency, so power grows with about the cube of the clock while speed grows only linearly. On that approximation, the last 20% of clock speed costs about half the power. I first noticed this in Etched's pitch about running its transformer chip at low voltage, which I had initially dismissed. Here is the trade-off, with real measurements on top:

*Figure 6 (interactive, see https://hanifleo.com/writing/post-hot-chips-2026/): Lowering the clock costs little speed but saves a lot of power, and decode barely notices because it's waiting on memory anyway.*

Notice that decode hardly slows down at all. Because it is waiting on memory, a lower clock costs almost nothing. Microsoft's measurements on Azure found that capping GPU power could save up to 20% of peak power for 7% or less in performance ([Patel et al.](https://homes.cs.washington.edu/~patelp1/papers/llmpower-asplos24.pdf)). In the Splitwise paper, an H100 doing only decode was capped from 700 W to 350 W with almost no added latency ([Splitwise](https://arxiv.org/abs/2311.18677)). Nvidia now ships a "Max-Q" mode for data centers on the same logic ([Nvidia](https://developer.nvidia.com/blog/optimize-data-center-efficiency-for-ai-and-hpc-workloads-with-power-profiles/)). In a power-capped building, more chips at a lower voltage can beat fewer chips at full speed.

## the hyperscalers

The hyperscalers are making the same bet, more quietly, and one ratio shows it: peak compute divided by memory bandwidth. For a dense model at 8 bits, each user's token does about two operations per byte of weights read, so a chip with a ratio of R operations per byte needs a batch of roughly R/2 before it stops waiting on memory. Nvidia's Rubin, at about 17.5 FP8 petaflops over 19.2 TB/s, has a ratio near 900, so it wants batches in the hundreds.

Each new inference chip moves that ratio down. Meta's MTIA 450 doubles memory bandwidth over MTIA 400, to 18.4 TB/s, while FP8 compute grows by about 17%, which takes the ratio from about 650 to about 380 ([Meta](https://ai.meta.com/blog/meta-mtia-scale-ai-chips-for-billions/)). In April Google split its eighth-generation TPU in two. The inference chip, 8i, has less peak compute than the training chip, 8t, but about 30% more bandwidth and three times the on-chip SRAM, 384 MB against 128 MB ([Google](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive)). These are chips built to serve small batches fast, which a training chip isn't designed to do. Owning them is also leverage against a supplier whose margins are famously high.

## the network, and the split

There is another part of the bet that the spec sheet hides: the network. For large mixture-of-experts models, how many chips can act as one machine matters as much as any single chip. Google's TPU 8t puts 9,600 chips on one inter-chip network, and its data-center fabric joins more than 134,000 of them. Nvidia's NVLink domain is 72 GPUs per rack today, with 576 planned for Rubin Ultra.

Not everyone thinks specialization ends well. Vincent Zhong's [Disaggregation Trap](https://vincentzed.com/writing/disaggregation-trap) is not against splitting prefill from decode; large serving systems already do that in software, on identical GPUs that can switch roles in minutes. His objection is to making the split in silicon. A chip that can only do one phase fixes the ratio between them when you buy the hardware, while the workload keeps shifting over a four-to-five-year life. You pay for every pool and get the throughput of the most starved one. Nvidia's own roadmap shows the tension: it announced a prefill-only GPU, Rubin CPX, appears to have since dropped it, and specialized the decode side through Groq instead ([The Register](https://www.theregister.com/2026/03/16/nvidia_lpx_groq_3/)). The power results above point the same way. A power cap on identical GPUs gets much of the benefit of a separate decode chip, without locking in the ratio.

The split is also moving deeper. Some systems now divide decode itself: attention, which reads each user's own memory and gains nothing from batching, stays on memory-rich chips, while the expert layers, whose weights every user shares, run on a separate pool ([MegaScale-Infer](https://arxiv.org/abs/2504.02263), [Step-3](https://arxiv.org/abs/2507.19427)). It is the same question as the rest of this essay, one layer down: which bytes belong to one user, and which are shared.

## the stranger bets

The remaining bets are stranger. Tensordyne, formerly Recogni, does its arithmetic in a logarithmic number system, which turns multiplications into additions ([The Next Platform](https://nextplatform.com/compute/2026/06/16/tensordyne-converts-ai-matrix-math-to-logs-to-crank-up-inference-oomph/5256558)). As far as I can find, it hasn't published accuracy data yet. Etched hard-wires the transformer into silicon ([The Register](https://www.theregister.com/2024/06/26/etched_asic_ai/)). Both are chasing a moving target, because the GPU keeps getting the same benefit from cheaper number formats, first FP8 and now FP4.

## what I ask now

The part I'm least sure about is whether the models will hold still long enough for any of these bets to pay off. Diffusion language models, for example, generate a whole block of text at once instead of one token at a time, which pushes decode back toward being limited by compute ([Mercury](https://arxiv.org/abs/2506.17298)). If that kind of model wins, a chip built around memory bandwidth looks less clever.

So I've stopped asking which chip is fastest. I ask what speed its buyer is paying for, where its bytes live, and how much power it burns to get there. If you have a better answer than mine, especially from running one of these in production, I'd like to hear it.