Research Lab
Attention Is a Yield Problem
Research note: the surprising overlap between transformer inference optimization and semiconductor yield engineering.
- transformers
- inference
- semiconductors
A working hypothesis I keep returning to: transformer inference optimization and fab yield engineering are the same discipline wearing different badges.
Both fields take a fixed, enormously expensive physical substrate and try to extract more useful output from it without changing the substrate itself.
The parallel, made concrete
| Yield engineering | Inference optimization |
|---|---|
| Die per wafer | Tokens per second |
| Process window tuning | KV cache management |
| Binning and speed grades | Quantization levels |
| Defect density reduction | Attention sparsity |
| Metrology feedback loops | Speculative decoding acceptance rates |
In both cases the winning strategy is identical: measure obsessively, find where the theoretical capacity is being wasted, and attack the largest waste term first.
Why KV cache is the new defect density
Defect density dominated yield economics for decades because it compounded: one particle killed a whole die. KV cache memory dominates inference economics the same way: it's the term that scales with context length and batch size simultaneously, and it decides how much of your expensive HBM is doing useful work.
Techniques like grouped-query attention, paged KV caches, and cache quantization are, functionally, yield improvement programs for compute.
Open questions I'm chewing on
- Does the fab concept of a "process window" have a rigorous analog in serving configs (batch size × context × quantization), a region where quality is stable and outside which it degrades non-linearly?
- Fabs have SPC charts for every step. What's the equivalent statistical process control for a production LLM serving stack?
If you're working on either side of this boundary, I'd genuinely like to compare notes.