- 210comments
- 174comments
- 193comments
- 28comments
- 181comments
- 12comments
- 104comments
- 77comments
- 58comments
- 105comments
- 5comments
- —discuss
- 38comments
- —discuss
- 115comments
- 299comments
- 2comments
- 386comments
- 55comments
- 146comments
- —discuss
- 8comments
- 30comments
- 64comments
- 72comments
- 301comments
- 204comments
- 9comments
- 140comments
- 282comments
Excellent article. I believe the majority of benchmark performance gains moving forward will come from this side of the stack enabling faster iteration/recursion.
I knew of this but not the $ amount. Wow
Big money from taking Tesla's place in line for an Nvidia shipment during a huge shortage without compensation.
Great read.
I like how the author uses the analogy of scrabble word creation to describe LLM training but unfortunately the analogy didn't continue to inference and I got lost trying to keep up.
The analogy is flawed - in Scrabble you get rewarded for unlikely word combos. Also causal attention attends to prior tokens (tiles already placed).
"If AI inference remains as desirable as Kimball expects, the evolution is likely to follow the same trajectory as the CPU. The CPU didn’t improve along a single axis but instead across simultaneously. Once transistor scaling slowed, chip and system architecture innovations of all kinds proliferated. The list of individual innovations that led to today’s ubiquitous, powerful personal compute could fill dozens of books. A few decades from now, the history of AI inference innovation will show similar depth."
Of the areas mentioned in the article, which are the most likely to have the most prominent innovative impact, and what will they entail?
https://a.co/d/0dMk8urP
which is the Hennesey and Patterson computer architecture book would serve the role of the "dozens of books" hyperbole rather well.
One of the biggest limitations right now is memory capacity (storing large models/contexts in memory) and bandwidth (transferring the relevant data/weights to the silicon that is performing the operations on that data). This would cover things like:
1. having more memory on the card/chip and/or faster access to that memory;
2. integrated memory and compute units optimized for matrix and vector multiply add operations;
3. optimized load circuitry to e.g. read memory in the stride and span (next row, next column) access patterns common to matrices or ensure that no/few parts of the chip are stalled waiting on data or operations to complete.
Another aspect is quantizations. These are similar to SIMD vector operations in that you are performing an operation on a block of n-bit data values at the same time, so can have optimized circuitry.
For 2 or 3 valued quantizations you can reduce various addition and multiplication operations to logic operations, avoiding circuitry for things like the half-adder, full-adder, and carry-lookahead.
Then there's adding specific circuitry for common operations such as ReLU like is done in hardware acceleration of image, video, etc. processing. There's a trade off here as optimized hardware would perform better at the specific operations but if those are too specific then they can't be used by different/newer model architectures. (Though it does make sense to try and optimize common operations/logic where possible.)
It would be interesting to see if these designs can/will benefit training as well, as that would bring down the time/cost/energy of training large models as well as making it easier for local fine-tuning.
A large fraction of the innovation in CPUs is driven by working around the memory wall. I anticipate AI inference will follow the same trend, and innovations that work around the autoregressive nature will be enormously impactful.
Speculative decoding is an example. An accurate draft model can reduce the number of times you stream through memory by a factor of 4x.