- 461comments
- 302comments
- 387comments
- 559comments
- 380comments
- 518comments
- 296comments
- 155comments
- 5comments
- 504comments
- 84comments
- 453comments
- 359comments
- 369comments
- 198comments
- 365comments
- 289comments
- 346comments
- 97comments
- 253comments
- 217comments
- 466comments
- 223comments
- 326comments
- 273comments
- 122comments
- 367comments
- 70comments
- 2comments
- 40comments
Nice work. I really appreciate this tool for enhancing my limited understanding the mechanism(s) behind attention and LLMs.
Damn that page took down my Chromebook, never happened before..
twice...
Text under the "Examples" section...
"Try examples while GPT-2 model is being downloaded (600MB)"
That's a hefty chunk of download and likely compute too.
Would be curious if https://spreadsheets-are-all-you-need.ai/gpt2/ works on your Chromebook. You need to download the weights and then drag and drop them back into the page but maximum compatibility was one of the design goals.
All good I’ll check on my main!
Thanks!
Great site, intuitive description. I also found [1] very useful in the past.
[1] https://bbycroft.net/llm
this is awesome, thanks for sharing
Discussed a couple times:
LLM Visualization - https://news.ycombinator.com/item?id=45130260 - Sept 2025 (46 comments)
LLM Visualization - https://news.ycombinator.com/item?id=38505211 - Dec 2023 (131 comments)
This is not at all what I was hoping for. Expected a lot more Unicron.
Definitely more than meets the eye
The page diagram looks like Cybertron
Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector through Dense layer of ordinary network where Attention matrix forms weights of that layer. So attention head is trained to construct this small single layer network dynamically during inference from Key and Query. And that's the point. That's rarely underlined in explanations of LLMs architecture and for me it's quite amazing that it works so well. This mechanism easy to observe in this particular visualization if you click through it.
Can you explain this more, maybe dumb it down a little? Sounds important. I have t quite been able to get the attention section to click for me.
Look at this poster [1] (its low-res, full res is paid). Also see this video for context [2] - it talks about deepseek's innovation, but explains attention well. Above I was pointing to the moment where multiplication AV happens. In normal Dense layer in MLPs when you go through the layer you compute y=Wx, where x is an input and y is output (before gactivation) and W is a weight matrix. This W is usually what's produced through training process. This multiplication AV looks like Wx. If we take one column from matrix V in the poster and call it v, we can pretend for one moment that W=A and x=v [3]. So this multiplication Av works like linear transformation in ordinary network Wx. But in case of attention heads A is not trained directly but produced during inference, and is not trained directly like W is in ordinary network. In this case network is trained to produce A during inference.
[1] https://www.welchlabs.com/store/mladeepseek-attention-poster...
[2] https://www.youtube.com/watch?v=0VLAoVGf_74
[3] When multiplying A by V, we perform the same linear transform Av_i for each i-th column of V.
Welch Labs has a beautiful visualization in their YouTube video about Vision Language Action Models, where the attention of the prompt can be traced to the exact origin pixels in the image from one of those matrices.
It's definitely part of many intro explanations, but it sits in a weird position where you have to know MLPs well but not Transformers for this to be enlightening, which is a shrinking set of people nowadays. Either someone is a novice to all this, or know both, so beginner Transformer explanations can rarely assume deep familiarity with MLPs.
But yes, a Transformer block can be thought of as basically input-dependently deciding the weight vector of a dense layer. In classical MLP-like networks there was never any multiplication between input-dependent values (there are exceptions, like Squeeze-and-Excitation layers). Multiplication is always between parameters (that don't depend on the input) and activations that depend on the input. With Transformers attention provides a lot of multiplicative interactions between input dependent activations.
Also obligatory: Schmidhuber talked about this a long time ago.
And again recently: https://arxiv.org/abs/2102.11174
That is basically axiomatic ;) (Seriously, Schmidhuber's body of research is mind-blowing)
The reason it's rarely explained is that Schmidhuber discovered this a long time ago (Fast Weight Programmers). And for whatever reason, people take perverse pleasure in not citing him.
LLM Visualization: https://bbycroft.net/llm
I'm a huge 3Blue 1Brown fan and his series on this was great: https://www.youtube.com/watch?v=wjZofJX0v4M
I get that this is for explaining GPT-2, but I really hope laymen don't use it as an example of how modern models work (ex. absolute positional encoding is no longer used)
edit: I know that it mentions its not modern, but these kinds of details have major implications in terms of the representations a model can learn, which is in many ways the most important part!
This is how pedagogy works. You must simplify something.
I'm talking about bringing up a specific outdated details, not about simplifying it
I think a better simplification would be to remove the position-encoding altogether, and just discuss transformers as set models.
Yeah it's a bit odd, especially since RoPe is a lot more conceptually simple imo.
Interestingly, I find absolute positional embeddings easier to explain to a layperson than rope, e.g. [1]. What am I missing?
[1] https://www.youtube.com/watch?v=ZuiJjkbX0Og&t=5712s
I never understood the thinking behind the separate key query value matrixes? What are they doing exactly?
Well the embedding itself is in some latent space.
Wq projects it to the space of queries. I.E What questions is this token asking?
Wk projects it to the space of keys. I.E What questions does this token answer.
Wk projects it to the space of values. I.E What are those answers?
Of course this explanation is prescribed onto the matrixes after the fact.
You can in fact do weird stuff like construct weights so attention calculates least squares, or sorts numbers, or other weird constructions like a transformer that calculates gradient descent steps. It seems to be very flexible in terms of what functions on data it can encode.
I guess I'll say it: despite your good intent and effort, this doesn't explain a thing, not to the majority of people asking the question like above.
This sort of explanation, whilst being technically correct, is right up there with "A monad is a monoid in the category of endofunctors. What's your problem?".
It's a hashmap; both the keys and the values are token positions.
It's a "fuzzy" hashmap; insead of hashmap.get("ball")=="threw" it assigns a probability to every pair of words.
Each hashmap captures some kind of relationship between words.
For example, every LLM has lots of heads whose relationship measures "is token1 the noun on which the verb token2 is acting"? So "I threw the ball" would have a high probability for ("ball", "threw").
But most of the hashmaps don't capture such easy-to-explain relationships. Some of them do. The rest probably capture relationships that we haven't figured out yet. This is the truly mysterious stuff.
But it's just hashmaps. Hashmaps all the way down.
Regarding the temperature explanation:
Safety is definitely the wrong word here.
Temperature 0 generated text actually has a weird "lack of surprise" character that makes it seem artificial. [1]
I'd completely drop the dropout explanation. It's just not part of the modern recipe anymore, AFAICT.
As for the ambitious goal of explaining transformers with a single interactive visualization, I just have a hard time imagining a person is going to newly understand both word embeddings (word2vec blew my mind in 2014) and also gain an understanding of attention.
I am making my own visualizations for a presentation on "Full Bandwidth Transformers"[2] that I am giving tomorrow at the Deep Learning Study Group (SF) (on zoom for the non-locals)[3]. It's not meant to be stand alone/context free, but I'd love some feedback.
https://rrenaud.github.io/fullbandwidth_transformer_viz/
[1] https://arxiv.org/abs/2202.00666 [2] https://arxiv.org/abs/2608.08888 [3] https://www.meetup.com/deep-learning-sf/events/316601593/
Safety is definitely the wrong word here.
Hey, take it easy on the guy. Words are just probabilities in some high-dimensional space; it's a probabalistic selection anyways; he probably just nudged the latent vector a little too hard. Spymarking and watermarking are for safety reasons, you know. There was zero loss of quality there.
Is there any other mechanism replacing it? As far as I understood, while following the LLMs from Scratch book, the dropout is a mechanism to prevent overfitting, and it makes a lot of sense (basically cripple some neuron connections randomly during training such that their weights aren't updated).
Edit: well, I could have searched on the internet, but other people may stumble on your post and you seem quite knowledgeable of the inner workings of these things. :)
As someone with an EE degree (though a sysadmin), this use of the term "transformer" is constantly confusing. :)
(Also "cryto" for cryptocurrency rather than cryptography.)
Seriously. I was expecting more Optimus Prime.
Should have been “energon” instead of “tokens”
transformer is climbing up in the most-overloaded-terms category, jockeying for a spot somewhere among “service”, “template” and “zone”
For the uninitiated, I can't recommend enough, The Illustrated Transformer:
https://jalammar.github.io/illustrated-transformer/
His book is also very good: https://www.amazon.com/Hands-Large-Language-Models-Understan...
seconded, the book is very good.
When people say a book is good what does that mean? Good for infotainment? To get an intuitive understanding? To be able to open up Openthropic and compete against the frontier labs? What can you get in the book thats not available via a chatbot? Interesting question
well i cant speak for anyone else, but I can tell you there are quite a few books on transformers etc and this one does the best job of explaining how the guts (attention heads etc) work. has it good visualizations too
DRM-free EPUB and PDF (for those who'll pay double to legally avoid Kindle DRM):
https://www.ebooks.com/en-us/book/211460386/hands-on-large-l...
Excellent resource for beginners
thank you
Why did the inner EE in me think this was going to be a power transformer lol
Would have been very interested. I know there are genius/insane people out there winding their own transformers.
I didn't realize this was a think until I heard of people doing it in a a CRT forum.
I suspect it will be getting more popular as flyback transformers get harder and harder to find.
I first heard of it here: https://ludens.cl/paradise/turbine/turbine.html (CTRL+F transformer), a pretty fascinating DIY medium distance power transmission setup.
Winding your own transformer is really not that big of a deal. Especially if you work in R&D as a power electronics engineer. No one waits for the custom transformer to be delivered to your bench when you were designing a new power supply.
For high power stuff though, you need some equipment to do that since copper will hard to bend neatly.
Could have been the cars that turn into robots for that matter
Great UI and it was very helpful to me as a newcomer
Hm, I wonder why my laptop is suddenly pulling about 5 fps while I'm just browsing HN?
The nefarious "2.2 GB RAM usage within 10 seconds" tab open in the background:
Can it do Jev too?
It seem that everyone is getting into details of how transformers work, but I am more interested in why other setups didn't work.
Or is it?
They do work, but transformers work better on the accuracy/compute tradeoff curve. Transformers let far away positions in the sequence exchange info in 1 step, while a dilated CNN will need many layers. Additionally, Transformers can be fairly efficiently trained.
I do think if Transformers weren't invented you would still be able to train powerful language models, but they would take more ram and be slower so nobody does this.
most people in ML have no idea what transformers actually are.
Traditional networks, at every layer, used to be output = [weights matrix][input], where input is a vector, and weights matrix is the weights, where each row corresponds to the set of weights for each neuron.
Transformers upscale the dimension of the data. Instead of the above, transformers do [output] = [input]
[weights_matrix]. When you multiply an input by a matrix, you get an output matrix back. Thats all that happens. Nothing fancy. You have weights matricies for K/Q/V, which when post multiplied with the input, give you the KQV vectors, and then you just simply multiply them together and apply a scaling factor.There is nothing magical about K/Q/V. There is nothing about any one doing any querying or any one representing some keys. The naming is just a carry over from how they that selection process is used in pre llm data science fields where you manually define the key and query matricies to define relationships between components.
The reason of why it works is because is an extension of something called kernel tricks from pre LLM machine learning days - you map a lower dimensional space to an extra dimension based on some equation, and it lets you apply some classifier on the combination of existing values and new value. Thats what transformers are doing - they are mapping the individual token to the dk x n_heads latent space, which allows for a higher dimensional representation of the data, capturing complex relationships.
You can do Transformers with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on. The thing is, there really isn't any way to tell if any of that gives you more advantage - it certainly would give you more granularity, but as of right now, in terms of training to generate a specific token given previous ones before it, it seems that you don't need any more dimentions than dk x n_heads. Interestingly enough, you also can mathematically represent any such transformer including the starting one with a sequence of linear layers like in traditional networks, the only thing is that it becomes computationally inefficient due to having duplicates of data.
The reason why RNNs and others and others didn't work is because RNN training is effectively trying to linearly regress on chaotic effects - i.e what set of starting conditions would evolve with a given process into what you want. This is an NP hard problem, and you can't really do it linearly.
Transformer models on the other hand, use breadth instead of compute to capture interactions. In those learned weight matrices, you have a latent space of a bunch of "knowledge" compressed, and an algorithm to search on that "knowledge".
But, its very possible that an RNN can be smarter than a frontier model while being much smaller in size - in the same way that its very possible that you can have the right set of prompts for an existing local inference smaller model that can basically be very close to AGI in terms of being able to solve any problem across any domain. Right now, the space is about exploring those prompts, which is the frameworks and harnesses, to get to there, as well as making the compute portion more efficient so you can explore that space faster.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
Thank you sir for this lengthy explaination.
As a outsider I have many dumb quesion like these. I am trying to understand transformers in a Occam's razor way. It's a complicated machinary after all.
Not dumb questions. These design decisions are based on years of applied testing, more than any theoretical result.
Imagine you have a soccer field, a ball with position x and y, a kick strength, and direction in an angle. Your job is to write a function that determines if the ball will end up in a goal. So that is 4 values. However the function itself will contain many intricacies, like trig functions, simulated drag, and so on.
In the contest of LLMs, you cant have these types of coded function. Your function has to be a mathematical equation that is smooth - i.e no discrete steps, no singularities. The reason for this is when any neural net is trained, you use backpropagation of the error to adjust weights, and how much you adjust them is directly proportional to the weights effect on the final output, and in order to compute this, you have to have smooth functions from start to finish.
So what you do instead is you add data to your 4 values, that capture different relationship between them. If your 4 values are x,y,k,and h, your first data point can be a1x + b1y + c1k + d1h. The second point can be a2x + b3y + c4k + d5h. And so on. You can have as many of those values as you want. And then you can add, combine, and scale those values in any way you chose.
This basically gives you a map of 4 values into a binary decision whether the ball will end up in a goal or not, after sufficient training. However, the total number of extra values that you chose has to be large enough to capture all possibilities - if you don't have enough, you will start to make mistakes for some initial conditions.
Hey I like your exposition of attention in terms of the kernel trick, but the big huge difference is that kernel methods use the inner product which is a commutative operation -- it's bidirectional (and both tokens are projected into the space by the same function prior to being dot-producted).
This means that it can't capture unidirectional relationships, like "ball" is the object on which the verb "threw" acts in the sentence "I threw the ball". This relationship is true in only one direction; it isn't true to say "threw" is the object on which the verb "ball" acts.
I do think it would be fair to say that transformers generalize the kernel trick to noncommutative relations by applying a different projection function (W^Q and W^K) to the two tokens being considered. This makes the overall operation (project then dot product) a noncommutative operation.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
Nirvana? Singularity? Paperclips? Vernor Vinge rising from the dead? I'm curious; please share!
I mean, given sentence construction, you don't really need to capture directionality, you just have a mapping of how sentences are constructed to the latent space of some representation.
Simulated evolution. Thats how you "solve" highly nonlinear chaotic systems. And generally, if you think about it, you have to have some secondary system on top of the knowledge embedded in LLMs to drive them to select certain tokens, which then starts to eerily resemble what humans call emotions in themselves.
edit:
...You have weights matricies for K/Q/V, which when post multiplied with the input, give you the KQV ** matricies **...
It isn't so much that "other setups didn't work". More like "the first thing we found that did work turned out to be the minimal thing that could work".
Transformers are conventional feed-forward neural networks alternated with attention blocks. You can think of them as big huge "ordinary" neural networks augmented with this new kind of block.
Attention blocks are basically just a differentiable hashmap. Think of it like a scratchpad memory.
It turns out that a hashmap/scratchpad is pretty essential to being able to untangle language. I don't find this too hard to believe. Somewhere in there, you have to build the graph of which object is acting via which verb on which object.
What is surprising is that this is all it takes! These simple little hashmap/scratchpad units (and massive scale) are really the only thing you need to tack on to a feed-forward neural network to get essentially general intelligence. This is totally surprising to me.
Because that is what experiments suggested and what current computational power can compute.
There is not much theoretical explanation yet. Or why others do not work.
Hand-waving mostly.
If I had to summarize this in with sentences then here is my attempt.
Transformer training can be parallelized easily, making it possible to use brute force to train the neural network quickly (bitter lesson rewards compute friendly scalable architectures).
Transformers have perfect retrieval, they re-read the entire context window from scratch for every token.
Explanation over.
If you extrapolate this, then the logical conclusion is that the next model architecture would use even more brute force.
Right now transformers can only append a token at the end. This means they can read any input, but write only one specific output.
If you wanted to extend this, you would want to make the transformer read from any input and write to any output, i.e make it capable of updating the entire KV cache every iteration.
One of the better visualisations I've seen with the exception of Q/K/V weights and how they are presented. I believe they should be put more upfront since they are the core learnable parameter of attention. IMO, they should also be part of the "Head N of M" block since each head has its own weights (although they all can be collapsed into one huge matrix computation).
I have no idea why but I was thinking about Optimus Prime and Megatron when I was clicking the link. I was a bit disappointed.
That is good too though.
This is not about Optimus Prime and the gang, is it?
I regret nothing.
Visualization is definitely a good way to learn new things. And I also would like to recommend https://bbycroft.net/llm . It has beautiful graphs, clear animations and good introductions, explainng the LLM inference cores well
Did something similar using a SimCity-like simulator for how LLMs work: https://laurentiugabriel.github.io/token-town/
Love this! Not having spent too much time on understanding the architecture I always struggled to see how transformers get away with compressing all context in a flat vector (e.g. 768 numbers here) between attention and the MLP when computing the current token. But now I think I understand that since you often alternate between attention and MLPs it regularly mixes/queries the information of other tokens into the current computation. Probably common knowledge for everyone that learned about transformers but this made it so much quicker to see.
Bummer it doesn't work on mobile, we used to be mobile first now it's whatever the clanker spits out. Before you push back I must remind you that https://ciechanow.ski/ has very complex visualization perfectly accessible from the mobile phone
Mobile first, almost never in my experience. Mobile also, usually.
What gives you the impression an LLM made this? It doesn’t have any overt signs.
Side scrolling and letting me zoom is also an acceptable compromise for mobile if you’re not going to put the time in to craft the experience for it imo. And some things just won’t be great on mobile, like downloading a 600MB GPT-2 model.
It's lazy, it doesn't look good on desktop too. The navbar screams justice. If it hasn't been made by an LLM it would still be a lazy human
One thing that was not intuitive for me is that attention is per token generator and in theory unbounded for context size. Per token approach does not generate attention matrix as presented here but attention vector. It could provide different view and I find that aproach easier to reason about.
I expected electrical transformers, then Transformers with a capital T, but not this. (still, very very cool!)
Exact same thoughts. It’s cool, still a bit disappointing as I wanted to see a cool visualization of magnetic fields and inductance!
save
Ah "semantic meaning", the new "illegal annexation"...