Attention in transformers, step-by-step | Deep Learning Chapter 6

What Hacker News said

Strong rec tomashm · 2024-07 · on “The Illustrated Transformer (2018)”

This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. [LINK] 1. [this video]

Strong rec onlyrealcuzzo · 2026-05 · on “Claude Opus 4.8”

Because it doesn't work like how you think at all. You're still thinking it works like Chain of Thought. It doesn't. And the difference is key! It works by introducing probabilistic noise, and exploring N paths fully (each with noise) in parallel (all compressed). It's reasoning at a much, much smaller (probabilistic) level than running everything through the expensive large model (deterministic) and sometimes catching that it said, "I think 1.12 is greater than 1.9 because 12 is bigger than 9, final answer". The easiest way to think about it is: if you understand how hyper words work, it's as if it's searching for different versions of the hyper words that probilisticslly would lead to bett

Recommends MAXPOOL · 2024-07 · on “The Engineer’s Guide to Deep Learning: Understanding the Transformer Model”

There are many others that are better. 1/ The Annotated Transformer Attention is All You Need [LINK] 2/ Transformers from Scratch [LINK] 3/ Andrej Karpathy has really good series of intros: [LINK] Let's build GPT: from scratch, in code, spelled out. [LINK] GPT with Andrej Karpathy: Part 1 [LINK] 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning [LINK] Attention in transformers, visually explained | Chapter 6, Deep Learning [this video] Full 3Blue1Brown Neural Networks playlist [LINK]

Recommends superkuh · 2024-10 · on “LLMs Aren't Thinking, They're Just Counting Votes”

> when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. It's less about understanding and more about pattern recognition. This is not too different from most human person's experience with basic education ( [LINK] ). Though it is obvious that humans can do more than that too. And while "voting" that may be a close-enough metaphor for the multi-layer perceptron layers, it is not what is happening in the attention layers. I suggest checking out 3blue1brown's videos on gradient des

Recommends wren6991 · 2026-06 · on “Can I Buy Your KV Cache?”

If you want the actual maths instead of handwaving, I recommend: [LINK] For something gentler, 3Blue1Brown: [this video] (this is part of a series)

Cites rahimnathwani · 2024-04 · on “Visualizing Attention, a Transformer's Heart [video]”

He lists Ring Attention and half a dozen other techniques, but they're not within the scope of this video: [this video]

Cites peterlai · 2025-06 · on “3Blue1Brown Follow-Up: From Hypothetical Examples to LLM Circuit Visualization”

About a year ago, I watched this 3Blue1Brown LLM tutorial ( [this video] ) on how a model’s self-attention mechanism is used to predict the next token in a sequence, and I was surprised by how little we know about what actually happens when processing the sentence “A fluffy blue creature roamed the verdant forest.” A year later, the field of mechanistic interpretability has seen significant advancements, and we're now able to "decompose" models into interpretable circuits that help explain how LLMs produce predictions. Using the second iteration of an LLM "debugger" I've been working on, I compare the hypothetical representations used in the tutorial to the actual representations I see when extra

Critical lolinder · 2024-05 · on “YouTube seems to once again be rolling out its widely hated new web redesign”

Eh, insightful and thoughtful videos tend to have better comments than the low brow videos, but they're still not useful . The best of them read like a decent-but-not-great subreddit's comment thread. Taking one example, 3Blue1Brown's Deep Learning series, episode 6. Here are the top few comments [0]: > I'm a university lecturer with a PhD in AI, and I cannot compete with the quality of this work. Videos like this put the entire higher education system to shame. Fantastic! This is a nice comment, but doesn't contribute anything to my day. I'm sure the creator is glad to see it and I'm glad someone said it, but I don't need to read it. > Are you kidding me? ONE WEEK FOR 2 MASTERPIECES?! > Tha