CS480/680 Lecture 19: Attention and Transformer Networks

What Hacker News said

Recommends abhgh · 2023-04 · on “Transformers from Scratch”

I would also recommend Pascal Poupart's talk [1] on Attention and Transformers. Doesn't seem to be cited often, but its quite well done and seeing him work out some of the details using chalk and board is very reassuring. [1] [this video]

Recommends elankart · 2023-05 · on “Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper”

This might be dumbing this down very very much but it all boils down to having some sort of a special lookup table and you give it a "query" and a bunch of "keys" and the "value" is the most likely next word. As you input something into the network, a network of these tables are consulted and you are given most likely next word. The novelty in this paper is this "query-key-value" relation that gets learned. A lot of previous work in this area was focused on learning a rough state machine to which you input a set of state transitions and it will give you the most likely next state. This will also work but training such networks is very slow and you also don't have the capability to train the

Recommends metanonsense · 2023-06 · on “Attention is Turing complete [pdf]”

What made attention mechanism click to me was the video [LINK] by Sebastian Raschka. Another good video on attention that tries to give a bit more intuition on the query/key/value stuff is [this video] In hindsight, what helped me most is understanding the dimensionality at each step of the attention computation (explained well in Raschka's video). Another think I simply did not get from the original Transformer paper is that the learning of self-attention happens in the linear layers. (At least, that's my current understanding. If that's wrong, I appreciate any correction :)

Cites godelski · 2023-05 · on “Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper”

> I have zero AI/ML knowledge This may make it difficult to explain and I already see many incorrect explanations here and even more lazy ones (why post the first Google result? You're just adding noise) > Steve Yegge on Medium thinks that the team behind Transformers deserves a Nobel First, Yegge needs to be able to tell me what Attention and Transformers are. More importantly, he needs to tell me who invented them. That actually gets to our important point and why there are so many bad answers here and elsewhere. Because you're both missing a lot of context as well as there being murky definitions. This is also what makes it difficult to ELI5. I'll try, then try to give you resources to ge