[1hr Talk] Intro to Large Language Models
Why Hacker News recommends it
HN recommends it as the dense, approachable starting point for understanding LLMs and often cites it for specific claims about scaling and training.
What it is
Andrej Karpathy gives a one-hour general-audience introduction to large language models, covering what they are, how they are trained, comparisons to operating systems, and security challenges such as jailbreaks.
Recommended by 9 HN commenters across 6 different months, 2023 to 2026.
What Hacker News said
Strong rec shrimpx · 2023-11 · on “Generative AI for Beginners”
Andrej Karpathy's "Zero to Hero" series on YouTube is the ultimate guide to building LLMs. Extremely information-dense but as complete as it gets: [LINK] Also, an amazing high-level overview of LLMs, including extensive discussion about attack vectors, that he published a couple days ago: [this video]
Strong rec FlyingAvatar · 2024-02 · on “Ask HN: Do LLMs get "better" with more processing power and or time per request?”
Short answer is No. I highly recommended watching Andrej Karpathy's Intro to LLMs talk, particularly the section on System 1 vs System 2 thinking. Long story short, what you are describing, using more processing to prepare a better response, is something that is an area of interest, but is not currently part of ChatGPT (or any other LLM that I am aware of). See: [this video]
Strong rec mindwok · 2024-06 · on “Why has LLM progress seemingly stalled around the GPT-4 level? Or has it?”
All of Andrej Karpathy's stuff on YouTube is absolutely top tier. He's a rare example of a brilliant contributor in his field who is an equally brilliant educator. Start with his short intro to LLMs, only 1 hour long but is dense with valuable stuff: [this video]
Recommends jpdus · 2023-12 · on “Ask HN: AI/ML papers to catch up with current state of AI?”
Hey, imho best overall technical intro to LLMs (I guess that´s your main interest as you mentioned qlora + llama) is by Simon Willis [1]. Additionally or if you prefer videos, the recent 1h "busy persons intro" by Andrei Karpathy is great + dense as well [2]. [1] [LINK] [2] [this video] EDIT: Maybe I misunderstood as you asked about papers, not general intros. I don´t think that reading papers is the best way to "catch up" as the pace is rapid and knowledge very decentralized. I can confirm what Andrej recently wrote on X [3]: "Unknown to many people, a growing amount of alpha is now outside of Arxiv, sources include but are not limited to: - [LINK] - HN - that niche Discord server - anime profil
Recommends kozikow · 2023-12 · on “Ask HN: AI/ML papers to catch up with current state of AI?”
My view is to focus on doing stuff. That's what I did. Pick up some task you want the model to do, try finetuning llama, playing with APIs from OpenAI, etc. Googling and asking GPT along the way. Foundational model training got so expensive that unless you can get hired by "owns nuclear power plant of GPUs" you are not going to get any "research" done. And as the area got white-hot those companies have more available talent than hardware nowadays. So just getting into the practitioner area is the best way to get productive with those models. And you improve as a practitioner by practicing, not by reading papers. If you're at the computer, your time is best spent writing code and interacting
Recommends sk11001 · 2024-04 · on “Ask HN: Why has ChatGPT a bias towards correctness?”
Look into the training process - [this video] The goal of the initial pretraining phase is to make it good at predicting the next word. The rest of the training process is aimed at making it (1) helpful and (2) as correct as possible. I think some people oversimplify things by calling LLMs "next token predictors" and they leave out the tuning towards helpfulness and correctness.
Recommends runjake · 2024-04 · on “Ask HN: 16 yo Nephew, in E. Africa, wants to train an LLM with on disk Wikipedia”
In addition to the other great suggestions, point him to Karpathy's YouTube channel[1]. Karpathy has an approachable communication style. Here's his "1 hour intro to LLMs" video: [this video] 1. [LINK]
Recommends tikkun · 2024-04 · on “Ask HN: AI Training for Executive Level”
I suspect these videos would be higher quality than most offerings from universities or business schools (given that it's a fast moving field, and most of the true experts are practitioners, and so if they're teaching, they're not teaching actual classes, at most they're doing youtube videos or interviews): - [LINK] (Jeremy Howard Hacker's Guide to LLMs) - [this video] (Karpathy Intro to LLMs)