Let's build the GPT Tokenizer
Why Hacker News recommends it
Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.
What it is
Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.
Recommended by 5 HN commenters across 5 different months, 2024 to 2025.
What Hacker News said
Strong rec cranium · 2024-10 · on “Probably pay attention to tokenizers”
I finally understood the weirdness of tokenizers after watching the video Andrej Karpathy made: "Let's build the GPT Tokenizer" ( [this video] ). He goes through why we need them instead of raw byte sequences (too expensive) and how the Byte Pair Encoding algorithm works. Worth spending 2h for the deeper understanding if you deal with LLMs.
Recommends moozilla · 2024-03 · on “Ask HN: Did you encounter any leap year bugs today?”
ChatGPT is specifically bad at these kinds of tasks because of tokenization. If you plug your query into [LINK] , you can see that"egregious" is a single token, so the LLM doesn't actually see any "e" characters -- to answer your question it would have had to learn a fact about how the word was spelled from it's training data, and I imagine texts explicitly talking about how words are spelled are not very common. Good explanation here if this still doesn't make sense: [LINK] , or check out Andrej Karpathy's latest video if you have 2 hours for a deep dive: [this video] IMO questions about spelling or number sense are pretty tired as gotchas, because they are all basically just artifacts of this i
Recommends Chirono · 2024-08 · on “Artificial intelligence is losing hype”
This is just an artefact of tokenisation though. The model simply isn’t ever shown the letters that make up words, unless they are spelled out explicitly. It sees tokens representing groups of words. This is a little like saying a human isn’t intelligent because they couldn’t answer your question that you asked in an ultrasonic wavelength. If you’d like to learn more this video is a great resource: [this video]
Recommends zellyn · 2025-04 · on “Why Does Claude Speak Byzantine Music Notation?”
Andrej Karpathy goes into quite some depth about how tokenization works here: [this video] tl;dr many of the LLMs use byte-pair encoding to create tokens. You take a set of documents, and then form tokens by repeatedly merging the most common pair of tokens. The initial set of tokens is 256 raw bytes. And the text is typically represented in utf-8. I expect that although the LLMs can understand arbitrarily but cleanly offset unicode code points by (eventually) noticing the final byte of each sequence, they would do markedly worse on actually processing and completing on them, because they will not have been reduced to the normal set of tokens. However, if the text is actually output cleanly conve
Recommends vismit2000 · 2025-09 · on “UTF-8 is a brilliant design”
Karpathy's "Let's build the GPT Tokenizer" also contains a good introduction to Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32 in the first 20 minutes: [this video]
Cites anonymouse008 · 2024-03 · on “Claude 3 model family”
I'm convinced GPT is running separate helper functions on input and output tokens to fix the 'tokenization' issues. As in, find items of math, send it to this hand made parser and function, then insert result into output tokens. There's no other way to fix the token issue. For reference, Let's build the GPT Tokenizer [this video]
Cites dist-epoch · 2024-03 · on “Chronos: Learning the Language of Time Series”
Tokenisation turns a continuous signal into a normalized discrete vocabulary: stock "went up a lot", "went up a little", "stayed flat". This smooths out noise and simplifies matching up similar but not identical signals. > We tokenize text because text isn't numbers. Text is actually numbers. People tried inputting UTF8 directly into transformers, but it doesn't work that well. Karpathy explains why: [this video]
Cites pquki4 · 2024-06 · on “Notebooks Are McDonalds of Code”
If there are complaints about code quality in a jupyter notebook, chances are that someone is doing it wrong -- either the author is using it for demonstration purpose, for production or near production environment, or the person looking at others' notebooks is taking them too seriously. Most code in jupyter notebook is bad, but that's fine, because nobody should waste time doing that unless they must reproduce the results or they are sharing the notebook with others. Notable example of high quality notebook: [this video]