Building makemore Part 3: Activations & Gradients, BatchNorm
Why Stack Overflow recommends it
Stack Overflow users link it when answering PyTorch questions about weight initialization, loss spikes in the first epoch, and implementing BatchNorm by hand, since it explains the reasoning and shows working code.
Recommended by 3 people across 3 different months, 2023 to 2025.
What it is
Part 3 of Andrej Karpathy's makemore series, a nearly two-hour live-coded lecture on the internals of multi-layer MLPs. It examines forward activation and backward gradient statistics, weight initialization (Kaiming init), diagnostic plots, and then builds up Batch Normalization from scratch in PyTorch.
What people said
Recommends Syla · 2023-02 · Stack Overflow · on “What causes neural network accuracy to sharply increase after only one epoch?”
As said by Murilo, hard to say much without more information but it can come from multiple things: Your network learns through the batches of each epoch, meaning that your ~12 batches (6500/512) are already enough to learn a good bit of classification. Your weights are not really well initialized, and produce a huge loss for the first epoch. The massive decrease in the loss is actually the solver 'squishing' the weights. The best explanation I found for this comes from A. Karpathy in his 'MakeMore' tutorial: [this video] Now this sudden decrease of the loss is not extreme here (from 0.5 to 0.2) so I would not care much. I agree with Murilo that low accuracy in validation can come from too few sam
Recommends Mircea · 2023-07 · Stack Overflow · on “How to implement Batchnorm2d in Pytorch myself?”
The video [this video] from Andrej Karpathy has a very intuitive explanation. Here is a code snippet with the 1D implementation, from the notebook [LINK] associated with the video: class BatchNorm1d: def __init__(self, dim, eps=1e-5, momentum=0.1): self.eps = eps self.momentum = momentum self.training = True # parameters (trained with backprop) self.gamma = torch.ones(dim) self.beta = torch.zeros(dim) # buffers (trained with a running 'momentum update') self.running_mean = torch.zeros(dim) self.running_var = torch.ones(dim) def __call__(self, x): # calculate the forward pass if self.training: xmean = x.mean(0, keepdim=True) # batch mean xvar = x.var(0, keepdim=True) # batch variance else: xmean =
Recommends dmitry_romanov · 2025-07 · Stack Overflow · on “How to decide which mode to use for 'kaiming_normal' initialization”
Andrej Karpathy discusses it here: [this video] . The point is to keep standard deviation around 1 during the forward pass, and not to kill the (ReLU) neuron accidentally. You can take a look on the whole discussion starting a bit earlier in the video: [this video]