Learning Rate
Q: What is the learning rate?
It’s a hyperparameter that defines how big the weight updates are during training. Think of it as the "step size" in gradient descent.
Q: Why does it matter?
Because it directly controls convergence:
- Too high → the model oscillates and never stabilizes.
- Too low → the model converges painfully slowly or gets stuck in suboptimal solutions.
Q: How do I pick the right learning rate?
There’s no universal rule. Typical values range between 0.001 and 0.1, depending on the optimizer and the problem. Researchers often use learning rate finders (e.g., in PyTorch or FastAI) to empirically determine a good starting point.
Q: Are there advanced techniques?
Yes:
- Learning rate schedules (step decay, cosine annealing).
- Adaptive methods (Adam, AdaGrad, RMSProp) which adjust learning rates per parameter automatically.
- Warm restarts to escape poor local minima.
Q: Real-world example?
In image recognition tasks with CNNs, starting with a learning rate of 0.01 and decaying it after a few epochs often balances speed and accuracy.
The learning rate is often called the most important hyperparameter because it dictates how the model “learns” from mistakes. A well-chosen rate ensures steady progress toward minimizing loss, while a poor choice can completely derail training.
One useful intuition is to imagine hiking down a mountain blindfolded: if your steps are too big, you risk overshooting the valley; if they’re too small, you’ll take forever to arrive. This is exactly the balance deep learning practitioners face.
In practice, tuning the learning rate is rarely done in isolation. It interacts with other factors like batch size, optimizer type, and weight initialization. Modern deep learning frameworks also provide visualization tools (e.g., loss vs. learning rate curves) that help spot the “sweet spot” before full training begins. These diagnostics save time and prevent wasted compute.
📖 References
- Stanford CS231n: Optimization notes.
