Hyperparameter Tuning example with Simple Python
Let’s train a tiny neural net (1 hidden layer) and test different learning rates.
import random
import math
# Generate dummy data: y = 2 * x + 1
X = [i for i in range(20)]
Y = [2 * x + 1 for x in X]
# Hyperparameters
LEARNING_RATES = [0.0001, 0.01, 0.1] # Try different learning rates
EPOCHS = 100
def train_model(learning_rate):
# Initialize weights and bias
w = random.random()
b = random.random()
print(f"\nTraining with learning rate: {learning_rate}")
for epoch in range(EPOCHS):
total_loss = 0
for x, y_true in zip(X, Y):
# Prediction
y_pred = w * x + b
# Compute error
error = y_pred - y_true
total_loss += error ** 2
# Gradients (MSE loss derivatives)
dw = 2 * error * x
db = 2 * error
# Update weights
w -= learning_rate * dw
b -= learning_rate * db
if epoch % 20 == 0 or epoch == EPOCHS - 1:
print(f"Epoch {epoch} - Loss: {total_loss:.4f} - w: {w:.4f} - b: {b:.4f}")
return w, b
# Try different learning rates
for lr in LEARNING_RATES:
train_model(lr)
Observations
- Too small learning rate (0.0001): Takes very long to reduce loss. Might look stuck.
- Too large learning rate (0.1): Loss fluctuates, may overshoot the optimal point.
- Just right (0.01): Gradual and stable convergence.
What Happens Without Tuning?
- Wrong learning rate can lead to under-training or overshooting.
- Too few epochs may not allow the model to learn.
- Incorrect batch size could lead to noisy or slow training.
- Inappropriate network size or activation can limit the model’s capacity.
Impact of Good Hyperparameter Tuning
- Faster convergence and better accuracy
- Prevents overfitting or underfitting
- Reduces training time and improves generalization
- Makes the network more robust to unseen data
How Do We Know Tuning is Needed?
Check these signs:
- High training accuracy but low validation accuracy → overfitting
- Both training and validation accuracy are low → underfitting
- Training loss fluctuates heavily → learning rate too high
- Training loss is reducing too slowly → learning rate too low
- Model does not improve with more data → tuning required
Conclusion
Hyperparameter tuning is like adjusting the oven settings while baking — even the best ingredients won’t help if the process is not right.
A tuned neural network performs faster, better, and smarter, just like Arya’s perfectly baked cake.