Gradient Descent in Neural Network

1. DEFINITION

Gradient Descent is an optimization algorithm used to minimize the error (or loss) in a neural network by adjusting the model’s weights and biases. It works by computing the gradient (partial derivatives) of the loss function with respect to each parameter and then updating the parameters in the opposite direction of the gradient.

PLAIN STORY: “Climbing Down the Hill Blindfolded”

Imagine someone is blindfolded and dropped somewhere on a mountain. His goal is to reach the lowest point (valley) — that’s where the error is minimum.

Every step he takes, he:

  1. Feel the ground (calculate the slope/gradient).
  2. Step slightly downhill (move opposite the slope).
  3. Repeat until he can’t go any lower — that’s his minimum error.

This process is Gradient Descent. The steepness he feels is the gradient of the loss, and his movement is updating the weights.

Gradient Descent in Neural Network – Gradient Descent example with Simple Python