Basic Math Concepts – Gradient Descent in Neural Network

BASIC MATH KNOWLEDGE REQUIRED

To understand gradient descent, you need:

  • Functions: Like f(x) = x² (Loss Function)
  • Derivatives: f'(x) = 2x gives the slope (rate of change)
  • Partial Derivatives: If there are multiple weights (e.g., w₁, w₂), we compute slope w.r.t. each.
  • Learning Rate (η): A small number that controls how big each step is.

GRADIENT DESCENT FORMULA

if:

  • w is a weight,
  • L(w) is the loss function,
  • ∂L/∂w is the derivative (gradient) of the loss w.r.t. the weight,

Next – Mean Square in Neural Network