Gradient descent is one of the core methods model developers use to pre-train neural networks and gradually reduce their errors. Its job is to find parameter values (weights and biases) at which the model makes the fewest errors on the data.
How does gradient descent work?
Gradient descent updates the network's parameters at every training step. During the process the model computes the error from the gap between the predicted and the true value, which it takes from preset datasets. It then uses derivatives to estimate the direction in which the weights need to be adjusted to reduce that error.
The main stages of gradient descent:
Parameter initialization:
At the start the neural network gets random initial values for its weights and biases. At this stage the network's predictions will be far from the truth, because the model has not been trained yet.
**
Computing the error**:
After the first prediction the neural network compares the result against the sets of correct data it already holds, to see how close its answer is to reality. The gap between the prediction and the correct answer is called the error (loss) function. The bigger the gap — the higher the error, and the model understands that the prediction was not accurate.
**
The gradient and its role**:
The gradient is the derivative of the error function with respect to the model's parameters. It shows the direction in which the weights need to change to reduce the error. Picture it as the slope of a hill: the gradient tells you which way to move to go downhill, toward the point of minimum error (that is, toward the most accurate predictions).
**
Updating the parameters**:
The model's parameters are updated according to the value of the gradient. The speed at which they are updated is set by the learning rate. If the learning rate is too large, the model will jump too far and overshoot the optimal parameter values. If it is too small — training will be slow and can get stuck in local minima.
**
Repeated iterations**:
The process repeats many times, and on each iteration the parameters get closer and closer to the optimal values. The more accurate the model becomes, the smaller its error.
Imagine you are training a model to tell fruit apart in images — apples and bananas, say. In the early stages of training the network may confuse them, because it has not yet learned to pick out the features of each fruit precisely. With gradient descent the model adjusts its parameters to distinguish the characteristic traits of each fruit better: the shape of an apple, the yellow color of a banana, or the differences in peel texture.
After many iterations the model starts to recognize these features more accurately, because its parameters have "tuned themselves" to the distinctive properties of each fruit. So when you feed it an image, the model will be able to say confidently whether what it sees is an apple or a banana, drawing on the features it learned earlier.