Neural network parameters are adjustable values that determine how a model learns and makes predictions. During training, the network tunes these parameters so it can better "understand" the input data and solve the task at hand. Parameters govern how information travels between layers of neurons and how the data is processed.
They play a key role in how accurately a network can handle tasks such as image recognition or text processing. Put simply, the more of them there are, the better, faster and in greater volume a model can process information. For example, the most advanced version of LLaMA 3.1 has 405 billion parameters, while ChatGPT 4o has 1.8 trillion.
The core parameters of a neural network are weights and biases.
Weights and biases
Weights determine how important the connection between neurons in different layers is. When a neuron passes a signal on, the weight multiplies that signal, which affects how strong the transmitted signal is. A high weight means the connection matters more to the model.
**
Biases** are added to the sum of the input signals to help the model cope with more complex patterns and to correct its predictions, letting the network work with a wider range of data.
Hyperparameters
Beyond weights and biases there are hyperparameters, which are not learned directly but set before training begins. They affect the network's performance, its speed and its training accuracy.
These include, for example, the learning rate, which determines how quickly the network changes its weights at each training step.
Another important hyperparameter is batch size, which specifies how many data examples are processed at once. It ranges from 16 to several thousand items. Note that, unlike a token, an item here is a single unit of data — an image, for instance.
Also important are parameters such as the number of epochs (training cycles), which can run from 10 to several hundred or even thousands depending on the difficulty of the task and the volume of data.
On top of that there is the number of layers, which can range from 3 (input, hidden, output) to several dozen depending on how complex a particular model's architecture is. For example, some unofficial sources claim that the ChatGPT 4 architecture may have as many as 128 layers.