In few-shot learning the order in which you present examples really does matter, because it can affect how the model behaves. For instance, if the model sees several positive examples in a row, that can prime it to keep generating positive answers even when the context turns negative. This effect can be called context bias. It leads the model to answer in line with the examples it was given earlier, ignoring the actual meaning of the request.
An experiment with changing the order of examples
The original order:
-
Q: I just got the best news of my life! A: Positive
-
Q: I just got some terrible news. A: Negative
-
Q: The weather outside is so gloomy. A: Negative
-
Q: I am so proud of what I achieved today. A: Positive
**The changed order:**
- Q: I just got the best news of my life! A: Positive
- Q: I am so proud of what I achieved today. A: Positive
- Q: The weather outside is so gloomy. A: Negative
- Q: I just got some terrible news. A: Negative
As a result of the change in order, the model may behave more predictably and lean toward giving positive answers when the positive examples come first.
How to avoid the bias
-
Random order. Present the examples in a random sequence to avoid skewing toward one label.
-
An equal number of positive and negative examples. Make sure the model sees the same number of examples with each label, so it does not get used to a dominant answer type.
-
Different contexts. Give the model varied examples covering different topics and contexts, so it learns to adapt better to specific situations.
Experimenting with the order in which you feed data and with how varied it is will help improve results and reduce the risk of bias in how the model works.