What is a token?
A token is the smallest unit of text that can pass through the layers of the architecture. A token can be a word, part of a word, or even a single character, depending on the model and the tokenization method in use. In language models such as GPT, text is first split into tokens, which are then fed into the model for processing.
Tokens are the foundation of how neural networks work, because they break text into more manageable pieces. They also matter for processing long texts efficiently: the more context a model can take in at once (that is, the more tokens it can process), the better its results.
Example:
In the sentence "Neural networks are changing the world" the tokens can be the individual words: "Neural", "networks", "are", "changing", "the", "world", or parts of those words.
How does tokenization work?
To break your request into pieces, the model runs a tokenization process. It is an important stage in neural networks, because models work better with data in the form of numeric sequences. Each token is converted into a numeric code, and from those codes the model learns to recognize patterns and produce answers.
Tokenization also helps with languages in which a single word can carry several meanings depending on context. The more precise the tokenization, the better the model can "understand" the text. There are different tokenization methods that split text into tokens in different ways depending on the task and the type of text.
The main tokenization methods
Word tokenization: each token is a separate word.
For example, the phrase "I study machine learning" would be split into ["I", "study", "machine", "learning"].
Subword tokenization: text is split into smaller pieces.
For example, "learning" can be split into ["learn", "ing"], which is useful when working with rare words.
Character tokenization: each character becomes its own token.
For example, the word "learning" would be split as ["l", "e", "a", "r", "n", "i", "n", "g"].
N-gram tokenization: text is split into groups of several words or characters.
For example, "machine learning" can be split into ["machine learning"] or ["mach", "ine", "learning"] depending on the settings.
Tokenization with BPE (Byte-Pair Encoding): a widely used method that first splits text into characters and then merges frequently occurring pairs of characters or subwords to shorten the text.
Say we have the sentence: "machine learning helps".
**
1. The first step** — the text is split into individual characters, since we start at the most basic level. We get the following tokens:
m, a, c, h, i, n, e, , l, e, a, r, n, i, n, g, , h, e, l, p, s
**
2. Merging frequently occurring character pairs** — at this stage we start looking for frequently occurring pairs of characters and merging them:
The most frequent character pair might be, say, "i" and "n", which merges into "in"
Next the pair "e" and "a" might be merged to give the token "ea"
Step by step the sentence starts grouping itself into subwords.
m, a, ch, in, e, , l, ea, r, n, in, g, , h, e, l, p, s
3. The next steps — merging of frequent pairs continues until larger subwords have formed. The result can look like this:
mach, ine, learning, helps
BPE therefore encodes text efficiently by splitting it into subwords, which helps shrink the vocabulary while preserving information about words and their parts — something that matters for neural networks.
How many tokens can models process?
The number of tokens a model can process depends on its context window, also known as the input window. For GPT-4o, for instance, it is 128,000 tokens. We cover this in more detail in the article "Transformers" in the section Architectures behind the models in GPTunneL.