Demystifying How AI Writes Text: Understanding the Next-Token Prediction Mechanism

If you have ever used artificial intelligence and entered the exact same prompt twice, only to get two different answers, that is not a system glitch—it is a core mechanism designed from the ground up. Large Language Models (LLMs) never process an entire sentence all at once. Instead, they repeatedly answer a single question over and over: given all the text up to this point, what should the next part be?

For example, when a writing assistant like DraftPal is filling in the blank in the sentence "The cat sat on the ___", the system does not know how the sentence will end. It calculates the probability of every single word it knows. For instance, the word "mat" might have a probability of 41%, "chair" at 19%, and "floor" at 12%, and so on.

A crucial detail that explains strange LLM behaviors is that the model does not always pick "mat" even though it has the highest score. Instead, it rolls a weighted die based on all probability proportions, meaning that sometimes "chair" might win instead. This exact dice roll is what creates response variety, even when using the exact same model and prompt.

Once the winning word is chosen, it is appended to the end of the existing text. Then, the original question is processed all over again from the beginning, with its length increased by one token. This process is called Next-Token Prediction. When repeated hundreds of times, it builds up into a miniature generated response.

Even a tiny context change can unexpectedly alter the entire output. If you add just four words of context before the original sentence, such as "write this like a horror story", the probability calculations shift drastically. The proportion for the word "mat" drops to under 1%, while the word "coffin" surges up to 99%, even though the model itself hasn't changed at all—only the input data context has.

This mechanism also helps explain two phenomena that developers frequently encounter:

• The balance between determinism and diversity: Forcing the model to always pick the highest-scoring word (Greedy Decoding or Temperature 0) results in identical answers every time, making it ideal for structured data retrieval. However, turning on random sampling introduces linguistic diversity.
• Per-token processing cost: Every single token must go through a full forward pass through the model, making a 500-token response significantly more costly and time-consuming to process than a 1-token response—it is not just a minor linear increase.

Additionally, AI hallucinations or providing incorrect information stem from this exact mechanism. Once a token is appended to the context, it is never retroactively corrected. The model cannot turn back to review the 40th token after generating the 41st token. Therefore, if an error is guessed early on, that mistake is carried forward as the foundation for the next rounds of generation without any fixes.

In system operation dashboards, we frequently see two metrics appear: TTFT (Time-to-First-Token) and TPOT (Time-Per-Output-Token), which are used to measure this exact working loop. This is also why batching techniques exist to group next-token requests from multiple users to process them simultaneously in a single round, because the performance bottleneck lies in loading model weights into GPU processing units rather than the mathematical calculations themselves.

Source: Dev.to