Chapter 2 · 5 min
Tokens: how AI breaks up text
One small word explains why AI can struggle with surprisingly simple tasks.
The last lesson ended with a secret: AI predicts pieces of words, not whole words. Let's look at those pieces.
Chapter 2 · 5 min
One small word explains why AI can struggle with surprisingly simple tasks.
The last lesson ended with a secret: AI predicts pieces of words, not whole words. Let's look at those pieces.