Tokens
Wait, AI Only Predicts Tokens?


When an AI answers you, it isn't writing a sentence. It's guessing one small piece of text, then the next, then the next. Not the next word, the next token. One guess at a time, it writes your emails, your code, your stories. So let's open the box: what exactly is a token?
So what is a token?
You might picture an AI reading whole words, the way you do. It doesn't. Before it answers, your text is cut into small pieces called tokens.
A token is a chunk of text. Sometimes it's a whole word, like "magic". Sometimes it's only part of one: "snowboarding" becomes "snow" and "boarding". Even a comma or a period can be a token on its own.
Why those cuts? The tokenizer, the tool that does the splitting, learned its pieces from a huge amount of text. Common words usually stay whole, and longer or rarer ones get broken into familiar parts. A rough rule from OpenAI: one token is about 3/4 of an English word. Each model family has its own tokenizer, so counts vary. The graphic below uses GPT-4o's.
- Please
- res
- chedule
- my
- dentist
- appointment
- to
- Friday
- .
Seven words become nine tokens. "reschedule" is two pieces, "res" and "chedule", and the period gets its own chip. The faint dot marks a space at the start of a token.
From pieces to numbers
Computers work with numbers, not words. So every token gets swapped for a number, like a page number in a giant dictionary: in our sentence, " Friday" is 9377. From here on, the model only sees numbers, and at the end the app turns them back into words.
We won't get too deep on that, but it's good to know what happens under the hood.
One guess at a time
To write a reply, the model looks at every token so far and scores every possible next one. It picks one, adds it to the end, and repeats.
The whole trip for another sentence: text becomes tokens, tokens become numbers, and the model guesses what comes next.
A short reply takes a few dozen of these steps, a long one hundreds or thousands. The model isn't planning the whole sentence at once. It guesses the next token, then the next, until it hits a stop token or a length limit.
Why the AI rereads everything
The AI doesn't remember your chat. Every time you send a message, it starts with a blank mind.
So how does it seem to remember? The app sends it the whole conversation again, every single time. Your new message plus everything said so far.
Picture this short chat:
- You: "Hi, my name is Ana."
- AI: "Nice to meet you, Ana!"
- You: "What's my name?"
When you ask "What's my name?", the AI doesn't recall Ana. It gets all three lines again, reads them from the top, and finds the answer there. If the app sent only your last line, the AI would have no idea who you are.
The AI doesn't remember. The app hands it the whole chat every time. (Counts are for the message text only; real apps also send hidden instructions, so the true numbers are higher.)
So every message makes the next one bigger. In a short chat, that's no big deal. But in a long chat, or after you paste in a long article, the AI rereads all of it on every turn, and every reread uses tokens.
The same thing happens when the AI searches the web for you. The pages it finds are added to the chat as text, and they get reread on every turn after, even once you've moved on.
But some AI apps take notes
Rereading everything isn't the only way to remember. Apps and tools like Claude Code, ChatGPT Codex or Grok Bot (check our post about Grok Bot) wrap the model with extra tools, and one of those tools is memory. You may also see this called long-term memory or agent memory. Grok Bot works like this: each bot keeps its own memory, a short list of notes. When something important comes up, the bot writes it down.
The app hands the model the short list of notes along with your new message. The notes are still text, so they still cost tokens, but a handful of lines costs far less than a whole history. Long chats get the same treatment: older parts are boiled down to a summary.
The model itself still remembers nothing. The memory lives in the app, which picks the few lines worth sending.
What a "usage pool" is
For a long time, AI plans were sold by the message: a set number of chats every few hours, then you wait. More and more plans now give you a budget of tokens instead, usually one that refills every week. That budget is your usage pool.
Most companies don't say exactly how many tokens are in the pool.
Teams often skip the pool and pay on demand. Say five developers share one account: every token they use is counted, and the company gets one bill at the end of the month. On demand, you pay a price per million tokens, and that price depends on the model:
API list prices per million tokens, standard tier, checked Nov 16, 2026. Writing costs more than reading: output tokens cost 2 to 5 times more than input.
Pool or bill, the math is the same. A quick question uses a few tokens. A long chat, a coding task or a web search uses far more.
The pool is often shared across a company's apps. On Grok's paid plans, chat, images, voice, Grok Bot and Cursor all draw from one weekly pool (x.ai pricing). Claude does the same for its chat and Claude Code (Claude help), and OpenAI says ChatGPT and Codex will share one token-based allowance starting in December (OpenAI help).
So two people on the same plan can run out at very different speeds. What matters isn't how many messages you send. It's how much text the AI has to read and write for you.
How does AI sound so smart if it only guesses the next token?
An AI only knows what it learned up to the day its training ended. So for anything newer, it uses tools, like a web search. It still only writes tokens, but instead of guessing, it can write a request: "search the web for this."
Say you ask: "Are Guns N' Roses playing in Rio this year?"
- The AI writes a search request instead of guessing.
- The app runs the search and adds the results to the chat as text.
- The AI reads them and answers: "No. Their Rio show on April 10 was canceled."
The AI asks, the app searches, the results become more text to read.
Everything the AI reads before it answers (your messages, the chat so far and any search results) is called its context. The search didn't make the model smarter. It put fresh facts into the context, and the model answered from them. Without the search, it might have said yes, because the show was announced months earlier.
What we learned
- AI reads and writes in tokens, small pieces of text, and each token becomes a number.
- It writes an answer one token at a time, always guessing the next one.
- It doesn't remember your chat. The app sends the whole conversation again on every turn, so long chats use more tokens.
- Some apps add memory: a short list of notes the app sends instead of the whole history.
- More and more plans count tokens, not messages, and writing costs more than reading.
- Tools like web search add fresh facts to the context, so the AI can answer with up-to-date information.



