Executive summary
A token is a billing unit, not a direct measure of GPU work. Input, cached input and output can carry different rates. As conversations continue, earlier material may be included repeatedly, increasing total token usage. Prompt caching can change the cost of those repeated inputs.
I have been writing about the economics of AI for a while - the projects that refuse to die, the business cases full of hope, and what happens when AI moves from experimentation into the real world. Somewhere along the way, I realized it might be worth going back to one of the basics: tokens. It is one of those terms we use constantly in AI, often with an assumed understanding of what it actually means. So before I get back to the bigger economics, I wanted to spend two short articles on something more fundamental: what a token is, what we are actually paying for when we buy them, and why the bill can behave very differently from what we might intuitively expect.
A token is the meter
A token is essentially the metering unit. It is not a direct measurement of GPU work. Text is broken into tokens - roughly 75 English words per 100 tokens, although this varies by language and tokenizer [OpenAI - Understanding and counting tokens]. The basic transaction looks like this:
ONE MODEL CALL
INPUT OUTPUT
┌───────────────────────┐ ┌──────────────────┐
│ System instructions │ │ Reasoning* │
│ Your prompt │ ── MODEL ──► │ Answer │
│ Conversation history │ │ Tool calls │
│ Retrieved documents │ │ etc. │
│ Tool results │ │ │
└───────────────────────┘ └──────────────────┘
INPUT TOKENS OUTPUT TOKENS
billed at X billed at Y
* depending on modelNow OpenAI, for example, formally distinguishes input, cached input, output and - in reasoning models - reasoning tokens. Reasoning tokens may not appear in the answer you see, but they can count toward output usage.
The generic bill is therefore:
Cost = (Input tokens × Input price) + (Cached input tokens × Cached price) + (Output tokens × Output price)
That is also how OpenAI describes its token-based billing [OpenAI - ChatGPT token-based rate card].
Why input and output cost differently
The surprising part is that input and output are generally not priced similarly. Output is usually considerably more expensive. There is a technical reason. When the model reads your input, it can process much of that input in parallel. This phase is commonly called the prefill.
When it generates an answer, however, generation is autoregressive:
Generate token 1
↓
use token 1 to generate token 2
↓
use tokens 1–2 to generate token 3
↓
use tokens 1–3 to generate token 4
↓
...It cannot simply calculate all 2,000 output tokens simultaneously because token 1 affects token 2, token 2 affects token 3, and so forth. Consequently, output consumes expensive GPU capacity for longer and is much more latency-sensitive. That is why you often see something like:
Input: $X / million tokens Cached input: perhaps $0.1X Output: $4X, $5X, $6X, etc.
For example, as of September 2026, OpenAI's GPT-5.6 Sol pricing illustrates exactly this relationship: $4 per million input tokens, $0.40 for cached input and $20 per million output tokens - a 5:1 output/input ratio [OpenAI - GPT-5.6 Sol model page].
Why earlier context can be billed again
But here is the part corresponding to what you were thinking about. Suppose you have this conversation:
Turn 1 You send: 10,000 tokens I answer: 1,000 tokens
You might naturally think that the next question costs only whatever your new question contains. It often doesn't. On Turn 2, the model may effectively receive:
Previous 10,000 input
Previous 1,000 answer
New 500 question
------------------------
11,500 input tokens
Then perhaps:
1,500 new output tokensAnd then Turn 3:
Everything above + previous response + new question ------------------ perhaps 13,500 input tokens
So the same earlier 10,000 tokens can participate in multiple model calls. That means that over a conversation you can get:
Original document = 10,000 tokens Call 1: 10,000 Call 2: 10,000 again Call 3: 10,000 again Call 4: 10,000 again Call 5: 10,000 again Logical information supplied once Potential token processing: ~50,000
This is probably the phenomenon you were imagining when you said input might be "double counted."
View the illustration at full sizeIt is not generally: "10,000 input tokens × 2 because the model has to process them."
It is closer to: "Those 10,000 tokens became input to five separate inference calls, so they were processed five times."
What caching changes
Prompt caching changes the economics considerably. If that repeated prefix qualifies for caching, those reused tokens can be classified as cached input and charged at a substantially lower rate rather than the normal input price.
So far, this explains the basic economics of the meter: what goes in, what comes out, why they are priced differently, and why the same information can end up being processed more than once. But this is still the relatively simple case. Once we move from a conversation with an AI model to agents that retrieve information, call tools, reason, retry and make multiple model calls, the economics become considerably more interesting. That is where I will pick this up in Part 2.
This is the first of two parts. In the next one I follow the meter past the single call, into agents, retries and the private thinking you never see, and to the only number that actually mattered. Part 2 is here.
Editorial note: The original article prose and text-based examples are presented in source order. The executive summary, section headings, takeaways and illustration are additions for this website edition. Technical prices mentioned in the historical article refer to September 2026 and are not current-rate guidance.
STRATAGEM LABS
