What Does AI Actually Cost — The Structure of Token Pricing
Why AI bills vary wildly between companies, what a token is and how input, output, and context create cost, and where costs blow up as usage grows — explained for small and mid-sized businesses.
They say AI is cheap. Until the day you see the invoice.
The "A Few Dollars a Month" Trap
Ask for AI quotes and the numbers are all over the place. One says thirty dollars a month, another says three thousand. Same technology — why the hundredfold gap?
The answer lies not in the price plan but in how it's used. And the unit of that usage is the token.
In How to Read an AI Project Quote we discussed the traps in quotes. This time we look at the real cost structure underneath them.
A Token Is a Meter
A token is the smallest unit AI handles text in. Think of it as a fragment smaller than a word. In English one token is roughly four characters.
Most AI pricing is charged by this token count. Like a taxi meter, the cost rises with the amount of text exchanged. So the honest answer to "How much is it?" is "How much are you going to use?"
Output Costs More Than Input
There are two kinds of tokens.
Input tokens — the text you feed AI (questions, attached documents) Output tokens — the text AI produces (answers)
Usually output costs several times more than input. Producing something "thought up" costs more than reading. So a task like "write me a long report" can cost more than a hundred short questions.
Real price lists show this gap plainly. When Anthropic launched Claude 3.5 Sonnet in June 2024, it priced the model at $3 per million input tokens and $15 per million output tokens. For the same amount of text, what the AI writes costs five times more.
Why Long Documents Cost More
AI has no memory. Every time you talk to it, the needed context must be fed in again, whole. This is called context. That the context window is wiped each session is documented in Claude Code's official docs, and how to handle it without ballooning cost is covered in our series on recording-based AI workflow.
Attach a 100-page contract and ask a question, and those 100 pages count as input tokens for every question. Ten questions means feeding the contract ten times. That's why "the longer the document, the longer the conversation," the more cost snowballs.
Saving: Caching and Narrowing
Fortunately there are ways to cut it.
Caching — let AI briefly "remember" a document you feed repeatedly, so from the second time it's reused at a discount. For work that keeps using the same material, this lowers cost dramatically.
When Anthropic announced prompt caching, it said the feature can reduce costs by up to 90% and latency by up to 85% for long prompts. Notion said it was adding prompt caching to Notion AI to make it faster and cheaper.
Narrowing — instead of feeding all 100 pages, select only the relevant few. Giving only what's needed is the surest saving.
Which model you call also changes the unit price. When Naver Cloud released HCX-DASH, a lightweight HyperCLOVA X model, in April 2024, it said customers could use it at about one-fifth the price of the existing model (HCX-003) (Korean). Routing simple tasks such as summarization and classification to a lighter model alone can change the bill substantially.
The Real Risk Is "Variable Cost"
The scary thing about AI cost is that it's not fixed but variable. The better it goes, the more you use it; the more you use it, the higher the cost.
What was fifty dollars a month in the pilot can become five thousand a month after company-wide rollout. More users, each using it longer. That's why, as stressed in Starting AI on a Small Budget, it matters to start small and measure the unit cost.
Three Principles for SMBs
1. Measure with one representative task Run the one task you'll use most for a month. Once you have "cost per case," a simple multiplication reveals your budget.
2. Set usage caps Set monthly limits and per-user limits. It's the safety valve that stops cost from quietly running away.
3. Self-hosting only after you've scaled At very high volume, running your own model can be cheaper. But that break-even point is higher than you'd think. For most SMBs, renting is still cheaper.
Cost Can Be Controlled
Token cost is not mysterious. It's just a meter charged by the amount of text.
Understand the structure and you can manage it. What you feed, how much you receive, how often you call — hold just these three and AI cost becomes not a surprise but a predictable number.