Decision Guide Technology
Token prices are only part of the bill. Model the full monthly cost of an AI feature, including retries, monitoring and human review, and measure cost per successful task.
The short answer: when you add up the real AI costs of a typical feature built on a hosted model API, the model calls are often not the largest item. A realistic monthly budget also includes retrieval and data storage, application hosting, monitoring and evaluation, failed or repeated requests, and the people who review outputs or handle cases the AI cannot. The most useful number to track is not cost per token or cost per request, but cost per successful task: total running cost divided by the number of tasks the system completes to an acceptable standard.
This guide breaks down each cost component, works through a hypothetical example with current published prices, and shows which levers reduce cost without degrading quality. It covers ongoing running costs only. If you are budgeting to build a first version, see our separate guide to MVP development cost.
| Component | What drives it | Typical billing unit |
|---|---|---|
| Model inference | Requests, input and output length, model choice, number of calls per task | Per million tokens (hosted APIs) or per GPU hour (self-hosted) |
| Retrieval and data | Document volume, embedding updates, query volume | Per GB stored, per query or per instance |
| Application hosting | Traffic, latency targets, background jobs | Per instance hour, per request or per GB transferred |
| Monitoring and evaluation | Logged requests, trace retention, test suite size | Per seat, per event or per GB ingested; model calls for evaluations |
| Retries and failures | Timeouts, invalid outputs, rate limits, guardrail rejections | Extra inference and compute |
| Human review and escalation | Sampling rate for quality checks, share of tasks escalated | Staff hours |
Hosted model APIs usually charge separately for input tokens (what you send: instructions, retrieved documents, conversation history and the user's message) and output tokens (what the model writes). Output tokens typically cost several times more than input tokens.
Prices differ widely between models from the same provider. On October 27, 2026, Anthropic's published API pricing listed Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output tokens, and Claude Haiku 5.5 at $0.10 and $0.50 for prompts up to 100,000 tokens, a twentyfold difference. Prices also change: Google's Gemini API pricing page showed scheduled increases for some models from December 2026 and January 2027. Re-check provider prices every quarter, and treat any figure in this article as a dated example.
Two pricing features matter for budgeting:
Applications that answer questions from your own documents, often called retrieval-augmented generation (RAG), need somewhere to store and search that content: a vector database, a search service or a database extension. Costs depend on how much you store, how often you re-index changed documents and how many searches you run. Embedding models that convert documents into searchable vectors carry their own per-token charges, mostly when content is first indexed or updated.
The model is only one part of the application. You still run an API layer, authentication, queues for background jobs, a database for conversation history and logs, and possibly a front end. For most SaaS products this adds a modest amount to existing cloud spend, but latency targets and traffic peaks can change that. Our guide to SaaS cloud cost optimization covers how to attribute these costs alongside the rest of your platform.
AI features need monitoring beyond normal uptime checks. You need to see what was asked, what context was retrieved, what the model returned, how long it took and whether the answer was acceptable. That usually means request tracing, cost tracking per feature or customer, alerting on error and latency spikes, and an evaluation set of example tasks that you re-run whenever you change prompts, models or retrieval. Evaluation runs consume model tokens too.
Requests fail for ordinary reasons: provider timeouts, rate limits, outputs that fail format validation, or guardrails that reject an answer. Each retry costs another model call. Agent-style workflows, where the model makes several calls to plan, use tools and check its work, multiply both cost and failure points.
Most business AI applications keep people involved, either reviewing a sample of outputs for quality or handling cases the system escalates. Those hours are a real running cost and, as the example below shows, are frequently the largest one.
The scenario and assumptions below are invented for illustration. Model prices are Anthropic's published list prices on October 27, 2026; all other costs are assumptions, not quotes or Growix pricing.
The workload: a SaaS product adds an assistant that answers customer questions from its help documentation.
| Item | Calculation | Monthly cost |
|---|---|---|
| Input tokens | 300 million tokens × $2 per million | $600 |
| Output tokens | 40 million tokens × $10 per million | $400 |
| Retries | 8% of model cost (assumed) | $80 |
| Retrieval and vector storage | Assumed | $300 |
| Application hosting | Assumed | $400 |
| Monitoring and tracing | Assumed | $250 |
| Evaluation runs | Assumed model and tooling cost | $100 |
| Human quality review | 1,000 sampled answers × 3 minutes × $40 per hour (assumed loaded rate) | $2,000 |
| Total | $4,130 |
Cost per successful task: $4,130 ÷ 85,000 successful answers = about $0.049, or just under 5 cents.
Model calls account for about $1,080 of the $4,130, roughly a quarter. The quality-review sample alone costs nearly twice as much. Note also that the 15,000 escalated questions still create support work; that cost belongs in your customer support budget, and it is why resolution rate matters as much as token price.
Suppose the same assistant is rebuilt as an agent that makes five model calls per question to plan, search, draft and check its answer, with an average of 6,000 input tokens and 400 output tokens per call as the conversation context grows.
Model cost rises from $1,000 to $8,000 a month, an eightfold increase, before retries. That may be worth paying if the agent resolves substantially more questions, but it has to be proven with evaluation data. Our guide to AI agents vs workflow automation covers when that extra autonomy is justified.
Now suppose testing shows that 60% of questions are simple enough for the smaller model, while the rest stay on the mid-tier model.
Model cost falls from $1,000 to $430. With retries at the same 8%, the monthly total falls to about $3,514 and cost per successful task to about 4.1 cents, provided the resolution rate holds. If the small model lowers the resolution rate, the saving can disappear in escalations.
| Factor | Hosted model API | Self-hosted open-weight model |
|---|---|---|
| Cost shape | Pay per token; scales with usage | Pay for GPU capacity whether used or not |
| Fits best when | Usage is variable or still growing | Volume is high and steady, or data must stay in your environment |
| Operational effort | Low; provider runs the model | High; you manage scaling, updates and availability |
| Model choice | Provider's catalog | Open-weight models you can run and tune |
| Key risk | Price changes and provider outages | Idle capacity and operations workload |
Self-hosting can lower the cost per token at high, predictable volume, but it adds capacity planning, GPU availability and on-call responsibilities. Most businesses should start on hosted APIs, measure real usage for a few months, and revisit the decision with data.
It depends on volume, model choice and how much human review the task needs. In this guide's hypothetical example, an assistant answering 100,000 questions a month costs about $4,130 a month, or roughly 5 cents per successful answer, and model calls are only about a quarter of that total.
Agents make several model calls per task to plan, use tools and check their work, and the context grows with each step. In the example above, moving to a five-call agent design raises model cost about eightfold before retries, so it only pays if the agent resolves noticeably more requests.
Measure cost and outcome per feature first, then trim retrieved context, cache repeated instructions, batch work that can wait, route simple requests to smaller models and validate outputs to cut retries. Check that resolution rates hold after each change, because extra escalations can erase the saving.
It can be at high, steady volume, but it adds capacity planning, GPU availability and on-call work. Most businesses should start on hosted APIs, measure real usage for a few months and revisit the decision with data.
Before launch, write down the cost per successful task you can afford, based on what the feature is worth: support cost avoided, revenue from a premium tier or time saved. Then build the model above with your own numbers, run a pilot to replace the assumptions with measurements, and set alerts if cost per task or failure rate drifts.
Growix's Cloud Infrastructure & Architecture team designs the hosting, scaling and cost governance around AI workloads, and our Infrastructure Reliability & Monitoring service sets up the monitoring, alerting and incident response that keep them stable. If you are still designing the feature itself, our guide to adding AI features to a SaaS product covers the build side.
Discuss your IT and infrastructure needs
Four weeks before Black Friday, the work is testing and ownership, not new features. A checklist for checkout testing, dependencies, alerting and incident response.
Lower the cost of serving each customer, not just the monthly bill. A four-step method for SaaS teams on AWS, Azure or Google Cloud, with a worked example.
Migration fixes platform limits; optimization fixes waste and performance. Diagnose which constraint you have, then sequence the work so you never migrate what you should retire.
