The short answer
Inference is the stage where a trained AI model is run on new input to produce an output, such as answering your prompt, writing code or generating an image. NVIDIA defines it as the process where a trained model generates new outputs by reasoning and making predictions on new data. Every prompt you send is an inference request, which is why longer thinking, higher effort settings and images use up plan limits faster.
| Plain meaning | Running a finished AI model to get an answer |
|---|---|
| Versus training | Training teaches the model once; inference happens every time anyone uses it |
| Speed measures | Time to first token (TTFT), time per output token (TPOT), throughput |
| What raises the cost | Longer outputs, reasoning or higher effort, images, video and audio |
| Where you feel it | Waiting time, usage limits and per-token API prices |
What inference means, in plain English
An AI model has two lives. First it is trained: it learns from huge amounts of data by adjusting billions of internal settings called weights. That happens once per model version, at great expense. Then it is used, over and over, by millions of people. Each use is called inference.
NVIDIA's glossary puts it simply: training is the process where a model learns to perform a task, and inference is the process where a trained model generates new outputs from new data. Training is like writing and printing a textbook; inference is every time someone opens it to look something up. When you type a question into ChatGPT, Claude or Gemini, the model is not learning from you in that moment. It is running inference.
How inference works, and how it is measured
A language model writes its answer one token at a time. It reads your prompt, predicts the next token, adds it, and repeats, until the answer is finished. Every token costs a little computing work on a chip such as a GPU. That is why NVIDIA lists these measures:
- Time to first token (TTFT): how long before the first word appears. This is what makes a tool feel quick or sluggish.
- Time per output token (TPOT): the average time for each following token, which sets how fast the answer streams.
- Throughput: how many tokens a system processes in a given time, which drives the cost of serving everyone.
Real-time products push these hard. ElevenLabs says its Eleven v4 Turbo voice model has median inference latency of around 100 milliseconds, which is what a phone agent needs to sound natural. Other makers change the method itself: Inception's Mercury diffusion language models generate tokens in parallel instead of one at a time, which the company says makes them several times faster. See what a diffusion model is.
Why thinking longer costs more
Reasoning models improve answers by producing more tokens before they reply. NVIDIA calls this test-time scaling: the model runs a sequence of inference passes and creates more output tokens, which needs significant test-time compute. You can see this in the tools you use. Google's Gemini help centre says each available Gemini model can be set to low, medium or high effort, that higher effort gives more thorough answers but uses more of your limit, and that limits are based on compute rather than a fixed number of prompts. Images, video, music, Deep Research and extended thinking use the limit faster. See what a reasoning model is.
Why inference matters at work
- It explains your limits. Plans meter inference. Spending your allowance on high-effort answers to simple questions means running out when you need it. Use low effort for rewrites and quick lookups, high effort for analysis.
- It sets the price of building with AI. APIs charge per token for input and output. Inception, for example, prices Mercury 3 per million input tokens and per million output tokens, with output costing more. A long system prompt sent with every request is paid for every time. See what an AI API is.
- It is where the money is going. Amazon reported spending $77.3 billion on property and equipment in the third quarter of 2026, up from $35.1 billion a year earlier, and said its fall in free cash flow mainly reflects investment in AI. Amazon does not split that figure between training and inference, but it shows the scale of computing being built to run AI.
- It decides whether you can run AI privately. Inference on your own hardware keeps data in house. Black Forest Labs says its open-weight FLUX 3 Image Klein runs on consumer GPUs down to an RTX 3090. Our guide to running AI locally covers the options, and private AI for sensitive data compares hosted choices.
An example you can try in ten minutes
You can feel inference trade-offs with any tool that lets you pick a model or effort level. Gemini is a good test because its effort setting is explicit.
- Choose a real question with some reasoning in it, such as "Which of these three supplier quotes is cheapest over three years, given these terms?" with the figures pasted in.
- Ask it at low effort. Note roughly how long before the first words appear and how long the full answer takes.
- Ask the same question in a new chat at high effort. Note the time again.
- Compare the answers. Did the slower one catch something the faster one missed? Check the maths yourself.
- Now try a simple rewrite task at both levels. You will usually find low effort is enough.
The habit to keep: match effort to the task. In ChatGPT and Claude, the same lesson applies to choosing a faster or a more capable model. Our guide to choosing the right AI model goes further.
Related terms worth knowing
- Token: the unit inference produces and APIs charge for.
- Reasoning model: models that spend more inference to think before answering.
- GPU: the chip most inference runs on.
- Fine-tuning: extra training, which is not the same as inference.
- Context window: everything the model reads per request, which adds to inference cost.
- AI API: how developers pay for inference by the token.
Common misconceptions about inference
- "The AI learns from my chat as I type." During inference the model's weights do not change. Some services may later use chats to train future versions, which is a separate privacy setting you can usually switch off.
- "Training is the big cost; using AI is cheap." Training is expensive, but inference happens every time anyone uses the model. NVIDIA notes that as models grow more complex, especially reasoning models, they need more compute for inference.
- "Slower answers are always better." Higher effort helps on hard problems. For simple tasks it mostly costs time and allowance.
- "Free plans run the same model, just with fewer prompts." Often not. Since 9 October 2026, Gemini's free plan sends most prompts to Flash-Lite, which Google describes as its fastest model.
- "Local AI needs a data centre." Smaller open-weight models, such as FLUX 3 Image Klein, can run inference on a good desktop GPU.
Prompts to copy
Pick the right effort for a task
I am about to ask an AI tool to [task]. Tell me whether this needs a fast model at low effort or a more capable model at high effort, and why, in two sentences. Then rewrite my request as a tight prompt that will get a good answer at the lowest effort that works.
You get: A prompt matched to the cheapest setting that will do the job
Estimate the running cost of an AI feature
We want to add an AI feature that [describe] for [number] users, each making about [number] requests a day. Each request has about [number] words of input and [number] words of output. Estimate the tokens per month, show your assumptions, and list three ways to cut inference cost without hurting quality.
You get: A rough token budget and practical savings