The short answer
Multimodal AI is AI that can take in or produce more than one kind of data, such as text, images, audio and video, in the same model. Google's Machine Learning Glossary defines a multimodal model as one whose inputs, outputs or both include more than one modality. In November 2026, Gemini reads images, audio and video, ChatGPT accepts audio uploads on paid plans and Qwen3.8-Omni-Flash handles text, images and audio, while DeepSeek-V4-Pro and GLM-5.3 are text only.
What multimodal AI is, in plain English
A "modality" is a type of data: text, images, audio, video, and sometimes others such as code or sensor readings. Early chatbots were text in, text out. A multimodal model can, for example, look at a photo of a receipt and answer in text, listen to a voice note and write a summary, or take a sketch and return an image.
Google's Machine Learning Glossary defines a multimodal model as a model whose inputs, outputs or both include more than one modality. The word covers both directions. Reading a chart image is multimodal input; generating an image from your text is multimodal output.
Inside the model, every modality ends up as tokens. Google's Gemini documentation gives the rates: an image up to 384 pixels on both sides is 258 tokens, larger images are cut into 768 by 768 tiles of 258 tokens each, video is 263 tokens per second and audio is 32 tokens per second. So a one-minute video clip uses about 15,800 tokens of its context window before you type a word.
Which AI models are multimodal, November 2026
| Tool or model | Maker | What it handles |
|---|---|---|
| Gemini and Gemini API models | Text, images, audio and video input (counted per image and per second); image creation with Nano Banana 2, free in the Gemini app; Gemini Live voice conversations on the free plan | |
| Gemini Notebook (formerly NotebookLM) | Sources can be PDFs, Slides, images, audio files and public YouTube videos with captions | |
| ChatGPT | OpenAI | Audio uploads to transcribe, summarise and question on paid plans; image creation and voice on all plans with limits on Free |
| Qwen Chat | Alibaba | Qwen3.8-Omni-Flash: native text, image, audio and audiovisual input; Qwen3.7-Plus: images, documents, video and audio |
| Meta AI | Meta | Live AI: point your phone camera and ask about your surroundings in real time; voice conversations |
| Cohere Command A+ | Cohere | Text and image input, 128K input context, open weights under Apache 2.0 |
| DeepSeek | DeepSeek | DeepSeek-V4.1-Flash-1017 understands images; DeepSeek-V4-Pro (Expert Mode) is text only |
| GLM | Z.ai | GLM-5.3 is text only; GLM-5.3-Flash has native visual input |
The last two rows are the useful warning: the same assistant can switch between a multimodal model and a text-only one depending on the mode you pick. In DeepSeek, turning on Expert Mode moves you to DeepSeek-V4-Pro, which does not read images.
Why multimodal AI matters at work
Most work information is not neat text. It is screenshots, scanned forms, slide decks, whiteboard photos, recorded calls and product videos. Multimodal models remove the conversion step.
- Meetings: upload the recording and get notes and actions. ChatGPT paid plans accept audio files directly; see how to transcribe a meeting with AI and the best AI for meeting notes.
- Documents with charts: ask about the chart on page 12 of a PDF, not only the text around it. See how to summarise a PDF with AI.
- Screenshots: paste an error message, a dashboard or a competitor's pricing page and ask what it means.
- Video: turn a training video or webinar into a written guide; see how to turn a video into notes.
- On the move: point a phone camera at a label, a machine or a sign and ask, as Meta AI's Live AI and Gemini Live allow.
Try it: three multimodal tests in ten minutes
- Whiteboard to action list. Photograph a whiteboard or handwritten notes. Upload it to Gemini or ChatGPT and use the first prompt below. Check that it read the handwriting correctly.
- Chart check. Screenshot a chart from a report. Ask for the three main takeaways and the exact values it reads off the chart. Compare with the source numbers: misread values are a common multimodal error.
- Voice note to email. Record a 60-second voice note describing an update, upload it to a ChatGPT paid plan or to Gemini Notebook, which accepts audio files as sources, and ask for a short email in your tone.
Avoid uploading images or recordings with other people's personal data unless your company policy allows it. Images carry more than you think: names on screens, faces, addresses. See how to keep your data private in AI tools.
Common misconceptions about multimodal AI
- "Every AI assistant can see images now." Not every model can. DeepSeek-V4-Pro and GLM-5.3 are text only, and the mode or plan you choose decides which model you get.
- "Multimodal means it can make videos." Reading a modality and generating it are different skills. Many models read images but cannot create them, and video generation is usually a separate tool or a paid feature, such as video generation in Gemini from Google AI Plus.
- "If it reads the image, it reads it accurately." Handwriting, small chart labels and dense tables are frequent sources of error. Ask it to list the values it read so you can check them.
- "Images and audio are free to add." They use tokens like text does, so long videos can fill a context window and use up plan limits.
- "Multimodal is a different kind of AI from LLMs." Many multimodal models are large language models extended to other inputs, which is why the same chat box handles all of them.
Related terms
- Generative AI: AI that creates text, images, audio and video.
- Token explains how images, audio and video are counted.
- Diffusion model is the method behind most image and video generators.
- Text to speech means generating spoken audio from text.
- Computer use covers agents that read the screen visually and act on it.
- Context window explains why long recordings may not fit.
Prompts to copy
Whiteboard photo to action list
This is a photo of our whiteboard from a planning meeting. First, transcribe everything you can read, marking any words you are unsure of with [?]. Then turn it into a table of actions with owner, task and deadline where shown. Do not invent owners or dates that are not on the board.
You get: A transcription with uncertain words flagged, plus an action table you can paste into your tracker.
Read a chart and show your working
Look at this chart. List every value you can read, with its label. Then give the three most important takeaways in one sentence each. If any value is hard to read, say so instead of estimating.
You get: The values the model read, which you can check, followed by short takeaways.
Voice note to email
Here is a voice note with a project update. Turn it into an email to [recipient] under 150 words, in a friendly, direct tone. Keep every date and number exactly as spoken and list anything I mentioned that sounds like an action for them.
You get: A short email drafted from your voice note, with actions pulled out.