Multimodal AI
Also called multimodal model, vision language model, multi-modal
AI that can work with more than one type of input or output, such as text, images, audio and video.
A multimodal model can take in, and sometimes produce, more than one kind of data. Most leading assistants now accept images alongside text: you can upload a screenshot, a chart, a photo of a handwritten note or a PDF with figures, and ask questions about it. Many also handle audio for voice conversations, and some accept video.
Multimodal input is useful for anything that is hard to describe in words: reading a graph, checking a design mockup, debugging from a screenshot of an error, transcribing a whiteboard, or describing images for alt text. Output can be multimodal too: some systems generate images, speech or video directly, while others call a separate specialised model, such as a diffusion model for images.
Multimodal models can make specific mistakes. They may misread small text, miscount objects, confuse left and right, or describe details that are not actually in the image. Giving context about what the image is and what to look for improves accuracy, as with text prompts.
Example
A user uploads a photo of a restaurant receipt and asks, "Split this between 3 people, but Sam did not have wine." The model reads the items, removes the wine from Sam's share, and shows each person's total.
Common mistake
Assuming the model sees an image the way you do. It may not read faint or tiny text reliably, and it can hallucinate details. For anything precise, such as numbers in a table or text in a document, ask it to quote what it read so you can check it against the original.