What are images as input?
To a computer a photo is millions of coloured dots with no letters in it. A model that accepts images, often called multimodal, can still read one. The picture is cut into small squares, and each square is turned into something the model handles the way it handles a token of text.
That means pictures cost tokens and take up room in the window, usually a few hundred to a couple of thousand each. Bigger images cost more, which is why apps shrink a photo before sending it: small enough to be cheap, large enough to stay readable.
A model can misread a picture just as it can get a fact wrong, and a creased digit looks as certain in its reply as a clear one. So anything extracted from an image is checked like any other reply, and a person is asked when the check fails.
Why do images as input matter?
Much of the information people deal with is visual: receipts, screenshots, forms, charts, photographs of a problem. A model that can take a picture as input can read all of these directly. That opens up tasks that used to need a person to look and type.
How do images as input work?
- The picture is resized so that it is small enough to be cheap and large enough to stay legible.
- It is cut into a grid of small squares.
- Each square is turned into something the model handles much like a token.
- The model reads those alongside any text, and writes its reply as usual.
When should you use images as input?
Use it when
- Reading documents that arrive as scans or photos
- Understanding a screenshot or a page layout
- Letting people show a problem instead of describing it
Think twice when
- The same information is available as text. Text is cheaper and more exact
- Fine detail matters and the picture is poor
Where do images as input sit in a real system?
The image is attached to a model call like any other input. Your code prepares it first: resizing, rotating, and for multi-page documents deciding which pages to send.
What goes wrong with images as input?
- Trusting every digit read from a blurry picture.
- Sending huge images and paying for detail that is not needed.
- Forgetting that text inside an image can carry instructions too.
What does images as input look like in practice?
A crumpled receipt is photographed for a club's accounts. The model returns the shop, the date and a total of $41.80. Code checks that the lines add up to the total. They do not, so the receipt is shown to a person, who sees the 1 was read as a 7.