A receipt becomes a row: AI Engineering, story 25
HM Nayem photographs a crumpled receipt for the hiking club's accounts. See how a picture turns into clean numbers in a spreadsheet.
How a photo becomes something a model can read, and then clean numbers.
The story, step by step
A crumpled receipt
HM Nayem bought supplies for the hiking club and needs the cost in the club's accounts. The receipt has been in a pocket all day. HM Nayem takes a photo of it and wants nothing more to do with it.
A photo is not text
To a person, that picture is full of words and numbers. To a computer, it is twelve million coloured dots. There is not a single letter in the file. Nothing in it says “total”.
How this used to be done
For years the answer was a program that hunts for letter shapes, followed by hand-written rules like “the total comes after the word TOTAL”. It worked on clean receipts from shops it had rules for, and broke on everything else.
Some models can look
Many models today accept pictures alongside text in the same request. They are called multimodal. The request looks just like the ones you have seen, with one new kind of part: an image.
Shrink it first
Before sending, the app makes the photo much smaller. A phone photo is far sharper than reading a receipt requires, and you are about to see that size costs money. Small enough to be cheap, big enough to stay readable.
How a model sees a picture
The model cuts the picture into small squares. Each square is turned into something it handles just like a token of text. So to a model, an image is a few hundred extra tokens that happen to describe what is in each square.
What a picture costs
Because a picture becomes tokens, it is billed like tokens and takes room in the window like tokens. This one costs about as much as a page and a half of text. The full-size photo would have cost many times more.
Ask for a form, not a description
Left alone, the model would describe the receipt in a friendly paragraph. A spreadsheet cannot use a paragraph. So, as in the returns story, the app asks for an exact shape, with the category picked from a fixed list.
The reply
Back comes exactly that shape, filled in. It looks perfect. But look at the total, and then at the items above it.
Check it, as always
The app never trusts a reply. One of its checks is plain arithmetic: do the items add up to the total? They come to 86.40. The model wrote 36.40. A crease in the paper ran straight through the 8.
Look again
The app sends the photo back with the problem spelled out: the items do not add up to the total, please look again. With that hint, the model reads the creased digit correctly.
When to ask a person
Sometimes a photo is simply too blurred, and no second look will help. Then the right move is not to guess. The app shows HM Nayem the one field it is unsure of and asks. One tap, and it is certain.
A row in the spreadsheet
The checked values go into the club's accounts as one clean row. Twelve million dots went in. Four tidy fields came out. HM Nayem never typed a number.
Now at real speed
Photo in, shrunk, read, checked, read again, saved. The same pattern handles invoices, forms, ID cards and handwritten notes. Anything on paper can become data.
A picture becomes tokens, and then data.
- A photo is only coloured dots until a model that accepts images reads it.
- The model cuts a picture into small squares and treats each like a token.
- Pictures cost tokens, so apps shrink them before sending.
- Whatever is read from an image is checked, and a person is asked when it stays unclear.
Ideas in this story
- Images as input: A model that accepts pictures cuts each one into small squares and reads them much like tokens.
- Structured output: Getting a reply in a fixed shape that code can read, checking it, and retrying when it is wrong.