AI Engineering, story 25A receipt becomes a row

A crumpled receipt

1 of 14

A receipt becomes a row: AI Engineering, story 25

HM Nayem photographs a crumpled receipt for the hiking club's accounts. See how a picture turns into clean numbers in a spreadsheet.

How a photo becomes something a model can read, and then clean numbers.

The story, step by step

  1. A crumpled receipt

    HM Nayem bought supplies for the hiking club and needs the cost in the club's accounts. The receipt has been in a pocket all day. HM Nayem takes a photo of it and wants nothing more to do with it.

  2. A photo is not text

    To a person, that picture is full of words and numbers. To a computer, it is twelve million coloured dots. There is not a single letter in the file. Nothing in it says “total”.

  3. How this used to be done

    For years the answer was a program that hunts for letter shapes, followed by hand-written rules like “the total comes after the word TOTAL”. It worked on clean receipts from shops it had rules for, and broke on everything else.

  4. Some models can look

    Many models today accept pictures alongside text in the same request. They are called multimodal. The request looks just like the ones you have seen, with one new kind of part: an image.

  5. Shrink it first

    Before sending, the app makes the photo much smaller. A phone photo is far sharper than reading a receipt requires, and you are about to see that size costs money. Small enough to be cheap, big enough to stay readable.

  6. How a model sees a picture

    The model cuts the picture into small squares. Each square is turned into something it handles just like a token of text. So to a model, an image is a few hundred extra tokens that happen to describe what is in each square.

  7. What a picture costs

    Because a picture becomes tokens, it is billed like tokens and takes room in the window like tokens. This one costs about as much as a page and a half of text. The full-size photo would have cost many times more.

  8. Ask for a form, not a description

    Left alone, the model would describe the receipt in a friendly paragraph. A spreadsheet cannot use a paragraph. So, as in the returns story, the app asks for an exact shape, with the category picked from a fixed list.

  9. The reply

    Back comes exactly that shape, filled in. It looks perfect. But look at the total, and then at the items above it.

  10. Check it, as always

    The app never trusts a reply. One of its checks is plain arithmetic: do the items add up to the total? They come to 86.40. The model wrote 36.40. A crease in the paper ran straight through the 8.

  11. Look again

    The app sends the photo back with the problem spelled out: the items do not add up to the total, please look again. With that hint, the model reads the creased digit correctly.

  12. When to ask a person

    Sometimes a photo is simply too blurred, and no second look will help. Then the right move is not to guess. The app shows HM Nayem the one field it is unsure of and asks. One tap, and it is certain.

  13. A row in the spreadsheet

    The checked values go into the club's accounts as one clean row. Twelve million dots went in. Four tidy fields came out. HM Nayem never typed a number.

  14. Now at real speed

    Photo in, shrunk, read, checked, read again, saved. The same pattern handles invoices, forms, ID cards and handwritten notes. Anything on paper can become data.

A picture becomes tokens, and then data.

Ideas in this story