[01]The demo is not the product
A demo answers one question: can this work at all? A product has to answer a different one: can this work every day, for every user, on every input, without anyone from the vendor in the room. Most AI projects we are asked to rescue were optimized for the first question and never re-planned for the second.
The clearest symptom is a team that measures success in impressive outputs rather than in boring ones. A customer-service agent that writes a beautiful paragraph is a demo. One that resolves the same refund request correctly a thousand times is a product.
[02]Real data is messy data
In the demo, the documents were clean PDFs. In production they are scans, photos of screens, three languages, and a spreadsheet someone exported in 2019. The model did not change; the input distribution did. Production-grade systems are built around that gap, with validation, fallbacks, and a path to a human.
"Software isn't the goal. A better-running business is. The same is true for AI — the model is never the point."
— from the BOYD systems audit
[03]Users do not read prompts
The people who will use the system are operations staff with a queue to clear. They will paste the wrong thing, skip the instructions, and judge the tool in the first ten minutes. Design for that person, not for the engineer who wrote the prompt.
- Log every input and output from day one, and read them weekly.
- Give every automated decision a visible confidence and an undo.
- Route anything the system is unsure about to a named person, not a shared inbox.
- Measure resolution, not generation.
[04]What production-grade actually means
Security, reliability, maintainability, documentation, monitoring, and deployment are not extras you add after the demo works. They are the product. If a system cannot be operated by your team after we leave, we have not finished.
- AI in production
- Operations
- Engineering
