In a photo-based ordering system, a simulated reading let me validate the entire flow without paying for a single real one. What it didn't catch taught me more than what it confirmed.
Anderson Silveira · September 16, 2026
A lubricant distributor received orders as photos of handwritten notes on WhatsApp, and someone retyped everything. It’s the kind of manual work that calls for automation: repetitive, with a known rule and an expensive mistake. The system I built keeps the gesture of taking a photo: the AI reads the note, matches each line against the catalog, and proposes the items. The person reviews line by line before confirming.
When you automate work like that, the temptation is to validate the eye-catching part, which is the AI reading handwriting. But the question that decided whether the automation worked was a different one: can the buyer review a proposal from the machine and confirm the right order?
To answer that, I didn’t need a real reading. The flow runs end to end with a simulated reading that returns a fixed extraction matched to the catalog, with no paid calls. The review, the uncertain line, the unrecognized line that locks the button, and the order that reaches the dashboard like any other: all of that could be exercised without spending anything on AI. In production, the feature stayed off at the client’s request until the real reading was turned on. Turning it back on means flipping a switch.
The moment the topic became clear came later. Running the whole flow again with the simulated reading, an “Estimated total R$0.00” showed up in the review footer. The client had asked to remove prices from every screen; the rule worked, but two places were still printing a zeroed-out amount. No test complained. And that same week, the entire suite passed green while the API, actually running, was returning an error on the catalog. Validating early solves a lot. It doesn’t help if what you’re validating isn’t the system that goes live. This article is about how to validate an automation before switching on its expensive part.
Start with the hypothesis
A prototype without a hypothesis becomes a presentation. A good hypothesis states what you believe, for whom, and what would prove the opposite:
Written the way I’d write it today, the one for photo ordering would be: the buyer can turn a photo of the note into a correct order without retyping; confirmed if they resolve every uncertain line without help; refuted if they confirm a wrong line without noticing.
It’s worth separating three questions, because they call for different tests:
- Value: does this solve something that matters? Here, removing the retyping from the back office.
- Usability: can people use it? Here, the review screen.
- Feasibility: does the technology deliver? Here, the reading accuracy.
The simulated reading was useful precisely because it separated the second from the third. It says nothing about the AI’s accuracy. It says everything about what happens around it.
Fidelity follows the question
| Low fidelity | High fidelity or code | |
|---|---|---|
| Cost to change | Minutes | Hours or days |
| Good for testing | Flow, order of steps, concept | Interaction, states, copy, trust |
| Risk | People fill gaps with their imagination | It looks finished, and the team gets attached |
The rule I use: the earlier the question, the lower the fidelity. The exception is when the interaction itself is what’s being tested. A review with suggested lines, unidentified lines, editable quantities, and a button that only unlocks at the end has too many states for static screens. There, a code prototype, with the expensive part swapped for a simulation, is worth more than any mockup.
The catch is that code looks like product. If it becomes the foundation without review, validation becomes a development shortcut.
People, tasks, and the limit of five
The most cited reference is Jakob Nielsen’s 2000 article: five participants find most of the usability problems in a design. The caveat that gets lost: it’s per round, within an iterative process, per user profile, and it’s for finding problems, not measuring success rates.
The task is the instrument. It describes the situation, never the button:
| Bad task | Better task (illustrative example) |
|---|---|
| Click “Send photo” and confirm the items | The repair shop needs three buckets and a case of oil. You wrote it down on paper. Place the order. |
| Check whether the yellow line is right | The order reached the store with an item you didn’t ask for. Find out where it came from. |
Define success beforehand. In photo ordering, success isn’t “confirmed.” It’s “confirmed the order that was on the paper.”
A green test isn’t a working system
This is the part the case added to the method. Three episodes, from three projects:
- In the ordering system, the tests read the source code of a shared package; the running API read its compiled version, which was stale. The suite passed in full, while the catalog and the buyer’s home screen were returning errors. Shipping on the strength of the green would have taken down the two most used screens.
- In Glint, a pass through every route found a contact-the-organizer form that sent to an address that didn’t exist. The call didn’t report the error, the screen showed success, and the message was lost.
- Also in Glint, a QA campaign with specialist agents took the suite from 378 to 1,049 tests. Along the way, it surfaced a funnel that always showed zero people (the data was written under one name and read under another) and an account suspension that only blocked at login: anyone already logged in kept their access.
The pattern is the same: the test confirmed something adjacent to what mattered. So after the green, I run the flow on the live system, with real or simulated data, looking at the screen. That’s how the “R$0.00” showed up.
Validate the cost and the trade-off, not just the screen
Two validation decisions that aren’t about usability.
Estimate before you run. In the legal AI agent, every analysis run has a real cost, so the screen shows the estimate before the button, against the budget remaining for the case. The creditor’s full track was validated end to end on a real case, from triage to an approved draft. And even so, when a lawyer ran one of his own cases, the draft failed twice; both failures were fixed the same day. In one of them, the chat had never worked on a large case file: the size calculation measured one unit and the real limit was in another. The validation was measuring the wrong thing.
Keep the cost on screen. In Glint, the rotating QR code keeps a screenshot of a valid ticket from circulating, but it requires the attendee’s phone to have internet at the door. Instead of turning it on for everyone, the option launched off by default, with the warning written next to the toggle. Validating here meant asking in what context the protection turns into a line.
The cycle
- 01Hypothesis
The buyer reviews the machine’s proposal and confirms the right order.
- 02Simulate the expensive part
Simulated reading, fixed extraction, no paid calls.
- 03Tasks
The note on paper becomes an order, without the words on the screen.
- 04Sessions
About five people per profile, per round.
- 05Live system
Run the flow on the running API, not just in the suite.
- 06Severity
Rate what came up and fix it before switching on.
Severity rating: not every finding weighs the same
I cross two axes: how many people the problem would affect and what it causes. The items below are real findings from the cases above; the positions are my reading, not a measurement.
| Severity | Criterion | What to do |
|---|---|---|
| 4 | Blocks the main task, or causes loss of data or money | Fix before launch |
| 3 | The person gets there, with a lot of effort, errors, or help | Fix this round |
| 2 | Noticeable hesitation, without blocking | Next cycle |
| 1 | Cosmetic | Log it; fix if cheap |
The “R$0.00” is a good example of misleading severity. It looks cosmetic. In a system where the client asked to hide prices because each buyer has their own negotiated terms, a zeroed-out amount on the confirmation screen creates doubt exactly at the moment of confirming.
What I’d do differently
- Test a rule change on every screen, with the simulated flow switched on. Hiding prices worked because the decision lived in one place. The two places that slipped through were the ones I didn’t look at: the pickup schedule and the review footer, in a feature that was switched off.
- Build the shared package before testing the running API. That became a rule after the scare.
- Put the review in buyers’ hands before turning on the real reading. The pain points were inferred from the process, not from interviews. The simulation makes that round free; it’s the step I’d take before flipping the switch back on.
- Test the counter’s vocabulary against the real catalog. In the simulation, “cx 24” (short for a box of 24) was matched to the drum, not to the box of 24 bottles. It may be an artifact of the test catalog, but it’s exactly the kind of costly error human review exists to catch.
Checklist before the next round
- Is the hypothesis written down, with criteria for confirmation and refutation?
- Are value, usability, and feasibility separated, each with its own test?
- Can the expensive part (AI, payment, integration) be simulated to validate the rest?
- Do the tasks describe situations, without the interface’s wording?
- Was what counts as success defined beforehand?
- Are there about five people per profile, with another round after the fix?
- After the green test, did someone run the flow on the live system?
- Are the cost and the trade-off visible to whoever decides to switch the feature on?
- Were the findings rated by severity, not just listed?
In an automation, the AI is almost always the expensive part. It’s also the part that least needs to be switched on to validate the flow around it. Simulating it lets you validate early. Looking at the running system is what keeps a green suite from passing for validation.