Notes ยท Munna Suprathik
The Prompt That Worked in Dev and Died in Prod
By Munna Suprathik, Generative AI Engineer.
It passed every test I had. All ten of them. I wrote all ten. In hindsight that is not a test set, it is a mirror, and mirrors are very encouraging.
Real users showed up with typos, half sentences, context I never imagined, and two extremely polite people who wrote paragraphs. The prompt that was "robust" fell over on inputs barely different from my own. Prompts are brittle in a way normal code is not, because a single reworded sentence can quietly shift the model into a different mode, and nothing throws.
Green in dev, quietly wrong the moment a real user typed.
So I built the least glamorous tool in my stack: a spreadsheet of real, messy inputs with the behavior I expect for each. Before any prompt change ships, it runs against that set. The rule that changed everything is simple. A change that wins on the easy cases and loses on the hard ones is not an improvement, no matter how good the diff feels.
A prompt tested only on the inputs you imagined works in the demo and nowhere a user actually lives.