Where to Draw the Line: What to Make Deterministic and What to Leave to AI

At work we have been using AI for a while now, mainly one-shot prompts connected to some of our processes. And most recently I've been exploring how to build useful agents. I've learned the importance of using good data to evaluate what the prompt or the agent is giving us. The more I compared different models and different prompts over multiple evaluation runs, something was becoming clearer to me. I was always looking at the wrong place.

The obvious answer

I worked on a customer support agent that would be able to correctly diagnose and recommend communication for the different types of complaints we receive. Luckily I have more than enough real tickets to use to evaluate my agent. I started with the shortest path. I identified the most frequent type of ticket that also had clear possible outcomes when reading the necessary data, using some tools I gave to the agent. I wrote an elegant system prompt, a custom version of a previously tested prompt for answering customer support knowledge, with some added prose to act as the agent that classifies and resolves this type of ticket. And enthusiastically went to measure the results.

Twenty-seven percent. And on top of that it invented that the tracking said delivered, when it hadn't.

Not happy with the results I went for the obvious answer. We need a bigger and meaner model, ready to beat those pesky tickets.

Fifty percent. The tuned smaller model was already doing forty-seven. A meagre improvement for what those tokens cost me.

So the problem was really clear and the numbers were really clear. I needed to make the classification more deterministic. I created a classifier function that would read the tools data and give a possible classification. I created a second tool that, given a certain type of issue, would try to give you the possible outcomes. Those parts I could unit test in code. And I can also easily explain and show the map to the domain owners for validation. I ran the evaluations once more and the numbers made it super clear. Eighty-seven percent, with no variance and nothing invented. And the small job I had left to the model scored fifteen out of fifteen, on the cheapest one available. The model was not the answer, but rather defining what needs to be deterministic and what does not.

The test that passed too well

So, with that lesson still warm, I was asked to review and fix a new process to automate the newsletter. This was a POC done some time ago in collaboration with marketing and tech. At a high level the process would select a list of categories and new listings and feed that into a prompt, which would output a selection of the 8 best candidates based on a weekly schedule of which categories to prioritize. The main complaint was that it was unreliable: wrong day selection, picking expired offerings, and so on. At first glance the bug was simple. The initial selection didn't account for the correct time window. The prompt was asked to select based on dates, so it did exactly that. Garbage in, garbage out.

So I set a base evaluation to establish a baseline, then fixed the bug and ran the eval, and I got slightly better results. Ok, probably the bug isn't the only problem here, I told myself. Right after, I do what any good engineer does these days. I wrote a prompt to an agent: here is the baseline, here is the original prompt, here is how to evaluate new results, go make it happen. The agent wrote such ugly code. Mapping words to categories, to days, it was a mess. I saw the test results. 100% of the tests passed. Good, problem solved, let's clean up the messy code and call it a day. If only it were that easy, the moment I saw the perfect result I knew I had the classical overfitted optimisation. That was definitely not the right answer.

So I designed a better solution, a deterministic backbone with the prompt acting as a judge of style. I also improved the test. I decided that my first evaluation was comprehensive and it covered many edge cases, but it didn't represent reality, so I also ran the process against one year of data, grading the results for each day. I compared the different variations, after I did a few more with both code and prompt improvements, trying to minimize the possibility of AI messing up the selection across the outlier days. As I compared the results something was clearer to me. The deterministic backbone without any prompt hands down had the best results of them all. At the end of the day I concluded it was better to remove AI altogether from the process. The tool needed was just simple good code, not better prompts or better models.

Why we all reach for the prompt

We all reach for the prompt first, and we probably go for more model after. But I think that before AI it was really clear where to draw the line. Non-deterministic is hard, so we make things deterministic. Now that AI makes non-deterministic easy, and it gets better every new model, the line is not that clear anymore. And if you don't evaluate the results the line becomes impossible to see. The prompt seems like the easiest place to optimize, the prose is sitting right there like a dial. But even with good evaluation data it is easy to overfit against it and fail miserably running live. And the statistical nature of foundation models is easy to forget when you get the right answers, while the numbers can be saying otherwise. That is why the prompt itself was never the first place to look.

Drawing it by hand

AI can do a lot of things, but that doesn't mean it needs to do everything.

Your guarantees should always live in code. The verifications, the counting, your policies. The model needs to be bounded when you use it: structured data in, validated output. That way you can shape the results without breaking any guarantee.

When things were not clear, I've learned through testing with numbers I could trust which things needed to be deterministic in code and what things needed to stay non-deterministic and on the model side. At first glance the line is not always so clear, but testing and putting numbers to the claims makes it clearer.

Before building or fixing anything we need to ask ourselves: is the rule knowable from the data we have? Is correctness binary and testable? Is getting it wrong a defect, or a matter of taste?

Can your test even tell the difference between a good answer and a lucky one? Making a good test feels more like an art than an exact science sometimes, but it is critical that we do it.

Sometimes AI will make things unpredictable and unreliable. It's ok to say it's not always the right tool. The same measuring that told me to remove AI from the newsletter is what told me to trust it with the wording of the support replies. Fifteen out of fifteen, on the cheapest model available.

Maybe it is the casino addiction we are all having in this space right now, about what the next roll of the tokens is going to bring. If you are not evaluating the AI output, you are gambling, and the house always wins.