At work I built an AI chat assistant that answers questions about how our software platform works by reading the code. Product managers, analysts and architects used it to check how the platform behaves without having to ask an engineer.
It had one dangerous habit. Sometimes it said "the platform doesn't do that" when it did. In one case the exact term was in the code hundreds of times, buried under thousands of looser matches, and the AI decided the feature didn't exist. People act on confident answers, so a confident wrong answer is the worst kind of failure.
This is how I fixed it, how I measured it, and what I would tell anyone building something similar. I've described it in general terms, without internal names or data.
Why didn't better prompts fix it?
My first instinct was to tell the AI to be more careful. It helped a little, then stopped helping. A prompt is a request, not a rule. On a long or awkward question the AI still took a shortcut, searched once, found nothing useful and said the feature wasn't there.
So I stopped asking and started checking.
What did I build?
- A search the AI must run. Before the AI can say something doesn't exist, it has to run a fixed search tool. The tool tries spelling and naming variants, searches all the code at once, and ranks the most specific matches first, so exact hits can't get buried. Every search is logged.
- An answer checker. A separate check runs on every final answer before the user sees it. Every source the answer quotes must exist. Any "this doesn't exist" claim must come with proof that the search ran.
- A visible result. Every answer shows a pass or a warning, and an answer that fails gets one automatic retry.
The lesson in one line: give an AI tools it must use and checks it must pass. Asking it to be careful doesn't hold up.
How did I measure it?
I ran four separate tests. They measure different things, so I keep their results separate.
- Blind audits. Three realistic scenarios, run by AI reviewers who had never seen the build and weren't told the expected answer. I fixed what failed and ran them again, five rounds in all. In the final round, 100% of the sources it quoted were real, it made up nothing, and average recall was 32%.
- A 22-question test set. Real regulatory questions in plain language, each answer scored for correctness, completeness, helpfulness and relevance. No made-up facts. The answers were good, but slow and wordy.
- Cold reviews of the knowledge behind it. The assistant sits on a map of the code that AI helpers rebuild each week. On the largest rebuild, 33 of 33 reviews by reviewers with no background passed, with no made-up facts.
- Checks per product area. Five to eight scenarios for each area. All passed with no made-up facts, and recall was about 95%. These scenarios were narrower than the blind audits, which is why the recall is so much higher.
Why publish a 32% recall?
Recall is how many of the relevant things the assistant found. 32% is low, and I wrote it down rather than hiding it. I chose an assistant that admits what it hasn't found over one that fills the gap with a guess. For people checking regulated behaviour, "I couldn't confirm that" is useful. "That doesn't exist", when it does, is harmful.
An honest low score you can act on is worth more than a high score nobody believes.
How do I keep the AI's checks honest?
An AI marking its own work tends to agree with itself. So the reviewers are always fresh sessions that haven't seen how the thing was built, and they are never told the expected answer. They're split by angle. One checks the logic. One checks what a user would see. One checks that every quoted source is real and every test checks what it claims to. "Did this make things worse?" gets answered by comparing test results against the last good version, not by anyone's opinion.
Where else did this work?
I used the same approach on a large AI-written regulatory test suite. Independent AI scorers rated a fixed 50-row sample before and after my fixes. Rows judged accurate went from 42% to 86%, and made-up rows went from 5 to 0. Measuring the same sample before and after is what showed the fix worked.
What did I learn?
- Put the rules in tools and checks, not just in the prompt.
- Use reviewers who don't know the answer.
- Keep results from different tests apart. Merging them hides the weak spots.
- Building the tool is half the job. The weekly rebuild behind this assistant once went stale because the job wasn't kept running, and the answers got worse with it.
What I'd do next is grow the test set from real questions people ask, so the score tracks how the assistant is actually used.