Parag Pandya

Published · Updated

Making an AI assistant tell the truth

At work I built an AI chat assistant that answers questions about how our software platform works by reading the code. Product managers, analysts and architects used it to check how the platform behaves without having to ask an engineer.

It had one dangerous habit. Sometimes it said "the platform doesn't do that" when it did. In one case the exact term was in the code hundreds of times, buried under thousands of looser matches, and the AI decided the feature didn't exist. People act on confident answers, so a confident wrong answer is the worst kind of failure.

This is how I fixed it, how I measured it, and what I would tell anyone building something similar. I've described it in general terms, without internal names or data.

Why didn't better prompts fix it?

My first instinct was to tell the AI to be more careful. It helped a little, then stopped helping. A prompt is a request, not a rule. On a long or awkward question the AI still took a shortcut, searched once, found nothing useful and said the feature wasn't there.

So I stopped asking and started checking.

What did I build?

The lesson in one line: give an AI tools it must use and checks it must pass. Asking it to be careful doesn't hold up.

How did I measure it?

I ran four separate tests. They measure different things, so I keep their results separate.

Why publish a 32% recall?

Recall is how many of the relevant things the assistant found. 32% is low, and I wrote it down rather than hiding it. I chose an assistant that admits what it hasn't found over one that fills the gap with a guess. For people checking regulated behaviour, "I couldn't confirm that" is useful. "That doesn't exist", when it does, is harmful.

An honest low score you can act on is worth more than a high score nobody believes.

How do I keep the AI's checks honest?

An AI marking its own work tends to agree with itself. So the reviewers are always fresh sessions that haven't seen how the thing was built, and they are never told the expected answer. They're split by angle. One checks the logic. One checks what a user would see. One checks that every quoted source is real and every test checks what it claims to. "Did this make things worse?" gets answered by comparing test results against the last good version, not by anyone's opinion.

Where else did this work?

I used the same approach on a large AI-written regulatory test suite. Independent AI scorers rated a fixed 50-row sample before and after my fixes. Rows judged accurate went from 42% to 86%, and made-up rows went from 5 to 0. Measuring the same sample before and after is what showed the fix worked.

What did I learn?

What I'd do next is grow the test set from real questions people ask, so the score tracks how the assistant is actually used.

Written by Parag Pandya, a Senior Product Manager in Melbourne with 18 years in Australian energy. Say hello on LinkedIn.

More writing

← All writing