The Tests Passed. The Meeting Was Still Missing.

For a while my way of working with an agent was one-on-one. I'd sit with it and correct it as it went, sometimes on technical decisions, and along the way it wrote tests. Many, many tests. They passed, and I kept looking at them and struggling to see what they actually verified, or whether they meant anything at all.

The tests passed

I was building a CLI at work for fetching information about meetings and their transcripts. At one point the CLI decided a meeting wasn't available because its recording wasn't available, and the tests were fine with that. The trouble was that the AI transcript was already there.

I only found out by testing for real. I went looking for a meeting I knew I should have a transcript for, the CLI told me it wasn't available, and only then did the agent work out why. Its tests had been checking its own assumption, so of course they passed.

Real meetings, made deterministic

So I gave the agent real meetings to work from, and it used them to build its own automated verification. It turned them into mock data built from those meetings, so it was testing against something deterministic but still very real. That got me to the point where I could trust the agent not to break real functionality, which is what I'd been checking by hand every time I sat with it.

I was the slow part

The idea I keep coming back to from pstack, a set of agent skills I've been using, is going deep and fully trusting the AI to produce high-quality output before going broad. I've come to think the more improvement iterations you can run, the better the result gets, and the way to run more of them is to take the human out of the loop. In my case that's me, and I'm slow. I was the slow part of every one of those one-on-one sessions.

That only works if the agent can tell on its own whether it got the right outcome. The quality of each iteration I'm less worried about. If an iteration produces mediocre code, I can make up for it by running more iterations, or by upgrading to a stronger model as one becomes available. What doesn't go away is the need to check that the output matches intent, and if the agent can't do that, more iterations just give me more of what I got with the meetings: tests that pass and an answer that's wrong.

Maybe mock data isn't the answer

I don't have this fully working yet. I'm also not sure mock data is the right way to do it, because it's still a test suite, and a test suite only knows the cases someone thought to put in it.

Look at how the bug actually got caught. I wasn't running a test. I knew what the CLI was supposed to do for me, I used it the way I'd use it for real, and I checked whether it did that. That's closer to what I mean by TDD: the same discipline, but with the tools to test like a human would.

I think that's the real gain: judgment based on intent, with the model using those same tools the way I did. Taking the human out of the loop doesn't mean taking out the human's way of checking. It means handing it to the agent.