A good LLM harness hides the failures it handles

If you run LLM agents in production, you build a harness around them.

The harness is all the code that isn’t the model. It catches a tool call that blows up. It stops a run that has gone on too long. It makes sure a worker that dies mid-job doesn’t leave that job stuck forever. It writes down what happened.

I spent weeks on mine. Then I shipped it, and it worked.

That turned out to be the problem. Here’s the short version of this post: a harness that handles failure well makes failure invisible. The better it gets, the quieter your error logs get - and the failures don’t go anywhere.

The first day in production

The job I’d built all this for looks like this:

Take these 135 questions. Answer each one using the documents in this project. Put the answers in a report, and link every answer back to where it came from.

One agent takes the list and fans out. Each question gets its own sub-agent with its own set of tools and its own limited budget of turns. They work in parallel, write their answers back, and the results get collected into a report.

Day one, that ran. Four of the five jobs finished. Each took between twenty and forty-six minutes. Two hundred and fifty-four sub-agents started, and every single one of them finished cleanly - none got stuck, none needed a human to go rescue it, nothing was left half-written.

I was happy with it.

Except users kept calling it fragile

So I went and looked at the error logs.

Fourteen days of production. One error. A single string that was too long for its database column, fixed in an afternoon.

One error in two weeks, and users telling me the thing was unreliable. Both of those were true at the same time, and it took me embarrassingly long to work out why.

What I had wrong

When a sub-agent fails, my harness catches it.

It writes the failure into the database, marks that item as failed, and moves on to the next one. No exception escapes. Nothing gets logged as an error - because from the harness’s point of view, nothing went wrong. It did exactly what I built it to do.

So the failures were never missing. They were sitting in a database table the whole time, in a column I wasn’t looking at.

Put simply:

• An error log tells you when your code breaks.
• It tells you nothing about whether the model’s work succeeded.

Those are two different things. In a normal system they’re close enough that you can watch one and mostly know about the other. In an agent system they come apart completely, because the model failing at its job is a completely normal event that your code handles correctly.

So I looked at the table instead

Every tool call the system has ever made is a row in the database. I’d been writing to it for weeks and never read it. Here’s what one day actually looked like:

What failed	How many	What it was
Reading a document that had no readable text	180	Sub-agents picking documents blindly. Nothing told them which ones they could actually read, so they found out by trying
Search queries that wouldn’t compile	55 calls - about a third of all searches	The model writes a query, gets an error back, tries again, often fails again
Sub-agents running out of turns	26 of 254 - one in ten	They hit their limit before finishing. And when that happened, everything they’d done was thrown away
Writing to a report	2 attempts, 0 worked
Made-up or stale document ids	~8
A job rejected before it started	1 of 5	A required field was missing from every item
An actual crash	1	The only one of these my monitoring told me about

That last row is the whole point. My monitoring caught the bottom line of that table and nothing above it.

Two of those rows are the same problem

Look at the first row and the third row together.

Each sub-agent gets a limited number of turns. If it wastes four of them opening documents that turn out to be unreadable, and two more on searches that won’t run, it hits its limit in the middle of the actual work.

And hitting the limit was a cliff. If a sub-agent ran out of turns without formally finishing, I threw away everything it had - all the reading, all the evidence it had gathered, the partial answer it was in the middle of writing. Gone.

So the 180 unreadable documents and the one-in-ten sub-agents dying are not two problems. They’re one problem seen from both ends. Small friction eats the budget, and running out of budget costs you the whole job.

Neither one logs an error.

Three fixes

Once I could see the numbers, the fixes stopped being guesswork. Each one is aimed at a specific row.

Tell the model what it can do with what you just gave it.

Those 180 failures happened because I handed the model a list of documents with their ids and names, and nothing else. It had no way to know which ones it could read. So it guessed, and a lot of the guesses were wrong.

Now every document in that list comes with the list of things you can actually do with it. The rule I wrote down afterwards: if you give the model an id, tell it what that id is good for. It should never have to learn what your tools can do by failing at them.

The one that surprised me.

I assumed the search failures were the model writing bad syntax, and I started writing better error messages to teach it the right syntax.

Before shipping that, I went back and read all 107 individual failures. Eighty-four of them - 78% - were the model trying to search by document id.

My search language didn’t have a field for document id. Every result I handed back included a document id. The model kept trying to use it, and the language kept saying no.

That’s not a syntax problem. No error message could have fixed it, because there was no right answer to point at. The fix was to add the field. Ten minutes of work, hiding behind an assumption I’d held for weeks.

Don’t throw away work when the budget runs out.

When a sub-agent hits its turn limit now, it gets one last turn with exactly one option available: hand in your best answer with what you’ve got. Its work reaches the report instead of the bin.

I also made failed calls cost less than successful ones. A sub-agent that burned four turns on cheap mistakes shouldn’t be out of budget for the real work.

What I should have built first

A yield report.

Not errors - yield. How many items finished. How many failed, and why. How many sub-agents ran out of turns. Broken down per job and per week, read straight off the same table I’d been ignoring.

It’s a small piece of code. It’s on the admin page now and behind a command I can run against any week. It took an afternoon, and it’s the single most useful thing in the whole system, because it’s the only thing that tells me whether the agent is doing its job.

And it changed what I alert on. Alert when yield drops, not when errors spike. There are no errors. There were never going to be any.

Fair warning on those numbers: they’re from the first production day of a brand new system. They’re the reason all this work exists. I’ve fixed the classes one at a time rather than re-running that exact day as a clean before-and-after, so read the table as the problem I started with, not a scorecard.

The takeaway

If you’re running LLM agents in production and your error dashboard is clean, that isn’t good news. It might just mean your harness is doing its job.

A harness turns model failure into a quiet row in a table. Quiet rows don’t page anybody. So go read the rows.

Errors tell you if your code is working. Yield tells you if your agent is working. Only one of those is the thing you shipped.

Mine runs in production at scoreboardai.net, with real customers paying for it - which is exactly why I needed to know which number I was looking at.