I rewrote my prompt five times. It got worse every time.

There’s a file in my project that nobody wanted to touch.

It’s the system prompt - not anything a user types, but the block of text the model reads before it sees a single word of the actual conversation. Rules, background, tool descriptions, the whole standing brief.

Mine had grown to about 34,000 characters, and that’s before adding a description and a JSON schema for each of two dozen tools on top. Nobody had ever measured it. It had no version history, so there was no way to see what changed last week or roll any of it back. Three environments held three different versions of it and I couldn’t have told you which one was right. Parts of it described behavior the system never actually had - instructions for something that got designed, written up, and never built.

And every time the agent did something I didn’t like, my first move was to open that file and add a paragraph.

This is the bug that broke me of the habit.

The bug

Give the agent a job with a long list in it - a couple hundred items, each one needing its own small piece of work - and it would open the first item, look at it, and then write me a paragraph about what it was planning to do next. Then stop. It didn’t do the work. It described the work.

About 40% of turns on jobs shaped like that. I could watch it happen. I had the full transcripts. There was nothing mysterious about this one.

So I did what you do. I told it not to.

Five times

First I added a line to the system prompt saying to act instead of announcing. Then I measured. The rate went up.

I rewrote the line to be blunter. Up again.

I added an example - here’s the wrong behavior, here’s the right one. Up.

I moved it near the top, thinking maybe position mattered. Up.

I made it a hard rule with the consequences written out. Up.

Five attempts, every one aimed at this exact behavior, every one carefully worded. All five made the number worse. Not “no effect” - worse.

At some point you have to stop blaming your wording.

Why adding more made it worse

I’d been treating the system prompt like a config file. You add a setting, you get the behavior. Add a rule, the model follows the rule.

That holds until the prompt gets big. After that, new instructions don’t stack on top of the old ones - they compete with them. The model is weighing everything you wrote against everything else you wrote, and each new sentence makes all the existing ones count for a little less.

Which explains the five failed attempts. I wasn’t writing on a blank page. I was writing over the top of every rule I’d already put in there, including a lot I’d forgotten about, including some that described features that didn’t exist.

The detector I built and then deleted

Before I understood any of that, I tried the other obvious thing: if the model stalls, catch it and re-run the turn.

It worked, sort of. I built it, measured it, and then took it out. Two reasons.

The first is that it was a regex looking for certain phrasings in the model’s output. Any check like that breaks the moment a model update changes how it words things, and it breaks quietly - it just stops catching anything, and your dashboard looks fine.

The second reason is the bigger one. A retry can rescue a single stalled turn. It can’t carry a job that needs the model to decide to keep going forty times in a row. Even at 95% per turn, forty turns in a row is a coin flip you lose. All I’d really built was something that made the failure take longer to show up.

What actually worked

Out of everything I wrote, one line survived: “proceed” means act, not announce. That halved the stall rate.

Why did that one work when five other versions didn’t? Partly because by then I’d deleted the other four. They had been competing with each other the whole time. Cutting them is what let the last one land.

But the real fix wasn’t in the prompt at all.

The problem was that the loop depended on the model choosing to continue, over and over, dozens of times. So I took the loop away from it. The model now proposes the whole job in a single call, and a database table drives the execution - one row per item, a background worker moving them along, results collected outside the conversation entirely. The model isn’t running the loop anymore, so it can’t stall it.

That’s the pattern I keep running into. If you’re writing the same instruction five different ways, the instruction usually isn’t the problem. Something in the design is asking the model to do a job it shouldn’t have to do.

Putting a number on it

Once I knew the size of the prompt was a live variable, I stopped letting it be an invisible one.

There are two limits checked in CI now, and I keep them separate on purpose:

• One on the whole thing. This one is about money. It’s roughly 25,000 tokens on every turn of every conversation, trivial ones included.
• One on just the base text and instructions. This one is about quality. It sits at 29,000 characters, which is the low end of where I measured things starting to go bad.

Why two? Because with only a total, I could add 4,000 characters of new instructions, save 4,000 somewhere in a tool schema, come in under the limit, and make the quality problem worse while the check stayed green. The instruction text is under a quarter of the total, so the total can absorb a lot of damage without noticing.

Then I went looking for things to cut, and the measurement told me something I didn’t expect. I’d assumed the weight was spread thin across two dozen tools and I’d have to shave a bit off each one. It wasn’t. It was sitting in two places that each explained something for a second time. One of them described every tool over again, after the tool’s own description and its schema, both of which the model already receives in the same request. The other re-taught a query language that the base text already covered.

I deleted the copies and kept the teaching. That took the prompt from 108,743 characters down to 96,900. About 11%.

Then I lowered the CI limit by the same amount. If you free up space and leave the ceiling where it was, you haven’t saved anything - you’ve just made room for the next thing to fill it quietly.

Two things I decided not to build

Both of these looked right on paper.

The first was progressive disclosure. Show the model a one-line summary of each tool, let it ask for the full schema when it actually needs one. It’s the biggest single saving available, and it’s the first thing anyone suggests. I skipped it, because it buys those characters with the model’s turns - it has to go and work out what its own tools do before it can use them. Turns are the scarcer resource here, and I’d spent a while clawing them back from exactly this kind of overhead.

The second was an automatic duplicate detector. If the problem is text saying the same thing twice, write something that finds the repetition. I prototyped it and ran it against the two duplicates I already knew about, which is the exact case it was built for. It found 17% overlap. The second copy had been rewritten in different words rather than pasted, so the checker walked straight past it. It would have sat in CI showing green while the thing it existed to catch went through untouched.

The size limits catch the same problem without caring whether the text was copied or reworded, so that’s what I kept.

Why I didn’t notice sooner

Five rewrites each made things worse, and it still took me five of them to see it. That’s because I had nothing to compare against. No history, no diff, no way to ask what the prompt looked like on a day when the number was good. I was hand-editing the most sensitive input in the system, in three separate places, with no record of any of it.

That’s fixed now - every change is a snapshot with an author and a timestamp, and I can diff any environment against the repo or against another one. If I’d had that from the start I’d probably have caught this at revision two.

Takeaway

A system prompt isn’t documentation and it isn’t a config file. It’s a fixed amount of space that everything you write has to share, and nothing warns you when it’s full. Every sentence you add takes something away from the sentences already in there.

So measure it. Put the number in CI so it can’t drift on you. Split the limit, so saving money in one place can’t hide a quality loss in another. Version it like code, because it is code. And when you win space back, lower the limit by the same amount.

And if you’re on your fifth rewrite of a rule aimed at one behavior - stop. You’re not going to find the right words. Go change the design.

(One note on the 40%: I measured it before all of this and haven’t re-run that exact job since, so read it as where I started rather than where I am. All of it runs in production at scoreboardai.net.)