A hundred decisions later

In part 1 I argued that agents follow a short list they always see far better than a long handbook. In part 2 I said the layer that holds a team’s own calls, case law, is nearly empty.

Suppose you fill it. Every time the team settles something, it goes into DECISIONS.md, next to CLAUDE.md, DESIGN.md and the rest. A year in, the file is long. Does the agent still follow it? I tested it.

The test

I used Deploy Board, an open-source app by Yuri Mandrikov at Evil Martians, built as the reference app for an AI design-system skill.¹ The thinking behind it is in Evil Martians’ post on design-system guardrails for AI.² It’s a small dashboard for deployments, and it comes with what a careful team gives an agent: a DESIGN.md, an AGENTS.md, 22 component contracts, patterns, tokens, the official shadcn skill and a design linter.

To that I added a DECISIONS.md with 50 or 100 team decisions. Five of them, in the middle of the file, cover the jobs I asked for.

The middle column is a baseline: ten builds with no decisions, two per job. All ten built the habit, not the team’s call. A rule only means something if the agent wouldn’t do it anyway.

Claude Code with Opus 5.5 did each job four times at each size, two ways: reading DECISIONS.md when it needed it, which is what a line in CLAUDE.md asks for, or with the whole file loaded into every session. That’s 80 builds. A script rebuilt each finished app, used it in a hidden browser, removing a deployment, typing a bad version, reading the colours, and recorded what had been built.

The file was already too long

Claude Code doesn’t hand the agent everything a command prints. In the version I used, anything over about 30,000 characters is set aside: the agent sees the first 2,000 and a note saying where the rest is. 50 decisions come to 33,000 characters, 100 to 65,000. Every build that read the file hit the limit.

This is what one agent got back when it printed the 100-decision file, along with DESIGN.md:

So it searched the file for its titles instead: 100 lines, one per decision. Line 281 read:

It opened four short stretches of the file, that entry among them, and built a button that says Remove.

It followed every one

That’s 80 of 80, against none of ten without the decisions. Every reply also named the decision it was following.

It makes its own cheat sheet

How it got there changed with size.

At 50 it mostly went around the limit and read the lot. At 100 it more often skimmed. That’s the cheat sheet from part 1, built on the spot out of the titles. It worked because each title was a rule in its own right, and the five that mattered matched their jobs. A title like “Remove flow” would have given it nothing to go on. I didn’t test vague titles, and that’s where I’d expect this to break first.

The titles are the cheat sheet. Write them as rules.

What grows with the file

Tokens are how much text the AI reads and writes. Cost is at pay-as-you-go prices; on a subscription it comes out of your usage limit instead.

Loading the file into every session bought nothing here: both ways scored 100%. At 100 decisions it made each build 38% more expensive than one with no decisions, because the agent carries the whole file through every step. Reading it when needed added 22%.

The other thing that grows is noise. Every reply named the right decision, plus about four others at 50 and five at 100 that hadn’t shaped anything. Whoever reviews the work has to read past them.

When the team and the design system disagree

A longer file means more chances to contradict the design system. Earlier, in 140 builds of a smaller app, I gave the agent team decisions that disagreed with its DESIGN.md.

The one sentence, added to CLAUDE.md, the instructions file the agent reads at the start of every session:

Without it, the team usually won, because nothing said otherwise. With it, the agent followed the policy every time, and all 60 replies said what disagreed. None of them put the clash on the list of decisions waiting for review, though. Each flag lived in one reply and scrolled away.

What we learned

1. 01
   You don’t have to keep the file short for the agent’s sake. At 100 decisions it could see only a fraction of the file at once, and it still followed every decision I tested. Beyond 100 is untested.
2. 02
   The title does the work. Past the cut-off, the agent chose what to open from the titles alone. A title that states the rule, and where it applies, gets found. A label like “Remove flow” wouldn’t. Writing the title is the most important part of recording a decision.
3. 03
   Loading everything up front is a cost, not a safeguard. Reading the file when needed worked just as well, and cost less as the file grew.
4. 04
   A decision only counts if it changes something. Without the decisions, the agent built its own habits every time. A decision that matches what it already does is agreed with, not followed, so check what the agent builds without one before trusting it.
5. 05
   What grows is the reading for people. The agent kept up with 100 decisions. What grew was everything a person has to read: more side decisions named in each reply, and clash flags with nowhere to go. The open question isn’t whether the agent keeps up with the file. It’s whether the team keeps up with the agent.

This is one model, Opus 5.5 in Claude Code, on one project, with up to 100 decisions, most of them drafted with AI and checked by me. A script and I did the scoring, with no second person checking. Treat it as a strong signal, not a law.

Finding the page

I build Arbiter for this layer. Its default limit is 40 decisions, and I tested two and a half times that.

I expected a long file to fail the way the handbook in part 1 does. It didn’t, because the agent never tried to read it like a handbook. It read the titles like a contents page and turned to the one it needed. Length wasn’t the problem. Finding the page was, and the title is how the agent finds it.

Next, I want to see what raising Arbiter’s limit would do: 200 and 400 decisions, and vaguer titles, like the ones people write in a hurry.