The rule nobody could break

Every night my system reads the notes I have written and proposes connections between them. A judge decides which pairs are worth linking. Its instructions include a rule I was rather proud of: each note carries a shape, a one-line description of its underlying pattern with the subject matter stripped out, and a real link needs the two shapes to agree. Two notes that merely share a word are noise.

That rule had rejected nothing. Not once, in five months.

I noticed because I was chasing a different problem. The judge kept linking notes that look alike and are not: two notes containing the word “Portuguese”, one about naming variables and one about how to address a group of people. I had a small hand-labelled set of pairs with traps like that in it, and the judge kept falling for all three.

So I checked how often the shape rule was actually firing, and found that the shape was missing from every single note the judge ever saw. All of them. The rule read “compare the shapes” and the shapes said, in every case, none.

That felt like a complete explanation. The judge had been told to use a discriminator that did not exist, so it fell back on matching words, so it linked homonyms. I wrote that up as the diagnosis, put it in a merged change, and filed the fix: generate the missing shapes.

Filling the field

I generated them. Two and a half thousand notes, a few hours, a real cost. Coverage went from zero to ninety-nine percent. Then I ran the same eleven pairs again, expecting the traps to fall away.

The traps did not move. All three still linked.

And something else had happened. A pair that the judge had always got right, two notes about the same holding in my portfolio, one about what I actually paid for it and one about the tax deadline for selling it, was now being rejected. I had made the thing worse in a place it had been fine.

I asked the judge why. It answered:

Both notes are highly specific data points within the same financial project, but their conceptual shapes describe different problems.

Read that twice. It saw that the two notes are about the same concrete thing. It said so. And then it rejected them anyway, because my rule told it the shapes must agree, and it stated that requirement flatly, with no exception for the obvious case where two notes are about one subject from two angles. What I paid for a share and when I must sell it are the same subject and two different patterns. Of course they are.

The rule had been wrong since the day I wrote it. It had simply never been in a position to demonstrate that, because it had never received an input it could evaluate. A conditional whose test always reads unknown is not a lenient rule or a strict one. It is not a rule at all. It is a comment with syntax.

The part I got wrong about my own work

I had thought of the backfill as a data task. Filling in a blank column. Housekeeping.

It was a deployment. There was a piece of logic sitting in production that had never executed its interesting branch, and the thing I did that afternoon was switch it on. Not by changing a line of code. By giving it, for the first time, something to look at.

Nothing about that is visible from where I was standing. The code had tests and they passed, because the tests supplied shapes and exercised the branch. The code had a consumer and it ran nightly. Every signal said working. The only thing missing was any real occasion for it to say no, and that absence is silent by construction.

So there is a question I now ask before backfilling anything, and it takes about a minute: who reads this field, and what do they do the moment it stops being empty? If the answer includes a branch that has never run against real data, then I am not filling in a column. I am shipping a feature, and I should measure it the way I would measure a feature, with a before and an after.

And the traps?

Not the shapes at all. My diagnosis had been confident and wrong twice over, so I stopped guessing and just measured four different models on the same eleven pairs. The small local model I had been using got three traps wrong. A mid-sized one got one wrong. A large local model got all three right, with perfect precision.

So did the frontier cloud model. Exactly the same score, right down to missing the same single subtle pair.

I had been braced for an uncomfortable trade: better judgment about my personal notes in exchange for sending them off my machine. That trade turned out not to exist. The local model ties the expensive one, and is faster than the small one it replaces, because it activates only a fraction of itself per token. The whole dilemma evaporated when I finally had numbers instead of an intuition about what big models are for.

Two things I would not have learned by reasoning harder. Both took an afternoon of measuring things I assumed I already knew.