The gate approved a file name

Sapix checks what the assistant posts in my name before it leaves. Nearly half the time, what it checked was a file name. The text lived in a file the check never opened, and the log said passed. The fix I had shipped the day before, for another hole in the same gate, had never run once, and its test was green.

Sapix has a gate in front of what the assistant sends out in my name: a comment on an issue, a message to a group, an email draft. It reads the text before it leaves and refuses anything that breaks a short list of rules I hold to. In July I found this gate blind in another client, because it matched on tool names that client never used. This time the names were right, and I took that to mean everything the tool sent went through the gate.

The gate reads the request, though, and a request can point somewhere else. The tool that writes comments and issues takes the body written into the call, or the name of a file to read it from. Long comments are easier to write to a file first, so the assistant does that often. When it did, the gate read the file name, found nothing wrong with a file name, and logged the check as passed. In my main client, 401 of the 876 writes made through that tool since the start of August went out that way. The log looked healthy the whole time: a check, a decision, a pass.

It surfaced while the assistant was auditing a fix from the day before. The route my own rules prescribe for creating an issue had never passed through the gate at all, so the assistant added it to the list of tools the gate is shown and to the list it knows how to read, and wrote a test that the two lists agree. The test was green. The fix never ran. Between those two lists sits a small dispatcher that hands each tool to its check, and its default branch let the new tools through without a word. Twenty-five issue captures went through after the fix, and the gate saw none of them.

The test could not have caught it, and that is the part I keep turning over. It built its own example call and filled in the field the gate reads, for every tool, including one whose real calls never carry that field. That tool approves a stored draft: all it sends is an id, and the text lives in the draft. The test proved the gate accepts the example. Nobody sends the example.

Once the shape had a name, it turned up once more. Another client lets the model write a short script that calls tools, and the gate in front of it sees one call to the script tool and nothing inside it. In one week of August, eleven of those scripts wrote to issues: fourteen writes, one check.

None of it did harm that I can find. The twenty-three file bodies still on disk pass the gate’s blocking rules. The gap was structural, which is the kind that waits.

The fix was to stop checking the request and check the send. Every one of these routes, the file, the stored draft, the script, ends in the same place: the code that actually sends. It is also the only place that holds the text exactly as it will go out. The gate lives there now for Sapix’s own tools, and I proved it on real calls built so that nothing could be published if it failed. The request-side check keeps only the tools that belong to other servers, which Sapix cannot reach inside. That leaves one route open: a script in that other client can still call another server’s tools, and those are checked only at the request. No such server is configured there today, which is a fact about today, not a fix.

A check that reads a request can only judge what the request carries. When it carries a pointer, a file name, an id, a script, the check is judging the pointer, and a pointer passes every rule. Tests have the same blind spot. A test that writes its own input proves the code handles that input, and the only input worth testing with is the one real callers send.