The safety check my biggest changes were skipping
This one is for the technically minded, and it is a decent reminder that a safety net is only as good as the bit of it nobody tested properly.
The setup
Belt and braces on every change.
The setup: I let an AI agent do the hands-on work on my systems, and nothing reaches a live site without a change I approve with my fingerprint. When I approve one, an authorisation for that exact change gets handed to the part of the system that guards the live servers. Belt and braces. The guard should never let a change through without it.
Where it quietly broke
A length limit nobody thinks about.
It worked a treat. Right up until it did not, in the quietest and most dangerous way it could have.
The authorisation was being passed along as a command line argument. If you work with systems you can already see it coming. There is a limit on how long a single command line argument can be. A small change made a small authorisation and it sailed through. A big change, the kind with a long list of edits, made an authorisation that went over that limit, and instead of failing loudly it just got dropped. The change still went out. The check simply did not run.
The hole was where the heavy stuff lands
The bigger the change, the more likely it skipped.
Sit with that a second. The bigger and more important the change, the more likely it was to skip the very check meant to catch it. The hole in the net was right where the heavy stuff lands.
Nothing came of it, because the discipline around it held and it got caught. But it is a textbook version of the failure that actually gets people. Not the obvious gap you would spot in a five minute demo, but the one that only opens up under size, under load, under the exact conditions that matter most. Your test change is small, so it passes. Your real change is big, so it never gets checked. You would never see it coming from a quick try on a slow afternoon.
The fix, and what is worth keeping
Test the ugliest input, not the tidy one.
The fix was simple once we saw it. Stop passing the authorisation as an argument with a length limit, and stream it instead, so size stops mattering. And make it shout if it ever cannot deliver, rather than shrug and carry on.
The bit worth keeping is dull and it works. Do not just check your safety net catches something. Check it catches the biggest, ugliest, most awkward thing you can throw at it, because that is where it will quietly let you down, and that is the worst possible place for it to happen.
For the techies
The short version of what actually sits under this.
This is how we build
If you want systems run with this much care, right down to the failure nobody else tests for, that is what I do.
Drop me a line