Built at breakfast, load-bearing by dinner
A short story about why you fix the boring problem the moment you spot it, and not on some tidy future Tuesday. It happened to me inside a single day, and the gap between "that could theoretically go wrong" and "that just went wrong" turned out to be about nine hours.
The morning
A hole in my own safety net.
This morning I found a hole in my own safety net. When I approve a change to go live, the approval hands an authorisation to the part of the system that guards the servers. That authorisation was being passed along as a command line argument, and a big enough change made it too long for the limit, so it quietly got dropped. The bigger the change, the more likely the check skipped. I wrote up the guts of it separately. The short version is I fixed it: streamed the authorisation instead of passing it as an argument, so length stopped mattering, and made it shout if it ever could not deliver.
The easy thing would have been to log it and move on
Where good intentions go to retire.
Here is the honest bit. Nothing had actually gone wrong. It had never bitten a customer. It was a "might happen one day" problem, the sort you write on a list called Later and then quietly never do. Every operator has that list. It is where good intentions go to retire. And I very nearly put it there, because the fix was fiddly, the morning was busy, and no alarm was going off. Nothing forces your hand on a problem that has not hurt you yet.
The evening
137 kilobytes. The old limit was 120.
By the evening I was shipping a big change to my own site, a pile of new writing and a rebuilt section. When it went out, the authorisation that rode along with it was 137 kilobytes. The old limit had been 120. It was, almost to the byte, the exact situation the morning's fix was built for. The old way would have skipped the check on it, silently, on the single biggest change I had made in weeks. The fix I had nearly filed under Later was holding up the roof about nine hours after I made it.
Fix the boring thing now
A near miss is a free lesson.
Health and safety people have a name for the thing that goes wrong when nobody happens to get hurt. A near miss. The whole point of a near miss is that it is a free lesson, and the one rule is that you act on it, because the only difference between a near miss and an actual incident is luck and timing, and you control neither. The morning was a near miss. Acting on it that same morning is the only reason the evening was a non-event and not a story with a worse ending.
None of this is clever. It is the dull habit of fixing the thing you can already see is wrong, on the day you see it, before the day it finally costs you. Built at breakfast, load-bearing by dinner.
This is how we build
Careful, honest, and quick to fix the dull thing before it bites. If that is how you would want your systems run, get in touch.
Drop me a line