I heard it again last week. A good senior engineer, pushing back on a proposal to replace a five year old batch job. "It works. If it ain't broke, don't fix it."
Sounds like discipline. Sounds like someone who has been burned by a rewrite before.
It is the most expensive sentence in software engineering.
"Not broke" is a guess, not a fact
The rule treats "working" as a property of the system. It is not. Working is a property of the system and the conditions it happens to be sitting in right now.
Your batch job works. Under this load. With this data shape. On this OS version. With the one engineer who understands it still employed. Change any of those and you find out whether the thing was ever sound, or whether it was lucky.
Nobody knows which one you have. "It ain't broke" means "nobody has stressed it hard enough to find out". Those are two different statements, and teams keep treating them as one.
The bill comes due, and it is enormous
Look at what happens when a whole organisation follows the rule for decades.
The US Government Accountability Office found federal agencies spend about 80 percent of their IT budget on operating and maintaining what already exists. Roughly 20 percent is left for anything new. The federal government spends over $100 billion a year on IT.
In 2019, GAO named 10 critical legacy federal systems. They ranged from about 8 to 51 years old. Together they cost around $337 million a year to keep breathing. Some ran COBOL. Some ran on hardware and software no vendor supports any more, with known security holes still open.
Every one of those systems worked. Every year, someone looked at it and said it ain't broke. Fifty one years of nobody fixing it, and the bill is $337 million annually plus a security posture no one would sign off on today.
Sit with the 80/20 split for a second. Four dollars in five goes to standing still. Your company is likely running a similar ratio and calling it prudence.
Netflix went the other way
In 2011, mid migration to AWS, Netflix built a tool to kill its own production servers at random. Chaos Monkey. Nora Jones, Casey Rosenthal and Greg Orzell were behind it.
The reasoning was blunt. Failures happen anyway. Waiting for one to happen at 3am is a terrible way to learn. So schedule them for Tuesday at 2pm, when everyone is awake and caffeinated.
They open sourced the code in 2012, then scaled the idea up. Chaos Gorilla kills an entire AWS availability zone. Chaos Kong drops a whole region.
Netflix did not invent this. Amazon got there first: Jesse Robbins started "Game Day" in 2003, deliberately causing major failures on a schedule, an idea he borrowed from firefighter training. Google started DiRT, its disaster recovery testing program, in 2006. Facebook runs Project Storm to simulate losing whole data centres.
Four of the most reliable engineering organisations on earth arrived at the same conclusion. Break it yourself, on your schedule, or the universe will break it on its own.
The discipline got a name and a definition. The Principles of Chaos Engineering put it like this:
"Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production."
Read the goal again. Build confidence. Not find bugs. Confidence.
You do not get confidence from a system nobody has ever stressed. You get hope.
Break it on purpose, on a schedule, with a plan
Breaking things at random with no method is called a Friday deploy. Do not do it.
The published principles give you the method. Four rules, plus one:
- Start with a hypothesis about steady state. Pick a number you already watch. Stream starts per second. Checkout completions per minute. Write down what normal looks like, then predict it holds when you pull the plug.
- Vary real world events. Kill a node. Fill a disk. Add 400ms of latency. Expire a certificate. Model the failures sitting in your own incident log.
- Run it in production. Staging lies to you. Different data, different traffic, different neighbours.
- Automate and repeat. A one off experiment tells you about one Tuesday. Continuous experiments tell you about your system.
- Minimise the blast radius. Start with one instance and one percent of traffic. You are hunting information, not headlines.
The last rule separates engineering from vandalism. Skip it and you deserve everything coming to you.
The tooling is not the interesting part. Any competent team ships a pod killer in an afternoon. The interesting part is the flip in default. Instead of "prove to me this is worth touching", the default becomes "prove to me this survives being touched".
The version you run with no tools at all
Most teams are nowhere near ready for Chaos Kong. Fine. The mindset scales down, and the cheapest experiments need no infrastructure whatsoever.
Hide the runbook for one drill. If your recovery depends on a document nobody has opened since the day it was written, you have a broken system wearing a working system's clothes.
Send your most critical person on holiday. Pick the engineer everyone routes questions to. Give them a week where nobody is allowed to contact them. Whatever falls over is your real architecture diagram.
Restore from backup this quarter. Not verify the backup exists. Restore it, into a real environment, and time it. Zero byte backup files, written faithfully every night for months, are a classic. Teams find out mid outage.
Kill a dependency and watch the alerts. If nothing pages, your monitoring is decoration.
Each of these costs an afternoon. Each one converts a guess into a fact.
The rule rots teams too, not only systems
Here is where I get less popular.
"If it ain't broke, don't fix it" is not confined to code. It is how most management operates. Nobody is shouting, so leave it. Attrition is normal for our sector, so leave it. She has always run standup like a status parade, so leave it.
Same rule. Same compounding bill. With people the failure mode is quiet, and by the time it surfaces the good ones have already gone.
My own research into bad bosses found 99.5% of respondents said they had experienced one or more types of bad boss. Near universal. And almost every one of those bosses was, by their organisation's own measure, not broke. Delivering. Hitting targets. Nobody escalated.
Nobody ran the experiment.
A team with no drill has no idea whether it recovers. A promotion process nobody has stress tested is not fair, it is untested. A culture nobody has poked is not healthy, it is unexamined. I work on this side of the problem at Step It Up HR, and the pattern matches the one in the server room exactly. Systems degrade quietly. Only deliberate stress surfaces the rot while you still have options.
Pick one thing this week
Do not rewrite the monolith. Do not run Chaos Kong on a Friday.
Pick one system everyone agrees is fine. Write down what "fine" means as a number. Then break one small piece of it on purpose, during working hours, with the team watching and a rollback ready.
One of two outcomes follows. It holds, and now you know instead of hope. Or it falls over, and you found out on a Tuesday afternoon with your best people at their desks, rather than at 4am with a furious customer on the phone.
Both outcomes are wins. Neither is available to a team who never touches anything.
So: which system in your stack has nobody dared touch in three years? And what does everyone assume about it, with no one ever having checked?