There is an interesting trap in IT and software development. Sometimes we get really good at fixing a problem. So good, in fact, that we stop asking why we keep having to fix it.
A ticket comes in. Someone recognizes the issue. They know exactly what SQL query to run, which service to restart, which configuration to change, or which workaround to give the user. Problem solved. Ticket closed. Everyone moves on.
Then it happens again. And again. Eventually, what started as an exception becomes routine. That is when fixing the problem can actually become more expensive than solving it.
We Often Measure the Wrong Thing
Most technology organizations measure things like response time, resolution time, ticket volume, SLA compliance, and backlog size. Those measurements are important, but there is another question we should be asking:
Why did this ticket exist in the first place?
Closing 100 tickets quickly may look great on a dashboard. Preventing 80 of those tickets from ever being created is much better.
Think of it like repeatedly putting water into a leaking bucket. At some point, instead of getting better at filling the bucket, you probably need to fix the leak.
Technology organizations aren't much different.
When the Workaround Becomes the Process
Workarounds aren't necessarily bad. Sometimes they're exactly what we need. Production is down. A customer can't work. A critical process has stopped. You get things running again first. That's responsible operations.
The problem begins when the temporary workaround quietly becomes permanent.
Someone documents the workaround. Someone else learns it. Eventually the support team knows it. Then a new employee is trained on it. Before long, the organization has created an entire business process around something that was never supposed to exist in the first place.
At that point, we've stopped treating the workaround as temporary.
We've designed around the defect.
Tribal Knowledge Can Hide the Real Cost
This becomes even more dangerous when experienced employees make recurring problems look easy.
You've probably heard something like, “Oh, when that happens, just ask John. He knows how to fix it.”
That might sound reassuring, but what happens when John is on vacation? What happens when John leaves the company? More importantly, why does John need to keep fixing it?
Good people often compensate for bad processes. Because they're good at it, the organization sometimes doesn't realize how broken the underlying process actually is.
Every Repeat Has a Cost
One recurring problem might only take 15 minutes to resolve. That doesn't sound significant. But suppose it happens four times a week. That's an hour a week, or roughly 50 hours over the course of a year.
Now multiply that across several employees, environments, customers, or systems.
That still doesn't account for interruptions, context switching, investigation time, customer frustration, QA, management attention, or the opportunity cost of what those people could have been doing instead.
Suddenly, the little 15-minute problem isn't so little anymore.
Recurring problems compound.
The Ticket Should Leave Something Behind
This is why I believe a support or production ticket should produce more than a resolution. Whenever possible, it should leave behind knowledge.
What happened? What was the root cause? How was it diagnosed? What evidence confirmed the problem? What was done to correct it?
And most importantly: What would prevent this from happening again?
Sometimes the answer will be a code change. Sometimes it will be architecture or configuration. Sometimes it will be documentation, monitoring, training, or a better business process.
And sometimes the answer will simply be that fixing it permanently isn't worth the investment. That's okay too. The important thing is that we made that decision intentionally instead of allowing a workaround to become permanent simply because nobody questioned it.
Don't Confuse Activity With Progress
There is something satisfying about closing tickets. You can see the queue getting smaller. You can measure it. You can report it.
But activity and progress aren't always the same thing.
A team that closes 1,000 recurring tickets isn't necessarily performing better than a team that identifies the root causes and prevents 500 of those tickets from ever being created.
One team became better at fixing problems. The other became better at eliminating them.
That's an important distinction.
The Goal Isn't Zero Problems
Software will have defects. Infrastructure will fail. People will make mistakes. Requirements will change. Unexpected things will happen.
That's technology.
The goal isn't to create an environment where nothing ever goes wrong. The goal is to make sure we're learning from the things that do.
Because the most expensive problem isn't always the catastrophic outage everyone remembers. Sometimes it's the tiny problem everyone has learned to live with.
The one we've fixed hundreds of times. The one everyone knows the workaround for. The one nobody stops to question anymore.
Sometimes the most expensive problems are the ones we've become really good at fixing.
Thanks,
Michael Cronin
Website: https://www.michaelcronin.info
LinkedIn: https://www.linkedin.com/in/michaeltcronin/details/experience/
