All Aboard! The Upgrade Train
(brought to you by AI... I mean the constant upgrades... not the article... that was all me... and my spell checker)
As IT professional(s), I (we) have been on the constant upgrade cycle. So much so, Microsoft created a monthly 'holiday' where we get 'gifts' from them.
The Problem
Different IT environments get different levels of tolerance for upgrades/patching. For Microsoft servers, there is usually an agreed upon window of time where 'upgrades' are done and the business knows to 'take a break'. However, additional IT environments are trying to join those Microsoft 'take a break' windows and a complete mess is ensuing. Storage patches need to go into place impacting the actual server storage the Microsoft servers rely on. And network upgrades are impacting both of them, leading to situation of:
No network,
No storage,
No servers...
No business
AI hasn't made that easier. Besides the flood of 'slop' patches, AI is actually finding real bugs and real vulnerabilities. But AI is doing it at a rate that neither the software developers nor the business users can handle.
Is there an Answer?
Honestly, not without a lot of work.
I've heard some people floated the idea that 'we just need to get thru this period and eventually all the bugs will be found and we will have perfect software'. In a static environment, that may be true. But who lives/works in a static environment? I think that there are several things that are working against this idea:
- Developer fatigue: Researching and understanding a bug and then developing a patch and patching a software is not an instant process. It takes time. Time no one has. With the amount of reports coming in, we have a choice... drowned or fight AI with AI. Call in the life line of AI to evaluate, propose, and patch the bugs. How soon will it be before we just press the 'I believe' button and tell AI... just go do it? How soon will it be before we can no longer reason about the software?
- The AI models keep getting better and finding more kinds of bugs. More reports, etc. etc. The key here is AI's ability isn't fixed.
- Model availability: Or the lack there of. Not all models are available to everyone. Many models have been restricted to just some 'elite' companies for special purposes. Everyone else... you just get the scraps. Good Luck!
- Model affordability: At some point in time the full cost of running models will come due. And those companies and/or individuals that cannot afford to pay, will not be allowed to play. At the risk of speaking something into existence... how soon we will start seeing 'Go Fund Me' campaigns for tokens?
Let's assume that none of those items manifests itself. Where does that leave us, the IT pro, who is trying to do the best we can, not having to upgrade the literal entire IT environment every... month, week, day? Here are my suggestions...
Short term: the business, as it is...
- The business will need to have more 'downtime' in a month. If server patches are done on Sunday, then Storage patches are done on Saturday and Network (inclusive of firewall, WAFs, proxies, etc) patches are done Friday.
- We will be introducing a lot of change. We will want to develop some good 'canary testing' methods to be reasonably sure when an upgrade is complete, everything is 'as we expect it'.
- If you are in a business whose size makes a single patching day per area impractical, then that business will need to have more patching days per month. Maybe geographically related. (East coast on first week, West cost third week)
There will be pushback. But remind those that push back that patching systems doesn't prevent the business from making money... it protects the business so that it can continue to make money. Think of it like an insurance. Bad stuff will happen. This is you paying your premiums so that when it does happen, the impact can be recovered. If that is too nuanced, just think of it like this... It's just the cost of doing business.
Long term: the business, as it will need to become...
The business (and IT), will need to transform its thinking. Instead of relying on servers, and storage, and network we need to start thinking in terms of services. No business buys a server or storage or network gear just for that gears sake. (IT geeks excluded) That gear is purchased to provide a service. We need to begin thinking about those services. Service is important. Servers are not.
How do we do this?
- Assume failure: Any server, storage, network device will fail. We have developed processes to try and weed out those failures early and to minimize them... but they do and will continue happen. Let us not 'hope they don't happen'. Let us assume they will and have a plan.
- Stop worrying about the shortest path: What is the shortest path between your machine and Google? Microsoft? You don't know. And you don't need to know. So stop trying to optimize it. The only thing our users care about is they can get to the service they desire, when they desire them.
- Re-architect applications: You will need to re-architect the application the business relies on. Just shoving the servers into a "cloud" and calling it "cloud enabled" is a lie. We can no longer lie to ourselves or allow ourselves to be lied to. What does re-architect mean? In a nutshell it means:
a) separating your data from its presentation layer (where your customers, employees, etc. interact with the data).
b) re-structuring how the data is organized so that it can be easily chunked up and replicated across physical locations.
c) and while we are at it, we need to plan in how we backup and restore the data when it is corrupted or lost.
"Does this mean I must move to the cloud and pay the cloud tax?"
No. You will need some multiple places to put servers, storage, network hardware in our environment. It also means you will need start thinking of your server rooms and data centers like the cloud providers do. Modular-ize. Container-ize.
"Now that I've done all this work what does my business get out of it. I still have to patch, upgrade, etc."
Correct. But now your application is setup to be "always on". Now your users can work from the office or their home or anywhere.
And now we're back to...
"How do I re-architect my application(s)?"
Well... AI. Can AI build a perfect application? No. AI can however, with good prompting, planning, validation and oversight, take an ancient historic application, and rebuild it in a modern, safe fashion using today's tools and methods. Slop is still possible. But maybe, a slop application is all that is needed for a specific service. Today's AI can build more than just slop. It can really customize an application for your specific business. Instead of what we all have done for the last 30 years which is change our business around the applications we could afford to rent.
Conclusion?!?
Instead of being stuck on the track and being run over by all the fallout of AI... can we use this time of change to help ourselves, our businesses, our families, achieve a better outcome?