There is a bit of a dilemma facing any company working in the cloud hosting industry: to upgrade or not to upgrade.
We've experienced the problem first hand recently when a storage area network (SAN) - a big data storage box that is core to hosting cloud platforms - suddenly went down without warning.
The cause? A firmware bug that told the server to shut itself down after 240 days' service. This would be a shameful enough bug to be found in an cheap, entry-level consumer platform, but this was a top dollar enterprise vendor.
To make matters worse, the vendor knew about the bug and had buried information about it in a release note issued months beforehand.
When we contacted the vendor in question they told us to upgrade... not to the most recent firmware version but a later version than the one we had. The implication of that was the latest version might come with its own bugs that had either not yet been found or not yet been resolved. The advice was couched in the usual vague language. Their engineer might as well have just said: "Don't sue us."
Since firmware upgrades are supposed to remedy defects and roll-out new features, it would be nice if vendors simply shipped finished product in the first place. In reality, this rarely happens. Even the biggest names in the industry - Cisco, Dell, Microsoft, the Linux Kernel, OpenSSL et al - regularly issue updates. The fact is that large scale debug testing is performed by customers.
Given the amount of money exchanging hands, the hardware industry really ought to be better at its testing. Given that isn't happening at the moment, the alternative is that they need to be more open about bugs.
Perhaps we have to share the blame at this point. Have us customers frightened the vendors with so many lawsuits that they can no longer afford to be transparent and tell the truth about flaws in their product?
How do we, as firmware buyers, know what to do if we can't get truthful advice from vendors? It is a quandary for the whole industry. So, what should we do when a new version becomes available. The main options are:
In theory this should be the most secure and advanced version. But it is also the most likely to have unresolved bugs.
If it ain't broke, don't fix it. Keep going with the release you know works until something forces your hand.
Hang around for a little while to see if you hear of any early adopters up in arms about the latest version, and apply the new patch only when you're confident it is without major flaws.
Each of those approaches has its problems. Which do you favour? We're genuinely interested to know, so feel free to leave a comment.
