Feels like starting over
Tuesday is “Patch-it day” in our office; it used to be a weekly feature, but a decision was made that monthly better fit our needs. So be it. However, there are often warnings and warning signs that slip by in these easier times. We have a number of servers that do experimental duty; the oldest in the farm is currently dedicated to some PHP/MySQL work. The machine also doesn’t have any strategy for recovery after something stops dead. Today, the machine had a lovely warning that a write failure had occurred; then the data drive was gone.
Of course, we now have no idea what data might be gone; the machine was there for the convenience of others. A cooler head prevailed, and the drive spent a couple of hours in cold storage. Not enough. About noon, the decision was made to simply start over.
So, when I get in tomorrow, I will re-activate that server, with some minor software upgrades. I will then notify the stakeholders that their months of effort are starting over. Nothing else will have changed. The oldest machine will carry on, with no backup strategy. The concerned parties will continue to believe, albeit with less conviction, that all is under control. And I will tiptoe past that machine with my fingers crossed. Isn’t that what all the pros do?
