Thursday 12 August 2010

Are your backup systems robust enough?

My previous blog talks about the problems getting Windows 7 workstations to attach to a Windows 2003 domain server. In that blog we talked a problem we had with our domain server. This blog is to add more detail to those comments.
Our server failed unexpectedly. No warning signs, just a shutdown of access for workstations. Believeing this to be one of thos unusual things we reset the server. After 10 minutes we had no service and after attaching a monitor we found that the server had not completed POST and BIOS setup.
Several attempts met with similar failure.
We rebuilt the server but when we attached the disks from the old server to the new one to copy off the data we could not access any data.
The last time we had problems with this disk system we had set it up without the mirrored RAID so one level of protection was missing.
However we do have an external hard disk which was set up to do a complete backup of the data EVERY night. No problem then. Attach the external disk. We see a perfectly good looking .bkf file but when we tried to recover it with the Windows Backup & Recovery tool it told us there was no recoverable data on the disk.
Never mind, we still have our online, in the Cloud, backup system. We have not stored everything but the major items are still up there. We download the backup client software to do a restore. It tells us that we have used 2GB of our 3GB allowance but wait.... When we try to recover the files it says there are no files selected for recovery or that there is no data to collect. The help desk lived up to the antithesis of their name and I still have no satisfactory answer from them.
From this it appears that we have lost all our Client files from the last 6 years, all the accounts data, all the management files (policy documents, logos, letter templates, contract templates etc), all our supplier files, and all the downloaded data (device drivers and software used to build client machines).
We have called in the services of a Disk specialist who has been able to recover most of our data but before you say 'What is all the fuss for' then consider this.
We limped along on Friday. We paid overtime on Saturday to rebuild the server (luckily we did not have to pay the cost of new hardware as we could use some kit that was redundant). We have not been able to work on Monday, Tuesday or Wednesday.
This doesn't affect many of us as there are only 3 of us in the organisation but how would it affect you?
What is the cost to you for paying someone to replace and rebuild your computer system?
What is the cost to have your staff being idle for 3 days?
What is the cost to you not being able to fulfill client orders?
Add to that the cost of calling in a specialist to attempt to recover you data.
This is not cheap, It could easily run into 5 figures. Put into perspective what it would cost to have a completely robust backup and recovery system in place.
This has happened to us. We might have got through it because we had second and third line backup systems in place. Don't let it happen to you. Having gone through all this we are now better placed to help you avoid the pitfalls that we have fallen into.
Call us!

No comments: