IT Ambulance Disaster Recovery Planning Saves the Day.

When the IT Ambulance monitoring and management system alerted our Network Operations Centre that one of our customers servers had gone off line suddenly, the IT Ambulance support team were put on standby to resolve the issue as a standard IT Support Issue. However even after 9am we were unable to raise the site.

Shortly after this we were contacted by one of our clients senior staff trying to get to work at their head office being hampered by a fire that had closed of the High Street in Staines, Middlesex. It was clear a major incident loomed. All Telecoms and Electrical connectivity to site seemed severed. Whilst the fire had not directly hit our customers premises their building was currently ringed with buildings on fire, in various states of collapse and at risk from volleys of exploding gas cylinders. Experience of this kind of incident told us that the Incident Fire Commander, who has total control, would not allow access for some days. Typically this kind of fire takes a matter of days just to damp down.

Our standard “denial of premises” planning had already kicked in. On site the servers would run on UPS support until the batteries were depleted and at a predetermined battery life remaining they would begin an automated shutdown, in an elegant manner, to preserve data integrity. As soon as the mail server became unavailable our email continuity solution had automatically kicked in, email being stored in the cloud until the servers became available again, with webmail available until that time. We were able to immediately give access to this vital communication media so that our customer could maintain contact with their customers throughout this event.

Our team at IT Ambulance, Park Royal, London NW10 began the process of preparing machines from of our stock of DR Loan Servers ready to restore from the off site back up. They delivered a full recovery onto this alternate hardware that day, ready to take to site if possible but the decision was made to host these in our own Data Centre with remote access to our customers team working from home. Our design utilising virtualisation of this infrastructure utilising Microsoft’s Hyper-V technology really lends itself to this kind of rapid recovery. IT Ambulance continuously maintains the capacity to host this business continuity infrastructure and provide connectivity on standby for just such situations.

The following day the Incident Commander granted very limited access to the site to assess damage and critically this offered the opportunity to recover the on site IT Infrastructure. IT Ambulance were able to support the checking and reconfiguration of the hardware to support our customers relocation to temporary accommodation nearby without any delay. This saw the customer back to “Business as Usual” in new premises within 48 Hours of being alerted to the event.

From an IT and operational perspective there was very little in the way of downtime throughout this event. Email communications were maintained and our customer had a number of options to choose from in how their business continuity was supported to suit their evolving situation. IT Ambulance has a well proven “Playbook” for these situations that really shows its worth in an incident of this scale.

It is estimated that the Industrial Estate will remain closed for some months.

How would your business have faired in the same event? Could you use IT Ambulances expertise in DR Planning or in execution of your plan mid event?

Some great pictures of the havoc this fire caused in Staines can be found here: