Major Disater Recovery Invocation at IT Ambulance

Well suddenly the reason the IT Ambulance Head office move date was being kept secret is now apparent.

It’s fair to say that IT ambulance is keen on “eating its own dog food” in other words testing and using our own products and procedures on itself so as to promote confidence in providing these to our large and happy customer base but this takes the biscuit.

On Thursday 23rd September the management team announced that far from moving on the 30th September or 1st of October we would be moving on Friday 24th September and that we were to have servers off site to DR locations that evening with the entire HQ team working from our planned DR locations or home until Tuesday 28th September. 3 days with no IT Ambulance HQ!

This was to be treated as a Denial of Premises Disaster Plan invocation. The performance of our plan was to be completely monitored end to end and lessons learned documented and built into our plan.

“This is one of those rare occasions that a company gets the opportunity to test its Disaster Preparedness in a full blown bugging out scenario. Our customers trust that we know our business. Lets show them that that trust is well placed. Our business now is in Longmead.” said IT Ambulance Managing Director, Frank Connelly addressing the team.

Well that day was a bit of a haze as crates delivered by the removal company were hastily packed with the contents of Desks and Cupboards. The Shaftesbury IT Ambulance Van assumed its role of ferrying our critical Server infrastructure including our primary Datavault servers to their bugging out locations.

Customers and Suppliers were mailed informing of the change, mail redirection and even business cards were ordered in accordance with our check list.

Our Technicians rose to the challenge immediately and with enthusiasm and by 6pm our servers were un-racked and in transit.

The Datavault Service was unavailable for less that 1 hour with back ups immediately resuming without any loss of data. Product Checklist -BIG TICK, although confirmation mails were disrupted. Lesson learned.

Our own mail service was down for nearly 2 hours. This was because having un-racked and disconnected our servers for transport we had an out of hours support call that required access to information on our encrypted service. Lesson Learned. However the customer comes first!

Mobile phone users were texted details of new connections for email services. Blackberry users were unaffected. Point to Blackberry as Iphone users reset server addresses.

We think that we can get this down to less than an hour too based on this experience.

The staff move went very well. Our phone system was specifically designed for portability and the hunt groups and transfers between handsets went off without trouble. Laptops from the office and home PCs were utilized to log in remotely to our Terminal server running RDP. True systems portability tested and working! Office VOIP phones were simply plugged into routers at home and one by one became available.

Meanwhile we moved office! We were very impressed with the service provided by  Finns Removals of Gillingham.  They had only been booked that week out of the local paper but came with glowing references. They arrived onsite on Friday at 8AM and had delivered to the new office by 2PM.

One difficulty in the plan and the secrecy was that the move brought into direct conflict the office move and the installation of the new network at our office. However this was thought to be worth it to test the validity of the plan. Luckily  and in no small part to the patience and skill of our fit out team the cabling was completed on Saturday a bit ahead of the scheduled Monday forecast. Nice clean job!

Saturday afternoon some of the office team arrived and moved the desks and crates to final position for unpacking back into desks and cupboards. This step would have been completely avoided if it were not for the scenario as the fit out team needed access around the desks.

Monday morning team member drifted in on a rota basis to sort their workspace out and some found nice new kit to play with! Also a simulation of rebuilding lost or damaged equipment. Clocks were ticking now to have all this running and in place before the end of the day.

BT came good for us bringing in the delivery date for all our new phone lines and we leveraged our great supplier relationships to ensure the rest of our services were migrated equally smoothly. Boy were we glad to see their engineers though! They rolled in Monday afternoon and brought sockets through to our super new server room. Broadband speeds about 10% improved by virtue of being closer to the exchange.

Ultimately, the test of how well we did is not with us though it is with our Customers. I wonder how many noticed that this occurred at all?

A big thank you to everbody involved in this exercise. See you at the debrief!

Is Disaster Recovery something you take seriously as a business?  Do you know how to deal with a Disaster of this scale? We know most of our competitors seem to stop at back ups. DR is burnt into the conscience of everything IT Ambulance provides.

To discuss how IT Ambulance can protect your Infrastructure investment and guide you towards an inclusive strategy for minimizing risk call 0800 0214 999