What the Verizon Services Outage During the Derecho Storms Teaches Us about Business Continuity…
Verizon is currently in a lot of heat with the Virginia State government for the 911 service outage that occurred during the Derecho storms in late June this year. Not only did their Internet and phone services go down for many customers in the DC-MD-VA metro area, but their critical 911 service went out as well. In the August 16, 2012 Washington Examiner article, it states that in a report coming out next week, “Verizon acknowledges that drained batteries, faulty generators, and human errors were the causes of service problems in Fairfax and Arlington counties and elsewhere,” which means that all the back-up methods they had in place to combat a large-scale power outage scenario did not prove to be effective or reliable.
So here are some important things to learn from Verizon’s Business Continuity (BC) failures:
- Even the biggest companies experience faulty back-up planning for emergency situations. That doesn’t make it ok, it simply means that no one is completely exempt from failures.
- You have to thoroughly plan your BC procedure and examine all possible destructive outcomes. Yes, your building is less likely to blow up – but that doesn’t mean you can’t have a plan in place for the least unlikely outcome.
- Analyze and determine what your critical functionalities are; then, make sure each one is covered by a comprehensive back-up plan. Not everything running in your office is 100% critical, so take that into account, especially when planning your disaster recovery budget.
- You must test your BC plan CONTINUALLY – at least once per month. Make sure all staff involved in your plans are thoroughly trained and involved in all testing activities as this will reduce the incidence of human error.
- BC can includeleveraging cloud solutions or implementing on-sitefailover solutions, such as a back-up generator. It all depends on how much you’re willing to spend, but one solution can work just as well as the other if used correctly.
- Protecting your business functions is key and so is protecting your data. The two tasks go hand in hand. Make sure that whatever solution you chose gives you the ability to protect, preserve, or restore your business data if necessary.
- Planning for failure is always the key to a successful project; planning for your BC plan to fail is an essential part of this process. What do you do when your back-up plan does fail or how do you communicateproperly to your customers or others who also depend on your services? Part of the reason why Verizon is under fire right now is not only because their services failed, but they neglected to inform anyone about the failure or what was being done to correct it – and initially they even tried to deny that it happened at all.
- Early-warning systems or comprehensive monitoring for your back-up solutions is another critical element. If something fails ordoes not work properly, you need to be aware of it immediately. The sooner you are aware of the issue, the sooner you can remediate it. The worstpossibilityis to have a scenariowhere your services are down while you are unaware (as what appeared to be happening inVerizon’s case).
The moral of this story – don’t allow sudden disruptions to affect your business when you can prepare a comprehensive back-up solution in advance. It’s no longer acceptable to live by the “thisis unlikely tohappen to me” mentality because, as we have seen many times in the past year alone, so many unexpected things can happen that will totally disrupt your operations.
If you feel yourself frozen like a deer in headlights at the thought of planning a Business Continuity or even a Disaster Recovery solution for your business, you’re not alone.Don’t hesitate to reachout to us at info@networkats.com with any questions, we wantto help!
