Disaster Recovery: Not Just Backup

"I've been warning my customers to implement disaster recovery plans for as long as I've been in business," said Amy Rao, CEO and founder of Integrated Archive Systems (IAS) (Palo Alto, CA). September 11 has forced Americans to take stock of their personal lives and American businesses to do the same with business continuity. In the same way attorneys across the nation have reported an increased demand for wills, businesses are finally starting to take the issue of disaster recovery seriously. No longer does the possibility of disaster seem as remote as being hit by lightning.

Rao's customers that had not seriously considered disaster recovery options are now kicking off projects to investigate their options. This is in stark contrast to a Comdisco (Rosemont, IL) study a year ago, which reported that few companies were taking the necessary steps to ensure availability during a disruption. Yet, nearly half of all organizations need their business-critical applications running at least 99% of the time. The Comdisco survey indicated that 67% of businesses did not have redundant computer hardware at alternate locations, 79% did not have critical applications on standby at an alternate location, and 86% did not have a plan in place to ensure availability of critical Web applications. Also, on average, businesses spent only 8% of their IT budgets on business continuity, which includes disaster recovery plans.

Add Value With Education And Site Assessments For Disaster Recovery
While cost and limited personnel resources still overshadow the need for disaster recovery protection, Rao said September was her company's biggest month of this year for sales. In November 2001, IAS hosted a disaster recovery seminar with VERITAS (Mountain View, CA) which normally draws about 100 people. This year's event drew more than 470 individuals.

VARs and integrators are key players in helping their customers develop disaster recovery plans. Disaster recovery is not just backup. It's not just a matter of sending tapes off-site. The key to disaster recovery is getting the customer's business up and running as soon as possible. The many facets of business continuity include recovery of entire systems, replacement of hardware, and possibly replacing key people. Then, down to the basics, communications may need to be restored. And this is as basic as getting a dial tone back. And finally, it even means making sure the employees have desks and chairs.

IAS provides disaster recovery assessment and solutions through a detailed procedure that can be applied and customized to each customer. First, IAS identifies the customer's critical business applications and their recovery requirements. It then applies backup and/or replication methodologies for data recovery.

"Disaster recovery manifests itself in the form of cold and hot sites," said Rao. She explained that cold sites can be as basic as extra servers kept in storage off-site. In the event of disaster, these servers are extracted from warehouses, powered up, and configured. Hot sites are much more expensive. However, for companies like financial institutions, telecommunications companies, and companies which fall under government regulatory requirements, their necessity can be compared to the necessity of life insurance for the consumer. A hot site usually takes the form of a second data center. It is usually used for day-to-day operations with employees processing data through its servers and storage. But it has enough bandwidth and capacity that the operations of another site could be transferred to it within minutes. This assumes that the systems and data are continually replicated to the hot site.

Recommend The Right Replication Software
VARs are essential in setting up data replication processes, which provide off-site duplication of critical data to alternate locations. They can recommend the kind of software that will enable replication, such as Volume Replicator from VERITAS. VARs can also evaluate the adequacy of a customer's standard operating procedure documentation, company contacts, and materials. They also assist with helping to develop policies and procedures for the business function and operation of each department. This includes the documentation of system interfaces and even an understanding of software licensing concerns. For instance, most software is licensed for only one location and one machine. It is not a given that the software can be used on a different location or machine. However, Rao doubts that any vendor would be so heartless as not to issue an emergency authorization.

Another part of the disaster recovery plan that Rao offers her customers is provisions for basics like printer support, e-mail services, Web services, and rerouting of 800 numbers. Cell phones, home phones, pagers, and alternate contact methodologies must be documented for all disaster recovery team members.

Offer Testing Services As Value Add
Perhaps the most important part of Rao's plan is the testing aspect. What good is a plan if there are holes in it? Rao said, "We advise our customers to review their plans quarterly and to test them at least twice a year. The testing-process duration can vary from half a day to several days. Pricing for this service varies. If it's a half-day project, IAS will charge an hourly rate. And for a longer testing project, the charge is flat, based on the extent of the project."

Rao's testing service starts out with the basics. First they pretend that the customer's building cannot be accessed. Using a cold site as an example, IAS will physically take the backup servers and storage devices out of storage, connect them to the network, load the operating systems and related data, and start recovering. From there, they perform random testing to determine if the data is present and evaluate its level of integrity. The testing may reveal that the customer is missing a software license, that there's not enough storage, or the network connectivity is faulty. "It's much better to discover these problems in a testing situation than in a high-stress disaster situation," she said.

Set Customers Straight About Backup
Since most companies' disaster recovery plans are sorely lacking, there is a lot of opportunity for VARs and integrators in this area. In Rao's experience, most customers she runs into only have a system of backup tapes that they keep off-site. Before September 11, it was assumed that they could bring the tapes back to the main location and reload them to get the system up and running. If the customer's site is gone, the process to set up a new system will be lengthy and filled with snags, panic, and frustration.

So, if customers recognize the need for disaster recovery, but economic factors are still holding some of them back from putting a plan in place, how can a VAR deal with this? Rao recognized this and overcomes it by spending time with her customers, looking at critical applications and determining their worth. She helps her customers develop a disaster recovery budget based on the level of coverage they need. The budget is based on the customer's acceptable level of risk. They must consider the opportunity cost of disaster on their business. Opportunity cost means: If the business goes down, what will it miss out on? Rao said the cost of a disaster on a company's business is measured in how much its reputation for customer satisfaction will suffer when it can't deliver services. She said it also can be measured by revenue loss when a company can't process and ship orders or handle currency transactions.

Rao maintains that customers need to prepare a robust solution that they can afford. At the same time, she advises other VARs to scour the market for new technology and find cheaper ways for their customers to prepare themselves. For instance, IAS spends extensive amounts of time with the research and development departments of vendors.

Most importantly, Rao delivers the message that planning is key. She said, "Having a well-thought-out plan in place is instrumental to execution. An effective plan is not reliant on one individual. Everyone in IT in all offices should be trained to execute the disaster recovery plan." At least now, more customers are listening to Rao's message.