From Behind The Eight Ball To Ahead Of The Curve

In the fall of 2000, the Public Service Company of New Mexico (PNM), supplier of all electricity and gas for the state of New Mexico, was trapped in a classic enterprise backup nightmare. Following years of steady growth, the company had 86 Windows NT servers on its network, each with its own DLT (digital linear tape) drive. Every night, a dedicated staff spread out over the enterprise with shopping carts, replacing tapes and backing up machines one by one. Sometimes they finished by morning, but often they didn't. What's more, each backup had to be checked manually for errors, on a server-by-server basis - another drain on PNM's network staff.

Heaven help the PNM employee who lost one of the CAD/CAM (computer-aided design/computer-aided manufacturing) drawings, document files, or e-mails stored on these machines. "At any given time, we had no more than four days' worth of backup on site," remembers Don Atwood, PNM's senior information center specialist. "Everything else was off-site and would take at least 24 hours to retrieve and restore from the recovery facility."

Doing The Math
PNM arrived at this impasse in the classic way: by implementing backup on a machine-by-machine basis - until things got out of control. "We had a centralized backup plan on the drawing board for more than a year, but it was a half-million-dollar project, and we kept canceling it for budgetary reasons," Atwood explains. "It was cheaper to add a $6,000 tape drive to each new server, so that's what we kept doing." Then a spike in PNM's projected network server and data growth forced a change of strategy. "Once we determined that we'd be adding 100 new servers in the next four years, suddenly the half-million-dollar solution looked cheap."

Atwood put out RFPs (requests for proposals), and enjoyed some instant popularity. Fifteen integrators, resellers, and OEMs (original equipment manufacturers) responded, including some big names. But Atwood wasn't impressed by household names. "We knew the big companies would have backup solutions, but we wanted companies who specialized in backup, that installed backup for a living, and spent their days and nights taking backup support calls. That narrowed the field in a hurry."

One of the bidders that made the cut was Intelligent Solutions, a Medford, MA-based systems integrator Atwood found while searching the Web for backup equipment information. At presentation time, Intelligent Solutions President Kevin Metcalfe stood out from the other finalists.

"Typically, these companies send teams - a salesman to make the pitch and techs to answer the difficult questions," says Atwood. "Kevin came in himself, presented by himself, and answered all the questions himself - not only my questions, but my bosses' questions, too, which are tougher to prepare for."

As compelling as Metcalfe's presentation was, so was the tape library technology he proposed. While the other companies were offering familiar DLT tape backup, Metcalfe's solution was built around Sony's PetaSite Scalable Mass Storage System and DTF-2 (digital tape format version 2) technology - a mainstay in the broadcast industry, but as yet a rarity in commercial or industrial enterprise networks.

"The DTF's throughput (86.4 GB per hour, sustained) was between four and five times faster than the fastest DLT option," says Atwood. "And the drive's native capacity (200 GB per cassette) was almost double anything else out there."

For all its relative power and capacity, Intelligent Solutions' backup was designed for management ease. "Our goal is always to enable backup of the entire enterprise by one person, from one GUI (graphical user interface) console to one set of tapes,'" says Metcalfe. "In the morning, all the manager has to do is check his e-mail to see which backups were successful and which weren't."

The Value Of Expertise
Once Atwood chose Intelligent Solutions, the real work, integrating the solution with PNM's network, began. He remembered, "The system was actually installed well within the time frame they specified. Once they installed the system, I identified some performance bottlenecks, which, in an enterprise backup system, could be anywhere in the network."

Atwood continued, "Pinpointing the problem involved dealing with multiple vendors like Cisco, Compaq, and Intel, all of which tend to point fingers at one another when problems crop up. But Dan Goodwin, Intelligent Solutions' director of systems architecture, took on the vendors himself, found the problem, and fixed it. As far as I'm concerned, he went way beyond his contractual responsibilities, something I've never experienced before with a solutions provider." Not surprisingly, Atwood has come to depend on Goodwin for support.

Delivering Measurable Benefits - And Exceeding Expectations
Service wasn't the only thing Atwood got more of than he expected. It turns out he bought even more backup capacity than he thought. "Each DTF-2 tape has an advertised 200 GB native capacity and 500 GB with compression," he says. "But we've compressed as much as 788 GB on a tape. Out of 100 tapes, at least 40 are holding more than 600 GB."

As a result, PNM has gone from having four days to four full months of nearline backup data in the library on-site. "We can find and restore anything in minutes," adds Atwood, "even remotely, over an Internet connection, if we happen to be out of the office." Atwood estimates that over the next three years, the backup solution will save PNM $300,000 in unnecessary tapes, tape drives, software, and networking hardware. That doesn't include the savings of reducing his backup staff from ten employees to one.

Because PNM's growth projections spiked once again since the backup system was installed, Atwood is especially happy with the scalability built into the system. "We already added a third DTF-2 tape drive to the PetaSite library," he said. "We just plugged it in without reengineering or increasing management duties. This kind of scalability is important to us, because the company is planning larger projects and an acquisition that will significantly increase our storage and backup requirements." When the time comes, PNM can even scale its bandwidth. They can upgrade from their current 100BaseTX target connection to gigabit Ethernet whenever they need it. That's another thing DTF does that DLT doesn't.

In all, Atwood counts himself a more-than-satisfied customer. "With six months under our belt, we're getting faster, higher-capacity backup than we expected. Both the Sony hardware and Intelligent Solutions are continually outperforming our expectations."