March 2014

Prepare for the Worst
By Lisa A. Eramo
For The Record
Vol. 26 No. 3 P. 10

Recovering HIT systems during a disaster takes foresight and fortitude.

It was surreal, like something out of a movie. When Hurricane Sandy struck the East Coast in the fall of 2012, terror and confusion ensued. New York City hospitals, many of which were evacuated, became inundated with patients seeking emergency care. Power outages spread quickly, and backup generators kicked into full gear. Providers needed access to crucial health technology, and they needed it quickly amidst the chaos of rising floodwaters.

“Not having access to critical systems and data is a matter of life or death in health care,” says Jim Gerrity, director of global industry marketing for Ciena, a network infrastructure provider. “I don’t think that’s being overly dramatic or overstating the consequences.”

“It’s a patient safety issue. You need to have access to that information quickly,” says Allison Viola, MBA, RHIA, vice president of policy and government affairs at the eHealth Initiative.

In health care, there potentially are hundreds of systems that must be restored in the event of a disaster. As the industry moves toward an environment that relies on EHRs and other highly integrated and sophisticated technology, disaster recovery is paramount.

The good news is that, unlike paper records that may have been strewn across streets and waterways in a hurricane, tornado, or other natural catastrophe, electronic information can be backed up off site or in a safe location and restored quickly and efficiently when needed.

The emergence of health information exchanges (HIEs) also has helped organizations retrieve data more easily. These entities potentially can serve as a valuable lifeline during a disaster or significant downtime, Viola says. “I think HIEs will enable the recovery process much more efficiently. You’ll have the ability to reach out to other facilities to recompile that information,” she adds.

Still, experts say many organizations aren’t prepared for these types of scenarios. “Unfortunately, you see the best efforts around disaster recovery postdisaster,” says Larry Sellers, principal consultant of technology services at CTG Health Solutions, an HIT solutions and services company. “It’s a good wake-up call.”

A false sense of immunity can be a health care organization’s worst enemy. Although natural disasters may be infrequent, no organization is exempt from unexpected system downtime due to human errors (eg, an employee in the data center accidentally pulls a power plug), hardware failure, unplanned power outages (eg, a construction company accidentally hits a power line on the street), or system viruses and other cyber attacks.

In addition to the consequences of unexpected interruptions, downtime due to system upgrades, software patching, and hardware replacements can paralyze an organization if not properly anticipated and planned for well in advance.

So the question isn’t when or how downtime or a disaster may occur, but rather whether an organization will be armed and ready when it happens. To reach that stage, experts recommend providers tackle several items.

Develop a Formal Disaster Recovery Plan
The HIPAA/HITECH Omnibus Final Rule section 164.308, which took effect on March 26, 2013, requires organizations to develop a contingency plan in the event of an emergency or other occurrence that damages systems containing electronic protected health information. The plan must address data backup, disaster recovery, emergency mode operations (which include the ability to protect the security of electronic protected health information while operating in emergency mode), testing and revision, and applications and data criticality analysis. In addition, The Joint Commission requires health care organizations to establish in writing and periodically test a disaster recovery plan.

It’s great to have a plan, but is it viable, asks Lee Fleming, advanced recovery strategy practice lead at SunGard Availability Services, a business continuity software provider. “Are there annual tests on every application? Are there quarterly tests on the updates and changes? That’s not as common as I would like it to be,” he says.

Sellers agrees: “I think the biggest mistake I see is that the plans are shelved and dusted off once a year. … I believe it’s more of a living document that must be visited and tied heavily into the change management activities in the organization.”

Create Appropriate Triggers
This is one of the more difficult challenges, according to Sellers, who notes that moving into disaster mode can be costly and result in data gaps. “It’s also a big disruption in operations,” he adds.

One of the biggest mistakes organizations make is not identifying every potential event that can jeopardize data, thus delaying the implementation of crucial recovery steps to ensure business continuity, Gerrity says. In addition to security and network threats, planners must consider threats unique to their geography, such as earthquakes, tornadoes, hurricanes, and frequent power interruptions.

For example, Sellers recalls working with one health care organization that was evacuated after nearly 46 inches of snow caused structural beams to buckle in a building that housed its data center. Despite being rare, such unexpected scenarios can’t catch staff off guard.

It’s fine to troubleshoot before moving into full disaster recovery mode, Sellers says, but there must be a specified time frame in place for initiating a response. “A lot of organizations leave it to chance and local decision making without having a documented number in mind in terms of the amount of downtime they’ll tolerate,” he says.

Establishing a predeclaration statement prior to initiating full disaster mode may be one solution that buys organizations time while also allowing them to prepare for the worst. “Everyone who is instrumental in the recovery is put on standby and made ready,” Sellers says. “If they’re on their way home, turn them back around. Folks should start gathering their recovery plans. It allows you to react more quickly.”

Maintain an Accurate Inventory
If a system is inadvertently left out of the disaster recovery plan, other more crucial systems to which it’s connected may be inoperable. Fleming says he encountered this predicament when refining the disaster recovery plans of several New York hospitals following Hurricane Sandy. “We had tested everything, but when the situation happened, it was so different,” he recalls. “A lot of systems were undocumented and not on our big radar picture.”

That’s because testing had involved only a small subset of applications and not the infrastructure as a whole. For example, tie-in applications such as credentialing software must be restored and working properly to enable physician names to appear in clinical applications. Ensuring that all systems are documented and tested is paramount, Fleming adds.

Consider Off-Site Options
When natural disasters strike, on-site data centers can become damaged or inaccessible. To combat this likelihood, experts say more health care organizations are moving data backup off site—even across the country—to potentially safer locations. “We have two on-site data centers in separate buildings on our campus with real-time data replication between the two to help protect against hardware failure,” says Ed Ricks, CIO at Beaufort Memorial Hospital, a 197-bed facility situated on the Atlantic Intracoastal Waterway in Beaufort, South Carolina. “We also have our primary backups in our newest data center and a near–real-time copy of that going to a leased site which is geographically far away from our hospital. In a worst-case scenario, I know that our data is protected and can be restored.”

Despite its location in a hurricane zone, Beaufort Memorial Hospital, which has an ongoing awareness of the importance of disaster recovery, has yet to experience a natural disaster. However, it has dealt with a network outage that caused two nursing units to lose connectivity to their electronic systems.

Like Beaufort Memorial, other institutions are opting for off-site storage. “With higher bandwidth speeds and the cloud becoming mainstream and more advanced daily, utilizing off-site storage of the data has become a way of life, not to mention [helpful] to meet mandated programs such as meaningful use and the HITECH Act,” says Jason Hawley, director of information services at Yuma District Hospital and Clinics, a 22-bed critical access hospital in Colorado. “With our ClearDATA disaster recovery solution, we can spin up our virtual machines in less than 24 hours and have access to our patient information and financials, facilitating quick recovery of critical business applications and data.”

All hospitals should consider the feasibility of moving to a highly available data model for clinical applications, Fleming says. “When the hospital switches to the backup mode … the end users don’t see a difference. That’s the model that hospitals are moving toward,” he says, adding that the concept is seamless, eliminates recovery time, and users never are aware the data center has been damaged.

“As we continue to migrate to a cloud-based disaster recovery solution, we are concurrently running our legacy solution,” Hawley says. “We have a network-attached storage device in one of our separate data closets away from the data center that stores duplicates of the day’s bare-metal backups. Our mission-critical data is backed to RDX cartridge, encrypted, and taken off site to a fire-proof safe.”

Despite its benefits, off-site data storage is not something every hospital can afford, says Brian Rogers, director of product management at Summit Healthcare Services. Noting the importance of having access to recent clinical data while waiting for the redundant data center to activate, he recommends connecting at least one machine per unit to a backup battery. Summit Healthcare provides a solution that pulls critical data from health information systems at specific intervals, encrypts them, and distributes them to strategically located downtime stations throughout an organization. These stations include user- and role-based authentication to ensure HIPAA and meaningful use compliance.

Downtime machines also should include copies of the disaster recovery policy, Rogers says. “Even in the case of a network loss or power loss, you’ve got a working machine with a physical printer plugged into it so you can print out critical information,” he says.

Don’t Skimp on Testing
“Most health care providers don’t test their disaster recovery plans because of budget restrictions, resource constraints, or a fear of the results,” Gerrity says.

However, testing—minimally, on an annual basis—is crucial. “Anytime there is an upgrade or infrastructure change, we recommend that you test again,” Fleming says. “It’s not a matter of just running the test; it’s also refining the process.” For example, testing helps identify gaps such as incorrect IP addresses.

“Currently, we test at least once a year, and it really should be once every six months,” Hawley says. “As HIT continues to evolve, I see us moving to every six months or possibly quarterly.”

Rehearsing or scripting much of the test in advance yields little benefit, Sellers says. “Folks perform last-minute training right before [the test] to make sure the procedures are tidied up,” he says. “That’s not a real-world scenario. Real world is that it could happen tomorrow with two minutes’ notice, and you have to be prepared with the folks you have right now.”

Fleming has witnessed similar missteps. “I think organizations test to test. What I mean by that is they set everything up in a clean, pristine environment and they script the test. That’s troublesome to me, as you should test for a disaster,” he says. “The way I like to run my test is to say, ‘OK, the system is down. Show me how you would break it all down and recover from that’ as opposed to … ‘Just turn on the machine and make sure [the process] works.’”

Ensure testing includes the actual steps to gain access to the redundant data center, Rogers says. “How will you swap your backup system into real time and validate the data? This is not just moving through the steps of accessing and understanding the policy,” he notes.

Gerrity says health care networks must be equipped with the proper infrastructure and capacity for testing and disaster response. “It’s not just about having a network in place; it’s about having a smarter network in place,” he explains while pointing to the following important considerations:

• Can the network provider quickly increase bandwidth if a complete system restore is necessary?

• Does the network provide pay-as-you-use circuit connectivity for testing?

• How frequently can data be backed up? Will the backup occur in real time or during regular intervals to protect information integrity?

• Does the network include redundancy through separate fiber conduit paths? “If you’re building a network, you want to make sure that those fiber connections are going through conduits in different geographies so that if one gets cut or flooded, the other one is available,” Gerrity says.

• Does the network ensure HIPAA-compliant data encryption?

Don’t forget to test communication-related elements. “If you’ve stated that you’ll provide a status update every 20 minutes, then you need to exercise that,” Sellers says. “Exercise the communications and all components, including the physical recovery.”

Plan for the worst-case scenario, Viola says. “Individuals must initiate the plan and carry it through. What will your organization do if people can’t get to work because of the disaster? You have to be able to adjust on the fly depending on who has or hasn’t been impacted,” she notes.

Gerrity suggests training a pool of employees who can respond in an emergency and assist when the disaster spans a large geographic area.

Monitor Data Backup
Fleming says organizations that don’t test backups face serious consequences. “At the time of a disaster, you try to recover, and then pieces are missing,” he says.
Ensure that the backup is functional and includes correct and complete data. “If you’re relying on that backup and it’s corrupted or didn’t complete the last time through or is a year old, it’s not going to do you any good,” Rogers says.

Yuma District Hospital and Clinics has found that being diligent pays off. “Our legacy backups are tested at least once a month for integrity/restoration, and we often have instances where a file or folder needs to be restored,” Hawley says. “To date, restorations have been met with 100% success. Our EHR databases are backed up every two hours so, at the most, we lose no more than two hours’ worth of data.”

Accessibility
A disaster recovery plan is effective only if employees can gain easy access. Gerrity recommends keeping multiple copies in secure locations and ensuring the plan clearly identifies priority IT systems and clinical application systems that must be recovered first. Many applications may not require immediate restoration because they don’t directly affect operations, he adds.

Hawley agrees: “We tabletop the disaster. What are our first steps? What do we need access to first? Prioritizing business-critical functions—the bread and butter of the facility—and how to get access to them immediately is the heart of the plan.”

Honesty’s the Best Policy
After completing a disaster recovery test, providers must not shy away from the results. “I’ve seen organizations sugarcoat the results so they won’t look bad to management, which I think is tragic,” Sellers says.

Organizations that fail to meet recovery time objectives must determine where their inefficiencies lie. For example, there may be gaps in the plan. Rogers recommends addressing the following questions:

• Which critical data must be available during a downtime?

• Can data be accessed quickly from multiple locations? If not, why? What were the barriers?

• How much time and effort are expended to manually reenter data into the system? How will this be handled in the event of an extended downtime affecting a large amount of data?

— Lisa A. Eramo is a freelance writer and editor in Cranston, Rhode Island, who specializes in HIM, medical coding, and health care regulatory topics.