October 2015

The Changing Face of Disaster Recovery
By Susan M. Lucci, RHIA, CHPS, CHDS, AHDI-F, and Tom Walsh, CISSP
For The Record
Vol. 27 No. 10 P. 14

On top of Mother Nature's wrath, health care organizations must be prepared to handle modern-day intrusions.

Like most aspects of health care, disaster recovery is evolving. In recent years, the diversity of incident types that can lead to disasters has changed enormously. Therefore, the need to plan, prepare, and practice has assumed added significance. Throughout the industry, the focus has shifted from "if" to "when" the next disaster strikes.

Is your health care organization ready?

Disaster Types
The purpose of security controls is to prevent, detect, assure, and recover. No matter what type of event occurs, health care organizations must be able to recover sufficiently to allow the continuum of patient care to carry on. One disaster concern that has always been high on the radar is system outages, the inability to access mission-critical applications and systems.

Traditionally, organizations have categorized disasters as internal or external. Natural disasters such as hurricanes, flooding, and tornadoes are examples of external disasters. Internal disasters include system crashes, failed software upgrades, and accidents such as a busted water pipe in the data center or the emergency power being mistakenly turned off.

On top of these circumstances, a new set of disaster recovery events has emerged. Tragic situations such as terrorist attacks, pandemic health issues, and organized cyber-crime hacking events are now a concern. While these aren't entirely new to health care, the expanding challenges that accompany preparedness and recovery from such events add complexities.

In many cases, certain internal and external factors cannot be easily predicted, making preparedness essential. Proper planning helps enable a swift and efficient return to normal activities while providing ongoing health care to the surrounding community.

Cyber attacks are typical of the modern threats faced by health care organizations. While traditional disaster recovery has focused mainly on system outages caused by acts of nature, cyber attacks present a new set of challenges. The following cases illustrate the dangers posed by these nefarious acts.

• In August 2012, a hacker encrypted the protected health information of the Lake County Surgeons practice and demanded a ransom be paid to obtain the password.

• In April 2014, Internet access at Boston Children's Hospital was cut off for almost two days when a group of hacktivists, suspected to be led by the group Anonymous, launched a directed denial-of-service attack in response to a child custody case.

• In December 2014, Clay County Hospital's internal systems were infected by a ransomware program that resulted in protected health information being held hostage. The data hijacker threatened to release the information to the public if a considerable payment wasn't made.

Disaster Recovery Planning
As required by the HIPAA Security Rule, a disaster recovery plan includes a series of systematic steps that must be followed in order to quickly return to normal. Without proper planning, some situations can significantly delay or even derail the recovery process. The key components of a plan address the following:

• activation (when and how to declare a disaster);

• assessment (determining the root cause and assessing the disaster's impact);

• containment and control (steps to prevent escalation);

• recovery (steps to get at least mission critical applications and systems quickly restored);

• emergency mode operation procedures (how the organization will function until operations return to normal);

• restoration (returning to normal operations may mean shifting data center operations from an alternate site to the primary data center); and

• exercising and testing (when and how the disaster recovery plan will be tested).

Still, even with excellent planning, there's the possibility that something unexpected occurs. For example, take communication, a key step in any recovery plan. Many organizations have implemented Voice-over Internet Protocol, better known as VoIP. One of the drawbacks of this system is if the network goes down, so do the telephones. Also, external disasters typically cause a flood of telephone traffic on cellular networks, causing the service to be unavailable to many callers.

A well-written disaster plan provides communication alternatives such as using text messaging to contact the disaster team.

Strategy

Maintain High Availability
Most health care organizations use virtualized servers and systems to maintain high availability. Virtualization allows multiple servers to run on one server host—sort of a mini mainframe. If a hardware or software problem is detected, the controller instantly moves a server from one host to another host server, thus avoiding potential downtime or a disaster.

Data Backup Plan
A well-defined data backup plan is an important component of any disaster recovery strategy. Besides being a requirement of the HIPAA Security Rule, implementing a data backup plan is also necessary to determine the recovery point objective (RPO), which is the amount of time elapsed between the replication of data.

In the past, daily backups to tapes were the prevailing practice, but that is no longer the case. A daily backup represents an RPO of 24 hours, meaning in the event of a system failure, up to 24 hours' worth of work may be lost. Imagine telling physicians or nurses that they'll have to reenter everything they did yesterday into each patient's chart—and please make sure it's exactly the same.

Now, organizations are lowering their RPO times to less than two hours and using disk-to-disk backup to make it happen quickly and efficiently.

Incident Response Team
A proper disaster recovery plan begins with identifying the key players who can provide insight into the needs of each department. The staff member with either the highest level of managerial responsibilities or the longest tenure onsite at the time of the disaster should temporarily assume the role of incident commander, who must immediately contact the incident response team (IRT). This individual should remain in the position until someone with more authority or seniority arrives at the scene.

The IRT usually comprises a cross-section of individuals who can assess damages and assist in the recovery process. Typical IRT members include the following:

• IT director;

• information security officer;

• house supervisor (after hours);

• privacy officer;

• public relations manager;

• telecommunications manager; and

• director of facilities (or plant engineering).

Depending on the type of disaster, other subject matter experts can be added to the team.

Mission Critical Systems
An inventory of applications and systems, along with their designated criticality to the mission, must be established. A business impact analysis identifies and prioritizes application systems as well as defines recovery objectives.

Emergency Mode Operation Procedures
When systems are offline, it's inevitable that the business of registering and caring for patients is going to be more complicated. An emergency mode operations plan provides guidance during a chaotic time. Many normal business processes and procedures may have to be temporarily suspended during a disaster recovery situation.

Recovery (Resynchronization)
The synchronization of information and systems following any type of disaster can be one of the most challenging parts of the recovery process. Information stored on paper must be reintegrated into electronic systems, plus the timing of documentation and events must be carefully coordinated.

Testing
Once a plan is created, it must be tested. Because it doesn't require having to actually bring down any systems, a tabletop exercise is an easy way to step through a plan without impacting patient care or business operations. Mistakes, incorrect assumptions, and missing plan elements can be uncovered during this process. Following the exercise, the plan should be updated to correct any false notions, omissions, or errors.

Points of Failure
When formulating a disaster recovery plan, keep in mind there's not nearly enough time to practice for every potential scenario. For many years, hospitals have devoted a large amount of time and resources to be ready for terrorism acts and random shootings, but these can still tax systems and create myriad potential missteps.

The disasters being experienced by today's health systems aren't just weather-related, but make no mistake; Mother Nature has intensified her efforts to make life miserable for hospital leaders.

Back-up generators may take over as planned, but recovery efforts can be delayed if flooded roads are impassable, making it impossible to access necessary fuel. To further complicate matters, some older hospitals station their emergency generators in prone-to-flooding basements.

In October 2012, Hurricane Sandy caused enormous damage when it hit the East Coast, forcing many health care organizations to activate disaster recovery measures. For example, Staten Island University Hospital was forced to use paper documents when power was shut down, severely disrupting operations, including its data center. With the storm approaching, Bellevue Hospital Center and NYU Hospital believed their generators were ready for the oncoming threat, but when water in the basement reached the eight-foot level, the machines failed. The only option was to evacuate several hundred patients using flashlights.

The conundrum is that while many security practices may meet the requirements for disaster preparedness, few hospitals are upgrading their infrastructure to the necessary extent. Many large city hospitals are more than 50 years old, and taking on a massive infrastructure project is just not possible at a time when budget concerns are paramount. Besides, few patients choose a hospital based on the location of its back-up generators.

In fact, only one-third of the hospitals that participated in a 2012 Joint Commission survey said they were planning to upgrade their infrastructure. Still, some are taking the plunge. For example, Mount Sinai Health System has slotted $12 million for infrastructure improvements, including moving four basement generators to higher floors.

HIM's Role

Organizational Workflows

Disaster recovery plans tend to focus on an IT department that resides in a local data center or server room. IT staff may not have as much of an in-depth understanding of organizational workflows as HIM professionals. The systematic recovery of systems must align with the business process steps used in providing treatment and care to patients. HIM professionals can provide assistance to the IT staff with application, system, and hospital department recovery prioritization to support workflows.

Documentation
In the wake of a disaster, HIM staff will be involved with the recovery process in a number of ways. HIM professionals understand the importance of good documentation and preserving critical information. If paper records are being used, decisions must be made about whether to backload the information. Orders and emergency records must be either uploaded when the power returns or scanned.

When a server outage at Boulder Community Hospital shut down the hospital's MEDITECH system, no one suspected it would be down for 10 days. The conversion to paper records was effective but prolonged. When full recovery was completed, only about eight hours of data were lost. Still, the impact of the transition to paper and the resumption of electronic processes created much additional work for caregivers and HIM staff.

Patient Identification
HIM experts, who know what information to capture in disaster situations, can play a key role in patient identification procedures. Readily available prenumbered tags and medical record charts, along with a process to store valuables and personal effects, can help reunite victims with family members.

Release of Information
Concerned family members and friends need a designated place where they can wait to obtain information on their loved ones. Since HIPAA regulations permit disclosure during a disaster regarding a patient's general condition or in the case of death, every effort should be made to accommodate family in the most comfortable and convenient way possible. HIM professionals and privacy officials can support public relations and administrative staff to make sure accurate, compliant information is released.

Impact on HIM Staff
In terms of preparedness, HIM staff will play an important role in the creation of an emergency mode operations plan.

On a more immediate level, depending on the disaster's scale, HIM work schedules may need to be altered. A disaster situation complicated by a nonworking EMR system means that everything from registration to treatment practices will be handled on paper records. As a result, HIM's routine is likely to be affected for some time even after the disaster has been resolved. The work may be more stressful and, depending on the type of disaster, long hours may become the norm.

A smoothly running HIM department will be a key factor in how quickly and efficiently the entire organization can return to normal operations.

Manage the Best You Can
There is no single solution to help hospitals prepare for every type of disaster. In today's health care environment, it takes a well-orchestrated combination of strategies, including addressing how to combat the rising incidence of cyber-crime.

In every health system, defending against disasters and implementing a successful recovery depends on a team composed of many players. Careful planning, practice drills, and exercises can lead to more "what if" questions to optimize the response during a crisis. This team approach gives organizations the best chance to return to full operations in the shortest amount of time. The protection and communication of health information is critical to ensure that timely care is provided and data integrity is maintained.

— Susan M. Lucci, RHIA, CHPS, CHDS, AHDI-F, is a consultant and chief privacy officer at Just Associates.

— Tom Walsh, CISSP, is president and CEO of tw-Security.