October 2018

Surviving the Storm — And Other EHR Downtime Disasters
By Elizabeth S. Goar
For The Record
Vol. 30 No. 9 P. 14

From hurricanes to hackers, forces of all kinds can disrupt medical record service. Is your organization prepared?

When Hurricane Ike slammed into Galveston in 2008, it swamped the University of Texas Medical Branch (UTMB) with eight feet of water on the first floor. The hospital campus had been evacuated in advance of the storm and it was several days before flood waters receded enough to allow Alexa Cross, who at the time was UTMB's associate director of HIM, and the HIM team to get back onto the island to survey the damage and retrieve current patient records.

Most of the patient records were on the health system's recently implemented Epic EHR and other medical record systems, which was beneficial once power was restored after the storm.

Despite the circumstances, Cross and the team quickly relocated the HIM department off-island—at Cross's home—where they were able to remotely access the EHR and keep critical information flowing.

"We turned my dining room, kitchen, and study into the medical records department, so 24 hours a day we had release of information going on," Cross says, adding that round-the-clock record access was critical because the island's evacuation meant patients were spread across the country. "Because patients were being actively treated, we were faxing things all over the US. … Patients were able to get their records in half an hour. It just worked very well."

When Hurricane Harvey hit in August 2017, Cross had moved to Houston Methodist, where she was the senior systems analyst for Epic HIM and Identity. Experience had taught the hospital the importance of contingency planning to maintain EHR access—specifically, applying lessons learned from Tropical Storm Allison, which swamped the city in 2001 when Cross was director of HIM at Texas Children's Hospital.

Houston Methodist had in place high-powered generators capable of keeping the facility and its information systems up and running, as well as a remote data center and a business continuity access plan. Epic's Care Everywhere HIE platform also came into play.

"The EMR gave access [to records] across the system, to all the practices that were affiliated with Methodist along with all other Epic facilities. If the patient didn't opt out, Care Everywhere was available for all other Epic facilities, even those in other cities so they could at least access an abstract. Patients also had access to a patient portal, which made a huge difference in this last storm," Cross says.

Her experience with three separate natural disasters has given Cross unique insights into how best to survive EHR downtime when Mother Nature unleashes her fury. Along with storing paper records and microfilm at least two stories above ground and having a paper back-up or flash drive with contact information for employees and disaster recovery vendors, she recommends the following:

• create a designated area for caregivers to store paper records of evacuated patients, preferably in the HIM department so they are available for requests for information;
• provide evacuees with a comprehensive continuity of care document and/or access to a robust patient portal where all their information can be easily accessed when/if needed;
• assign each department a dedicated cell phone that can be passed off to each shift—the number should either be the department's main phone number or calls to that number should be forwarded to the cell phone;
• establish remote access for employees who normally don't have it, especially those in HIM roles;
• ensure that the IT plan is kept updated and can be prepared for downtime; and
• set up mirroring at any remote data sites.

"We needed a lot less recovery time for Harvey, which was so huge and so impactful of the coverage areas. I was working remotely the whole time, but it was relatively simple compared with the other storms," Cross says. "The EMR made a huge difference. … It routinely takes a snapshot of the patient information and puts it in a business continuity report. When a disaster is anticipated, caregivers can print out this report for all patients on their unit so they have basic clinical information to facilitate treatment during downtime."

Manmade Disasters
EHR downtime procedures were implemented at Houston's St. Luke's Episcopal Hospital/CHI (Catholic Health Initiatives) St. Luke's, when the city was devastated by heavy flooding and tornados in May 2015.

"Our disaster planning included assigning employees every year to either a prep or a recovery team," recalls Sarah Glass, RHIA, CCS, FAHIMA, who was director of coding at the time. "The prep team remained at the facility during the storm and managed paper records while the EHR was down. The recovery team evacuated and returned when the storm was over to help with recovery efforts."

However, not every disaster is the work of Mother Nature. Glass says unplanned EHR downtime can be caused by anything from destructive malware and ransomware to power outages and hardware malfunctions. For example, Boston Children's Hospital was hit with a distributed denial of service attack in 2014 that flooded the hospital's servers with data to disrupt services. Two years later, Hollywood Presbyterian in California suffered a ransomware attack on its EHR for which it paid a $17,000 ransom in Bitcoin to restore its systems.

More recently, Missouri's Cass Regional Medical Center's EHR was subject to a ransomware attack that forced the vendor to shut down the system while the hospital instituted downtime procedures. In that situation, the EHR vendor was able to play a role in recovering from the unplanned downtime, but Glass points out that's not always the case, particularly when the outage is due to the hardware, the network, power loss, or loss of internet access.

"The EHR vendor can't help you if your system is down due to an internet outage," she says. "If you can get into e-mail and the internet but not the EHR, then it may be EHR software, not the server [hardware]. If you are using a web-based EHR, you can't access the EHR, but you also can't access the internet; it may be an ISP issue."

Ron Moser, CISA, CRISC, CCSFP, senior site reviewer and practitioner for the Electronic Healthcare Network Accreditation Commission, went a step further, noting that any time an organization depends on a third party for EHR-related services, unplanned downtime can be particularly problematic. Often, he says, outages are caused by poor implementation planning, connectivity issues, or even poor user training. It can also be a pure capacity issue.

"A lot of times [EHR downtime] does come down to the earlier-on activities, so it's really critical right from the get-go to have a strong risk management plan and process," Moser says. "Organizations that are dependent on third parties need to know what key things they will require out of their EHR and what threats they might run into. There are other things, too. How can I get in contact with the vendor? What if the system is [up] but it's not responding? Are they really going to meet [the standards] they say they will?"

The best way to emerge relatively unscathed from unplanned EHR downtime is to plan for worst-case scenarios.

"Getting outside the typical disaster recovery scenarios [is crucial] because the worst disasters are the ones you don't normally think about," Moser says, adding that testing internal and vendor responses is crucial. "Without actually testing, when [downtime] happens, you'll find out there are lag times you never dreamed would be there."

Moser recommends organizations develop a solid risk management plan, including input from IT, clinicians, and anyone else who can provide insights into what happens to workflows when the EHR is down. The best plans identify key assets, processes, and people, and evaluate the likelihood various threats will come to fruition.

"When it comes to your EHR system, you should expect to be able to do disaster response testing with your vendor where you can [determine], 'If these pieces go down, what is the plan for putting another piece in place?'" Moser says. "Cloud, connectivity, etc, are things that should be tested. Keep looking for the weakest links and keep testing those."

He also recommends having local systems where the most immediate information is cached and available when the EHR is down or have local copies of records on separate vendor services.

"Anytime your fail-over plan is dependent on a single vendor, you could be [in] trouble," Moser says. "You need something that you can take over yourself if you need to."

Cyberattacks on the Rise
With cyberattacks on health care organizations taking place at double the rate of other industries, no discussion of EHR downtime is complete without a specific focus on surviving malicious breaches. According to Ed Cabrera, CISSP, CISA, chief cybersecurity officer at Trend Micro, medical records on the black market can often be sold at a higher price than even personally identifiable information (PII).

Because they contain a unique blend of PII and medical insurance and financial information, medical record data can be sold and used for a long period of time. Hackers can use medical records to produce counterfeit IDs, tax returns, and birth certificates, procure drugs through prescription information, and create false medical insurance claims.

"Additionally, many health care organizations have growing attack surfaces because of digital transformation efforts in cloud, mobile, and IoT [internet of things]," Cabrera says. "Internet-connected medical devices transmitting and storing sensitive data have systemic and technical vulnerabilities that provide adversaries the perfect access to hospital records. The operational risk of these systems will also increase as health care organizations face increasing digital extortion attacks. Exposed devices and systems will put more patients at risk and thus create the potential for bigger profits for cybercriminals."

Cabrera recommends facilities vigorously test their disaster recovery strategy to reduce or eliminate the risk of digital extortion. And while Hollywood Presbyterian ultimately paid the ransom, he advises against doing so because "paying cybercriminals will only incentivize future attacks, and, in the end, there is no guarantee of the completeness and integrity of the data you recover."

Instead, Cabrera recommends the following steps to minimize the risk of an attack shutting down an EHR system:

• Identify: Continuously identify, map, and assess critical data and systems to identify their corresponding threats and vulnerabilities.
• Protect: Use a connected threat defense strategy to securely configure, monitor, and patch critical hospital data systems.
• Detect: Develop and deploy proactive strategies (ie, a next-generation Security Operations Center) and partnerships (law enforcement and information sharing and analysis organizations) internally and externally to actively hunt for indicators of attacks and compromise.
• Respond: Develop and deploy an internal incident response plan supported by a third-party vendor to quickly respond to and mitigate threats.
• Recover: Develop and deploy a multidisciplinary recovery plan to quickly reduce the overall impact to patient care and prevent damage to operations and the hospital's reputation.

"The best way to minimize the damage a cybersecurity attack can cause is to proactively have a multidisciplinary crisis response plan and test it frequently. We know an attack is possible, so plan for it and make sure the right processes are in place to react quickly," Cabrera says.

— Elizabeth S. Goar is a freelance writer based in Tampa, Florida.

 

TESTING FOR SUCCESS
Two of the most common oversights hospitals make when it comes to disaster recovery and business continuity planning are failing to test and refresh. That's according to ClearDATA Chief Technology Officer Matt Ferrari, who says that running these plans through a simulated scenario on a consistent basis is key to containing the damage.

"Getting all of the accountable parties in the room, from application development owners and IT to marketing and communications, is key. Then have a facilitator run through a potential scenario as if it were truly happening and have each party collaborate on how to execute against the plan," he says. "While technology is a key component of executing a failover within a disaster recovery or business continuity plan, it is not the sum of all parts. How did you communicate with the staff, patients, public? What systems are the most important to bring up first vs less important?"

Hospitals must review plans when changes are underway, not just when new systems are deployed. "You are likely onboarding new doctors [and] nurses [or] opening new offices and therefore firewall ports and IP access controls," Ferrari says. "The truth is your systems are changing constantly. What are you doing to keep your plan up to date as changes occur? Is the [plan] updating a core consideration when you make changes in the hospital environment?"

— ESG

 

RESOURCES AND GUIDANCE
There are numerous resources available to help hospitals implement plans that mitigate the risk of EHR downtime. Sarah Glass, RHIA, CCS, FAHIMA, recommends the Data Integrity Guidelines, which are part of the National Institutes of Standards and Technology (NIST) Data Integrity Project; the EHR SAFER Guides from HealthIT.gov, which enable health care organizations to identify recommended practices to optimize EHR safety; and the Contingency Planning SAFER Guide, which identifies recommended safety practices associated with planned or unplanned EHR unavailability.

Cherie Holmes-Henry, chair of the HIMSS Electronic Health Record Association (EHRA), advises developers and provider organizations to employ a risk-based privacy and security framework based on NIST's Cybersecurity Framework. Also, the Office of the National Coordinator for Health Information Technology developed the Guide to Privacy and Security of Electronic Health Information, a primer for health professionals to understand how to comply with privacy and security policy requirements.

"We encourage the development of implementation guides and best practices to apply the Framework, specific to the many facets of health care and health IT to improve adoption," says Holmes-Henry, adding that "EHRA recognizes that the majority of health care–related cybersecurity breaches are due to the lack of privacy and security best practices at the user level. To improve this, we support programs that provide awareness and basic education of cybersecurity best practices for health professionals, and that develop a culture of seeing privacy and security as an enabler of improved patient trust and better health outcomes."

— ESG