Disaster Recovery for Enterprise Imaging
What Is Disaster Recovery?
Disaster recovery (DR) includes best practices and strategies for preventing or minimizing data loss and business disruptions should catastrophic events occur—equipment malfunctions, power outages, cybersecurity breaches, war or military attacks, natural disasters, or other significantly disruptive events occur. A recent ESG survey stated nine out of ten respondents reported that their organization could not withstand more than an hour’s worth of lost data before experiencing significant business impact.
Infrastructure failure can cost as much as USD 100,000 per hour, and critical application failure costs can range from USD 500,000 to USD 1 million per hour. More than 40% of small businesses will not reopen after experiencing a disaster, and among those that do, an additional 25% will fail within the first year after the crisis.
Disaster recovery planning involves strategizing, deploying technology, and implementing continuous testing. In addition, maintaining backups of data is a critical component, however, a backup and recovery process alone does not constitute a disaster recovery plan. It also includes ensuring that adequate storage and computing are available to maintain robust failover and failback procedures. Failover is the ability to switch automatically and seamlessly to a reliable backup system. When a component or primary healthcare infrastructure system fails, either a standby operational mode or redundancy should achieve failover. Failback involves switching back to the original primary systems. Failback is the second stage of a two-part system for safeguarding information in a crisis mode during natural disasters or other events that can compromise an IT operation.
The Most Common Types of Disaster Threats to Healthcare:
Winter storms, floods, tornados, hurricanes, wildfires, earthquakes, and other severe weather that poses a significant threat to human health and safety, property, critical infrastructure, and homeland security are considered natural disasters. These occur both seasonally and without warning, causing periods of insecurity, disruption, and economic loss. The way healthcare organizations care for patients during these circumstances has become complicated especially if they have to unexpectedly close or evacuate.
Two examples of the devastating impact natural disasters inflicted upon healthcare organizations are Hurricane Katrina and Hurricane Sandy. During the investigation of the preparation and response to Hurricane Katrina, the Select Bipartisan Committee found that nearly all issues arose from information gaps. Numerous communication failures and information-sharing gaps may have led leaders to fail to act promptly or act blindly, without adequate intelligence and analysis. In addition, after the hurricane made landfall, much of the communication infrastructure was damaged, and backup systems were unavailable.
Ariel View of Memorial Medical Center after Hurricane Katrina. Source: Nurse Labs
To further complicate matters, during both hurricanes, evacuation, and transportation of medical patients occurred and created more confusion. During Hurricane Katrina, 65 hospitals across the country took in evacuated patients. However, many facilities lacked electronic medical records (EMRs), and sending patients with paper records was inefficient and unworkable. As a result, many patients arrived at new locations with incomplete medical records.
By 2031, global ransomware damages are predicted to exceed $265 billion with a new attack expected every two seconds.
In 2021, there were a number of major healthcare-related data breaches, with over 40 million patient records compromised in the USA. This led to several warnings from the FBI about the risk of cybercrime in the healthcare sector. In the wake of Russia’s invasion of Ukraine, the FBI has released further warnings of Russian hacks on US healthcare organizations.
A report on the impact of healthcare-related data breaches from Protenus found that over 50 million patient records were compromised last year, with a total of 905 incidents reported. This reflected a 44% rise in the number of hacking incidents with healthcare organizations.
Healthcare-related data breaches affected over 22.6 million total patients in 2021, with the single largest data breach reported affecting more than 3 million individuals. This was the Accellion FTA breach, which we’ll cover in more detail later.
In total, there were over 600 reported healthcare breaches last year in the USA. As part of the HITECH act, the US government publishes a list of all reported healthcare breaches including 500 or more individuals.
Three Biggest Healthcare Cybersecurity Attacks In 2021
Accellion Data Breach
Accellion (now known as Kiteworks), suffered a data breach in late 2020 when its 20-year-old File Transfer Appliance system was hacked with a zero-day exploit. The breach was the single largest healthcare-related hack in 2021, affecting over 3.51 million people. At least 10 different healthcare organizations suffered data breaches, including hospitals, medical schools, and clinics.
Florida Healthy Kids Corporation
A Florida-based healthcare facility found in February 2021 that its web hosting provider had failed to patch over seven years’ worth of vulnerabilities, affecting over 3.5 million patients, making it one of the biggest healthcare-related data breaches of all time. Unauthorized users gained access to the unsecured system and tampered with the data of thousands of applicants to the facility.
20/20 Eye Care Network
20/20 Eye Care Network was involved in a breach of Amazon AWS web servers that gave cybercriminals access to social security numbers, identification numbers, dates of birth, and health insurance information. This breach reportedly affected over three million people and a lawsuit is pending as a result of the data loss.
The ransomware attack on Planned Parenthood in Los Angeles led to records of 400,000 people being breached. Information access included personal addresses, insurance information, and highly sensitive medical records.
Power outages affect communication infrastructure, technology, power station transformers, and transportation. Hospitals, long-term care facilities, primary care offices, clinics, and emergency medical services are the most vulnerable when a power outage occurs as the healthcare providers’ ability to communicate with their patients is compromised. Power outages often concern refrigeration for certain medications or medical devices. If a power outage occurs due to flooding, it can prevent patients from traveling to receive necessary care, deliveries, and medical workers from getting to work. Lack of generator fuel can be connected to power outages causing the inability of heating and life support systems to operate.
Paramedics move an injured person toward an ambulance during a terror attack. Source: The Telegraph
Healthcare organizations are targets for terrorist attacks. Such attacks have far-reaching consequences, including decreased accessibility, possible casualties, and fear. The extent, incidence, and characteristics of terrorist attacks against hospitals are unknown. The Global Terrorism Database (GTD) was tasked with analyzing all terrorist attacks against hospitals from 1970-2019. Analyses were performed on temporal factors, location, attack and weapon type, and the number of casualties or hostages. Chi-square tests were performed to evaluate trends over time and differences in attack types per world region. This analysis of the GTD identified 454 terrorist attacks against hospitals over a 50-year period.
How do healthcare organizations deal with disasters?
The Centers for Medicare & Medicaid Services issued a regulation in 2016 requiring adequate planning which is the most important thing that an organization can do to mitigate the potential impact on patient care should a disaster occur. Drills, preparedness exercises, and training that focuses on how the organization will continue to provide patient care during health IT downtime are all part of a disaster plan. For example, using paper as a backup for patient intake should an organization’s EHR system have an outage. The organization should also determine how to resume normal health IT-based operations once the downtime passes, and how to integrate all data orders that were generated during the downtime period.
There are resources available to help healthcare organizations comply with HIPPA Security Rule as they plan their contingencies, backup protocols, and operations should a disaster occur. The Office of the National Coordinator for Health Information Technology (ONC), in concert with the HHS Office for Civil Rights (OCR), created a HIPAA security risk assessment tool. This tool contains a series of helpful questions for an organization, from a preparedness standpoint, to ensure the availability and integrity of electronic patient health information.
Another resource available to healthcare organizations is a series of ONC tools known as SAFER (Safety Assurance Factors for Electronic Health Record (EHR) Resilience) guides. These interactive guides are meant to help organizations perform a self-assessment of their health IT systems to optimize them from a patient safety standpoint.
The importance of planning
What is the difference between backup and disaster recovery?
Backup is the process of making an extra copy (or multiple copies) of data to protect it. For example, restoring backup data if an accidental deletion occurs, database corruption, or a problem with a software upgrade.
After an outage, disaster recovery refers to the plan and processes for reestablishing access to applications, data, and IT resources. The plan might involve switching over to a redundant set of servers and storage systems until your primary data center is functional again. It is common for organizations to confuse backup with disaster recovery. But as they may discover after a serious outage, simply having copies of data doesn’t mean you can keep your business running. To ensure business continuity, you need a robust, tested disaster recovery plan.
Enterprise Imaging Disaster Recovery: Key terms
There are a few key terms used to shape strategic decisions and enable evaluation backup and disaster recovery solutions.
- Availability according to the Society of Imaging Informatics (SIIM) is the measure of time when a system is fully available for the business functions for which it was designed. For a PACS, for example, this would mean support of DICOM image management, archival, and visualization. All of the essential system functions must be operating for the system to qualify as available. (For example, image management and archive, but not visualization, would not qualify a PACS as “available” in the proper sense of the term.) Availability in IT systems is often measured in “nines,” using the percentage of availability per unit of time as the basis.
- Disaster recovery as a service (DRaaS) is a managed approach to disaster recovery. A third party hosts and manages the infrastructure used for disaster recovery. Some DRaaS offerings might provide tools to manage the disaster recovery processes or enable organizations to have those processes managed for them.
- Failback refers to switching back to the original systems. Once the disaster has passed and the primary data center is functioning, a failback occurs.
- Failover is the process of automatically offloading tasks to backup systems seamlessly for users. For example, an organization could fail over from the primary data center to a secondary site, with redundant systems that are ready to take over immediately.
- Recovery point objective (RPO) refers to the amount of data you can afford to lose in a disaster. This includes copying data to a remote data center continuously so that an outage will not result in any data loss. Some organizations determine that losing five minutes or one hour of data is acceptable.
Restore is transferring backup data to your primary system or data center. The restore process is generally considered part of backup rather than disaster recovery.
Prioritizing Workloads: Disaster Recovery Policies and Processes
Once key concepts of disaster recovery are understood, organizations can apply them to create workflows and protocols. Many organizations have multiple RTOs and RPOs that reflect the importance of each workload to their business.
A common and generally accepted model for Disaster Recovery levels derives from the early work of SHARE, a consortium of mainframe users, to classify relative levels of Disaster Recovery Readiness. This defines seven potential tiers of readiness, ranging from no preparation to complete business integration of preparation. This is referred to as the “Seven Tier” model.
- Tier 0: No off-site data – Possibly no recovery
- Tier 1: Data backup with no hot site
- Tier 2: Data backup with a hot site
- Tier 3: Electronic vaulting
- Tier 4: Point-in-time copies
- Tier 5: Transaction integrity
- Tier 6: Zero or near-Zero data loss
- Tier 7: Highly automated, business-integrated solution
Evaluate deployment options
However, for disaster recovery, a plan that relies wholly on an on-premises environment would be challenging. If a natural disaster or power outage strikes, the entire data center (primary and secondary systems) will be affected. That is why most disaster recovery strategies employ a secondary site, otherwise known as Disaster Recovery Site, some distance away from the primary data center. A disaster recovery site is a location used by an organization for restoring its IT infrastructure and business-critical operations when a primary production center is affected by a natural or man-made disaster. It may be located across town, across the country, or across the globe depending on the organization. Creating a disaster recovery site allows an organization to continue conducting operations and delivering services without disruption until the primary location is restored. Some factors to consider when deciding on a secondary site are performance, regulatory compliance, and physical accessibility to the proposed secondary site. The Disaster Recovery Site usually contains fully functional servers (mirrors of the production servers) with reduced capacity. In most cases, disaster recovery sites don’t have redundant servers, but some organizations choose to have a fully mirrored site as an insurance policy.
With the growing adoption of cloud-based solutions, cloud computing enables greater integration and collaboration between hospitals, medical organizations, and healthcare providers. Cloud-based backup and disaster recovery solutions are becoming increasingly popular among healthcare organizations of all sizes. Many cloud solutions provide the infrastructure for storing data and, in some cases, the tools for managing backup and disaster recovery processes.
Cloud-based backup and disaster recovery solutions can support both on-premises and cloud-based production environments. An organization may decide, for example, to store only backed-up or replicated data in the cloud while keeping the production environment in its own data center. With this hybrid approach, the organization would gain the advantages of scalability and geographic distance without having to move its production environment. In a cloud-to-cloud backup model, both production and disaster recovery are located in the cloud, but at different sites to ensure enough physical separation. Healthcare organizations often use different cloud providers to achieve extra resilience against disasters. Cloud agnosticism is becoming increasingly important for software vendors.
As organizations adopt mobile applications, storing and backing up clinical data in the cloud provides users with complete access. In a cloud-based format, confidential patient-based information is protected by a third party, which is continually updating firewall security and other protection measures to ensure HIPAA compliance. Dicom Systems has leveraged the healthcare cloud to deploy enterprise imaging initiatives and advance interoperability for a number of clients globally. Our Cloud Partners include AWS, Google, and Azure.
Traditional tape backups
Traditional magnetic tape storage has been utilized since the 1950s and can still play a role in an organization’s backup plan. With a tape solution, a large amount of data can be stored reliably and cost-effectively. A tape drive provides sequential access storage, unlike a hard disk drive, which provides direct access storage. A disk drive can move to any position on the disk in a few milliseconds, but a tape drive must physically wind tape between reels to read any one particular piece of data. As a result, tape drives have very large average access times.
In 2022, IBM introduced the Diamondback Tape Library, reaffirming its position that a magnetic tape is a form of data storage that is still relevant today. According to IBM, magnetic tape provides physically air-gapped isolation to increase resiliency against cyber security threats such as ransomware. IBM’s new tape storage offering is aimed at organizations that need to securely store hundreds of petabytes of data, both traditional and “new wave” hyperscalers – global enterprises aggregating massive customer data sets.
Snapshot-based replication can be used for backup or disaster recovery. Of course, the data is only as complete as your most recent snapshot. If snapshots are taken every hour, an organization must be willing to lose an hour’s worth of data.
Many organizations are moving toward continuous replication for disaster recovery as well as for backup. With this method, the latest copy of a disk or application is continuously replicated to another location or the cloud, minimizing downtime and providing more granular recovery points. Continuous replication solves the problem of the “backup window”, where organizations are at risk of losing data created between two scheduled backups, as is the case with snapshot-based replication.
Securing Medical Images for disaster preparedness
An effective disaster recovery plan allows healthcare organizations to quickly restore all medical data and resume normal processes while minimizing downtime and data loss. Due to complex infrastructure requirements, radiology departments need specialized disaster preparedness protocols, particularly for natural disasters or other emergencies that increase the demand for and volume of imaging exams. For instance, after an earthquake, the number of casualties may require an increased level of imaging as part of patient diagnostics and triage.
Having a data backup plan is a must for effective medical imaging disaster recovery. A comprehensive data backup plan dictates which medical data to back up, how frequently the data should be backed up, as well as how long the data can be stored. These considerations are dictated by the type of medical data as well as the storage capacity of the secondary data storage sites. As a best practice, mission-critical data such as medical imaging should be backed up the most frequently.
Another consideration of medical imaging data backup is patient privacy. Title II of the Health Insurance Portability and Accountability Act (HIPAA) of 1996 mandates that all medical information should be managed in such a manner that ensures that patient privacy is adhered to. Failing to abide by this law can lead to financial as well as other penalties from the government. When planning data backup solutions and processes as part of a disaster recovery plan, HIPAA compliance is imperative.
Benefits of disaster recovery in the cloud
Migrating to the cloud is an opportunity for medical imaging environments to ensure availability, performance, and security while reducing infrastructure complexity and cost. When access to on-prem systems becomes limited, such as during a natural disaster or a pandemic, a cloud-based health IT platform has a significant advantage as part of a disaster recovery response plan.
Hot vs. Cold vs Warm Storage
Hot storage refers to data that is frequently used and accessed, such as documents on your hard drive, which means it requires faster (and more expensive) hardware. Hot storage is for an immediate and reliable access. All data that you need to be able to access immediately must therefore be placed in hot storage. This can include data that is:
- Known to change
- Used for clinician query purposes
- Used in any current projects
Cold storage is the opposite of hot storage: this is data that you want to keep, but rarely need to access. The rate of retrieving data and response time for cold storage data systems is slower than the services intended for managing active data. Cold storage is typically housed on a service such as Amazon Glacier, where it is ready for you to access if it is ever required. This is data that may need to be retained for legal reasons or compliance The legal requirement for storing imaging studies varies by jurisdiction. In the U.S., many regions require providers to retain medical images for five to seven years, with certain cases requiring studies to be kept for over 20 years, such as in the case of minors. Cold storage is also appropriate for data that is no longer updated but is still queried. This data is also known as “dormant data.” Cold storage is similar to investing in a homeowner’s insurance policy in case of a disaster. In most cases, a healthcare organization won’t need to fully restore many files and may only need to retrieve a few if a disaster occurs. Most of the costs are associated with the retrieval of files, so keeping copies of images as an insurance policy becomes the economical choice.
Data that requires continuous access without the restrictions forced by cold storage is fit for warm storage. Warm storage is for data that does not need to be accessed as quickly as hot data, so it can be stored in a slightly slower, capacity-optimized environment. If you are concerned about overloading the hot storage, files can be stored in warm storage.
Comparison of different data storage types.
On-Premise, Cloud, or Hybrid Data Recovery Considerations
- Radiologist and physician viewing workflow(s)
- HL7 and DICOM protocols behave differently with high latency
- Faster connections cost more but can increase data access
- Migrating to cloud storage demands security and typically an express route or encrypted end-to-end network.
- The workflow needs change and AI demands faster data access.
- AI algorithms may require increased investments in more performant storage.
- Data security is paramount.
- Enterprise Imaging vendors leveraging S3 or equivalent can gain security with data encrypted in flight and at rest (vendor dependent).
Enterprise imaging considerations before deploying on-premise, Cloud, or hybrid.
Dicom Systems Unifier Cloud Archive
When a healthcare system’s picture archiving communication system (PACS/MIMPS) goes down, the need for disaster recovery and business continuity must go hand in hand, so that mission-critical clinical workflows can continue. Unifier Cloud Archive ensures uninterrupted patient imaging operations, whether PACS/MIMPS downtime was planned or unplanned. Our solution lets our customers not only store the data in any chosen cloud in natively Dicom format, but it also enables the Dicom objects to be queried, retrieved, and viewed in a zero-footprint viewer. As such, the Unifier Cloud Archive can be used as a real-time source for relevant prior images. When it launched in 2019, Dicom Systems’ Unifier Cloud Archive was the first turnkey solution for backing up HIPAA-compliant medical imaging to the cloud.
Dicom Systems Cloud Partners include AWS, Google Cloud, Azure, and Life Image. The collaboration with these partners allows us to deploy the Unifier platform to serve as a bridge between on-premises systems and the cloud, allowing organizations to access data from a highly secure infrastructure.