In May 2024, Ascension, an extensive 142 U.S.-based hospital system (with many hundreds of associated medical facilities), experienced a severe cyberattack. The breach affected patients and operations across its numerous facilities, prompting 37 days of downtime procedures and in-depth investigations. As of this post, full operational recovery is likely still underway to populate all of the data captured in paper records and to resolve significant gaps that allowed the attack to be successful.
In this deep dive, we examine some fundamental questions about the attack: How could a large and well-known nonprofit health system allow for such a complete disruption of care and risk to patient safety? Did threat actors damage critical systems and steal patient data? Was a complete shutdown of the Ascension EHR network even needed to stop the attack? What is the likely cost of the attack and how much could preventing similar attacks from happening again cost?
Through publicly available information we try to answer these questions and examine what we can learn from the Ascension attack about how to better defend healthcare from cyberattacks.
Here’s a summary of what we learned in our review:
- Gaps in core security systems could have contributed to the initial incident and delayed recovery efforts.
- From Ascension’s report, no clarity has been given on why malware downloaded by a user wasn’t detected or blocked by in-place security tools, or if the employee wasn’t educated on downloading and installing valid software.
- Given that only 7 of ~25,000 servers were impacted and no EHR systems were affected, we conclude that the Ascension team may have had a difficult time eliminating systems from the scope of the attack or using other security tools to contain the attack and threat actor movement.
- Poor network segmentation likely contributed to the size of the outage. Without segmentation, Ascension couldn’t limit isolation to just impacted systems. Ultimately Ascension determined many critical systems should be taken offline as the only means to prevent ransomware spread.
- A solid initial response plan doesn’t prevent the need for substantial downtime and recovery plans. Ascension’s incident response plan was enacted quickly, but it didn’t prevent an extended outage due to lengthy recovery procedures, and Ascension teams weren’t prepared to care for patients in a degraded state for more than a few hours or days.
- The outage highlights large and concerning patient and financial impacts. However, the technology improvement costs needed to prevent similar incidents might significantly exceed the outage costs.
- The best defense is having a good defense. Better detection and containment capabilities could have reduced the impact on patient care. A fully capable security operations team with advanced threat detection and response, automation, and AI in use may have helped reduce the footprint of what needed to be taken offline, thus reducing the impact on patients.
A Quick Overview of Ascension and Ascension Technologies
Founded in 1999, Ascension Medical was created through a merger of two health systems and grew through mergers and acquisitions over 15 years to become the most extensive nonprofit Catholic health system in the United States. Due to Ascension’s history of hospital acquisitions across 19 states, the complexity of EHR integrations supporting multiple associate and patient portals was vast at the time of the attack.
Ascension Technologies, part of Ascension, leads data centralization across markets and provides low-cost IT infrastructure and software application services focused on rapid and effective clinical decision-making, information sharing across the care continuum, and other operational benefits. It enables seamless access to data across all applications and connects provider, patient, and consumer experiences.
Left of BOOM – What may have been happening in the Ascension IT environment before the attack?
By studying Black Basta observed tactics, techniques, and procedures, and publicly available information about Ascension’s technology, we’ve identified that previously existing technical weaknesses were likely exploited during the Ascension breach. We’ve extrapolated the following potential issues that allowed the attack and extended outage to occur:
- Weak Access Controls: Ascension’s access control measures were insufficient. This included weak passwords and an ongoing but possibly incomplete initiative to roll out multi-factor authentication (MFA) across all critical systems.
- Unsecured Remote Access: Remote access points were complex and not adequately secured, a critical issue that provided more opportunities for accidental download of malicious files and for attackers to infiltrate the system without detection.
- Outdated Software: Some systems ran obsolete software with known vulnerabilities that had not been patched, creating easy entry and lateral movement points for attackers.
- Poor Network Segmentation: The lack of proper network segmentation allowed the attackers to move freely within the network once they gained access. Once inside, attackers could easily reach critical systems and data.
- Data Visibility Gaps: Data stored across multiple environments is frequent in healthcare and, according to IBM Cost of a Data Breach Report 2024, is one of the leading factors in increasing the number of days to scope and contain a breach.
- Potential Complexity of Security Tools: Ascension confirmed compromised footprint of the attack was just 7 out of ~25,000 servers. Given the duration of containment and recovery, we can assume there were gaps and complexities in security monitoring that prevented Ascension from confirming how far the attack spread without outside forensics.
Right of BOOM – A Timeline of Events
The attack resulted in a complete shutdown and isolation of core EHR instances and took most portals offline. This meant that associate and patient portals could no longer access medical records. It also disconnected integrated systems that enable care operations, from connected medical devices and labs to dictation, coding, and payments. Most functions required some sort of manual workaround during the outage.
Here’s how the Ascension events unfolded:
- BOOM—An Unclear Date: A person working at an Ascension facility downloaded a file that they believed was legitimate but contained malware, allowing threat actor Black Basta backdoor access. Ascension has commented this download was “an honest mistake”, or it may have occurred in conjunction with spearphishing, which is a common technique used by Black Basta.
- It’s unclear if the malware wasn’t caught by endpoint detection and response (EDR) agents or network detection tooling due to tool or deployment gaps or if the threat actor deployed tools and further masquerading to avoid detection.
- Ultimately, Black Basta was able to gain access to the server network. Through a likely combination of exploiting vulnerabilities and privilege escalation techniques, the attackers moved laterally and installed ransomware on (at the time) an undetermined number of Ascension servers.
- May 8, 2024: Ascension Medical discovered unusual activity in their systems.
- May 9, 2024: The breach was publicly acknowledged, stating that systems were compromised and downtime procedures were enacted. Diversions (i.e. sending emergency ambulances to other hospitals) were underway in many markets.
- May 13, 2024: Ascension reported that there was no clear timeline for full system restoration.
- Ascension later confirmed that Mandiant was brought in to investigate the attack, and Palo Alto Networks Unit 42 and CYPFER helped supplement rebuild and restoration efforts.
- May 24, 2024: Two class-action lawsuits were filed on behalf of affected patients.
- May 29, 2024: 21 days later, Ascension reported it restored EHR access to its first market.
- May 30, 2024: Unions petition Ascension due to deep concerns about staff strain and patient safety.
- June 12, 2024: Ascension confirms that PHI was stolen from just 7 of its ~25,000 servers, which were impacted by ransomware, and that the attack is not believed to have compromised any core EHR systems.
- June 14, 2024: 37 days later, Ascension reported all EHR and Patient Portal access was restored, however, medical records created during the outage may be missing until populated.
Ascension has also mentioned that remediation of additional systems is still needed. In addition to delayed/missing electronic records, patients likely can expect issues and gaps in billing, scheduling, and other key administrative functions that occurred during the outage.
Patients and Care Providers Impacted
According to multiple news reports and statements from Providers and Nurses, the duration of the outage pushed downtime procedures well past the breaking point. These patient care and patient safety issues were repeatedly reported from multiple markets:
- Prescription and Pharmacies: Most markets experienced pharmacy and prescription refill issues. If pharmacies didn’t close completely, providers used fax as a workaround to send prescriptions to pharmacies for manual processing. Delays in prescription refills were exceptionally high presenting significant risks for patients with chronic conditions.
- Emergency Room Diversions and High Wait Times: All emergency departments remained open for walk-ins, but some markets diverted ambulances to the next closest hospital. News outlets and social media reported long waits. While no metrics have been shared on the mortality impacts of the incident, diversions, and ED crowding have long been found to increase mortality in critical patients.
- Delayed and Canceled Imaging and Scheduled Procedures: Multiple markets reported canceling imaging and scheduled elective and outpatient procedures to focus on inpatient care.
- Mistakes and delays in inpatient care and medication: Cited in news reports and by the Local 40 Union, improper staffing and support lead to frequent patient safety issues, including missing orders, wrong or delayed medications, and potential deaths. Some nurses reported quitting due to the unbearable strain experienced.
Containment & Recovery
While the initial response was quick, the remaining investigation and road to recovery have been extended and complex (and still ongoing).
- Confirming and Isolating Breached Systems: Because Ascension couldn’t confirm threat actor access and indicators of compromise on all systems, it likely took days of work with outside experts and forensic review to identify and then further isolate the 7 compromised systems, necessary to prevent re-compromise and remove the threat actor from the network.
- Market by Market Recovery: Once EHR systems by market were confirmed not impacted or accessed by threat actors, and didn’t contain vulnerabilities that could be newly exploited, Ascension Technology, IT, and Security teams likely began the process of restoring access to systems.
- Where critical vulnerabilities existed, such as weak network configurations and emergency patches, those likely also needed to be fixed first before systems could be brought safely back online.
- Confirming Integrated/Interoperable Systems: Critical system integrations and data connections needed to be validated before full functionality could be restored. This required careful coordination of restoring secondary systems or establishing connectivity with external parties.
- Migrating Paper Records: Data reconciliation to correct time-sensitive records, such as scheduling, and moving downtime procedures to normal operations likely occurred over several days. For some markets, capturing and entering paper records created during the 37-day outage may still be ongoing.
Scope of The Breach
The exact patients and Ascension staff PHI information stolen is still being investigated, but the following scope of the data breach has been shared so far:
- Records Affected: An estimated 3 million patient records were compromised, but this is still to be confirmed.
- Potential Types of Data Compromised: Patient names, Addresses, Dates of birth, Medical histories, Social Security numbers, Insurance details
The Long Right Tail – Further Mitigation Efforts:
The following mitigations are potentially underway and commonly required in health system environments after an attack that impacts patient care:
- Identity protection services are currently offered to any Ascension patient, regardless if patients have been confirmed impacted.
- Enhancing system monitoring by implementing security tools where missing, upgrading tools that didn’t detect or block the initial attack, and increasing attack surface discovery efforts to prevent future gaps.
- Increasing security protocols in network, identity, and infrastructure configurations is critical to preventing a threat actor from moving laterally from non-EHR systems to compromise critical EHR systems.
- Removing easily exploited vulnerabilities will likely take much longer due to the required planning, testing, and downtime to key clinical and operations systems.
- Improving recovery time objectives and testing by market, including critical integrations with external parties.
- Improving extended downtime procedures, including staffing necessary support and operations roles during system outages.
Financial Magnitude of BOOM
Ascension reported net patient revenues of nearly $20.3 billion for the nine months ending March 31, 2024. We expect to see the financial impact of this event in the financial statements eventually, but we can start to make some early guesses about how this incident could impact Ascension.
By reviewing public financial statements for the three months ending March 31, 2024, we can estimate that approximately $3 billion in expected Patient Revenues were put at risk during the outage and its aftereffects. If Ascension saw a 10% drop in patient care during this time, then a minimum loss of $300 million could be expected before accounting for the additional incident and operating costs incurred – more than 30 times the average healthcare breach cost.
Based on the last quarterly financial performance, it’s reasonable to expect this will translate mainly to a net operating loss and potentially a net loss for the period. Ascension does maintain approximately $16 billion in cash and cash equivalent investments, which will be needed to absorb the short and long-term impacts of the incident, such as any lawsuits for failure to ensure patient safety.
On July 25th, 2024, Ascension announced the intention to sell nine Illinois hospitals to Prime Healthcare, which intends to invest $250 million to improve facilities, make substantial technology investments, and upgrade systems.
The 9 Illinois hospitals represent just 6.3% of the Ascension hospital network. So while it’s impossible to fully estimate how much investment the remaining 93.7% of the network requires in technology and system upgrades, this transaction does raise some significant concerns. Can Ascension maintain its focus on its nonprofit mission, keep its commitment to return to profitability from prior poor performance, and remove substantial technology and cybersecurity risks? It’s a lot to juggle across a very complex technology and healthcare delivery environment.
A Cyber Defense-Focused Future
In reviewing what we know about this most recent incident, the Ascension team’s fast response did limit ransomware spread, but it may have verged on using a sledgehammer to crack a nut, when the ideal response would have contained only the impacted systems and network segment. Given the gaps in security posture, Ascension likely had few alternative choices, and the extended outage was the only secure response.
Of course, Ascension’s new goal should be to prevent significantly broad and extended system outages in the future, which may require an eye-watering investment in substantial technology upgrades and organizational disruption. Time will tell us the full impact and whether Ascension will invest in improvements. It’s clear improvements are needed in 3 key areas:
- building the security operations team and the tools needed to defend the organization more effectively,
- improving clinical operations procedures and resourcing for extended downtime scenarios, and
- making the likely significant investments needed in technology upgrades to improve the recovery of critical technology.
Regardless of Ascension’s choice, there seems to be a little path forward that doesn’t improve resiliency to prevent wide-scale patient harm from happening again.
Blackwell is cybersecurity made for healthcare.
Our comprehensive security platform and expert managed services provide the most complete protection for healthcare organizations. Leading providers across North America trust Blackwell to reduce cyber risk, stay compliant, and build trusted experiences across the care continuum.
© 2024 Blackwell Security | Privacy Policy | Sitemap