Hey, there! Log in / Register

Global security-software glitch forces cancellation of all elective surgery and office visits at Mass General Brigham hospitals

By adamg on Fri, 07/19/2024 - 9:17am

An attempt by a computer-security firm called CrowdStrike to send out an update for its Windows security software has taken down computers and mobile devices around the world, including at Mass General Brigham, which sent out an urgent message to staff this morning not only canceling all non-emergency procedures and visits, but telling doctors and nurses they shouldn't even try to access the hospital patient-information sytem:

We continue to experience a major Digital incident that is affecting all Mass General Brigham hospitals and sites as well as many major businesses across the country. All Mass General Brigham sites should be using downtime procedures for all systems. Due to the severity of this issue, all elective and non-emergent surgeries, procedures, and non-urgent ambulatory care at Mass General Brigham sites will be cancelled for Friday, July 19, 2024.

All staff who were scheduled today should report to work even if their work area clinics and cases are canceled, unless otherwise directed by their manager.

If you have a MGB device and you have not already used it today, please do not turn on the device or log in. This will help to expedite our resolution to the issue.

We have activated the incident command across the system. All leadership and technical teams are working on addressing this incident.

The problem relates to an update to CrowdStrike's software and how it doesn't play nicely with the Windows platforms on which it was installed.

Mass General Brigham IT sent a second message:

All staff who were scheduled today should report to work even if their work area clinics and cases are canceled, unless otherwise directed by their manager.

If you have a MGB device and you have not already used it today, please do not turn on the device or log in. This will help to expedite our resolution to the issue.

This is a reminder that MGB is on downtime procedures. DO NOT ATTEMPT TO ACCESS EPIC UNTIL THE ALL CLEAR IS SENT.

More details on the problem.

Topics: 
Technology
Free tagging: 
Mass General Brigham


Ad:


Like the job UHub is doing? Consider a contribution. Thanks!

Comments

By cybah on Fri, 07/19/2024 - 9:40am.

Thankfully my work.. a cloud provider.. is not affected.

But ouch. What a way to bring down a hospital to nearly a halt.

But the cloud is suppose to be better. LOL Remember, the cloud is just someone else's computer and not this magical thing.

up
17

By CuteUsername on Fri, 07/19/2024 - 10:06am.

I don't think the cloud has anything directly to do with this glitch, though. The cloud is involved in what the software is supposed to do, but the software runs locally on companies' computers, and that software updated itself in a way that causes Windows to crash. (Downloading updates from a server might be one of the many things tech companies try to sell as "the cloud" but I don't think that's what CrowdStrike means by it.)

By BostonDog on Fri, 07/19/2024 - 10:28am.

It was a normal update that installed fine but caused the computer to hang on reboot. Fairly easy fix for someone who knows how to boot windows in safe mode but that's a huge number of systems to fix for the few (if any) techs who are in person at these places.

I've always been skeptical of things like Crowdstrike. It seems like trading one risk for another.

By cybah on Fri, 07/19/2024 - 10:46am.

As someone who used to work for a cybersecurity company...

Nope It is not. Why? It will take one breach that is caught by crowdstrike to pay for itself for years. If a breach is not caught, it could cost a company millions in lost revenue.

I've watched breaches happen in real time, its amazing how fast this stuff works. And when you have a product like crowdstrike, it will intercept it and/or at least be able to log everything so the forensics are easy to do.

Fwiw, not a huge fan of CrowdStrike. I prefer Rapid7 instead. Local company, solid product, and their CEO can play a fine game of table tennis (when they were in at 100 summer, we had a challenge with them.. as the company I worked with was across the hall)

up
10

By BostonDog on Fri, 07/19/2024 - 11:13am.

You point is valid which is why I'm strongly encouraged or required to use CS on my systems. (And do so willingly.) But today's failure is going to cost some firms just as much as a breach.

Last night's Crowdstrike failure wasn't that bad, all told, but the failure shows how it would be possible for a malicious group to push out an attack via CS if they gained access to that firm's tools.

It's basically how security guards are useful to protect against intruders. But if the security guards become the intruders themselves, you have even bigger problems.

By cybah on Fri, 07/19/2024 - 10:41am.

Yes its the cloud because CloudStrike lives in the cloud (the calling home server). And its the lack of update coming from it is the issue and why things are crashing..

Today cloud can be anything that isn't local to your computer. Cloud does not have to be things like Azure or AWS and such. If it lives somewhere else... its "cloud"

(remember I work for a cloud outfit.. and we aren't AWS or Azure or etc. Yet we are still "cloud")

up
10

By Rwgfy on Fri, 07/19/2024 - 10:38am.

Isn't redundancy in IT systems of the state's health care monopoly a public health issue? Will there be an investigation or at least a 311 report?

By cybah on Fri, 07/19/2024 - 10:42am.

You're special aren't you.

No it is not because there are other hospitals that are in functioning order like BI-Lahey and (sadly) Steward Healtcare hospitals.

up
13

By HenryAlan 2.0 on Fri, 07/19/2024 - 12:39pm.

State healthcare monopoly? Last I checked, MGB is an independent corporation. And it's far from a monopoly. I work for a different large healthcare provider and our systems are fine. The biggest threat to health care tech wouldn't be something attacking Windows anyway, but something attacking EMR systems, such as the EPIC system mentioned in the MGB text alerts.

Regardless, you clearly have no idea what you are talking about. If you did, you'd be embarrassed by this sad attempt to troll. Trolling is an art form, and you are no artist.

up
10

By SwirlyGrrl on Fri, 07/19/2024 - 1:13pm.

Guess what, chump - this was NOT limited to MA.

I have family hospitalized in VT and they were not discharged on time due to this glitch. I also have family working in other state's hospitals and a VA facility and they have had issues, too.

By necturus on Fri, 07/19/2024 - 1:22pm.

So did all the display boards at North and South Stations. They were just coming back online this morning when I got on my train.

By Frelmont on Fri, 07/19/2024 - 11:37am.

So, CrowdStrike feller gives a full-throated apology for his service glitch. That’s good. And, we’re confident it’s a matter of the system not playing nicely with the environment and not a long undetected adversary, or allied agency?

By anon on Fri, 07/19/2024 - 11:38am.

But I wanted to get my lipo NOW!

up
10

By mg on Fri, 07/19/2024 - 1:08pm.

Elective just means it isn't an emergency. My cataract surgery was considered elective, as was my lumpectomy and my polyp removal. They all had to happen, preferably very soon, but I wasn't going to die if they were postponed a few days.

up
15

By necturus on Fri, 07/19/2024 - 1:20pm.

... can shut down so many vital resouces by accident.

Imagine what a malicious actor could do in that position. If this country ever goes to war, we'll probably learn of it when all our TVs go blue-screen.

By SwirlyGrrl on Fri, 07/19/2024 - 4:33pm.

Uh. No.

Over $1 billion in sales is definitely non-obscure.

By SmallJB on Fri, 07/19/2024 - 4:40pm.

"Obscure?" Solely because you haven't heard of it before today?

By Ron Newman on Fri, 07/19/2024 - 5:01pm.

I was able to log into it today, and found everything there that I expected to find.