Hospital hires outside consultant to probe power outage

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Station Home Page & More Local Information

The hospital lost power for hours following a transformer explosion. Though not the only hospital to lose power, RI Hospital was the last to come back on line. One patient on life support died during the blackout.

Hospital hires outside consultant to probe power outage PROVIDENCE  Lifespan has announced that it is hiring an outside consultant to look into the power outage and generator failure at Rhode Island Hospital that contributed to the death of a patient there.

A transformer explosion just after midnight Friday morning at Narragansett Electrics South Street Station blew out power to a large section of the city including Rhode Island Hospital. A patient who was on life support in the hospitals critical care unit passed away during the blackout  but hospital officials had denied any connection between the two events. The patient, said the hospital, was brought in with 100 percent expectation of mortality, and just happened to die during the blackout. Now the hospital is admitting that the outage contributed to 74-year-old Clinton Dionnes death, and Mayor Vincent Cianci is calling for an investigation. Rhode Island Hospital was without power for about two hours. Im asking for an investigation. It affected the most vulnerable of populations  young people at Hasbro, people in the intensive care unit, people on respirators, Cianci said. Lifespan announced that its hiring then outside consultant to investigate the outage Friday. The state medical examiner released a report Friday that said that Dionne died from a lack of ventilatory support. Dionne was on life support in the hospitals coronary care unit when he died.

Cianci held a press conference during the night. He said he was going to bed at about 12:10AM when there was an explosion and my bedroom was lit up. Narragansett Electric says about 7000 customers were affected by this particular power outage, and they were able to get everyone else back on line in about 8 minutes  including two other hospitals. St. Josephs hospital suffered no interruption. Women and Infants says its emergency generators kicked in almost immediately, and the hospital was back on line within 10 minutes. Hasbro Childrens Hospital also got back on line fairly quickly. So what went wrong at Rhode Island hospital? Not only did emergency generators fail, the hospital could not get back on line when the other hospitals did. We had, literally, electricity on their doorstep, but they were unable to connect, said Mike Ryan of Narragansett Electric. Narragansett Electric believes it is an internal problem with Rhode Island Hospitals system. The hospital, in turn, believes the problem was external. Elevators and machines came to a grinding halt, as nurses and doctors rushed to bedsides. It was a blessing that extra staff were on hand because of Tropical Storm Floyd  some even had to use hand ventilators to keep patients breathing. Dionnes family has retained an attorney, meaning they will likely seek a settlement from the hospital, or file suit. Cianci said the transformer explosion also cut power to two pumps at the hurricane barrier. This is not the first time that the electric company has had transformers blow up this summer, Cianci said.

http://www.msnbc.com/local/WJAR/219609.asp

-- Homer Beanfang (Bats@inbellfry.com), September 20, 1999

Answers

Homer Beanfang (Bats@inbellfry.com), thanks for posting all these interesting, Y2K-pertinent articles.

Only 1 fatality -- they had extra staff -- if juice is out for long periods staff will get exhausted, more fatalities. Of course many ppl on vents are going to pass on soon.

Homer, it seems you were posting a few months ago ... [trying to remember] ...

-- Ashton & Leska in Cascadia (allaha@earthlink.net), September 20, 1999.


Ashton & Leska,

Here's another squirrel story for ya.

Published Monday, September 20, 1999

Power returns to downtown Minneapolis

Associated Press

The lights are starting to come back on in downtown Minneapolis.

A power outage this morning left around 18-hundred Northern States Power customers in the dark at the start of the workweek.

N-S-P spokesman Paul Adelmann (AD'-ehl-muhn) says a squirrel apparently got into the company's downtown equipment, causing the outage.

Power started returning just after eight a-m, about a half-hour after the outage began.

) Copyright 1999 Associated Press. All rights reserved.

http://www2.startribune.com/stOnLine/cgi-bin/article?thisStory=8093733 9

-- Homer Beanfang (Bats@inbellfry.com), September 20, 1999.


aHa! Thanks. LOL. The Invasion continues ... ;^)

Homer, we know you from somewhere, it's driving us crazy.

-- Ashton & Leska in Cascadia (allaha@earthlink.net), September 20, 1999.


It sounds like they had a bad transfer switch. That would explain why their generator didn't kick in (assuming they kept it up) and why they couldn't get back on line when everyone else did.

What does this have to do with Y2K? Hate to tell you but stuff like this happens everyday. Did in the past, will in the future.

-- The Engineer (The Engineer@tech.com), September 20, 1999.


Testing. The back-ups and contingency plans, Engineer. Part of the Y2K preparedness scene... this year.

And next year? Lawyers. For those that didn't prepare well.

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), September 20, 1999.



Yes! Another blow brought against the tyrannical Furless Apes! Today the Forest, Tommorow the World!!! Long live the Rodent Revolution!!!

-- The Squirrel King (Just Nuts@up in a.Tree), September 20, 1999.

Diane,

Hospitals and such are usually required to test their generators and switches at least once a year, sometimes once a month. Again what does this have to do with Y2K?

And, again, I hate to tell you this but small animals are one of the largest causes of power outages in the US. Lightning is #1 but small animals are #2.

-- The Engineer (The Engineer@tech.com), September 20, 1999.


Engineer,

Let's see if I can make it simple...

Y2K potential problems ==> power out ==> emergency back-ups fail ==> hospital staff overloaded ==> critical patients unattended ==> people die.

All for want of testing, and back-ups, to their back-ups. GI?

But of course, from where you sit there will be no Y2K power outages... right?

Engineer, you should know... expect the unexpected. Sheesh!

Diane

(YOU should know better).

-- Diane J. Squire (sacredspaces@yahoo.com), September 20, 1999.


Diane:

From the information in the story it appears to be the hospital's fault as Engineer said.

It would be prudent to wait for the results of the investigation.

However, based on the statement:

" A patient who was on life support in the hospitals critical care unit passed away during the blackout  but hospital officials had denied any connection between the two events."

which was later shown to be incorrect, I wouldn't trust anything this hospital told me.

They went into denial mode before the smoke cleared.

Seems to be reflex these days.

-- Tom Beckner (tbeckner@erols.com), September 20, 1999.


Diane,

You could leave out the first part of your equation concerning Y2K and it would still be valid. There have been problems in the past, there will be problems in the future. Even if (as I strongly suspect) Y2K comes and goes without so much as a bump in the night there will still be the possibility (to the point of certainty) of ice storms and more hurricanes. There will still be the lack of generation and transmission capacity in the Northeast and Midwest. What happen to that hospital could easily happen again this winter if they dont find the problem and fix it.

These are real problems and arent going to go away on 1/2/2000. Probably unlike this forum and most of its posters when they find out how wrong theyve been.

-- The Engineer (The Engineer@tech.com), September 20, 1999.



Engineer:

Inflamatory remarks aside, have you been involved with remediation of generation or distribution control systems that gives you the confidence to make such sweeping generalizations?

Given the hint of type testing as indicated by ABB and others, I wouldn't bet two cents on the thourghness of the control testing to date unless you have seen otherwise.

Add that to NRC statements that nukes don't have date dependant controls and your industry comes up short on the credibility index.

-- Tom Beckner (tbeckner@xout.erols.com), September 20, 1999.


Agree Engineer.

But there is a Y2K lesson to be learned, for the terminally dense (not you) with this story... this year... for hospitals, et. al.

i.e. CHECK THE BACK-UP SYSTEMS!

The uniqueness about Y2K is potenial problems are not necessarily local or regional, but global, with rippling repercussions. However you know this already.

Perhaps, if I was sitting in your corner of the Western Power grid, in your seat, I might feel complacent. However, I'm not. And many aren't.

And even if you're "Y2K Ready" the unexpected does happen. A squirrel is relatively easy to recover from, but frankly, I'm a tad more concerned about the nut cases that may choose to use Y2K as an excuse to exacerbate the Y2K local problems (not the fuzzy-tailed critters that gather the "other" kind of nutz). Your territory has a hot-bed of 'em. *Sigh*

For myself... in Silicon Valley we depend on PG&E.... they offer "no guarantees." In the normal course of events, that would not concern me overmuch. But, Y2K... is "different."

Scope... magnitude... duration... and the out-of-left-field curve balls... are the great Y2K unknowns.

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), September 20, 1999.


Tom,

First, as for inflammatory remarks. Well this seems to be the board for them, doesnt it? Though there does seem to be a double standard. You can make inflammatory remarks in favor of Y2K bringing about the end of the world or close to it but not the other way around. People seem to make all sorts of predictions about Martial Law and other things with specific dates. I was just wondering if nothing happens the same people will go: OOPS I was wrong. or what? Ive noticed that all the predications of portents of doom indicators starting back in January of this year that have failed to materialize. Yet people press on regardless. What happens if the predictions fail to materialize next January? Will this board be full of apologies? Somehow I dont think so.

The interesting thing about this (and other forums) is how people who are not involved in a particular business speak with great assurance. Yet when you go to check out their facts you usually wind up with references (URLs) that are just posts of someones similar opinion. Saying: I believe X will happen and in support of my opinion Joe Blow believes X will happen too. Of course neither person may know anything about what they are talking about and have no experience on which to judge anything in that particular line of work but that doesnt seem to matter. It doesnt mean youve supported your position with a fact. Youve just supported it with another opinion.

As for the specifics of your post:

Like ABB wasnt trying to drum up some business? I dont think anyone took that seriously. We hired ABB to do some for some equipment they manufactured and installed. Ditto with GE. But we hardly gave them a blank check to look at everything, as much as either company would have liked it.

First your question implies that there must have been lots of fixes because there must have been lots of things to fix. You assume that as a given. Its not true of distribution control system. What ever they are. Can you tell me what a distribution control system is? Are you thinking of SCADA? SCADA isnt (in our system) used to control distribution.

There has been some remediation in things like SCADA systems and EMS. Though a lot less (because there was a lot less to remedial) then people think. However even if the SCADA system hadnt been remediated it still would work. The common assumption errors I see with regard to Y2K:

Everything has a chip in it. Not true.

Everything is automated. Not true.

Everything will fail in January (or latter if not then). Not true in the Y2K sense. True in that everything fails sooner or later.

Then you get: You cant operate it manually or at least not for long. Of course most things are operated manually every day. But I guess thats neither here nor there.

As for Nuke plants, they arent my forte but again can you be specific? Controls arent date dependent.

-- The Engineer (The Engineer@tech.com), September 20, 1999.


Engineer:

I am a utility customer in search of direct answers to specific questions.

Your response indicates the value of your opinions.

Thanks,

-- Tom Beckner (tbeckner@xout.erols.com), September 21, 1999.


Tom:

Specific Questions?

Your specific questions were based on assumptions, such as the control systems for distribution. But distribution usually doesn't have a control system. I cant say categorically that there are no distribution control systems in the country because it varies from utility to utility. That's the hard thing for people who claim to GI to get. There isn't some big computer somewhere controlling the distribution system to your house.

I can say that most likely the distribution feeder to your house is probably protected by a fuse. There may be a recloser up stream. Again, it will be self contained. Upstream of that will be a breaker. That may or may not be controlled via SCADA. It depends on the utility. Some have a lot, some dont. There is no national standard for what is or is not remotely controlled. The equipment your utility chooses to automate and at what voltage level may have nothing to do with what another has decided. One of the myths that a lot of Y2K is based on is that everything is controlled via automation from somewhere. Not true.

-- The Engineer (The Engineer@tech.com), September 21, 1999.



So, Engineer... does that mean there are *more* or *less* potential points-of-failure? Depending upon the locals, of course.

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), September 21, 1999.


I'm not sure what you mean by more or less. If your worry is automated systems it means less. If your worry is there having to send someone out to replace a fuse or close a recloser that locks out it means more. Of course that depends on the service area. Some are quite small and any point can be gotten to in less then half an hour. Others are large and it may take a crew an hour or so to get there. It all depends on where you live.

-- The Engineer (The Engineer@tech.com), September 21, 1999.

Engineer:

Your experience is valuable, it can only be aquired over time as opposed to read the text and take a test.

If you would be kind enough to answer my first question so that I can have a frame of reference for your conclusions in re control systems:

"Inflamatory remarks aside, have you been involved with remediation of generation or distribution control systems that gives you the confidence to make such sweeping generalizations?"

If yes then it would be a help if you could give some examples of the the tests you have observed and a description of the systems being controlled and / or monitored.

Thanks,

-- Tom Beckner (tbeckner@xout.erols.com), September 22, 1999.


"Control", by definition, requires at least: time-related sensing of a parameter + calculate new response + change input to process + change process, followed by: secondary time point sensing + calculate change in response + change input to process + change process + ....

ALL these require time sensing, and many (not all, but many) senors and controller, this involves controllers and their programs that invoke both date and time measurements. Thus potentially every control process could be threatened by control point failures: consistent experience in every industry indicates that 2-4% of every processor will abolsutely fail to perform as expected.

Now, as an engineer, I'd not recommend walking under a building designed by somebody with an attitude that "thing fail - live with it."

I'd rather design anything with an attitude STARTING at what is required? What are the limits? What are implications to the public;s life, safety, and future impacts on the client and his/her actual requirements: environmental, financial (if it costs too much - nobdy can use it!), practicality, ease of use, ease of manufactor, ease of repair, ability to actually function, etc.

Acepting failure is NOT an option of any engineer, at any time, under any culture from the Eqyptian pyramid builders to the Roman road and aqueduct designers to today's civil, mechanical, or electrical engineers.

Accepting threats to the public's life, health, or safety; or to violations of legal requirements in environmental, reporting, consrtuction codes, or contracts will you and your company (and quite properly SHOULD GET YOU and your company) fired and quite possibly faced with with civil, professional, and perhaps criminal penalties.

A simple programmer or file clerk or secretary might accept failure - that's why a "programmer" cannot be given the responsibilty to legally "sign off" on a contract or drawing to actually build it. Neither can a lawyer, or a politician, or a priest. Clinton's whole group of advisors on Y2K has nobody who has ever been actually responsible for anything: they have only been "salesmen": as politicans, as lawyers, as staffers, or as "committee members" and "fund raisers".

And a salesmen is only responsible for the story he weaves......the higher the sales goal, the bigger the story.

You, by your degree and your training, are required to understand that a conservative design, a conservative process, and a successful process are the only adequate levels of performance.

-- Robert A. Cook, PE (Marietta, GA) (cook.r@csaatl.com), September 22, 1999.


Tom, Yes I have observed tests. Mostly having to do with relays and the protection system. Some control system tests involving SCADA. It involved the standard roll over drill and key dates. However let me remind you that not everything needed to be remedied. A lot of things (most of them) didnt have any problems. Those that did were fixed. However even some of those could have been fixed on the spot simply by rebooting.

Robert, No you are wrong. While control does involve some type of time sensing it doesnt necessarily involve any type of date sensing. If your supposition were true then the analog controls that relied on Agastats, P&Bs, etc would have never been able to work. Lots of controls just look at a signal, start time (relative), elapsed time, and then see if the signal goes away. Breaker failures use this type of timing. They look at trip current and start timing. If the trip current doesnt go away within X number of cycles a trip signal is issued to the bus differential. A date doesnt enter into it at all. Some SCADA timing is relative. Commands have to be issued within a certain window (double clicking as it were). If the time is too long or too short between the first command and the second, the command is not followed. Again date doesnt enter into it.

Accepting failure is a part of life. Which is why there are backup systems. Most relays are installed in pairs so if SET 1 fails, SET 2 can still trip the breaker. Many utilities have two sets of batteries in their substations. Still things fail. If a contractor digs up a cable that contains the telephone circuits for the current differential relaying the circuit can fail. As you stated in the paragraph above you have to consider the financial aspects. You cant spend infinite money to protect against failure. I believe the expression is reasonable and prudent. When you buy a car it comes with a spare tire because getting a flat is not an unusual experience. It doesnt come with 4 spare tires.

-- The Engineer (The Engineer@tech.com), September 22, 1999.


Moderation questions? read the FAQ