The good news, and the Bad.greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread |
After running our largest hydro power station on manual control for 18 days during the Y2K upgrade of our SCADA system, I am pleased to report that the manual control worked just fine. Although it was a little inconvient not having everything on a screen in front of the controller, the actual operational side went very smoothly with no problems.We are now running on SCADA once again, but not on our new system, rather its back on the reliable 14 year old system. *The new SCADA failed its tests*. The year rollover part worked OK, but there were some programming faults that made the system very hard to use, so its off line while the manufacturers come up with a programming fix. This is estimated to be 3 to 4 weeks away, so its not too bad.
Have any other facilities reported similar glitches in their upgrades?
Malcolm
-- Anonymous, June 25, 1999
Malcolm,Bless you many times over. That is some of the "best" information that I have seen in a long time. It's nice to get the actual report of how the manual operation went, as well as the disappointing news about the new system. Hopefully, the new SCADA will be corrected. I can't speak further about our own systems, but others here will, I hope, and with as straightforward a commentary as you have given. I have seen similar problems with new software in other major business/government programs. In some cases the newer software is not quite "ready for prime time" as we sometimes say here in the US. Keep the updates coming. It sure helps to know actual details of how things are working out in the real world, as opposed to the media reported world that seems to be our own major source of details.
-- Anonymous, June 25, 1999
Malcom, I've never seen a whole new SCADA system go in without SOME kind of bugs. If this were just a "fix", I don't believe that you would have had such problems. Where there's new software, there's new bugs. In some cases, the new bugs are worse than the y2k bugs ever were! Thats what kills me about Y2K, everyone gets wild over a y2k bug, no matter how minor the effects of it are, yet companies are throwing in new software left and right, and having to work out the bugs, some potentially serious. Regards,
-- Anonymous, June 25, 1999
"Everyone". "Wild". Yeah. Sure. Right.Do you ever see any real people, or only straw men?
-- Anonymous, June 25, 1999
factfinder,
f)Malcom, I've never seen a whole new SCADA system go in without SOME kind of bugs. If this were just a "fix", I don't believe that you would have had such problems.
m) you are truly amazing. let me ask you a question. is it easier to wire a new building or rewire an old buiding?
f) In some cases, the new bugs are worse than the y2k bugs ever were!
m) now this is one of the more insightful statements that you've written. this is a big part of the problem... new bugs being created as a result of attempts to remediate. and you are right... they can be much worse, particularly if there is no source code. kinda like dancing in the dark.
-- Anonymous, June 25, 1999
What happens when, in 6 months, they can't go back to an old working system while they repair the new one? In fact, what happens in 5 months, if the new one takes 6 weeks to repair, or what happens in 4 months if the new one takes 10 weeks to repair? The window is shrinking daily and the criticality growing more and more, even before y2k.
-- Anonymous, June 26, 1999
"Thats what kills me about Y2K, everyone gets wild over a y2k bug, no matter how minor the effects of it are, yet companies are throwing in new software left and right, and having to work out the bugs, some potentially serious."Couldn't have said it better myself. And yes it may get you killed. The reason all us "doomers" are so scared about all these "minor" and "potentially serious" problems is that we know the reality of how IT implementations don't ever work without some probs. I don't think I've ever seen a system implemented that didn't cause some problems somewhere initially. The problem with 2K is that we're doing the largest implementation ever. Accross industry etc. While I am glad to see some compliance statements rolling in, I'm scared that I don't see more.
There should be more at this point.
-- Anonymous, June 26, 1999
Jim,"There should be more at this point."
Exactly right. In fact, in the nuclear area alone there should be 4-5 per week rolling in right now. Factor in the thousands of non-nuke providers or distributors and the number we should be seeing now is staggering.
-- Anonymous, June 26, 1999
Valve might have limited pipeline spill, investigators say Saturday, June 19, 1999 By SCOTT SUNDE SEATTLE POST-INTELLIGENCER REPORTERThe gasoline spill that culminated in a deadly explosion and fire in Bellingham might have been greatly limited had a valve worked properly, federal regulators say.
A block valve, which is intended to close a section of pipeline, is located just one-tenth of a mile north of where the rupture occurred near Whatcom Creek.
But the valve "either malfunctioned or was not closed in a timely fashion," investigators found. As a result, up to 130,200 additional gallons were released, the U.S. Transportation Department wrote in describing the preliminary findings of an investigation into the accident.
The Transportation Department estimated the total spill at 151,200 to 277,200 gallons.
After fuel began leaking June 10, gasoline vapor ignited, killing two 10-year-old boys. A third teenager was overcome by fumes and drowned in Whatcom Creek.
Patricia Klinger, a spokeswoman for the department, said the findings about the block valve are preliminary but still significant. That is why the Transportation Department has ordered Olympic Pipe Line Co. to test valves on the upper 37 miles of its pipeline. Regulators also ordered the company to focus on valves in populated or environmentally sensitive areas and add more if they are needed.
The block valve near the spill can be closed to stop the flow of gasoline or other liquid fuel coming from refineries in Cherry Point and Ferndale. It works by remote control, closed by operators in a Renton office.
Olympic officials are reviewing the preliminary findings and aren't certain that the valve played a major role, said Joann Hamick, a spokeswoman for Olympic Pipe Line Co.
"We don't know," she said. "It's something they (regulators) put there and it needs to be looked at."
She said the company has already begun complying with the Transportation Department's order, including reducing pressure on some segments of the pipeline.
Investigators trying to determine a cause of the spill and to assess its aftermath will pay attention to the valve, said Keith Holloway, a spokesman for the National Transportation Safety Board.
"We are looking at the SCADA (computer) system and operations of valves," he said.
SCADA is a computerized system by which Olympic controls, monitors and detects leaks in the pipeline.
It wasn't working right before and during the spill, according to preliminary investigative evidence cited by the Transportation Department. "System response time was reported as slow, and the system's recording of some data was not consistent with normal operation."
Olympic officials say about an hour before they had the first reports of a possible leak their computer system crashed. ***********************************
What I really want to know here, is why the computer system crashed? This example, although in the oil industry is a very real demonstration of how multiple fail safe systems can still be overcome by Mr. Murphy (of Murphy's law fame).
Time and again I hear about the wonderful progress of the electric industry regarding Y2K. If the industry were on target to meet Y2K and every utility in the country had already completed it's remediation, I'd still be nervous come Jan 1. They haven't. Why not? Why are the actual percentages of completion of "readiness" so darn low? You guys have posted 10 or 20 press releases which does me no great comfort. Ahh...but we still have 6 months left right?
-- Anonymous, June 26, 1999
Six months is a long time, Jim. Having the benefit of experience, I wouldn't be afraid to take on a power plant that was starting from scratch right now if I had full management support (excluding major corporate mainframe programs).Regards
-- Anonymous, June 26, 1999
Factfinder if there were more of you around the country we'd probably be seeing more readiness announcements.Talk to your boss and see if they'll lease you out to some of these other slackers and get'em up to snuff.
And no, i'm not being sarcastic, i'm serious.
-- Anonymous, June 26, 1999
Lane, No straw men, not at all! When I refer to everyone getting wild at the least little y2k bug, while appearing to be oblivious to often much more signficant bugs found in new software code, I had YOU in mind - a real and genuine person who tends to follow me around from thread to thread ...lol :)Regards,
-- Anonymous, June 26, 1999
You get things fixed today by calling to locate a replacement part, and then having it shipped out by Fedex for installation tomorrow. This may not be possible after January 1, and repair may take considerably longer.
-- Anonymous, June 27, 1999
This appears to be proof of serious Y2K bugs. 3 to 4 weeks for repairs may be too long for some companies, wouldn't you think?
-- Anonymous, June 29, 1999
off
-- Anonymous, June 29, 1999
off dammit !
-- Anonymous, June 29, 1999
Sorry Marianne (oops, i mean marianne), nice analogy, but it really doesn't work. In the case of computer programs, starting from scratch really is usually easier than re-doing existing. What works in wiring a house doesn't work in writing software. P.S. There are a lot of other fields where this also holds true. You should ask something like "Which is easier to repair an old copier, or wheel in a new one?"
-- Anonymous, June 30, 1999
well walt... you got it all wrong. you and i are in complete agreement. i was responding to factfinders contention that he never saw a new scada system go in without some problems and his assumption that "if it were just a fix," it would be alot less of a problem. he states that were there are new problems there are new bugs... my arguement is that there are *more* problems and bugs with the reworked software.
my contention was, and still is, that rewriting is much more of a problem than starting from scratch.
this from factfinder:
Malcom, I've never seen a whole new SCADA system go in without SOME kind of bugs. If this were just a "fix", I don't believe that you would have had such problems. Where there's new software, there's new bugs.
my response:
m) you are truly amazing. let me ask you a question. is it easier to wire a new building or rewire an old buiding?
btw walt... it is easier to wire a new building than rewire an old one which is why i used that analogy.
the new building is ready for the electricans to run the wires and they are not impeded with closed walls and renovations that are not on the original plans.
it is easier to write a new program from scratch... horrendous problems trying to rewrite someone elses programs and second guess every little jot and tittle.
this is what factfinder believes to be true... it is easier to "fix" an existing program. i find this a scary deduction on the part of one who is remediating the code for the electrical plants.
-- Anonymous, June 30, 1999