Folks: There appears to be a fair number of highly qualified engineers that have stated with some excellent facts that "all is well". I've seen this on a number of forums and even at this forum on June 04 under the title "How is electric Industry completing remediation so quickly". Lane Core exchanged blows with the "Engineer" and after about 5 rounds, I thought Lane was the victim of a TKO. The engineer used good solid facts and not a lot of rhetoric. He was backed up by another engineer who also came across as credible. If this had been a TV debate, people would have turned off their sets saying there is no problem in the elctric industry. I can understand why the corporate suits preach the "all is well" spin, but I'm at a loss to understand why the guys with the hands-on experience are asserting that there will not be much of a problem. Comments PLS ?

-- Anonymous, June 12, 1999

Answers

Hi Chuck: The answer is simple: The reason we who are actually testing and remediating systems say that all is well, is because all really is well.

We've Y2k tested hundreds of kinds of devices (relays, RTU's, controllers, etc.) and all of them pass our tests with flying colors. Further, we've performed integrated tests, and even some have used EPROM readers to examine the vendor's code. All tests come up the same...Y2k ready. We forward the DCS system clock with the power plant on line and connected to the grid...the power plant stays on line.

However, we are not giving up just yet...we are continuing to test any new devices (or old devices with new firmware revisions) that come up on the market, and are now testing non mission-critical software programs to see if we can find that elusive Y2k bug. So the final verdict is still not out yet.

In summary, it is not spin...It is reality.

-- Anonymous, June 12, 1999

I'm still waiting for you engineers to post a "WE'RE Y2K COMPLIANT". Let me know when you do.

And if you're really, really ready then step out and tell management to tell the lawyers to back off.

Make the statement.

-- Anonymous, June 13, 1999

Chuck, there are some engineers who have posted the "all is well" message on this forum since about February, if I remember correctly. I have also read posts on this forum (and others) from industry insiders which express less confidence and exhibit a measure of doubt about how well remediation is proceeding in some places.

I look at it this way. There are 10, 421 generating plant units in the United States, 136 control areas, 64 operating centers, 8 regional ISO's, as well as power brokers, non-generating cooperatives, and all their attendant suppliers and vendors. (telecoms, fuel, water, and parts and upgrades supply) If you access the most recent NERC monthly survey summary for April of this year (http://www.nerc.com/y2k) and go to the General Planning tab of the spreadsheet, you can read other comments written by _industry_ people as well.

For instance, in answer to the NERC question, "List the greatest obstacles your organization faces in achieving Y2K readiness by December 31, 1999," we do NOT find a consensus of "there are no problems". (And notice the question is not talking about meeting the NERC June 30 deadline, but being ready by the end of the year.) Some examples of what industry people did answer to this question include:

"Difficulty in finding time to do the Y2K tasks."

"Verifying telecommunication status."

"...timely availability of required software and embedded chip upgrades. This is compounded by the need for the upgrades to be implemented during scheduled outages."

"FUNDING; STAFFING RESOURCES TO ADEQUATELY TEST EVERYTHING!" [caps used on spreadsheet, I did not add them)

"Awaiting solutions from system manufacturers. If they do not come through with corrective action and upgrades, there may not be sufficient time or availability of new equipment to meet the year end. Testing -- production schedules make time allowance for testing very limited/difficult."

"THE AVAILABILITY OF COMPANIES TO ASSIST IN REMEDIATION OF EQUIPMENT." [again, caps original, not mine)

"PROGRAM CODE REMEDIATED BY CONTRACTOR NOT OPERATING SUCCESSFULLY." (original caps)

(getting)"Major combustion turbine DCS supplier information and support."

"Integrated testing of systems may be difficult if not impossible on actual production systems. The interface and verification of readiness between EMS and Telecommunications is probably the most critical area."

"The biggest challenge we face is getting information from our critical vendors and suppliers. Another concern is whether this information is correct."

"Maintaining sufficient resources (in-house, contract and vendor) to complete Y2K tasks."

Now, the utility people who wrote these comments were guaranteed anonymity for their companys by NERC, and there are more answers than I have posted here. (There are also other answers to a question about concerns beyond organizational control.) So you see, Chuck, it's a mixed bag of news from inside the industry. And we have not counted the Canadian and Mexican components which are part of the U.S. electrical grids. Does it seem to be common sense to you that with over ten thousand generating units and lots of other control areas, and all the different company sizes, financial situations, staffing capabilities, etc., that there would be variations? Personally, I would be very surprised if there weren't a lot of variables, but every one has their own outlook on both that subject and the differences in human nature and competency. You'll have to apply your own perpective to the overall picture.

-- Anonymous, June 13, 1999

And there is this (NRC Y2K Information Notice 99-12):

There are mission critical non-safety-related functions such as digital feedwater controls, moisture separator reheater controls, reactor recirculating coolant controls, and motor generator set controls that are affected by the Y2K concern and have required remediation. These balance-of-plant functions are critical for power generation.

-- Anonymous, June 14, 1999

I am a mechanical engineer by profession,however, I have very little knowledge concerning power generation other than what I've read lately. I think we can all agree that the grid staying up is going to be the one of the most important issues in the coming months. It could be the difference between simply a difficult time versus catastrophic problems.

I have a few questions that hopefully someone can clarify for me. If these have been covered elsewhere I apologize.

1. I see quite a bit of positive feedback (which if accurate is quite encouraging) on this forum saying that few if any systems that have been tested would have caused critical failures even if they had been left uncorrected (am I misstating this?). If this is the case then why are there constant news reports concerning how horrible things are going to be (with reference to nuclear and non-nuclear power) in Asia, Eastern Europe, South America, Africa and other areas of the world but not in the US? What makes these systems different from ours? I find it hard to believe that there are that many manufacturers of the necessary equipment used in power plants, and would tend to think they have common systems.

2. Does this rosy scenario hold true for ALL types of power generation plants or just certain types? Are some types more prone to failure than others?

If someone could clarify these questions for me I would be most grateful.

John

-- Anonymous, June 14, 1999

First let me second what Dan wrote.

Second because we actually work on the equipment and know. We don't have to read NERC statements etc. When you know how the system(s) work and work together you have a real understanding. Not a: "I read this NERC statement" understanding. Doing something for 20 or 25 years, or more gives you a much better understanding that what someone picks up just by reading a few industry handouts.

As for why we aren't saying we are Y2K compliant. I for one don't have a clue as to what that really means anymore. It's become more of "charge". Like saying you have an "agenda". People tend to take it as meaning everything in total compliance, not just the important stuff. Also there is the chance that if there is an outage completely unrelated to Y2K, except by timing, it will still get blamed on Y2K. I expect anything from now until July of 2000 to be blamed on Y2K. And since junk science seems to be the rule in court rooms these days why take a chance? The people writing these statements also dont have a hands on perspective from working on the equipment. So they tend to not understand the difference between something essential and something nonessential.

Also you have to realize that in a lot of cases the people working on Y2K finished up some months ago but are still charging work to the accounts. Its a hot ticket so you can get all sorts of work done by charging it off to Y2K where it wouldnt be approved on its own.

-- Anonymous, June 14, 1999

**John, a question like yours was asked on this forum a while ago by the reporter Drew Parkhill. It was titled, "Question for Optimists: If U.S. Won't Have Power Problems, Why Should Any Other Country?" The link for this thread is at: http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000nY1

Dan the Power Man is the only industry person who posted some possible reasons for the reported concerns over potential electric industry problems in various places overseas. His comment is the second one from the bottom of the thread. The last comment from Mr. Parkhill contains other questions about the "showstopper" issue. Perhaps more comments will be forthcoming in answer to your question here.

**Engineer, I respect the information you give on what you have personally found re systems, but by stating, "The people writing these statements also dont have a hands on perspective from working on the equipment," (re the NERC survey statements) you're asserting that those who are giving survey answers are not Y2K project leaders, or are not even talking to project engineers to get an assessment of the company's situation. Following that logic would lead us to believe that all industry engineers have this big "no problem" secret -- but they're not telling anybody else in their organization! You're saying the people answering the NERC survey questions don't really know what's going on? You want us to believe that what you, as an individual, writes, applies to every single engineer's situation in the 10,000 plus U.S. generating units, and a hundred plus control centers, but we aren't supposed to believe what anybody else in the industry says to NERC if it portends potential problems? That's a huge "we" you're using. (Industry engineers aren't telling anybody else in their own companies about the difference between what's essential and what's not? Or they are but nobody is listening to them?)

Not only that, if, as you assert, there truly are no Y2K problems in the electric industry which *could* affect power getting to homes and businesses, even if they are not remediated in time, then you're telling us that the industry people are ripping off their customers and stockholders by expensing completely unnecessary testing work and undertaking completely unnecessary contingency plans (no vacations, stockpiling fuel, training, back up communications systems, etc.) just because they don't have a "real understanding" either? And all this at a time when deregulation issues have made fiscal cost effectiveness for utilities of critical importance, by their own admission? Utility executives are just throwing away money because they don't understand there's really no good reason these expenses are not needed? Due diligence only has to go so far to provide legal coverage. Or are you saying that Y2K engineers just haven't been able to convince utility management, NERC, APPA, NRECA, the NRC etc. that it's not really important whether a utility is "ready" or not, because there are no problems anywhere anyway?

When you speak for your own findings, fine. When you imply that you're aware of everything that's going on in every part of the entire electric industry, and that you're sure that all of the ongoing testing everywhere has and will not show any critical problems at all (or that any potential problems written about by those in the industry are from utility people and oversight agencies with no understanding) then that comes closer to a claim of omniscience than a personal assessment.

The uncommunicative industry scenario you imply raises my concerns higher than one in which there are variables in design, programming, vendors, etc. from one plant to the next, which all need to be taken into account and in which not everyone holds the same airtight assurances but are addressing potential problems as best they can.

-- Anonymous, June 14, 1999

Ahhhh Bonnie, I love you,

I was about to take a couple more Excedrin but can put that down now. As usual, you cut through the smoke and refocus the attention where it belongs. I too welcome any good news from any segment of any industry, but when I read the sweeping "all's well" statements such as above, my head throbs. I can never figure who I'm listening too, a super ego or a PR Pro. I really appreciate it when you step in and re- establish a basic framework and basic logic regarding all the reports.

-- Anonymous, June 14, 1999

I can never figure who I'm listening too, a super ego or a PR Pro.

That just about sums up my reaction, too.

-- Anonymous, June 14, 1999

There are two reasons for automating any process - human or mechanical. One - is simply to save labor, cut costs, eliminate needless drudgery. The second, and for this discussion, FAR MORE IMPORTANT reason for automation, is increased PRECISION and reliability over time. Yes, lets assume Engineer is right, and that everything can be run manually. The very next thing we have to ask is, WHY HAS IT BEEN AUTOMATED?? If it was automated just to save money and avoid needless drudgery, then no big deal. Hire more people and have the crews put in more overtime. BUT, if the purpose of automation was increased precision and reliability over time, then suddenly "we have a problem Houston." For instance, a computer automatically monitors temperature. No problem, any human can do that by watching an analog thermometor. Right? Sure, of course. But where Y2K comes in is this. In "normal" times, it would be rare to have two distinctly separate systems go down simultaneously. Even more "extraordinarily" rare for three systems. And so on down the line - forming a nice bell curve of the probability for simultaneous failures. In the normal world, when the computer that monitors temperature fails, you just have someone babysit the thermometor until the computer comes back on line. Since that thermometor is "important", its not likely to get overlooked. Computer is probably fixed in a couple days - or even weeks - and things return to normal. But what if six automated systems fail? Now you have do a LOT more babysitting. And here's the rub. WHEN SYSTEMS ARE AUTOMATED FOR PRECISION PURPOSES, THEY ALSO HAVE A LABOR REDUCING EFFECT. What happens with automation is you end up with fewer and fewer trained people monitoring more and more systems - because the computers are helping them. Even assuming that everything could be run manually, it would only take three or four automated systems to go offline to spark a "labor shortage" crisis. Where are all the trained people who have been gradually phased out over the years going to come from? Where???? Sure Engineer may know how to run things manually, but he can't be in all places at once. It gets right back to Gary North's "breakdown in the division of labor." If power fails, WE ALL FAIL. If ANY industry needs contingency plans and extra people trained in case things have to be run manually, ITS THE POWER INDUSTRY!! So WHERE ARE THE DETAILS OF THESE PLANS?? I am SICK of happy-face, I want FACTS!!

-- Anonymous, June 15, 1999

Brian,

That was a good overview of the problem in the "real world." I'm retired from the airline industry (pilot) and the same situation you are describing applies there. Lose the autopilot while cruising at high altitude and you *can* fly it manually, but not with the precision that the autopilot provided. It becomes a handfull and very tiresome after an hour or two. The supersonics, such as the Concorde, can not be precisely managed by hand flying at all. Then we go to the newest computer driven, fly-by-wire equipment, in which they have reduced the flight crew from 3 to 2. Even more problems if you try to go manual, and some of the technology is designed without any manual backups now, just redundant computer control. If any pilot of modern equipment was to suggest that it's really no big problem to fly a long distance international flight using only the pilot's own inputs, I'd be as disappointed in their statement as I am with those we get from people like Engineer. And I'd wonder just where that pilot was coming from, and what their agenda was.

-- Anonymous, June 15, 1999

Gordon,

Operating T&D system without SCADA is kinda like operating a Cessna without the use of a trim wheel. Or perhaps like having to hand crank landing gear.

-- Anonymous, June 15, 1999

CL,

Come on, give me a break. You go into the control room of any large electric utility, see all the puter screens, and all the people, busy keeping that complex machine rolling, and you want to tell me it's like operating a little Cessna. Sheeesh. Now, if you want to tell me about how simple your own 8kw portable home generator is to operate, I could get into *that* with you.

-- Anonymous, June 15, 1999

Oh, come on, Gordon. Isn't it obvious? Thirty-five years of computerization has been, basically, irrelevant. Geesh. Get with it, man.

-- Anonymous, June 15, 1999

Gordon, Sheesh, I was referenced T&D SCADA systems not generation. And, if you've never been in a Cessna before, it looks pretty intimidating. The analogy to the SCADA system is really quite applicable, in scale. Have you ever been in a substation??? And to answer a challenge, my personal opinion is that my company's T&D system is Y2K Ready. oh yea, FOR IMMEDIATE RELEASE.

-- Anonymous, June 15, 1999

As a software developer at a major power entity, I would have to say that I think that the industry still has a major problem. Even if the grid hardware automation is simple, the information systems are not.

If it can be proven that the power companies don't need information, then I think that we could safely assume that Y2K won't be a problem. Until then, I need more evidence that the information systems problems have been eliminated. Shouldn't I expect an answer other than "Trust me" from management and lawyers that don't understand what I do? What kind of comfort do you think I can take? Do you think I have any concerns for my family when I am not given any reason to think that the systems are ready? What is the wisdom in saying "We will be fine" when the people who say it don't understand software projects?

I have been saying that the world may not loose it's function, but it just might loose it's mind. (I'm not so sure it hasn't already)

-- Anonymous, June 15, 1999

Reporter, Thanks.

CL, I wish I knew you personally and you could take me through your facility to see all the stuff you are talking about. I sincerely love to see complex electo-mechanical systems. Probably a Rube Goldberg trait in me, where the more complex and intimidating it looks, the bigger I grin. Now, one other thing I forgot to ask. You do have a personal portable generator of your own ready, don't you? I have heard that many of your compatriots in the industry are doing that.

-- Anonymous, June 15, 1999

Gordon,

Complex and electro-mechanical in the same sentence??? Either you are a serious engineering geek (a compliment by the way!!), or very adept at subtle sarcasm. I'm afraid the e/m stuff is really not much to look at.

To answer your question, no I don't have a portable generator. I'm debating whether to pick up a slightly used one at the massive "yard sale" come January 5th.

-- Anonymous, June 15, 1999

CL,

I think I detected a Freudian slip there. Just what makes you think those portable gen sets are going to *used* by Jan 5th. Come on now, be honest.

-- Anonymous, June 16, 1999

I wonder if CL is one of those people that thinks everyone employed in all these power companies are all interchangable-cog "competant profesionals". I.e., all these people around the country who've been taking automation for granted for the last quarter century, are going to be to step up to the plate and "hand crank down that ole landing gear" within 5 minutes of midnight so that I don't have time to turn my new "in the box" generator into a "used one" so that he could buy it off me at half price during halftime of the Rose Bowl (Go MSU!!)?? And if going manual was as easy as turning a crank, than power companies are competing with McDonald's for employees?

-- Anonymous, June 16, 1999

Brian,

I think you done broke the code. You've got the picture just fine. But CL is still working on his picture. I can sense that he is getting real close, but still has few more dots to connect before he can step back and say "Oh, *I* get it now!" It just takes some of us a little longer to see it. The fact that he is even thinking about buying some backup power for himself is a positive sign. I'm sure he knows some folks in his own company that are doing, or have already done, just that. Then there's this business about some railroads installing large diesel gen sets at some river bridge crossings. CL will see that sort of thing and rightfully begin to wonder what they might know that he does not. We don't all progress in our understanding of these things at the same pace.

-- Anonymous, June 16, 1999

1. No mission critical Y2K devices failed in a manner that would in any way jeopardize power transmission and distribution. These results I have in my posession and have seen and heard from countless utilities that have independently tested with the same results. 2. We can operate the system manually. We do it to a large extent now, as so ably noted by Dan, Engineer etc. We have drilled operating with a failure of an external provider (telecom) that would defeat SCADA. The drill proved that this can be done. We are ready to operate this way. Name the system that will failed (based on testing) to cause us have to operate this way. Describe the sequence of events that will lead to you cranking your high dollar toys. This challenge has been put forth before. Describe your scenario and name the faulty devices and their failure modes. 3. You misinterpret the analogy directed to Gordon. The difficulty of flying a plane without a trim wheel relative to using a trim wheel - the difficulty of landing an airplane with manual gear vs. motorized doesn't indicate the level of competence necessary to fly - just the relative difficulty of some contingencies. Not that I'm knocking the abilities or contributions of all MSU grads at the McDonalds around the country. (grin - GO IRISH)

-- Anonymous, June 16, 1999

Oh yea, I forgot one clarification. I don't own a generator. Might get one for the camper (to run the A/C at place that don't have hookups) if the price is right next year.

This is not meant to discourage anyone from purchasing one. Y2K will not impact the reliability of the electric system. If you have unreliable power now, and winters are touch and go with reliability, then by all means do what you must. Please get advice from your local utility or a liscensed electrician to ensure your safety and the safety of your utility workers. But if your power is fairly reliable now, expect the same (there will always be sporadic outages, and no utility will guarantee "no outages" now, tomorrow, Y2K, or beyond - never have, never will).

-- Anonymous, June 16, 1999

CL,

Well, I can see you are struggling with this stuff. I am glad to see you are thinking about a generator. Just for your own information, you will need at least a 4kw gen set, with a 30amp 120v outlet, to start up that AC unit, and those units weigh at least 100 lbs. So keep that in mind. See, I'm really trying to help with your contingency plans. But, why don't you seriously think about buying that gen set right now. You have the perfect excuse for your friends and neighbors (who you have been telling that there is no problem). Just say you're doing it for *this* summer's camper season. After all, if you *are* wrong about the economy and electric reliability, there won't be any gen sets available next January because the National Guard will have them all. And, there won't be a camping season either. So why not just play it safe? Nothing to lose. Just move your current rationalization up to this season, and be happy.

-- Anonymous, June 16, 1999

Since I'm kinda late to this discussion, I'd like some clarification from CL, is he just refering to traditional coal-fired, hydro, etc, utilities, or is he trying to tell us that the seventy-some nukes in this country are also as "safe as they ever have been" - avec Y2K ...

come back ...

-- Anonymous, June 16, 1999

CL,

I've been rereading your above posts and think I may have overestimated the type of facility that you are running. You said, "I'm afraid the e/m stuff is really not much to look at." So, I'm wondering, are you running one of those small rural plants that have maybe a big waterwheel in the river, and one of those long fat leather belts that turn a generator set? The kind that only have a few analog instruments, some switches and relays, housed in a small wood shack? You don't have to name the place, cause I know you feel sensitive about that information, but if I'm right here, or pretty close, it would go a long way toward explaning your current thinking.

-- Anonymous, June 16, 1999

I read this from "somebody" "called" "cl":

1. No mission critical Y2K devices failed in a manner that would in any way jeopardize power transmission and distribution. These results I have in my posession and have seen and heard from countless utilities that have independently tested with the same results.

And I read this from the NRC:

There are mission critical non-safety-related functions such as digital feedwater controls, moisture separator reheater controls, reactor recirculating coolant controls, and motor generator set controls that are affected by the Y2K concern and have required remediation. These balance-of-plant functions are critical for power generation.

Somebody's lying to me. I don't think it's the NRC auditors.

-- Anonymous, June 16, 1999

Lane, I gotta jump in here. Your post, as written, compares apples and oranges.

I'm not defending CL's position. That's up to CL. But CL is talking about power transmission and distribution. You're talking about power generating facilities (nuclear, in particular). These are two completely separate and distinct segments of a power delivery system. The NRC's comments, in this case, are not applicable to the T&D world.

-- Anonymous, June 16, 1999

Rick,

When an intelligent point is made that questions one of my assertions, I'll defend it. In the mean time, how did that poem of yours go... about the trolls and such???

-- Anonymous, June 16, 1999

gosh, hi cl... yourrrrrr baaccccckkkkkkk

i noticed a funny pattern with you guys. when one of you takes an inordinate amount of heat... he kinda fades away for awhile then, voila... another takes his place.

sort of like a game... or an attack team.

-- Anonymous, June 17, 1999

Thanks, Rick. Sorry if I jumped to conclusions. If that's what I've done....

-- Anonymous, June 17, 1999

From Dan the Power Man:

We've Y2k tested hundreds of kinds of devices (relays, RTU's, controllers, etc.) and all of them pass our tests with flying colors. Further, we've performed integrated tests, and even some have used EPROM readers to examine the vendor's code. All tests come up the same...Y2k ready. We forward the DCS system clock with the power plant on line and connected to the grid...the power plant stays on line.

Does that refer only to transmission and distribution?

Thanks....

-- Anonymous, June 17, 1999

Having worked in over 30 high tech companies over 20 years as a consultant and contractor, I have come to two basic conclusions concerning what I can and can not believe from people in the industry.

1. Managers, PR people, sales people, and other non-technical types lie and mislead as a matter of course. Its almost like they are incapable of correctly presenting the facts. 2. Engineering types rarely intentionally lie or mislead people about the technical details of what they do. Its not that they are on any higher moral ground, just that they might be found out and proven to be wrong or incompetent. Engineers HATE to be wrong, and will avoid even the appearance of being wrong at all cost. Ego is very much a part of what we do. (This does not apply to engineers that have succumbed to the dark side and gone into management).

Applying this to the current discussion I am convinced that the experts on this board are being honest with respect to their area of expertise. Both sides truly believe what they are saying. What disturbs me deeply is that they have diametrically opposed conclusions.

The things that concern me are how true their person experiences are industry wide and across the various type of power generation. Engineers also tend to be somewhat tunneled-visioned, and will almost never admit that they can solve a particular problem, and often believe that there expertise in their particular field translates into expertise in other fields. This is rarely the case.

Being an engineer I am uncomfortable with not being able to form accurate conclusions. I have spent way too many hours trying to reach a decision on my risk factor with power next year. I am coming to the conclusion that I will not be able to form an intelligent decision until the time gets here, at which point it will be moot. Far too much conflicting data.

Therefore I will err on the side of caution.

It is completely impractical for most people to generate their own power for any extended length of time. The logistics are far too complex. Therefore I will not be buying a generator, my preparations will be more passive. I will simply be prepared to do without power for a short period. If things go worst case scenario a generator will make little difference in the long run. I prefer to spend my time and resources preparing for things that in MY personal experience that I know for a fact are a weak link, and simply hope that the power stays on. Im going to be really POed if Im being lied to.

I expect many, many problems both large and small that have the more immediate possibility of adversely affecting my life. I see a very rough ride ahead, and I see no way to predict with any accuracy what these problems will be.

I hope that the experts who believe the power grid will stay up are being honest and are correct, without power everything else becomes far more complicated.

-- Anonymous, June 17, 1999

John,

Thanks for that very straightforward commentary. It rings true, and is well presented. I couldn't have said it better myself.

-- Anonymous, June 17, 1999

I agree with Gordon. Thanks for a thoughtful post.

1. Managers, PR people, sales people, and other non-technical types lie and mislead as a matter of course. It's almost like they are incapable of correctly presenting the facts. 2. Engineering types rarely intentionally lie or mislead people about the technical details of what they do....

Exactly. And that's why it has disturbed me so much when I have seen articles by engineers posted here that read like press releases.

-- Anonymous, June 17, 1999