CL and I were discussing embedded systems and testing a while back, and he sort of disappeared from the thread. We basically came to the conclusion that in order to test a back-box embeddeded system, you need to basically set the date, exercise the functions of the device, and watch what happens.

My point is that, having done that, you cannot possibly know that you have tested every logic path in the software running on the embedded device. Software bugs are fiendishly hard to find sometimes, even when you have the source code sitting in front of you and can run the code through a symbolic stepping debugger.

I maintain that you can't adequately test a black-box embedded device. Certainly not in an afternoon. You can try, but there may be some combinations of dates and input sensor values that cause it to do something it shouldn't. I've seen some really strange bugs that shouldn't happen, but do because some obscure combination of conditions make something happen that wasn't accounted for in the code, or by the users of the system.

I think the reason some utilities are getting done much quicker than we thought they would last year is that they are not testing sufficiently. They are doing what amounts to superficial testing on some devices, and trusting that. I'm probably going to get nasty flames for that statement, but that's okay. I'm a big tough guy, I can handle it :-)

Jon

-- Anonymous, May 20, 1999

Answers

Jon

You mean you ASSume that utilities are doing superficial testing. You have absolutely no competence to gauge the complexity of discreet protectin and control devices. You do not know the complexity of the processes - you guess, you do not know the complexity of normal acceptance and design testing - you guess. This type of unsubstantiated (YES, UNSUBSTANTIATED) charge in such a public forum is downright irresponsible. It is actually in the category of being beneath worthiness of a reply. (I only reply because of the damage your recklessness can cause to my hard work - and I hope the real intellectuals here (FF, Engineer, Dan) respond to this in an actually coherent manner.)

The actual reason we are done (and if you have been doing this as long as I have you would not say quickly - only finally)is that there were not very many date aware devices, and these had few problems.

As you stated, the protective relays that I test sound pretty simple to you - I guess they're not rocket science. But they represent 95% of the date aware devices on the T&D system. And the other 5% are simpler by orders of magnitude.

You see, the importance of electric utility grid is something that might be a new relevation to you Y2K folks, but it has been a way of life for utility engineers. The sense of conservatism in engineerng design at utilities is unreal. Electromechanical relays still protect and control the majority of the grid! A saying we have when installing something "new" and innovative - "we're throwing caution to the wind and jumping headlong into the 80's". We do not quickly jump on new technology because the existing stuff is proven and is at its heart, rarely improved upon by computer algorithms. Heck, can you name me one microprocessor based relay that is as quick as the electromechanical KD relay, or the TTL/static technology Uniflex system? Rick, step in and back me up on this at any time... (How many protection and control engineers does it take to change a lightbulb? 4. 1 to change the bulb and 3 to discuss how good the old one was).

And ANOTHER thing, at least one utility is doing chip level validation of our "black box" tests. A uP simulator is replaced in the socket, and test leads are clipped on pins of RTC's and communications ports. All data is monitored to look for patterns that could be dates. This method has corroborated the "black box" method in each case. The only finding of caution was that one particular device did not access the RTC to update time/date for a full 2 min after rollover. This is not a problem, as we anticipated it in our test procedures.

Suggestion, why don't you search out a GM test plan and look it over. By the way, what problem do YOU have with my test approach, either the description I gave for a protective relay, or the theoretical RTD example you posed?

One last point. What is the degree of certainty with which you make this rather serious charge? How did you arrive at this degree of certainty? How do you explain the utilities that have generation and entire T&D systems operating daily in the Year 2000 now without problems?

-- Anonymous, May 20, 1999

See, I knew that would get an answer :-)

CL, you make it sound like I'm the president of the US, and just dumped on you personally. I'm a nobody, and I merely stated my opinion, for what it's worth.

From the testing methods that you described to me in an earlier thread, I stated my views on the fact that what you stamp as compliant using that testing method may not be.

It's very simple. I don't care what kind of testing you do with a black-box, you can't possibly know that you've tested every possible piece of code in it unless you do an exhaustive source code review, and even then things can come back and bite you.

Now, please understand when I say "you" above I'm not talking about you CL in particular, but Y2K remediation people in the utilities in general. I understand in the T&D neck of the woods there's not much in the way of digital devices, but on the generation side of things there are.

You say the reason they haven't found any problems is because there aren't any -- I say maybe they're not testing enough. In fact, I don't believe it is possible to test a black-box system with any large degree of certainty for something like this. You can test it and be mostly sure, but there may be some combination of factors that don't come up until the system is in place, and has been stressed a certain way, and so on.

I'm not a utility engineer, but I have been programming computers for 15 years, and I do know what I'm talking about.

Jon

-- Anonymous, May 20, 1999

Well hello CL and Jon. I see you are in the midst of an interesting discussion. FWIW, Jon, these very issues were hotly debated amongst utilities back in 1997. Here's what the Power Man says (which is pretty much another way of saying what CL and The Engineer have said):

First, we really are not doing "black box" testing. We know that T&D equipment, the relatively few out there that are microprocessor- based, use a separate real time clock chip. This RTC is NEVER used in an essential function in mission critical devices. So from a strictly technical standpoint, there is no compelling reason even to Y2k test such a device.

But we have to be sure about this. So we talked to those folks who write the code for these devices--programmers like you, Jon. A few utilities purchased PROM readers and scanned the software for use of date functions. Their results were the same as our tests: Date is NOT used in an essential function, and devices maintain essential functions regardless of the date. Those same experts who write code confirmed our suspicion that the few date functions used CAN be tested for Y2k readiness by forwarding the clock.

Still undaunted, we contacted the manufacturers and vendors of the equipment we buy. They further confirmed everything I've said thus far. Remember, Jon...we don't buy chips, we buy devices with chips, so ultimately the manufacturer knows what is inside the box.

And so we continued to test. Thousands of tests have been performed, confirming ours and the experts' hypothesis. Tests continue to this day; all commonly used devices have been tested, and each utility is responsible for testing any one of a kind devices.

If there was anything else we could reasonably do, we would Jon. Did we miss anything?

Dan

-- Anonymous, May 21, 1999

Dan wrote:

> If there was anything else we could reasonably do, we would Jon. > Did we miss anything?

Well, I don't know. I don't suspect you do, either. At least not until a device causes a problem.

The problem is, the way you described how you do testing is a world apart from how you described it in an earlier thread. You said you basically connect a laptop via a serial port, try the various dates, and it's all wrapped up for a given device in about half a day.

I'm sorry, but you can't adequetely test a black-box device in half a day. You might be 95% sure that it will work, but life (and software) is never that simple.

So, what you're telling me is that in most cases, you are running fairly straight-forward functionality tests on them, plus listening to what the vendor says about them.

Is that correct?

Jon

-- Anonymous, May 21, 1999

Jon,

Might I add some history. Electromechanical technology (as I assert above) has done an admirable job of protecting the complex power system. The engineering used magnets, coils and flux interaction equations to build these E/M devices.

The technology to migrate this protection to uP based devices existed long before it was economically feasable. For many years, engineers developed the algorithms for power system protection, but never actually made a comercially viable product until memory began to get cheap in the 80's. My point is that the protective functions of these devices are well known and age old. It is quite possible for a protection engineer to know that a device is operating properly after a rollover.

Perhaps you are thinking more in the terms of process controls where large parts of the functionality is user developed. The protective algorithms in our relays are fixed by the vendor. Our user flexible programming is basically limited in most cases to mapping outputs.

Some utilities may have built PLC based programs that are as complex as you imply, but I doubt it. Most application papers I've seen use multiple controllers to segregate functions (unlike computer apps folks, our design criteria includes an analysis of common mode failures and single points of failure). We try to avoid putting all the eggs in one basket from a contingency point of view and also for ease of maintenance.

-- Anonymous, May 21, 1999

CL, Dan, Jon, and anyone else interested in this topic...

TransAlta Utilities (Canada) provided a "black box" testing procedure that has been resident in the euy2k.com toolbox for quite some time. I'd appreciate your review / feedback / response regarding the accuracy and usefulness of the document, and whether TransAlta's approach matches yours.

-- Anonymous, May 21, 1999

CL,

I think is it is very 'responsible' of Jon to bring this up on a public forum. I've been wondering how you can test these embedded chips, myself. On a public forum, people (like yourself) can correct this "unsubstantiated" claim, and maybe prevent some panic. Or on flip side of the coin, help to bring this up with the testers, so that someone will start to work on finding ways to test them. We all come to this site, and sites like it, to find out what is really going on behind the headlines. I'm glad that there are people in this forum, who are knowledgable enough to answer the tough questions.

-- Anonymous, May 21, 1999

Hi to all, it's been a while since I've posted, but I like this topic.

In order of my preference in testing: System Black Box (equipment/device level) Descreet components (chip level)

You can't exactly test a "chip" for Y2K, you're gonna need the whole RTC circuit for that, and even this kind of testing offers absolutely no assurance that the firmware programming (or software programming for PCs) properly processes date/time data. Far better to test on the black box level. You guys go ahead and check each spark plug, I prefer to crank her up drive off anyday ;)

No testing or simulation is perfect, whether a vendors or a utilities, but based on the findings, there are zero examples that I know of "strange combinations" of sensor signals and firm/software logic causing a Y2K problem that would have been missed in standard Y2k testing. I'll go with the factual findings of testing over mythical speculation anyday.

Now, back to no testing is perfect - lets say we miss one out of a hundred y2k bugs in testing. Since it is highly likely that other utilities have the same or very similar devices and are testing with somewhat differing methods, we can and have found out about problems identified at other utilities through industry organizations and through peer contacts at other utilities.

Lets say we still miss 1 in 200, even after using all the industry and vendor findings and our own testing. By the results I have seen personally and through industry organizations, I estimate that 98% to 99.9% of Y2K bugs are minor in nature and will NOT have a significant impact on function of a given device. Even more, I estimate that FAR LESS THAN 1/10,000 Y2K bugs if left unidentified, are used in plant systems and are severe enough to cause a plant shutdown or a decrease in generation. (This is 1/10,000 BUGS, not devices.)

1/200 Bugs Missed * 1/10,000 Catestrophic Bugs = 1/2,000,000 Catestrophic Bugs Missed.... (I would put the range at 1/200,000 to 1/20,000,000 for a 95% confidence level)

I think that I am being way too pesimistic with these figures, by the way ;)

Regards,

-- Anonymous, May 21, 1999

So, Dan, CL, Fact:

From your experiences, and what you know of the power industry's Y2K efforts in the US, do you then- in your *opinion* (not a prediction)- expect no problems of any consequence next year? This is serious question, not anything else. Just sincerely inquiring.

-- Anonymous, May 22, 1999

Drew, Based on the actual industry findings concerning the types and severity of y2k bugs in the electric industry, I am confindent that there will be no disruptions of any significance in the delivery of electrical power caused by y2k bugs during the transition to Jan.1, 2000. This is an easy call. I wish I had as much factual information for other engineering tasks ahead of me:)

Now, I have a question for you. What disruptions to society and the infrastructure do you forsee due to the flaming of y2k fears (such as the upcoming 60 Minutes fiction on Y2K). Do you think all of the exaggeration will lead to massive hording, panic, etc, before January 1 even gets here? Will the facts slowly take hold, or will Y2K newsmongers continue to ignore them and milk this puppy to the end???

Regards, FactFinder

-- Anonymous, May 22, 1999

<<<>>>

Allow me to interrupt this discussion with a point of order. The illusion of objectivity and open-mindedness is highly compromised when a person calls something "exaggeration" and "fiction" before one has even seen it. Have you seen an advanced copy of what 60 Minutes is going to air this weekend?

And as for "facts", I believe that is what this forum attempts to uncover (especially when it comes to some 'warriors for truth' like Bonnie Camp). I can understand some level of frustration if you really feel like you've got a handle on things, but let's remember it is your own industry that allowed us to get to the brink of 2000 without having clearly established *years ago* that this was going to be your perceived "non-event." And also I hope you can admit that you are one person, with a limited amount of knowledge, and it is at least theoretically possible that there is something about Y2k's potential impact on electric power that you don't know.

I believe I can say without fear of contradiction that everyone here-- EVERYONE--hopes and prays you are correct in your assessment of the future as it pertains to power and Y2k. I think we can also agree that without electric power our world would be a very different place. So we are agreed that IF Y2k would have been (or is to be) a threat to power, it is a very significant fact that impacts all of us.

All of us want to get to the bottom of this. And none of us--NONE-- has all the answers, because we're not omniscient.

Please continue to challenge error where you truly see it, but be careful about making statements that are obviously unsupportable (such as passing judgment on something none of us has seen).

And finally, as for the panic and fear, yes it could be a problem. But remember that electricity is just one link in the chain, and that which some are preparing for is much broader than just a power industry problem. Please wait until about June of 2000 to ask me whether I feel foolish about any preparations that I have made, and to tell me that I was wrong. By then I will be prepared to agree with you... or perhaps the accuracy of our risk-assessment will be obvious.

-- Anonymous, May 22, 1999

Drew,

I think I'll even up the ante on FactFinder. I believe that there will be no lights out due to Y2K. Reason 1 - All the major utilities have put enough resources to catch and fix DCS systems (some will not be done until scheduled unit outages in the fall). Reason 2 - All the major utilities have put enough resources to verify that there are no operational problems in protective relaying and control devices. T&D system was never an issue, except for maybe home-grown SCADA and EMS programs. These have been addressed by major utilities and either were not a problem or have been remediated. Reason 3 - This leaves small, independent power generators and electric co-ops that may not have put the necessary resources into Y2K. Technology on cogen units I have seen are older, not date-aware. I have no data to suggest that a co-gen will trip for Y2K, but I won't rule it out either. I predict the low-load period of Y2K and the dilligence of the major utils will result in a net generation surplus, even in the absence of co-gens. Electric co-ops use the same protective relays that the major utilities use for T&D, so if they haven't tested it is no big deal. They would have just found the same results that FF, Dan, and I have found (over, and over, and over again).

Bob - you assert that noone wants Y2K to be a problem. I would disagree. Even though they would never admit it in public (and maybe not even to themselves), some who have invested significant $$$ in Y2K prep don't want to look foolish and "fringe". They need some degree of Y2K failure to validate themselves.

You also say that panic and fear 'could" cause a problem. No one, anywhere has any factual basis to predict widespread outages or problems with the electric system. Look close at all statements and you will see they are based on the CONSEQUENCES of electic utility failure and NOT probability. On the other hand, panic and fear will CERTAINLY cause problems. Johnny Carson made one offhand joke about toilet paper shortages and bingo...instant shortage. If every car in California buys just ONE additional gallon of gas (above normal consumption) the refineries of California will be emptied. No testing required - this is a certainty! Time for folks to think add probability into the risk assessment equation.

-- Anonymous, May 22, 1999

Some excellent comments about hording, CL...I think I'll make a few preparations for the results of the Y2K HYPE problem as a result...will pick up extra pack of toilet paper this week, and start riding around with my gas tank topped off ;)

Bob, I would have tried to be more open minded concerning the Y2k story that 60 Minutes will be airing, unfortunately a few quotes from the show clearly demonstrate that this report will itself not be open minded. To wit (or to nit wit):

'60 MINS' PLANNING NEW Y2K SCARE STORY; WATER, ELECTRICITY MAY BE CUT, SAYS MAG

Y2chaos on CBS-TV this weekend! Despite the billions being spent to deal with the Y2K computer bug, many cities in America are uncertain they'll be able to continue to provide basic services like water and electricity on Jan. 1, 2000, reports 60 MINUTES in a new shock story being readied for Sunday.

CBS' Steve Kroft is putting the finishing touches on his controversial report, according to network sources.

Mary Ellen Hanley, a computer systems specialist hired by the District of Columbia, tells wide-eyed Kroft that she believes Washington will continue to function on Jan. 1, 2000, but it must be prepared for what many cities could face.

"We think there will be some disruptions... localized in many cases if the supply chain works," Hanley tells the cameras.

"If power works, if gas works, if Bell Atlantic works... all of those are big ifs."

Another "expert" from the IT crowd tries to pretend to have a handle on y2k in industry, sigh...."A big if??" There are very strong reports from all of the above industries -power, gas, and telephones- indicating that the y2k bugs being found are not as significant as first beleived and will not cause signficant disruptions. I will go with the real experts anyday.

And Bob, I will not pass judgement on your personal preparations for Y2K, I have tried not to do that to anyone here...the only ones I have taken issue with are those who dispense advice to take extreme measures for y2k.

Taking the advice of those not directly working on Y2K problems such as the above quoted "expert"(and who do not have first hand knowledge) in the above cited industries, now THATS foolish..;)

Regards,

-- Anonymous, May 22, 1999

[Repost of above message. My source information was omitted during posting, presumably for using html tag symbols]

Some excellent comments about hording, CL...I think I'll make a few preparations for the results of the Y2K HYPE problem as a result...will pick up extra pack of toilet paper this week, and start riding around with my gas tank topped off ;) Bob, I would have tried to be more open minded concerning the Y2k story that 60 Minutes will be airing, unfortunately a few quotes from the show clearly demonstrate that this report will itself not be open minded. To wit (or to nit wit):

*** snip from Matt Drudge at www.drudgereport.com *****

'60 MINS' PLANNING NEW Y2K SCARE STORY; WATER, ELECTRICITY MAY BE CUT, SAYS MAG

Y2chaos on CBS-TV this weekend! Despite the billions being spent to deal with the Y2K computer bug, many cities in America are uncertain they'll be able to continue to provide basic services like water and electricity on Jan. 1, 2000, reports 60 MINUTES in a new shock story being readied for Sunday.

CBS' Steve Kroft is putting the finishing touches on his controversial report, according to network sources.

Mary Ellen Hanley, a computer systems specialist hired by the District of Columbia, tells wide-eyed Kroft that she believes Washington will continue to function on Jan. 1, 2000, but it must be prepared for what many cities could face.

"We think there will be some disruptions... localized in many cases if the supply chain works," Hanley tells the cameras.

"If power works, if gas works, if Bell Atlantic works... all of those are big ifs."

**** end snip ****************

Another "expert" from the IT crowd tries to pretend to have a handle on y2k in industry, sigh...."A big if??" There are very strong reports from all of the above industries -power, gas, and telephones- indicating that the y2k bugs being found are not as significant as first beleived and will not cause signficant disruptions. I will go with the real experts anyday.

And Bob, I will not pass judgement on your personal preparations for Y2K, I have tried not to do that to anyone here...the only ones I have taken issue with are those who dispense advice to take extreme measures for y2k.

Taking the advice of those not directly working on Y2K problems such as the above quoted "expert"(and who do not have first hand knowledge) in the above cited industries, now THATS foolish..;)

Regards,

-- FactFinder (FactFinder@home.com), May 22, 1999.

-- Anonymous, May 22, 1999

Aw, c'mon, FF - you've said (eloquently, I might add, on more than one occasion) that you have to consider the source - I put ol' Matt in pretty much the same league as World Net Daily. Take anything in either of those places with a laaarrrrggggeeee grain of salt. I'll reserve judgement on the 60 Min piece until I see it.

-- Anonymous, May 22, 1999

lol Rick, you do have a point there...but you gotta admit, Matt has been the first with a LOT of stories ....and he was right on the money with the blue dress....;.

Now if I were a journalist, I would have tried to get a second source, or at least linked to an ubsubstantiated post in a newsgroup...;)

Regards,

-- Anonymous, May 22, 1999

Jon: You ask if we are doing "straightforward tests and along with vendor statements" to verify the readiness of power equipment. My answer is, "Kind of". The tests are relatively straightforward (and do take less than half a day in most cases) because power devices don't use their date functionality (if they even have any) for anything more than date/time stamping.

Consider an example that would be far more serious. Say you had a phase-distance relay that appeared to be over-reaching (it tripped for a disturbance or fault outside of its zone of protection). Since this is a serious problem, you would have to simulate dozens of faults of various types (three phase, two phase, single line to ground, close in faults, far away faults, etc.) on this relay to verify its proper operations. This would appear to take considerable time; however, because we must test new relays as they arrive in this manner, there are programs already set up to perform the tests in a relatively short amount of time (a few hours or less). These programs use very expensive equipment and associated software (in the tens of thousands of dollars), and are the same kinds of equipment we used in performing the y2k tests. We don't claim to have all the answers, but we think we have this thing licked.

Rick Cowles: Transalta is one of the utilities that is digging much deeper into the testing methodology, and has been used in the industry to confirm that our tests are sufficient to verify y2k readiness.

Drew: Yes, in my opinion, at this time, that Y2k will have no serious consequences on power...however, the problems won't be "of no consequence". There will be as a minimum some headaches caused by improper logging, or non-essential programs not working properly. I would like to ask you and a few others for some assistance on another thread...may I?

-- Anonymous, May 22, 1999

Dan, I'm assuming that the complex and expensive testing equipment and associated software you mentioned was also assessed for possible Year 2000 problems. It would be really nice, though, if you could confirm this assumption? Was any remediation in code or hardware necessary for this testing equipment? If so, how did you test the testing systems?

-- Anonymous, May 22, 1999

Hello Bonnie. Yes, the testing equipment has been assessed. Keep in mind that it uses dates like a lot of other equipment, only for recordkeeping purposes, and that date information is NOT passed from the test equipment to the device being tested.

Vendor statements have been reviewed, and they state compliance (or readiness); since this equipment won't shut off your power, at many companies it might get tested after the June 30 deadline for mission critical equipment.

FWIW, Bonnie, I enjoy reading your thoughtful posts.

-- Anonymous, May 22, 1999

Rick,

Why single out World Net Daily? I believe them before I believe ABC, NBC, CBS or CNN - and their liberal Democrat media slant. WND has been more on the ball with y2k reports than any of the above.

Regards, BB

-- Anonymous, May 23, 1999

Well, I'm on vacation- and I really *would* like a vacation, so maybe I'll be brief (famous last words).

FF,

>>Based on the actual industry findings concerning the types and severity of y2k bugs in the electric industry, I am confindent that there will be no disruptions of any significance in the delivery of electrical power caused by y2k bugs during the transition to Jan.1, 2000. This is an easy call. I wish I had as much factual information for other engineering tasks ahead of me:)

Okay, that's what I wanted to know.

>>Now, I have a question for you. What disruptions to society and the infrastructure do you forsee due to the flaming of y2k fears (such as the upcoming 60 Minutes fiction on Y2K). Do you think all of the exaggeration will lead to massive hording, panic, etc, before January 1 even gets here? Will the facts slowly take hold, or will Y2K newsmongers continue to ignore them and milk this puppy to the end???

What I wonder is why you would believe anyone should ever take you seriously, based on statements such as this? I realize that later on you posted some excerpts from the publicity for the story, but sorry- the fact that *you* don't agree with parts of the story hardly makes it a work of "fiction." It's well-known that DC is the worst-prepared major city for Y2K in terms of its own systems; they're basically hoping for the best at this point. If the power & phones work perfectly, DC is still set to run into all kinds of problems. As far as "newsmongers" goes- of all the TV newsmagazines, only 60 Minutes, so far, has done any Y2K stories at all. The nightly news shows (from which the majority of Americans get their news) rarely bother with it. "Newsmongering?" Give me a break. Do you mean the Red Cross, GartnerGroup, Consumer Reports? The "newsmongers" and nutcases like them?

BTW, you promised a thorough report on why the utility industry in such good shape months ago. I still haven't seen it.

Finally, you like to mention "unsubstantiated" posts on newsgroups- yet these frequently consist of simply reposting news stories. When such stories fit your point of view, I haven't seen you have any problem with them.

Also, see another question I have for you, CL, Dan & anyone else at the end of my post. I do appreciate your response.

CL,

Much of what you say agrees with the positive side of the power issue as I've read it as well. My one hesitation is that you believe there's no T&D issue, whereas I know of another EE with *longgggg* experience (include design experience) who disagrees. He is worried about dirty power for a period of 6-8 months. And my memory may fail me, but did NERC mention something about T&D in its most recent report?

Dan,

Thanks for your answer. I'll help you if I can on the other thread, but I *am* on vacation (and I certainly need it). I'll float around on the Net some, but certainly not as much as normal. Just post what you need, or if you want to keep it off the record, and need me to e- mail you (you can't reach me by e-mail where I am), I'll send you an e-mail with an address where I can be reached.

Everyone,

As I've said before, I have no problem believing that the US grid will have zero Y2K difficulties, if sufficient evidence leads me to that conclusion. However, the word of 3 to 5 people (almost all of whom post anonymously, although I know who Dan is) is not enough. When someone like Jon Arnold says "Y2K should be a non-event," (or words to that effect), I certainly sit up & take notice.

At the same time, you must remember that not everyone in the industry shares your optimism. Some are public with their views (Rick, Dick Mills, Roleigh Martin- and that is *not* to call them "doomers"- labels are pathetic); but most pessimists are not. Their concerns are varied- one worries over fuel for the plants in his system, for instance.

That said, no one should think I'm out looking for any sign the power companies will have problems. In fact, last year, when Senator Bennett said he thought there was a 40% chance of Y2K taking the grid down (if Y2K happened at that time), I said on the air I felt there was *no* chance of that happening. Hardly the statement of a perpetual antagonist.

One other point, though, is that, at least in my opinion, Y2K & electric power is really *not* the "lynchpin" issue of Y2K. It only would be *if* we had significant power failures due to Y2K. My primary concern has always been the economic impact of Y2K; the power issue has always been a subset of that (unless, of course, we had found ourselves looking at widespread global power failures, and the subsequent humanitarian issues).

In any event, to wrap up: a second question for FF, CL, Dan & anyone else: what is your response to the ABB report, and to the NRC note Bonnie posted recently about Y2K problems in nuclear plants? Thanks again for your answers.

-- Anonymous, May 23, 1999

Sean,

>I think is it is very 'responsible' of Jon to bring this up on a public forum. I've been wondering how you can test these embedded chips, myself. On a public forum, people (like yourself) can correct this "unsubstantiated" claim, and maybe prevent some panic.

I do not think that discussing testing methods and questioning their thoroughness and validity are irresponsible - I agonize over these issues and will continue to review my methods looking for the unturned stone (remember, we are trying to prove a negative - an impossible feat).

What I do object to is the manner in which the issue is raised - especially given the tendencies of people who are much concerned about Y2K to hold statements of fact to different standards (bad news easily accepted, successes continually questioned). Please note the DEFINITIVE tone of Jon's original post with statements like "you cannot possibly know that you have tested..." and "They are doing what amounts to superficial testing on some devices, and trusting that." These statements are pure speculation by someone who is capable of intelligent evaluation of utility test methods, but is totally lacking the knowledge needed to make the analysis. Jon doesn't know the complexity of the devices, the normal level of product evaluation testing, QA acceptance testing, or maintenance testing. He doesn't have the schematics or understand the application of these devices to the level necessary to make such bold assertions. This is what I have a problem with. Had he stated them as a question, OK with me.

Bob made the following statement addressed to Fact Finder: >Please continue to challenge error where you truly see it, but be careful about making statements that are obviously unsupportable (such as passing judgment on something none of us has seen).

I propose that this statement applies more to Jon than to FF.

Drew - I'd be interested in hearing what power system events will cause the "dirty power" that he is predicting. Why don't you start another thread to address this (after your vacation)?

Jon - I oppose personal attacks on this forum, and would never slander you by equating you to the president of the United States.

-- Anonymous, May 24, 1999

CL,

I believe you're taking my words out of context. Given the previous thread where we discussed a hypothetical testing situation, I believe that what I say is correct.

I am almost always very careful to qualify direct statements that I make. For instance, you quoted "They are doing what amounts to superficial testing on some devices, and trusting that."

What I said in the first part of that sentence is "having done that, ...", which refers to the testing methodology I described in the first paragraph.

Similarly, in the last paragraph, I started it with "I think...".

You're right, I am lacking in the knowledge to make the analysis, which is why I grilled you at length in a previous thread to see exactly how you make the analysis.

I still maintain that you can't adequately test a moderately complex black-box device. So far, people have thrown all kinds of accusations that I don't know what I'm talking about. FF maintains that because lots of utilities are testing the same devices, any problems should surface. You seem to just point out that your testing methods are very complex, after explaining to me how simple they are.

Dan made a reasonable stab at an answer, but he seems to be talking only about T&D stuff, not generation.

Jon

-- Anonymous, May 24, 1999

Jon,

Out of context is a speciality here - that is why caution in wording must be an extreme concern in this imperfect communications medium - hence my indignation.

Lets try again. Power systems will respond the same for the range of faults that are possible. The possible combinations are large, but finite. Phase-phase, three phase, phase-gnd, phase to phase faults at various points on a line. The algorithms to protect for these types are well-proven and were modeled by a battalion of Dilbert's in the 70's and early 80's, just to see if they could be implemented on a computer (even though it was not economically feasable). The ability to test for all fault types is not time consuming, but to test EVERY POSSIBLE fault location is impossible. If you could not see the code for a hand-held calculator, would you test the multiplication function or every possible multiplication problem? Now make the calculator an old mechanical adding machine with an LCD readout - that is the scope of the problem for relays.

These algorithms are converted from E/M technology. They do not use dates, because the coils and magnets they are replicating did not use dates (go look at your electric meter - same concept). In mid-80's, a vendor introduced a uP based relay, with an addition - fault locating and SOE functions to aid in fast restoration. These used software clock derived from crystal oscillator pulses, and ran a new, fault locating algorithm. This is seperate from protection. Huge wall seperating the two. Fault locating runs in background and idle times when primary functions are not being processed.

So, the primary function has years of testing. The algorithms work, and don't use dates - never did. Consider the "black box" to be 2 distinct devices. Our test methods served 2 functions: 1. Test the fault locating black box. This worked, we found many problems, mostly cosmetic and 100% totally unrelated to protection.

2. Test the protection and observe if fault locating problems could somehow interact with the critical fault protection. These faults are simulated with computerized test equipment that is very efficient once configured. There is no user code in a relay to present the utilities an opportunity to "build" a faulty clock/calendar application. Only task is to test the algorithms that are largely unchanged from the E/M relays of the 40's, 50's & 60's and that were tested by computer modelling in the 70's and early 80's. The vendor designers have certified that there is no interaction, we apply faults to verify this. There is no code to have a "loaded gun" except the computer algorithms that are totally date unaware.

Picture an old relay ladder logic controller. No dates ok? Now add a module with SOE and time date stamping. You can't see the ladder logic for the RLL, so you test even though the vendor says no date interaction. Very close analogy.

This rambles, and my kids are calling for me. Sorry, wish we could talk about this, and show you a test but that is quite impossible. (Can you imagine where the utilities would be if the utilities accommodated a request for EVERYONE that wanted to review and validate each test? )

-- Anonymous, May 24, 1999

CL,

I understand that protective relays do not have much in the way of Y2K issues. I have no interest in validating any of your testing. I am interested in testing methods in general, and problems that I see with testing as a concept under certain conditions.

However, my main point of interest lies with generation. So far, nobody has talked very much about how embedded systems in generating plants are tested. The TransAlta document was interesting (especially given that they provide my electricity), but basically it seems to say you need to break up the embedded system into layers and different groups test the layers, and the interfaces. Most of the layers are covered at the vendor level or below, which is not really aprt of black-box testing as far as the end user is concerned (i.e., the end user still has to take the word of the vendor that the system is okay).

Jon

-- Anonymous, May 24, 1999

Jon,

You are fortunate to have TransAlta as your provider. They have been a leader in the Y2K effort and their work doing chip level code analysis has been a big part of validating our test methods. The person I have conversed with from them is top notch and is doing testing that benefits the entire industry (and you).

-- Anonymous, May 24, 1999

For any moderately complex system it is not possible to test all combinations of input and output values. This said, there are two ways that tests can be conducted: Test the system with values that define the boundaries of the system or define THE safe operating state of the system and flag any deviations from this safe state as an error and process the required shutdown or recovery steps as defined by the process. This can still be a large number of test cases to run.

A coworker related that mathmatical proof was required of programs written during the Apollo program. I have not seen this used commercially.

Jim

-- Anonymous, May 27, 1999

CL said The actual reason we are done (and if you have been doing this as long as I have you would not say quickly - only finally)is that there were not very many date aware devices, and these had few problems.

If this is true then the power industry cannot have much experience in these date aware devices. Apparently there are not very many. Would it not be better to have an outside party verify the compliance of these black boxes?

-- Anonymous, May 28, 1999