I had the opportunity to once again participate in one of Ed Yardeni's "Y2k Action Day" updates, and it is now online (you need the RealPlayer plugin) at http://www.y2kactionday.com/

The lineup of speakers reads like a "who's who" of Y2k, so I think it's worth your while to take a bit of time and listen to the interviews if you get the chance.

Rick

-- Anonymous, June 14, 1999

Answers

Rick,

Thanks for the link. I was a little disappointed that Ed Yardini did'nt ask the participants really hard questions. Mr. Hall, the embedded chip expert testified before congess last fall that a company absolutely could not do "type testing", to do so could result in disaster! Now not only was the question of type testing not brought up, but by it's absense appears to support factfinder, the engineer, Dan the Power Man and CL position that it is not an issue. The only embedded system issue that Hall could point to was the Hydro plant in Brazil whose control panel went haywire. I am not discounting this issue but it is only one example. The point that I'm trying to make is this. If embedded sytems were a serious industry issue than Hall should have been able to point to several examples here in the U.S.

Can anyone describe a situation of how a power plant may trip or how a utility may fail to provide power due to a Y2K problem?

-- Anonymous, June 14, 1999

Bill .... A Y2K problem that WILL stop the power plants is OIL; 1) Pumping in Saudia Arabia, Venezuela and Nigeria. 2) Non compliant chips and computers in PORT facilities of above. 3) Inability of US refineries the process ('crack') same . 4) Therefore, insufficient oil for utilities; insufficient desiel for the railroads to haul the coal to other refineries AND insufficient desiel AND GASOLINE for the suburban high paid engineers, plant managers, etc. to GET TO WORK !! Can you fix that, ' Engineer '? Eagle .... circling .

-- Anonymous, June 14, 1999

Hal,

Are you a embedded sytems expert? Believe me, I'm not trying to be a wise guy by asking such a question. The point I'm trying to make is this. I find it extremely difficult to find the truth when it comes to Y2K issues. To say that I'm disappointed in David Hall's audio statements would be an understatement. Remember this was the same guy who testified before congress last fall that the sky is falling. Remember the only concrete example he could give about any real life example was the hydro plant in Brazil. This is public infomation and is hardly new infomation.

I do believe in giving credit when it is due however. Factfinder, Cl, Dan, the power man, the engineer, and other utility insiders have all shed light on the embedded systems issue. I would like to thank them for their time and effort.

Hal, will we have supply chain problems at our electric utilities? I don't know the answer to this question, do you? If you do, please describe the scenario in detail and list the error sequences by system and their impact on product delivery. Again, I'm not trying to be a wise ass but am just trying to point out that no one seems to have this infomation.

-- Anonymous, June 15, 1999

Bill W. ... Thanks for keeping the " afterburner " off as you addressed my points. I can only answer that about three months ago, I read a long thread by a IT person who visited with a Japanese IT specialist in Japan to discuss the Japs problems. He was astonished to learn by patient listening , that this same person had supervised the whole Saudia Arabian oil industry computerization a number of years ago , and this same Japanese stated (obstensively) " It is too complicated AND too late to fix now." The other two countries have been in the news repeatedly of late , stating in no uncertain terms that THEY WILL NOT GET THEIR PORT FACILITIES AND OIL FIELD COMPUTERS FIXED IN TIME !!! As we import 55% of our oil now, basicly from these three countries, can you come to any OTHER conclusiion ? I lived the 70's gas crunch, AND, we only lost about 5% of the oil coming in for refining . Stop thinking about what you WANT to happen; START adding up the facts and you'll get a different picture. BTW, why would our government be buying extra oil for our reserves if there was NO PROBLEM ??? Eagle P.S. I have an "outside" source for KNOWING it will be a NINE , NOT a bump in the road.

-- Anonymous, June 15, 1999

Hal,

I appreciate and share your Y2K concern. In the above post you raise some valid points. Especially about why would the Government add oil to the reserves if there were no problem. I believe one reason is because at the time they passed the legislation everyone thought embedded chips would be a huge problem.

Hal, are you basing your dire predictions on the embedded systems problems? It appears to me that you must be doing this. A few months ago I was reading the same infomation. As a matter of fact, the turning point for me was the testimony of David Hall, an embedded chip expert. After I read his testimony before congress, I started preparing immediately. I am not an embedded chip expert. I must depend on others that claim to be. Before congress, Hall painted a very grim picture of embedded chips and Y2k. In his audio he has toned down his statements dramitically.

You tell me to stop thinking about what I want to happen and to start adding up the facts. Believe me, Hal, that is exactly what I'm trying to do. My analysis of the facts tell me that Y2K has two "Glitch Tracks"; paperwork and embedded systems. Of the two, embedded systems clearly hold the potential for dire consequences. David Hall put forth the idea that "type testing" would not work. I believed him. Factfinder, cl, and other utility insiders held a different view. Now David Hall is saying embedded system problems are not such a big deal. In other words his view is now coinciding with the majority view.

-- Anonymous, June 15, 1999

I don't know whether this answers your question or not, but my power company is compliant. Since they are an integrated company - they own everything between the dam and the wall socket - their experiences might be helpful.

They identified more than 8,000 systems that could be at risk from among hundreds of thousands of systems. They identified 3200 of these as mission critical. They tested these systems - and yes, they type tested.

They report a 7% failure rate in mission critical embedded systems between the dam and the wall socket.

They took five years to identify and replace the systems.

http://www.bchydro.com/

Tom

-- Anonymous, June 15, 1999

I wouldn't be too hard on Mr.Hall. I believe he has come to the same conslusion as a lot of the other y2k intelligentsia. If you run around screaming about how awful it will be, pretty soon no one is listening. It's only the TV evangilist that can scream and shout and have his message appreciated (by some).

You might want to research a (rather large) story that was printed in the Houston Chronicle about 4 months ago regarding y2k and the shipping ports/terminals/cacking plants there. And REALLY read it...both stated and implied points. I have been there during a drill and quite frankly, the DRILL for a breakdown in that place scared the bejabbers out of me (and it wasn't even about y2k).

You may well be like the rest of us. Looking for concrete information when there is none to be had.

James Moore (email address is real)

-- Anonymous, June 15, 1999

Tom:

I be interested in knowing whether your power company commenced with their embedded systems projects at the same time they took on thier mainframe projects five years ago. Most utilities I'm aware of were essentially blind-sided by the thought of embedded systems benig a problem and didn't even start dealing with them until 1997. Any knowledge of that?

-- Anonymous, June 16, 1999

Tom, thank you for posting the status of BCHydro. It was remiss of me not to have re-checked BCHydro's web site in a while, as the information is posted there also. The web site also defines mission critical, except the words used there are "high impact".

"BC Hydro had successfully remediated 223 "high impact" devices across the province. High impact devices are those that could have impacted safety, the environment, or Hydros ability to generate and deliver electricity to its customers."

In the "Embedded Systems" section we read, "Our inventory of embedded systems identified over 8,000 embedded systems. Of those 8,000 embedded systems, approximately 3,300 were identified as critical embedded systems, in that their failure could directly affect our ability to deliver electricity to our customers or result in public safety or environmental concerns." (223 is 7% of the 3,300)

There is also a description of lower priority devices: "Lower priority devices, for example, those that would function normally but print an incorrectly-dated status report, are now being addressed."

I would say this answers some often-asked questions on this forum, the first being that a utility has stated that there are high impact (mission critical) devices which did need remediation, or the safety, environment or generation/distribution of electricity would have been compromised. (And these were differentiated from incorrect date-printing problems.)

This information also might answer why there have been reports of concern over the stability of electric grids in countries outside of the United States. The utilites in many countries have not been addressing the Y2K issue nearly as long as BCHydro has. In fact, there are _very_ few U.S. utilities which began a Y2K project five years ago either.

I suppose since BCHydro is a Canadian utility, the arguments may still continue about there being no critical embedded problems in U.S. utilities, but BCHydro is nevertheless connected to the U.S. grid. I welcome their announcement and send them congratulations on having finished remediating all their "high impact" systems! Since Tom stated compliancy, perhaps the web site has not yet been updated to indicate even the date-stamping problems have now been dealt with -- in which case, even more congrats are due!

-- Anonymous, June 16, 1999

One of the few times I have been to this forum and Tom B. is posting! I hope things are going well for you Tom!

Below is testimony from BC Hydro to Industry Canada.

INDYEV124-e

Mr. Bob Steele (Chief Information Officer, B.C. Hydro):
Good afternoon. Overall, Hydro has been working on the year 2000
problem since about 1994-95. We're in good shape, and we do not
expect any power outages due to the year 2000 issues. We'll be ready
with all our critical functions by the May 31 of this year.

We divided our project into four major areas: the inventory, testing and
remediation of our business systems; the inventory, testing and
remediation of our operational control devices; the business partner
reviews; and the preparation of contingency plans. My comments today
are restricted to the operational control devices and the contingency
plans that have been prepared that relate to the operation of the electric
system, as well as our suppliers and grid partners that impact the
operation of the electric system.

Hydro has undertaken two inventories of the operational control
devices and our generation, transmission, distribution and
communication systems. These devices include the relays, the circuit
breakers, metering devices, the sensors, and so on, that monitor and
control the electric system. The additional inventory was taken to
identify devices that were likely to require testing or could likely have a
Y2K problem, so that we could commence our remediation as quickly
as possible.

 1840

The second inventory was a much more detailed inventory. It involved
identifying, labelling, and adding to the inventory database every piece
of equipment in our system that could possibly have a year 2000
problem.

Inventories were taken using a combination of both local and central
specialized resources. They were signed off by the responsible local
manager after the inventory was taken.

The inventory and the process for collecting inventory was audited by
an external firm, an international engineering firm. The audit included
sample checks from a number of our generating and substation sites.
And we have a change control process in place to ensure that any new
equipment, replacement equipment, etc., is updated in the inventory
and tested if necessary. The audit reviews were generally positive, and
Hydro's taken steps to address any issues that were raised by the
audits.

The inventory as of this month contains just over 8,000 devices that
could potentially have a Y2K consequence. These devices were then
assessed as having a high, medium, or low impact should they fail.
Potential high-impact devices were classified as such if they could
threaten public safety, employee safety, the environment, or our ability
to generate and deliver electricity to our customers. And of the 8,000
and some devices, 3,307 were categorized as potentially having a high
impact.

Those 3,307 devices were evaluated again to determine if the Y2K
dates affected their operation. In that evaluation and testing, 221 of the
devices were determined to be affected by the Y2K problem. However,
none of them, if they failed, would fail in such a way as to cause an
interruption to electricity. So even though they potentially had a high
impact, the nature of the failure during the testing would be such that
they would be benign to causing an interruption of electrical
distribution.

Each evaluation or test was supported in document and signed off by a
engineer of record. Each document was also reviewed by an additional
level of senior technical staff with the responsibility for the Y2K efforts
in their specialized area.

Remediation work and retesting has been completed for 201 of the 221
devices that were impacted, and the balance will be completed by the
end of this month. To date the above processes have been reviewed by
external engineers on three occasions and by internal auditors on two
occasions. Plans are in place for two additional external reviews, as
well as two more internal reviews by our internal audit department.
These reviews are in addition to the Y2K teams' biweekly quarterly
review board that reviews the quality of the work that's being done.

B.C. Hydro has also been very active with a number of the utility
organizationsNERC, which is the North American Electricity
Reliability Council, EPRI, which is the Electric Power Research
Institute, and CEA, which is the Canadian Electrical Associationas
well as with other utilities to compare the test results and to share
information on equipment that could cause Y2K problems.

Additional systems integration and tests will be carried out over the
next few months. Plans include testing protection equipment on both
ends of a major transmission line by advancing the clocks and at the
same time allowing the clocks to roll through the critical dates. That test
was completed last month. We intend to advance the clocks on all 13
major generating facilities and allow the clocks to roll through the
critical dates. That will be completed by the end of July.

We're practising a black start at one of our most critical generating
facilities. We will be doing that in the June timeframe.

We'll be advancing the clocks at our supervisory control and data
acquisition systems and allowing the clocks to roll through the critical
dates. We'll be doing that for the supervisory control systems in May
and June.

We'll be taking all of the equipment in a geographic areaand that
would include generation, substation, and supervisory controland
advancing the clocks simultaneously and allowing the clocks to roll
over through the critical dates.

The integrated contingency plans we've developed are at three levels at
our system control centre. We have four area control centres and 340
generation substations and communication stations, and we have
completed the contingency plans for each of those stations.

 1845

Contingency staffing for low-risk datesand there are a number of
those, including last December 31 to January 1, April 9, August 22 and
23 for us, and September 9, as well as the February 28 to 29 period in
the year 2000are all in place and we've been operating with
additional staff over those periods of time. And we have plans in place
to have the contingency staffing ready within the next month or two for
the extra staffing we'll have on in the December 31 to January 1
timeframe.

We have a number of exercises and drills to practise, evaluate, and
improve our contingency plans. We took part in the April 9 NERC
drill, which is the North American Electricity Reliability Council. That
was to test our back-up communications systems, and was undertaken
successfully. We have two additional drills that are planned to exercise
our contingency plans. These drills will include simulating various
Y2K scenarios and testing the responses of both the regular and the
Y2K contingency staff and their reaction to these various scenarios.

We're participating on May 19 with other utilities within B.C. for one
of the exercises and in September we're participating with the North
American Electricity Reliability Council and all the other electric utilities
for a second drill.

At the same time, Hydro is working with our grid partners, both
directly and through the Western Systems Coordinating Council, and
with our major customers, the independent power producers, and our
suppliers to ensure all risks are minimized with respect to Y2K issues.

In summary, B.C. Hydro recognizes the criticality of ensuring a
continued supply of electricity. We've been working on the Y2K issue
for many years, and while the nature of our service never allows us to
provide a 100% guarantee of service, we are confident that the risks of
disruption to the normal operation of the electric system from year
2000 issues is very low.

Additionally, should we suffer unforeseen outages, we will have
additional staff and equipment in place to mitigate any impact and
restore services as promptly as possible.

Thank you.

The Chair: Thank you very much, Mr. Steele.

-- Anonymous, June 16, 1999

Brian, thank you for posting the testimony transcript from May 12. It added information that was not included on BCHydro's website. In the testimony it's stated that none of the designated "high impact" systems found to have Y2K problems would "fail in such a way as to cause an interruption to electricity". So this does correspond with some of the other assessments we've read about.

This is more good news, but unfortunately, while the claim of "the nature of the failure during the testing would be such that they would be benign to causing an interruption of electrical distribution", the same claim was not made about whether the Y2K problems found in these systems would "be benign to" the other two criteria of high impact systems - that of "public safety and environmental concerns". We do know that those systems have now been fixed (replaced, patched, upgraded, etc.) but are left not knowing exactly why they needed fixing.

However, then I came across a question which addressed part of this issue and was asked of Mr.Steele after all the individual testimonies had been given. Part of Mr. Steele's answer was, "When I classified the devices as having a high impact, if a device of that nature, protected relay, for example, was year 2000 susceptible and we knew there was a problem, it could fail in a number of ways. It could fail by opening a circuit breaker, which would impact the flow of electricity. It could also fail by just stopping to work, not open a circuit breaker but with the protection it provides in case of overfrequency or something along those lines, the network wouldn't be protected. But it wouldn't immediately impact the flow of electricity. So I hope that explains a little bit more."

I thought it explained quite a bit! It gives an example of why a detected problem could be classed as not impacting the flow of electricity but still needed to be fixed. In the example given, it would be because the network would still not be protected even if there was no immediate impact to electrical flow -- the difference between a device failing by cutting off flow and just plain failing to work at all. This filled in a lot of blanks for me as far as understanding why some devices need to be fixed even though they don't immediately shut down the power.

The testimony transcript also offered something I found curious. Mr. Steele, the BCHydro Chief Information Officer, stated at the beginning there were four major areas of the company's Y2K project -- Business Systems, Operational Control Devices, Business Partner Reviews and Contingency Plans. Yet he then went on to say, "My comments today are restricted to the operational control devices and the contingency plans.." Was there not enough time to say anything about the Business Systems and Business Partner Review areas? These areas were addressed in other companys' testimony and follow-up questions, so any restriction was not on the part of the committee hearing the testimonies. It seemed an odd omission to me, but it may not mean anything.

-- Anonymous, June 16, 1999

Bonnie,

You never fail to pick up the weak spot in any given text. Yes, relays can fail in a number of ways, as was explained by the BC Hydro rep. Unintended trip output is often the first considered because it so immediately impacts the system. This failure mode would interrupt service to customers.

The second mode is not as sexy, but still quite devastating. If a protective relay is not performing properly as a "silent sentinel" of the power system, a fault will not be removed with the normal speed and precision. This will cause an extended area of outage, and can threaten system stability - minimizing or eradicating the positive benefits of an interconnected grid. This can lead to a cascading outage - similar to SF, but with different catalysts.

To operate properly, relays must be fast and accurate, always tripping for faults in its zone of protection. Relays that meet this objective are Dependable. They also must not falsely trip when there is no fault, or if the fault is outside the zone of protection of the relay - they must be Secure.

Reliability = Security + Dependability.

It is possible, from the wording of the statement, that some relay's dependability was destroyed by Y2K issues, but my experience tells me that this is not a concern. I will restate all my test result claims in the more precise language normally reserved for protection engineers - I did not find a mission critical device that failed in a manner that would compromise the dependable and secure operation of the device (remember - T&D with some generation protective relays). We test for all failure modes, and no show stoppers were reported at EPRI by any utilities. Perhaps some other utility folks can enhance their "no lights out" statements here to include failure to operated or reduced dependability.

I won't speak for others, but I used the lights out language simply because it was most easily related to by people who want to know if they'll need a generator. I was once criticized here for being too technical - arrogance, condescending and all that. This seemed like a reasonable compromise given the subtle nuance of reliability issues in protective relaying. You obviously are one that no one should fear talking technical with. Congrats.

Believe me, a failure to operate would be considered a SHOWSTOPPER. And we have had none in T&D.

-- Anonymous, June 16, 1999

A question to Tom from bc hydro and to Rick

Other utilities are probably not as far along as Tom's bc hydo.

Do all the Y2K electric utility people working on embedded modules have some Internet mechanism for immediately notifying all their counterparts at other utilities about Y2K problems that they have discovered.

Is there a (publically accessible) bulletin board where heads up utility guys like Tom can electronically post immediate notices for his counterparts at all other utilities like : Item X from Company Y is Y2K non-compliant and must be replaced or else Z will occur ?

Or have the lawyers in utility companies been afraid of being sued by some vendor for disparaging remarks about Y2K problem products ?

I am concerned that each utility's Y2K testing team must discover everything by themselves. That could be deadly for all of us.

I have searched NERC and can't find anything.

How long did it take to get replacement embedded modules for the items that bchydro discovered needed to be replaced ? One week ? N months. Did they have to be customized for bc hydro ?

What typical lead times (order to delivery) might one expect in the 3rd or 4th quarter.

Tom, please stay involved in this thread. We really need guys like you and Rick.

Thanks alot Ron Sander Electronics Engineer and strong GI

-- Anonymous, June 17, 1999

Ron - Tom B. doesn't work for BC Hydro - just an "interested customer" as far as I know. ;-) I'll answer as best I can; perhaps some of the industry folks can chime in as well:

Other utilities are probably not as far along as Tom's bc hydo.

This would be correct. To what extent? Unknown.

Do all the Y2K electric utility people working on embedded modules have some Internet mechanism for immediately notifying all their counterparts at other utilities about Y2K problems that they have discovered.

The only such mechanism that I'm aware of is the industry data base maintained by the Electric Power Research Institute, which is accessible only to member companies (roughly 120, if I recall correctly). To the best of my knowledge, there is no "mail list" or listserver that industry folks can automatically broadcast such operating experience to their industry counterparts.

Is there a (publically accessible) bulletin board where heads up utility guys like Tom can electronically post immediate notices for his counterparts at all other utilities like : Item X from Company Y is Y2K non-compliant and must be replaced or else Z will occur ?

See previous.

Or have the lawyers in utility companies been afraid of being sued by some vendor for disparaging remarks about Y2K problem products ?

While this may be an issue, I don't think it's a widespread concern in the corporate counsel offices.

How long did it take to get replacement embedded modules for the items that bchydro discovered needed to be replaced ? One week ? N months. Did they have to be customized for bc hydro ?

You might want to go to BC Hydro's website and ask that question - I think there's a direct link where you can send email to their Y2k coordinator.

What typical lead times (order to delivery) might one expect in the 3rd or 4th quarter.

It's gonna vary, depending on the complexity of the equipment, and whether or not "Y2k compliant" replacement parts / software upgrades are available. The 3rd or 4th quarter of this year is going to be pretty late to get into most vendor's order and production ques.

-- Anonymous, June 17, 1999

Rick,

EPRI has such an information clearing-house. No broadcast, but a user can suscribe to a device and be alerted by e-mail whenever anything is posted regarding that device.

As for the lawyers, we have been required to sign non-disclosure agreement with at least one vendor BEFORE they would release their detailed test procedure/results.

-- Anonymous, June 17, 1999