Y2K problems in substations PLCs?

greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

Okay, here is what I have been told about this. The problem goes like this:

The problem is a firmware problem (NOT an RTC problem). It is related to what could be called "rollover trigger times." The firmware in the PLC tracks time or date intervals for a wide variety of functions- maintenance intervals, testing operational parameters, etc. If the PLC firmware is tracking date intervals, and the date rolls into 2000 (for, let's say, maintenance)- but the firmware thinks its 1900- then the maintenance has "never been done," and the PLC will go into an error mode. The effect will range from nothing to potentially shutting down a substation. I am told this (second-hand) from two different power companies.

Reactions? Thoughts? Is this off the wall, or in the ballpark? I admit I may not have described this well - but I think it's more or less close enough for discussion, I hope.

-- Anonymous, June 04, 1999

Answers

Drew, I don't know the answer to your question, but I do know that there are Y2K problems in some of the Alstom (worldwide transmission and distribution vendor) "PCB Substation computer family of products", listed under their "T&D Protection and Control" business line.

They describe "minor problems", "minor limitations", "date stamping" problems, and "no influence on processes" for some of their products. There are other substation products, however, which only have a "Call us." notation after them. I've noticed this on other product status lists - only minor problems are described on the site, but other components have only a "Call us." (In my mind, the implication is that a company is not going to air any more serious defects except to valid customers who need to know. I can think of no other explanation for stating online those minor problems with simple solutions or which do not affect processes, and then only saying "Call us" about others.)

In the Alstom substation product list, there are 2 "Call us - no upgrade available" notations, 4 "Call us - under investigation", 4 "Call us - upgrade required, or new version in progress", and 5 just plain "Call us" notes. These can be found at:

http://www.tde.alstom.com/cgi-bin/y2k/y2k.cgi

On the Alstom list of Protection and Control systems manufactured outside of France, if I counted right, there are 22 "Call us." notations, 3 "test on site required" notes, and 6 products "under investigation". These are at:

http://www.gecalsthomgpc.co.uk/ProdServ/Y2Klist.htm

Alstom has also put an "Important Note" at the bottom of their switchgear pages:

"The Medium Voltage switchgear supplied by +Electrical Distribution Business; is Year 2000 compliant. Some switchgear, however, might have been equipped with digital protection and control equipment. In case such equipment from ALSTOM origin, please look under +Protection and Control; in this web site. In case such equipment is from a different origin, please contact the ALSTOM entity having supplied the switchgear. In case the switchgear was included in a system supply, please contact the relevant supplier or contractor, whether ALSTOM or not."

-- Anonymous, June 04, 1999


Hello Drew, and welcome back from vacation.

I have not personally seen a "rollover trigger" type design the way you describe it. Programmable Logic Controllers (PLC's) are generally not used in substations; I've only seen them in a few specialized applications in 1% of our substations. Nevertheless, PLC's are being used more in brand new subs.

Again, the issue goes back to how a reasonable person would use a PLC for the type of function you are talking about. It would be a very poor design to use a PLC to trip a circuit breaker because a real (or apparent) maintenance cycle had run out. The proper design would use this function to send an alarm to the operations center as a "level 2" alarm, so the operator could call out a crew to take a look at the breaker the next business day (level 1 alarms require immediate attention). So, if a whole bunch of PLC's did mistakenly send an alarm, it is possible that the operators could get inundated with alarms, and take an improper action.

Usually circuit breakers are maintained not on a regular schedule (every year, e.g.), but based on the number of times they have operated and the magnitude of fault current they had to interrupt. If the calendrical function of the PLC was wacky, I could see how a PLC might mistakenly call for maintenance (or delay sending the alarm), but the actual records for breaker maintenance would catch this (hey, this breaker was maintained two months ago...why is there an alarm already to have it re-maintained?).

And hello, Bonnie. The GEC site you referenced appears to be very good. If they can tell you by product line what action you should take, this completes the assessment phase of y2k very quickly. Those wacky British folk seem to have beat us Yanks to the punch...the GEC stuff I've dealt with had 4 digit dates (even in a product we bought back in 1988!), or didn't use a real time clock in even their microprocessor based products. All of the Alsthom breakers I've seen don't have any date functionality in them, FWIW.

-- Anonymous, June 04, 1999


All,

I sensed a little wierdness going on in this thread, so after some investigation, have selectively weeded some irrelevant responses (including my own). I learn something new every day. If this seems a bit cryptic, it's meant to be.

Besides the wierdness, only two responses came close to having relevance to Drew's question. Those responses have been retained.

Have a nice day. And be careful out there.

-- Anonymous, June 04, 1999


Well, Rick, you're right about cryptic, because I'm sure confused! I got some e-mail earlier tonight asking about this thread, but I only had time to take a quick look before I went out this evening. I hope this is not going to be inappropriate to whatever went on, but I do want to publicly answer a couple of the e-mail questions about Alstom. (If you feel it is better to delete this also, I'll completely understand - even though I don't really understand "the weirdness"?)

It is true that Alstom products do not have a major presence in the United States compared to some other vendors. They do have a larger presence in countries outside the U.S., however, including (in part) Mexico, Argentina, Chile, Denmark, France, Germany, The Netherlands, El Salvador, Paraguay, Brazil and Canada (which as many of us are aware is part of the U.S. electrical grid). They are also doing work for the United Arab Emirates and the Ukraine and also have built power plants in S. Africa, Morocco, and Tunisia.

Their recent and current major projects include:

The largest combined cycle power plant in Europe at Eems in the Netherlands (1775 MW). Two nuclear power plants which will produce a total of almost 4000 MW at Daya Bay and Ling Ao in China. A. Lopez Mateos a 4x350 MW fossil steam power plant in Mexico.

(Readers may recall Menno, one of the industry people in Holland who has posted here, mentioning the new Eems plant.) In fact, a part of the reason I post information that has both outside-the-U.S. and inside applications as well, is for the simple reason that readers here are not only Americans - at least that is what my e-mail indicates. Nor do I find it inappropriate for us to be paying attention to any potential effects of Y2K outside of this country, as well as inside it. This may be debated by some; it's only my personal view.

This year it was also announced that Alstom and ABB are consolidating:

"The Company will fulfill this ambition today through two simultaneous strategic moves - the sale of our heavy duty gas turbine business to General Electric, with whom ALSTOM had a licence agreement, and the creation, on joint and equal terms with ABB, of the world's number one player in the energy sector. ALSTOM will retain 100% ownership of its industrial gas turbine business.

This new company, which will be called ABB ALSTOM POWER, will have a presence in 100 countries, a turnover of approximately 10 billion euros and will employ about 54,000 people."

--Hi to you, Dan! I've found the European vendors such as ABB and Alstom to be more upfront about their products than any U.S. based vendor. (It's far less like pulling teeth to get info.) I also admire their system of listing their products; it must have saved industry people a lot of assessment time, and time is certainly at a premium with Y2K projects. Perhaps it's due to a less litigious environment than we have here in the U.S.? Alstom even defines its terms, for instance: "A substation control system is an example of a hardware-software system." The original questioning on this thread had to do with substations, but I'll put in a bonus just for you. *wink* The Alstom Power Generation family of products is listed as all compliant. It's mainly the Protection and Control product lines where problems are to be found. They do have a Disclaimer saying any info is provided "as is" and "is subject to change", but it's such a tiny one compared to American legal jargon it's rather refreshing!

One of the Y2K challenges Alstom states is of concern is: "Adding to the complexity of the Year 2000 challenge are interfaces between systems. Of particular concern is the synchronization of data when the system at one end of the interface has corrected some problems and the system at the other end has not. Coordination between systems will be critical and filters may have to be used at those interfaces until both systems have corrected any problems."

-- Anonymous, June 05, 1999


Drew:

Last November, I asked a similar question. You can find it under Embedded Controls -- "Are power plants (and other) testing ....." A.J. Edgars posted an interesting, technical response. I don't think Dan was around then, so it would be interesting to get his take on Edgars' response.

Dan,

A few follow up questions to your response. If there is no response to a level 2 alarm within a given period of time, does it automatically get bumped to a level 1? If so, what is a normal time frame? Hours, days ?? I guess it depends on the hardware or application, eh? Also, is there ever an automatic shut down if no timely response is made, maybe at a level 1 alarm?

-- Anonymous, June 09, 1999



Hi Ralph,

It has been mildly surprising to me how few people have followed up on my posts or asked critical questions of them. The only ones that come to mind are Bonnie's comments and FactFinder's challenges.

In summary my position has been, is and remains, that: 1. there are far too many embedded systems for them all to be remediated 2. some small percentage of embedded systems will fail causing monetary damage 3. some smaller percentage of embedded systems will fail causing bodily injury 4. the combined effect of cascading glitches in automated control systems consisting of many embedded systems networked together will cause much greater problems 5. manufacturers around the world will still be shipping products with known Y2K problems well into 2000

I'm here to elaborate on any of these points, but nobody likes to challenge me, except FactFinder... :-)

I think the simple answer is that I am thinking globally and cross-industry whereas this forum is specifically focused on Electric Utilities. It sounds like everything is hunky-dory in the Electric Utilities now.

--aj

-- Anonymous, June 09, 1999


Drew,

Here is a more specific answer to your original question.

Yes, you are in the ball park. A line drive actually. :-)

The problem with PLCs and other industrial automation equipment that include time/date functionality is typically with the firmware, kernel /executive and/or application code.

Here is a real honest to goodness example of a popular PLC with a real Y2K bug: Siemens S5/115 PLC: The Siemens executive (Operating System) on these PLC'S simulates an RTC function (time and date) as there is no on-board RTC on this device. When this executive runs through the turn of the century it returns ?? (2 question marks) when reading the date back as it does not recognise a year with a leading 0 (zero). Any device, SCADA System, equipment or MES system reading this date will be confused resulting in reaction which is in certain circumstances a fail safe routine.

It was exactly this type of failure that had caused part of a Texaco oil refinary to shutdown. A data logger didn't recognize the '??' data and failed-safe causing a shutdown.

-- Anonymous, June 17, 1999


I hadn't checked this out in a while. There are quite a few new entries. For you curious types, take a look and read the PLC and SCADA cases. Let your imagination dream up a few scenarios.

http://www.iee.org.uk/2000risk/Casebook/eg_index.htm

I wonder what type of plant needs a robot to change air filters?

Who would trust their alarm monitoring to Windows'95 and VisualBasic (yikes!) ?

Or this one: http://www.iee.org.uk/2000risk/Casebook/eg-44.htm A manufacture shuts down over the Jan 1st holiday as a precaution and on Monday can't start his assembly lines.

Let's hope this plant isn't manufacturing candles, batteries, or kraft dinner... ;-)

-- Anonymous, June 18, 1999


Moderation questions? read the FAQ