We've read quite a bit about the contingency plans to go to manual operations if there are failures in telecoms, or some of the utility monitoring and control systems, such as SCADA systems and others. The NERC April Drill and the upcoming September Drill focus on potential loss of telecommunications and the manual operations which would be needed.

In the Powerpoint presentation on the April 9 Drill Review, to be found at

ftp://ftp.nerc.com/pub/sys/all_updl/docs/y2k/

the background for the drill shows the "Coordination Matrix" for the U.S. at 3000+ electric utilities and 1400+ telecom providers. Actual participants in the drill are listed as 200 power systems, 500 facilities, and 2500 personnel involved. In this presentation there is something under "Other Lessons Learned" which didn't make it into the press releases I saw about the drill. This is:

"Operating by manual telemetry is arduous and time consuming - prioritize further".

In the MAPP Region Powerpoint review of the Lessons Learned from the April 9 Drill, there is: "--Alternate communications systems require more organization, time, and staff than expected --Not all the available data is critical, Not all the critical data is available."

Going to the International Atomic Energy Agency's Summaries of the July 1999 Y2K Workshop they recently hosted for nuclear power plant personnel from countries around the world, we can read this:

"Plants cannot continue to operate without their PPCs. [Power Plant Controls] They must shutdown within a specified numbers of hours if certain functions of the PPC are not available (.5 - 8 hours depending on function and reactor design). In general, the PPCs are monitoring tools, not control mechanism. However, there are other computer systems that have control functions and which could have Y2K vulnerabilities."

The next paragraph after the above is particularly in contrast to what we have been hearing about manual operations in the U.S. ---

"In general, plants noted that operations without network and computer systems would be unlikely. Even if not required by operating requirements, some systems have become part of the operators expectation and environment (e.g., safety parameter display system (SPDS)" ( http://www.iaea.org/ns/nusafe/y2000/wwersess.html )

This succinct summary of overseas power plant people's opinions at the Workshop that "some systems have become part of the operator's expectation and environment" is in line with concerns expressed previously on this forum. Ironically, this opinion is from personnel who, on average, have less computer control and monitoring systems in their plants than do those in the U.S., and who have had less years since implementation of those systems in which to grow into a dependency on them -- yet they admit that that is the case.

Hence I was not surprised to also find in the U.S. April 9 post-drill assessments of what still needs to be done these words, in orginal caps: "TRAIN, TRAIN, TRAIN."

Think about this. How many people, not only in the electric industry, but in all other segments of the infrastructure are in the position where "some systems have become part of [their] expectation and environment"?

-- Anonymous, August 03, 1999

Answers

Bonnie,

The reservations expressed in your post are similar to those that we had before we took our SCADA down in June.

Prior to SCADA, (when our hydro stations were fully manned and only controllable manually) we had a minimum staffing requirement of three people on shift around the clock. With the advent of SCADA we reduced that manning level to one person around the clock in the control center, with one additional support person during normal working hours (8:00 to 5:00). We envisaged that during our 3 weeks on manual control that we would require something similar to our previous staffing requirements, but we just didn't have that number of staff available. However when we looked closer at the issue we found that we could still get by with only one operator at the remote power station, and our controller on duty at our main site had to relearn some of his basic operating skills and do double duty as both controller and operator. Thus only one additional staff member was required.

Fortunately our control center, and the (old manual) control room for our larger hydro station are in adjoining rooms so that part was not too hard to manage.

Another issue that we thought would be a major problem was related to the comment: "In general, plants noted that operations without network and computer systems would be unlikely. Even if not required by operating requirements, some systems have become part of the operators expectation and environment (e.g., safety parameter display system (SPDS)"

Our older station which used to be all manual was recently upgraded to allow for remote control. Part of the upgrade involved replacing the old mechanical governors with modern electronic units (both from Woodward), but the abililty to manually adjust things like gate limits, speedo-motor settings, goverspeed droop and dashpot settings etc. were all removed. Also removed was the breaker control and the manual synchroscope. Fortunately we were able to find a way around each of these problems by allowing for a seperate local input to the governor PLC and having a simple local/remote switch on the auto-synch relay.

As far as no longer having access to information that we had become used to having; Well it was just a matter of learning to live without it.

A hydro station is a fairly simple type of plant compared to our fossil fired plant, but even there they have managed to handle local control with only three people per unit, and their normal compliment is four people per unit anyway. However once again the thermal station controller has had to learn to manage with less timely information than he is used to.

One point to remember about using staff to control a station manually is that it is more labour intensive, and there is a greater chance of making errors. So while on manual control only essential operating is carried out. ie starting and stopping plant, and/or emergency operating only. There is no ability for carrying out routine or scheduled maintenance.

Malcolm

-- Anonymous, August 03, 1999

Malcolm,

As a layman, this is an encouraging post. There is a confounded varaible in all this though and that is there will is an increased potential for unusual load patterns in 2000. This assumption rests on 1) potential for Y2K related failures in large electrical customers, 2) Planned shutdown of large consumers in accordance with contingency plans and neccessity for comunicating this to producers. 3) Currently unpredictable social situations. In an evironment of increased anomalies 2o to Y2K, unusual load patterns both increase the likelihood of required reversion to manual operations and increase the difficulty of that task. I won't pretend to know whether this is non-issue or not however for argument's sake let's assume manual operation is neccessary. Can this be accomplished if there are bizzare electrical demand loads? Just as interruptible customers are shed to preserve grid stability, I know I for one would rather that all residential electric power be cut off for 48 hours around the millenium if this would enhance the stability of the grid. Indeed, if I thought the great spin monster could change its spots I would think it make make sense to tell people now that their fears will come true and all residential electricity will be shut off around the millenium. Would enhance preparation I suspect, would increase generator reserves, wouldn't make a whit of sense from a technical standpoint if generation reserves and bizarre load patterns are not an issue.

-- Anonymous, August 03, 1999

To clarify I am suggesting shedding residential loads temporarily if this enhances the chances of preserving power to 1) vital social services (Police, fire, water, hospitals etc) 2) Those industries which would would suffer safety or devastating equipment failure from power interruption (oil refining, chemical plants etc.)

-- Anonymous, August 03, 1999

>I know I for one would rather that all residential electric power be cut off for 48 hours around the >millenium if this would enhance the stability of the grid.

I would not be surprised if "for the sake of our childr... Grid" they do implement a mandatory power outage across the board. This would naturally fall within the 2 to 3 days inconvenience window that they've been harping about all along.

-- Anonymous, August 04, 1999

>I know I for one would rather that all residential electric power be cut off for 48 hours around the >millenium if this would enhance the stability of the grid.

I would not be surprised if "for the sake of our childr... Grid" they do implement a mandatory power outage across the board. This would naturally fall within the 2 to 3 days inconvenience window that they've been harping about all along.

-- Anonymous, August 04, 1999

To clarify I am suggesting shedding residential loads temporarily if this enhances the chances of preserving power to 1) vital social services (Police, fire, water, hospitals etc) 2) Those industries which would would suffer safety or devastating equipment failure from power interruption (oil refining, chemical plants etc.)

Is that really a good idea? Off-hand, I can't think of a better way to induce an extreme reaction from the public than to deliberately "shed" residential "loads".

But, hey, this is all moot. The utilities are done. They never really had to do anything in the first place, but they're done now. So, we shutting down this "board"? Why not?

-- Anonymous, August 04, 1999

Lane,

I don't want to take a stance here of breaking a few eggs to make an omelete. If we take as given that there are difficulties with manual operations, unusual demand patterns might furher exacerbate these difficulties. As reported recently,

"The regulator-general, Dr John Tamblyn, has urged householders not to switch their electric appliances on and off on New Year's Eve because the electricity supply depends on consistent demand."

http://www.theage.com.au/daily/990804/news/news15.html

Within the context of these possible problems is there a scenario which might minimize suffering?

I haven't the expertise to weigh the complex variables involved and fortunately I don't have the difficult decisions ahead that await NERC, NRC and others.

One objection to this scenario is that the public would behave irresponsibly to such an announcement. Sadly, this appears a valid objection, this speaks upsetingly though to the spirit of American resiliance that you wrote about in a recent column. Possibly the most irresponsible activity the populace is making at present is a generalized lack of any preparation. Such a scenario might also minimize the possibility for societal upheaval from large scale gatherings along the lines of times square as well as numerous families taking holidays far from their homes. Finally, it might (might) allow discussion of a frightening topic to enter the public disourse in a more thoughful manner.

I realize that utlilites already prioritize their load sheding to minimize disruption. Thus residential loads will be shed first while power continues to the water treatment facility and to the nuclear power plant. Is there from a technical standpoint any advantage to proactively simplifying the grid? Backing further off the "canyon rim" as Dick Mills puts it, ( and informing the populace of this action early on ) in order to lessen the likelihood of a, how is it phrased, "station black", situation at a nuclear power plant, or similarly a water treatment facility without power. It would take only one large scale mishap at a toxic chemical plant secondary to unexpected power surges or outages to question whether such actions might save lives. It may be true, but how upsetting to think that as citizens we couldn't stomach the idea of a day or two without power even if it would be to our advantage. Then again if everything's fixed this is thankfully just crazy talk and we should just party on.

-- Anonymous, August 04, 1999