9/9/1999 came and went. No major failures happened, at least that were reported in the public domain.

Are you surprised? I'm not.

The NERC drill is being characterized as a smashing success.

Are you surprised? I'm not.

I was there.

To set the stage, a little background is in order. On behalf of a client, I was invited by a major bulk power transmission provider to observe the latest NERC sponsored Y2k drill from their main control center during the evening and early morning of 9/8 and 9/9/1999. I was given free rein of the entire facility, and wasn't shadowed by a public affairs person, beyond an initial facility tour. I was allowed to speak with anyone I wished, and overhear all communications during the course of the drill. And I was also pleased to see that one staple of all electric industry drills in which Ive participated in the past is still alive and kicking  free food for the participants and observers.

From the outset, it was quite apparent that a lot of effort had been expended in preparing for this drill. The drill procedure listed six objectives:

1. Demonstrate the ability to effectively deploy resources related to the staffing allocations for the 12/31/1999 transition
2. Demonstrate the ability to implement operating and administrative procedures, as necessary to maintain the reliability of the Bulk Electric System
3. Demonstrate the ability to effectively use backup (alternate) communications systems.
4. Demonstrate the ability to effectively deploy elements of the Y2k Contingency Response Plans
5. Demonstrate the ability to observe and document drill activities and provide an objective analysis of the overall response by the participants.
6. Train drill participants in Y2k specific activities

In reading the above objectives, it should be clear to the most casual observer that the drill was not intended to actually exercise any equipment, or perform any actual rollovers of time / date sensitive components, either embedded controls or computer system based. This is not a criticism of the drill scope, but rather, a necessary and important clarification of drill scope that has been totally absent from the impressions being provided in the media over the past two days. Alternate communication systems were tested, and some personnel were dispatched to the field from remote locations, but no equipment was actually tested or exercised during this process.

The drill began with a coordinators teleconference at 8:30PM to confirm readiness for drill activities and status of remote facilities staffing, and the drill ended with a critique around 4AM the following day. In between, there was a lot of satellite telephone communications going on from the region, and NERC performed a roll call of all participating organizations as midnight approached. Drill activities were suspended between 11:30PM and 12:30AM so that all participants were in a standby mode for the 9/9/1999 rollover.

And honestly, the whole thing was about as exciting as watching paint dry.

The region that I was observing from did not have any simulated problems thrown at them by NERC, so it was impossible to gauge response to even a single potential problem, much less multiple problems arriving at basically the same time. Communications appeared to be adequate, but satellite based telecommunications systems are somewhat flaky on a good day, and this day was no exception. Using a sat telephone is just not the same as using your cell phone. Hopefully, the regular phone lines will continue to work during the actual rollover

The personnel in the facility were very confident of their ability to work through the drill. I dont know if Id characterize the attitude as cocky, but it was apparent from the outset that they didnt consider this to be a very big deal. Most of the system operators and supervisors continued with their regular responsibilities, sandwiched around an occasional drill related phone call. To sum up in a very short sentence, they were confident not only in their abilities, but also in the outcome of the drill prior to the drill.

Heck, they were thinking about the press releases even before the drill started. The entire Public Affairs group was manned, the fax machines were preprogrammed with media contact numbers, and the Business Wire internet connection was warmed up and ready to go.

This is, in my mind, a good thing and a bad thing. It has been noted in this forum before that hubris kills  and if the drill really accomplished anything, it was to reinforce the emerging attitude that Y2k is not going to be a big deal for the electric industry. I continue to be concerned that this will lead to significant complacency on the part of the electric industry in general, and individual participants in particular, because I dont believe that this attitude is confined to simply the facility or region that I observed.

I am concerned that the drill was not a full-blown exercise. If the purpose of the drill was to challenge the operating status quo (which it wasnt), drill scenarios would introduce multiple simultaneous failures to 1) gauge how field personnel would respond and 2) test the command and control coordination of recovery efforts. For example, when training pilots in simulators, and testing nuclear plant emergency response, multiple concurrent failures are always thrown at the participants. If such a thing occurred during this drill, it wasnt apparent to me as an observer.

So, were the regional objectives of the drill (listed above) met?

• Demonstrate the ability to effectively deploy resources related to the staffing allocations for the 12/31/1999 transition

  From the standpoint of being able to pre-position personnel in key strategic locations, this objective was met. From the standpoint of being able to dispatch multiple technical teams, in order to deal with simultaneous microprocessor based failures and perform manual workarounds in multiple geographic locations, the objective was not satisfied.

• Demonstrate the ability to implement operating and administrative procedures, as necessary to maintain the reliability of the Bulk Electric System

  To the extent that this objectives 1 and 2 overlapped, this objective was met. However, it was not apparent that any written Y2k specific contingency plans were utilized, from a command and control perspective, either in the control center or remote locations. Even if the contingency plans were utilized, again, no specific operating procedures could have been exercised because no equipment was exercised or disabled during the course of the drill.

• Demonstrate the ability to effectively use backup (alternate) communications systems

  Per the discussion above, this objective was met.

• Demonstrate the ability to effectively use backup (alternate) communications systems

  Per the discussion above, this objective was met.

• Demonstrate the ability to observe and document drill activities and provide an objective analysis of the overall response by the participants

  Im not totally sure what the purpose of this objective was meant to convey, other than the ability of the organization to have observers taking notes. A consultant who has worked extensively with the transmission providers Y2k program was the designated observer in the facility  he was there and taking notes. Does this satisfy the objective? I suppose. There were also counterparts to this individual stationed in the remote locations. However, Im still not sure that this objective meant anything to begin with, from a standpoint of operationally exercising equipment, personnel and procedures.

• Train drill participants in Y2k specific activities

  A lot of training took place prior to the drill, particularly in using satellite communication equipment, and this training paid off. Communications was not a major problem. However, communications is only one aspect of this objective  theres another part that wasnt exercised, and that is the ability of personnel to adequately respond to simulated multiple (and simultaneous) system and/or equipment failures.

---

Ill close my comments on this drill by relating an interesting conversation that transpired with one of the system operators. The operator had gone through the big northeast U.S. / Canadian ice storm a few years back, when power was lost for weeks in many locations. Fortunately, the operator was only without power for three days. Still, even during the three days, life without water (an electric well pump supplies water to the operators house) was difficult at best. Imagine, the operator said to me, flushing the toilets with snow. It wasnt pretty.

The operator then described some personal preps to me, including the purchase of a generator for the well. I dont know how bad Y2k is going to be, but Id rather be ready than not. I learned my lesson.

Others before me have said it best  listen to what the folks in the trenches say, and pay even closer attention to what they are doing.

-- Anonymous, September 10, 1999

Answers

Rick-

Just wanted to thank you for all of your hard work. You're one of the guys who help us all see a little farther into the Millennial Fog.

Best regards,

-- Anonymous, September 10, 1999

rick,

it appears that the skids were greased for this media show. since no actual equipment/systems were exercised, i guess that it would be hard for them to fail. imagine that, and i thought this was going to be a grid wide test. silly me, you know that the spin doctors would never bet on anything but a sure thing.

thanks for the hard work and making this bbs as good as it can be.

again dont listen to what they say, watch what they do.

now watch me pull a rabbit out of my hat... nothing up my sleeve...

al

-- Anonymous, September 10, 1999

Wanna hear a funny one? We're so confident in the oil industry, we're not even making the effort to spin out a press release, we just have some shill from the API draft a White Paper saying the whole thing is not a problem.

Thanks again Rick for going there and posting this.

-- Anonymous, September 10, 1999

Thanks for the insider's view Rick, I wasn't involved in this at all. I did see that the NERC press release was pretty good in stating the real purpose of the drill, but the press as usual hasn't a clue. The NERC press release seemed "ready to go" as well...

Regards,

-- Anonymous, September 10, 1999

The NERC Press Release _was_ ready to go, I have no doubt. It was posted to the NERC ftp site at 2:42 P.M. yesterday, 09/09/99. For items besides press releases it normally takes from 2 to 6 weeks before they're posted. Whatever else we might think about NERC's capabilities, they get an A+ in public and media relations.

Thanks, Rick, for the report. I hope you managed to get some sleep! While I was reading about this drill, I couldn't help contrasting in my mind Malcolm's report of the "surprise" drill their controller had in the middle of the night. I think if NERC or any utility were really serious about making sure of emergency response (for anything, not just Y2K) they'd pull a surprise drill TONIGHT, even if was just in communications. It seems to me that would be a whole lot more telling and profitable for all concerned. Even the military practices surprise mobilization drills to make sure they can contact people efficiently in the event they need to.

Any of you who work in the U.S. electric industry -- have you ever had a surprise drill of any kind done? Or been called at home to verify your contact number? Or are the only surprises real emergencies?

-- Anonymous, September 10, 1999

Bonnie, Regarding NERC vs. the suprise Drill that Malcom spoke of, how about his post concerning the low rate of utility Y2K Readiness in NZ? Say what you will about NERC, they have been a postive factor in the Y2K Readiness successes in the US.

Regards

-- Anonymous, September 11, 1999

I was an observer, participant, and director of our drills. We had over 200 people that were involved. And I'm still tired, 3 days later!

In my personal opinion, the main function of the drill was to do the "dress rehearsal", and get folks used to the fact that they'll be working at night, in the dark, when the rollover comes. The operators are of course already used to doing this, so it wasn't a big deal for them, But it was a big deal for the big brass office junkies, and the field personnel.

Bonnie, to answer your question, we did surprise several folks with drills (and I was surprised to learn that I was to be a participant in one of them). They knew that something might be coming, but not the specifics. We tried to create scenarios that would get folks out of their "ordinary" emergencies and into some that Y2k might bring. For example, if there is an outage, some customers are bound to call in and ask, "is this a Y2k problem"? So how do customer service, distribution operations, and field personnel quickly find an answer to this question? That was a drill we performed.

One down side to the drills was that very few power companies have their satellite phones yet, so we won't be able to test that until later.

Finally, Rick, glad to see you had an opportunity to be an observer. FWIW, my experience on the personal generator thing has been exactly the opposite; everyone I talk to in the business does not have one.

-- Anonymous, September 11, 1999

Gee. Rick Cowles says industry personnel are buying generators. But Dan... er... Dan... ummm... Dan, just Dan, only Dan, merely Dan, says they're not.

Gee.... What a quandary.... Whom to believe? My, my, my, what a difficult decision....

-- Anonymous, September 12, 1999

Dan,

I can *guarantee* you that some people in the power industry are buying/have bought generators. Write it down & take it to the bank. A minority, I'm sure. But they do exist. I'd be stunned beyond belief if anyone who's done even a minimum of Y2K/utility research couldn't find such people. It sure didn't take me long.

-- Anonymous, September 14, 1999

rick,

this is not a loaded question...

you commented:

"Communications appeared to be adequate, but satellite based telecommunications systems are somewhat flaky on a good day,"

this led me to consider... what effect would the anticipated solar storms have on the successful performance of satellite phones, if any? would it render them useless or have little or no effect.

i feel that this is an important issue limited not only to the power industry... our whole country's contingency planning is geared around satellite communication.

-- Anonymous, September 15, 1999