I don't post very often, but read every thread each day. Thanks to all of you GIs for the valuable input. I've started using this alias because of the company I work for, and as salve for my general paranoia.

Although my company has a "drop dead" date of June 1, 1999 for all Y2K projects, and although we have been doing exhaustive regression testing on the big-iron programs and data, it seems like a new bug pops up every day.

Form our local Y2K meeting in Wichita. Most of it sounds like the same old line everybody else is spouting, but this is the first mention I have seen of a major utility stating publicly that they will pull off of the national grid and instigate their own Black Start Plan.

It also sounds like the local water boys are genuinely worried: Youre only going to get to flush the toilet once (if the power fails).

Does the whole thing sound sort of naove to you? It does to me.

http://www.wichitaeagle.com/technology/y2k/y2k0324.htm

LOL

Mike

-- Planebuilder (Y2KOldgeek@aol.com), March 24, 1999

Answers

All utilities have "black start plans" even well before Y2K.

-- zzzzzzzzzzzzzzzzzzz (zzzzzzzzz@zzzzzzz.com), March 24, 1999.

I just saw this article summarized in Sanger's Y2K Review!

Sorry if this post seems redundant for some, but I did not expect Wichita, Kansas to make the Big Time.

-- Planebuilder (Y2KOldgeek@aol.com), March 24, 1999.

Dear zzzzzzzzz:

Thanks for that condescending response, you asshole.

I know they all have black start plans. This is the first time I have seen a utility PUBLICLY ANNOUNCE that they would go off the national grid and restart in "two to four" hours, however.

Go back to sleep now.

-- Planebuilder (Y2KOldgeek@aol.com), March 24, 1999.

Mike -

I have a neighbor who is a local supervisor for our regional power company. He told me a month or so ago that our power company was compliant as far as mission critical systems, but admitted that he wasn't at all sure about the grid itself. He also said, and I wish someone could verify this, that the rest of the grid (in our case the Western portion) had to be up and running to black start our utility. In other words, one utility cannot take itself off the grid and run itself independantly because most utilities do not produce enough of their own power to run their entire utility without outside input from the rest of the grid they are connected to.

-- Valkyrie (anon@please.net), March 24, 1999.

pb: I thought the same thing when I read this story today. The media has gone from warning that "your toaster may not work!" to wondering if "we should be able to restart the Grid in 24 hours".

Quite a transition, IMO.

-- a (a@a.a), March 24, 1999.

Val,

the info I have *somewhat* substantiates your claim. Older facillities, especially in smaller utilities, often require assistance to blackstart, while newer (say in the last 15-20 years or so) construction, especially in larger utilities often are designed with blackstart capability from the ground up. Locally here in the DC metro area the utilities have folks who are trained to blackstart their plants - it's a team effort.

Arlin

-- Arlin H. Adams (ahadams@ix.netcom.com), March 24, 1999.

My utility is so clueless that they probably do not know what black start is. The Public Utilities Commission said that this power "provider" is 25 per cent completed with renovation with a scheduled completion date of June 1999. A generator is prudent in this area. When are the regulators going to wake up?

-- Tom (amazed@aaa.com), March 24, 1999.

Excuse me, but what is the definition of 'Blackstarting' ? I thought it was being able to start without external power. Seems like I was mistaken from what you folks are saying.

-- (someone@somewhere.com), March 24, 1999.

Someone, the def'n hasn't changed. The scenario is that the relevant portion of the grid starts to suck enough power from the specific utility, that it goes down. If you bting yourself back up, the grid demand knocks you down again. What tis utility is talking about is the publicly dreades "Islanding", where the utility sees the draw increasing, pulls teh breaker bars (believe me, they will probably have to do it by hand) and brings itself down. they then make sure that there are no connections to the grid, and restart their generating capacity, energize their own lines and YOU have power.

THIS, is what I think Ko-Skin-Em had in mind in his now infamous "private enterprise may not be able to allocate...and we'll be ready" to do it for them statement.

The first thing that scares them sleepless is bank panics. The second, and this opne dries the mouth, is the possibility that as the grid becomes unstable, and damaging to those that contribute power to it, they will elect out, creating islands of haves and have-nots. this being unfair and undemocratic, what they might do is nationalize the haves, and make them go back to contributing to their part of the grid. Disregarding, for the time being that they are destroying the units they are using.

Robert Cook or someone might be by to fill in any holes or correct anything I've got backwards.

Chuck

-- Chuck, a night driver (reinzoo@en.com), March 25, 1999.

Still using external power, but total grid is black.

-- cant (start@no.grid), March 25, 1999.

Couple of good comments above, and a few (right up there at he start) that are simply confusing - "go back to sleep...,etc.?" I'm not sure what he implies by that....

Regardless of those people, you're correct Chuck - but we still need to go a little a deeper into the problem. (like most stuff in year 2000 issues, there is a short general answer that is wrong and too simplistic; a detailed exact answer that is correct in one case, and utterly wrong in another case; and a long boring, equally wrong answer that is too detailed; so I'll try to condense some useful information out of all the related stuff. Sorry people, the problem is very complex, and from I've seen of the testing to date, the powers-that-be aren't testing in enough detail to uncover all the protential (much less all the probable) problems.

Literally, a utility has to be seen as several related, but independent entities - you (at your house) only see the voltage at your wire's connection to the street power's line transformer.

So you've got to look at several levels:

First - look downstream from the utility to the house - one, get power to the high voltage transformer yard at the utilities' power plants themselves. Then distribure that power through the high voltage lines looping across a single region (city munipality, EMC, or county are typical sizes), these loops connect many individual substations and a complex distribution network from the high voltage loops down through medium voltage transformer stations then to neighborhood transformer and control stations to the overhead or buried local lines that you see.

If a utility (such as my own: Cobb EMC) or a city such as Marietta Power Company don't own a power plant, they have to begin by connecting to a neighboring distribution system (EMC) that has power in its "loop" from the originating source (Oglethorpe Power), measuring and buying that power, then distributing the current through its local loops and networks. In this case, Marietta and Cobb EMC could not "island" themselves - there is no independent power source - but still need to be able to test their "blackstart" procedures. The city of Atlanta, south of us, is on a different series of distribution loops, and buys its power from GA Power. Thus, it is very likely that Marietta may have power, but Atlanta is blacked out; or vice versa. (Given that Marietta completed remediation and testing in Sept 1998, and the City of Atlanta hasn't finished assessment yet, I think it more likely that Marietta will have lights, and Hartsfield Airport and Atlanta will be very, very dark.)

In this case at this local level, a "black start" would mean identifying sources of power, identifying how and where to connect, making the connection itsself, then "restarting" all downstream services safely without tripping off critical upstream supplier services as load surges and increases. Possible, certainly. If you're prepared and have practiced the technique. But the actual methods of "blackstarting" are very different from "re-connecting" after a local power loss - consider the difference between being at your car dealer and disconnecting and replacing your car battery at 10:00 am on a Tuesday, and being in the middle of a farm road in Kansas at 2:00 am during a blizzard with no jumper cables and no farmhouse in sight. In the first case, you need to be able to pay for the battery; and after the mechanic has used his wrenches to remove the clamp, loosen to connectors, replaced the battery, and retightened everything; all you need do is restart the engine and reset the radio.

In the second case, you have to start by figuring out where the nearest farmhouse is that has people, a battery or a car or truck or tractor, and can you get there. Do you have jummer cables? At night, (no lights) do you have wrenches if you need to replace the battery? Can you find the rigth positive/negative terminals in his car, in your car with no lights? Does he use a 12v battery in his tractor, or is it 24v, 6 volts? Did his his dog bite you walking to the front door? Did he have jumper cables? If not, what will you do? Can he get his car close enough to make the hookup? If not, can you push your car out of the ditch? Can you safely leave the car at all in the snowstorm? Should you leave the car? What about passengers in the car, if any? Are they safe?

Do you see the difference in degree of problems - though both involve the same root cause, the degree of troubles in a "routine" hookup after year 2000 could be very, very difficult, compared to the same task down now. The command, control, and sensing problems are increased at every level. A blackstart, in this case, can be done, but it is made much more difficult if the control center is "blacked out". Feedback of the newly varying loads "upstream" to the providing network and power plant is made more difficult too.

Assume a utility has a power plant - the problems of a "blackstart" remain equally true as the utility tries to re-connect to the grid. (Because the time to restart the power plant is not known when it first trips off lines.) So even with a power plant, procedures on all three shifts need to be rehearsed at each utility distribution center.

At every power plant, the "blackstart" process must be checked aas well - because no two plants are identical, and each will require a different method. At the power plant, a "blackstart" assumes teh lights are out, and the plant is shutdown. No pumps, no coal crushers, no hydraulics, no steam, no condensate water, no power, no fuel pumps, no burners, no boilers - or the boilers are cold and full, or in hot stand-by, or whatever (the process changes in every different initail condition.) The plant staff must properly get each load and system restarted in the proper order and under safe conditions so they can resume generating power. The easiest to start off are the emergency diesel-gen sets, then aircraft-derived turbine generators, the most difficult are old coal-burning steam plants.

But the small diesel gen sets and gas turbines can't support the whole local network - so the loads have to be managed very, very carefully after they come on line. Else, they overload, and the blackout resumes. As a big coal plant comes up, or attempts to come up. its restart may easily exceed availble power (to turn covneyor belts, start up feed pumps, condensate pumps, filters, crushers, etc.) Each time the limit is exceeded or someone goofs, the startup procedures must be aborted, and then restarted. It's why I expect interrupted power for a long period. We hope that manual restarts won't actually break equipment or systems, but cannot assume that will be true in all cases.

Going "upstream" towards the national grids - combined, many of these "loops" make up a network that may span one or more counties - some cross state boundaries. The regional high voltage lines ultimately cross the "grid boundaries" at specific points between the four "national grids" that people generally think of when they mention the "grid" - but electric power is exchanged and sold at many levels below these points.

The shear size of the load represented by the national grids represents a stabilizing force. As loads come on and off, the national load remains relatively stabile. If a utility isolates itself, the relative magnitude of load changes and surges increases with respect to the availbe generating capacity. Hence, to a certain degree, the local system becomes soemwhat more unstable as islanding increases - and of course, the loss of any single power plant greatly increases the problems becuase backup sources are eliminated by islanding.

In my opinion as an observer no tinside enough control and distribution centers, not enough attention and drills are being made to this whole problem of restarting from conditions of some local power (somple black recovery), no local power ("black connects") to the networks, "black starts" at the power plant level, and accomodating the effects of "black connects" to upstream (feeder) power plants as whole regions come back on line at once, potentially with the power plants themselves under less reliable, slower manual control.

-- Robert A. Cook, P.E. (Kennesaw, GA) (cook.r@csaatl.com), March 25, 1999.

Mr. Cook -

Thank you for your black start explanation. Makes it so much easier to see the problem. Another question if I might. A while back I read that if the telecom lines are down if/when a rolling blackout starts, that the power lines will burn out and have to be replaced. Is this true? Is this caused by the shut down of local power companies and the resultant excess power being fed downline? If so, I am afraid here in Western Washington our chances of having power restored in a month or so would be doomed as our transmission lines come over the Cascades and this year anyway, they have 400 + inches up snow up there. I would think access except maybe by helicopter (high altitude birds and pilots at that)would be nearly impossible. Therefore, we would be looking at June earliest for power (unless they can reroute through the Columbia Gorge).

-- Valkyrie (anon@please.net), March 25, 1999.

Seems a bit far-fetched to me. "Wire" (line) troubles don't appear to be likely under even the biggest surge and restart condition I can image.

Compare this to the transformer - control station - distribution controller failures that would be more likely to fail (literally to blow up) under stressful conditions of high current, high voltage, voltage peaks, high-low frequencies. More problems likely, but also easier to find and isolate, then to get to and fix. An individual control circuit in a single given processor could fail open, fail shut, fail stuck (mid-position) if a chip chip were bad - then the gadget or process it is controlling or monitoring could fail more dramatically by overflowing, losing fluid and going empty, breaking or whatever.

the cummulative effect could be cascading faiures of different types totally unrelated to the originatig "bad controller" - like a loose nut - by itself, any single loose nut might not matter. But if the loose nut were in your oil drain pan, then plug loosens and may fall out, the oil drains out, the engine runs hot, siezes, and then the car stops. But the actual problem is the loose nut, not the overheated engine.

A single loose tire lug nut might run fine for many thousand miles, until you hit a bad pothole - then the wheel comes off and the car rolls over because of the extra stress on the remaining nuts. The problem was the loose nut, not the pothole. By itself, the loose nut was not dangerous. By itself, the pothole is not dangerous, at 5 mph. By itself, driving at 60 mph is not dangerous. A pothole at 5 mph is not dangerous, and probably won't damage the stud - even if the nut were loose.

But the rollover at 60 mph is deadly. This is the stunning illogic of those like the original poster, and some others like ANP, who claim by extrapolation:

"Nothing has happened in the past - therefore nothing will happen in the future."

Not true - the system has not failed in the past ONLY because it has NOT ever been tested in such a way as to make failure possible. Any given bridge will fail - if loaded with too great a load weight. The Tacoma Narrows Bridge failed catastrophically under conditions of NO WEIGHT at all - but failed utterly under light winds only slightly higher than previously experienced during construction.

Yet up until the very day it broke in half, a highway administrator could rightfully claim - the Bridge will stand up to any load - its been under winds before, its been under loads before, and has never failed during the past four years of construction. Its designed with a factor of safety for loads four times higher thatn ever expected, and so "I assure you it is safe to drive on tommorrow."

But "tommorrow" it broke in two, a falling twisted mass of steel and concrete into the water. Despite what the bridge administrator said. It was simply tested under different conditions than he expected.

-- Robert A Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), March 25, 1999.

Check submit.

-- Robert A Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), March 25, 1999.

Check passed.

-- Robert A Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), March 25, 1999.

Thanks Robert,

That explanations a keeper.

As I recall, in the San Francisco Blackout last December, it took them longer than expected to turn the city back on. They had to do it manually, and small section by small section (specific neighborhoods).

Would that be an example of a Black Start situation?

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), March 25, 1999.

Yes - that was a "black start" from neighboring regional loops - but even that was an "easy" recovery compared to potential year 2000 issues.

Consider: "They took longer than expected to recover" - from a "live" network, with the national grid grid up and the local "loops" up and under normal "fully automated and regulated" power control. With the area power plants not affected by the blackout, nor were the area power plants troubled by any year 2000 disturbances or having to use any manual controls. No telephone, comm's or satellite difficulties in the region. Power out short enough that secondary and tertiary difficulties (gasoline pumps out, radio and cell phone batteries still charged, forklifts, cranes, trucks and rail was not affected - other than the BART trains. Emergency generators functional and not in danger of running out of fuel. Food, water pressure and fire fighting still up because it was only a short time, but HVAC and building services out - no business that day, but a "holiday" atmosphere amongst the population = Utility crews were the heroes, they didn't need police or state troopers to protect them from rioters. No looting. No weather danger - the peninsula doesn't get hot, doesn't get dangerously cold - ever, so losing house power put no one in danger. )

Only one small region of the whole Bay area out. A single known cause at only only one failure point - that did not repeat its failure elsewhere. Imagine trying to "bring back up neighborhoods" when each time you do the neighborhood immediately trips off again - sometimes - but you have to bring the power back on to try to figure out why it is trippig off, and figure out what areas don't trip off, and which ones do, and why. (The answer? Same controller, same maufacturor, same part number, same embedded controller - but the chip itself was made by a different Tawanese plant in two different production runs to supply two different assembly plants in Singapore that happened to shipped the parts to the vender in the same crate. Sound plausible? Guess which one the vender "tested" when he "certified 2000 compliance".)

That's why even that level of "regional simulation" is not really a valid example of recovery. It was "too easy" a recovery to even serve as an training example (a drill) to compare to the potential, very realistic troubles likely to simultaneously happen for longer periods after year 2000.

Drills must be thorough, realistic, and cover the degree of troubles reasonably possible. Training and simulation must go way past that level of drills - because operators MUST know all the details of their systems so they can make the systems work despite the "routine" support structures and systems that ARE NOT WORKING during the emegency.

Despite the bland comment above the "sleeping observer" about Black starts as "routine training" - I don't know of ANY "blackstart" training that ANY power company has done at ANY generating plant, ANY distribution system, or ANY grid. If anyone has evience to the contrary, please enlighten us - that utility might be the only one withits lights on.

And such training, in my humble opinion, must be done at all power plants, on all three shifts, and for all networks and all utilities. Unless they intend to use to "on the job training" on all three shifts next Jan and Feb with your power.

-- Robert A Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), March 26, 1999.

Thanks Robert.

*Sigh*

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), March 26, 1999.