I've been back at work for two days after a lovely two week mid winter break, and I have both good and bad progress reports to make.

Our new MITS Scada system is nearing the end of its bench trials, and appears to be working up to specifications. There are still one or two items that I'm not happy with, but they are minor in nature and we should be able to learn to live with them.

The Honeywell SCAN3000 sytem has had a patch applied which is now allowing it to work as well as the pre Y2K upgrade, however it is only a patch, and a more permanent fix is still required.

One sytem which we believed to be Y2K ready has proved to have a problem. It is a remote control and alarm system (but not a full SCADA) called MacroView that is used to communicate and supply data from RTUs and PLCs at a remote site. Our initial testing, and the manufacturers documentation showed that it was Y2K ready. However one of our technicians ran a Y2K test that caused it to fail. Further tests as a result have showed that its performance during the rollover could not be relied on. The suppliers will not be able to provide an update of the software before our close off of new systems next month, so we have trialled turning the clock back. This method will provide a temporary solution untill after the Y2K rollover when we can then implement a permanent upgrade.

I have been talking to a friend of mine who works for one of our competitors, and he (accidently or on purpose?) let slip that their NEW SCADA and EMS system is giving huge problems with its installation, and it is not Y2K compliant. The company that he is working for is trying very hard to ensure that they have sufficient trained staff to manually control all nine of their power stations during the rollover, and possibly for some weeks afterwards. I think that he was hinting that I should change employers as I have had a lot of experience in manual control, but I'm happy right where I am. :-)

Malcolm

-- Anonymous, August 24, 1999

Answers

Thanks, Malcolm.

-- Anonymous, August 24, 1999

Thanks for the info Malcolm. What do you think of these "facts" factfinder?

-- Anonymous, August 24, 1999

Anyone who has ever hung a door knows it is pretty much a thankless task. There's an awful lot that can go wrong, much more than most people would imagine. If you get everything right, then people open the door, walk through it and don't even notice it. However; if you get something wrong then everyone who uses the door will notice it.

Most people don't want to know all about hanging doors, what can go wrong, how to correct it, etc. They don't even really want to know why it's not working properly. They just want it fixed and to forget about it.

Malcom, the way you post here is like one of those things that we take for granted when it works perfectly. There's absolutely nothing to complain about! It's just right!

Thank you tremendously. If we all get through this in good shape it will be because of people such as yourse

-- Anonymous, August 24, 1999

Malcolm, do you know if, when, and how the NZ government and populace have been and/or will be informed about your competitor's situation?

-- Anonymous, August 24, 1999

Malcolm,

You wrote:

>>One sytem which we believed to be Y2K ready has proved to have a problem. It is a remote control and alarm system (but not a full SCADA) called MacroView that is used to communicate and supply data from RTUs and PLCs at a remote site. Our initial testing, and the manufacturers documentation showed that it was Y2K ready. However one of our technicians ran a Y2K test that caused it to fail. Further tests as a result have showed that its performance during the rollover could not be relied on.

If you could not have implemented this solution, what would have been the consumer impact of this problem?

Also, in your estimation (to the extent you can guess), is it possible that customers of your competitor might lose power if they find themselves unable to manually operate their systems (for whatever reason)?

The point behind these questions is simply to attempt to gauge the level of risk which may exist in non-remediated or insufficiently remediated systems.

Thanks for your report.

-- Anonymous, August 24, 1999

The company that he is working for is trying very hard to ensure that they have sufficient trained staff to manually control all nine of their power stations during the rollover, and possibly for some weeks afterwards.

I think this comment raising an interesting and often overlooked element of Y2k. People. The man-machine interface. I don't think anyone should be surprised by the industry assertion that it is possible to run an electrical system without computers. After all, we had an power grid before we had computers.

The problem with this idea is that it ignores the worker. Essentially you change the job description suddenly. Experience does not count for as much. The job can be done, but this worker in this plant has never done his job without the technology before.

The opportunity for human error will skyrocket and therefore the number of human errors will skyrocket. The real disasters will happen in places where a computer failure is compounded by one or more human errors.

Tom

-- Anonymous, August 24, 1999

Malcolm, I just posted a comment to another thread about the average customer just wanting straightforward info. Some of them have written to me that they wish American utility people could be as communicative of the good and bad as "that Malcolm guy". (Smile) Even if those Y2K aware people who write me don't have an engineering or IT background, their common sense tells them that with any major assessment and changeover of systems in any business, there _has_ to be problems encountered, both expected and unexpected. That's probably why the "no problem" reports so endemic here in the U.S. are looked askance at and engender mistrust.

Reading your reports is like a breath of fresh air for most of us, even though we recognize the incredible legal tangles here which tend to prevent getting this kind of report from U.S. utilities. Yes, you're working hard at finding and solving any Y2K problems. Yes, some things have gone very well, others not so well. Yes, such and such was a surprise, but we're dealing with it the very best we can. These situations are what most people's common sense tells them _any_ Y2K project would be like, since good, bad, and in-between is just the way things are in life, with a surprise or two usually happening along the way.

Most of those I hear from who have decided to make some individual/family risk-management preparations have done so not because they necessarily think everything's going to fall apart, but because their experience in life tells them that there are always unforeseen surprises which _may_ upset the apple cart and affect them. "The best laid plans of mice and men oft' gang agley." I might not have gotten that quote completely accurate since I'm going by memory and don't have my reference books handy in this hotel room as I would at home, but the idea comes through, and it's an idea that's been around long, long before Y2K.

I hope you had a great two weeks off, Malcolm, and thanks for the update. And give the guy who did the extra testing and found that unexpected problem a pat on the back from us, too, will you?

-- Anonymous, August 24, 1999

I get an email notification with every post. Since most people don't bother signing "Susan Sarandon" or whatever at the bottom of the post, you can't tell who wrote what. This is an interesting experience for a while - evaluating the quality of the argument and comments without knowing who or what you are evaluating.

I discovered that posts made by very sexy grandmothers are very easy to pick out because they always make good common grandmotherly sense.

Y2k is not going to be about how well the leaders, the best and the brightest do. Y2k is going to be about how well the laggards manage the trick. Malcom's reports do have the ring of truth for exactly the reason Bonnie suggests. BC Hydro's reports carry the same mantle of credibility. Transparency is not that difficult to spot.

It is as obvious as a nice pair of legs. Hubba, hubba.

Mel

-- Anonymous, August 24, 1999

Bonnie - I believe it goes this way.

"The best laid schemes of mice and men gang aft agley And leave us naught but grief and pain for promised joy"

Robert Burns - "To a Mouse"

Burns if one of my all time favourites and that is a stunning poem - a whole philosophy of life. I lurk here every day and enjoy all the posts. Thank you a thousand times over. Because of my research, I am the office's Y2K research person (I work in a local Councillor's office) and I actually helped my Councillor fight for and get generators for our police stations - an awful oversight!

Haven't read Burns for months - thanks for the reminder!

-- Anonymous, August 24, 1999

Now this is the kind of thread I like - a technical discussion of actual Y2K problems. Thanks for the update, Malcom, however your post leaves me wanting to know much more! I believe Drew gets right to the heart of the matter on this with his questions and his closing comment:

Drew: "The point behind these questions is simply to attempt to gauge the level of risk which may exist in non-remediated or insufficiently remediated systems."

While I am familiar with many types of equipment used in US power utilities, I am not familiar with all equipment, especially some of the equipment other countries such as yours.

My questions: 1. MacroView "Scada" In what application is this software used? I am not familiar with the MacroView, but the information I read on the website indicates that this is actually software used for monitoring purposes, not control, is this correct? You stated, "One of our technicians ran a Y2K test that caused it to fail." What was the nature of the failure, did it lock up, or just fail a test date? What were the effects of the failure on plant operations (if left unfixed)? Is the problem with the MacroView software, or with the platforms its run on?

2. "competitors...NEW SCADA and EMS system is giving huge problems with its installation, and it is not Y2K compliant. The company that he is working for is trying very hard to ensure that they have sufficient trained staff to manually control all nine of their power stations during the rollover, and possibly for some weeks afterwards."

I'm not quite certain of the context here...SCADA/EMS indicates that you are talking about load dispatch control, not actual plant controllers (other than the standard KW control the dispatchers can perform). Or are you also talking about SCADA systems that actually control the plant? Is the sufficient trained staff needed for manual load control by the dispatchers (i.e, telephone dispatch to the plant operators for load management)? Or is this for the generating plant manual control? --

A few comments: For the US plants I am familiar with, the larger fossils use high-end DCS systems for control, not SCADA. It is common in the US to use SCADA systems for supervisory controls of smaller combustion turbines and hydro plants, but even then the equipment has primary controllers running the plant (dedicated combustion controls by the maufacturer, PLCs for hydros, etc) with SCADA performing only supervisory tier control. In some applications such as small hydro plants, there is sometimes NO manual backups to the PLC /MMI controls.

I tested some PC based test equipment (not installed in the plant) that the vendor said would function but with minor date errors. This was "custom" equipment, about a dozen or so units in use in the industry. Wrong, I had a hard failure, the software crashed due to divide by zero errors - I remediated by installing a patch on the PC platform it was run on. The minor date errors were dealt with administratively, since it was desired to continue using this equipment. It goes without saying that it's important to test all mission critical devices with date functions. We even tested non- mission critical devices installed in the plant that were remediated.

Regards,

-- Anonymous, August 24, 1999

Malcolm,

Welcome back from vacation. FactFinder makes a point that I would like to comment on, regarding a plant system with *no* manual backups. This has happened in aviation as a result of both being seduced by computer technology and a management desire to maximize profits. It used to be, years back, that all primary flight controls were capable of being used in a manual mode. They had the hydraulic boosts for normal operation, but if hydraulics failed the system reverted to cables and pulleys and you now physically moved the control surface with brute force. Of course this is the only way that the smaller aircraft operate anyway, such as the Cessna's, cable and pulley and human force. In a similar way your automobile with power steering could be an example. Normally you are not actually moving the wheels with the steering wheel, you are moving the hydraulic assist valves which govern all the work being done. But if you lose the power steering, you *can* physically force the wheels to move, by using a lot more force, through the steering shaft and gears alone. That's what I mean by manual backup.

In recent years the aviation designers and engineers have removed all that cable/pulley hardware, to save weight (and cost) and improve fuel range. Sure makes the model look better on paper. But no more manual backups. Now they have redundancy instead. 2 or 3 hydraulic systems available and 2 or 3 computer systems available. In theory, the probability of all things failing at the same time is extremely small and is "an acceptable risk" to the engineers. Great. For me, give me some cable/pulley backups. I love the way the new technology works, and it does work well. Still, call me old fashioned, but cables/pulleys are kinda nice too, if you need them. Betcha a lot of passengers have no idea that circumstances could arise that would leave the pilots with no manual way to control the airplane. It has happened, more than once. Don't you just love the engineers who dream that stuff up?

-- Anonymous, August 25, 1999

Drew and factfinder,

I'm still catching up on everything that has been going on with our testing, but I'll try to answer your questions as well as I can at this stage. As I find more information I will make it available.

The issue with our Macroview system is a software problem. I'm not yet sure just what the cause is, but the result is more than just a date issue. The system is mainly for monitoring, but there is some remote control functionality as well. It is used to monitor the smaller of our hydro stations, with the ability to start, stop and change load on the generators. It is not critical to the running of the station, but does allow us to run the station unmanned 16 hours per day. (We have 2 qualified staff members living within 5 minutes walking distance of the site for any unscheduled work outside normal hours.) The fault that has surfaced appears to be a date incompatibility between the main server and the backup logging server which can at times cause the system to run so slowly as to be unusable. It doesn't fail completely, but it might as well be out as far as its usablility is concerned. The solution of turning the clock back is not an ideal one, but it does work, and will see us through in the meantime. If this solution had not been possible it would just require us to man the site 24 hours per day untill it was fixed.

With our competitors system, they are in the process of installing a completely new SCADA system (replacing their existing 12 year old system), and incorporating an energy management/optimisation capability at the same time. 7 of their 9 stations have been remotely controlled by their old SCADA, and the other 2 stations are being added. Their system allows them to start, stop and change load on all plant, Carry out all HV switching operations, report all plant status, trends etc. The EMS addon is desinged to allow the controller to enter a load setpoint, and not only will the load be split among available plant, but the loading will be optimised in such a way as to maximise efficiency. The failure of this system during our summer will not be noticed by the power consumers as long as the three largest stations in this group can be manned. I would be suprised if the government or general public would be informed of this issue unless they are unable to man the largest of these stations. They do have another very large station in another area, and combined with their largest station, and our companies hydro plant there will be no shortage of generation.

Malcolm

-- Anonymous, August 25, 1999

Perhaps I should have added that if our competitors largest station is running, that they must run 2 other stations for water management.

Malcolm

-- Anonymous, August 25, 1999

Mel,umm..Tom, if I was at my home computer right now I could pull up the exact time frame, but early this year I got an e-mail from someone who was a customer of B.C. Hydro. She had done her own status research and just asked me for my "take" on what B.C. Hydro was saying. I told her that after reading scores of utility Y2K web pages, you could get a good feel for the standard P.R. and anything else rather stood out. I also wrote that my gut hunch was that B.C. Hydro was telling it straight, that they'd get the job done in the necessary time frame, and they'd be good to go barring things outside of their control. The interesting thing is that this lady replied and basically said, "Good. That's just what I got from reading their information, too! Now if only my other utilities were the same.." I thought you'd like to know that some others agree that "transparency is not that difficult to spot."

Citygirl, thanks for supplying the exact line! I'm also very glad for all your efforts. The very nice thing is that you and your Councillor have accomplished a permanent improvement in the emergency capacity of your police force, regardless of what does or does not happen with Y2K. I shouldn't have been surprised, but I still was, when towns and cities for the most part did not take advantage of the Y2K issue to push through infrastructure, communications and emergency improvements the tax payers might not otherwise have wanted to pay for. It was a Win-Win political move (even if they weren't very sure about Y2K itself). "We're ready for any possible Y2K failures, and if they don't materialize we're ready for any other community emergencies which might come our way." Unfortunately, the same lack of foresightedness which got us into this Y2K situation seems to have held true throughout. Missed opportunities abound. It's great to hear somebody was thinking and making positive safety improvements. My hat's off to you both!

-- Anonymous, August 25, 1999