What did our government know and when did it know it? [oil embedded chips problem]

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Remember Watergate? Remember the Senate Watergate committee hearings? Well, some of you may, others may be a bit too young. The real key question in those hearings (and in the impeachment process that was started against Richard Nixon) was the following question:

"What did he know and when did he know it?"

It was asked repeatedly of witnesses of what they knew and when they knew it. It was the most famous question of the decade and we heard it crop up again during the Clinton scandals.

Now I ask the question in light of the more recent news stories about the NIST report, the Kosky memo, Mr CEO story which raises the issue of when did our government really learn the status on the embedded systems problems. Did they learn the "truth" just recently or have they known all along and the "spin" has been a desperate coverup of the facts? While we can't know for sure, I'd like to get your thoughts on this. Not to prove anything but to get your opinion of whether or not you the people think that our gov't has been deliberately misleading us or if they really were boneheads enough not to monitor the embedded chip problem when you or I knew it was a critical factor 2 years ago or more.

-- R.C. (racambab@mailcity.com), December 20, 1999

Answers

When did they know? I dont think we will ever know that but I do know the decision to keep quiet and lie was made two years ago behind closed doors.

-- Susan Barrett (sue59@bellsouth.net), December 20, 1999.

They will know as the same date we all will know:

Feb, 29th 1900 very late at night, (sorry, I'm not y2k compliant)

-- Rainbow (Rainbow@123easy.net), December 20, 1999.


Someone, somewhere, "knew", but was it reported high enough up and believed?

-- mushroom (mushroom_bs_too_long@yahoo.com), December 20, 1999.

I think when you read through Russ Kelly's "Ask the Experts" and particularly check out Paula Gordon's statements in the latter months, it really depicts Clinton's comprehension being limited in this area, he seemed to be more willing to let Al Gore/Koskinem handle things. I guess Clinton has had his mind on other pressing matters earlier this year...Anyway, Gordon gives the impression Koskenim has known for a while too. Hmmm.

-- Clueless (McCyber1@webtv.net), December 20, 1999.

 RC

Don't know if this will help but this is from the recent Press Club speach.

And it should be noted that the Y2K.gov site had a brief rundown of Embedded systems in the spring and it mentioned issues that the NIST document brought up. Unfortunately I can't find it. I would have the info in my hard drive (wow somewhere!) if you want me to look it up.

Excerpts: Y2K Czar Koskinen Speaks to National Press Club
 

A shorthand for those embedded chips is what it really meant was a shift in focus from information processing concerns -- financial systems, payroll systems, the things that you envision a big mainframe computer processing or your own computer processing, and it was a shift to focus to operating systems: What happens on the platform, what happens with a power plant, with an oil refinery, with pipelines? Were the operators focused on the fact that if you go into a lot of manufacturing or operating places or even ships you see somebody sitting in front of a computer running the place. And what they're doing, that computer is doing, that control system is doing, is responding to information coming from all of those chips out there that are monitoring hot and cold, on and off, whatever else might be going on.

 Smart building systems -- you go into a lot of new buildings, you go down to the basement, the janitor is not there with a shovel shoveling coal. He is sitting there in front of a computer monitoring all the systems which may be in fact set to go on and off according to the calendar.

 Fortunately it turns out that very few chips by themselves contain date-sensitive calculations. But a lot of them are connected to control systems. . . that in the process are sensitive to date-control systems. And so people have been working on that for a long, long time over the last couple of years as people became aware of it.

 The question is how many of them are at risk, and how many people really understand they are at risk? And so we have been talking about - - the industry has been talking about it. So a few weeks ago instead of having people -- there are a lot of people abstractly talking about it who have never actually done anything with embedded chips. We brought in industry experts who had been actually dealing with this problem in the electric power industry, the oil industry, the gas industry, as well as several experts from companies who have been actually working on embedded chips and government experts, to sit down and say, Okay, let's review the bidding: What's been done? What are the issues that we know and need to in fact focus on? And the consensus was that the major infrastructure industries in this country have in fact as a general matter understood the issue about chips and understood the need to test for them. They have understood that the fact that one device looks like it's compliant doesn't mean that the next one that look exactly like it is the same, because it may have a different either chip configuration or the chips may have been made by someone else, or it may be connected in a different way.

 What grew out of that meeting was all of this information is up on the Web, and we sent it around to industry groups to all of the people we deal with, including the civic action and active groups who have been interested in this issue. The consensus was that if there was a risk it's probably not in the critical infrastructure industries but it's in industries and manufacturing systems that are complicated -- simple systems don't have these issues -- complicated systems where even though it doesn't look like it's a date-sensitive process, there may be in that process, particularly in the software, but in the control systems, date sensitivities that you don't see. So everybody with a complicated mission-critical system should be testing it for date sensitivities. And that's why we put that out. The example given by one of the testers was looking at five bottling plants -- not exactly a critical infrastructure industry but important in a lot of ways -- they tested and rolled the clocks forward looking for these control issues. And four of the plants worked just fine. But in the fifth plant the sterilizing system stopped when you rolled everything forward. Nobody could quite figure out where in that complicated system was the issue and why, but it was clear that people needed to be aware of the fact that if you're running a complicated operating system you need to make sure you've done the testing. But again I would stress that the consensus around the room was that the electric power, oil, gas industries all were aware of that and of the need to continue to test.

 The other issue that grew out of that is an issue that we have been talking about for some time, and that is that manufacturers and vendors to their credit continue to update the status of Y2K compliance of their devices. And it's not that they were lying or cheating the first time out; it is that as we have had continual testing people have discovered that a device may work perfectly fine 80 percent of the time, but if you put it in a certain configuration it no longer works and needs an additional patch.

 And some software or hardware devices are into their third, fourth, or fifth patches or upgrades. And the concern we have, and it grew out of that meeting as well, but we have talked about it for the last six months, is that when you think you're done you are never done, that basically our concern is anybody who finished in June and said, "Now I can go on to other things," and didn't keep monitoring what are the upgrades, what's the updated information, may find that their system works just fine except for the fact in certain situations it is now not going to work because they don't have the last patch.

 So, again, the major industries represented said they all knew that they had people who are monitoring a regular basis what goes on. But our concern was to send a message to people -- and we talked about it in the federal government -- but outside the government that even when you think you're done, and if you have got manufacturer statements about Y2K compliance, make sure you keep monitoring that because their manufacturer may put out another statement saying you need another upgrade, or we have discovered more information about it.

 All of that information has been around. As I say, the major industries were not either surprised -- they were actually participants in trying to figure out what the lessons learned are. So there is not a reinterpretation. We don't think that the problem -- in fact, the consensus was bigger than people thought as we have been moving through it. But it was that there are these important issues that ought to be focused on as we go forward, particularly in industries which may not have paid the same amount of in-depth attention to the problem that the major infrastructure industries have.

Regardless what Kosky new or didn't know this thread is the most alarming of all his speaches.

A must read from April.

  TRANSCRIPT: KOSKINEN COMMENTS TO SINGAPORE APEC Y2K SYMPOSIUM - USIA

-- Brian (imager@home.com), December 20, 1999.



R.C.,

unfortunately in our society it doesnt matter!!!

The days of accountability have been over for some time. We may be returning to them soon. but at this stage the system is set up so that no one is accountable.

Janet Reno tried it in the beginning but even she caved eventually. Poor thing, the parkinsons really got her. prayers. To err is human to forgive divine. Those that have not sinned cast the FIRST Stone. Its too late to expect accountability!! from anyone.

-- d----- (dciinc@aol.com), December 20, 1999.


It's very important for .gov to assert that THEY are ready. That way, any embedded issues in Mideast can be "detached" for purposes of spin, control and morale at home ("it's them, not us, let's pull together").

I am coming around to the view that provoking worldwide preparation would have required disclosure of "evidence" that would have crashed the markets. I'm wondering if I have been too harsh on the spinners. IOW, if I "knew" that embeddeds were majorly exposed but the market crash would be the price of preparation and wouldn't prevent embedded breakdown anyway, what would I "do"?

That's a tough question.

Where it becomes tricky is the question you raise, "when was this known?"

If two years ago, there was not only time to prep but time to do major repair on embeddeds by raising the alarm. I put my money on "within the past six months" -- just about the time your own intel on this began to firm up.

If that is true, then they really were between a rock and a hard place. They could have put on a 14-day siren instead of a 3-day siren, but they might have judged that ANY serious siren would have spooked the markets WITHOUT any majorly useful result.

Putting it another way, if a depression is on the way, two weeks of food and water instead of three days aren't going to make a hell of a lot of difference.

BTW, I would still insist that, IF SO, our .gov will be much happier to have a peaceful U.S. (relatively, relatively) so the country can focus on the nightmarish risks we'll face internationally than come down with jackboots on our own citizens while STILL facing nightmarish risks internationally.

-- BigDog (BigDog@duffer.com), December 20, 1999.


R.C.,

FWIW, I believe that our "servants" are also divided into GI, DGI and DWGI, just like the rest of us. There's another factor in the high echelons, especially in gov't, and that is ARROGANCE ("I ordered it fixed, therefore it will be"). When the arrogant finally GI (my guess is this year sometime), the only thing left to plan was how to take advantage of the situation, since what takes place during any "crisis" is an enormous transfer of wealth. It is this transfer of wealth that has been controlling the "spin", making sure the sheep get sheared to the max.

-- ivan (ivan1776@ivnet.net), December 20, 1999.


Thanks for more of you sharing your thoughts and insights.

Brian, on Kosky...I detect just more continuing spin control. I wonder if he spins in his sleep too. :-) Frankly, I found the NIST report to shed more light really. As a former news reporter and PR person myself I know what it is like to spin positive news, but frankly I didn't have to try and twist negatives to positives.

I hope more of you who stumble into this thread will stop and post your thoughts too.

-- R.C. (racambab@mailcity.com), December 20, 1999.


R.C., I posted this IEE link on your earlier thread as well, but just in case you don't see it there, if you follow their "updates" link, you will find one on "reasonableness" that might attempt to answer your original question. My guess is that if I, an ordinary person, knew about the problem with embedded systems almost two years ago, people in high places did too.

-- Rachel Gibson (rgibson@hotmail.com), December 20, 1999.


I have tried in my White Paper on Y2K and Embedded Systems to address question of what persons in roles of responsibility in the U.S. government have understood concerning the nature and scope of embedded systems problems. I have tried to address the question of who failed to understand the problems, and who also failed to appreciate the implications of those problems for policy and action. See especially Parts 1, 2, 5, and 6 of my White Paper at http://www.edu/~y2k/keypeople/gordon.

I have also addressed these issues in the "Comments, Essays, and Op-ed Pieces" which can be found at that same website. In addition, the real video website accessible from the GW website has videos of panels and briefings that address these issues. The November 23 panel program at GW University on national and global Y2K and embedded systems initiatives that are still needed is also available online on video in the C-SPAN archives at their website at http://www.c-span.org. (When you get to the C-SPAN archives, click on the November 23 GW panel program.)

I recognize that this is hard to accept, but the fact is that people in key roles of responsibility in agencies of the Federal government, responsible for preventing disasters, responding to disasters, or both, have simply been inadequately informed concerning embedded systems. They have typically had no one on staff with the expertise to help them understand these matters. This includes especially individuals who have responsibilities for any of a wide range of technological man-made disasters. And some of the people in roles of responsibility in the government who now as of November 1999 have a better comprehension of the seriousness of embedded systems problems have yet to demonstrate a full appreciation of the implications that this new understanding should have for policy and action. They have failed to take the action that should follow from their new found understanding. It is as if the failure to understand technological complexities has served as a major barrier to sound policy and action.

On top of this few of those who have been responsible for the manufacture and distribution of embedded systems that are destined to fail have come forward to help educate and persuade government officials from the President on down about the seriousness of the situation we are in. And until November of 1999, few who have significant hands on embedded systems expertise have either tried to be heard or succeeded in being heard by persons in key policymaking positions.

Since May of 1998, I have been looking for people in government, including individuals in the Congress and on Congressional staffs, who had an understanding of embedded systems and their implications. I have organized meetings between people with embedded systems expertise and individuals in roles of responsibility in these branches of government. To my knowledge, White House advisors were not in effect successfully "briefed" on these matters until November 9, 1999. Also, to my knowledge, the briefings that the President and Vice President had on these matters as early as 1995, did not result in sufficiently increasing the comprehension and concern of either. It also does not appear that either the President or the Vice President is aware of the implications of the statements and article released by the Department of Commerce (DOC), the National Institute for Standards and Technology (DOC), and the President's Council on Year 2000 Conversion. Neither the President nor the Vice President has made any statement since November 9 concerning the scope and seriousness of the embedded systems problems that have yet to be addressed. Indeed, the comments on November 10 by the President reflected no knowledge of such matters whatsoever.

I have found very, very few individuals in policymaking roles in government who have a basic understanding of the seriousness of embedded systems problems. And most of those who have exhibited such understanding have failed for one reason or another to initiative the kinds of policies and actions need to help prevent and minimize the potential disasters that could result from malfunctioning embedded systems. For whatever reasons, only a very few of those who have had such an understanding or who have recently attained such an understanding have tried to make their influence felt. Only a very few have applied their understanding to action or at the very least a call to action. These individuals have not been in White House positions.

In the coming months, as the seriousness of embedded systems problems becomes a reality, one can only hope that those in leadership roles will be jarred into recognizing that it is incumbent upon them to apply adequate resources to these most serious threats and take action. I describe the course of action that they could take in my White Paper.



-- Paula Gordon (pgordon@erols.com), December 21, 1999.


Big Dog brings up an interesting point. Let's say that .gov is assuming things are okay and working and wake up say 6-12 month's to go, Hey Maybee just maybe we won't make it. Let's say they do some real quiet analysis and it shows that sure enough we won't make it. .gov now has the choice to tell the .public or not to tell .public.

If you tell the public you risk panic and almost certainly risk a recession. The panic is the peasants who cares, now the recession means that economic activity shrinks which means fewer resources available to remediate what can be fixed. That is a possible out for .gov we kept the bubble going to maximize the fixes BEFORE the event.

The problem is this .gov has used its creditability to maintain power. There is no second shot its gone so there is no believing for most of the population and that will make things interesting very soon. People will feel that anything more than a few hours or a single day tops is a betrayal. It will not be pretty when the Sheeple wake up, that is a sight that I don't want to see.

We shall all see soon enough, and we never know what they know and when they knew it until 50+ years later.

-- Squid (ItsDark@down.here), December 21, 1999.


Paula,

Thank you for taking the time (out of what must still be a busy schedule) to provide input on this question.

You indicated that the gov't leaders "have simply been inadequately informed concerning embedded systems." This suggests passivity on their part. In other words, how aggressively did they pursue this? I ask this but I doubt that any of us could provide a documentable answer. Your post does lead me to think that gov't leadership on this issue of embeddeds must have therefore been rather passive and certainly lacked follow through. This is amazing to me because as a layman I did quickly come to consider the embeddeds problem to be a more insurmountable issue than simply software code remediation. I came to recognize the enormity and complexity of the embeddeds problem in early 1998 (though I was coming to terms with the overall Y2K issue in late 1997 and was substantially aware as early as 1995/96. [Though I remember the issue first broached in a computer class in 1969! but we were told not to worry, someone would fix it in 30 years! yeah, right!]

My point is that if I, as a layperson inquiring and educating myself in my spare time can figure this out (and I did so way back in time) surely those whose job called for such an understanding should have done so way back then. It is nearly inconceivable to me that those tasked with the issues of Y2K would have not aggressively pursued a thorough education on this important aspect of Y2K. I know if I were in that position of responsibility I would exhaustively research the issue. In my case as a layman, I didn't have to get exhausted to pick up the necessary elements to realize that embeddeds needed special monitoring and that self-reporting was untrustworthy.

It is hard for me to conceive that presidential advisors would not have had intelligence personnel extensively research all aspects of embeddeds issues and that the results would have been so benign in their conclusions. IF this is so, then it speaks volumes concerning the levels of incompetence within the Intelligence community. I find that a little hard to believe that they didn't know. I can accept the notion that intelligence sources knew and simply failed to educate top leadership personnel effectively. I can see that political minds would attempt to ignore it because they prefer to focus upon "happy" thoughts and thus ignore vital issues that might confuse and derail their "positive" approach to solutions.

In looking at your historical recap (and its very good, I take no issue with your work) I am amazed that leadership didn't call these people into the loop until last month. These people should have been identified by intelligence sources, contacted and brought forward by 1995 at the first Presidential briefing.

I guess the bottom line here Paula is this:

If the top levels of the Administration and Congress were unaware of the embeddeds problems until 1999 then there has been gross dereliction of duty and a complete failure by top leaders including the President. They were unworthy of the responsibilities entrusted to them. This applies equally to members of both parties. This is really a case of gross negligence and incompetence from the top all the way down to the bottom.

Frankly, I suspect that the buck passing started long ago. If so, as it seems by Koskinen's statements, then that these people really have known all along but refused to admit it and have been spinning excuses for their failures. If the gov't is just now finding out the full scope of embedded systems issues it only points to gross negligence on their part. They deserve the blame. IF, however, they knew long ago and even then realized it was hopeless and suppressed that information then we have another instance of gross incompetence and mismanagement in regards to preparations.

No matter how I look at it. The government is at fault here as much if not more so than industry. It's really the case of what happens when a government gets corrupted by big business in a bi-partisan manner.

Based upon the way this government has conducted itself so far, I am not optimistic that they will provide correct leadership to solve the problems that are about to descend upon us.

On a different note: What was your opinion of John Koskinen's recent comments at a National Press Club appearance regarding the embeddeds memo he recently made?

Again, thank you Paula for sharing this important information.

-- R.C. (racambab@mailcity.com), December 21, 1999.


R.C.:

In July 1998, Capers Jones presented his study on Y2K and the introduction of the Euro to the US Government. Jones concluded in July 1998, based on the start date of remediation, that the Best Case scenario for the USA was 85% successfully remediated.

At the same time Beach Olesin published the Pain Index which is based on failure rising as a function of the number of interconnections in the system.

IMHO, the .gov has been well aware of the full extent of Y2K probabilities since mid-1998 or possibly earlier. General news, public awareness and preparative action, peaked in late February- early-March following .gov hiring of Public Relations firms to "manage the public perception" of Y2K.

I do agree that they faced two options:

a. Declare war on Y2K and commit a full frontal mobilization. OR,

b. Downplay Y2K concnerns to buy time to optimize the fix rates under continuity of normal conditions.

I believe they selected "b" to avoid a panic early in 1999, as the Y2K impacts would be exacerbated by panic and delay getting more fixes done in the time available.

Unfortunately, the .gov were perhaps too good at quieting concern as we are now under prepared for anything greater than a BITR.

-- Bill P (porterwn@one.net), December 21, 1999.


high government officials are no different than average middle class citizens when it comes to y2k. My friends mostly cannot or will not admit that next year their stock funds will not go up and their net worth will not climb. All is well and will remain so. Suggest it may not be and a slight tremor of the neck and a second of eyeglaze over. Then the face returns to normal and the mouth makes noises of how next year will be even better. This from people who cluck over the lack of life planning by their 20 something children. Truly the gods make crazy those they are about to destroy.

-- Noone (Noone@none.co), December 21, 1999.


R.C.

I wanted to respond to a few of the very thoughtful comments you made.

Quoting you:

"My point is that if I, as a layperson inquiring and educating myself in my spare time can figure this out (and I did so way back in time) surely those whose job called for such an understanding should have done so way back then. It is nearly inconceivable to me that those tasked with the issues of Y2K would have not aggressively pursued a thorough education on this important aspect of Y2K. I know if I were in that position of responsibility I would exhaustively research the issue. In my case as a layman, I didn't have to get exhausted to pick up the necessary elements to realize that embeddeds needed special monitoring and that self-reporting was untrustworthy."

Even comments about embedded systems that Bill Gates has made reflect a failure on his part to comprehend the problems that embedded systems failures can pose, including multiple, near simultaneous failures. One must spend at a bare minimum at least ten to twenty minutes with someone who not only knows about the nature and scope of the problem, but who is also able to communicate in understandable terms. If a person with Bill Gates' intelligence has not applied himself to learning about embedded systems, it becomes less of a surprise that persons in the government who had little or no expertise in technology to begin with, would do so. They might not even know what questions to begin to ask beyond: "What's an embedded system?" Indeed, I talked with several people in roles of responsibility in the Federal government in 1998 who have asked me that very question. One, in fact, was in the Office of the Vice President.

As for a sense of responsibility, I certainly agree that one just assumes that surely everyone who serves in government has such a sense of responsibility. It is a tragedy for our country that there are persons serving in government today who do not have such a sense of responsibility.

Quoting you again:

"It is hard for me to conceive that presidential advisors would not have had intelligence personnel extensively research all aspects of embeddeds issues and that the results would have been so benign in their conclusions. IF this is so, then it speaks volumes concerning the levels of incompetence within the Intelligence community. I find that a little hard to believe that they didn't know. I can accept the notion that intelligence sources knew and simply failed to educate top leadership personnel effectively. I can see that political minds would attempt to ignore it because they prefer to focus upon "happy" thoughts and thus ignore vital issues that might confuse and derail their "positive" approach to solutions."

I can personally attest to the fact that there are people who are knowledgeable concerning embedded systems who have either tried or volunteered to "educate" the President and the Vice President concerning the entire Y2K problem, including embedded systems. To my knowledge, such efforts began as long ago as 1995. That would be educators were not successful may say far more about the capacity or interest of the President and the Vice President to grapple with this subject, than it does with the competency of those who tried to educate them.

I personally know that the closest advisors of both the President and the Vice President have been provided materials on the subject of embedded systems since July of 1998. I have no idea whether those materials were ever read, or if read, if they were understood. I know that offers of technical briefings and invitations to panels on embedded systems were given to Presidential advisors beginning in the early summer of 1998 and again in December 1998, as well as several times in early 1999, including as late as May. The specific offers and invitations that I know about were not accepted.

In May of 1999 I learned that the President's Council had asked the National Institute for Standards and Technology to provide the Council with a kind of definitive review of embedded systems issues. Mr. Koskinen was seeking clarification concerning some specific issues. Several of these same issues turned out to be ones that I had brought up with him. Mr. Koskinen released a statement summarizing the concerns discussed at the November 9th meeting. That summary covered many of the issues that I, among others, had raised with him.

Quoting you again:

"In looking at your historical recap (and its very good, I take no issue with your work) I am amazed that leadership didn't call these people into the loop until last month. These people should have been identified by intelligence sources, contacted and brought forward by 1995 at the first Presidential briefing."

The President's Council's decision to seek clarification on embedded systems concerns did not start in November. The November 9th meeting was the culmination of efforts begun in April or May. I would add, however, that such efforts were long overdue even in April or May, let alone November.

Again quoting you:

"I guess the bottom line here Paula is this:

If the top levels of the Administration and Congress were unaware of the embeddeds problems until 1999 then there has been gross dereliction of duty and a complete failure by top leaders including the President. They were unworthy of the responsibilities entrusted to them. This applies equally to members of both parties. This is really a case of gross negligence and incompetence from the top all the way down to the bottom."

Your conclusions are quite similar to my own. Senator Bennett seemed to understand the embedded issue for a time between June and July of 1998 and early in 1999. Then in early 1999 the Senator became convinced by corporate leaders that he spoke with that embedded systems problems were not as great as he had previously been led to believe. For a variety of reasons, including, apparently, the political riskiness of holding onto such an unpopular point of view, Senator Bennett accepted the more sanguine appraisal and at times, seems to have all but declared a premature victory. Meanwhile the President does not seem to have comprehended the problem of embedded systems and the Vice President and his highest level staff seem to understand the significance of embedded systems even less.

Quoting you:

"Frankly, I suspect that the buck passing started long ago. If so, as it seems by Koskinen's statements, then that these people really have known all along but refused to admit it and have been spinning excuses for their failures.

If the gov't is just now finding out the full scope of embedded systems issues it only points to gross negligence on their part. They deserve the blame. IF, however, they knew long ago and even then realized it was hopeless and suppressed that information then we have another instance of gross incompetence and mismanagement in regards to preparations."

From my explorations of relevant background material here, I think that we are looking less at the suppression of information and far more at the failure to both gather the most pertinent information and assess its full significance. One needs to first grasp the significance of information before making a decision to suppress it. They truly did not grasp the significance of the information. True, this is unbelievable. It is nonetheless the conclusion that my information, knowledge, and experience compel me to believe. Quoting you:

"No matter how I look at it. The government is at fault here as much if not more so than industry. It's really the case of what happens when a government gets corrupted by big business in a bi-partisan manner."

I think that the failure to understand the complexities of the threats and challenges that face the nation and the world is more attributable to simple ignorance and lack of effort to try to understand. There has been a widespread failure on the part numerous people in key roles of responsibility to learn about the nature and scope of the technological aspects of the problems facing us. They have not gathered persons around them who have adequate technical expertise. The strategies that they have developed for addressing those problems are consequently less than adequate. The problem definition is simply an inadequate one. Even now the level of comprehension of technological complexities, let along organizational, sociological, psychological, and managerial aspects of the Y2K and embedded systems crisis are greating wanting. Perhaps, if the problem being faced were likely to have only a level 1 or 2 impact on the impact scale, the approach they have advocated would have been appropriate. Perhaps, too, their optimism might have some nearterm justification if there were no embedded systems issues, and if all the remediation and testing of all information systems, mission critical and non-mission critical, in both the public and private sectors were complete and tested. Such a sanguine view, however, would not be justified over the long run since the rest of the world has dealt with the remediation of information systems far less effectively than we have. The impacts that will be felt in other parts of the world will have major ripple effects that will have the most profound impacts on the U.S.

You wrote:

"Based upon the way this government has conducted itself so far, I am not optimistic that they will provide correct leadership to solve the problems that are about to descend upon us."

In Part 5 of my White Paper, I describe some alternatives in the event that the current leadership of the Federal government does not rise to the occasion. There is still an outside chance that they could. It would require incredible integrity to admit their failure to understand the nature and scope of the problem to date. It would require including persons in the decisionmaking process who possessed needed kinds of expertise. It would require applying adequate resources to addressing the set of problems facing us now and for the coming months and years.

You wrote:

"On a different note: What was your opinion of John Koskinen's recent comments at a National Press Club appearance regarding the embeddeds memo he recently made?"

If you are referring to the questions that were asked of Mr. Koskinen on November 10th (during and after the Press Conference), I felt that his comments demonstrated that he had taken a major step in the right direction. However, this new found realization that the embedded problem is far more serious than previously recognized, does not as yet appear to have been incorporated into any plan of action or change in overall strategy. Such a plan of action is going to have to be undertaken sooner or later if potential future impacts are to be significantly prevented or minimized over the coming months and years. There are embedded systems that will continue to be "ticking timebombs" until they are remediated. The high hazard areas have to be addressed proactively and head-on. There are still many opportunities to avoid having to "fix on failure". In fact, such opportunities will continue to exist for months, if not years to come.

A major task right now is that of getting the powers that be to recognize that we are not facing a single period of time when there are apt to be disruptions, unless you consider that the rollover period as a period of time that could last for months or years.

I hope to have a chance to address many of the issues discussed here in Part 7 of my White Paper. I will likely use there as well some of the thoughts developed here.

Best wishes,

-- Paula Gordon (pgordon@erols.com), December 21, 1999.


Paula Gordon:

This seems to be Koskinen's most recent comment on the NIST paper. He wants us to believe that the critical infrastructure remediators were esp. aware of the embedded system problem and that consequently the critical infrastructure should be OK. This seems to contradict Mr Cherry's statement that utilities etc were chiefly vulnerable (not merely chiefly aware and hence fixed). Any thoughts?

15 December 1999 Excerpts: Y2K Czar Koskinen Speaks to National Press Club NATIONAL PRESS CLUB LUNCHEON WITH JOHN KOSKINEN, CHAIRMAN, PRESIDENT'SCOUNCIL ON YEAR 2000 CONVERSIONNATIONAL PRESS CLUB BALLROOMWASHINGTON, D.C.DECEMBER 14, 1999Today

MR. LIPMAN: There's several questions about the Council's recentstatement on embedded systems. Has there been a reinterpretation ofrisks? Have the statements been placed on the Internet? And commentabout pipeline companies saying that they will be fixing on failure.And for those who may be watching on television or listening on theradio who don't understand what embedded systems are, do you want toexplain that also?

MR. KOSKINEN: All right. A multi-faceted question here. We're having asale. We're doing three for one. Embedded chips are basically, if youthink about it, if you have got a hand-held calculator or a VCR oranything that has a digital display has a chip in it that in factproduces that result. And as you think about that, then that's whypeople originally said, Well, gee, if the embedded chips areeverywhere -- they're in microwave ovens -- what if it doesn't work?It became, as I said, the growth industry, the problem, because wemanufactured and distributed over the last several years billions ofthem. Nobody knows exactly how many, but the estimates are 3 to 50billion.And the concern initially was that the chips themselves -- somepercentage of them -- had clocks in the chips -- so that they would beout buried around somewhere, and they would be sensitive to therollover, and there were enough of them that we would never be able tofind them all.A shorthand for those embedded chips is what it really meant was ashift in focus from information processing concerns -- financialsystems, payroll systems, the things that you envision a big mainframecomputer processing or your own computer processing, and it was ashift to focus to operating systems: What happens on the platform,what happens with a power plant, with an oil refinery, with pipelines?Were the operators focused on the fact that if you go into a lot ofmanufacturing or operating places or even ships you see somebodysitting in front of a computer running the place. And what they'redoing, that computer is doing, that control system is doing, isresponding to information coming from all of those chips out therethat are monitoring hot and cold, on and off, whatever else might begoing on.Smart building systems -- you go into a lot of new buildings, you godown to the basement, the janitor is not there with a shovel shovelingcoal. He is sitting there in front of a computer monitoring all thesystems which may be in fact set to go on and off according to thecalendar.Fortunately it turns out that very few chips by themselves containdate-sensitive calculations. But a lot of them are connected tocontrol systems. . . that in the process are sensitive to date-controlsystems. And so people have been working on that for a long, long timeover the last couple of years as people became aware of it.The question is how many of them are at risk, and how many peoplereally understand they are at risk? And so we have been talking about-- the industry has been talking about it. So a few weeks ago insteadof having people -- there are a lot of people abstractly talking aboutit who have never actually done anything with embedded chips. Webrought in industry experts who had been actually dealing with this problem in the electric power industry, the oil industry, the gasindustry, as well as several experts from companies who have beenactually working on embedded chips and government experts, to sit downand say, Okay, let's review the bidding: What's been done? What arethe issues that we know and need to in fact focus on? And theconsensus was that the major infrastructure industries in this countryhave in fact as a general matter understood the issue about chips andunderstood the need to test for them. They have understood that thefact that one device looks like it's compliant doesn't mean that thenext one that look exactly like it is the same, because it may have adifferent either chip configuration or the chips may have been made bysomeone else, or it may be connected in a different way.What grew out of that meeting was all of this information is up on theWeb, and we sent it around to industry groups to all of the people wedeal with, including the civic action and active groups who have beeninterested in this issue. The consensus was that if there was a riskit's probably not in the critical infrastructure industries but it'sin industries and manufacturing systems that are complicated -- simplesystems don't have these issues -- complicated systems where eventhough it doesn't look like it's a date-sensitive process, there maybe in that process, particularly in the software, but in the controlsystems, date sensitivities that you don't see. So everybody with acomplicated mission-critical system should be testing it for datesensitivities. And that's why we put that out. The example given byone of the testers was looking at five bottling plants -- not exactlya critical infrastructure industry but important in a lot of ways --they tested and rolled the clocks forward looking for these controlissues. And four of the plants worked just fine. But in the fifthplant the sterilizing system stopped when you rolled everythingforward. Nobody could quite figure out where in that complicatedsystem was the issue and why, but it was clear that people needed tobe aware of the fact that if you're running a complicated operatingsystem you need to make sure you've done the testing. But again Iwould stress that the consensus around the room was that the electricpower, oil, gas industries all were aware of that and of the need tocontinue to test.The other issue that grew out of that is an issue that we have beentalking about for some time, and that is that manufacturers andvendors to their credit continue to update the status of Y2Kcompliance of their devices. And it's not that they were lying orcheating the first time out; it is that as we have had continualtesting people have discovered that a device may work perfectly fine80 percent of the time, but if you put it in a certain configurationit no longer works and needs an additional patch.And some software or hardware devices are into their third, fourth, orfifth patches or upgrades. And the concern we have, and it grew out ofthat meeting as well, but we have talked about it for the last sixmonths, is that when you think you're done you are never done, thatbasically our concern is anybody who finished in June and said, "Now Ican go on to other things," and didn't keep monitoring what are theupgrades, what's the updated information, may find that their systemworks just fine except for the fact in certain situations it is nownot going to work because they don't have the last patch.So, again, the major industries represented said they all knew thatthey had people who are monitoring a regular basis what goes on. Butour concern was to send a message to people -- and we talked about itin the federal government -- but outside the government that even whenyou think you're done, and if you have got manufacturer statementsabout Y2K compliance, make sure you keep monitoring that because theirmanufacturer may put out another statement saying you need anotherupgrade, or we have discovered more information about it.All of that information has been around. As I say, the majorindustries were not either surprised -- they were actuallyparticipants in trying to figure out what the lessons learned are. Sothere is not a reinterpretation. We don't think that the problem -- infact, the consensus was bigger than people thought as we have beenmoving through it. But it was that there are these important issuesthat ought to be focused on as we go forward, particularly inindustries which may not have paid the same amount of in- depthattention to the problem that the major infrastructure industries have.

-- Paul (Paul@bina.com), December 22, 1999.


To the Top

-- top (top@top.top), December 22, 1999.

For the last 12 months, the federal government and most individual states have passed some form of legislation limiting or completely excluding lawsuits against both government and industry based on damages caused by Y2K-related failures.

To me this is a very strong clue that while the public facade was "don't worry, be happy", in fact, many of the more influencial people in government and industry had a pretty good idea more than a year ago of the problems that might arise due to a variety of Y2K-related problems. While there might not have been a full understanding of embedded systems at the upper levels of our government, clearly many felt the risk of problems serious enough to pass such legislation.

There are a couple of ironies here. First, I would have supported such legislation had it be part of an larger overall plan designed to inform and protect the American people. But it was not. It was clearly meant to protect government and industry from their own negligence and to hide such negligence from the public. Such an arrogant and self-serving approach I cannot and will not support.

Second, have we not now painted ourselves into a corner with such legislation? If a problem occurs at your company, do you (1) claim it was NOT a Y2K problem and thereby open yourself to litigation, or (2) claim it WAS a Y2K problem and thereby protect your company from litigation but in so doing, open yourself to a loss of confidence that could also spell big trouble company? Heads, I win, tails you lose.

For the time being, it seems that option (1) is winning out. Once the lawyers get involved however, things may not be quite as simple.

-- Arnie Rimmer (Arnie_Rimmer@usa.net), December 22, 1999.


Paul,

You wrote:

"This seems to be Koskinen's most recent comment on the NIST paper. He wants us to believe that the critical infrastructure remediators were esp. aware of the embedded system problem and that consequently the critical infrastructure should be OK. This seems to contradict Mr Cherry's statement that utilities etc were chiefly vulnerable (not merely chiefly aware and hence fixed). Any thoughts? ......"

The short response: I think you are absolutely right. Mr. Koskinen's assessment is in contradiction to that of Michael Cherry's and Chip Schaffer, both of who were among the experts at the November 9 meeting. For whatever reason, Mr. Koskinen is either not recognizing the contraction or simply not aware of it.

Regarding Mr. Koskinen's comments concerning internal Federal agency efforts to assess, remediate, and test their own embedded systems: In March, as I have noted elsewhere, I became aware of some problems wth the protocols that the General Service Administration was using to assess, remediate, and test embedded systems in Federal buildings. These protocol problems included some of the same issues that were eventually raised at the November 9, 1999 meeting. To my knowledge there was no intention on the part of GSA as of March to go back and make sure that all embedded systems were properly assessed, remediated and tested. Not only had they used short cut methods, known to result in some failures, but not all of the systems that should have been assessed were assessed.

I brought these matters to Mr. Koskinen's attention in March. Apparently he did not become convinced that these problems were indeed problems until November. His first public statements concernin the issues came out after that meeting.

With regard to vendor certification and type testing approaches, it seems unlikely that GSA would have gone back and applied more sound protocols. Indeed, in March I was told by an official at GSA and another official at GAO, that it would be too costly and too time consuming to do so.

The technicians and operations manager of the building I toured in March were also unaware of the need to test non-date sensitive embedded systems that were connected with systems that were date senstive. I do not know if Mr. Koskinen recognized this problem as a problem prior to November 1999. I have no idea if GSA has.

For these reasons, I am not at all certain that Federal agencies have in fact been operating on the basis of the kinds of sound approaches that were identified during the November 9 meeting. If they have been, it is for a very short period of time.

It is of even greater concern that the North American Electric Reliability Council recommended short cut measures to assessment and testing in the April report. (See page 35 of the NERC April 30, 1999 quarterly report).

People working on the front lines, such as Michael Cherry and Chip Schaffer (both participants in the November 9 meeting) who know for a fact which of these short cut measures have been and still are being used. They also have some sense of the extent to which those in field know about the need to test non-date sensitive systems that are integrated with systems that are date sensitive. Their assessments seem to me to be at odds in many ways with statements made by Mr. Koskinen.

-- Paula Gordon (pgordon@erols.com), December 22, 1999.


Paula,

I am so glad that you stated:

"The short response: I think you are absolutely right. Mr. Koskinen's assessment is in contradiction to that of Michael Cherry's and Chip Schaffer, both of who were among the experts at the November 9 meeting. For whatever reason, Mr. Koskinen is either not recognizing the contraction or simply not aware of it."

"People working on the front lines, such as Michael Cherry and Chip Schaffer (both participants in the November 9 meeting) who know for a fact which of these short cut measures have been and still are being used. They also have some sense of the extent to which those in field know about the need to test non-date sensitive systems that are integrated with systems that are date sensitive. Their assessments seem to me to be at odds in many ways with statements made by Mr. Koskinen."

That was certainly my take on it. I was beginning to wonder if I you or others were thinking the same thing. The Dec 15 Nat'l Press Club comments by Koskinen just "blew me away" as it seemed he was back- tracking away from his own memo and the NIST Report. But I wanted to hear it from others a little closer to the action than myself. This quote is a definite "keeper" for the archives.

-- R.C. (racambab@mailcity.com), December 23, 1999.


Moderation questions? read the FAQ